Information processing device, information processing method, and program

Abstract

To obtain a parallax image for achieving stereoscopic vision while reducing the possibility of causing discomfort.SOLUTION: Correspondence data that links corresponding pixels in a pair of images that constitute parallax images for achieving stereoscopic vision is generated. The vertical displacement of corresponding pixels in the pair of images is corrected on the basis of the generated correspondence data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a technique for obtaining parallax images for realizing stereoscopic vision. [Background technology]

[0002] It is important to suppress vertical deviation in the left-eye image and right-eye image constituting the parallax image for realizing stereoscopic vision in order to suppress the user's eye fatigue, inability to fuse images, etc. In this regard, in Patent Document 1, the amount of deviation between the images is calculated using the left-eye image and right-eye image obtained by capturing an oblique line, and the entire image is rotated or translated based on the calculation result to suppress the vertical deviation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5297899 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, head-mounted displays that are worn on the user's head and directly view images displayed in front of the user's eyes have been widely used. For example, head-mounted displays may display parallax images generated by performing geometric transformation of images or mapping to a mesh on input images obtained by imaging using a parallax imaging device equipped with two fisheye lenses. Even if the same subject is included in the left-eye image and the right-eye image that constitute the input image, the image is recorded as a light beam with a different incidence angle depending on the distance to the subject and the angle of the subject relative to the optical axis direction of the camera, causing a vertical position shift. In this case, even if the entire image is rotated or translated by applying Patent Document 1, if the vertical shift amount differs depending on the position of the pixel in the image, it cannot be appropriately corrected, and there is a possibility that an uncomfortable feeling will be caused when viewing an image displayed on a head-mounted display or the like. [Means for solving the problem]

[0005] An information processing device according to one aspect of the present disclosure is characterized in having an acquisition means for acquiring a pair of images that constitute a parallax image for realizing stereoscopic vision, a matching means for generating correspondence data that links corresponding pixels in the pair of images that constitute the parallax image acquired by the acquisition means, and a correction means for correcting vertical misalignment between corresponding pixels in the pair of images based on the generated correspondence data. Effect of the Invention

[0006] According to the technology of the present disclosure, it is possible to obtain parallax images for realizing stereoscopic vision with reduced possibility of causing discomfort. [Brief description of the drawings]

[0007] [Figure 1] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. [Diagram 2] 2 is a block diagram showing an example of a functional configuration of an information processing device; [Diagram 3] 4 is a flowchart showing a flow of processing executed by the information processing device. [Figure 4] FIG. 13 is a diagram showing an example of a 180° equirectangular parallax image. [Diagram 5] 1A and 1B are diagrams illustrating a configuration of an imaging device and an example of an imaging scene. [Figure 6] 1A and 1B are diagrams for explaining the cause of deviation in the elevation angle in 180° parallax images. [Figure 7] FIG. 13 is a diagram illustrating an example of correction weights. [Figure 8] FIG. 13 is a diagram showing an example of a 180° equirectangular parallax image after correction. [Figure 9] 2 is a block diagram showing an example of a functional configuration of an information processing device; [Figure 10] 4 is a flowchart showing a flow of processing executed by the information processing device. [Figure 11]1A and 1B are diagrams illustrating an example of the relationship between a fisheye image with parallax and a mesh for projection. [Figure 12] FIG. 13 is a diagram showing the coordinate system of the UV coordinates of a mesh. [Figure 13] FIG. 13 is a diagram showing an example of mesh data after correction. [Figure 14] FIG. 11 is a diagram illustrating an example of a correction amount table. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, embodiments for carrying out the technology of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the scope of the claims, and all of the combinations of features described in the embodiments are not necessarily essential to the solution of the technology of the present disclosure.

[0009] <<Embodiment 1>> 1 is a diagram showing an example of the hardware configuration of an information processing device according to this embodiment. The information processing device 100 has a CPU 101, a RAM 102, a ROM 103, an HDD interface (hereinafter, the interface is referred to as "I / F") 104, an input I / F 106, an output I / F 108, and a network I / F 110. These are connected to a system bus 112 so as to be able to transmit and receive data. The HDD I / F 104 is connected to a hard disk drive (HDD) 105. The input I / F 106 is connected to an input device 107. The output I / F 108 is connected to an output device 109. The network I / F 110 is connected to an external server 111.

[0010] The CPU 101 executes programs stored in the ROM 103 and the HDD 105 using the RAM 102 as a work memory, and controls the operation of each block described later via the system bus 112. The HDD I / F 104 connects a secondary storage device such as an optical disk drive in addition to the HDD 105. The HDD I / F 104 is, for example, an I / F such as a serial ATA (SATA). The CPU 101 can read data from the HDD 105 and write data to the HDD 105 via the HDD I / F 104. Furthermore, the CPU 101 can load data stored in the HDD 105 onto the RAM 102, and conversely, can also save data loaded onto the RAM 102 onto the HDD 105. The CPU 101 can execute the data loaded onto the RAM 102 as a program. The input I / F 106 connects to an input device 107 such as a keyboard, a mouse, a digital camera, or a scanner. The input I / F 106 is, for example, a serial bus I / F such as a USB or IEEE1394. The CPU 101 can read data from the input device 107 via the input I / F 106. The output I / F 108 connects the information processing device 100 to a head mounted display, which is an output device, that is worn on the user's head. The output I / F 108 is, for example, a video output I / F such as DVI or HDMI (registered trademark). The CPU 101 can send data to the head mounted display via the output I / F 108 and display a predetermined video on the head mounted display.

[0011] (Functional configuration of information processing device) 2 is a block diagram showing an example of a functional configuration of the information processing device 100. The information processing device 100 of this embodiment generates a corrected parallax image (corrected image) in which the vertical shift between corresponding pixels is suppressed, from a parallax image represented by input image data.

[0012] The information processing device 100 of this embodiment has an image data acquisition unit 201, an image processing unit 202, and an image output unit 203. The image processing unit 202 has an image correspondence calculation unit 211, a shift amount calculation unit 212, a correction weight calculation unit 213, a correction amount calculation unit 214, and an image correction unit 215.

[0013] In this embodiment, the image processing unit 202 generates association data linking corresponding pixels in a pair of images constituting a parallax image represented by image data acquired by the image data acquisition unit 201. Next, the image processing unit 202 calculates the vertical shift amount between corresponding pixels in the pair of images based on the generated association data. Next, the image processing unit 202 corrects each pixel in the pair of images based on the calculation result so as to suppress the vertical shift between corresponding pixels in the pair of images. By outputting a display image generated based on a corrected parallax image composed of the pair of images after correction to a head mounted display, the following effects can be obtained. That is, a user wearing a head mounted display can see an image that is less likely to cause discomfort compared to the parallax image before correction.

[0014] In the image processing unit 202, first, the image correspondence calculation unit 211 performs correspondence between pixels in a pair of left and right images that constitute a parallax image, and generates correspondence data that links corresponding pixels. That is, the image correspondence calculation unit 211 performs correspondence between pixels in each of the left-eye image and the right-eye image that constitute a parallax image, and calculates correspondence information that indicates the correspondence result. The shift amount calculation unit 212 calculates the vertical shift amount between corresponding pixels in the pair of images based on the correspondence data generated by the image correspondence calculation unit 211.

[0015] The correction weight calculation unit 213 calculates a correction weight based on the incident angle of the captured light ray. The correction amount calculation unit 214 calculates a correction amount for each pixel of the image based on the vertical shift amount calculated by the shift amount calculation unit 212 and the correction weight calculated by the correction weight calculation unit 213. The image correction unit 215 performs correction for each pixel in the paired images so as to suppress vertical shift between corresponding pixels in the paired images based on the correction amount for each pixel in the paired images, and generates a corrected parallax image (corrected image).

[0016] Then, the image output unit 203 outputs the corrected parallax image generated by the image processing unit 202 to the HDD 105 or the like as a file.

[0017] (Corrected parallax image generation process) Fig. 3 is a flowchart showing the flow of the process of generating a corrected parallax image. The functions of each block and each step in Fig. 2 and Fig. 3 are realized by CPU 101 reading program code stored in a storage area such as ROM 103 into RAM 102 and executing it. Alternatively, some or all of the functions of the blocks and steps in Fig. 2 and Fig. 3 may be implemented by hardware such as ASIC or electronic circuitry. Each symbol S below means a step in the flowchart.

[0018] In S301, the image data acquisition unit 201 acquires image data from the HDD 105 or the input device 107. The image data is image data representing a left-eye image and a right-eye image having a parallax between each other, for example, image data representing a 180° equirectangular image (hereinafter referred to as a 180° equirectangular parallax image). That is, the left-eye image and the right-eye image can be said to be a pair of images constituting a parallax image. Note that the image data is not limited to a 180° equirectangular parallax image, and may be image data representing a stereo image, a panoramic image (wide-angle image), or the like. The acquired image data is sent to the image correspondence calculation unit 211.

[0019] Hereinafter, an example will be described in which a 180° equirectangular parallax image is acquired as the image data. The 180° equirectangular parallax image is an image obtained by converting two images (a left-eye image and a right-eye image) with parallax captured by a parallax imaging device equipped with two fisheye lenses into an equirectangular image.

[0020] (180° equirectangular parallax image) 4 is a diagram showing an example of a 180° equirectangular parallax image. Image data showing the 180° equirectangular parallax image shown in FIG. 4 is image data that stores colors representing incident light rays from a direction of 180° horizontally and 180° vertically from a certain reference position. In this embodiment, a 180° equirectangular image for the left eye (hereinafter also referred to as an image for the left eye) is generated by I L The 180° equirectangular image for the right eye (hereinafter also referred to as the right-eye image) is I R In this embodiment, the left eye and right eye images I L ,I R is treated as an input disparity image. In FIG. 4, the left-eye and right-eye images I L ,I R As shown in FIG. 1, the left-eye image I L and right eye image I R When the same subject is displayed at the same elevation angle on both the left and right eyes, the subject can be viewed on the head mounted display without any vertical deviation on the left and right eyes. L and right eye image I R If the same subject is shown in different vertical positions (positions with different elevation angles) in the two images, it may be difficult to fuse the images or a sense of incongruity may occur. Note that since the vertical axis in equirectangular projection represents the elevation angle, it can also be said that the vertical position shift represents the shift in the elevation angle of the incident light rays.

[0021] (Configuration of imaging device and imaging scene) Left-eye and right-eye images I shown in Fig. 4 L ,I RThe following describes the configuration of an imaging device and an example of an imaging scene with reference to FIG. 5. As shown in FIG. 5, left-eye and right-eye images I L ,I R The imaging device that captures the above images is a stereo camera (parallax imaging device) equipped with two imaging systems, and is a stereo fisheye camera 510 equipped with two fisheye lenses 511 and 512 arranged in the left-right direction. When imaging is performed with such a stereo fisheye camera 510, as in FIG. 4, a subject 501 located in front of the stereo fisheye camera 510 is imaged with a small deviation in elevation angle by the left and right fisheye lenses 511 and 512 of the stereo fisheye camera 510. In contrast, the larger the absolute value of the azimuth angle of a subject, such as a subject 502 located away from the front of the stereo fisheye camera 510, the larger the deviation in elevation angle of the subject is when the subject is imaged with the left and right fisheye lenses 511 and 512 of the stereo fisheye camera 510. In other words, the further away in azimuth angle is from the center of the left and right fisheye lenses 511 and 512 of the stereo fisheye camera 510, the larger the deviation in elevation angle of the same subject is when the subject is imaged.

[0022] (Cause of elevation angle deviation in 180° equirectangular parallax images) Fig. 6 is a diagram for explaining the cause of the deviation of the elevation angle in a 180° equirectangular parallax image. Fig. 6(a) shows a state in which a stereo camera captures an image of a subject, and Fig. 6(b) shows a state in which a person views the subject. It is assumed that the positional relationship between the stereo camera and the subject shown in Fig. 6(a) is the same as the positional relationship between the person and the subject shown in Fig. 6(b). It is assumed that a person views a subject from the front, which is a desirable state when viewing the subject.

[0023] In FIG. 6A, the optical axis directions 621 and 622 of the left and right lenses 611 and 612 of a stereo camera (parallax imaging device) 610 are taken as references. L , θ RFor example, suppose that a rectangular parallelepiped object 630 is located at a position where the absolute value of azimuth becomes large. In this case, the difference between the distance from the left lens 611 to the position 631 of the vertex A of the object 630 and the distance from the right lens 612 to the position 631 of the object 630 tends to become large. Therefore, in a region where the absolute value of the azimuth angle becomes large in the 180° equirectangular parallax image, the deviation of the elevation angle becomes large. In addition, the deviation of the elevation angle also changes depending on the distance from the stereo camera (parallax imaging device) to the object. In other words, the closer the distance from the stereo camera to the object, the larger the deviation of the elevation angle, but the farther the distance from the stereo camera to the object, the smaller the deviation of the elevation angle.

[0024] 6(b), it is desirable that a person 640 looks at a subject 650 from the front, and there is no difference between the distance from the left eye 641 of the person 640 to a position 651 of the vertex A of the subject 650, and the distance from the right eye 642 of the person 640 to the position 651 of the subject 650. In other words, when the person 640 looks at the subject 650, the elevation angle φ' L and an elevation angle φ′ when the right eye 642 of the person 640 sees the position 651 of the subject 650. R It is desirable that the above should be the same.

[0025] A user wearing a head-mounted display sees a left-eye image with his / her left eye and a right-eye image with his / her right eye, so that the same subject in each image is seen from the front. Therefore, in the left-eye image and the right-eye image, it is desirable that the corresponding elevation angles when the position P of the subject is seen by the left eye and the right eye of the user are the same, as in the case of FIG. 6(b). On the other hand, if the elevation angles of the pixels corresponding to the left-eye image and the right-eye image do not match when the subject is seen, it causes difficulty in fusing the images when the images are seen, which causes discomfort when the images are seen. Therefore, in this embodiment, in order to create a desirable state when a person sees the subject, the deviation of the elevation angle of the incident light between the pixels corresponding to the left-eye image and the right-eye image is corrected. This reduces the difficulty in fusing the images when the images are seen, and reduces the possibility of discomfort when the images are seen.

[0026] In S302, the image correspondence calculation unit 211 calculates the left-eye and right-eye images I L ,I R Based on this, the unit 212 generates association data (association information) indicating a correspondence relationship linking a pixel of the left eye image with a pixel of the right eye image that corresponds to the pixel of the left eye image. The generated association data (association information) is sent to the displacement amount calculation unit 212.

[0027] In this embodiment, as a method of associating pixels, a method of associating pixels of a left-eye image with pixels of a right-eye image that correspond to the pixels of the left-eye image by stereo matching is adopted. R For each pixel in the left eye image I L In the right eye image I R In the left eye image I, the coordinates (x, y) of the pixels corresponding to the coordinates (x', y') of the pixels are calculated. L Based on the coordinates of each pixel in the right eye image I R The coordinates of the corresponding pixels in the right-eye image I R Pixel P ini Let A be an N×M pixel area centered on the coordinates (x, y) of (i=1, 2, …, w×h). R In this area A R is used as a template. Left eye image I L Scan the whole area and find area A. R The similarity between the two regions is calculated, and the region with the highest similarity is A. L The center pixel of the corresponding pixel is defined as (x', y'). Here, the width of the equirectangular image is w and its height is h. R The left eye image I is matched with all pixels in the left eye image I, and the corresponding coordinates are calculated. The similarity evaluation index may be any index that can evaluate the similarity between regions, such as SSD (Sum of Squared Difference) or SAD (Sum of Absolute Difference). L Although the case of scanning the entire image has been described, the present invention is not limited to this. For example, the scanning range is set to the left eye image I L Alternatively, only the epipolar line of the pixel may be used. In this case, calculation can be performed quickly and with high accuracy. Although the correspondence by stereo matching has been described, the present invention is not limited to this. For example, feature points may be detected and correspondingly performed, and pixels may be correspondingly performed based on the result of the feature point correspondence. For pixels for which no feature point is detected, the corresponding coordinates may be calculated by interpolating from the result of the correspondence of surrounding feature points.

[0028] In S303, the displacement amount calculation unit 212 calculates the vertical displacement amount of the right-eye image based on the association data acquired in S302. The calculated displacement amount is output to the correction amount calculation unit 214. R Pixel P i For the coordinates (x,y) of the left eye image I L The shift amount d of the corresponding pixel coordinates (x', y') in i is calculated as the difference between corresponding coordinates. That is, d i = y' - y.

[0029] In S304, the correction weight calculation unit 213 calculates the correction weight of each pixel of the 180° equirectangular image. The calculated correction weight of each pixel is output to the correction amount calculation unit 214. The correction weight of each pixel is calculated according to the angle of the incident light recorded in the pixel from the positional relationship of the left and right imaging devices (lenses) provided in the parallax imaging device. That is, the correction weight is calculated to be smaller in the front direction where the absolute value of the azimuth angle is small than in the case of directions other than the front direction, and is calculated to be larger than in the case of the front direction when the absolute value of the azimuth angle is large. For example, the front direction has an azimuth angle θ in the range of -π / 4 to π / 4. This makes it possible to reduce the influence of the calculation error of the amount of deviation for a frontal subject where the deviation of the elevation angle is small and the effect of correction is small. Also, in an area where the absolute value of the azimuth angle is large and the effect of correction is large, the effect of correction can be obtained by increasing the weight.

[0030] In addition, in the elevation angle direction, the weight of the correction is calculated so that the weight is larger in areas where the absolute value of the elevation angle is small compared to other areas, and the weight is smaller in areas where the absolute value of the elevation angle is large compared to other areas. In areas near the poles of the zenith and nadir, which are areas where the absolute value of the elevation angle is large, there is a possibility that all azimuth angle areas will be included in the display range when the image is viewed, so the weight of the correction is calculated to be small to prevent deviation due to correction.

[0031] Right eye image pixel P i The weight of correction w i is pixel P i The azimuth angle θ of the incident ray i The weight function w1(θ i ) and pixel P i The elevation angle φ of the incident ray i The weight function w2(φ i ) is calculated as the product of w i =w1(θ i )×w2(φ i The X-axis, which is the horizontal axis of the equirectangular image (image for the right eye), corresponds to the azimuth angle of the incident light, and the Y-axis, which is the vertical axis of the equirectangular image (image for the right eye), corresponds to the elevation angle.i Azimuth angle θ i is θ i = π / 2 × (xw / 2), and the pixel P i Elevation angle φ i is φ i =-π / 2×(yh / 2).

[0032] (Calculated correction weight for 180° equirectangular image) FIG. 7(a) is a diagram for explaining the correction weight in the calculated 180° equirectangular image. In FIG. 7(a), the brightness of the color represents the magnitude of the correction weight, and the darker the color, the greater the correction weight, and the brighter the color, the smaller the correction weight. The maximum value of the correction weight is a numerical value smaller than 1, and the minimum value of the correction weight is a numerical value larger than 0. As shown in FIG. 7(a), in the region where the absolute value of the elevation angle φ is small and the absolute value of the azimuth angle θ is large, a large value is calculated as the correction weight, and strong correction is performed. In the front direction and polar regions, a small value is calculated as the correction weight, and weak correction is performed.

[0033] (Weight function) Fig. 7(b) is a diagram showing an example of a weighting function w1(θ) for an azimuth angle θ. Fig. 7(c) is a diagram showing an example of a weighting function w2(φ) for an elevation angle φ. The weighting functions w1 and w2 are expressed as follows using a sigmoid function.

[0034]

number

[0035] By adjusting the correction amount using this weight, it is possible to reduce the influence of a mistake in pixel correspondence between the left-eye image and the right-eye image, while reducing the possibility of a sense of incongruity caused by a vertical shift (vertical parallax shift) when viewing these images. Also, an important area in an image, such as a face area, may be detected, and a weight that is a numerical value larger than the weight calculated using the weight function may be set for that area. As a method for detecting a face area, for example, an existing image recognition method may be used. Specifically, a method may be used in which a face detector that has learned face images in a neural network is used to detect a face area that indicates the position and size of a face included in an image. Alternatively, a method may be used in which a face area that indicates the position and size of an image included in an image is detected by a matching process using a template of a face image. This makes it possible to strongly correct the deviation in the elevation angle of an important area. In order to prevent image deviation due to a sudden change in the correction amount, when the weight of a part of the area is set large, the magnitude of the weight in the surrounding area of ​​the detection area may be adjusted so that the weight changes smoothly between the detection area and the surrounding area of ​​the detection area.

[0036] In S305, the correction amount calculation unit 214 calculates the right-eye image I R Based on the shift amount of each pixel and the correction weight of each pixel, the right eye image I R The correction amount for each pixel of the right eye image I is calculated. The calculated correction amount for each pixel is sent to the image correction unit 215. R Pixel P i Correction amount c for (x,y) i is the deviation of the corresponding coordinates d i and the correction weight w i It is calculated by multiplying with c i =d i ×w i It is calculated as:

[0037] In S306, the image correction unit 215 corrects the right-eye image I R Based on the correction amount of each pixel, the corrected right-eye image I R Then, the left eye image I L and the generated corrected right-eye image I R' is sent to the image output unit 203.

[0038] Corrected right-eye image I R Each pixel P i The value of ' is the right-eye image I R Pixel P i From the correction amount c i The pixel values ​​are obtained by sampling the pixel values ​​shifted in the Y-axis direction by 1. For example, an interpolation method such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation may be used for the interpolation of the pixel values ​​when sampling the pixel values, but is not limited to these.

[0039] (180° equirectangular parallax image after correction) FIG. 8 is a diagram showing an example of a corrected 180° equirectangular parallax image obtained by correcting the vertical shift of the 180° equirectangular parallax image shown in FIG. 4. When the subject 403 and 404 shown in FIG. 4 has different elevation angles between the left-eye image and the right-eye image, the correction of this embodiment reduces the deviation of the elevation angle between the left-eye image and the right-eye image, as shown in FIG. 8 for the subjects 803 and 804. In addition, the correction amount is small for the subjects 801 and 802 in front, and no deviation due to the correction occurs. In this way, by performing correction based on the weight according to the shift amount for each pixel and the angle of the incident light, it is possible to obtain an image in which the difficulty of fusing the images when viewing these images is reduced and the possibility of causing discomfort when viewing these images is reduced. In addition, since the images can be fused even at a place where the absolute value of the azimuth angle is large and the distance of the subject is close, the imaging range in which the possibility of causing discomfort is reduced can be expanded. In this embodiment, the aspect of correcting only the right-eye image has been described, but the present invention is not limited to this. For example, the amount of correction may be divided in half to correct both the right-eye image and the left-eye image paired with the right-eye image, or only the left-eye image paired with the right-eye image may be corrected without correcting the right-eye image.

[0040] In S307, the image output unit 203 outputs the left eye image I L and the corrected right-eye image I RBy outputting the image to the HDD 105 as an image file in this manner, the image can be used as an image for a predetermined display device such as a head mounted display.

[0041] As described above, according to this embodiment, it is possible to generate an image in which the deviation of the elevation angle of the 180° equirectangular parallax image is corrected. Therefore, even if the orientation of the parallax imaging device and the direction in which the user views the image are deviated by a certain angle, it is possible to reduce the vertical deviation of the subject in the left eye display image and the right eye display image displayed on the head mounted display. As a result, it is possible to obtain an image that is less likely to cause discomfort when viewed on a specific display device such as a head mounted display that allows the user to perceive a stereoscopic image.

[0042] <<Embodiment 2>> In this embodiment, a parallax image and mesh data for displaying the parallax image are input, and the UV coordinates of the mesh data are corrected based on the parallax image to reduce the vertical deviation of the displayed image when the image is viewed. In this embodiment, the differences from the first embodiment will be mainly described.

[0043] The information processing device 100 according to this embodiment has the same hardware configuration as the information processing device according to the first embodiment shown in FIG. 1, and a detailed description thereof will be omitted.

[0044] The mesh data is data used when displaying a parallax image on a predetermined display device such as a head-mounted display, which is an output device. The mesh data has information for three-dimensionally arranging the mesh data in a three-dimensional computer graphics (3DCG) space so that the angle between the origin of the three-dimensional space and each vertex of the mesh data is the angle of the incident light recorded in each pixel of the parallax image. The parallax image is mapped onto the mesh data as a texture, arranged in the 3DCG space, and rendered based on a virtual viewpoint (virtual camera), thereby generating an image to be displayed on an output device (display). When displaying on a head-mounted display, a conversion process is further performed on the rendered image, taking into account the characteristics of the device, such as distortion of the eyepiece lens. That is, in this embodiment, mesh data is generated to reduce the vertical deviation between the image for the left eye and the image for the right eye that occurs in the display image. In addition, in this embodiment, a fisheye image obtained by imaging using a parallax imaging device equipped with two fisheye lenses (parallax fisheye lenses) as the parallax image will be described as an example, but is not limited to this. For example, images other than fisheye images, such as stereo images and panoramic images (wide-angle images), may be used.

[0045] (Functional configuration of information processing device) 9 is a block diagram showing an example of a functional configuration of the information processing device 100 according to this embodiment. The information processing device 100 according to this embodiment corrects the UV coordinates of the mesh data so as to suppress deviation in the vertical direction of the display image based on the input parallax image and the input mesh data.

[0046] The information processing device 100 of this embodiment has an image data acquisition unit 901, a mesh data acquisition unit 902, an information processing unit 903, and an output unit 904. The information processing unit 903 has an image correspondence calculation unit 911, a shift amount calculation unit 912, a correction weight calculation unit 913, a correction amount calculation unit 914, and a UV coordinate correction unit 915.

[0047] In this embodiment, the information processing unit 903 corrects the UV coordinates of the mesh data so as to suppress a vertical shift of the display image based on the parallax image, which is image data acquired by the image data acquisition unit 901, and the mesh data acquired by the mesh data acquisition unit 902. By outputting a display image generated based on the corrected mesh data and parallax image to the head mounted display, the following effects can be obtained. That is, a user wearing the head mounted display can see an image that is less likely to cause discomfort compared to a case where mesh data before correction is used.

[0048] In the information processing unit 903, first, the image correspondence calculation unit 911 performs correspondence between pixels in a pair of left and right images that constitute the parallax image, and generates correspondence data that links corresponding pixels. That is, the image correspondence calculation unit 911 performs correspondence between pixels in the left eye image and the right eye image that constitute the parallax image, and calculates correspondence information that indicates the correspondence result. The shift amount calculation unit 912 calculates the vertical shift amount of the pixels that are associated with the vertices of the mesh data based on the correspondence data generated by the image correspondence calculation unit 911.

[0049] A correction weight calculation unit 913 calculates a correction weight based on the incident angle of a light ray captured at a pixel associated with a vertex of the mesh data. A correction amount calculation unit 914 calculates a correction amount for the UV coordinates of the vertex based on the vertical shift amount calculated by the shift amount calculation unit 912 and the correction weight calculated by the correction weight calculation unit 913. A UV coordinate correction unit 915 corrects the UV coordinates of the mesh data based on the correction amount for each UV coordinate, and generates corrected mesh data.

[0050] Then, the output unit 904 outputs the parallax image acquired by the image data acquisition unit 901 and the correction mesh data generated by the information processing unit 903 to the HDD 105 or the like as files.

[0051] (Correction mesh data generation process) Fig. 10 is a flowchart showing the flow of the process of generating correction mesh data. The functions of each block and each step in Fig. 9 and Fig. 10 are realized by CPU 101 reading program code stored in a storage area such as ROM 103 into RAM 102 and executing it. Alternatively, some or all of the functions of the blocks and steps in Fig. 9 and Fig. 10 may be implemented by hardware such as ASIC or electronic circuitry.

[0052] In S1001, the image data acquisition unit 901 acquires image data from the HDD 105 or the input device 107. The acquired image data is sent to the image correspondence calculation unit 911. The image data is image data obtained by combining two images (a left-eye image and a right-eye image) having parallax captured by a parallax imaging device equipped with two fisheye lenses in a state of being aligned in the horizontal direction. Hereinafter, an example will be described in which the image data is obtained by combining two images having parallax captured by a parallax imaging device equipped with two fisheye lenses in a state of being aligned in the horizontal direction. Details of fisheye parallax images will be described later with reference to the drawings.

[0053] In S1002, the mesh data acquisition unit 902 acquires mesh data from the HDD 105 or the input device 107. The acquired mesh data is sent to the image correspondence calculation unit 911 and the correction weight calculation unit 913.

[0054] (Fisheye parallax images and mesh data) 11 is a diagram showing an example of a fisheye parallax image and example mesh data according to the present embodiment. A fisheye parallax image 1101 has an image located in the left half and captured by a left imaging device (left lens) of the parallax imaging device, and an image located in the right half and captured by a right imaging device (right lens) of the parallax imaging device.

[0055] Lines 1102 pointing from the vertices of the left-eye mesh and the right-eye mesh to the fisheye parallax image shown in Fig. 11 indicate the correspondence between the vertices and the coordinates of the image based on UV coordinates. Here, the mesh data has three-dimensional coordinates representing the positions of multiple vertices, UV coordinates that associate the vertices with coordinates in the image, and topology information that represents the combination of vertices that constitute a triangle that represents the surface of the mesh. The UV coordinates represent the correspondence between the vertices of the mesh data and the positions of the fisheye parallax image, which is the texture. For example, when the vertex coordinates that represent the three-dimensional position of a vertex of the mesh data are (X, Y, Z) and the corresponding UV coordinates are (u, v), the vertex (X, Y, Z) is associated with one point on the texture indicated by the coordinates (u, v).

[0056] (Texture) 12 is a diagram showing an example of a texture. In this embodiment, the UV coordinates are expressed in a coordinate system in which the horizontal direction is the U axis, the vertical direction is the V axis, the bottom left of the image is the origin (0,0), the right edge coordinates (U,V) are (1,V), and the top edge coordinates (U,V) are (U,1).

[0057] The range that the UV coordinates can take is [0 to 1]. If the same subjects on the left and right have the same vertices at the same elevation angle, such as subjects 1103 and 1104 captured in front of the fisheye parallax image, the image can be viewed without any vertical deviation between the left and right when wearing a head-mounted display. On the other hand, if the left and right vertical positions are different, such as subjects 1105 and 1106 on the right side of the fisheye parallax image, it may be difficult to fuse the images or a sense of discomfort may occur when viewing the image while wearing a head-mounted display.

[0058] In S1003, the image correspondence calculation unit 911 generates correspondence data indicating a correspondence linking pixels of the right-eye image area of ​​the image data with pixels of the left-eye image area that correspond to the pixels of the right-eye image area, based on the image data and the mesh data. Note that the image data is the data acquired in S1001. The mesh data is the data acquired in S1002. The generated correspondence data (correspondence information) is sent to the deviation amount calculation unit 912.

[0059] Similar to the first embodiment, stereo matching is performed between the left and right half regions of the parallax image, and correspondence data (correspondence information) is generated that associates pixels (x, y) in the right half region with pixels (x', y') in the left half region that correspond to the pixels (x', y') in the left half region that correspond to the pixels. Here, the correspondence information used in S1004 and subsequent steps is only pixels that correspond to the vertices of the mesh data for the right eye by UV coordinates, so that the pixels to be matched may be limited to only those that correspond to the vertices, thereby reducing processing costs.

[0060] In S1004, the displacement amount calculation unit 912 calculates the vertical displacement amount of the coordinates of the parallax image corresponding to the vertices of the mesh data based on the association data acquired in S1003. The calculated displacement amount is sent to the correction amount calculation unit 914.

[0061] The amount of deviation d between the coordinates (x, y) of a pixel in the right half of the image and the corresponding coordinates (x', y') of a pixel in the left half of the image v is calculated by converting the difference between the corresponding coordinates into the V coordinate value of the UV coordinate system. That is, d v =-(y'-y) / h, where v is the index of the vertex (v=0,1,…,n r ), where n r is the number of vertices of the right-eye mesh, and h is the height of the fisheye parallax image.

[0062] In S1005, the correction weight calculation unit 913 calculates the correction weight of the V coordinate of each vertex of the mesh. The calculated correction weight is output to the correction amount calculation unit 914.

[0063] In the first embodiment, the angle of the incident light is obtained from the pixels of the equirectangular image to calculate the correction weight. In the present embodiment, however, the azimuth angle θ of the vertex v viewed from the origin is calculated. v , elevation angle θ v The correction weight w is the angle of the incident light. v The weight of the correction based on the azimuth angle and elevation angle of the incident light is calculated using the same weighting function as in the first embodiment.

[0064] In S1006, the correction amount calculation unit 914 calculates the amount of correction of the V coordinate of each vertex of the mesh based on the amount of deviation of the V coordinate of each vertex of the mesh calculated in S1004 and the weight of the correction calculated in S1005. The calculated amount of correction is sent to the UV coordinate correction unit 915. The amount of correction c of the V coordinate corresponding to the vertex of the mesh for the right eye v is the deviation of the corresponding coordinates d v and the correction weight w v It is calculated by multiplying with c v =d v ×w v It is calculated as:

[0065] In S1007, the UV coordinate correction unit 915 generates corrected mesh data based on the correction amount of the V coordinate corresponding to each vertex of the right-eye mesh data. That is, it can be said that the coordinates indicated by the right-eye mesh data are corrected. The generated corrected mesh data is sent to the output unit 904. Note that, by applying the corrected mesh data, the vertical deviation between corresponding pixels in a pair of images constituting a parallax image is corrected.

[0066] The V coordinate value of each vertex of the mesh data for the right eye after correction is calculated by subtracting the correction amount c from the original V coordinate value. v The value is shifted by only

[0067] (Mesh data after correction) FIG. 13 is a diagram showing an example of mesh data after correction. The position of the image referred to by the vertices of the right-eye mesh data is changed by the correction of this embodiment, and the same height of the subject is referred to by the same vertices of the left and right meshes. In this way, by correcting the V coordinate for each vertex based on the weight according to the shift amount of the reference point and the angle of the incident light, it is possible to reduce the difficulty of fusion when viewing the image and the possibility of causing discomfort. In addition, since fusion can be performed even at a place where the absolute value of the azimuth angle is large and the distance of the subject is close, it is possible to expand the imaging range that is less likely to cause discomfort. In this embodiment, the aspect of correcting only the right-eye mesh data has been described, but is not limited to this. For example, both the right-eye mesh data and the left-eye mesh data paired with the right-eye mesh data may be corrected by dividing the correction amount in half. In addition, it is also possible to correct only the left-eye mesh data paired with the right-eye mesh data without correcting the right-eye mesh data.

[0068] In S1008, the output unit 904 outputs the fisheye parallax image and the generated corrected mesh data as a file to the HDD 105. By outputting the data as a file to the HDD 105 in this manner, the data can be used as data for a predetermined display device such as a head mounted display.

[0069] As described above, according to the present embodiment, mesh data for displaying fisheye parallax images after correcting the deviation in the elevation angle can be generated. Therefore, even if the orientation of the parallax imaging device and the direction in which the user views the image are offset by a certain angle, the vertical deviation of the subject in the left eye display image and the right eye display image displayed on the head mounted display can be reduced. As a result, data can be obtained that reduces the possibility of causing discomfort when viewed on a display device that allows the user to perceive a stereoscopic image, such as a head mounted display.

[0070] [Other embodiments] The present disclosure is not limited to the above embodiment, and various embodiments are possible. In the above embodiment, the shift amount of the object is calculated for each pixel, but the present disclosure is not limited to this. For example, the correction amount may be calculated without calculating the shift amount, assuming that all objects are present at the object distance calculated from the focus position at the time of image capture. In addition, the correction amount may be calculated by assuming the object distance from the image capture conditions such as the image capture mode, instead of the focus position.

[0071] In the above, the manner in which the correction amount is calculated for each pixel according to the displacement amount of the object and the weight of the correction has been described, but the present invention is not limited to this. A table storing the correction amount for each angle of the incident light may be prepared in advance, and the correction amount obtained from the correction amount table may be used during correction. FIG. 14 is a diagram showing an example of the relationship between the azimuth angle θ and the elevation angle φ and the weight of the correction. The correction amount table 1400 is a table showing the relationship between the azimuth angle θ and the elevation angle φ 1401 and the weight of the correction 1402. In addition, in the case of the correction amount for the vertices of the mesh, the correction amount of the V coordinate determined from the angle of the vertex from the origin may be obtained and used for the correction. That is, different correction amounts may be set according to the position in the parallax image. In setting the correction amount, the correction amount may be set so that the weight of the correction becomes smaller as the absolute value of the angle of the elevation angle of the imaging device that captures the parallax image recorded in each pixel of the parallax image becomes larger. In addition, in setting the correction amount, the correction amount may be set so that the weight of the correction becomes larger as the absolute value of the angle of the azimuth angle of the imaging device becomes larger.

[0072] In the above, the input is described as an image, but the present invention is not limited to this. When the input is a time-series video image, a predetermined frame may be selected from the video image to match the left and right images. In other words, matching data may be generated that links corresponding pixels in the left and right images (paired images) based on the acquired predetermined frame.

[0073] In the above, the input was an image, but if the input is a time-series video image, a background image may be generated from the video image excluding moving objects that are moving subjects, and the left and right images may be associated using the generated background image. That is, association data may be generated that links corresponding pixels in the left and right images (paired images) based on the generated background image. The background image may be generated by arranging outputs obtained by applying a median filter in the time direction to each pixel of the video image. The background image may also be generated by detecting moving objects in each frame of the video, and calculating the average color of the image in the time direction only in the background area excluding the detected moving object area.

[0074] In the above, the embodiment has been described in which association data linking corresponding pixels in a pair of images constituting a parallax image is generated, and vertical misalignment between corresponding pixels in the pair of images is corrected based on the generated association data, but is not limited thereto. That is, association data linking corresponding regions formed of a plurality of pixels in the pair of images may be generated, and vertical misalignment between corresponding regions in the pair of images may be corrected based on the generated association data.

[0075] In the above, the aspect of generating a parallax image for realizing a stereoscopic view with the correction weighting factored in has been described, but the present invention is not limited thereto. For example, a parallax image for realizing a stereoscopic view without the correction weighting factored in may be generated. In addition, a mesh data for displaying a parallax image without the correction weighting factored in may be generated.

[0076] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0077] The disclosure of this embodiment includes the following configuration examples. (Configuration 1) An acquisition means for acquiring a pair of images constituting a parallax image for realizing a stereoscopic vision; a matching unit that generates matching data that links corresponding pixels in a pair of images that form the parallax image acquired by the acquisition unit; a correction means for correcting a vertical shift between corresponding pixels in the pair of images based on the generated correspondence data; 13. An information processing device comprising:

[0078] (Configuration 2) The correction means corrects one or both of the pair of images. 2. The information processing device according to configuration 1.

[0079] (Configuration 3) acquiring the parallax image and mesh data for displaying the parallax image; The correction means corrects the vertical misalignment between pixels in correspondence with each other in a pair of images constituting the parallax image by correcting the coordinates indicated by the mesh data acquired by the acquisition means based on the generated correspondence data. 2. The information processing device according to configuration 1.

[0080] (Configuration 4) The method further includes a derivation means for deriving a correction weight for each pixel constituting the parallax image, The derivation means derives the correction weight in accordance with an angle of a light ray recorded in each pixel of the parallax image with respect to an optical axis direction of an imaging device that captures the parallax image. 4. The information processing device according to any one of configurations 1 to 3.

[0081] (Configuration 5) The derivation means derives the correction weight that decreases as the absolute value of the elevation angle of the light ray recorded in each pixel of the parallax image increases. 5. The information processing device according to configuration 4.

[0082] (Configuration 6) The derivation means derives the correction weight that increases as the absolute value of the azimuth angle of the light ray recorded in each pixel of the parallax image increases. 6. The information processing device according to configuration 4 or 5.

[0083] (Configuration 7) The deriving means derives the correction weights using a sigmoid function. 6. The information processing device according to configuration 5.

[0084] (Configuration 8) The deriving means derives the correction weights using a sigmoid function. 7. The information processing device according to configuration 6.

[0085] (Configuration 9) The parallax images are time-series images, and the apparatus further includes a generating unit for generating a background image by excluding a moving subject from the parallax images; The association means generates the association data based on the background image generated by the generation means. 9. The information processing device according to any one of configurations 1 to 8.

[0086] (Configuration 10) The acquiring means acquires a predetermined frame when the parallax image is a time-series image, The association means generates the association data based on the predetermined one frame acquired by the acquisition means. 10. The information processing device according to any one of configurations 1 to 9.

[0087] (Configuration 11) A setting unit is further provided for setting a correction amount for each pixel constituting the parallax image, The setting means sets the amount of correction such that the weight of the correction decreases as the absolute value of an elevation angle of an imaging device that captures the parallax image recorded in each pixel of the parallax image increases, and the weight of the correction increases as the absolute value of an azimuth angle of the imaging device increases. 11. The information processing device according to any one of configurations 1 to 10.

[0088] (Configuration 12) The association means comprises: performing a matching process on all pixels in the pair of images constituting the parallax image, and generating the association data based on a result of the matching process; Alternatively, a feature point detection process is performed on the pair of images constituting the parallax image, and the association data is generated based on the feature point detection result. 12. The information processing device according to any one of configurations 1 to 11.

[0089] (Configuration 13) the association means generates the association data linking regions formed of a plurality of corresponding pixels in the pair of images; The correction means corrects a vertical misalignment between corresponding regions in the pair of images based on the generated correspondence data. 13. The information processing device according to any one of configurations 1 to 12.

[0090] (Configuration 14) The parallax image is an image obtained by subjecting an image captured by an imaging means equipped with two fisheye lenses to equirectangular processing. 14. The information processing device according to any one of configurations 1 to 13.

[0091] (Configuration 15) The pair of images includes a left-eye image and a right-eye image that is paired with the left-eye image and has a parallax with the left-eye image. 15. The information processing device according to any one of configurations 1 to 14.

[0092] (Configuration 16) An acquisition step of acquiring a pair of images constituting a parallax image for realizing a stereoscopic vision; a matching step of generating matching data that links corresponding pixels in a pair of images that constitute the parallax image acquired in the acquisition step; a correction step of correcting a vertical shift between corresponding pixels in the pair of images based on the generated correspondence data; 13. An information processing method comprising:

[0093] (Configuration 17) A program for causing a computer to function as each step of the information processing method according to configuration 16.

Claims

1. An acquisition means for acquiring a pair of images constituting a parallax image for realizing a stereoscopic vision; a matching unit that generates matching data that links corresponding pixels in a pair of images that form the parallax image acquired by the acquisition unit; a correction means for correcting a vertical shift between corresponding pixels in the pair of images based on the generated correspondence data; 13. An information processing device comprising:

2. The correction means corrects one or both of the pair of images.

2. The information processing apparatus according to claim 1,

3. the acquiring means acquires mesh data for displaying the parallax image, The correction means corrects the vertical misalignment between pixels in correspondence with each other in a pair of images constituting the parallax image by correcting the coordinates indicated by the mesh data acquired by the acquisition means based on the generated correspondence data.

2. The information processing apparatus according to claim 1,

4. The method further includes a derivation means for deriving a correction weight for each pixel constituting the parallax image, The derivation means derives the correction weight in accordance with an angle of a light ray recorded in each pixel of the parallax image with respect to an optical axis direction of an imaging device that captures the parallax image.

2. The information processing apparatus according to claim 1,

5. The derivation means derives the correction weight that decreases as the absolute value of the elevation angle of the light ray recorded in each pixel of the parallax image increases.

5. The information processing apparatus according to claim 4.

6. The derivation means derives the correction weight that increases as the absolute value of the azimuth angle of the light ray recorded in each pixel of the parallax image increases.

5. The information processing apparatus according to claim 4.

7. The deriving means derives the correction weights using a sigmoid function.

6. The information processing apparatus according to claim 5,

8. The deriving means derives the correction weights using a sigmoid function.

7. The information processing apparatus according to claim 6,

9. The parallax images are time-series images, and the apparatus further includes a generating unit for generating a background image by excluding a moving subject from the parallax images; The association means generates the association data based on the background image generated by the generation means.

2. The information processing apparatus according to claim 1,

10. When the parallax image is a time-series image, the acquisition means acquires a predetermined frame, The association means generates the association data based on the predetermined one frame acquired by the acquisition means.

2. The information processing apparatus according to claim 1,

11. A setting unit is further provided for setting a correction amount for each pixel constituting the parallax image, The setting means sets the amount of correction such that the weight of the correction decreases as the absolute value of an elevation angle of an imaging device that captures the parallax image recorded in each pixel of the parallax image increases, and the weight of the correction increases as the absolute value of an azimuth angle of the imaging device increases.

2. The information processing apparatus according to claim 1,

12. The association means comprises: performing a matching process on all pixels in the pair of images constituting the parallax image, and generating the association data based on a result of the matching process; Alternatively, a feature point detection process is performed on the pair of images constituting the parallax image, and the association data is generated based on the feature point detection result.

2. The information processing apparatus according to claim 1,

13. the association means generates the association data linking regions formed of a plurality of corresponding pixels in the pair of images; The correction means corrects a vertical misalignment between corresponding regions in the pair of images based on the generated correspondence data.

2. The information processing apparatus according to claim 1,

14. The parallax image is an image obtained by subjecting an image captured by an imaging device equipped with two fisheye lenses to equirectangular processing.

2. The information processing apparatus according to claim 1,

15. The pair of images includes a left-eye image and a right-eye image that is paired with the left-eye image and has a parallax with the left-eye image.

2. The information processing apparatus according to claim 1,

16. An acquisition step of acquiring a pair of images constituting a parallax image for realizing a stereoscopic vision; a matching step of generating matching data that links corresponding pixels in a pair of images that constitute the parallax image acquired in the acquisition step; a correction step of correcting a vertical shift between corresponding pixels in the pair of images based on the generated correspondence data; 13. An information processing method comprising:

17. A program for causing a computer to function as each step of the information processing method according to claim 16.

Images