Image processing device, image processing method, and program

Abstract

This method aims to generate images that suppress vertical displacement of the subjects in the left and right images of a stereo image, rather than simply aligning them. [Solution] The image processing device includes an image acquisition means for acquiring multiple parallax images of a subject; a first projection information generation means for generating first projection information based on calibration values ​​for stereo parallelization of images; a positional shift information generation means for generating positional shift information including vertical shift information of images based on the first projection information; a second projection information generation means for generating second projection information based on the first projection information and positional shift information; and an image generation means for performing warping processing based on the second projection information and generating an output image from the images acquired by the image acquisition means.

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program. [Background technology]

[0002] With the evolution of HMD (Head-Mounted Display) technology, it has become possible to input wide-angle images with a field of view exceeding 180 degrees and display the cropped image in real time within the display field of view in front of the HMD. This allows users to easily view so-called VR (Virtual Reality) images. Furthermore, by capturing images in stereo and displaying them on the left and right displays of the HMD, stereo vision is possible, improving the sense of realism. For capturing wide-angle images, for example, a fisheye lens is used, and this image will be referred to as a fisheye image below. The image input to the HMD is a fisheye image converted into an equirectangular image. Stereo images are captured, for example, by two cameras equipped with monocular lenses, or by a camera equipped with a lens unit having two lens optical systems (hereinafter, the lens optical system will be simply referred to as a lens).

[0003] When capturing stereo images, misalignment of the optical axes can occur between the left and right lenses. When viewing a stereo image with misaligned optical axes, vertical misalignment occurs in the left and right images, hindering fusion. Furthermore, even if a caption is superimposed at the same vertical coordinate position on the left and right images of a stereo image with misaligned optical axes, a difference will occur in the relative position of the object and the caption. Therefore, it is important to generate an image that has been corrected to minimize vertical misalignment. Horizontal misalignment is a misalignment necessary for calculating parallax to perceive depth, and even if there are some errors in horizontal misalignment or parallax, it only changes the perceived depth and is unlikely to hinder the fusion of the left and right images. Another purpose of correcting vertical misalignment is to speed up the calculation of parallax. The parallax of pixels in a centrally projected stereo image is the difference in the horizontal coordinates of corresponding images or pixels in the left and right images. Pixel correspondence can be achieved, for example, by searching the right image for blocks with a small difference (SAD, etc.) of surrounding pixels centered on the pixel of interest in the left image. In this case, if the vertical positions of corresponding pixels in the left and right images match, the search can be limited to one dimension in the horizontal direction, enabling faster calculation of disparity.

[0004] One method for correcting vertical displacement is called stereo parallelization. Stereo parallelization calculates camera parameters so that corresponding pixels on the left and right are placed on the epipolar line, and then projects the left and right images. For example, an F matrix is ​​used for stereo parallelization of a centrally projected image. Non-patent documents 1 and 2 disclose projection methods for stereo parallelization of equidistant projected images (fisheye images). Furthermore, in recent years, it has become possible to capture high-resolution video with horizontal resolutions of 8K and 16K, and vertical displacement tends to increase when viewed at the pixel level, thus increasing the demand for vertical displacement correction. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Fish-Eye-Stereo Calibration and Epipolar Rectification,Steffen Abraham, W. Forstner,Published 1 August 2005,Computer Science, Engineering Isprs Journal of Photogrammetry and Remote Sensing [Non-Patent Document 2] Fisheye stereo camera using equirectangular images, Akira OHASHI, Fumito YAMANO, Gakuto MASUYAMA, Kazunori UMEDA, Daisuke FUKUDA, Kota IRIE, Shuzo KANEKO, Junya MURAYAMA, Yoshitaka UCHIDA, Journal of the Japan Society for Precision Engineering, 2017 Volume 83 Issue 12 Pages 1095-1100 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] Even after calibrating and aligning stereo images using global camera parameters, vertical misalignment of the left and right images remains. [Means for solving the problem]

[0007] The image processing apparatus according to the present invention includes: an image acquisition means for acquiring a plurality of parallax images of a subject; a first projection information generation means for generating first projection information based on calibration values ​​for stereo parallelization of images; a positional shift information generation means for generating positional shift information including vertical shift information of images based on the first projection information; a second projection information generation means for generating second projection information based on the first projection information and the positional shift information; and an image generation means for performing warping processing based on the second projection information and generating an output image from the images acquired by the image acquisition means. [Effects of the Invention]

[0008] According to the present invention, it is possible to generate an image in which vertical displacement of the subjects in the left and right images of a stereo image is suppressed more effectively than with stereo parallelization. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram illustrates an example of the hardware configuration of an image processing device. [Figure 2] This diagram illustrates an example of the functional configuration of an image processing device. [Figure 3] This is a flowchart illustrating an example of image processing in Embodiment 1. [Figure 4] This is a flowchart explaining the positional misalignment detection process. [Figure 5] This is a diagram illustrating image segmentation. [Figure 6] This figure shows the epipolar lines for each latitude in an equirectangular image. [Figure 7] This diagram illustrates an example of the functional configuration of an image processing device. [Figure 8] This is a flowchart illustrating an example of image processing in Embodiment 1. [Figure 9] This is a flowchart illustrating an example of image processing in Embodiment 2. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0011] <Embodiment 1> An example of generating an image in which vertical displacement of a stereo image is suppressed will be described below. The fisheye image in the present embodiment is an equidistant projected image with a viewing angle of 180 degrees, and the description is given on the assumption that the resolution is 4096×4096, and the image circle of the fisheye is adjacent to a 4096×4096 rectangular area. However, the resolution of the image is not limited to this and may be any value. A stereo fisheye image is a set of left and right fisheye images.

[0012] FIG. 1 is a diagram illustrating an example of the hardware configuration of an image processing apparatus according to the present embodiment. The image processing apparatus according to the present embodiment includes a bus 101, a RAM 102, a graphic processor 103, a display 104, a CPU 105, a user I / F 106, an external storage 107, and an input / output I / F 108.

[0013] The bus 101 controls data flow within the image processing apparatus. A RAM (Random Access Memory) 102 is a writable memory, and functions as, for example, a work area for the CPU 105. The graphic processor 103 performs arithmetic processing and the like required when displaying an image on the display 104. The graphic processor 103 is capable of encoding and decoding images in real time. The display 104 is a display device that displays commands input from the user I / F 106, response outputs of the image processing apparatus to the commands, and the like.

[0014] A CPU (Central Processing Unit) 105 cooperates with other components based on computer programs such as an operating system (OS) and application programs, and controls the operation of the entire image processing apparatus. In the present embodiment, the description is given on the assumption that the image processing apparatus includes one CPU 105, but the configuration is not limited to this, and the image processing apparatus may include a plurality of CPUs 105. In this case, each process can be operated in parallel by multithread processing.

[0015] The user I / F 106 is, for example, an interface to which a touch panel, a keyboard, a mouse and the like are integrally connected, and a user can input commands and the like through operations. The external storage 107 is a non-volatile external storage medium, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive) or the like. The input / output I / F 108 is an interface for exchanging data via a network or the like.

[0016] In this embodiment, the program and data to be executed are recorded in external storage 107, input and stored in RAM 102, and executed by CPU 105. The program and data are input and output via bus 101 and RAM 102. Hereafter, when referring to the acquisition of images and data, it refers to data read from external storage 107 to RAM 102, data generated by each module and stored in RAM 102, or data acquired directly from each module. The internal format in this embodiment is an 8-bit RGB image, and image projection and conversion are performed by inputting an 8-bit RGB image and outputting an 8-bit RGB image. However, there are no limitations on the format, and YUV images or monochrome luminance images may also be used. Furthermore, the position shift detection described later is performed using a luminance image, and if the internal format is an RGB image or a YUV image, it is converted to a luminance image before position shift detection is performed. If the image data is encoded, it will be explained that the encoded data is sent to the graphics processor 103, decoded within the graphics processor 103, converted to the internal image format, and then acquired by CPU 105. The UI (user interface) screen and image processing results can be displayed on the display 104 via the graphics processor 103. The graphics processor 103 is capable of performing geometric transformations on input images and can input / output images to / from RAM 102 or output directly to the display 104. The processed data can be recorded in external storage 107 or stored in RAM 102 and shared with other programs.

[0017] Figure 2 is a diagram illustrating an example of the functional configuration of the image processing apparatus in this embodiment. The image processing apparatus in this embodiment includes an image acquisition unit 201, a calibration value acquisition unit 202, a gravity direction information acquisition unit 203, a parameter calculation unit 204, a first projection information generation unit 205, and a first warping processing unit 206. The image processing apparatus also includes a position shift information generation unit 207, a filtering unit 208, a second projection information generation unit 209, a second warping processing unit 210, and an image output unit 211. In this embodiment, the output of various data is performed via the bus 101 and RAM 102, but there are no limitations on the method, and it can be used at any timing on the input side.

[0018] The image acquisition unit 201 acquires a stereo image consisting of multiple images with parallax taken of a subject. In this embodiment, the image acquisition unit 201 acquires a stereo fisheye image. The calibration value acquisition unit 202 acquires calibration values ​​related to the stereo image acquired by the image acquisition unit 201. For example, the calibration value acquisition unit 202 acquires calibration values ​​for stereo parallelization of the stereo image. The gravity direction information acquisition unit 203 acquires gravity direction information indicating the deviation between the vertical direction and the gravity direction of the stereo image acquired by the image acquisition unit 201.

[0019] The parameter calculation unit 204 calculates image conversion parameters for the stereo image acquired by the image acquisition unit 201 based on the calibration values ​​acquired by the calibration value acquisition unit 202 and the gravity direction information acquired by the gravity direction information acquisition unit 203. The first projection information generation unit 205 generates first projection information to be used for warping processing in the first warping processing unit 206 based on the image conversion parameters calculated by the parameter calculation unit 204.

[0020] The first warping processing unit 206 uses the first projection information generated by the first projection information generation unit 205 to perform image transformation processing (warping) on ​​the stereo image acquired by the image acquisition unit 201. In this embodiment, the first warping processing unit 206 generates an equirectangular image from the stereo fisheye image acquired by the image acquisition unit 201 through warping processing.

[0021] The positional displacement information generation unit 207 estimates a two-dimensional positional displacement for each pixel based on the first projection information and generates positional displacement information including information on vertical displacement. The filtering unit 208 performs filtering on the positional displacement information generated by the positional displacement information generation unit 207. Alternatively, the filtering unit 208 may use the stereo image generated by warping based on the first projection information by the first warping processing unit 206 to perform filtering on the positional displacement information.

[0022] The second projection information generation unit 209 generates second projection information to be used for warping processing in the second warping processing unit 210, based on the first projection information generated by the first projection information generation unit 205 and the filtered positional shift information. The second projection information generation unit 209 generates projection information that performs vertical shift correction and image conversion in a stereo image all at once as the second projection information.

[0023] The second warping processing unit 210 uses the second projection information generated by the second projection information generation unit 209 to perform image conversion processing (warping) on ​​the stereo image acquired by the image acquisition unit 201 to generate an output image. The second warping processing unit 210 is an example of an image generation means. In this embodiment, the second warping processing unit 210 generates an equirectangular image as an output image from the stereo fisheye image acquired by the image acquisition unit 201 through warping processing. The image output unit 211 outputs the stereo image (output image) generated by warping based on the second projection information by the second warping processing unit 210.

[0024] Figure 3 is a flowchart illustrating an example of image processing performed by the image processing device in Embodiment 1. The image processing device calculates calibration information and performs image transformation based on the calculated calibration information.

[0025] In step S301, the CPU 105 acquires a stereo image. In this embodiment, the stereo image is assumed to be a stereo fisheye image, and the CPU 105 acquires the stereo fisheye image from the external storage 107. A stereo image is a set of images that are simultaneously captured by lenses positioned horizontally apart, resulting in parallax between the left and right images.

[0026] In step S302, the CPU 105 acquires calibration values. In this embodiment, there are no limitations on the stereo parallelization model or calibration method, and the method may be, for example, the techniques disclosed in Non-Patent Literature 1 or Non-Patent Literature 2. In this embodiment, for example, the calibration values ​​for stereo parallelization calculated in advance for the model described in Non-Patent Literature 2 are recorded in the external storage 107 in association with the stereo fisheye image (for example, as image metadata). Then, when step S302 is executed, the CPU 105 reads the recorded calibration values ​​from the external storage 107.

[0027] In step S303, the CPU 105 acquires gravity direction information. In this embodiment, for example, the gravity direction is detected from the information of the accelerometer built into the camera when an image is taken. The difference between this detected gravity direction and the vertical direction of the captured image is recorded as gravity direction information in the external storage 107 in the form of an appended image to the stereo fisheye image (for example, as image metadata). Then, when step S303 is executed, the CPU 105 reads the recorded gravity direction information from the external storage 107.

[0028] In step S304, the CPU 105 calculates image transformation parameters based on the calibration values ​​obtained in step S302 and the gravity direction information obtained in step S303. The calculated image transformation parameters are the parameters for performing equirectangular transformation on the fisheye image in the process of step S306, which will be described later. In step S304, the CPU 105 calculates the parameters of the transformation formula that incorporates the acquired calibration values ​​and gravity direction information into the formula for performing equirectangular transformation from the fisheye image. In this embodiment, for example, the correction of the gravity direction is performed in the polar coordinate calculation process of Non-Patent Literature 2 by rotating each of the left and right images in the direction of the optical axis (Z axis) to maintain the horizontal alignment of the left and right images.

[0029] In step S305, the CPU 105 generates first projection information (equirectangular transformation map) based on the image transformation parameters calculated in step S304. The equirectangular transformation map is a two-dimensional map that contains information about which coordinate pixels in the fisheye image are referenced when calculating the pixel values ​​of the output equirectangular image. The width and height of the map are the same as those of the equirectangular image, respectively. The elements of the map are represented as a two-dimensional real vector (X,Y). In this embodiment, a map is generated for each of the left and right images in the stereo image. The specific transformation formula from the fisheye image to the equirectangular image is disclosed in Non-Patent Literature 2, for example, so its explanation is omitted here.

[0030] In step S306, the CPU 105 generates an equirectangular image from the stereo fisheye image by warping using the first projection information (equirectangular transformation map) generated in step S305. Since the coordinates, which are elements of the equirectangular transformation map, are represented by real numbers, the CPU 105, for example, obtains pixel values ​​by referencing the pixel values ​​of the four neighboring pixels (coordinate values ​​are integers) of the coordinates referenced by real numbers and synthesizing them using bilinear interpolation.

[0031] In step S307, the CPU 105 generates positional displacement information (positional displacement map including vertical displacement) based on the first projection information (equirectangular transformation map) generated in step S305. The CPU 105 estimates the two-dimensional positional displacement for each pixel from the first projection information (equirectangular transformation map) and generates a positional displacement map including vertical displacement. In this embodiment, as an example, the displacement of corresponding pixels in the right image is mapped using the left image as a reference. For the positional displacement detection method, positional displacement detection is performed by executing the process described with reference to Figure 4, for example. In this embodiment, two-dimensional positional displacement is estimated, but only the map of one-dimensional vertical displacement information is stored and used in the processing of later steps.

[0032] In step S308, the CPU 105 performs filtering on the positional displacement information generated in step S307. In this embodiment, the CPU 105 performs filtering on the vertical component of the generated positional displacement map. The CPU 105 applies an edge-preserving smoothing filter to the positional displacement map generated in step S307. One method of edge-preserving smoothing is a bilateral filter. There are also guided filters and joint bilateral filters that use an image as guide information. The guide information will be obtained and used from an equirectangular image.

[0033] In step S309, the CPU 105 generates a second projection information (vertical displacement correction / equirectangular transformation map) based on the first projection information (equirectangular transformation map) generated in step S305 and the position displacement map filtered in step S308. The vertical displacement correction / equirectangular transformation map is a map that performs vertical displacement correction and equirectangular transformation in one step. In this embodiment, the difference value obtained by halving the vertical position displacement element of the filtered position displacement map is subtracted from the corresponding element of the equirectangular transformation map of the left image and added to the corresponding element of the equirectangular transformation map of the right image. This is equivalent to finding the average value of the vertical coordinate of the pixel in the left image and the vertical coordinate of the corresponding pixel in the right image, and subtracting and adding half of the difference value to each.

[0034] In step S310, the CPU 105 uses the second projection information (vertical shift correction / equirectangular transformation map) generated in step S309 to generate a stereo equirectangular image (output image) for output from the stereo fisheye image by warping. The image warping method is the same as the processing in step S306.

[0035] In step S311, the CPU 105 outputs correction identification information and the stereo image (stereo equirectangular image for output) generated in step S310. In this embodiment, for example, the image is output to an HMD, and the HMD generates a projected image that matches the characteristics of the lens and display within the HMD and displays it on the display, allowing the user to view the image. The correction identification information indicates whether or not correction was performed (whether or not local vertical shift correction was performed) and, if so, the correction method (what model was used for correction). This information can be used, for example, to suppress the HMD from performing duplicate processing or as a hint when re-correcting to the characteristics of the HMD.

[0036] Next, we will explain the positional displacement detection in the image processing device. The method for calculating the two-dimensional positional displacement for each pixel is the same as the optical flow estimation algorithm for calculating time-series motion. In this embodiment, the vertical center of the equirectangular image is set to 0 degrees, the absolute value of the vertical position (elevation angle) is expressed as latitude, and the azimuth angle with the optical axis direction as 0 degrees is expressed as longitude. In an equirectangular image, areas with higher latitudes are magnified horizontally, and the parallax expressed in terms of the number of pixels also increases. Therefore, the search range widens, and it is difficult to directly apply the optical flow estimation algorithm that is executed assuming the maximum search range. Below, with reference to Figure 4, we will explain the process for detecting the positional displacement of an equirectangular image that addresses this problem. Figure 4 is a flowchart that explains the positional displacement detection process for an equirectangular image.

[0037] In step S401, the CPU 105 receives an equirectangular image of the object to be detected for positional displacement. In step S402, the CPU 105 vertically divides the equirectangular image input in step S401 into horizontal slices. In this embodiment, the equirectangular image is divided into 36 vertically oriented sections of 5 degrees each. This division is schematically represented in Figure 5(a). Note that in Figure 5(a), the division into horizontal slices is schematically represented, so the number of divisions is shown as 8.

[0038] In step S403, the CPU 105 reduces each slice (divided image) according to a reduction ratio calculated based on the position of the slice (divided image) and the first projection information, such that the width of the slice decreases as the latitude increases. In this embodiment, the CPU 105 reduces the slice by bilinear interpolation using the following equations (1) and (2), where s is the horizontal reduction ratio when y is the vertical center coordinate of the slice.

[0039]

number

[0040] Figure 5(b) schematically represents this division state. This shows that slices (divided images) at higher latitudes are horizontally compressed.

[0041] In step S404, the CPU 105 detects the two-dimensional displacement for each pixel in each slice and generates a displacement map slice. The CPU 105 detects the displacement for each slice (divided image unit). The method for calculating the two-dimensional displacement for each pixel is the same as the optical flow estimation algorithm for calculating time-series motion. Various optical flow estimation algorithms are known, including methods for filling in occlusion regions, and there are no restrictions on the algorithm used for this flow. The algorithm should be selected according to the target accuracy and processing time, so a detailed explanation is omitted. Note that pixels with y=0 and y=4095 cannot have their disparity calculated and are therefore excluded from displacement detection.

[0042] In step S405, the CPU 105 expands each displacement map slice generated in step S404 in the horizontal direction. The expansion ratio is the reciprocal of the reduction ratio s calculated in step S403.

[0043] In step S406, the CPU 105 synthesizes the horizontally expanded displacement map slices from step S405 to generate a displacement map. This generates a displacement map with a vertical and horizontal field of view of 180 degrees. The generated displacement map is used in the processing shown in the flowchart in Figure 3.

[0044] In the case of stereo images, vertical displacement of images of the same subject can be reduced by performing stereo parallelization projection using global calibration values ​​such as camera parameters. However, such projection alone cannot completely eliminate vertical displacement of the left and right images. For example, if the lens used for shooting is not pan-focus, the focus will shift, requiring correction for each image to achieve accurate calibration. Furthermore, if distortion exists in the lens and needs to be corrected, it is difficult to improve the accuracy of the correction, and external factors such as thermal expansion may cause the optical axis to shift, meaning that the pre-calculated calibration value may not be optimal. Also, if the image sensor is driven by a rolling shutter system, rolling shutter distortion occurs. Moreover, parallelism cannot be guaranteed when the image is rotated using gravity direction information. Thus, there are limitations to suppressing vertical displacement solely through projection using global deformation parameters.

[0045] In this embodiment, the image processing device converts a fisheye image into an equirectangular image, detects the amount of positional displacement in two dimensions for each pixel of the left and right equirectangular images, and warps the image by shifting the pixel position using the vertical displacement component. Compared to projection using global calibration values, the vertical displacement is reduced while the horizontal displacement remains unchanged, and it is possible to generate an image that suppresses the vertical displacement of the subject in the left and right images more effectively than stereo parallelization. In equirectangular images, the horizontal magnification is large in high latitude regions (regions far from the vertical center), meaning that the horizontal displacement is large, making it difficult to detect positional displacement using general optical flow detection algorithms. In this embodiment, the image is sliced ​​according to latitude, and the horizontal reduction ratio is changed for each slice to enable positional displacement detection in the equirectangular image, and this positional displacement information is used to correct the vertical displacement. In addition, when pixels of the left and right images are matched in a stereo image, there are regions where no corresponding pixels exist, so-called occlusion regions. Even if the positional displacement in these regions is estimated, the accuracy is low, so the roughness tends to be noticeable when the image is warped. Therefore, in this embodiment, the image processing device applies a smoothing filter to the misalignment map in order to suppress high-frequency fluctuations in the misalignment correction amount, which are easily noticeable due to warping. Since occlusion regions do not occur across the boundary between foreground and background objects, an edge-preserving smoothing filter is preferable if the increased computational load is tolerable in images with clearly defined object boundaries. In this embodiment, an optical flow estimation algorithm is used to detect misalignment, but a machine learning-based optical flow estimation algorithm with occlusion countermeasures may also be applied. Occlusion region countermeasures estimate information from the surrounding area, and can achieve an effect equivalent to applying an edge-preserving smoothing filter. By doing so, it becomes possible to view images with less vertical misalignment without changing the parallax component when viewing stereo images with an HMD.

[0046] The image processing device in Embodiment 1 generates first projection information based on calibration values ​​for stereo parallelization of the image, and generates positional displacement information including vertical displacement information of the image based on the first projection information. Then, based on the first projection information and positional displacement information, it generates second projection information for correcting the vertical displacement of the left and right images, and generates an output image from the acquired stereo image by performing warping processing based on the second projection information. With such an image processing device, it is possible to generate an image in which the vertical displacement of the subjects in the left and right images of the stereo image is suppressed more than with stereo parallelization.

[0047] In this embodiment, the image is described as being displayed on an HMD, but it is not limited to this. The image processing device may output an equirectangular image to a file or display the equirectangular image directly on a flat-panel display. When superimposing text overlays, objects, etc., onto the output equirectangular image using an image editing application, there is no vertical misalignment between the left and right images, so the text overlays and objects can be placed at the same vertical coordinate position in the left and right images, leading to improved efficiency in stereo image editing.

[0048] In this embodiment, one of the objectives is to position text overlays and objects at the same vertical coordinate positions in the left and right images on an equirectangular image. To achieve this, an example is shown in which displacement in the Y-axis direction (elevation / depression angle) on the equirectangular image is treated as vertical displacement and corrected. However, displacement perpendicular to the epipolar line may also be treated as vertical displacement and corrected. Figures 6(a) and 6(b) show epipolar lines for each latitude in an equirectangular image. In Figure 6(a), the vertical and horizontal axes represent elevation / depression angle and azimuth angle, respectively, and each curve shows an epipolar curve in 15-degree increments from 0 to 75 degrees. The larger the absolute values ​​of the elevation / depression angle and azimuth angle, the more vertical the epipolar line on the figure tends to be. Note that in the process of step S307 in Figure 3, only one-dimensional vertical displacement information is stored. However, when correcting displacement perpendicular to the epipolar line as vertical displacement, this vertical displacement can be represented and stored in two dimensions. Vertical shift correction for vertical shift perpendicular to the epipolar line can more accurately suppress vertical shift within the display field of view in cases where the HMD display does not follow head movements and only the gaze moves. A similar effect is achieved when the HMD adjusts the position of 0 degrees azimuth to track head movements in order to suppress vertical shift in real time, and displays a cropped field of view, for example, an azimuth angle of ±60 degrees. This problem arises because, while the azimuth angle is set with a 180-degree field of view centered on each of the left and right lenses during imaging, the azimuth angle is set with the head as the center for the HMD, resulting in a difference in the geometric model. Even if the definition of vertical shift differs depending on the purpose, this embodiment allows for the definition and suppression of a desired vertical shift on the left and right images. For example, one could define and correct vertical shift by considering both vertical shift due to elevation / depression angle and vertical shift perpendicular to the epipolar line. Alternatively, the vertical shift due to elevation / depression angle and vertical shift perpendicular to the epipolar line could be corrected on average. Alternatively, as shown in Figure 8(b) as an example, the vertical deviation relative to the elevation angle may be corrected for azimuth angles up to ±60 degrees, and for angles where the absolute value of the azimuth angle is greater than that, the deviation perpendicular to the epipolar curve may be considered as the vertical deviation and corrected accordingly.Furthermore, the correction identification information output in step S311 of Figure 3 can also be information that identifies the method, such as whether it is a vertical shift with respect to the elevation angle, a vertical shift with respect to the epipolar line, or a vertical shift with respect to an arbitrary model as shown in Figure 8(b). When outputting to a file, in addition to whether or not correction is performed, information identifying the method may be output to the file as metadata for the image along with the image data. In the case of a vertical shift with respect to an arbitrary model, the mathematical formula or curve data that represents that model will also be included in the information identifying the method. In this embodiment, the component perpendicular to the epipolar line of a two-dimensional image is defined as the vertical shift, but the component perpendicular to the epipolar line of a sphere projected into three-dimensional space may also be defined as the vertical shift. When arranging text vertically in the direction of line of sight, assuming viewing with an HMD, suppressing this vertical shift makes it possible to superimpose with less discomfort due to the shift.

[0049] In this embodiment, the calibration parameters for stereo parallelization were described as being calculated in advance. However, instead of calculating the calibration values ​​in advance, feature points may be detected from the input image, and the calibration values ​​may be calculated using the coordinate information of the feature points. In this case, if information on the direction of gravity is also taken into account when calculating the calibration values, it is possible to calculate calibration parameters that achieve both horizontal alignment of the image and parallelization of the epipolar lines. In this embodiment, images are described as being input from a file, but the invention is not limited to this; captured images from an image sensor may also be input in real time. In this embodiment, the processing in step S309 of Figure 3 involves modifying both the left and right images, but this is not the only option. For example, the elements related to the left image may be fixed, and only the elements related to the right image may be modified, or conversely, the elements related to the left image may be modified, and the elements related to the right image may be fixed, and the same result can be obtained. In this embodiment, the process shown in the flowchart in Figure 4 is described as generating a displacement map and then performing a filter. However, the filter may be performed on the displacement map slice generated in step S404 of Figure 4.

[0050] In this embodiment, an example of processing a pair of stereo fisheye images is shown, but moving images may also be processed by processing them sequentially. In this case, if the position shift detection process is performed by using the position shift of the previous frame as the initial value and calculating the difference, the amount of processing required for position shift detection can be reduced. Furthermore, it is not limited to stereo images consisting of a pair of images, but images consisting of three or more images taken in the same direction may also be processed. In this embodiment, the processing of each step has been described as being performed by the CPU 105 of the image processing device, but the device configuration may include dedicated hardware for each process, with each piece of hardware performing the processing. Alternatively, for example, the ISP (Image Signal Processor) may be equipped with the image conversion function described in this embodiment, and the ISP may perform the image conversion.

[0051] In this embodiment, positional displacement detection was performed using an equirectangular image, but it is not limited to this, and cylindrical images or perspective projection images may also be used. Cylindrical images cannot represent a wide field of view such as ±90 degrees vertically, and perspective projection images also cannot represent a wide field of view of ±90 degrees vertically and horizontally, making it impossible to detect positional displacement at high latitudes. However, in the case of cylindrical images, it is possible to change the projection direction for each latitude and create sliced ​​images divided by latitude as shown in Figure 5 to accommodate a wide field of view. Similarly, in the case of perspective projection images, it is possible to accommodate a wide field of view by dividing them in the direction of latitude and longitude. In this embodiment, a stereo fisheye image with a 180-degree field of view was used as the input, but the system is not limited to this. It is possible to use images captured with lenses of various fields of view, including stereo 360-degree cameras. In this embodiment, it was explained that the map used for equirectangular image transformation and the equirectangular image are the same size, but this is not limited to this. The map size may be reduced to reduce computational complexity and memory usage. A small map can be enlarged using bilinear interpolation or similar methods, although the accuracy will decrease, making it usable as a map. It is not necessary to generate an enlarged map and store it in memory; if the elements to be referenced are calculated when needed, the processing is equivalent and memory usage does not increase.

[0052] (modified version) In the example described above, filtering was performed on the positional misalignment information (positional misalignment map) according to the configuration shown in Figure 2 and the processing shown in Figure 3. However, filtering may also be performed on the positional misalignment information (positional misalignment map) according to the configuration shown in Figure 7 and the processing shown in Figure 8.

[0053] Figure 7 illustrates another example of the functional configuration of the image processing apparatus in this embodiment. In Figure 7, components having the same function as those shown in Figure 2 are denoted by the same reference numerals. In the image processing apparatus shown in Figure 7, the positions of the filtering unit 208 and the second projection information generation unit 209 are swapped compared to the image processing apparatus shown in Figure 2.

[0054] In the image processing apparatus shown in Figure 7, the second projection information generation unit 209 generates second projection information based on the first projection information generated by the first projection information generation unit 205 and the positional displacement information generated by the positional displacement information generation unit 207. The second projection information generation unit 209 generates second projection information (vertical displacement correction / equidistinctive transformation map) based on the first projection information (equidistinctive cylindrical transformation map) and the positional displacement information (positional displacement map).

[0055] The filtering unit 208 applies filtering to the second projection information generated by the second projection information generation unit 209. Alternatively, the filtering unit 208 may apply filtering to the second projection information using the stereo image generated by the first warping processing unit 206 based on the first projection information. The second projection information filtered by the filtering unit 208 is output to the second warping processing unit 210.

[0056] Figure 8 is a flowchart illustrating an example of image processing performed by the image processing device shown in Figure 7. The processes in steps S301-S307 and S310-S311 in Figure 8 are the same as those in steps S301-S307 and S310-S311 in Figure 3, respectively, so their explanation is omitted.

[0057] In step S801, which follows the processing in step S307, the CPU 105 generates provisional second projection information based on the first projection information generated in step S305 and the positional displacement information generated in step S307. For example, the CPU 105 combines the two maps, the first projection information (equidistinctive cylindrical transformation map) and the positional displacement information (positional displacement map including vertical displacement), to generate provisional second projection information (vertical displacement correction / equidistinctive cylindrical transformation map). The specific method is the same as the method described in step S309 in Figure 3, so the explanation is omitted.

[0058] Next, in step S802, the CPU 105 performs a filtering process on the provisional second projection information (vertical displacement correction / equidistinctive cylindrical transformation map) generated in step S801 to generate a second projection information (vertical displacement correction / equidistinctive cylindrical transformation map). In this embodiment, the CPU 105 performs a filtering process on the vertical component of the generated provisional second projection information (vertical displacement correction / equidistinctive cylindrical transformation map) to generate a second projection information (vertical displacement correction / equidistinctive cylindrical transformation map). The specific method is the same as the method described in step S308 in Figure 3, so the explanation is omitted. After executing the process in step S802, the process proceeds to step S310.

[0059] In equirectangular transformation, the coordinate values ​​referenced to calculate the pixel value of the target pixel in the equirectangular image do not change significantly between the target pixel and its vicinity. Therefore, even if filtering is performed on a map that combines the positional shift map and the equirectangular transformation map, as in this modified example, the same effect as filtering the positional shift information can be obtained.

[0060] <Embodiment 2> Embodiment 2 describes an example of generating an image with suppressed vertical shift of stereo images for video processing. The hardware configuration and functional configuration of the image processing device in Embodiment 2 are the same as those in Embodiment 1. However, in Embodiment 2, the operation of the image acquisition unit 201, the positional shift information generation unit 207, and the filtering unit 208 differs from that of Embodiment 1. In addition, the image processing device in Embodiment 2 operates continuously as a whole for video processing.

[0061] The image acquisition unit 201 acquires a stereo image consisting of multiple images with parallax taken of the subject. The image acquisition unit 201 acquires each frame image in a stereo video composed of multiple frames as a stereo image. In the process of step S901 in Figure 9, which will be described later, the stereo image of one frame acquired by the image acquisition unit 201 is output to the first warping processing unit 206. In addition, in the process of step S909 in Figure 9, which will be described later, the stereo images acquired continuously by the image acquisition unit 201 are output to the second warping processing unit 210.

[0062] The positional displacement information generation unit 207 estimates a two-dimensional positional displacement for each pixel based on the first projection information and generates two-dimensional positional displacement information (positional displacement map). The filtering unit 208 applies a filtering process to the two-dimensional positional displacement information (positional displacement map) generated by the positional displacement information generation unit 207 and outputs the filtered positional displacement information (positional displacement map) to the second projection information generation unit 209. In this embodiment, the filtering unit 208 applies a filtering process to the vertical component of the positional displacement map based on the amount of lateral displacement in the generated positional displacement map. Alternatively, the filtering unit 208 may apply a filtering process using a stereo image generated by warping based on the first projection information by the first warping processing unit 206.

[0063] In Embodiment 2, the second warping processing unit 210 receives and processes the same second projection information (vertical displacement correction / equidistinctive cylindrical transformation map) generated by the second projection information generation unit 209 every frame.

[0064] Figure 9 is a flowchart illustrating an example of image processing performed by the image processing device in Embodiment 2. In step S901, the CPU 105 acquires one frame of the stereo video as a stereo image. In this embodiment, as an example, the CPU 105 acquires the image of the first frame of the stereo video. In this embodiment as well, the stereo image is assumed to be a stereo fisheye image.

[0065] In step S902, the CPU 105 acquires a calibration value in the same manner as in step S302 in Embodiment 1. In step S903, the CPU 105 acquires gravity direction information in the same manner as in step S303 in Embodiment 1. In step S904, the CPU 105 calculates image transformation parameters based on the calibration values ​​obtained in step S902 and the gravity direction information obtained in step S903, in the same manner as in step S304 in Embodiment 1.

[0066] In step S905, the CPU 105 generates first projection information (equidistinctive cylindrical transformation map) based on the image transformation parameters calculated in step S904, in the same manner as in step S305 in Embodiment 1. In step S906, the CPU 105 generates an equirectangular image from the stereo fisheye image by warping processing, using the first projection information (equirectangular transformation map) generated in step S905, in the same manner as in step S306 in Embodiment 1.

[0067] In step S907, the CPU 105 generates two-dimensional positional displacement information (positional displacement map) for each pixel based on the first projection information (equidistinctive cylindrical transformation map) generated in step S905. For the positional displacement detection method, for example, positional displacement detection is performed by executing a process that will be described with reference to Figure 4, similar to step S307 in Embodiment 1.

[0068] In step S608, the CPU 105 performs filtering on the two-dimensional positional displacement information (positional displacement map) generated in step S907. In this embodiment, the CPU 105 performs filtering on the vertical component of the positional displacement map based on the amount of horizontal displacement. In this embodiment, the filtering is performed using a filter that gives a greater weight to the vertical displacement component of a pixel as the horizontal displacement of the pixel decreases, that is, as the depth position increases. An example of the filter function in this embodiment is described below.

[0069] Here, d vert (y,x) is the value of the vertical displacement at coordinate (y,x). horz (y,x) is the value of the horizontal displacement at coordinate (y,x). x =255, r y If we set =255, σ=42, and ε=0.01, then d' is the value of the vertical displacement of the output in the coordinate (y,x). vert (y,x) is expressed based on the following equations (3) and (4).

[0070]

[0071] Here, Equation (3) calculates the total sum of weights. Further, Equation (4) calculates the value of vertical shift after filtering such that the central weight is increased. r, which corresponds to the filter diameter, x , r y corresponds to the kernel size of the filter. When performing filter processing while scanning a pixel window to be referenced using a filter kernel, in the present embodiment, a sufficiently large value is set such that the filter kernel includes both a foreground and a background. Since an image is captured larger when the subject is at a closer position, it is necessary to adjust the values of r x , r y according to the shortest shooting distance. Note that since an equirectangular image is enlarged in the horizontal direction as the latitude increases, it is conceivable to increase the value of r x in accordance with the latitude; however, in the present embodiment, a fixed value is used to equalize the processing load.

[0072] In step S909, the CPU 105 continuously acquires stereo fisheye image frames. Successive frames are acquired one after another each time the processing of step S909 is executed. In step S910, the CPU 105 generates second projection information (vertical shift correction / equirectangular conversion map) in the same manner as in step S309 of the first embodiment. Specifically, the CPU 105 generates the second projection information (vertical shift correction / equirectangular conversion map) based on the first projection information (equirectangular conversion map) generated in step S905 and the position shift map filtered in step S908.

[0073] In step S911, the CPU 105 performs warping on the stereo image frame acquired in step S909, in the same manner as in step S310 in Embodiment 1. Using the second projection information (vertical shift correction / equirectangular transformation map) generated in step S910, the CPU 105 generates a stereo equirectangular image for output from the stereo fisheye image by warping. In step S912, the CPU 105 outputs the correction identification information and the stereo image (stereo equirectangular image for output) generated in step S911, in the same manner as in step S311 in Embodiment 1.

[0074] In step S913, the CPU 105 determines whether processing of all frames in the stereo video has been completed. If the CPU 105 determines that processing of all frames has been completed (YES in S913), the process shown in the flowchart in Figure 9 is terminated. On the other hand, if the CPU 105 determines that processing of all frames has not been completed, i.e., that there are unprocessed frames (NO in S913), the process returns to step S909. The process from steps S909 to S913 is repeated until it is determined that processing of all frames has been completed.

[0075] According to Embodiment 2, similar to Embodiment 1, it is possible to generate an image that suppresses vertical displacement of the subjects in the left and right images more effectively than stereo parallelization. Furthermore, when the position of the camera capturing stereo motion images is shifted, the images of subjects at different depth positions will shift more significantly the closer the depth distance. When performing image conversion to reproduce the shift movement by adjusting external parameters, it is not possible to reflect the difference in depth position, so it is necessary to adjust each pixel as in Embodiment 1. However, if the subject is sufficiently far away, the difference in depth position will result in a shift of less than one pixel, even in an 8K image. If driven by a global shutter method, there is no effect of rolling shutter distortion. Therefore, the main cause of vertical shift after stereo parallelization is optical distortion. In this embodiment, the processing in step S908 creates a vertical shift map by filtering, which has a high contribution to vertical shift corresponding to pixels with small parallax, that is, pixels with a large depth distance. This can be said to be a vertical shift map with low frame dependency. Although there are limitations on the shooting conditions, by applying this single vertical shift map to all frames, it is possible to reduce the load of detecting positional shifts for each frame compared to applying the processing described in Embodiment 1 to all frames.

[0076] In this embodiment, positional displacement detection is described as being performed once per video sequence, but this is not limited to this. Positional displacement maps for multiple frames in a video sequence may be generated and averaged to create a positional displacement map with reduced frame dependence, which can then be used to correct the vertical displacement of all frames. This demonstrates that smoothing in the temporal direction is effective as well as spatial smoothing.

[0077] In this embodiment, a filter is applied after generating a misalignment map, but the filter may also be applied to the sliced ​​images during the process shown in Figure 4, which is performed in step S907.

[0078] (Other embodiments of the present invention) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0079] It should be noted that the embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0080] The disclosure of this embodiment includes the following configurations and methods, etc. (Composition 1) Image acquisition means for acquiring multiple parallax images of a subject, A first projection information generation means that generates first projection information based on calibration values ​​for stereo parallelization of an image, A positional displacement information generation means generates positional displacement information including vertical displacement information of the image based on the first projection information, A second projection information generation means generates second projection information based on the first projection information and the positional displacement information, An image processing apparatus characterized by comprising: an image generation means that performs warping processing based on the second projection information and generates an output image from the image acquired by the image acquisition means. (Configuration 2) The system includes filtering means for filtering the vertical component of the positional displacement information or the second projection information, The image processing apparatus according to configuration 1, characterized in that the image generation means performs warping processing on the image acquired by the image acquisition means based on the second projection information generated based on the filtered positional displacement information or the filtered second projection information. (Composition 3) The image processing apparatus according to configuration 2, characterized in that the filtering means uses a filter in which pixels with smaller parallax have a greater weight in the vertical component. (Composition 4) The image processing apparatus according to any one of configurations 1 to 3, characterized in that the first projection information generation means generates the first projection information based on the calibration value and the deviation between the vertical direction and the gravity direction of the image. (Composition 5) The image processing apparatus according to any one of configurations 1 to 4, characterized in that the image generation means generates an equirectangular image from a fisheye image acquired by the image acquisition means. (Composition 6) The image processing apparatus according to any one of configurations 1 to 5, characterized in that the second projection information generation means generates the second projection information by taking the component perpendicular to the curve calculated based on the epipolar line of the equirectangular image generated from the fisheye image acquired by the image acquisition means based on the first projection information as the vertical displacement. (Composition 7) The image processing apparatus according to any one of configurations 1 to 5, characterized in that the second projection information generation means generates the second projection information by taking a component perpendicular to an arbitrary curve showing the relationship between the elevation angle and azimuth angle of an equirectangular image generated from a fisheye image acquired by the image acquisition means based on the first projection information as a vertical displacement. (Composition 8) The image processing apparatus according to any one of configurations 1 to 7, characterized in that the position shift information generating means divides an image generated from an image acquired by the image acquisition means based on the first projection information, reduces the divided image according to the position of the divided image and the reduction ratio calculated from the first projection information, and detects the position shift in units of the reduced divided image. (Composition 9) The image processing apparatus according to any one of configurations 1 to 8, characterized in that it outputs the output image and information regarding the presence or absence and method of vertical displacement correction. (Method 1) The image acquisition process involves obtaining multiple images with parallax from a photograph of the subject, A first projection information generation step that generates first projection information based on calibration values ​​for stereo parallelization of the image, A positional displacement information generation step that generates positional displacement information including vertical displacement information of the image based on the first projection information, A second projection information generation step that generates second projection information based on the first projection information and the positional displacement information, An image processing method characterized by comprising: a warping process based on the second projection information, and an image generation step that generates an output image from the image acquired in the image acquisition step. (Program 1) A program to cause a computer to function as one of the means of an image processing device described in any one of items 1 to 9 of the configuration. [Explanation of Symbols]

[0081] 101: Bus 102: RAM 103: Graphics Processor 104: Display 105: CPU 106: User Interface 107: External Storage 108: Input / Output Interface

Claims

1. Image acquisition means for acquiring multiple parallax images of a subject, A first projection information generation means that generates first projection information based on calibration values ​​for stereo parallelization of an image, A positional displacement information generation means generates positional displacement information including vertical displacement information of the image based on the first projection information, A second projection information generation means generates second projection information based on the first projection information and the positional displacement information, An image processing apparatus characterized by comprising: an image generation means that performs warping processing based on the second projection information and generates an output image from the image acquired by the image acquisition means.

2. The system includes filtering means for filtering the vertical component of the positional displacement information or the second projection information, The image processing apparatus according to claim 1, characterized in that the image generation means performs warping processing on the image acquired by the image acquisition means based on the second projection information generated based on the filtered positional displacement information or the filtered second projection information.

3. The image processing apparatus according to claim 2, characterized in that the filtering means uses a filter in which pixels with smaller parallax have a greater weight in the vertical component.

4. The image processing apparatus according to claim 1, characterized in that the first projection information generation means generates the first projection information based on the calibration value and the deviation between the vertical direction and the gravity direction of the image.

5. The image processing apparatus according to claim 1, characterized in that the image generation means generates an equirectangular image from a fisheye image acquired by the image acquisition means.

6. The image processing apparatus according to claim 1, characterized in that the second projection information generation means generates the second projection information by taking the component perpendicular to the curve calculated based on the epipolar line of the equirectangular image generated from the fisheye image acquired by the image acquisition means based on the first projection information as the vertical displacement.

7. The image processing apparatus according to claim 1, characterized in that the second projection information generation means generates the second projection information by taking a component perpendicular to an arbitrary curve showing the relationship between the elevation angle and azimuth angle of an equirectangular image generated from a fisheye image acquired by the image acquisition means based on the first projection information as a vertical displacement.

8. The image processing apparatus according to claim 1, wherein the positional displacement information generating means divides an image generated from an image acquired by the image acquisition means based on the first projection information, reduces the divided image according to the position of the divided image and the reduction ratio calculated from the first projection information, and detects the positional displacement in units of the reduced divided image.

9. The image processing apparatus according to claim 1, characterized in that it outputs the output image and information regarding at least one of the presence or absence and method of vertical displacement correction.

10. The image acquisition process involves obtaining multiple images with parallax from a photograph of the subject, A first projection information generation step that generates first projection information based on calibration values ​​for stereo parallelization of the image, A positional displacement information generation step that generates positional displacement information including vertical displacement information of the image based on the first projection information, A second projection information generation step that generates second projection information based on the first projection information and the positional displacement information, An image processing method characterized by comprising: a warping process based on the second projection information, and an image generation step that generates an output image from the image acquired in the image acquisition step.

11. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 9.

Images