Information processing device, method and program

Abstract

To estimate a depth of a subject in a highly accurate and high-speed manner.SOLUTION: An information processing device comprises: extraction means for extracting an area of a subject from each of two images photographed in two viewpoints by the information processing device; detection means for detecting corresponding points from the area of the subject for each of the two images; and estimation means for estimating a depth of the subject from the two viewpoints based on positions of the two viewpoints and positions of the corresponding points in each of the two images.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a technique for estimating the arrangement of an object. [Background technology]

[0002] In recent years, research has been conducted into mixed reality, which presents information from a virtual space to users by overlaying it on the real space in real time. A rendering processing device used in mixed reality displays a composite image in which all or part of a real image captured by an imaging device such as a video camera is overlaid on a virtual space image (CG) generated according to the position and orientation of the imaging device.

[0003] In this case, a specific subject area is detected from the image in real space and the three-dimensional shape of the subject is estimated, allowing a real object to be synthesized in the virtual space. One method for estimating the three-dimensional shape is the stereo measurement method using multiple cameras. In stereo measurement, camera parameters such as focal length and the relative positions and orientations of the cameras are estimated by calibrating the imaging device, and depth can be estimated from corresponding points in the captured images and the camera parameters using the principles of triangulation.

[0004] Such depth estimates need to be updated in real time at a frequency equivalent to the frame rate, i.e., both accuracy and speed are required.

[0005] To solve this problem, in Patent Document 1, first, block matching is performed on the entire stereo image to detect corresponding points between the stereo images. The depth is estimated based on the parallax, and the distance from the subject to be measured to the camera is determined as the estimated distance range. The search range for block matching is set to this estimated distance range and the depth is measured again. This is based on the idea that, for example, if the position of the face is determined, the distance range where the hands are located can be estimated, and the range can be narrowed down to this range. By narrowing the range and performing block matching in this way, highly accurate detection of corresponding points and therefore highly accurate depth estimation are realized. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2017-45283 A [Non-patent literature]

[0007] [Non-Patent Document 1] H. Hirschmuller.Stereo processing by semiglobal matching and mutual information.IEEE Transactions on Pattern Analysis and Machine Intelligence(PAMI),30(2):328-341,Feb 2008. Summary of the Invention [Problem to be solved by the invention]

[0008] Since the stereo images are captured at different positions, a structure drawn in one image may not be drawn in the other image. For example, FIG. 1 shows an example of left and right images captured by a stereo camera, where FIG. 1(A) is an image captured by the left camera, and FIG. 1(B) is an image captured by the right camera. In FIG. 1(A), a cube 101 is captured in the background of a hand, which is a subject, but in FIG. 1(B), the cube 101 is not captured. As described above, since the stereo images are captured at different positions by the cameras, the structures drawn in each image may differ. In such a case, false matching may occur in the stereo matching, and corresponding points between the stereo images may be erroneously detected. This is also the case when the technology of Patent Document 1 is used, and information other than the subject from which the depth is estimated may adversely affect the stereo matching, lowering the accuracy of the depth estimation. [Means for solving the problem]

[0009] In order to solve the above problem, according to one aspect of the present invention, an information processing device includes an extraction means for extracting a subject area from each of two images captured from two viewpoints, a processing means for processing each of the two images based on the subject area in each of the two images, a detection means for detecting corresponding points from the subject area in each of the two images after processing by the processing means, and an estimation means for estimating a depth of the subject from the two viewpoints based on the positions of the two viewpoints and the positions of the corresponding points in each of the two images. The processing means adds structural information of the subject to the two images. do. Effect of the Invention

[0010] According to the present invention, the depth of a subject can be estimated with high accuracy and at high speed. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a diagram showing an example of images captured by each camera of a stereo image. [Diagram 2] FIG. 2 is a block diagram showing an example of a functional configuration of the system. [Diagram 3] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. [Figure 4] 11 is a flowchart illustrating an example of a process executed by an information processing device. [Diagram 5] FIG. 13 is a diagram showing an example of an image in which the background region is filled with a single color. [Figure 6] 1 is an example image illustrating the problem of filling background regions with a single color. [Figure 7] 13A and 13B are diagrams illustrating examples of filters for adding subject structural information to a background region. [Figure 8] FIG. 13 is a diagram showing an example of an image in which structural information of a subject is added to a background region. [Figure 9] FIG. 13 is a diagram showing an example of an image in which correspondence information between images is added to a background region. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. Note that the configurations described in the following embodiments are representative examples, and the scope of the present invention is not necessarily limited to those specific configurations.

[0013] (Embodiment 1) 2 is a block diagram showing an example of the functional configuration of a system according to this embodiment. As shown in FIG. 2, the system according to this embodiment has an information processing device 200 connected to an imaging device 210 and a display device 220.

[0014] First, the information processing device 200 will be described. Fig. 3 is a hardware configuration diagram of the information processing device 200 in this embodiment. In the figure, a CPU 301 comprehensively controls each device connected via a bus. The CPU 301 reads out and executes processing steps and programs stored in a read-only memory (ROM) 302. An operating system (OS), each processing program according to this embodiment, device drivers, etc. are stored in the ROM 302, temporarily stored in a random access memory (RAM) 303, and appropriately executed by the CPU 301.

[0015] The input I / F 304 receives an input signal from the external device (imaging device) 210 in a format that can be processed by the information processing device 200. The output I / F 305 outputs an output signal to the external device (display device) 220 in a format that can be processed.

[0016] 2, the imaging device 210 includes an imaging section 211 and an imaging section 212, and inputs images acquired from each of them to the information processing device 200. In this embodiment, the image acquired by the imaging section 211 is an image for the left eye (image from the left viewpoint), and the image acquired by the imaging section 212 is an image for the right eye (image from the right viewpoint).

[0017] The image acquisition unit 201 acquires images captured by the imaging units 211 and 212 of the imaging device 210 as stereo images, and stores the acquired stereo images in the data storage unit 202 .

[0018] The data storage unit 202 stores the stereo images and virtual object data input from the image acquisition unit 201, as well as color and shape recognition information used for subject extraction.

[0019] The object extraction unit 203 extracts a specific object region from the stereo images. For example, color information of the object is registered in advance, and an area corresponding to the registered color information is extracted from each of the stereo images.

[0020] A background modification section 204 sets the area other than the object area extracted by the object extraction section 203 as a background area, and generates a background-modified stereo image by modifying the background area in the stereo image.

[0021] The corresponding point detection unit 205 uses the background-changed stereo images generated by the background change unit 204 to perform stereo matching for matching identical points between the stereo images.

[0022] A depth estimation unit 206 estimates the depth from the corresponding points detected by the corresponding point detection unit 205 based on triangulation.

[0023] The output information generating unit 207 performs appropriate processing according to the purpose of use, such as further performing drawing processing on the stereo image captured based on the depth estimated by the depth estimating unit 206. For example, a polygon model may be generated based on the depth, and a composite image in which an image and a virtual object are occlusion-expressed may be generated from the data of the virtual object stored in the data storage unit 202. Furthermore, it may be determined whether the three-dimensional position acquired from the depth is in contact with the virtual object, and the determination result may be displayed. The processing performed here is not particularly limited, and may be appropriately switched according to an instruction from the user, a program to be executed, etc. The data of the output image obtained as a result of the processing is output to the display device 220 and displayed.

[0024] 4 is an example of a flowchart showing a process in which the information processing device 200 changes the background region of a stereo image and estimates the depth. In the following, each process (step) will be explained with an S added to the beginning of each reference number.

[0025] In step S400, the image acquisition unit 201 acquires stereo images captured by the imaging units 211 and 212. The image acquisition unit 201 is, for example, a video capture card that acquires images obtained from the imaging units 211 and 212. The acquired stereo images are stored in the data storage unit 202.

[0026] In step S401, the object extraction unit 203 extracts an object region from each image of the stereo image stored in the data storage unit 202. For example, the characteristics of the object may be learned in advance by machine learning, and an area having the learned characteristics may be determined as the object region and extracted. For example, the color of the object may be registered and the object may be extracted. Here, the object region in the image is defined as the object region, and the area other than the object is defined as the background region.

[0027] In step S402, background modification unit 204 generates background-changed stereo images by filling in areas determined to be background areas by subject extraction unit 203 with a single color. Fig. 5 shows an example of an image when the background is changed by background modification unit 204 with a hand as the subject. Fig. 5(A) shows the result of background modification performed on Fig. 1(A), which is an image captured by the left camera, and Fig. 5(B) shows the result of background modification performed on Fig. 1(B), which is an image captured by the right camera. By generating background-changed stereo images with the background areas modified in this way, it is possible to eliminate the difference in structures in the background areas between images, which was a problem in Fig. 1.

[0028] In step S403, the corresponding point detection unit 205 uses a stereo matching process to detect corresponding points from a pair of background-changed stereo images, which are processed images. For this stereo matching process, for example, semi-global matching (SGM) as described in Non-Patent Document 1 may be used. Note that this embodiment is not limited to using SGM for stereo matching. A method may be used in which sampling points are drawn on the left eye image and epipolar lines (scanning lines) for associating the sampling points with the right eye image, correlation is calculated using local areas on the epipolar lines as clues, and the most highly correlated points are detected as corresponding points. Alternatively, a method may be used in which the matching cost between images is expressed as energy and the energy is optimized by graph cut.

[0029] In step S404, the depth estimation unit 206 determines the depth values ​​of the corresponding points using triangulation. That is, the depth values ​​of the corresponding points are determined based on the correspondence information of the corresponding points detected by the corresponding point detection unit 205, the relative position and orientation of the imaging units 211 and 212 of the imaging device 210, and camera internal parameters (lens distortion, perspective projection exchange information). Corresponding point information linking the depth value information of the corresponding points with the three-dimensional position of the imaging device is stored in the RAM 303.

[0030] (Embodiment 2) In the first embodiment, the case where the background region is filled with a single color is exemplified. For example, FIG. 6(B) shows an expanded search block range centered on a focus point 601 for stereo matching in FIG. 6(A), which is a background-changed stereo image, and FIG. 6(C) shows an expanded search block range centered on a focus point 602. In this way, by filling the background with a single color, The area around points 601 and 602 is This can lead to mismatching during stereo matching.

[0031] In this embodiment, in consideration of such a case, structural information of the object may be added to the background. That is, the object extraction unit 203 creates an image in which the extracted object region and background region are binarized, and the background change unit 204 performs a convolution operation with the filter shown in FIG. 7 to check whether the object region is nearby. judgement This filter is a filter that is slightly larger than the block of the SGM used for detection by the corresponding point detection unit 205, and outputs a negative value if the subject is to the left of the point of interest, and a positive value if the subject is to the right.

[0032] Here, FIG. 8(A) shows an image obtained by binarizing FIG. 6(A). FIG. 8(B) shows an image obtained by enlarging the filter range around a focus point 801, which is at the same position as the background region in FIG. 6(B), and FIG. 8(C) shows an image obtained by enlarging the filter range around a focus point 802, which is at the same position as the background region in FIG. 6(C). Looking at the vicinity of the focus point in a larger area than the search blocks in FIG. 6(B) and FIG. 6(C), it can be seen that FIG. 8(B) has an object on the right side, but FIG. 8(C) has no object. In such a case, when the binarized image is convoluted with the filter in FIG. 7, the background region in FIG. 8(B) has a value close to 0, and the background region in FIG. 8(C) has a negative value. That is, blocks that could not be distinguished in FIG. 6(B) and FIG. 6(C) can be distinguished by the corresponding point detection unit 205 due to the difference in the background region.

[0033] As described above, when the background region is filled with a single color, the corresponding point detection unit 205 may mistakenly detect very similar subject regions, but by adding structural information about the subject to the background region, it is possible to correctly detect these regions.

[0034] (Embodiment 3) In the first embodiment, the background region is filled with a single color, and in the second embodiment, the structure information of the subject is added to the background region. In contrast, in the present embodiment, the correspondence information between images (epipolar line information) is added to the background region. For example, rectification is performed on the stereo images acquired by the image acquisition unit 201 based on the relative positions and orientations of the imaging units 211 and 212 and the camera internal parameters. By utilizing the fact that the epipolar line is horizontal in the stereo images after rectification, the epipolar line information is added to the background. That is, when the image coordinates are expressed as (x, y) as in FIG. 9(A) which is an image for the left eye and FIG. 9(B) which is an image for the right eye, the background change unit 204 sets the background color based on the y coordinate of the background region, that is, the vertical position.

[0035] As described above, when the background region is filled with a single color, very similar subject regions that would be erroneously detected by the corresponding point detection unit 205 can be correctly detected by adding correspondence information between images to the background region.

[0036] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0037] 200 Information processing device 201 Image Acquisition Unit 202 Data storage unit 203 Subject extraction section 204 Background Change Section 205 Corresponding point detection unit 206 Depth Estimation Unit 207 Output information generation unit 210 Imaging device 220 Display device

Claims

1. An extraction means for extracting a subject area from each of two images captured from two viewpoints; a processing means for processing each of the two images based on a region of the subject; a detection means for detecting corresponding points from subject areas of the two images after processing by the processing means; an estimation means for estimating a depth of the subject from the two viewpoints based on the positions of the two viewpoints and the positions of the corresponding points in each of the two images, The information processing apparatus according to claim 1, wherein the processing means adds structural information of the subject to the two images.

2. 2. The information processing apparatus according to claim 1, wherein the processing means changes a color of an area other than the area of ​​the subject.

3. 3. The information processing apparatus according to claim 2, wherein the processing means paints areas other than the area of ​​the subject in a single color.

4. 2. The information processing apparatus according to claim 1, wherein said processing means adds to said two images, as structural information of said subject, a state of said subject in the vicinity of a point of interest in said image.

5. The information processing device according to claim 4, characterized in that the processing means adds to the two images, as structural information of the subject, the state of the subject in the vicinity of a point of interest in the image and a state in the vicinity of the point of interest which is globaler than the vicinity.

6. An extraction means for extracting a subject area from each of two images captured from two viewpoints; a processing means for processing each of the two images based on a region of the subject; a detection means for detecting corresponding points from subject areas of the two images after processing by the processing means; an estimation means for estimating a depth of the subject from the two viewpoints based on the positions of the two viewpoints and the positions of the corresponding points in each of the two images, The information processing apparatus according to claim 1, wherein the processing means adds, to the two images, information on the correspondence between the two images.

7. 7. The information processing apparatus according to claim 6, wherein said processing means adds information on an epipolar line to said two images as corresponding information between said two images.

8. The information processing apparatus according to claim 7 , wherein the processing means corrects the two images so that the epipolar line is horizontal, and sets a color of the area other than the area of ​​the subject based on a position in a vertical direction.

9. 9. The information processing apparatus according to claim 1, wherein the extraction means extracts the area of ​​the subject from each of the two images based on color information.

10. The information processing apparatus according to claim 1 , further comprising a generating unit that generates an output image based on the depth estimated by the estimating unit.

11. 11. The information processing apparatus according to claim 10, wherein the generating means generates an image by combining the two captured images with a virtual object based on the estimated depth.

12. 2. The information processing apparatus according to claim 1, further comprising a determination unit that determines whether the subject is in contact with a virtual object based on the depth estimated by the estimation unit.

13. An extraction step of extracting a subject area from each of two images captured from two viewpoints; a processing step of processing each of the two images based on a region of the subject; a detection step of detecting corresponding points from subject areas of the two images after the processing; an estimation step of estimating a depth of the subject from the two viewpoints based on the positions of the two viewpoints and the positions of the corresponding points in each of the two images, The information processing method according to the present invention, wherein in the processing step, structural information of the subject is added to the two images.

14. An extraction step of extracting a subject area from each of two images captured from two viewpoints; a processing step of processing each of the two images based on a region of the subject; a detection step of detecting corresponding points from subject areas of the two images after the processing; an estimation step of estimating a depth of the subject from the two viewpoints based on the positions of the two viewpoints and the positions of the corresponding points in each of the two images, The information processing method according to the present invention, wherein in the processing step, correspondence information between the two images is added to the two images.

15. A program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 12.

Images