Indoor stereoscopic video fusion method based on multiple planes

Abstract

The invention discloses an indoor stereoscopic video fusion method based on multiple planes. The indoor stereoscopic video fusion method based on multiple planes comprises the following steps: step 1, image distortion removal; step 2, straight line detection; 3, selecting multi-plane feature points; step 4, fusion parameter calculation; according to the indoor stereoscopic video fusion method based on the multiple planes, under the condition that it is assumed that all indoor scenes meet the Manhattan world hypothesis, calculation is carried out through point and line information of the multiple planes of the ground and the wall, fusion parameters of all the planes are obtained, and finally all the planes are merged and fused with a three-dimensional model.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, three-dimensional reconstruction, image correction and video fusion, and in particular to a multi-plane-based indoor stereoscopic video fusion method. Background Art

[0002] There are two main traditional video fusion methods: one is to manually adjust the camera image posture (position and angle) in the renderer or rendering interface, and directly back-project the image onto the 3D model. The other is to manually select the same feature points based on the feature point information of the image and the 3D model, calculate the homography matrix based on the matching information, and fuse the camera image into the model;

[0003] Both of the above methods require 3D reconstruction of the scene to obtain a relatively accurate 3D model, and then perform video fusion, which is usually applicable to outdoor scenes. For video fusion of indoor scenes, it is difficult to obtain an accurate 3D model with texture. In addition, video fusion is usually based on the plane assumption (the fusion scene is a scene that is approximately flat), which is usually difficult to meet in indoor scenes (there are many scenes such as indoor walls). Without an accurate model, video fusion and projection cannot be performed in indoor scenes. Summary of the invention

[0004] The purpose of the invention is to provide a multi-plane based indoor stereoscopic video fusion method to solve the problems mentioned in the background technology.

[0005] The present invention is implemented in this way: a method for indoor stereoscopic video fusion based on multiple planes, the method for indoor stereoscopic video fusion based on multiple planes comprises the following steps:

[0006] Step 1: Image dedistortion;

[0007] Step 2: straight line detection;

[0008] Step 3: Multi-plane feature point selection;

[0009] Step 4: Fusion parameter calculation.

[0010] A further technical solution of the present invention is: the image dedistortion in the step 1 is: using the camera intrinsic parameters to correct the image distortion and remove the influence of lens distortion on fusion; the camera intrinsic parameters can be obtained by collecting multiple calibration plate images and calibrating the camera according to the Zhang Zhengyou method.

[0011] A further technical solution of the present invention is: the straight line detection in the step 2 is: extracting line features from the image to obtain all line segment information L in the image; the straight line detection method can adopt the current mainstream methods such as LSD (Line SegmentDectctor), CannyLines (CannyLines: AParameter-Free Line Segment Detector) and the like.

[0012] A further technical solution of the present invention is: the multi-plane feature points in step 3 are selected as follows: according to the plane characteristics, the plane is divided into four types, namely, ground, front, left and right. And 4 points are selected for each plane for calculation; ground: 4 quadrilateral ground points are selected; front, left and right: two ground points and two points on the wall are selected.

[0013] A further technical solution of the present invention is: the fusion parameter calculation in step 4 is as follows: according to the line segment detection result in step 2, three vanishing points of the image are calculated, namely, horizontal, vertical, and front and back, to obtain vk; after respectively calculating the homography matrices of the ground and three walls, the images are respectively transformed to obtain target images of the four surfaces, and the target images are fused to obtain the final three-dimensional model; the coordinates Vk of the vanishing point vk in the 3D space are calculated according to the following formula:

[0014]

[0015] For a ground point, calculate its coordinates in 3D space:

[0016]

[0017] For floor type: In the floor type, all points are ground points. According to the four ground space 3D points, P1, P2, P3, P4, select two points to set the x-axis and y-axis. And calculate the x-axis width x_size, and the y-axis width y_size. According to the width and height, calculate the target point:

[0018] [[0.,0.],[x_size,0],[x_size,y_size],[0,y_size]]

[0019] According to the target point and the four pixels selected in the image, the homography matrix H is calculated and used to correct the image.

[0020] For wall type: In the wall type, there are two ground points and two wall points. The 3D point calculation method of the ground point is based on the above method. The 3D point calculation method of the wall point is as follows:

[0021]

[0022] Where h and λ are unknowns, V1 is the 3D coordinate of the vanishing point, p2 is the 2D coordinate of the wall, and p1 is the 2D coordinate of the ground.

[0023] The homography matrix is ​​as follows:

[0024] Pmap=H*Pimg

[0025] Among them, H is the homography matrix, Pimg is the point in the image, and Pmap is the calculated target point.

[0026] Beneficial effects of the present invention: The multi-plane based indoor stereo video fusion method of the present invention, assuming that all indoor scenes satisfy the "Manhattan world hypothesis", calculates through the point and line information of multiple planes on the ground and walls to obtain the fusion parameters of each plane, and finally merges each plane and fuses it with the three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flowchart of the multi-plane-based indoor stereoscopic video fusion method provided by the present invention;

[0028] Figure 2 It is a straight line detection result diagram of the multi-plane-based indoor stereo video fusion method provided by the present invention;

[0029] Figure 3 It is a schematic diagram of ground selection points of the multi-plane-based indoor stereo video fusion method provided by the present invention;

[0030] Figure 4 It is a schematic diagram of the front selection points of the multi-plane-based indoor stereoscopic video fusion method provided by the present invention;

[0031] Figure 5 Schematic diagram of the left selection point of the multi-plane-based indoor stereo video fusion method provided by the present invention;

[0032] Figure 6 It is a schematic diagram of right selection points of the multi-plane-based indoor stereoscopic video fusion method provided by the present invention;

[0033] Figure 7 It is a schematic diagram of a ground target after conversion according to the multi-plane-based indoor stereo video fusion method provided by the present invention;

[0034] Figure 8 It is a schematic diagram of the front target after conversion of the multi-plane-based indoor stereo video fusion method provided by the present invention;

[0035] Fig. 9 is a schematic diagram of a left target after conversion in the multi-plane-based indoor stereo video fusion method provided by the present invention;

[0036] Fig.10 is a schematic diagram of a right target after conversion in the multi-plane-based indoor stereo video fusion method provided by the present invention;

[0037] Fig.11 It is a schematic diagram of the final result after fusion of the multi-plane based indoor stereo video fusion method provided by the present invention. DETAILED DESCRIPTION

[0038] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0039] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention, so they have no substantial technical significance. Any modification of the structure, change of the proportion relationship or adjustment of the size should still fall within the scope of the technical content disclosed by the present invention without affecting the effects and purposes that can be achieved by the present invention. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" quoted in this specification are only for the convenience of description, and are not used to limit the scope of the implementation of the present invention. The change or adjustment of their relative relationship should also be regarded as the scope of the implementation of the present invention without substantially changing the technical content.

[0040] Embodiment 1: Figure 1 The invention shows a method for indoor stereoscopic video fusion based on multiple planes, which comprises the following steps:

[0041] Step 1: Image dedistortion;

[0042] Step 2: straight line detection;

[0043] Step 3: Multi-plane feature point selection;

[0044] Step 4: Fusion parameter calculation.

[0045] A further technical solution of the present invention is: the image dedistortion in step 1 is: using the camera intrinsic parameter to correct the image distortion and remove the influence of lens distortion on fusion; the camera intrinsic parameter can be obtained by collecting multiple calibration plate images and calibrating the camera according to Zhang Zhengyou's method;

[0046] The line detection in step 2 is: extracting line features from the image to obtain all line segment information L in the image; the line detection method can adopt the current mainstream methods such as LSD (Line Segment Dectctor) and CannyLines (CannyLines: A Parameter-Free Line Segment Detector);

[0047] The multi-plane feature points in step 3 are selected as follows: according to the plane characteristics, the plane is divided into four types, ground, front, left and right. And 4 points are selected for each plane for calculation; ground: select 4 quadrilateral ground points; front, left, right: select two ground points and two points on the wall;

[0048] The fusion parameter calculation in step 4 is as follows: according to the line segment detection result in step 2, the three vanishing points of the image are calculated, namely, horizontal, vertical, and front and back, to obtain vk; after the homography matrices of the ground and the three walls are calculated respectively, the images are transformed respectively to obtain the target images of the four surfaces, and the target images are fused to obtain the final three-dimensional model; the coordinates Vk of the vanishing point vk in the 3D space are calculated according to the following formula:

[0049]

[0050] For a ground point, calculate its coordinates in 3D space:

[0051]

[0052] For floor type: In the floor type, all points are ground points. According to the four ground space 3D points, P1, P2, P3, P4, select two points to set the x-axis and y-axis. And calculate the x-axis width x_size, and the y-axis width y_size. According to the width and height, calculate the target point:

[0053] [[0.,0.],[x_size,0],[x_size,y_size],[0,y_size]]

[0054] According to the target point and the four pixels selected in the image, the homography matrix H is calculated and used to correct the image.

[0055] For wall type: In the wall type, there are two ground points and two wall points. The 3D point calculation method of the ground point is based on the above method. The 3D point calculation method of the wall point is as follows:

[0056]

[0057] Where h and λ are unknowns, V1 is the 3D coordinate of the vanishing point, p2 is the 2D coordinate of the wall, and p1 is the 2D coordinate of the ground.

[0058] The homography matrix is ​​as follows:

[0059] Pmap=H*Pimg

[0060] Wherein, H is the homography matrix, Pimg is the point in the image, and Pmap is the calculated target point. The multi-plane based indoor stereo video fusion method of the present invention assumes that all indoor scenes meet the "Manhattan world hypothesis", calculates through the point and line information of multiple planes on the ground and wall, obtains the fusion parameters of each plane, and finally merges each plane and fuses it with the three-dimensional model.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-plane based indoor stereo video fusion method, characterized in that: The multi-plane based indoor stereo video fusion method comprises the following steps: Step 1: Image dedistortion; Step 2: straight line detection; Step 3: Multi-plane feature point selection; Step 4: Fusion parameter calculation.

2. The method for indoor stereoscopic video fusion based on multiple planes according to claim 1, characterized in that: The image dedistortion in step 1 is to use the camera internal parameters to perform distortion correction on the image to remove the influence of lens distortion on fusion.

3. The method for indoor stereoscopic video fusion based on multiple planes according to claim 1, characterized in that: The straight line detection in step 2 is: extracting line features from the image to obtain all line segment information L in the image.

4. The method for indoor stereoscopic video fusion based on multiple planes according to claim 3, characterized in that: The multi-plane feature points in step 3 are selected as follows: according to the plane characteristics, the planes are divided into four types: ground, front, left and right. And 4 points are selected for each plane for calculation.

5. The method for indoor stereoscopic video fusion based on multiple planes according to claim 4, characterized in that: The fusion parameter calculation in step 4 is as follows: according to the line segment detection result in step 2, the three vanishing points of the image are calculated, horizontally, vertically, and front and back, to obtain vk; after respectively calculating the homography matrices of the ground and the three walls, the images are transformed respectively to obtain the target images of the four surfaces, and the target images are fused to obtain the final three-dimensional model.

Images