Automatic video fusion method in complex scene

Abstract

The invention discloses an automatic video fusion method in a complex scene. The automatic video fusion method in the complex scene comprises the following steps: step 1, image preprocessing; step 2, depth estimation, region segmentation and plane fitting; 3, calculating a front-view projection map; 4, feature matching and parameter calculation; according to the automatic video fusion method in the miscellaneous scene, the fusion precision is improved through a calibration and distortion removal method; the complex scenes are classified by utilizing depth estimation and region segmentation, and fusion operation is independently performed on different classes, so that the fusion precision is improved; and finally, automatically calculating and marking matching information to realize automatic fusion.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, three-dimensional reconstruction, image correction and video fusion, and in particular to an automatic video fusion method in complex scenes. Background Art

[0002] There are two main traditional video fusion methods: one is to manually adjust the camera image posture (position and angle) in the renderer or rendering interface, and directly back-project the image onto the 3D model. The other is to manually select the same feature points based on the feature point information of the image and the 3D model, calculate the homography matrix based on the matching information, and fuse the camera image into the model;

[0003] Both of the above methods require some manual intervention (adjusting angles and selecting matching points), which is labor-intensive, complicated, and requires certain technical skills of the operator. In addition, they can only handle some simple scenes (approximately flat scenes, such as roads, etc.). When there are areas of different heights and shapes in the scene, they are usually unable to accurately fuse and calculate, resulting in large fusion offset errors. Summary of the invention

[0004] The purpose of the invention is to provide an automatic video fusion method in complex scenes, which solves the problems in the background technology.

[0005] The present invention is implemented in this way: an automatic video fusion method in a complex scene, the automatic video fusion method in a complex scene comprises the following steps: the automatic video fusion method in a complex scene comprises the following steps:

[0006] Step 1: Image preprocessing;

[0007] Step 2: Depth estimation, region segmentation and plane fitting;

[0008] Step 3: Orthographic projection map calculation;

[0009] Step 4: Feature matching and parameter calculation.

[0010] A further technical solution of the present invention is: in step one, the image preprocessing is: calibrating the camera, calculating the camera's internal distortion parameters; and dedistorting the image to eliminate the fusion inaccuracy problem caused by the distortion.

[0011] A further technical solution of the present invention is: in step 2, the depth estimation is: using a convolutional neural network (CNN) or a recurrent neural network (RNN) network to perform depth estimation on the video image and calculate the depth information of the image.

[0012] A further technical solution of the present invention is: in step 2, the region segmentation is: segmenting the image, dividing different contents in the image into different regions according to the scene and type; scene segmentation usually adopts deep learning methods, such as full convolutional networks, deep convolutional neural networks and other network models.

[0013] A further technical solution of the present invention is: in step 2, the plane fitting is: after obtaining the depth calculation result and the instance segmentation result of the image, plane fitting is performed on all N segmented objects using the depth map; for a certain instance object Ni, all 2D points it contains are: P2_uv, the corresponding depth is: P2_depth, for a certain pixel point i, its 3D coordinate in space is calculated according to the following formula:

[0014] P3D i =K -1 *di*P2 uvi

[0015] Among them, K is the camera intrinsic parameter matrix; di is the depth value corresponding to pixel i (in P2_depth); P2_uvi is the 2D point coordinate.

[0016]

[0017] After obtaining the 3D point coordinates P3D of all points in each instance object Ni, plane fitting is performed on the 3D points. Commonly used methods include the least squares method, SVD decomposition method (singular value decomposition) and Lagrange multiplier method. In this solution, the least squares method is used for plane fitting.

[0018] Assume the plane equation is: z = a0x + a1y + a2

[0019] For all 3D points, construct the least squares cost function:

[0020] S=min∑[(a0x i +a1y i +a2)-z i ] 2

[0021] Finally, the plane parameters a0, a1, a2 are calculated, and the distance d from all 3D points in the instance object to the plane is calculated to obtain the maximum distance dmax.

[0022] Instance object fusion: In actual situations, different instance objects may belong to the same plane or an approximate plane. Therefore, this solution uses a plane fusion method to fuse different instance objects of the same plane or a near plane. First, for any two example objects i and j, their plane equations zi and zj and the maximum plane distances dmaxi and dmaxj are calculated. Compare the normal vectors of the two planes. If the angle between the normal vectors is less than the threshold T, superimpose all 3D points of the two planes, fit the plane Zij, and calculate the maximum distance dmaxij. If the maximum distance dmaxij satisfies:

[0023] dmaxij<s*max(dmaxi,dmaxj)

[0024] Then instance objects i and j are considered to belong to the same plane. Where s is the scale factor.

[0025] Finally, all instance objects are traversed and all example objects on the same plane are merged to obtain the final plane set P.

[0026] A further technical solution of the present invention is: in step three, the orthographic projection map is calculated as follows: the 3D model to be fused is imported into the rendering engine, the camera perspective (orthographic perspective) is set, the model is rendered, and the orthographic projection map of the model is calculated.

[0027] A further technical solution of the present invention is: in step 4, the feature matching and parameter calculation are as follows: feature matching is performed on all plane regions of the plane set P in the image and the projection map, and the homography matrix between the plane and the map is calculated according to the feature matching result, and the mask of each plane is calculated according to the segmentation result for regional display; the homography matrix is ​​as follows:

[0028] Pmap=H*Pimg

[0029] Among them, H is the homography matrix, Pimg is the matching point in the image, and Pmap is the matching point of the projection map.

[0030] Finally, according to the relationship between the projection map and the 3D model, the image is mapped into the 3D model to achieve video fusion.

[0031] Beneficial effects of the present invention: The automatic video fusion method in complex scenes of the present invention improves the fusion accuracy by using calibration and dedistortion methods; and uses depth estimation and region segmentation to classify complex scenes, and performs fusion operations on different categories separately to improve the fusion accuracy. Finally, the annotation matching information is automatically calculated to achieve automatic fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1It is a calibration plate format diagram of an automatic video fusion method in a complex scene provided by the present invention.

[0033] Figure 2 It is a schematic diagram of calculation results of depth estimation of an automatic video fusion method in a complex scene provided by the present invention;

[0034] Figure 3 It is a schematic diagram of the segmentation result of the region segmentation of the automatic video fusion method in a complex scene provided by the present invention;

[0035] Figure 4 It is a main flow chart of an automatic video fusion method in a complex scene provided by the present invention. DETAILED DESCRIPTION

[0036] Embodiment 1: Figure 1-Figure 4 An automatic video fusion method in a complex scene is shown, and the automatic video fusion method in a complex scene includes the following steps:

[0037] Step 1: Image preprocessing;

[0038] Step 2: Depth estimation, region segmentation and plane fitting;

[0039] Step 3: Orthographic projection map calculation;

[0040] Step 4: Feature matching and parameter calculation;

[0041] In step 1, the image preprocessing is: calibrating the camera, calculating the camera's internal distortion parameters; and dedistorting the image to eliminate the fusion inaccuracy caused by the distortion; the calibration plate is in the form of AprilTag, such as Figure 1 The AprilTag calibration plate shown in the figure determines the image feature coordinates by detecting each small square. When the calibration plate image is blocked, each small square can also be identified individually. Collect multiple calibration plate images and calibrate the camera according to Zhang Zhengyou's method;

[0042] In step 2, the depth estimation is to use a convolutional neural network (CNN) or a recurrent neural network (RNN) network to perform depth estimation on the video image and calculate the depth information of the image. The calculation result is as follows: Figure 2 As shown;

[0043] In step 2, the region segmentation is as follows: segmenting the image, dividing different contents in the image into different regions according to the scene and type; scene segmentation usually adopts deep learning methods, such as full convolutional network (FCN), deep convolutional neural network (DCNN) and other network models; the segmentation result is as follows Figure 3 As shown;

[0044] In step 2, the plane fitting is as follows: after obtaining the depth calculation result and instance segmentation result of the image, plane fitting is performed on all N segmented objects using the depth map; for a certain instance object Ni, all 2D points it contains are: P2_uv, the corresponding depth is: P2_depth, for a certain pixel point i, its 3D coordinates in space are calculated according to the following formula:

[0045] P3D i =K -1 *di*P2 uvi

[0046] Among them, K is the camera intrinsic parameter matrix; di is the depth value corresponding to pixel i (in P2_depth); P2_uvi is the 2D point coordinate.

[0047]

[0048] After obtaining the 3D point coordinates P3D of all points in each instance object Ni, plane fitting is performed on the 3D points. Commonly used methods include the least squares method, SVD decomposition method (singular value decomposition) and Lagrange multiplier method. In this solution, the least squares method is used for plane fitting.

[0049] Assume the plane equation is: z = a0x + a1y + a2

[0050] For all 3D points, construct the least squares cost function:

[0051] S=min∑[(a0x i +a1y i +a2)-z i ] 2

[0052] Finally, the plane parameters a0, a1, a2 are calculated, and the distance d from all 3D points in the instance object to the plane is calculated to obtain the maximum distance dmax.

[0053] Instance object fusion: In actual situations, different instance objects may belong to the same plane or an approximate plane. Therefore, this solution uses a plane fusion method to fuse different instance objects of the same plane or a near plane. First, for any two example objects i and j, their plane equations zi and zj and the maximum plane distances dmaxi and dmaxj are calculated. Compare the normal vectors of the two planes. If the angle between the normal vectors is less than the threshold T, superimpose all 3D points of the two planes, fit the plane Zij, and calculate the maximum distance dmaxij. If the maximum distance dmaxij satisfies:

[0054] dmaxij<s*max(dmaxi,dmaxj)

[0055] Then instance objects i and j are considered to belong to the same plane. Where s is the scale factor.

[0056] Finally, traverse all instance objects and merge all example objects on the same plane to obtain the final plane set P;

[0057] In step 3, the orthographic projection map is calculated as follows: the 3D model to be fused is imported into the rendering engine, the camera viewing angle (orthographic viewing angle) is set, the model is rendered, and the orthographic projection map of the model is calculated;

[0058] In step 4, the feature matching and parameter calculation are as follows: feature matching is performed on all plane regions of the plane set P in the image and the projection map, and the homography matrix between the plane and the map is calculated based on the feature matching results, and the mask of each plane is calculated based on the segmentation results for regional display; the homography matrix is ​​as follows:

[0059] Pmap=H*Pimg

[0060] Among them, H is the homography matrix, Pimg is the matching point in the image, and Pmap is the matching point of the projection map.

[0061] Finally, according to the relationship between the projection map and the 3D model, the image is mapped into the 3D model to achieve video fusion.

[0062] Beneficial effects of the present invention: The automatic video fusion method in complex scenes of the present invention improves the fusion accuracy by using calibration and dedistortion methods; and uses depth estimation and region segmentation to classify complex scenes, and performs fusion operations on different categories separately to improve the fusion accuracy. Finally, the annotation matching information is automatically calculated to achieve automatic fusion.

[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. The automatic video fusion method in complex scenes is characterized by: The automatic video fusion method in the complex scene includes the following steps: Step 1: Image preprocessing; Step 2: Depth estimation, region segmentation and plane fitting; Step 3: Orthographic projection map calculation; Step 4: Feature matching and parameter calculation.

2. The automatic video fusion method in complex scenes according to claim 1, characterized in that: In step 1, the image preprocessing is to calibrate the camera and calculate the intrinsic distortion parameters of the camera; and to dedistort the image to eliminate the fusion inaccuracy caused by the distortion.

3. The automatic video fusion method in complex scenes according to claim 2, characterized in that: In step 2, the depth estimation is: using a convolutional neural network (CNN) or a recurrent neural network (RNN) network to perform depth estimation on the video image and calculate the depth information of the image.

4. The automatic video fusion method in complex scenes according to claim 3, characterized in that: In step 2, the region segmentation is: segmenting the image, and dividing different contents in the image into different regions according to scenes and types.

5. The automatic video fusion method in complex scenes according to claim 4, characterized in that: In step 2, the plane fitting is: after obtaining the depth calculation result and the instance segmentation result of the image, plane fitting is performed on all N segmented objects using the depth map.

6. The automatic video fusion method in complex scenes according to claim 5, characterized in that: In step three, the orthographic projection map is calculated as follows: the 3D model to be fused is imported into the rendering engine, the camera viewing angle (orthographic viewing angle) is set, the model is rendered, and the orthographic projection map of the model is calculated.

7. The automatic video fusion method in complex scenes according to claim 6, characterized in that: In step 4, the feature matching and parameter calculation are as follows: feature matching is performed on all plane areas of the plane set P in the image with the projection map, and the homography matrix between the plane and the map is calculated based on the feature matching results, and the mask of each plane is calculated based on the segmentation results for regional display.

Images