A multi-view video stitching method based on deep learning

Abstract

The application discloses a kind of multi-view video splicing methods based on deep learning, steps are as follows: first, using Airsim simulator is collected at the image and depth data of set virtual public viewpoint, generates the dataset for video splicing task, and the image is cylindrical projection and so on Preprocessing.Then, using convolutional neural network designs artifact elimination module and smooth transition module respectively, the former considers the feature correlation of overlapping area, and the overlapping area is aligned by viewpoint regression to eliminate the artifact after fusion;The latter according to the feature information of image, the deformation rule obtained in the overlapping area is propagated to the non-overlapping area, to guide the smooth transition between regions to improve visual perception.Finally, according to the predicted displacement field, the original viewpoint image is distorted and transformed, and the splicing result is obtained by weighted linear fusion.The application can achieve real-time splicing performance while removing splicing artifacts, and can meet the online splicing needs in practical applications.

Description

Technical Field

[0001] This invention relates to a video stitching technology, belonging to the field of computer vision technology. Background Technology

[0002] Video stitching technology holds significant theoretical research value and plays a crucial role in various application fields such as virtual reality, security monitoring, autonomous driving, video conferencing, and drone aerial photography. Video stitching is typically used to synthesize two or more videos captured by cameras in different poses. It can reduce the requirements for video acquisition equipment and provide a wider field of view. Although image and video stitching has a long history of research, existing video stitching methods are not perfect. Long computation time, poor performance in wide-baseline, high-parallax scenes, and insufficient algorithm robustness are some of the challenges currently faced. Algorithms based on global homography alignment are commonly used in video stitching. When the camera optical centers are basically aligned or the scene depth changes are small, they are not affected by parallax and can achieve good results; otherwise, obvious artifacts will occur. However, in practical applications, it is difficult to achieve the condition of perfectly aligned camera optical centers, and distributed camera arrangements are required in some scenarios such as vehicle surround-view systems. To reduce artifacts, methods based on optimal stitching lines are commonly used, but these methods may produce transition irregularities, and the computational efficiency of minimizing the energy function is still relatively low.

[0003] The development of deep learning technology has provided a completely new dimension for video and image stitching technology, and adopting appropriate methods can help improve the quality of the stitched video. Convolutional Neural Networks (CNNs) have powerful feature extraction capabilities. Using CNNs instead of traditional feature extraction methods can provide better robustness in scenarios such as low lighting, low texture, or repetitive textures. Correspondingly, there are also deep learning-based homography estimation methods applied to stitching tasks for images with small parallax. However, the lack of suitable datasets is a challenge for applying deep learning methods to video and image stitching tasks. Some methods use parallax-free synthetic datasets, which are often inconsistent with real-world application scenarios. Summary of the Invention

[0004] Technical Problem: In view of the above-mentioned prior art, this invention proposes a multi-view video stitching method based on deep learning, which can eliminate artifacts caused by parallax; improve robustness in challenging scenarios such as low light, low texture or repetitive texture; and have high computational efficiency to meet the needs of online real-time stitching in practical applications.

[0005] Technical solution: To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0006] A deep learning-based multi-view video stitching method includes the following steps:

[0007] Step 1: Collect images and depth data at the set virtual common viewpoint, generate a dataset for video stitching tasks, and perform cylindrical projection preprocessing on the images in the dataset according to the camera viewpoint.

[0008] Step 2: Obtain 3D information of the scene from the depth data and convert it to obtain pixel-level displacement field.

[0009] Step 3: Design an artifact removal module using a convolutional neural network, considering the feature correlation of overlapping regions, align the overlapping regions, and return the viewpoint to the overlapping virtual optical center to eliminate the artifacts generated after fusion.

[0010] Step 4: Design a smooth transition module using a convolutional neural network. Based on the image's feature information, propagate the deformation rules of the overlapping areas to the non-overlapping areas to guide the smooth transition between regions and reduce the visual disjointedness.

[0011] Step 5: Transform the original viewpoint image according to the displacement field distortion transformation, and perform weighted linear fusion to obtain the stitching result.

[0012] Furthermore, the specific method of step 1 is as follows:

[0013] Video stitching from different viewpoints is treated as a viewpoint regression problem. Images acquired at the original viewpoint are mapped to an arbitrary common virtual viewpoint to handle parallax caused by non-coincident camera optical centers. To build an ideal optical center coincidence model at the virtual viewpoint and obtain reliable depth data, a camera model is built in a virtual 3D environment using the Airsim simulator, generating a dataset for training.

[0014] Furthermore, the specific method for step 2 is as follows:

[0015] The pixel displacement field is obtained by transforming the depth information in the scene. After acquiring the depth information corresponding to the two cameras at the virtual viewpoint, the 3D coordinates of the pixels are obtained. The image at the virtual viewpoint is transformed to the original viewpoint, and the displacement field flow during the viewpoint transformation process is calculated using solid geometry. gt .

[0016] Furthermore, the specific method for step 3 is as follows:

[0017] In video stitching tasks, overlapping areas are generally few. To minimize this by filtering out invalid areas and reducing computational load, the maximum binary mask M containing the overlapping areas is obtained based on the camera configuration. ov_max Extract this part from the input image and then input it into the current module.

[0018] For potential overlapping regions, an encoder-decoder structure was designed. The encoder stacks two images along their channel dimensions and extracts features using a series of convolutional layers for downsampling. The decoder consists of a series of upsampling layers and convolutional layers, employing skip connections. The features of the corresponding layer in the encoder and the displacement field output from the previous layer in the decoder are input and progressively upsampled for optimization. When a 1 / 4 resolution displacement field is obtained, it is directly upsampled using bilinear interpolation to obtain a displacement field for the overlapping region of the same size as the input resolution.

[0019] To train the artifact removal module, displacement field loss, content loss, and perceptual layer loss were defined.

[0020] By transforming the binary mask of the original viewpoint using the displacement fields of the two images, we obtain the binary mask M of the actual overlapping region. ov Based on the pixel displacement field flow predicted by the network O Construct an L1 loss function for the overlapping regions:

[0021]

[0022] Content loss calculation of the image at the virtual viewpoint I gt and the network output image I O L1Loss at the overlapping region:

[0023]

[0024] The purpose of the perceptual layer loss is to keep the features of the transformed image as consistent as possible. Deep, high-level semantic features are extracted using the conv5_3 layer in the pre-trained VGG-19 feature extraction network. This process is defined as F(·). The MSE Loss on this layer is calculated, and the overlapping region mask M is used. ov Extract:

[0025]

[0026] The total loss function for this module is:

[0027]

[0028] Furthermore, the specific method for step 4 is as follows:

[0029] The smooth transition module aims to create a smoother connection between overlapping and non-overlapping regions, resulting in a better visual experience. For non-overlapping regions, the design goal is to propagate a displacement field from the overlapping region to the non-overlapping region, guided by the image features of the original viewpoint. To achieve this propagation, the input consists of the original viewpoint image and the displacement field of the overlapping region predicted in the previous stage. The original viewpoint image is set to 1 / 4 resolution to accommodate the size of the displacement field. This submodule comprises a series of convolutional layers and residual blocks. Dilated convolution is used in the residual blocks to expand the receptive field; a total of six residual blocks are used, with dilation parameters set to [1,2,4,8,1,1]. This regression structure predicts the pixel displacement fields of all regions in both images.

[0030] To train this module, displacement field loss, displacement field consistency loss, and perception layer loss were defined.

[0031] In non-overlapping regions, the parts closer to the overlapping region should be given more attention, while the parts farther from the overlapping region should receive less attention. Therefore, it is not suitable to apply the same weight to each pixel using a loss function. Consider constructing the weights W using a Gaussian function. k The displacement field loss is obtained as follows:

[0032]

[0033]

[0034] The displacement field consistency loss function is used to ensure that the output of the second module in the overlapping region is consistent with the output of the first module.

[0035]

[0036] For the perceptual layer loss, the MSE Loss on the conv5_3 layer of the VGG-19 network is also calculated. The input is the deformed image, and the binary mask containing the original viewpoint image content at the virtual viewpoint is M.

[0037]

[0038] The overall loss function for this module is defined as:

[0039]

[0040] Furthermore, the specific method for step 5 is as follows:

[0041] Based on the final prediction results, the original view is transformed, and the two images are subjected to a simple weighted linear fusion process to obtain the stitched result I. o :

[0042] Io =W·warp(I A ,flow A )+(1-W)·warp(I B ,flow B )

[0043] Among them, flowg A flowg B represents the output pixel displacement field, warp is the transformation function, and W is the set linear fusion weight.

[0044] Beneficial Effects: This invention proposes a deep learning-based multi-view video stitching method that utilizes deep convolutional neural networks to process video stitching, providing a novel approach to solving this type of problem. This method can be applied to wide-baseline camera arrangements, eliminating parallax artifacts through viewpoint regression and improving the quality of the stitched video. Furthermore, due to the advantages of convolutional neural networks in extracting image features, it exhibits better robustness than traditional methods in challenging scenarios such as low-light, low-texture, or repetitive textures. The designed module has a short runtime, meeting the performance requirements for online real-time stitching. Attached Figure Description

[0045] Figure 1 This is an overall flowchart of the deep learning-based multi-view video stitching method provided by the present invention.

[0046] Figure 2 This is a schematic diagram of the camera arrangement in this invention.

[0047] Figure 3 This is a camera configuration diagram in the virtual 3D environment of this invention.

[0048] Figure 4 This refers to the design of the overall network architecture in this invention.

[0049] Figure 5 This is a comparison of the stitching results of the present invention under different methods. Column a is the reference true value, column b is the multi-band fusion method, column c is the APAP method, and column d is the method of this patent. The stitching results in each column are shown from top to bottom under different test scenarios. Detailed Implementation

[0050] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0051] As shown in the figure, a deep learning-based multi-view video stitching method includes the following steps:

[0052] Step 1: Collect images and depth data at the set virtual common viewpoint, generate a dataset for video stitching tasks, and perform cylindrical projection preprocessing on the images in the dataset according to the camera viewpoint.

[0053] Step 2: Obtain 3D information of the scene from the depth data and convert it to obtain pixel-level displacement field.

[0054] Step 3: Design an artifact removal module using a convolutional neural network, considering the feature correlation of overlapping regions, align the overlapping regions, and return the viewpoint to the overlapping virtual optical center to eliminate the artifacts generated after fusion.

[0055] Step 4: Design a smooth transition module using a convolutional neural network. Based on the image's feature information, propagate the deformation rules of the overlapping areas to the non-overlapping areas to guide the smooth transition between regions and reduce the visual disjointedness.

[0056] Step 5: Transform the original viewpoint image according to the displacement field distortion transformation, and perform weighted linear fusion to obtain the stitching result.

[0057] In this embodiment, the specific method of step 1 is as follows:

[0058] Video stitching from different viewpoints is treated as a viewpoint regression problem. Images acquired at the original viewpoint are mapped to an arbitrary common virtual viewpoint to handle parallax caused by non-coincident camera optical centers. To build an ideal optical center coincidence model at the virtual viewpoint and obtain reliable depth data, a camera model is built in a virtual 3D environment using the Airsim simulator, generating a dataset for training.

[0059] In this embodiment, the specific method of step 2 is as follows:

[0060] The pixel displacement field is obtained by transforming the depth information in the scene. After obtaining the depth information corresponding to the two cameras at the virtual viewpoint, the 3D coordinates of the pixels can be obtained. The image at the virtual viewpoint is transformed to the original viewpoint, and the displacement field flow during the viewpoint transformation process can be calculated using solid geometry. gt .

[0061] In this embodiment, the specific method of step 3 is as follows:

[0062] In video stitching tasks, overlapping areas are generally few. To minimize this by filtering out invalid areas and reducing computational load, the maximum binary mask M containing the overlapping areas is obtained based on the camera configuration. ov_max Extract this part from the input image and then input it into the current module.

[0063] For potential overlapping regions, an encoder-decoder structure was designed. The encoder stacks two images along their channel dimensions and extracts features using a series of convolutional layers for downsampling. The decoder consists of a series of upsampling layers and convolutional layers, employing skip connections. The features of the corresponding layer in the encoder and the displacement field output from the previous layer in the decoder are input and progressively upsampled for optimization. When a 1 / 4 resolution displacement field is obtained, it is directly upsampled using bilinear interpolation to obtain a displacement field for the overlapping region of the same size as the input resolution.

[0064] To train the artifact removal module, displacement field loss, content loss, and perceptual layer loss were defined.

[0065] By transforming the binary mask of the original viewpoint using the displacement fields of the two images, we obtain the binary mask M of the actual overlapping region. ov Based on the pixel displacement field flow predicted by the network O Construct an L1 loss function for the overlapping regions:

[0066]

[0067] Content loss calculation of the image at the virtual viewpoint I gt and the network output image I O L1Loss at the overlapping region:

[0068]

[0069] The purpose of the perceptual layer loss is to keep the features of the transformed image as consistent as possible. Deep, high-level semantic features are extracted using the conv5_3 layer in the pre-trained VGG-19 feature extraction network. This process is defined as F(·). The MSE Loss on this layer is calculated, and the overlapping region mask M is used. ov Extract:

[0070]

[0071] The total loss function for this module is:

[0072]

[0073] In this embodiment, the specific method of step 4 is as follows:

[0074] The smooth transition module aims to create a smoother connection between overlapping and non-overlapping regions, resulting in a better visual experience. For non-overlapping regions, the design goal is to propagate a displacement field from the overlapping region to the non-overlapping region, guided by the image features of the original viewpoint. To achieve this propagation, the input consists of the original viewpoint image and the displacement field of the overlapping region predicted in the previous stage. The original viewpoint image is set to 1 / 4 resolution to accommodate the size of the displacement field. This submodule comprises a series of convolutional layers and residual blocks. Dilated convolution is used in the residual blocks to expand the receptive field; a total of six residual blocks are used, with dilation parameters set to [1,2,4,8,1,1]. This regression structure predicts the pixel displacement fields of all regions in both images.

[0075] To train this module, displacement field loss, displacement field consistency loss, and perception layer loss were defined.

[0076] In non-overlapping regions, the parts closer to the overlapping region should be given more attention, while the parts farther from the overlapping region should receive less attention. Therefore, it is not suitable to apply the same weight to each pixel using a loss function. Consider constructing the weights W using a Gaussian function. k The displacement field loss is obtained as follows:

[0077]

[0078]

[0079] The displacement field consistency loss function is used to ensure that the output of the second module in the overlapping region is consistent with the output of the first module.

[0080]

[0081] For the perceptual layer loss, the MSE Loss on the conv5_3 layer of the VGG-19 network is also calculated. The input is the deformed image, and the binary mask containing the original viewpoint image content at the virtual viewpoint is M.

[0082]

[0083] The overall loss function for this module is defined as:

[0084]

[0085] In this embodiment, the specific method of step 5 is as follows:

[0086] Based on the final prediction results, the original view is transformed, and the two images are subjected to a simple weighted linear fusion process to obtain the stitched result I. o :

[0087] Io =W·warp(I A ,flow A )+(1-W)·warp(I B ,flow B )

[0088] Among them, flow A flow B represents the output pixel displacement field, warp is the transformation function, and W is the set linear fusion weight.

[0089] Example

[0090] The deep learning-based multi-view video stitching method of the present invention sets up the following in the Airsim simulator: Figure 3 The camera configuration shown demonstrates the construction of a camera acquisition model. The camera's field of view (FOV) was set to 90 degrees, its resolution to 1280x720, and the angle between two cameras was 60 degrees, resulting in an image overlap of less than 33%. A total of four cameras were placed. The baseline distance between the first two cameras was 1 meter, used to capture the original viewpoint image. The latter two were placed at the same virtual viewpoint to capture the set ground truth image and depth data. This method synthesized over a thousand sets of images from multiple scene maps and weather conditions to construct a dataset for training the network. During training, only the artifact removal module was trained first, with its parameters fixed, before training the smooth transition module.

[0091] Experiment: Test video clips were collected in map scenes different from the training data. Some frames were selected from the video stitching results and compared with existing stitching methods. The proposed method achieved good stitching results, verifying its effectiveness.

[0092] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-view video stitching method based on deep learning, characterized in that, Includes the following steps: Step 1: Collect images and depth data at the set virtual common viewpoint, generate a dataset for video stitching tasks, and perform cylindrical projection preprocessing on the images in the dataset according to the camera viewpoint. Step 2: Obtain 3D information of the scene from the depth data and convert it to obtain pixel-level displacement field; Step 3: Design an artifact removal module using a convolutional neural network, consider the feature correlation of overlapping regions, align the overlapping regions, and return the viewpoint to the overlapping virtual optical center to eliminate the artifacts generated after fusion. Step 4: Design a smooth transition module using a convolutional neural network. Based on the image's feature information, propagate the deformation rules of the overlapping areas to the non-overlapping areas to guide the smooth transition between regions and reduce the visual disjointedness. Step 5: Transform the original viewpoint image according to the displacement field distortion transformation, and perform weighted linear fusion to obtain the stitched result; The specific method for step 3 is as follows: In video stitching tasks, overlapping areas are few. To reduce the filtering of invalid areas and decrease computational load, the maximum binary mask M containing the overlapping areas is obtained based on the camera configuration. ov_max Extract this part from the input image and then input it into the current module; For possible overlapping regions, an encoder-decoder structure was designed. For the encoder, two images are stacked together along the channel dimension, and a series of convolutional layers are used to downsample and extract features. The decoder consists of a series of upsampling layers and convolutional layers, with skip connections used. The features of the corresponding layer of the encoder and the displacement field output of the previous layer of the decoder are input and upsampled step by step for optimization. When a displacement field with a resolution of 1 / 4 is obtained, it is directly upsampled through bilinear interpolation to obtain the displacement field of the overlapping region with the same size as the input resolution. To train the artifact removal module, displacement field loss, content loss, and perceptual layer loss were defined. By transforming the binary mask of the original viewpoint using the displacement fields of the two images, we obtain the binary mask M of the actual overlapping region. ov Based on the pixel displacement field flow predicted by the network O Construct an L1 loss function for the overlapping regions: Among them, flow gt This represents the displacement field during the viewpoint transformation process; Content loss calculation of the image at the virtual viewpoint I gt and the network output image I O L1 Loss at the overlapping region: The purpose of the perceptual layer loss is to keep the features of the transformed image as consistent as possible. Deep, high-level semantic features are extracted using the conv5_3 layer in the pre-trained VGG-19 feature extraction network. This process is defined as F(·). The MSE Loss on this layer is calculated, and the overlapping region mask M is used. ov Extract: The total loss function for this module is:

2. The deep learning-based multi-view video stitching method according to claim 1, characterized in that, The specific method for step 1 is as follows: Video stitching from different viewpoints is treated as a viewpoint regression problem. The images captured from the original viewpoint are mapped to an arbitrary common virtual viewpoint to handle parallax caused by the non-coincidence of camera optical centers. To build an ideal optical center coincidence model at a virtual viewpoint and obtain reliable depth data, a camera model was built in a virtual 3D environment using the Airsim simulator, generating a dataset for training.

3. The deep learning-based multi-view video stitching method according to claim 1, characterized in that, The specific method for step 2 is as follows: The pixel displacement field is obtained by transforming the depth information in the scene; after acquiring the depth information corresponding to the two cameras at the virtual viewpoint, the 3D coordinates of the pixels are obtained; the image at the virtual viewpoint is transformed to the original viewpoint, and the displacement field flow during the viewpoint transformation process is calculated using solid geometry. gt .

4. The deep learning-based multi-view video stitching method according to claim 1, characterized in that, The specific method for step 4 is as follows: The smooth transition module aims to create a smooth connection between overlapping and non-overlapping regions, resulting in a visually appealing image. For non-overlapping regions, the design focuses on propagating a displacement field from the overlapping region to the non-overlapping region, guided by the image features of the original viewpoint. To achieve this propagation, the input consists of the original viewpoint image and the displacement field of the overlapping region predicted in the previous stage. The original viewpoint image is set to 1 / 4 resolution to accommodate the size of the displacement field. This submodule comprises a series of convolutional layers and residual blocks. Dilated convolution is used in the residual blocks to expand the receptive field. A total of 6 residual blocks are used, with dilation parameters set to [1,2,4,8,1,1]. This regression structure is used to predict the pixel displacement fields of all regions in both images. To train this module, displacement field loss, displacement field consistency loss, and perception layer loss were defined. In non-overlapping regions, the parts closer to the overlapping region should be given more attention, while the parts farther from the overlapping region should be given less attention. Therefore, it is not suitable to apply the same weight to each pixel; consider using a Gaussian function to construct the weight W. k The displacement field loss is obtained as follows: The displacement field consistency loss function is used to ensure that the output of the second module in the overlapping region is consistent with the output of the first module. For the perceptual layer loss, the MSE Loss on the conv5_3 layer of the VGG-19 network is also calculated. The input is the deformed image, and the binary mask containing the original viewpoint image content at the virtual viewpoint is M. The overall loss function for this module is defined as:

5. The deep learning-based multi-view video stitching method according to claim 1, characterized in that, The specific method for step 5 is as follows: Based on the final prediction results, the original view is transformed, and the two images are subjected to a simple weighted linear fusion process to obtain the stitched result I. o : I o =W·warp(I A ,flow A )+(1-W)·warp(I B ,flow B ) Among them, flow A flow B represents the output pixel displacement field, warp is the transformation function, and W is the set linear fusion weight.

Images