Video stitching method and apparatus

Abstract

The application provides a video splicing method and device, the method comprising: extracting a target feature map of a target video frame sequence and a reference feature map of a reference video frame sequence; calculating a matching strength map between the two, determining a displacement expectation field, constructing an intermediate homographic mapping, and transforming the target feature map and the reference feature map to an intermediate view for coarse alignment; based on the target feature map after coarse alignment and the reference feature map after coarse alignment, constructing a local correlation body, regressing a dense deformation field, superimposing the dense deformation field and a rigid grid to generate a control grid; determining a grid time path according to a grid increment field between adjacent frames, and performing time domain smoothing processing on the control grid; based on the rigid grid and the smoothed grid, using a thin plate spline transformation to map and fuse the target video frame sequence and the reference video frame sequence, and outputting a spliced video. The application improves the accuracy, stability and robustness of video splicing through two-stage alignment and time sequence smoothing.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision technology, and in particular to a video stitching method and apparatus. Background Technology

[0002] With the rapid development of video surveillance, drone aerial photography, and panoramic live streaming, real-time multi-channel video stitching technology has become a core requirement. Currently, mainstream video stitching solutions typically calculate the homography transformation matrix based on feature point matching, projecting multiple video streams onto the same plane to achieve stitching. However, this method has drawbacks in practical applications: traditional homography transformation assumes that the images are on the same plane, which easily leads to severe ghosting and tearing when there are moving objects in the scene; existing methods mostly perform independent alignment based on single frames, resulting in cumulative drift and a breathing effect in the output image, affecting the viewing experience; furthermore, in complex scenes with missing textures or drastic lighting changes, the stability of feature point matching is poor, leading to a high stitching failure rate. Summary of the Invention

[0003] This invention provides a video stitching method and apparatus to address the shortcomings of existing video stitching methods, such as low accuracy, poor stability, and poor robustness.

[0004] This invention provides a video stitching method, comprising: Extract the target feature map from the target video frame sequence, and extract the reference feature map from the reference video frame sequence; Calculate the matching strength map between the target feature map and the reference feature map, determine the displacement expectation field, construct an intermediate homography map based on the displacement expectation field, and transform both the target feature map and the reference feature map to an intermediate viewpoint for coarse alignment based on the intermediate homography map. Based on the coarsely aligned target feature map and the coarsely aligned reference feature map, a local correlation volume is constructed, a dense deformation field is regressed, and the dense deformation field is superimposed with a preset rigid mesh to generate a control mesh. The rigid mesh is a set of regular mesh nodes that have not shifted. Based on the grid increment field between adjacent frames in the target video frame sequence and / or the reference video frame sequence, the grid time path is determined, and the control grid is smoothed in the time domain to obtain a smoothed grid. Based on the rigid mesh and the smoothed mesh, thin plate spline transformation is used to perform pixel mapping and fusion on the target video frame sequence and the reference video frame sequence to output a stitched video.

[0005] In some embodiments, constructing the intermediate homography map based on the displacement expectation field includes: The corner offset value is estimated based on the displacement expectation field. The corner offset value is the two-dimensional offset of the target video frame relative to the reference video frame at the four corners. An intermediate homography mapping is constructed based on the corner offset values.

[0006] In some embodiments, estimating the corner offset value based on the displacement expectation field includes: The displacement expectation field is input into a regression network, which includes convolutional layers and fully connected layers. The local context features of the displacement expectation field are extracted based on the convolutional layers, and the local context features are mapped to corner offset values ​​based on the fully connected layers.

[0007] In some embodiments, constructing an intermediate homography map based on the corner offset values ​​includes: Obtain the original four-corner coordinates, which are the coordinates of the four corner points of the reference video frame; Add the original four-corner coordinates to half of the corner offset value to obtain the coordinates of the middle four corners; The intermediate homography is calculated based on the original four-corner coordinates and the intermediate four-corner coordinates.

[0008] In some embodiments, calculating the matching strength map between the target feature map and the reference feature map to determine the expected displacement field includes: The target feature map and the reference feature map are subjected to L2 normalization in the channel dimension; Using the normalized target feature map as a local matching template, a position-by-position inner product operation is performed on the normalized reference feature map to obtain the matching strength map; Based on the first preset search radius, enumerate each candidate displacement corresponding to each position in the matching intensity map; The matching intensity map is normalized by introducing a softmax function with a temperature coefficient to obtain the weight of each candidate displacement at each of the aforementioned locations. Based on the weights of each candidate displacement at each position, a weighted sum is performed on each candidate displacement at each position, and the expected horizontal displacement and the expected vertical displacement at each position are calculated to obtain the expected displacement field.

[0009] In some embodiments, the adjacent frames include the current frame and the frame preceding the current frame, and determining the grid time path based on the grid increment field between adjacent frames in the target video frame sequence and / or the reference video frame sequence includes: The feature map of the current frame and the feature map of the previous frame are L2 normalized in the channel dimension to obtain the normalized feature map of the adjacent frame. With the current grid node position on the control grid as the center, within a two-dimensional window of the second preset search radius, enumerate each candidate displacement of each grid node position of the control grid; Calculate the channel inner product similarity between the normalized feature maps of adjacent frames; By introducing a softmax function with a temperature coefficient to normalize the channel inner product similarity, the weights of each candidate displacement at each grid node position are obtained. Based on the weight of each candidate displacement at each grid node position, the candidate displacements at each grid node position are weighted and summed to calculate the expected displacement at each grid node position, thereby obtaining the grid increment field of the current frame relative to the previous frame. The grid time path is obtained by accumulating the grid increment field in chronological order.

[0010] In some embodiments, the time-domain smoothing of the control grid includes: Obtain the target control grid and the corresponding target grid time path for the current consecutive T frames. Concatenate the target control grid and the target grid time path in the channel dimension to obtain a spatiotemporal feature map, where T is a positive integer greater than 1. The spatiotemporal feature map is input into the spatiotemporal modeler, which aggregates contextual information in the time dimension and regresses the correction amount of each node of the target control grid in the current frame. Subtract the correction amount from the target grid time path to obtain a smooth path; Subtract the correction amount from the target control grid to obtain a smoothed grid.

[0011] In some embodiments, the step of performing pixel mapping and fusion on the target video frame sequence and the reference video frame sequence using thin-plate spline transformation based on the rigid mesh and the smoothed mesh to output a stitched video includes: The source control point coordinates of the rigid mesh and the target control point coordinates of the smoothed mesh are normalized to the [-1,1] interval by the same scale in the pixel coordinate system to obtain the normalized source control point coordinates and the normalized target control point coordinates, respectively. Based on the normalized source control point coordinates and the normalized target control point coordinates, a thin plate spline basis function is constructed. The thin plate spline basis function is solved to obtain the thin plate spline coefficients, which include an affine matrix, bias, and kernel weights. Based on the thin plate spline coefficients, a mapping function is constructed to calculate the sampling position of each pixel in the target video frame sequence and the reference video frame sequence under the smoothed grid, thus completing pixel mapping. In the overlapping region of the mapped target video frame sequence and the mapped reference video frame sequence, a linear gradient weight is constructed along the geometric direction of the overlapping band; The target video frame sequence and the reference video frame sequence are weighted and fused using the linear gradient weights to output the stitched video.

[0012] In some embodiments, before performing pixel mapping and fusion on the target video frame sequence and the reference video frame sequence using thin-plate spline transform, the method further includes: Obtain the extreme coordinate values ​​of the smoothed grid at each node position in the first frame or first window of the current stitched video. The extreme coordinate values ​​include the global minimum horizontal coordinate, the global maximum horizontal coordinate, the global minimum vertical coordinate, and the global maximum vertical coordinate. The width, height, and top-left corner offset of the output canvas are calculated based on the coordinate extreme values. The width, height, and top-left corner offset are determined as fixed canvas parameters that are consistent across the entire video.

[0013] The present invention also provides a video splicing device, comprising: The feature extraction unit is used to extract the target feature map of the target video frame sequence and the reference feature map of the reference video frame sequence. The coarse alignment unit is used to calculate the matching strength map between the target feature map and the reference feature map, determine the displacement expectation field, construct an intermediate homography map based on the displacement expectation field, and transform both the target feature map and the reference feature map to an intermediate viewpoint for coarse alignment based on the intermediate homography map. Fine alignment unit is used to construct a local correlation volume based on the coarsely aligned target feature map and the coarsely aligned reference feature map, regress the dense deformation field, and superimpose the dense deformation field with a preset rigid mesh to generate a control mesh, wherein the rigid mesh is a set of regular mesh nodes that have not shifted. The smoothing unit is used to determine the grid time path based on the grid increment field between adjacent frames in the target video frame sequence and / or the reference video frame sequence, and to perform time-domain smoothing on the control grid to obtain a smoothed grid. The stitching unit is used to perform pixel mapping and fusion on the target video frame sequence and the reference video frame sequence based on the rigid grid and the smoothed grid, and output the stitched video.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video stitching method described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video stitching method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the video stitching method described above.

[0017] The video stitching method and apparatus provided by this invention extract target feature maps from the target video frame sequence and reference feature maps from the reference video frame sequence; calculate the matching intensity map between the two, determine the displacement expectation field, construct an intermediate homography mapping, and transform both the target feature map and the reference feature map to an intermediate viewpoint for coarse alignment. This reduces the dependence on a single feature point, thus maintaining stable alignment effects even in complex scenes with weak textures, high dynamics, and varying lighting. Based on the coarsely aligned target feature map and the coarsely aligned reference feature map, a local correlation volume is constructed, a dense deformation field is regressed, and the dense deformation field is superimposed on a rigid mesh to generate a control mesh that can adapt to local deformation, effectively eliminating ghosting and tearing of dynamic objects. According to the mesh increment field between adjacent frames, the mesh time path is determined, and the control mesh is smoothed in the time domain, effectively suppressing cumulative drift during long-term stitching. Based on the rigid mesh and the smoothed mesh, thin-plate spline transformation is used to map and fuse the target video frame sequence and the reference video frame sequence to output the stitched video, thereby improving the accuracy, stability, and robustness of video stitching. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the video splicing method provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the process for constructing intermediate homography mapping based on displacement expectation field provided by the present invention.

[0021] Figure 3 This is a structural schematic diagram of the video splicing device provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Figure 1 This is a flowchart illustrating the video splicing method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 110: Extract the target feature map of the target video frame sequence and extract the reference feature map of the reference video frame sequence.

[0025] The target video frame sequence and the reference video frame sequence can be multiple video frame sequences from different cameras. The reference video frame sequence refers to the single video frame sequence selected as the spatial reference from the multiple video frame sequences, such as a video frame sequence from the center viewpoint or a video frame sequence captured by the main camera. The target video frame sequence refers to one or more video frame sequences to be stitched to the reference viewpoint, such as a video frame sequence from the side viewpoint or a video frame sequence captured by a secondary camera. The reference video frame sequence includes multiple reference video frames arranged in chronological order, and the target video frame sequence includes multiple target video frames arranged in chronological order.

[0026] The reference feature map refers to the multi-channel feature representation extracted from the current frames of the reference video frame sequence; the target feature map refers to the multi-channel feature representation of the same dimension extracted from the corresponding frames of the target video frame sequence.

[0027] Optionally, a deep learning-based convolutional neural network can be used to perform multi-level feature extraction on the target video frame sequence and the reference video frame sequence.

[0028] Step 120: Calculate the matching strength map between the target feature map and the reference feature map, determine the displacement expectation field, construct an intermediate homography map based on the displacement expectation field, and transform both the target feature map and the reference feature map to the intermediate viewpoint for coarse alignment based on the intermediate homography map.

[0029] The matching intensity map characterizes the matching heat between the target feature map and the reference feature map at each pixel position and its neighborhood displacement. The displacement expectation field is the average expected displacement at each pixel position obtained based on the matching intensity map. The intermediate homography mapping is a transformation relationship that brings the target feature map and the reference feature map closer to the intermediate viewpoint.

[0030] Optionally, based on cross-channel local correlation calculation, the matching strength between the target feature map and the reference feature map is calculated to obtain a matching strength map.

[0031] Specifically, based on intermediate homography mapping, the target feature map and the reference feature map are simultaneously moved toward a reference system in an intermediate state, so that the two feature maps achieve preliminary spatial alignment under this reference system, thereby effectively narrowing the search range for subsequent fine alignment and reducing matching ambiguity.

[0032] Step 130: Based on the coarsely aligned target feature map and the coarsely aligned reference feature map, construct a local correlation volume, regress the dense deformation field, and superimpose the dense deformation field with the preset rigid mesh to generate a control mesh. The rigid mesh is a set of regular mesh nodes that have not shifted.

[0033] The local correlation volume refers to the similarity between feature vectors calculated per displacement within the local neighborhood window for each spatial location on the coarsely aligned target feature map and reference feature map. The similarity under all displacements is organized into a three-dimensional tensor, which is called the local correlation volume. This three-dimensional tensor includes height, width and the square of the search radius.

[0034] The dense deformation field refers to the set of two-dimensional non-rigid offsets predicted for each control grid node position on the target feature map and reference feature map plane, representing the position relative to the initial position. It is usually represented as a two-dimensional tensor with the same spatial resolution as the feature map.

[0035] Locally correlated volumes can be input into a regression network to obtain a dense deformation field output by the regression network.

[0036] The rigid mesh is a uniformly distributed set of initial mesh nodes that has not undergone any deformation adjustment. It overlays the image planes of the target and reference video frame sequences and consists of two-dimensional coordinate points distributed at fixed row and column intervals. The node positions are only related to the pixel coordinate system ratios of the target and reference video frame sequences and do not contain any displacement information caused by scene content or camera motion. Superimposing the dense deformation field onto the rigid mesh at different coordinate scales generates a control mesh containing local non-rigid deformation information, which is used for subsequent image deformation.

[0037] Step 140: Determine the grid time path based on the grid increment field between adjacent frames in the target video frame sequence and / or reference video frame sequence, and perform temporal domain smoothing on the control grid to obtain the smoothed grid.

[0038] In this context, adjacent frames refer to multiple video frames that are adjacent in the time dimension within the video sequence. The grid increment field represents the minute grid node displacements between adjacent frames caused by slight camera movement or objects moving through the scene.

[0039] Optionally, the grid increment of the current frame is estimated using the previous frame as a reference, and then accumulated frame by frame in the time dimension, thereby forming a grid time path that evolves continuously over time. The time context information of a historical frame can be combined to perform jitter correction on the grid time path, thereby obtaining a smoothed grid with a coherent trajectory and the removal of instantaneous noise, so as to eliminate inter-frame trajectory jumps caused by low texture or large-area occlusion.

[0040] Step 150: Based on the rigid mesh and the smoothed mesh, thin plate spline transformation is used to perform pixel mapping and fusion on the target video frame sequence and the reference video frame sequence to output the stitched video.

[0041] Optionally, a non-rigid mapping function is established between the smoothed mesh and the rigid mesh based on the principle of minimum bending energy. This mapping function is used to find the precise sampling position for each pixel in the target and reference video frame sequences, thereby achieving full-resolution distortion alignment of the image. After pixel mapping, data-level pixel-weighted fusion is performed in the overlapping areas to eliminate brightness abrupt changes and stitching breaks, ultimately outputting a stitched video with a wide viewing angle and natural transitions.

[0042] In this embodiment of the invention, target feature maps of the target video frame sequence and reference feature maps of the reference video frame sequence are extracted; the matching intensity map between the two is calculated, the displacement expectation field is determined, and an intermediate homography mapping is constructed. Both the target feature map and the reference feature map are transformed to the intermediate viewpoint for coarse alignment, reducing the dependence on a single feature point. This allows for stable alignment even in complex scenes with weak textures, high dynamics, and varying illumination. Based on the coarsely aligned target feature map and the coarsely aligned reference feature map, a local correlation volume is constructed, the dense deformation field is regressed, and the dense deformation field is superimposed on a rigid mesh to generate a control mesh that can adapt to local deformation, effectively eliminating ghosting and tearing of dynamic objects. According to the mesh increment field between adjacent frames, the mesh time path is determined, and the control mesh is smoothed in the time domain, effectively suppressing the cumulative drift during long-term stitching. Based on the rigid mesh and the smoothed mesh, thin-plate spline transformation is used to map and fuse the target video frame sequence and the reference video frame sequence to output the stitched video, thereby improving the accuracy, stability, and robustness of video stitching.

[0043] Figure 2 This is a schematic diagram of the process for constructing intermediate homography mapping based on the displacement expectation field provided by the present invention. Figure 2 As shown, in some embodiments, step 120, which constructs an intermediate homography map based on the displacement expectation field, includes: Step 121: Estimate the corner offset value based on the displacement expectation field. The corner offset value is the two-dimensional offset of the target video frame relative to the reference video frame at the four corners.

[0044] The displacement expectation field is used to characterize the local pixel-level motion trend between the target feature map and the reference feature map. The four corner points are the top left, top right, bottom left, and bottom right corners. The corner offset values ​​reflect the overall viewpoint difference between the target video frame and the reference video frame in the edge region.

[0045] In some embodiments, estimating corner offset values ​​based on the desired displacement field includes: The displacement expectation field is input into a regression network, which includes convolutional layers and fully connected layers. The local context features of the displacement expectation field are extracted based on the convolutional layers, and the local context features are mapped to corner offset values ​​based on the fully connected layers.

[0046] Understandably, by using convolutional layers to extract local contextual features of the displacement expectation field and using fully connected layers to map the local contextual features into corner offset values, dense local expected displacements can be effectively transformed into global alignment parameters, thereby improving the accuracy and stability of coarse pose estimation.

[0047] Step 122: Construct intermediate homography mapping based on corner offset values.

[0048] In some embodiments, constructing an intermediate homography map based on corner offset values ​​includes: Obtain the original four-corner coordinates, which are the coordinates of the four corner points of the reference video frame; Add the original four-corner coordinates to half of the corner offset value to obtain the coordinates of the middle four corners; Based on the original four-corner coordinates and the intermediate four-corner coordinates, the intermediate homography is calculated.

[0049] The original four-corner coordinates represent the positions of the four boundary vertices of the reference video frame without any transformation, while the middle four-corner coordinates represent the positions where only half of the corner offset value has been moved. Obtaining the original four-corner coordinates and adding them to half of the corner offset value means that when constructing the intermediate homography mapping, the target video frame is not directly hard-fitted to the viewpoint of the reference video frame, but rather the feature maps on both sides are simultaneously moved towards a viewpoint in an intermediate state.

[0050] In this embodiment of the invention, the intermediate four-corner coordinates are obtained by adding the original four-corner coordinates to half of the corner offset value, and the intermediate homography is calculated so that the features of the two video streams are close to the middle viewpoint when spatially aligned, making the pose after coarse alignment more natural and effectively reducing the local shearing error caused by multiple depth layers and slight rotation in non-planar areas.

[0051] In some embodiments, calculating the matching strength map between the target feature map and the reference feature map to determine the expected displacement field includes: L2 normalization is performed on the target feature map and the reference feature map along the channel dimension; Using the normalized target feature map as a local matching template, position-by-position inner product operation is performed on the normalized reference feature map to obtain the matching strength map; Based on the first preset search radius, enumerate each candidate displacement corresponding to each position in the matching intensity map; By introducing a softmax function with a temperature coefficient to normalize the matching intensity map, the weights of each candidate displacement at each position are obtained. Based on the weights of each candidate displacement at each position, the candidate displacements at each position are weighted and summed, and the expected horizontal displacement and expected vertical displacement at each position are calculated to obtain the expected displacement field.

[0052] Specifically, L2 normalization is performed on the feature map along the channel dimension to eliminate interference caused by differences in brightness and contrast between different video streams. Position-wise inner product operations are then performed using the target feature map as a local matching template to obtain a matching strength map that reflects feature similarity.

[0053] The preset search radius limits the two-dimensional spatial range for finding the optimal displacement, and the softmax function with temperature coefficient can perform soft maximum normalization on the matching intensity map, highlighting candidate displacements with higher probability.

[0054] In this embodiment of the invention, L2 normalization of the target feature map and the reference feature map in the channel dimension can eliminate the photometric difference. The weight of each candidate displacement at each position is calculated by combining the inner product operation and the softmax function with the temperature coefficient. The candidate displacements at each position are weighted and summed to obtain the displacement expectation field. This can effectively resist the matching ambiguity caused by sudden changes in illumination and local occlusion, and improve the robustness of correspondence recall in low-texture scenes.

[0055] In some embodiments, adjacent frames include the current frame and the frame preceding the current frame. Determining the grid time path based on the grid increment field between adjacent frames in the target video frame sequence and / or the reference video frame sequence includes: The feature map of the current frame and the feature map of the previous frame are L2 normalized in the channel dimension to obtain the normalized feature map of the adjacent frame. With the current grid node position on the control grid as the center, within a two-dimensional window of the second preset search radius, enumerate each candidate displacement of each grid node position of the control grid; Calculate the channel inner product similarity between normalized feature maps of adjacent frames; By introducing a softmax function with a temperature coefficient to normalize the channel inner product similarity, the weights of each candidate displacement at each grid node position are obtained. Based on the weights of each candidate displacement at each grid node position, the candidate displacements at each grid node position are weighted and summed to calculate the expected displacement at each grid node position, thus obtaining the grid increment field of the current frame relative to the previous frame. The grid time path is obtained by accumulating the grid increment field in chronological order.

[0056] Adjacent frames refer to the current frame and the previous frame that are immediately following each other in time in a streaming video.

[0057] Optionally, feature extraction is performed on the current frame to obtain the feature map of the current frame, and feature extraction is performed on the previous frame to obtain the feature map of the previous frame.

[0058] Specifically, L2 normalization is performed on the feature maps of the current frame and the previous frame in the channel dimension to reduce the impact of brightness fluctuations and contrast changes. By enumerating candidate displacements within a two-dimensional window with the current grid node position as the center and a second preset search radius, the search for minute grid displacements is limited to a local range, avoiding interference from transient noise in large-scale searches. The channel inner product similarity between the normalized feature maps of adjacent frames is calculated to measure the degree of matching between features.

[0059] In this embodiment of the invention, the channel inner product similarity between normalized adjacent frame feature maps is calculated, and a softmax function with a temperature coefficient is introduced to normalize the channel inner product similarity to obtain the weight of each candidate displacement. Then, the expected displacement is calculated by weighted summation based on the weights to obtain the grid increment field of the current frame relative to the previous frame. The grid increment field is accumulated frame by frame in chronological order to form a grid time path that evolves smoothly over time. This can effectively resist matching interference caused by sudden exposure changes and local occlusion, and enhance the robustness and temporal continuity of displacement estimation between adjacent video frames.

[0060] In some embodiments, temporal smoothing of the control grid includes: Obtain the target control grid and the corresponding target grid time path for the current consecutive T frames. Concatenate the target control grid and the target grid time path in the channel dimension to obtain the spatiotemporal feature map, where T is a positive integer greater than 1. The spatiotemporal feature map is input into the spatiotemporal modeler, which aggregates contextual information in the time dimension and regresses the correction amount of each node of the target control grid in the current frame. Subtract the correction amount from the target grid time path to obtain the smooth path; Subtract the correction amount from the target control grid to obtain the smoothed grid.

[0061] The spatiotemporal feature map includes the spatial grid deformation structure and the temporal evolution trend. The spatiotemporal modeler can be a network model built using a three-dimensional convolutional architecture.

[0062] Specifically, by inputting the spatiotemporal feature map into the spatiotemporal modeler, the contextual information of the most recent multiple frames can be aggregated in the time dimension using three-dimensional convolution, thereby regressing the jitter deviation that each node of the target control grid should correct in the current frame, i.e., the correction amount.

[0063] In this embodiment of the invention, by subtracting the correction amount from the target mesh time path to obtain a smooth path, and by subtracting the correction amount from the target control mesh to obtain a smoothed mesh, the trajectory jitter and burrs caused by low texture or motion blur can be effectively eliminated, making the final output stitched mesh smoother and more stable.

[0064] In some embodiments, based on a rigid mesh and a smoothed mesh, thin-plate spline transformation is used to perform pixel mapping and fusion on the target video frame sequence and the reference video frame sequence to output a stitched video, including: The source control point coordinates of the rigid mesh and the target control point coordinates of the smoothed mesh are normalized to the [-1,1] interval by the same scale in the pixel coordinate system to obtain the normalized source control point coordinates and the normalized target control point coordinates, respectively. Based on the normalized source control point coordinates and the normalized target control point coordinates, a thin plate spline basis function is constructed. The thin plate spline basis function is solved to obtain the thin plate spline coefficients, which include the affine matrix, bias, and kernel weights. A mapping function is constructed based on the thin plate spline coefficients. The sampling position of each pixel in the target video frame sequence and the reference video frame sequence under the smoothed grid is calculated to complete the pixel mapping. In the overlapping region of the mapped target video frame sequence and the mapped reference video frame sequence, a linear gradient weight is constructed along the geometric direction of the overlapping band. The target video frame sequence and the reference video frame sequence are weighted and fused by linearly gradient weights to output a stitched video.

[0065] Specifically, normalizing the source control point coordinates of the rigid mesh and the target control point coordinates of the smoothed mesh to a unified range can avoid the risk of numerical overflow during subsequent linear matrix operations.

[0066] Optionally, a thin-plate spline basis function is constructed based on the normalized source control point coordinates and the normalized target control point coordinates, and its augmented linear system is solved to obtain thin-plate spline coefficients including the affine matrix, bias, and kernel weights. A non-rigid mapping function is then constructed based on these thin-plate spline coefficients, and the precise sampling position of each pixel in the two video frame sequences under a smooth grid is calculated according to the principle of minimum bending energy to complete the pixel mapping.

[0067] In this embodiment of the invention, a precise non-rigid mapping function is established by solving the thin plate spline coefficients to complete pixel-level deformation. Combined with linear gradient weights, the overlapping areas are weighted and fused, which can effectively eliminate structural gaps caused by multiple views and reduce edge tearing and brightness step phenomena, making the transition of video stitching images more natural and seamless.

[0068] In some embodiments, before performing pixel mapping and fusion on the target video frame sequence and the reference video frame sequence using thin-plate spline transform, the method further includes: Get the extreme coordinates of the smoothed grid at each node position in the first frame or first window of the current stitched video. The extreme coordinates include the global minimum x-coordinate, global maximum x-coordinate, global minimum y-coordinate, and global maximum y-coordinate. The width, height, and top-left corner offset of the output canvas are calculated based on the coordinate extreme values. Define the width, height, and top-left corner offset as fixed canvas parameters that are consistent across the entire video.

[0069] The extreme values ​​of the coordinates represent the maximum bounding box boundary of the smoothed grid in global space. This set of canvas parameters is reused in subsequent video stitching processes and is no longer dynamically adjusted.

[0070] In this embodiment of the invention, by setting uniform fixed canvas parameters based on the grid coordinate extreme values ​​of the first frame or the first window, the phenomenon of field breathing and boundary jumping caused by the dynamic and frequent adjustment of the canvas size in subsequent video stream output can be avoided, thus ensuring the global field stability of long videos and streaming output images.

[0071] The video splicing device provided by the present invention is described below. The video splicing device described below and the video splicing method described above can be referred to in correspondence.

[0072] Figure 3 This is a structural schematic diagram of the video splicing device provided by the present invention. Figure 3 As shown, in some embodiments, the present invention also provides a video stitching device, the device 300 comprising: The feature extraction unit 310 is used to extract the target feature map of the target video frame sequence and extract the reference feature map of the reference video frame sequence. The coarse alignment unit 320 is used to calculate the matching intensity map between the target feature map and the reference feature map, determine the displacement expectation field, construct an intermediate homography map based on the displacement expectation field, and transform both the target feature map and the reference feature map to the intermediate viewpoint for coarse alignment based on the intermediate homography map. The fine alignment unit 330 is used to construct a local correlation volume based on the coarsely aligned target feature map and the coarsely aligned reference feature map, regress the dense deformation field, and superimpose the dense deformation field with the preset rigid mesh to generate a control mesh. The rigid mesh is a set of regular mesh nodes that have not been offset. The smoothing unit 340 is used to determine the grid time path based on the grid increment field between adjacent frames in the target video frame sequence and / or the reference video frame sequence, and to perform temporal domain smoothing on the control grid to obtain the smoothed grid. The stitching unit 350 is used to perform pixel mapping and fusion on the target video frame sequence and the reference video frame sequence based on the rigid grid and the smoothed grid, and output the stitched video.

[0073] Optionally, an intermediate homography map is constructed based on the displacement expectation field, including: The corner offset value is estimated based on the displacement expectation field. The corner offset value is the two-dimensional offset of the target video frame relative to the reference video frame at the four corners. Construct an intermediate homography mapping based on corner offset values.

[0074] Optionally, the corner offset value is estimated based on the displacement expectation field, including: The displacement expectation field is input into a regression network, which includes convolutional layers and fully connected layers. The local context features of the displacement expectation field are extracted based on the convolutional layers, and the local context features are mapped to corner offset values ​​based on the fully connected layers.

[0075] Optionally, an intermediate homography mapping is constructed based on the corner offset values, including: Obtain the original four-corner coordinates, which are the coordinates of the four corner points of the reference video frame; Add the original four-corner coordinates to half of the corner offset value to obtain the coordinates of the middle four corners; Based on the original four-corner coordinates and the intermediate four-corner coordinates, the intermediate homography is calculated.

[0076] Optionally, a matching intensity map between the target feature map and the reference feature map is calculated to determine the expected displacement field, including: L2 normalization is performed on the target feature map and the reference feature map along the channel dimension; Using the normalized target feature map as a local matching template, position-by-position inner product operation is performed on the normalized reference feature map to obtain the matching strength map; Based on the first preset search radius, enumerate each candidate displacement corresponding to each position in the matching intensity map; By introducing a softmax function with a temperature coefficient to normalize the matching intensity map, the weights of each candidate displacement at each position are obtained. Based on the weights of each candidate displacement at each position, the candidate displacements at each position are weighted and summed, and the expected horizontal displacement and expected vertical displacement at each position are calculated to obtain the expected displacement field.

[0077] Optionally, adjacent frames include the current frame and the frame preceding the current frame. The grid time path is determined based on the grid increment field between adjacent frames in the target video frame sequence and / or the reference video frame sequence, including: The feature map of the current frame and the feature map of the previous frame are L2 normalized in the channel dimension to obtain the normalized feature map of the adjacent frame. With the current grid node position on the control grid as the center, within a two-dimensional window of the second preset search radius, enumerate each candidate displacement of each grid node position of the control grid; Calculate the channel inner product similarity between normalized feature maps of adjacent frames; By introducing a softmax function with a temperature coefficient to normalize the channel inner product similarity, the weights of each candidate displacement at each grid node position are obtained. Based on the weights of each candidate displacement at each grid node position, the candidate displacements at each grid node position are weighted and summed to calculate the expected displacement at each grid node position, thus obtaining the grid increment field of the current frame relative to the previous frame. The grid time path is obtained by accumulating the grid increment field in chronological order.

[0078] Optionally, the control grid is smoothed in the time domain, including: Obtain the target control grid and the corresponding target grid time path for the current consecutive T frames. Concatenate the target control grid and the target grid time path in the channel dimension to obtain the spatiotemporal feature map, where T is a positive integer greater than 1. The spatiotemporal feature map is input into the spatiotemporal modeler, which aggregates contextual information in the time dimension and regresses the correction amount of each node of the target control grid in the current frame. Subtract the correction amount from the target grid time path to obtain the smooth path; Subtract the correction amount from the target control grid to obtain the smoothed grid.

[0079] Optionally, based on the rigid mesh and the smoothed mesh, thin-plate spline transformation is used to perform pixel mapping and fusion on the target video frame sequence and the reference video frame sequence to output a stitched video, including: The source control point coordinates of the rigid mesh and the target control point coordinates of the smoothed mesh are normalized to the [-1,1] interval by the same scale in the pixel coordinate system to obtain the normalized source control point coordinates and the normalized target control point coordinates, respectively. Based on the normalized source control point coordinates and the normalized target control point coordinates, a thin plate spline basis function is constructed. The thin plate spline basis function is solved to obtain the thin plate spline coefficients, which include the affine matrix, bias, and kernel weights. A mapping function is constructed based on the thin plate spline coefficients. The sampling position of each pixel in the target video frame sequence and the reference video frame sequence under the smoothed grid is calculated to complete the pixel mapping. In the overlapping region of the mapped target video frame sequence and the mapped reference video frame sequence, a linear gradient weight is constructed along the geometric direction of the overlapping band. The target video frame sequence and the reference video frame sequence are weighted and fused by linearly gradient weights to output a stitched video.

[0080] Optionally, the above apparatus further includes a canvas parameter determination unit, which performs the following steps before performing pixel mapping and fusion on the target video frame sequence and the reference video frame sequence using thin-plate spline transform: Get the extreme coordinates of the smoothed grid at each node position in the first frame or first window of the current stitched video. The extreme coordinates include the global minimum x-coordinate, global maximum x-coordinate, global minimum y-coordinate, and global maximum y-coordinate. The width, height, and top-left corner offset of the output canvas are calculated based on the coordinate extreme values. Define the width, height, and top-left corner offset as fixed canvas parameters that are consistent across the entire video.

[0081] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communications bus 440. The processor 410 can call logical instructions in the memory 430 to execute a video stitching method, which includes: extracting target feature maps of the target video frame sequence and extracting reference feature maps of the reference video frame sequence; calculating the matching strength map between the target feature map and the reference feature map, determining the displacement expectation field, constructing an intermediate homography map based on the displacement expectation field, and coarsely aligning the target feature map and the reference feature map to an intermediate viewpoint based on the intermediate homography map; constructing a local correlation volume based on the coarsely aligned target feature map and the coarsely aligned reference feature map, regressing the dense deformation field, superimposing the dense deformation field with a preset rigid grid to generate a control grid, wherein the rigid grid is a set of regular grid nodes that have not shifted; determining the grid time path according to the grid increment field between adjacent frames in the target video frame sequence and / or the reference video frame sequence, performing temporal domain smoothing on the control grid to obtain a smoothed grid; and performing pixel mapping and fusion on the target video frame sequence and the reference video frame sequence using thin plate spline transformation based on the rigid grid and the smoothed grid to output the stitched video.

[0082] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video stitching method provided by the above methods. The method includes: extracting target feature maps of a target video frame sequence and extracting reference feature maps of a reference video frame sequence; calculating a matching strength map between the target feature map and the reference feature map; determining a displacement expectation field; constructing an intermediate homography mapping based on the displacement expectation field; and transforming both the target feature map and the reference feature map to an intermediate homography mapping based on the intermediate homography mapping. The corners are coarsely aligned; based on the coarsely aligned target feature map and the coarsely aligned reference feature map, a local correlation volume is constructed, the dense deformation field is regressed, and the dense deformation field is superimposed with a preset rigid grid to generate a control grid. The rigid grid is a set of regular grid nodes that have not shifted; according to the grid increment field between adjacent frames in the target video frame sequence and / or the reference video frame sequence, the grid time path is determined, and the control grid is smoothed in the time domain to obtain a smoothed grid; based on the rigid grid and the smoothed grid, thin plate spline transformation is used to perform pixel mapping and fusion on the target video frame sequence and the reference video frame sequence to output a stitched video.

[0084] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the video stitching method provided by the above methods. The method includes: extracting target feature maps of a target video frame sequence and extracting reference feature maps of a reference video frame sequence; calculating a matching strength map between the target feature map and the reference feature map, determining a displacement expectation field, constructing an intermediate homography mapping based on the displacement expectation field, and coarsely aligning both the target feature map and the reference feature map to an intermediate viewpoint based on the intermediate homography mapping; constructing a local correlation volume based on the coarsely aligned target feature map and the coarsely aligned reference feature map, regressing a dense deformation field, superimposing the dense deformation field with a preset rigid grid to generate a control grid, wherein the rigid grid is a set of regular grid nodes that have not shifted; determining the grid time path according to the grid increment field between adjacent frames in the target video frame sequence and / or the reference video frame sequence, performing temporal domain smoothing on the control grid to obtain a smoothed grid; and performing pixel mapping and fusion on the target video frame sequence and the reference video frame sequence using a thin plate spline transformation based on the rigid grid and the smoothed grid to output a stitched video.

[0085] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0086] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video stitching method, characterized in that, include: Extract the target feature map from the target video frame sequence, and extract the reference feature map from the reference video frame sequence; Calculate the matching strength map between the target feature map and the reference feature map, determine the displacement expectation field, construct an intermediate homography map based on the displacement expectation field, and transform both the target feature map and the reference feature map to an intermediate viewpoint for coarse alignment based on the intermediate homography map. Based on the coarsely aligned target feature map and the coarsely aligned reference feature map, a local correlation volume is constructed, a dense deformation field is regressed, and the dense deformation field is superimposed with a preset rigid mesh to generate a control mesh. The rigid mesh is a set of regular mesh nodes that have not shifted. Based on the grid increment field between adjacent frames in the target video frame sequence and / or the reference video frame sequence, the grid time path is determined, and the control grid is smoothed in the time domain to obtain a smoothed grid. Based on the rigid mesh and the smoothed mesh, thin plate spline transformation is used to perform pixel mapping and fusion on the target video frame sequence and the reference video frame sequence to output a stitched video.

2. The video stitching method according to claim 1, characterized in that, The construction of the intermediate homography mapping based on the displacement expectation field includes: The corner offset value is estimated based on the displacement expectation field. The corner offset value is the two-dimensional offset of the target video frame relative to the reference video frame at the four corners. An intermediate homography mapping is constructed based on the corner offset values.

3. The video stitching method according to claim 2, characterized in that, The estimation of corner offset values ​​based on the displacement expectation field includes: The displacement expectation field is input into a regression network, which includes convolutional layers and fully connected layers. The local context features of the displacement expectation field are extracted based on the convolutional layers, and the local context features are mapped to corner offset values ​​based on the fully connected layers.

4. The video stitching method according to claim 2, characterized in that, The construction of the intermediate homography mapping based on the corner offset value includes: Obtain the original four-corner coordinates, which are the coordinates of the four corner points of the reference video frame; Add the original four-corner coordinates to half of the corner offset value to obtain the coordinates of the middle four corners; The intermediate homography is calculated based on the original four-corner coordinates and the intermediate four-corner coordinates.

5. The video stitching method according to claim 1, characterized in that, The step of calculating the matching strength map between the target feature map and the reference feature map to determine the expected displacement field includes: The target feature map and the reference feature map are subjected to L2 normalization in the channel dimension; Using the normalized target feature map as a local matching template, a position-by-position inner product operation is performed on the normalized reference feature map to obtain the matching strength map; Based on the first preset search radius, enumerate each candidate displacement corresponding to each position in the matching intensity map; The matching intensity map is normalized by introducing a softmax function with a temperature coefficient to obtain the weight of each candidate displacement at each of the aforementioned locations. Based on the weights of each candidate displacement at each position, a weighted sum is performed on each candidate displacement at each position, and the expected horizontal displacement and the expected vertical displacement at each position are calculated to obtain the expected displacement field.

6. The video stitching method according to claim 1, characterized in that, The adjacent frames include the current frame and the frame preceding the current frame. Determining the grid time path based on the grid increment field between adjacent frames in the target video frame sequence and / or the reference video frame sequence includes: The feature map of the current frame and the feature map of the previous frame are L2 normalized in the channel dimension to obtain the normalized feature map of the adjacent frame. With the current grid node position on the control grid as the center, within a two-dimensional window of the second preset search radius, enumerate each candidate displacement of each grid node position of the control grid; Calculate the channel inner product similarity between the normalized feature maps of adjacent frames; By introducing a softmax function with a temperature coefficient to normalize the channel inner product similarity, the weights of each candidate displacement at each grid node position are obtained. Based on the weight of each candidate displacement at each grid node position, the candidate displacements at each grid node position are weighted and summed to calculate the expected displacement at each grid node position, thereby obtaining the grid increment field of the current frame relative to the previous frame. The grid time path is obtained by accumulating the grid increment field in chronological order.

7. The video stitching method according to claim 1, characterized in that, The time-domain smoothing process for the control grid includes: Obtain the target control grid and the corresponding target grid time path for the current consecutive T frames. Concatenate the target control grid and the target grid time path in the channel dimension to obtain a spatiotemporal feature map, where T is a positive integer greater than 1. The spatiotemporal feature map is input into the spatiotemporal modeler, which aggregates contextual information in the time dimension and regresses the correction amount of each node of the target control grid in the current frame. Subtract the correction amount from the target grid time path to obtain a smooth path; Subtract the correction amount from the target control grid to obtain a smoothed grid.

8. The video stitching method according to claim 1, characterized in that, The process involves using thin-plate spline transformation to perform pixel mapping and fusion on the target video frame sequence and the reference video frame sequence based on the rigid mesh and the smoothed mesh, outputting a stitched video, including: The source control point coordinates of the rigid mesh and the target control point coordinates of the smoothed mesh are normalized to the [-1,1] interval by the same scale in the pixel coordinate system to obtain the normalized source control point coordinates and the normalized target control point coordinates, respectively. Based on the normalized source control point coordinates and the normalized target control point coordinates, a thin plate spline basis function is constructed. The thin plate spline basis function is solved to obtain the thin plate spline coefficients, which include an affine matrix, bias, and kernel weights. Based on the thin plate spline coefficients, a mapping function is constructed to calculate the sampling position of each pixel in the target video frame sequence and the reference video frame sequence under the smoothed grid, thus completing pixel mapping. In the overlapping region of the mapped target video frame sequence and the mapped reference video frame sequence, a linear gradient weight is constructed along the geometric direction of the overlapping band; The target video frame sequence and the reference video frame sequence are weighted and fused using the linear gradient weights to output the stitched video.

9. The video stitching method according to claim 1, characterized in that, Before performing pixel mapping and fusion on the target video frame sequence and the reference video frame sequence using thin-plate spline transform, the method further includes: Obtain the extreme coordinate values ​​of the smoothed grid at each node position in the first frame or first window of the current stitched video. The extreme coordinate values ​​include the global minimum horizontal coordinate, the global maximum horizontal coordinate, the global minimum vertical coordinate, and the global maximum vertical coordinate. The width, height, and top-left corner offset of the output canvas are calculated based on the coordinate extreme values. The width, height, and top-left corner offset are determined as fixed canvas parameters that are consistent across the entire video.

10. A video splicing device, characterized in that, include: The feature extraction unit is used to extract the target feature map of the target video frame sequence and the reference feature map of the reference video frame sequence. The coarse alignment unit is used to calculate the matching strength map between the target feature map and the reference feature map, determine the displacement expectation field, construct an intermediate homography map based on the displacement expectation field, and transform both the target feature map and the reference feature map to an intermediate viewpoint for coarse alignment based on the intermediate homography map. Fine alignment unit is used to construct a local correlation volume based on the coarsely aligned target feature map and the coarsely aligned reference feature map, regress the dense deformation field, and superimpose the dense deformation field with a preset rigid mesh to generate a control mesh, wherein the rigid mesh is a set of regular mesh nodes that have not shifted. The smoothing unit is used to determine the grid time path based on the grid increment field between adjacent frames in the target video frame sequence and / or the reference video frame sequence, and to perform time-domain smoothing on the control grid to obtain a smoothed grid. The stitching unit is used to perform pixel mapping and fusion on the target video frame sequence and the reference video frame sequence based on the rigid grid and the smoothed grid, and output the stitched video.

Images