An unsupervised video stitching method based on reliability perception and collaborative smoothing

Abstract

The application discloses an unsupervised video stitching method based on reliability perception and collaborative smoothing, and relates to the technical field of computer vision. In the method, a reliability prior module is introduced, a rigid texture is focused on by suppressing interference in space and combining attention, outliers are filtered in time to enhance trajectory robustness, and a feature perception system of filtering first and enhancing later is formed; a cross-view collaborative smoothing network is designed, a space-time cross-attention mechanism is used to promote active negotiation and interaction between double views, and non-physical grid deformation caused by local forced alignment is suppressed. Without introducing labeled data, the application can effectively eliminate stitching artifacts, significantly reduce the geometric distortion rate of the video, and improve the alignment accuracy and space-time stability while maintaining the network reasoning efficiency.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an unsupervised video stitching method based on reliability perception and collaborative smoothing. Background Technology

[0002] The statements in this section are merely to provide background information in relation to this application to aid in understanding it, and such background information does not necessarily constitute prior art.

[0003] Video stitching is the process of combining synchronized video streams from multiple overlapping cameras into a seamless, high-resolution panoramic view. This technology breaks the physical field-of-view limitations of a single-lens sensor and plays a crucial role in fields such as immersive virtual reality, wide-area surveillance, autonomous driving, and aerial photography. However, when multimedia applications expand to unconstrained natural environments, video stitching faces severe challenges such as large parallax, dynamic occlusion, and severe jitter.

[0004] Existing video stitching methods are mainly divided into optimization-based methods and deep learning-based methods. Traditional optimization methods offer geometric interpretability but suffer from high latency and rely on handcrafted features, making them very vulnerable in dynamic scenes. Deep learning methods improve robustness through high-level semantic feature extraction, but most existing frameworks treat video stitching as independent image stitching, ignoring crucial temporal coherence. This leads to the model's inability to maintain visual stability when processing handheld or moving camera footage, resulting in noticeable warping shake and frame jumps.

[0005] While state-of-the-art unsupervised online stitching frameworks mitigate distortion and jitter by jointly optimizing spatial alignment and temporal stability, they still exhibit inherent flaws in complex environments, such as low-light scenes or when the camera is moving rapidly. Specifically, this manifests as severe stitching artifacts, primarily caused by two factors: First, indiscriminate use of features; the network lacks sufficient semantic discriminative power and incorrectly relies on dynamic interference and specular noise for spatial alignment and temporal trajectory estimation. Second, insufficient view interaction; independently processing the trajectories of each view leads to direct geometric conflicts, forcing the network to resort to aggressive, non-physical mesh deformation to force local pixel alignment, resulting in significant structural distortion in the final video.

[0006] Therefore, how to solve the feature interference problem in extreme scenarios while maintaining unsupervised lightweight training, and ensure the geometric consistency of multi-view interaction, is a key challenge that current seamless video stitching technology urgently needs to overcome. Summary of the Invention

[0007] This application aims to address the issues of artifacts and severe mesh distortion in existing deep learning stitching frameworks in complex scenes, and to solve the feature interference problem in extreme scenarios while maintaining unsupervised lightweight training, and to ensure geometric consistency of multi-view interaction.

[0008] To achieve the above objectives, this application redesigns the spatial alignment, temporal trajectory estimation, and smoothing stages based on the characteristics of interference information in natural scenes. The specific technical solutions are as follows:

[0009] According to the first aspect of this application, an unsupervised video stitching method based on reliability awareness and collaborative smoothing is provided, comprising: Step 1, acquiring a first video sequence and a second video sequence to be stitched, wherein the first video sequence and the second video sequence are synchronously acquired by two overlapping image acquisition devices; Step 2, simultaneously inputting the first video sequence and the second video sequence into a trained spatial alignment network, wherein the spatial alignment network performs coarse alignment and fine alignment processing on each synchronous frame image of the two video sequences to obtain a first local spatial alignment grid corresponding to the first video sequence and a second local spatial alignment grid corresponding to the second video sequence, wherein a spatial reliability prior module for generating a spatial confidence mask is introduced into the spatial alignment network to characterize the alignment reliability of each pixel position; Step 3, using a trained temporal trajectory estimation network to process the first video sequence and the second video sequence respectively to generate a first camera trajectory corresponding to the first video sequence and a second camera trajectory corresponding to the second video sequence. The process involves several steps: Step 4: Obtaining the first stitching trajectory corresponding to the first video sequence based on the first local spatial alignment grid and the first camera trajectory; obtaining the second stitching trajectory corresponding to the second video sequence based on the second local spatial alignment grid and the second camera trajectory; and using a trained cross-view collaborative smoothing network with a spatiotemporal global flattening and symmetrical cross-view cross-attention mechanism to jointly optimize the first stitching trajectory and the second stitching trajectory to obtain the first smooth stitching trajectory and the second smooth stitching trajectory; Step 5: Deforming the first video sequence based on the first smooth stitching trajectory and the first local spatial alignment network; deforming the second video sequence based on the second smooth stitching trajectory and the second local spatial alignment network; and fusing and stitching the deformed first video sequence and the second video sequence to obtain a stitched video with spatiotemporal consistency.

[0010] Preferably, step 2 includes: step 201, using images from the first video sequence as target images and images from the second video sequence as reference images, and simultaneously inputting the target image and reference image of the same frame into a trained spatial alignment network, wherein the spatial alignment network includes a first feature extraction backbone network and parallel coarse alignment branches and fine alignment branches; step 202, using the first feature extraction backbone network to extract a multi-scale feature pyramid based on the target image and reference image; step 203, using the coarse alignment branch to estimate the depth-sensing homography moments from the target image to the reference image based on the multi-scale feature pyramid. Step 204: Using the fine alignment branch, construct a local cost volume based on the multi-scale feature pyramid, and predict the first local grid flow corresponding to the target image and the second local grid flow corresponding to the reference image based on the local cost volume; Step 205: Superimpose the first local grid flow onto the first global grid flow to obtain a first local spatially aligned grid, and superimpose the second local grid flow onto the second global grid flow to obtain a second local spatially aligned grid.

[0011] Preferably, step 203 includes: obtaining deep semantic features at the H / 16 scale from the multi-scale feature pyramid, and estimating the depth-aware homography matrix from the target image to the reference image based on the deep semantic features at the H / 16 scale using a 4-point representation method; employing a bidirectional decomposition strategy, illuminating an intermediate plane, and obtaining a first homography matrix from the target image to the intermediate plane based on half of each offset in the depth-aware homography matrix; using the depth-aware homography matrix and the first homography matrix, obtaining a second homography matrix from the reference image to the intermediate plane through matrix operations; and using the first homography matrix as a first global grid flow and the second homography matrix as a second global grid flow.

[0012] Preferably, step 204 includes: the fine alignment branch includes a spatial reliability prior module, a feature enhancement module, and a fine alignment module; wherein, the spatial reliability prior module outputs a spatial confidence mask based on the multi-scale feature pyramid; semantic features at the H / 4 scale are obtained from the multi-scale feature pyramid, and the feature enhancement module modulates the semantic features at the H / 4 scale based on the spatial confidence mask, and a spatial attention mechanism is used to enhance the modulated features; the fine alignment module includes a cost volume construction unit and a grid regressor, the cost volume construction unit combines the enhanced features with the first homography matrix and the second homography matrix respectively to construct corresponding first local cost volumes and second local cost volumes, and the grid regressor predicts the first local grid flow based on the first local cost volume and the second local grid flow based on the second local cost volume.

[0013] Preferably, step 3 includes: inputting the first video sequence and the second video sequence into a trained temporal trajectory estimation network, wherein the temporal trajectory estimation network includes a second feature extraction backbone network, a temporal reliability prior module, a temporal cost volume module, and a structure-preserving regressor; for each video sequence: using the temporal reliability prior module to output the corresponding temporal confidence mask; using the second feature extraction backbone network to extract the corresponding multi-scale feature pyramid, obtaining H / 4 scale semantic features from the multi-scale feature pyramid, and modulating the H / 4 scale semantic features with the temporal confidence mask to obtain masked filtered features; using the temporal cost volume module to construct a temporal cost volume based on the masked filtered features and encoding it as latent motion features; using a structure-preserving regressor with global average pooling to obtain residual mesh deformation based on the latent motion features; and using a chain rule to accumulate the residual mesh deformation corresponding to each frame image to generate the camera trajectory corresponding to the video sequence.

[0014] Preferably, the spatial reliability prior module and the temporal reliability prior module have the same structure, each consisting of multiple stacked convolutional layers and a Sigmoid activation function, and the output spatial confidence mask and temporal confidence mask are in the range of 0 to 1.

[0015] Preferably, the spatial alignment network, temporal trajectory estimation network, and cross-view collaborative smoothing network are trained in the following manner: First, the spatial alignment network and temporal trajectory estimation network are pre-trained independently; after pre-training, the outputs of the trained spatial alignment network and temporal trajectory estimation network are used as input to train the cross-view collaborative smoothing network; wherein, the training of the spatial alignment network and temporal trajectory estimation network is implicitly driven by a joint loss consisting of photometric alignment and grid regularization, and the training of the spatial reliability prior module and temporal reliability prior module is implicitly driven only by photometric error; the training of the cross-view collaborative smoothing network is driven by a combined loss consisting of trajectory data approximation term, second derivative motion smoothing term, grid shape preservation term, trajectory overlap area consistency term, and online alignment term.

[0016] Preferably, in step 4, the trained cross-view collaborative smoothing network is used, employing a spatiotemporal global flattening and symmetrical cross-view cross-attention mechanism, to jointly optimize the first and second splicing trajectories, obtaining a first smooth splicing trajectory and a second smooth splicing trajectory. This includes: Step 401, the cross-view collaborative smoothing network includes a shallow encoder, a symmetrical parallel cross-attention layer, a smoothing backbone network, and a decoder; Step 402, the shallow encoder maps the first and second splicing trajectories to a first dense feature embedding and a second dense feature embedding, respectively, and then spatiotemporally flattens the first and second dense feature embeddings; Step 403, the first and second dense feature embeddings are flattened in the spatiotemporal dimension using the shallow encoder. 403. Using a symmetrical parallel cross-attention layer, a symmetrical cross-view cross-attention mechanism is employed to align the splicing trajectories of the two video sequences based on the flattened first dense feature embedding and second dense feature embedding, thereby obtaining joint features; Step 404. The joint features are input into a smooth backbone network composed of 3D dilated residual convolutions stacked with different dilation rates to obtain smooth features; Step 405. The smooth features are decoded into a first residual smoothing increment and a second residual smoothing increment using a decoding head, and the first residual smoothing increment is superimposed on the first splicing trajectory to generate a first smooth splicing trajectory, and the second residual smoothing increment is superimposed on the second splicing trajectory to generate a second smooth splicing trajectory.

[0017] Preferably, step 403 includes: linearly projecting the flattened first dense feature embedding and the second dense feature embedding respectively to obtain the corresponding query vector, key vector and value vector; calculating two symmetric attention weights based on the query vector, key vector and value vector using a normalized exponential function; aligning the splicing trajectories of the two video sequences based on the flattened first dense feature embedding and the second dense feature embedding using the two symmetric attention weights to obtain a first alignment feature corresponding to the first video sequence and a second alignment feature corresponding to the second video sequence; and fusing the first alignment feature and the second alignment feature to obtain a joint feature.

[0018] According to a second aspect of this application, an unsupervised video stitching system based on reliability awareness and collaborative smoothing is provided, comprising: a video acquisition module, a spatial alignment module, a temporal trajectory estimation module, a trajectory smoothing module, and a video stitching module; wherein, the video acquisition module is configured to acquire a first video sequence and a second video sequence to be stitched, the first video sequence and the second video sequence being synchronously acquired by two overlapping image acquisition devices; the spatial alignment module is configured to simultaneously input the first video sequence and the second video sequence into a trained spatial alignment network, the spatial alignment network performing coarse alignment and fine alignment processing on each synchronous frame image of the two video sequences, obtaining a first local spatial alignment grid corresponding to the first video sequence and a second local spatial alignment grid corresponding to the second video sequence, wherein a spatial reliability prior module for generating a spatial confidence mask is introduced into the spatial alignment network to characterize the alignment reliability of each pixel position; the temporal trajectory estimation module is configured to process the first video sequence and the second video sequence respectively using the trained temporal trajectory estimation network to generate a first video sequence pair The temporal trajectory estimation network is configured to generate a temporal reliability prior module for generating a temporal confidence mask to characterize the motion consistency of cross-frame correspondences. The trajectory smoothing module is configured to obtain a first stitched trajectory corresponding to the first video sequence based on a first local spatial alignment grid and the first camera trajectory, and a second stitched trajectory corresponding to the second video sequence based on a second local spatial alignment grid and the second camera trajectory. Using a trained cross-view collaborative smoothing network, a spatiotemporal global flattening and symmetrical cross-view cross-attention mechanism is employed to jointly optimize the first and second stitched trajectories, resulting in a first smoothed stitched trajectory and a second smoothed stitched trajectory. The video stitching module is configured to deform the first video sequence based on the first smoothed stitched trajectory and the first local spatial alignment network, deform the second video sequence based on the second smoothed stitched trajectory and the second local spatial alignment network, and fuse and stitch the deformed first and second video sequences to obtain a stitched video with spatiotemporal consistency.

[0019] According to a third aspect of this application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of the first aspect of this application.

[0020] According to a fourth aspect of this application, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method of the first aspect of this application.

[0021] Compared with the prior art, this application has the following advantages:

[0022] This application utilizes spatiotemporal flattening and cross-attention mechanisms to fully negotiate information between two views. Through cross-view interactive negotiation and smoothing, it effectively resolves the conflict problem of local meshes. Compared to existing baseline models, this application can significantly reduce geometric distortion in non-overlapping video regions. Simultaneously, it can effectively eliminate structural artifacts and image fragmentation caused by highlights and noise interference in low-light areas, ensuring strict spatiotemporal geometric consistency while suppressing non-physical mesh folding distortion, even under conditions of rapid camera movement or low light.

[0023] This application introduces a dual reliability prior module (RPM) in both the spatial and temporal branches. This RPM acts as an interference suppressor, performing a filtering-then-enhancing mechanism. It filters out dynamic and environmental interference information from the feature level, providing a high-purity signal for the camera trajectory estimation algorithm. This allows the network to rigorously anchor spatial global homography, local mesh deformation, and cross-frame motion estimation to a reliable, rigid background. In various complex scenes, including normal, low-light, weak-texture, and fast-moving scenarios, it maintains smoother and more consistent camera motion trajectories, significantly improving video stability.

[0024] While effectively avoiding the forced radical non-physical mesh deformation of local pixels, the average alignment accuracy of this application is still effectively improved compared with the state-of-the-art baseline method. It can automatically learn feature selection and alignment based solely on photometric error without any manual ground truth annotation, and has strong environmental adaptability and robustness. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0026] Figure 1 This is a flowchart illustrating an unsupervised video stitching method based on reliability awareness and collaborative smoothing according to an embodiment of this application.

[0027] Figure 2 This is a schematic diagram of the architecture of an unsupervised video stitching system based on reliability awareness and collaborative smoothing according to an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Furthermore, the technical features involved in the various embodiments described below can be combined with each other as long as they do not conflict with each other.

[0029] Figure 1 This is a flowchart illustrating an unsupervised video stitching method based on reliability awareness and collaborative smoothing according to an embodiment of this application. Figure 2 This is a schematic diagram of the architecture of an unsupervised video stitching system based on reliability awareness and collaborative smoothing according to an embodiment of this application. Figure 2 As shown, the architecture of this application presents a three-stage cascaded structure, including a spatially aligned network (such as...). Figure 2 (A) in the text), time trajectory estimation network (e.g. Figure 2 (labeled B in the text) and cross-perspective collaborative smoothing networks (such as...) Figure 2 The structure consists of three parts (C), where the spatial alignment network and the temporal trajectory estimation network each embed a reliability prior module (RPM). Figure 2 (Mark D in the text). The following is combined with... Figure 1 and Figure 2 The embodiments of this application will be described in detail.

[0030] like Figure 1 As shown, the method includes:

[0031] Step 1: Obtain the first and second video sequences to be stitched together. The first and second video sequences are synchronously acquired by two overlapping image acquisition devices.

[0032] Step 2: Input the first video sequence and the second video sequence simultaneously into the trained spatial alignment network. The spatial alignment network performs coarse alignment and fine alignment on each synchronized frame image of the two video sequences to obtain the first local spatial alignment grid corresponding to the first video sequence and the second local spatial alignment grid corresponding to the second video sequence.

[0033] In some embodiments, step 2 may include:

[0034] Step 201: Use the images in the first video sequence as the target images. Images from the second video sequence are used as reference images. The target image and reference image of the same frame are simultaneously input into the trained spatial alignment network, wherein the structure of the spatial alignment network includes a first feature extraction backbone network and parallel coarse alignment branches and fine alignment branches.

[0035] Step 202: Use the first feature extraction backbone network (e.g., ResNet-18) to extract a multi-scale feature pyramid based on the target image and the reference image.

[0036] Step 203: Estimate the depth-sensing homography matrix from the target image to the reference image based on the multi-scale feature pyramid using the coarse alignment branch. To mitigate projection stretching distortion caused by unidirectional deformation, a bidirectional decomposition strategy is employed to obtain the first global mesh flow corresponding to the target image based on the depth-aware homography matrix. and the second global grid flow corresponding to the reference image. .

[0037] In some embodiments, step 203 may specifically include:

[0038] Deep semantic features at the H / 16 scale are obtained from a multi-scale feature pyramid, where H is the original height of the input image. A 4-point representation is then used to estimate the depth-aware homography matrix from the target image to the reference image based on the H / 16 scale deep semantic features. The depth-sensing homography matrix from the target image to the reference image can also be represented as: ,in, Representing the image number The horizontal displacement vector components of each corner point during the homography transformation process. Representing the image number The vertical displacement vector components of each corner point during the homography transformation process. A bidirectional decomposition strategy is employed, creating a hypothetical intermediate plane. The image is warped to this virtual intermediate plane, and the first homography matrix from the target image to the intermediate plane is obtained by dividing the offset of each element in the depth-aware homography matrix by half. .

[0039] Using depth-sensing homography matrix and the first homography matrix The second homography matrix from the reference image to the intermediate plane is obtained through matrix operations. .

[0040] The first homography matrix is ​​used as the first global grid flow. The second homography matrix serves as the second global grid flow. .

[0041] Step 204: Construct a local cost volume based on the multi-scale feature pyramid using the fine alignment branch, and predict the first local grid flow corresponding to the target image and the second local grid flow corresponding to the reference image based on the local cost volume.

[0042] In some embodiments, step 204 may specifically include: the fine alignment branch includes a spatial reliability prior module, a feature enhancement module, and a fine alignment module. Wherein,

[0043] The spatial reliability prior module is used to output a spatial confidence mask in the single-frame spatial dimension based on the multi-scale feature pyramid (i.e., extracted in step 201). This characterizes the alignment reliability of each pixel position;

[0044] Semantic features at the H / 4 scale (i.e., high-resolution feature maps at the 1 / 4 scale) are obtained from the multi-scale feature pyramid, and then the feature enhancement module is used based on the spatial confidence mask. Semantic features at the H / 4 scale Element-by-element modulation is performed to obtain the modulated features. ; By multiplying the modulated features element-wise with the original features, interference from occlusion, projection distortion, and low-texture regions on subsequent fine alignment is suppressed. To recover potentially attenuated weak textures that have been filtered out, a spatial attention mechanism is used to enhance the modulated features, highlighting structural edge details, resulting in enhanced features. ; ,in This represents the spatial attention calculation function.

[0045] The fine alignment module includes a costly convolutional unit and a mesh regressor. The costly convolutional unit is used to align the enhanced features. Combined with the first homography matrix Second homography matrix Construct the corresponding first local cost volume Second local cost volume Using a grid regressor According to the first partial cost volume Predicting the first local grid flow According to the second partial cost volume Predicting the second local grid flow .

[0046] Step 205: Create the first local mesh. Flow superimposed on the first global grid flow The first local spatially aligned mesh is obtained. The second local mesh flow Superimposed on the second global grid flow The second local spatial alignment mesh is obtained above. .

[0047] Step 3: Use the trained temporal trajectory estimation network to process the first video sequence and the second video sequence respectively, and generate the first camera trajectory corresponding to the first video sequence and the second camera trajectory corresponding to the second video sequence.

[0048] Existing methods for time-frame matching are easily interfered with by dynamic objects. This application extends the reliability prior module to time trajectory estimation. In some embodiments, step 3 may specifically include:

[0049] The first video sequence and the second video sequence are respectively input into the trained temporal trajectory estimation network, wherein the structure of the temporal trajectory estimation network includes a second feature extraction backbone network, a temporal reliability prior module, a time cost volume module, and a structure-preserving regressor.

[0050] For each video sequence :

[0051] 1. A time reliability prior module is used to output a corresponding time confidence mask based on the temporal correlation between adjacent frames in the inter-frame time dimension. To characterize the motion consistency of cross-frame correspondence, confidence attenuation is applied to temporal inconsistencies caused by dynamic occlusion, sudden changes in illumination, and rapid motion, thereby constraining error propagation during cross-frame information fusion.

[0052] 2. Utilize a second feature extraction backbone network (e.g., ResNet-18) to extract the corresponding multi-scale feature pyramid, and obtain H / 4 scale semantic features from the multi-scale feature pyramid. and time confidence mask Semantic features at the H / 4 scale Modulation is performed to obtain the features filtered by the mask. ; .

[0053] 3. Utilize time cost and volume modules based on the characteristics of the masked filter. Constructing Time Cost Volume And encode it as a latent motion feature .

[0054] 4. Instead of using fully connected layers, it utilizes a structure-preserving regressor based on underlying motion features using global average pooling (GAP). Residual mesh deformation is obtained , ,in, This represents the global average pooling function. This indicates a dimension reshaping operation, in which the feature tensor after GAP aggregation is transformed into the precise dimensional shape required for the local residual mesh. This represents a 1x1 convolution operation.

[0055] 5. Using a chain rule, accumulate the residual mesh deformation corresponding to each frame to generate the camera trajectory corresponding to the video sequence. , Indicates the image frame index. For video sequence indexing.

[0056] Step 4: Align the mesh according to the first local space. and the trajectory of the first camera Obtain the first stitching trajectory corresponding to the first video sequence. Align the mesh according to the second local space Second camera trajectory The second stitching trajectory corresponding to the second video sequence is obtained. Furthermore, utilizing a pre-trained cross-view collaborative smoothing network, and employing a spatiotemporal global flattening and symmetrical cross-view cross-attention mechanism, the first splicing trajectory is... Second splicing trajectory Joint optimization is performed to obtain the first smooth stitching trajectory. Second smooth splicing trajectory Specifically, this can include:

[0057] Step 401: The cross-view collaborative smoothing network includes a shallow encoder, a symmetric parallel cross-attention layer, a smoothing backbone network, and a decoding head.

[0058] Step 402: Use a shallow encoder to stitch the first trajectory. Second splicing trajectory Projecting each into a high-dimensional dense feature embedding matrix yields the first dense feature embedding. Second dense feature embedding The purpose of this operation is to map the low-dimensional coordinate offset to a semantic feature space with rich representational capabilities. Then, the first dense features are embedded... Second dense feature embedding To flatten the spacetime dimensions, the flattening formula is: This endows the network with a global receptive field, enabling it to retrieve motion cues throughout the entire time window. Among these, This represents the dense feature embedding before flattening. This represents the flattened dense feature embedding. Batch size B represents the number of samples processed in parallel during a single forward propagation of the network, time dimension T represents the number of consecutive video frames in the input sequence, height dimension H represents the height of the spatial feature map (dense feature embedding) generated after projection by the shallow encoder, width dimension W represents the width of the projected spatial feature map, and number of channels C represents the dimension of the dense feature embedding, used to encode the deep semantic information of the motion field.

[0059] Step 403: Employ a symmetrical parallel cross-attention layer and use a symmetrical cross-view cross-attention mechanism to embed the first dense feature after flattening. And the second dense feature embedding after flattening By aligning the splicing trajectories of the two video sequences, joint features are obtained. .

[0060] In some embodiments, step 403 may specifically include:

[0061] After feature flattening, the network needs to allow the trajectory information from the two views to interact and correct, embedding the flattened first dense feature. And the second dense feature embedding after flattening Perform linear projections to obtain the corresponding query matrix, key matrix, and value matrix;

[0062] Calculate two symmetric cross-attention weights based on the query matrix, key matrix, and value matrix using a normalized exponential function: , Among them, vision Figure 1 (Images from the first video sequence) Looking towards Figure 2 The attention weights for alignment (of images in the second video sequence) are: It is through vision Figure 1 query matrix With vision Figure 2 key matrix The result, calculated after scaling the dot product and normalizing it using the Softmax activation function, quantifies the feature similarity and geometric correspondence between two views in the global spatiotemporal flattened feature space. It defines the view... Figure 2 What proportion (weight) should be used to extract and integrate the various spatial location information into the view? Figure 1 In the corresponding position. Similarly, view Figure 2 Towards Figure 1 The attention weights for alignment are It is through vision Figure 2 query matrix With vision Figure 1 key matrix The result is obtained by scaling the dot product and normalizing it using the Softmax activation function. Here, d represents the feature channel dimension of the query matrix and the key matrix, and the superscript... This indicates transpose.

[0063] The stitching trajectories of two video sequences are aligned using two symmetrical attention weights based on the flattened first and second dense feature embeddings. Specifically, this is achieved using two equations: , Soft guidance corrects an unreliable region in one view by referencing a reliable region in another view. For example, in the equation In the middle, it indicates guiding the view. Figure 1 Unreliable area reference view Figure 2 Correction is performed on reliable areas. Unreliable areas were Reliable area correction, As the subject of correction, it preserves the initial motion representation of the view, which may contain distortions or inaccurate trajectory estimates due to local disturbances; and These represent the views (i.e., reference sides) to which the corresponding reliable regions belong during correction, and the regions in the views with high confidence. Y represents the enhanced view features after correction, and the subscript indicates the index of the video sequence to which the view belongs, i.e., the first alignment feature corresponding to the first video sequence. (i.e., already seen) Figure 2 Aligned views Figure 1 Features) and second alignment features corresponding to the second video sequence. (i.e., already seen) Figure 1 Aligned views Figure 2 feature).

[0064] Fusion of first alignment features Second alignment feature Obtain joint features .

[0065] Step 404: Combine features The input consists of a smooth backbone network composed of stacked 3D dilated residual convolutions with different dilation rates. By setting a stepwise dilation rate, the network can perform deep aggregation of the global spatiotemporal context with a large receptive field without increasing additional computational overhead, and output highly smooth joint feature maps, i.e., smooth features. ,in, A function representing a smoothed backbone network.

[0066] Step 405: Using the decoding head Smoothing features The image is then reprojected back into the two-dimensional physical motion space. The decoder directly outputs the residual mesh offset between the reference view and the target view, i.e., the first residual smoothing increment. Second residual smoothing increment And smooth the first residual increment Superimposed on the first stitching trajectory Generate the first smooth stitching trajectory. The second residual smoothing increment Superimposed on the second stitching trajectory Generate a second smooth stitching trajectory. This process refines the original coarse trajectory, resulting in a final stable and aligned stitching path.

[0067] Step 5: Based on the first smooth splicing trajectory Aligning the network with the first local space The first video sequence is deformed and then processed according to the second smooth stitching trajectory. Second local spatial alignment network The second video sequence is deformed, and the deformed first and second video sequences are merged and spliced ​​to obtain a spliced ​​video with spatiotemporal consistency.

[0068] Continue to refer to Figure 2 In some embodiments, the spatial reliability prior module and the temporal reliability prior module (hereinafter collectively referred to as the RPM module) have the same structure, consisting of multiple stacked convolutional layers and a sigmoid activation function, respectively. The output spatial confidence mask and temporal confidence mask are in the range of 0 to 1, that is, the output is a reliability confidence map with the same spatial dimension as the input and a value range of 0 to 1. Each network uses the reliability map to modulate the original features by multiplying them element by element to perform dynamic masking.

[0069] In some embodiments, the spatial alignment network, the temporal trajectory estimation network, and the cross-view collaborative smoothing network are jointly optimized and trained in an unsupervised manner using a multi-objective loss function as follows:

[0070] First, the spatial alignment network and the temporal trajectory estimation network are independently pre-trained based on the first and second training video sequences in the training set. The first and second training video sequences are synchronously acquired by two overlapping image acquisition devices.

[0071] Among them, the spatial alignment network and the temporal trajectory estimation network are trained by photometric alignment (overlapping regions). The joint loss consists of photometric error and TPS mesh regularization loss. Implicitly driven, RPM also automatically learns the filtering strategy through backpropagation under this objective. The specific form of the joint loss is:

[0072] ;

[0073] in, and These represent the original reference image and the target image, respectively. This represents the deformed grid coordinates of the reference and target images predicted by the network. This indicates an image deformation operation based on the deformation mesh coordinates to generate an aligned image. This indicates the calculation of the L1 distance. It is an overlapping region mask. The first term calculates the L1 difference within the mask. This indicates that inter-mesh and intra-mesh constraints are combined to maintain structural regularization. yes The weight hyperparameters, Represents the set of vertices of the mesh (referring to the mesh in both the reference view and the target view).

[0074] The training of the spatial reliability prior module and the temporal reliability prior module is implicitly driven only by photometric error, specifically the pixel-level color or brightness difference between the two aligned images in the overlapping region. This is represented in the formula as the L1 norm difference between the two deformed images within the mask region, requiring no manual annotation. Since light reflection and textureless regions lead to extremely high photometric errors, the network automatically learns to predict very low reliability scores for these outlier regions to minimize global deformation loss during training. To ensure that the RPM can receive high-fidelity gradient flow for effective updates under unsupervised conditions, the temporal trajectory estimation branch abandons the standard fully connected layer and introduces a structure-preserving regressor based on global average pooling.

[0075] After pre-training, the outputs of the trained spatial alignment network and temporal trajectory estimation network are used as input to train the cross-view collaborative smoothing network. The training of the cross-view collaborative smoothing network utilizes a term approximated by trajectory data. Second derivative motion smoothing term Grid shape retention items Consistency item of trajectory overlap area and online alignment items Combination loss drive.

[0076] Trajectory data approximation terms: ,in This indicates the calculation of the L2 norm.

[0077] Second derivative motion smoothing term:

[0078] ;

[0079] Where k represents the frame offset within the time sliding window. When calculating the motion smoothing term of the second derivative, the current frame is used as the reference. Centered on the frame, k is used to traverse adjacent frames within a given time radius r, thereby constraining the trajectory smoothness within a local time period.

[0080] Grid shape reserved items ;

[0081] in, , These represent the deformed grid coordinates corresponding to the first and second video sequences, respectively, obtained after updating based on the smooth trajectory.

[0082] Trajectory overlap region consistency item: ;

[0083] Online alignment items: ;

[0084] in, The full name is Online Hard Example Mining, which represents a training strategy operation. It means that when calculating the photometric alignment loss of overlapping areas, instead of treating all pixels equally, it dynamically selects the pixel areas with the largest alignment error for key penalty and optimization, thereby more effectively eliminating stitching gaps and artifacts.

[0085] To further illustrate the beneficial effects achieved by this application, the calculation methods of the specific performance evaluation indicators (PSNR / SSIM, stability, distortion rate) of the video stitching model of this application are described.

[0086] The specific evaluation metrics used are alignment accuracy (including PSNR and SSIM), stability score, and geometric distortion rate to evaluate the overall performance of video stitching.

[0087] Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are calculated in the overlapping regions of the images to quantify the sharpness and geometric consistency of the stitching seams. Statistical stability score and geometric distortion rate are used to measure the model's ability to suppress video jitter and preserve the structural shape of the images; lower values ​​indicate better performance.

[0088] Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are primarily used to evaluate the pixel-level and structural consistency between the target view and the reference view in the overlapping alignment region.

[0089] The calculation process for PSNR is as follows:

[0090] ;

[0091] Where MSE is the mean square error of the overlapping region, and m and n represent the total width and total height of pixels in the image overlap evaluation region (i.e., the total number of columns and rows involved in the calculation). Indicates the pixel width index. This indicates the pixel height index.

[0092] ;

[0093] in, This is the maximum possible value of the image pixels (e.g., 255 in an 8-bit image).

[0094] The calculation process for SSIM is as follows:

[0095] ;

[0096] Where x and y are image patches representing the overlapping regions of the two images, respectively. and These are their pixel averages. and For its variance, For covariance, and To maintain a stable constant, the closer the SSIM value is to 1, the better the alignment.

[0097] The Stability Score, defined as the average smoothness loss of the smoothed trajectory, is used to evaluate image stabilization performance; a lower score indicates better stability. It is primarily measured by calculating the second derivative of the smoothed stitched trajectory ($S$) within a sliding window, penalizing sudden jitter and acceleration. The calculation process is as follows:

[0098] ;

[0099] in, This represents the total number of frames in the video. This represents the smoothed continuous camera trajectory (also known as the final smoothed trajectory or smooth stitched trajectory).

[0100] Geometric distortion rate is used to evaluate a model's ability to preserve shape in non-overlapping regions. It is quantified by calculating the regularization constraints of a smooth mesh (including intra-grid and inter-grid deformations). A lower value indicates that the video content retains its natural structure more effectively, without non-physical stretching or folding distortion. The calculation process is as follows:

[0101] ;

[0102] in, and represents the predicted reference view and smoothed mesh vertices in frame t, respectively.

[0103] The test results, based on tests conducted on the StabStitch-D benchmark dataset across four categories—Regular, Low-Texture, Low-Light, and Fast-Moving—are shown in Tables 1 to 3.

[0104] Table 1. Quantization Comparison of Alignment Accuracy (PSNR / SSIM)

[0105]

[0106] Table 1 shows the comparison of the image alignment accuracy of this application with existing traditional image stitching methods (LCP), deep learning-based image stitching methods (UDIS, UDIS++), and video stitching methods (StabStitch, StabStitch++). The results show that this application achieves the best PSNR and SSIM average score in all four complex scenarios, verifying that the proposed method can achieve the most accurate stitching quality and geometric consistency of overlapping areas while maintaining lightweight computation.

[0107] Table 2. Quantitative Comparison of Camera Motion Trajectory Stability (Stability Score)

[0108]

[0109] Table 2 compares the video stability scores of the current state-of-the-art unsupervised video stitching baseline model with those of this application (lower scores indicate smoother and more stable trajectories). Thanks to the Reliability Prior Module (RPM) effectively filtering out interference features, this application achieved the lowest stability error scores in all test scenarios. The advantage in trajectory smoothness is particularly significant in low-light and fast-motion scenarios.

[0110] Table 3. Comparison of Geometric Distortion Rate in Non-overlapping Regions

[0111]

[0112] Table 3 evaluates the model's shape preservation ability in non-overlapping regions (lower distortion scores are better). The comparative results demonstrate that the baseline model often causes severe mesh stretching and deformation in complex scenes due to forced alignment, while this application effectively suppresses distortion peaks through the active negotiation mechanism of the cross-view collaborative smoothing network, achieving the lowest average distortion scores in all four scenes, significantly enhancing the natural visual experience of the video output.

[0113] The RPM module exhibits spontaneous task adaptability: spatial RPM automatically learns to activate corners and edges; temporal RPM automatically learns to mask high-contrast dynamic light reflections (such as water surface reflections), greatly improving generalization robustness to complex scenes.

[0114] According to one embodiment of this application, an unsupervised video stitching system based on reliability perception and collaborative smoothing is provided, including: a video acquisition module, a spatial alignment module, a temporal trajectory estimation module, a trajectory smoothing module, and a video stitching module; wherein, the video acquisition module is configured to acquire a first video sequence and a second video sequence to be stitched, the first video sequence and the second video sequence being synchronously acquired by two overlapping image acquisition devices; the spatial alignment module is configured to simultaneously input the first video sequence and the second video sequence into a trained spatial alignment network, the spatial alignment network performing coarse alignment and fine alignment processing on each synchronous frame image of the two video sequences, obtaining a first local spatial alignment grid corresponding to the first video sequence and a second local spatial alignment grid corresponding to the second video sequence, wherein a spatial reliability prior module for generating a spatial confidence mask is introduced into the spatial alignment network to characterize the alignment reliability of each pixel position; the temporal trajectory estimation module is configured to process the first video sequence and the second video sequence respectively using the trained temporal trajectory estimation network to generate the first video sequence. The system includes a first camera trajectory and a second camera trajectory corresponding to the second video sequence. A temporal reliability prior module for generating a temporal confidence mask is introduced into the temporal trajectory estimation network to characterize the motion consistency of cross-frame correspondences. A trajectory smoothing module is configured to obtain a first stitched trajectory corresponding to the first video sequence based on a first local spatial alignment grid and the first camera trajectory, and a second stitched trajectory corresponding to the second video sequence based on a second local spatial alignment grid and the second camera trajectory. A trained cross-view collaborative smoothing network is used, employing a spatiotemporal global flattening and symmetrical cross-view cross-attention mechanism, to jointly optimize the first and second stitched trajectories, resulting in a first smoothed stitched trajectory and a second smoothed stitched trajectory. A video stitching module is configured to deform the first video sequence based on the first smoothed stitched trajectory and the first local spatial alignment network, deform the second video sequence based on the second smoothed stitched trajectory and the second local spatial alignment network, and then fuse and stitch the deformed first and second video sequences to obtain a stitched video with spatiotemporal consistency. Since the system implementation is basically similar to the method implementation, details of the relevant technical features and their effects can be found in the corresponding descriptions of the method implementation provided above.

[0115] The embodiments provided in this application can also be computer program products. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application. A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. Computer-readable storage media may include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0116] The related equipment, media, and program products of this application, by adopting the above-mentioned model and training method, can achieve excellent seamless video generation quality while maintaining efficient inference on limited hardware platforms such as UAV wide-area monitoring and handheld mobile video acquisition.

[0117] It should be understood that the above description of the preferred embodiments is quite detailed and should not be construed as a limitation on the scope of protection of this application. Those skilled in the art, under the guidance of this application, may make substitutions or modifications without departing from the scope of protection of the claims of this application, all of which fall within the scope of protection of this application. The scope of protection claimed in this application shall be determined by the appended claims.

Claims

1. An unsupervised video stitching method based on reliability awareness and collaborative smoothing, characterized in that, include: Step 1: Obtain the first and second video sequences to be stitched together. The first and second video sequences are synchronously acquired by two overlapping image acquisition devices. Step 2: Input the first video sequence and the second video sequence simultaneously into the trained spatial alignment network. The spatial alignment network performs coarse alignment and fine alignment on each synchronized frame image of the two video sequences to obtain the first local spatial alignment grid corresponding to the first video sequence and the second local spatial alignment grid corresponding to the second video sequence. A spatial reliability prior module for generating a spatial confidence mask is introduced into the spatial alignment network to characterize the alignment reliability of each pixel position. Step 3: Use the trained temporal trajectory estimation network to process the first video sequence and the second video sequence respectively, and generate the first camera trajectory corresponding to the first video sequence and the second camera trajectory corresponding to the second video sequence. In the temporal trajectory estimation network, a temporal reliability prior module is introduced to generate the temporal confidence mask to characterize the motion consistency of the cross-frame correspondence. Step 4: Obtain the first stitching trajectory corresponding to the first video sequence based on the first local spatial alignment grid and the first camera trajectory; obtain the second stitching trajectory corresponding to the second video sequence based on the second local spatial alignment grid and the second camera trajectory; and use the trained cross-view collaborative smoothing network, employing a spatiotemporal global flattening and symmetrical cross-view cross-attention mechanism, to jointly optimize the first stitching trajectory and the second stitching trajectory to obtain the first smooth stitching trajectory and the second smooth stitching trajectory. Step 5: Deform the first video sequence according to the first smooth stitching trajectory and the first local spatial alignment network, deform the second video sequence according to the second smooth stitching trajectory and the second local spatial alignment network, and fuse and stitch the deformed first video sequence and the second video sequence to obtain a stitched video with spatiotemporal consistency.

2. The unsupervised video stitching method based on reliability awareness and collaborative smoothing according to claim 1, characterized in that, Step 2 includes: Step 201: Use the images in the first video sequence as target images and the images in the second video sequence as reference images, and simultaneously input the target image and reference image of the same frame into the trained spatial alignment network. The spatial alignment network includes a first feature extraction backbone network and parallel coarse alignment branches and fine alignment branches. Step 202: Use the first feature extraction backbone network to extract a multi-scale feature pyramid based on the target image and the reference image; Step 203: Utilize the coarse alignment branch to estimate the depth-sensing homography matrix from the target image to the reference image based on the multi-scale feature pyramid, and use a bidirectional decomposition strategy to obtain the first global grid flow corresponding to the target image and the second global grid flow corresponding to the reference image based on the depth-sensing homography matrix. Step 204: Construct a local cost volume based on the multi-scale feature pyramid using the fine alignment branch, and predict the first local grid flow corresponding to the target image and the second local grid flow corresponding to the reference image based on the local cost volume; Step 205: Superimpose the first local mesh stream onto the first global mesh stream to obtain the first local spatially aligned mesh, and superimpose the second local mesh stream onto the second global mesh stream to obtain the second local spatially aligned mesh.

3. The unsupervised video stitching method based on reliability awareness and collaborative smoothing according to claim 2, characterized in that, Step 203 includes: Deep semantic features at the H / 16 scale are obtained from the multi-scale feature pyramid, and the depth-sensing homography matrix from the target image to the reference image is estimated based on the deep semantic features at the H / 16 scale using a 4-point representation method. A bidirectional decomposition strategy is adopted, an intermediate plane is imagined, and the first homography matrix from the target image to the intermediate plane is obtained according to 1 / 2 of each offset in the depth-sensing homography matrix; Using the depth-sensing homography matrix and the first homography matrix, a second homography matrix from the reference image to the intermediate plane is obtained through matrix operations; The first homography matrix is ​​used as the first global grid flow, and the second homography matrix is ​​used as the second global grid flow.

4. The unsupervised video stitching method based on reliability awareness and collaborative smoothing according to claim 3, characterized in that, Step 204 includes: the fine alignment branch includes a spatial reliability prior module, a feature enhancement module, and a fine alignment module; wherein, The spatial reliability prior module outputs a spatial confidence mask based on the multi-scale feature pyramid. Semantic features at the H / 4 scale are obtained from the multi-scale feature pyramid, and the H / 4 scale semantic features are modulated according to the spatial confidence mask using the feature enhancement module. The modulated features are then enhanced using a spatial attention mechanism. The fine alignment module includes a cost volume construction unit and a grid regressor. The cost volume construction unit combines the enhanced features with the first homography matrix and the second homography matrix to construct the corresponding first local cost volume and second local cost volume. The grid regressor predicts the first local grid flow based on the first local cost volume and the second local grid flow based on the second local cost volume.

5. The unsupervised video stitching method based on reliability awareness and collaborative smoothing according to claim 4, characterized in that, Step 3 includes: The first video sequence and the second video sequence are respectively input into the trained temporal trajectory estimation network, which includes a second feature extraction backbone network, a temporal reliability prior module, a time cost volume module, and a structure-preserving regressor. For each video sequence: The time reliability prior module is used to output the corresponding time confidence mask; The corresponding multi-scale feature pyramid is extracted using the second feature extraction backbone network. The semantic features at the H / 4 scale are obtained from the multi-scale feature pyramid, and the H / 4 scale semantic features are modulated by the time confidence mask to obtain the masked features. The time cost volume module is used to construct a time cost volume based on the features filtered by the mask and encode it as latent motion features. The residual mesh deformation is obtained based on the potential motion characteristics using a structure-preserving regressor employing global average pooling. The residual mesh deformation corresponding to each frame of the image is accumulated using a chain rule to generate the camera trajectory corresponding to the video sequence.

6. The unsupervised video stitching method based on reliability awareness and collaborative smoothing according to claim 5, characterized in that, The spatial reliability prior module and the temporal reliability prior module have the same structure, consisting of multiple stacked convolutional layers and a sigmoid activation function, respectively. The output spatial confidence mask and temporal confidence mask are in the range of 0 to 1.

7. The unsupervised video stitching method based on reliability awareness and collaborative smoothing according to claim 6, characterized in that, The spatial alignment network, temporal trajectory estimation network, and cross-view collaborative smoothing network were trained as follows: First, the spatial alignment network and the temporal trajectory estimation network are pre-trained independently; After pre-training, the outputs of the trained spatial alignment network and temporal trajectory estimation network are used as input to train the cross-view collaborative smoothing network. Among them, the training of the spatial alignment network and the temporal trajectory estimation network is implicitly driven by the joint loss consisting of photometric alignment and grid regularization, and the training of the spatial reliability prior module and the temporal reliability prior module is implicitly driven only by photometric error. The training of the cross-view collaborative smoothing network is driven by a combined loss consisting of trajectory data approximation, second derivative motion smoothing, grid shape preservation, trajectory overlap consistency, and online alignment.

8. The unsupervised video stitching method based on reliability awareness and collaborative smoothing according to claim 1, characterized in that, In step 4, the trained cross-view collaborative smoothing network is used, employing a spatiotemporal global flattening and symmetrical cross-view cross-attention mechanism, to jointly optimize the first and second stitching trajectories, resulting in a first smooth stitching trajectory and a second smooth stitching trajectory, including: Step 401: The cross-view collaborative smoothing network includes a shallow encoder, a symmetrical parallel cross attention layer, a smoothing backbone network, and a decoding head; Step 402: Use a shallow encoder to map the first splicing trajectory and the second splicing trajectory into a first dense feature embedding and a second dense feature embedding, respectively, and flatten the first dense feature embedding and the second dense feature embedding in the spatiotemporal dimension. Step 403: Using a symmetrical parallel cross-attention layer, a symmetrical cross-view cross-attention mechanism is employed to align the splicing trajectories of the two video sequences based on the flattened first dense feature embedding and the second dense feature embedding, thereby obtaining joint features. Step 404: Input the joint features into a smooth backbone network composed of 3D dilated residual convolutions stacked with different dilation rates to obtain smooth features; Step 405: Use the decoding head to decode the smoothing feature into a first residual smoothing increment and a second residual smoothing increment, and superimpose the first residual smoothing increment onto the first stitching trajectory to generate a first smooth stitching trajectory, and superimpose the second residual smoothing increment onto the second stitching trajectory to generate a second smooth stitching trajectory.

9. The unsupervised video stitching method based on reliability awareness and collaborative smoothing according to claim 8, characterized in that, Step 403 includes: The flattened first and second dense feature embeddings are linearly projected to obtain the corresponding query vector, key vector and value vector. Two symmetric attention weights are calculated using a normalized exponential function based on the query vector, key vector, and value vector. By using two symmetrical attention weights based on the flattened first dense feature embedding and second dense feature embedding, the splicing trajectories of the two video sequences are aligned to obtain the first alignment feature corresponding to the first video sequence and the second alignment feature corresponding to the second video sequence. The first alignment feature and the second alignment feature are combined to obtain the joint feature.

10. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements the unsupervised video stitching method based on reliability awareness and collaborative smoothing as described in any one of claims 1-9.

Images