Real-time stitching method and system for finished camera video based on heterogeneous computing

Abstract

The application provides a finished product camera video real-time splicing method and system based on heterogeneous computing, which comprises the following steps: processing double-channel fisheye video streams through offline calibration to achieve global coarse alignment, and obtaining double-channel equirectangular projection video images containing target video images and reference video images; performing feature matching and inter-frame jitter suppression processing based on optical flow estimation on the target video images and the reference video images to obtain a pixel set matching point set; sequentially performing local homography matrix solving based on grid division and image warping processing on the pixel set matching point set to obtain a warped image; performing image fusion on the warped image and the reference video image through brightness matching and distance weight processing to obtain a fused image; and obtaining a fused video stream through the fused image. The application realizes pixel-level weighted fusion by defining the range through an overlapping area mask, correcting color brightness deviation through histogram matching, and generating dynamic weight coefficients based on distance transformation.

Description

Technical Field

[0001] This invention relates to the field of panoramic video stitching technology, and in particular to a method and system for real-time stitching of finished camera videos based on heterogeneous computing. Background Technology

[0002] With its 360-degree full-scene visual presentation capability, panoramic video is increasingly widely used in fields such as virtual reality (VR), intelligent monitoring, autonomous driving assistance, and immersive live streaming. Its core requirement is to generate high-definition, seamless, and real-time panoramic images through dual-channel image stitching technology.

[0003] Existing panoramic image stitching methods are mainly divided into two categories: traditional methods and deep learning-based methods. Traditional methods often rely on manually designed feature extraction algorithms such as SIFT and SURF for point matching, and then achieve image alignment through a global homography matrix. However, these methods have low feature point extraction efficiency and high mismatch rate in complex scenes, and the global homography matrix is ​​difficult to adapt to local deformations in areas with complex parallax, which easily leads to problems such as stitching misalignment and boundary breaks. Although deep learning-based methods improve the robustness of feature matching through models such as optical flow estimation, they suffer from high computational overhead and cannot meet the requirements of real-time stitching scenarios.

[0004] Current mainstream panoramic image stitching technologies still suffer from multi-dimensional performance bottlenecks, limiting their application effectiveness in demanding scenarios: First, the inter-frame jitter suppression mechanism is not perfect. Even minor vibrations of the shooting equipment or small movements of dynamic targets in the scene can easily lead to non-rigid jitter, local pixel shifts, and image flicker in the stitched panoramic image, severely affecting visual continuity. Second, the overlapping area fusion strategy lacks adaptive adjustment capabilities, making it difficult to dynamically adapt to brightness imbalances and color deviations caused by differences in lighting conditions and shooting parameters between two frames. Furthermore, it cannot effectively handle unnatural texture transitions, easily leaving obvious artificial traces at the stitching boundaries. Third... First, the alignment robustness in complex scenes is insufficient. When the shooting scene has significant parallax or extreme conditions such as weak texture / low light, the existing alignment algorithm is difficult to accurately capture the local features and motion patterns of the image, which easily leads to problems such as alignment misalignment and regional distortion, resulting in a decrease in the overall consistency of the stitching. Second, the adaptability to real-time and high-definition processing requirements is insufficient. Existing technologies either rely on CPU serial computing architecture, and when processing dual-channel video with resolutions of 4K and above, the single-frame stitching time far exceeds the real-time threshold, or although parallel computing is introduced, it is not deeply coupled with the task characteristics of the stitching process, resulting in low parallel efficiency and difficulty in meeting real-time processing requirements while ensuring high-definition stitching quality. Summary of the Invention

[0005] In view of the above, the main objective of this invention is to propose a real-time video stitching method and system for finished cameras based on heterogeneous computing, so as to solve the above-mentioned technical problems.

[0006] This invention proposes a real-time video stitching method for finished cameras based on heterogeneous computing, the method comprising the following steps:

[0007] Step 1: Process the dual-channel fisheye video streams through offline calibration to achieve global coarse alignment, and obtain the target video image and the reference video image;

[0008] Step 2: Perform feature matching and inter-frame jitter suppression processing based on optical flow estimation on the target video image and the reference video image to obtain a pixel-level matching point set;

[0009] Step 3: Sequentially solve the local homography matrix based on grid partitioning and perform image distortion processing on the pixel-level matching point set to obtain a distorted image;

[0010] Step 4: Based on the distorted image and the reference video image, perform image fusion by processing brightness matching and distance weights to obtain a fused image; obtain the fused video stream from the fused image.

[0011] This invention also proposes a real-time video stitching system for finished cameras based on heterogeneous computing, the system comprising:

[0012] Image preprocessing module, used for:

[0013] Offline calibration is used to process the dual fisheye video streams to achieve global coarse alignment, resulting in the target video image and the reference video image.

[0014] The feature point matching and inter-frame jitter suppression module is used for:

[0015] Feature matching and inter-frame jitter suppression based on optical flow estimation are performed on the target video image and the reference video image to obtain a pixel-level matching point set;

[0016] The local homography matrix solving and image distortion module is used for:

[0017] The pixel-level matching point set is sequentially subjected to local homography matrix solving based on grid division and image distortion processing to obtain a distorted image;

[0018] Image fusion module, used for:

[0019] Based on the distorted image and the reference video image, image fusion is performed through brightness matching and distance weighting to obtain a fused image; the fused video stream is then obtained from the fused image.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0021] 1. This invention captures pixel-level motion vectors through pixel-by-pixel correlation analysis, which not only provides underlying motion information support for accurate feature point matching, but also suppresses inter-frame jitter through dynamic filtering of historical frame optical flow. At the same time, it breaks through the rigid assumption of global homography matrix, avoids alignment deviation in complex parallax regions, and improves the robustness of feature matching and scene adaptability.

[0022] 2. This invention, for each grid cell, uses an inverse power method to fit the local homography matrix based on the coordinate correspondence of its internal feature points, and ensures the stability of the solution by setting convergence conditions. In the image fusion stage, the range is defined by overlapping region masking, color and brightness deviations are corrected by histogram matching, and dynamic weight coefficients are generated based on distance transformation to achieve pixel-level weighted fusion.

[0023] 3. This invention combines the features of mesh partitioning and GPU parallel computing to design a new stitching algorithm flow. By leveraging the single instruction multiple data execution mode of GPU thread bundles, the speed of solving the local homography matrix, pixel geometric distortion, and pixel-by-pixel image fusion is accelerated, so as to balance video stitching quality and computational efficiency. Attached Figure Description

[0024] Figure 1 This is a flowchart of the real-time video stitching method for finished cameras based on heterogeneous computing proposed in this invention;

[0025] Figure 2 This is a schematic diagram of the heterogeneous processing framework of the real-time video stitching method for finished cameras based on heterogeneous computing proposed in this invention.

[0026] Figure 3 This is a schematic diagram of the framework of the real-time video stitching system for finished cameras based on heterogeneous computing proposed in this invention. Detailed Implementation

[0027] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0028] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0029] Please see Figure 1This invention proposes a real-time video stitching method for finished cameras based on heterogeneous computing, which includes the following steps:

[0030] Step 1: Process the dual-channel fisheye video streams through offline calibration to achieve global coarse alignment, and obtain the target video image and the reference video image;

[0031] In step 1, the dual-channel fisheye video streams are processed through offline calibration to achieve global coarse alignment, resulting in the target video image and the reference video image. The specific steps are as follows:

[0032] The intrinsic and extrinsic parameters of the dual fisheye camera were obtained through offline calibration technology;

[0033] The intrinsic and extrinsic parameters of a dual fisheye camera are used to process the dual fisheye video streams to obtain the target video image and the reference video image.

[0034] Furthermore, the dual fisheye cameras are stacked back-to-back to form a panoramic camera system, covering a 360-degree field of view. Camera parameters are obtained through offline calibration to achieve global coarse alignment of the two video streams.

[0035] Step 2: Perform feature matching and inter-frame jitter suppression processing based on optical flow estimation on the target video image and the reference video image to obtain a pixel-level matching point set;

[0036] In step 2, feature matching based on optical flow estimation and inter-frame jitter suppression are performed on the target video image and the reference video image to obtain a pixel-level matching point set. The specific steps are as follows:

[0037] Based on the preset input size of the optical flow model, the target video image and the reference video image are scaled respectively to obtain the scaled target video image and the scaled reference video image.

[0038] The pixel-level optical flow vector between the scaled target video image and the scaled reference video image is calculated using a pre-trained optical flow estimation model to obtain the initial optical flow field;

[0039] Based on the dimensions of the target video image, the reference video image, the scaled target video image, and the scaled reference video image, the scaling factor for height and the scaling factor for width are calculated.

[0040] The initial optical flow field is scaled using scaling factors for height and width to obtain the scaled optical flow field.

[0041] The scaled optical flow field is optimized using an interpolation algorithm to obtain an optimized optical flow field;

[0042] The difference between the optimized optical flow field and the historical optical flow field is compared. If the difference is greater than a preset threshold, the historical optical flow field is updated. Otherwise, the historical optical flow field smoothed by the previous frame is used to obtain the optical flow field smoothed by the historical frame.

[0043] Based on the smoothed optical flow field of historical frames and the pixel coordinates in the target video image, the coordinates of the matching points in the reference video image corresponding to the target video image are calculated to obtain a set of pixel-level matching points.

[0044] In the process of calculating the pixel-level optical flow vector between the scaled target video image and the scaled reference video image using a pre-trained optical flow estimation model to obtain the initial optical flow field, the following relationship exists:

[0045] ;

[0046] in, This represents the scaled target video image. This represents a scaled-down reference video image. Represents the initial optical flow field. This indicates the optical flow estimation operation. Indicates the mapping relationship. Represents the coordinates of pixels in an image. Represents pixels Offset in the x-axis direction, Represents pixels Offset in the y-axis direction, Indicates the transpose symbol;

[0047] In the process of calculating the scaling factors for height and width based on the dimensions of the target video image, the reference video image, the scaled target video image, and the scaled reference video image, the following relationship exists:

[0048] ;

[0049] in, The scaling factor representing the height. The scaling factor representing the width. Indicates the height of the target video image and the reference video image. This represents the width of the target video image and the reference video image. This represents the height of the scaled target video image and the scaled reference video image. This indicates the width of the scaled target video image and the scaled reference video image;

[0050] In the process of scaling the initial optical flow field using scaling factors for height and width to obtain the scaled optical flow field, the following relationship exists:

[0051] ;

[0052] in, This represents the scaled optical flow field;

[0053] In the process of optimizing the scaled optical flow field using an interpolation algorithm to obtain the optimized optical flow field, the following relationship exists:

[0054] ;

[0055] in, This represents the optimized optical flow field. Indicates the interpolation operation;

[0056] The process of comparing the optimized optical flow field with the historical optical flow field, updating the historical optical flow field when the difference exceeds a preset threshold, and otherwise using the historical optical flow field smoothed from the previous frame, yields the optical flow field smoothed from the historical frame. The corresponding relationship is as follows:

[0057] ;

[0058] in, This represents the optical flow field after smoothing through historical frames. Indicates the preset threshold. This indicates that it has undergone L2 norm processing;

[0059] In the process of calculating the coordinates of the matching points in the reference video image corresponding to the target video image based on the smoothed optical flow field of historical frames and the pixel coordinates in the target video image, in order to obtain the pixel-level matching point set, the following relationship exists:

[0060] ;

[0061] in, Represents the pixel coordinates in the target video image. This represents the coordinates of the matching point in the reference video image corresponding to the target video image.

[0062] It should be noted that the interpolation algorithm used in this step is specifically bilinear interpolation.

[0063] Furthermore, in this step, based on the optical flow estimation model, the pixel-level optical flow field distribution between the two images is calculated and feature point matching is completed to ensure the robustness of feature point matching in complex scenes such as weak texture and low illumination. At the same time, pixel-level dynamic filtering is used to suppress inter-frame jitter.

[0064] Specifically, in a video sequence, subtle errors in frame-by-frame optical flow estimation accumulate and are passed to the subsequently calculated local homography matrix, resulting in unpleasant temporal jitter in the stitched video. Given that the human visual system is often more sensitive to dynamic jitter than static artifacts, this type of jitter can severely impair the viewing experience.

[0065] To address this issue, this step designs a smoothing strategy based on motion consistency checks. The core of this strategy lies in maintaining a dynamically updated historical optical flow field. For each pixel in the current frame, its instantaneous displacement vector is compared with the historical flow data for that point. If the displacement change does not exceed a preset threshold... If the motion at a point is stable, the historical optical flow value is used to replace the current estimate to maintain stability; otherwise, significant motion is determined to have occurred, and the historical optical flow data is updated accordingly. This method can effectively suppress high-frequency jitter caused by estimation noise while preserving effective real motion.

[0066] Therefore, this invention can obtain any matching point with constant time complexity. This feature significantly simplifies the design of heterogeneous computing frameworks and the decision-making process for the number of sampling points. Specifically, increasing the number of sampling points helps to capture subtle changes in local areas, thereby providing more accurate constraints for the optimization of the homography matrix and improving the splicing accuracy. Conversely, if the current splicing quality already meets the requirements, reducing the number of sampling points can effectively reduce computation and storage overhead, achieving a flexible trade-off between efficiency and quality.

[0067] Step 3: Sequentially solve the local homography matrix based on grid partitioning and perform image distortion processing on the pixel set matching point set to obtain a distorted image;

[0068] In step 3, the pixel-level matching point set is sequentially subjected to local homography matrix solving based on grid partitioning and image distortion processing to obtain a distorted image. The specific steps are as follows:

[0069] The target video image is uniformly divided according to the preset grid size to obtain a grid set;

[0070] A GPU thread is allocated to each grid in the grid set, and the GPU thread is used to establish a linear relationship between the coordinates of matching points in the target video image and the reference video image through constraint equations; all matching point constraints are vertically stacked to form a total coefficient matrix;

[0071] The diagonal weight matrix is ​​constructed by weighting the points based on their distance from the grid center. The local homography matrix is ​​obtained by minimizing the objective function using the diagonal weight matrix and the total coefficient matrix.

[0072] Using the local homography matrix, the homogeneous coordinates of the pixel-level matching points in each grid are projected and transformed to calculate their coordinates in the reference video image coordinate system, so as to obtain the distorted image.

[0073] In establishing a linear relationship between the coordinates of matching points in the target video image and the reference video image using constraint equations, the constraint equations are expressed in the form of a matrix-vector product, and the corresponding relationship is as follows:

[0074] ;

[0075] In constructing the diagonal weight matrix by weighting the points based on their distances from the grid center, the following relationship exists:

[0076] ;

[0077] in, This represents the diagonal weight matrix. This indicates the operation of constructing a diagonal matrix. Both represent column vectors of the diagonal weight matrix. This represents the total number of matching points. Indicates the index of the matching points. This indicates the operation of retrieving the maximum value. This indicates processing via the natural exponential function. Represents the coordinates of the grid center point. Indicates the first The coordinates of the matching points This indicates the degree to which the weight decays with distance. Represents the regularization term;

[0078] In the process of solving the homography matrix by minimizing the objective function using the diagonal weight matrix and the total coefficient matrix to obtain the local homography matrix, the following relationship exists:

[0079] ;

[0080] in, Represents the local homography matrix. Represents the homography matrix. This means finding the expression that minimizes the expression. The value, Represents the total coefficient matrix;

[0081] In the process of using the local homography matrix to perform a projection transformation on the homogeneous coordinates of pixel-level matching points within each grid, and calculating their coordinates in the reference video image coordinate system to obtain the distorted image, the following relationship exists:

[0082] ;

[0083] in, Both represent row vectors of the local homography matrix. Represents the homogeneous coordinates of the matching point in the target video image. This represents the homogeneous coordinates of the matching point in the reference video image.

[0084] Specifically, by matching key feature points in a reference video image and a target video image, this invention can solve for a homography matrix. This matrix accurately describes the projection transformation relationship between the two image planes. Specifically, it is a 3x3 matrix that can map the homogeneous coordinates of any pixel in the reference image to the corresponding homogeneous coordinates in the target image through a linear transformation, thereby quantifying the geometric relationship between the two in terms of viewpoint, rotation, or scale.

[0085] Since estimating the homography matrix of each pixel in the target image is computationally expensive and the differences between solutions of adjacent pixels are small, this step divides the target image into grids and applies the homography estimated at the center of each grid cell to all pixels within that cell. This approach allows for flexible adjustment of the grid size and directly reduces the complexity of homography estimation for the target image.

[0086] Step 4: Based on the distorted image and the reference video image, perform image fusion by processing brightness matching and distance weighting to obtain a fused image; obtain the fused video stream from the fused image;

[0087] In step 4, image fusion is performed based on the distorted image and the reference video image through brightness matching and distance weighting to obtain a fused image; the fused video stream is then obtained from the fused image, with the specific steps as follows:

[0088] Within the overlapping region, histogram matching is performed on each color channel in the distorted image to adjust the brightness distribution of the distorted image to match the brightness distribution of the reference video image, thus obtaining the adjusted image.

[0089] Calculate the shortest Euclidean distance from each pixel in the adjusted image and the reference video image within the overlapping region to the boundary of the non-overlapping region of its own image; calculate the fusion weights based on the shortest Euclidean distance using the distance map;

[0090] By using fusion weights, each pixel in the overlapping region is summed in a weighted manner to obtain a fused image; the fused image is then used to obtain a fused video stream.

[0091] In the process of using fusion weights to perform a weighted summation of each pixel in the overlapping region to obtain the fused image, the following relationship exists:

[0092] ;

[0093] in, Represents the pixel coordinates of the merged image. This represents the fusion weights of corresponding pixels in the target video image. This represents the pixel coordinates of the adjusted image in the overlapping region. This indicates the pixel coordinates of the reference video image in the overlapping region.

[0094] Furthermore, to eliminate seams in video stitching, this step uses a boundary distance-weighted fusion algorithm. This algorithm fully utilizes the massive parallel computing power of the GPU. Its core lies in dynamically processing the fusion weights of each pixel in the reference frame and the precisely aligned target frame. The algorithm dynamically assigns fusion weights to pixels based on their distance from the boundary of the overlapping area: pixels closer to their own image have higher weights, while pixels closer to the seam are more likely to fuse information from the other image. Through this smooth weight transition, pixel-level seamless fusion is ultimately achieved, significantly improving the overall consistency and visual appeal of the synthesized image.

[0095] Please see Figure 2 The present invention also designed a heterogeneous processing framework image stitching process based on CPU-GPU collaborative scheduling. Through differentiated allocation of hardware resources and refined decomposition of task processes, a significant improvement in stitching performance was achieved.

[0096] First, a binary mask is generated for the reference image and the target image. The overlapping and non-overlapping regions are divided using the mask, and the available matching points are selected accordingly. In addition, the grid structure of the distortion space of the target image is defined. The core objective of the entire preparation stage is to decompose the problem to be solved into several data classes that can be executed in parallel and to perform data preprocessing in advance.

[0097] Subsequently, all core computations are efficiently mapped to the GPU, including: solving multiple local homography matrices in parallel based on pre-computed matching points; and performing parallel warping transformations on the target image based on the mesh and the solved homography matrices.

[0098] Finally, a parallel fusion algorithm is used to synthesize the final image. This design is in line with the GPU's single instruction multiple data architecture, transforming the originally serial geometry and photometric calculation process into large-scale parallel processing, thereby improving throughput.

[0099] Furthermore, this invention uses a 2D thread model for GPU parallel computing, with the thread block size set to... ,in and Both are designed to be 32, representing the number of threads in the H and W dimensions of the thread block, respectively; this design is highly compatible with the GPU's warp scheduling mechanism, ensuring that all thread blocks can be divided into complete warps, avoiding resource waste caused by thread branch divergence;

[0100] To achieve complete coverage of the input data dimensions, the number of blocks in the grid is determined through a dynamic calculation strategy. For a size of... (Size after being divided into grids) or The input data, among which, This indicates the number of grid cells in the image height direction. This represents the number of grid cells along the image width. The total number of threads in a thread block within a GPU is... The number of grid blocks along the height direction is or Along the width direction or Even if the data dimension is not... or Even when the coordinates are integer multiples of the coordinates, edge regions can still be effectively processed. The resulting 2D mesh completely covers all data coordinates, establishing a one-to-one mapping between threads and data elements.

[0101] Please see Figure 3 This invention also provides a real-time video stitching system for finished cameras based on heterogeneous computing, the system comprising:

[0102] Image preprocessing module, used for:

[0103] Offline calibration is used to process the dual fisheye video streams to achieve global coarse alignment, resulting in the target video image and the reference video image.

[0104] The feature point matching and inter-frame jitter suppression module is used for:

[0105] Feature matching and inter-frame jitter suppression based on optical flow estimation are performed on the target video image and the reference video image to obtain a pixel-level matching point set;

[0106] The local homography matrix solving and image distortion module is used for:

[0107] The pixel-level matching point set is sequentially subjected to local homography matrix solving based on grid division and image distortion processing to obtain a distorted image;

[0108] Image fusion module, used for:

[0109] Based on the distorted image and the reference video image, image fusion is performed through brightness matching and distance weighting to obtain a fused image; the fused video stream is then obtained from the fused image.

[0110] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0111] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0112] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for real-time video stitching from finished cameras based on heterogeneous computing, characterized in that, The method includes the following steps: Step 1: Process the dual-channel fisheye video streams through offline calibration to achieve global coarse alignment, and obtain the target video image and the reference video image; Step 2: Perform feature matching and inter-frame jitter suppression based on optical flow estimation on the target video image and the reference video image to obtain a pixel-level matching point set. The specific steps are as follows: Based on the preset input size of the optical flow model, the target video image and the reference video image are scaled respectively to obtain the scaled target video image and the scaled reference video image. The pixel-level optical flow vector between the scaled target video image and the scaled reference video image is calculated using a pre-trained optical flow estimation model to obtain the initial optical flow field; Based on the dimensions of the target video image, the reference video image, the scaled target video image, and the scaled reference video image, the scaling factor for height and the scaling factor for width are calculated. The initial optical flow field is scaled using scaling factors for height and width to obtain the scaled optical flow field. The scaled optical flow field is optimized using an interpolation algorithm to obtain an optimized optical flow field; The difference between the optimized optical flow field and the historical optical flow field is compared. If the difference is greater than a preset threshold, the historical optical flow field is updated. Otherwise, the historical optical flow field smoothed by the previous frame is used to obtain the optical flow field smoothed by the historical frame. Based on the smoothed optical flow field of historical frames and the pixel coordinates in the target video image, the coordinates of the matching points in the reference video image corresponding to the target video image are calculated to obtain a set of pixel-level matching points. Step 3: Sequentially perform local homography matrix solving based on grid partitioning and image distortion processing on the pixel-level matching point set to obtain a distorted image. The specific steps are as follows: The target video image is uniformly divided according to the preset grid size to obtain a grid set; A GPU thread is allocated to each grid in the grid set, and the GPU thread is used to establish a linear relationship between the coordinates of matching points in the target video image and the reference video image through constraint equations; all matching point constraints are vertically stacked to form a total coefficient matrix; The matching points are weighted according to their distance from the grid center to construct a diagonal weight matrix; Using the diagonal weight matrix and the total coefficient matrix, the homography matrix is ​​solved by minimizing the objective function to obtain the local homography matrix; Using the local homography matrix, the homogeneous coordinates of the pixel-level matching points in each grid are projected and transformed to calculate their coordinates in the reference video image coordinate system, so as to obtain the distorted image. Step 4: Based on the distorted image and the reference video image, perform image fusion by brightness matching and distance weighting to obtain a fused image; obtain the fused video stream from the fused image.

2. The real-time video stitching method for finished cameras based on heterogeneous computing according to claim 1, characterized in that, In step 1, the dual-channel fisheye video streams are processed through offline calibration to achieve global coarse alignment, resulting in the target video image and the reference video image. The specific steps are as follows: The intrinsic and extrinsic parameters of the dual fisheye camera were obtained through offline calibration technology; The intrinsic and extrinsic parameters of a dual fisheye camera are used to process the dual fisheye video streams to obtain the target video image and the reference video image.

3. The real-time video stitching method for finished cameras based on heterogeneous computing according to claim 2, characterized in that, In the process of calculating the pixel-level optical flow vector between the scaled target video image and the scaled reference video image using a pre-trained optical flow estimation model to obtain the initial optical flow field, the following relationship exists: ； in, This represents the scaled target video image. This represents a scaled-down reference video image. Represents the initial optical flow field. This indicates the optical flow estimation operation. Indicates the mapping relationship. Represents the coordinates of pixels in an image. Represents pixels Offset in the x-axis direction, Represents pixels Offset in the y-axis direction, Indicates the transpose symbol; In the process of calculating the scaling factors for height and width based on the dimensions of the target video image, the reference video image, the scaled target video image, and the scaled reference video image, the following relationship exists: ； in, The scaling factor representing the height. The scaling factor representing the width. Indicates the height of the target video image and the reference video image. This represents the width of the target video image and the reference video image. This represents the height of the scaled target video image and the scaled reference video image. This indicates the width of the scaled target video image and the scaled reference video image; In the process of scaling the initial optical flow field using scaling factors for height and width to obtain the scaled optical flow field, the following relationship exists: ； in, This represents the scaled optical flow field; In the process of optimizing the scaled optical flow field using an interpolation algorithm to obtain the optimized optical flow field, the following relationship exists: ； in, This represents the optimized optical flow field. Indicates the interpolation operation; The process of comparing the optimized optical flow field with the historical optical flow field, updating the historical optical flow field when the difference exceeds a preset threshold, and otherwise using the historical optical flow field smoothed from the previous frame, yields the optical flow field smoothed from the historical frame. The corresponding relationship is as follows: ； in, This represents the optical flow field after smoothing through historical frames. Indicates the preset threshold. This indicates that it has undergone L2 norm processing; In the process of calculating the coordinates of the matching points in the reference video image corresponding to the target video image based on the smoothed optical flow field of historical frames and the pixel coordinates in the target video image, in order to obtain the pixel-level matching point set, the following relationship exists: ； in, Represents the pixel coordinates in the target video image. This represents the coordinates of the matching point in the reference video image corresponding to the target video image.

4. The real-time video stitching method for finished cameras based on heterogeneous computing according to claim 3, characterized in that, In establishing a linear relationship between the coordinates of matching points in the target video image and the reference video image using constraint equations, the constraint equations are expressed in the form of a matrix-vector product, and the corresponding relationship is as follows: ； In constructing the diagonal weight matrix by weighting the points based on their distances from the grid center, the following relationship exists: ； in, Represents the diagonal weight matrix. This indicates the operation of constructing a diagonal matrix. Both represent column vectors of the diagonal weight matrix. This represents the total number of matching points. Indicates the index of the matching points. This indicates the operation of retrieving the maximum value. This indicates processing via the natural exponential function. Represents the coordinates of the grid center point. Indicates the first The coordinates of the matching points This indicates the degree to which the weight decays with distance. Represents the regularization term; In the process of solving the homography matrix by minimizing the objective function using the diagonal weight matrix and the total coefficient matrix to obtain the local homography matrix, the following relationship exists: ； in, Represents the local homography matrix. Represents the homography matrix. This means finding the expression that minimizes the expression. The value, Represents the total coefficient matrix; In the process of using the local homography matrix to perform a projection transformation on the homogeneous coordinates of pixel-level matching points within each grid, and calculating their coordinates in the reference video image coordinate system to obtain the distorted image, the following relationship exists: ； in, Both represent row vectors of the local homography matrix. Represents the homogeneous coordinates of the matching point in the target video image. This represents the homogeneous coordinates of the matching point in the reference video image.

5. The real-time video stitching method for finished cameras based on heterogeneous computing according to claim 4, characterized in that, In step 4, image fusion is performed based on the distorted image and the reference video image through brightness matching and distance weighting to obtain a fused image; the fused video stream is then obtained from the fused image, with the specific steps as follows: Within the overlapping region, histogram matching is performed on each color channel in the distorted image to adjust the brightness distribution of the distorted image to match the brightness distribution of the reference video image, thus obtaining the adjusted image. Calculate the shortest Euclidean distance from each pixel in the adjusted image and the reference video image within the overlapping region to the boundary of the non-overlapping region of its own image; calculate the fusion weights based on the shortest Euclidean distance using the distance map; By using fusion weights, each pixel in the overlapping region is summed in a weighted manner to obtain a fused image; the fused image is then used to obtain a fused video stream.

6. The real-time video stitching method for finished cameras based on heterogeneous computing according to claim 5, characterized in that, In the process of using fusion weights to perform a weighted summation of each pixel in the overlapping region to obtain the fused image, the following relationship exists: ； in, Represents the pixel coordinates of the merged image. This represents the fusion weights of corresponding pixels in the target video image. This represents the pixel coordinates of the adjusted image in the overlapping region. This indicates the pixel coordinates of the reference video image in the overlapping region.

7. A real-time video stitching system for finished cameras based on heterogeneous computing, characterized in that, The system employs any one of the heterogeneous computing-based real-time video stitching methods for finished cameras according to claims 1 to 6, and the system comprises: Image preprocessing module, used for: Offline calibration is used to process the dual fisheye video streams to achieve global coarse alignment, resulting in the target video image and the reference video image. The feature point matching and inter-frame jitter suppression module is used for: Feature matching and inter-frame jitter suppression based on optical flow estimation are performed on the target video image and the reference video image to obtain a pixel-level matching point set; The local homography matrix solving and image distortion module is used for: The pixel-level matching point set is sequentially subjected to local homography matrix solving based on grid division and image distortion processing to obtain a distorted image; Image fusion module, used for: Based on the distorted image and the reference video image, image fusion is performed through brightness matching and distance weighting to obtain a fused image; the fused video stream is then obtained from the fused image.

Images