A real-time panoramic video stitching method based on embedded platform

Abstract

The present invention discloses a panoramic video real-time stitching method based on an embedded platform. The panoramic stitching device based on the method includes a visible light camera, a video processing board and a structural member. The visible light camera is evenly distributed on the side elevation of the structural member. The video processing board is used to receive the visible light camera video, stitch the video in real time, and output the real-time panoramic video. The method includes the following steps: 1) camera image registration: collect visible light camera images, and establish a geometric correspondence between images by matching the features of the images; 2) global optimization of multi-camera parameters: use the Levenberg-Marquardt algorithm for global optimization to minimize the reprojection error; obtain an accurate homography matrix and camera parameters; 3) image fusion: perform a projection transformation on the image before stitching, and map all pixels to a unified plane; then use an image fusion method based on Laplace pyramid transformation to obtain the final fused image. The present invention realizes the output of real-time panoramic video on an embedded platform.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a real-time panoramic video stitching method based on an embedded platform. Background Art

[0002] Panoramic video stitching technology mainly solves the problem that a single imaging device has a limited viewing angle and cannot fully present the real scene. Through the stitching algorithm, multiple videos are stitched and encoded into one panoramic video, retaining the information in the original video and removing duplicate redundant information, overcoming the physical limitations of the sensor, and providing large-viewing-angle, high-resolution video information. This technology has very wide applications in both civil and military fields. In the civil field, it can be used for panoramic video monitoring, virtual reality scene construction, event live broadcast, and car assisted driving systems. In the military field, it can be used for closed-cabin vehicles such as tanks and submarines. While ensuring the safety of the cabin occupants, it can reduce blind spots and improve the full-space perception ability. With the continuous development of computer vision technology, panoramic video stitching technology has shown broad application prospects and important research significance.

[0003] The key to panoramic video stitching is the image fusion stitching algorithm, and the performance of the algorithm determines the quality of the stitched video. Currently, there are multiple commercial software that can stitch image sequences with overlapping areas to generate wide-angle images. At the same time, several 360° panoramic cameras that support panoramic stitching have been launched one after another. They build panoramas based on a series of images. However, these applications and cameras have restrictions on the acquisition environment and images, and the input images in actual engineering applications contain many potential defects, such as large baselines between cameras, large image parallax, scene illumination changes, etc. These factors seriously affect the stitching quality, and the algorithm is highly complex. It is difficult to run in real time on resource-limited embedded platforms and cannot meet the requirements of real-time processing of video stitching. The key steps of panoramic stitching mainly include image registration based on feature matching, optimal stitching seam search based on graph optimization, and image fusion based on optical flow. Traditional image feature matching is inefficient and time-consuming, and cannot be processed in real time on embedded platforms. Summary of the invention

[0004] The object of the present invention is to provide a panoramic video real-time stitching method based on an embedded platform, so as to realize output of real-time panoramic video on the embedded platform.

[0005] The present invention is achieved by adopting the following technical solutions:

[0006] A panoramic video real-time stitching method based on an embedded platform, wherein the method is based on a panoramic stitching device comprising a visible light camera, a video processing board and a housing, wherein the visible light cameras are evenly distributed on the side elevation of the housing, and the video processing board is used to receive the visible light camera video, stitch the video in real time, and output a real-time panoramic video;

[0007] The method comprises the following steps:

[0008] 1) Camera Image Registration

[0009] At the same time, visible light camera images are collected, and features are extracted for each image. By matching the feature points between adjacent images, the geometric correspondence between the images is established, and the homography matrix between the images is calculated based on the feature points in the overlapping areas of the images.

[0010] 2) Global optimization of multi-camera parameters

[0011] The homography matrix between every two adjacent images is obtained from the previous step. The homography matrix reflects the transformation relationship between images and the relative position relationship between cameras. The Levenberg-Marquardt algorithm is used for global optimization to minimize the reprojection error function. The Levenberg-Marquardt algorithm is used for global optimization to obtain accurate homography matrix and camera parameters.

[0012] 3) Image Fusion

[0013] Using the homography matrix and camera parameters obtained in the previous step, the images are projected before stitching, and all images are projected into a unified coordinate system. The final stitched image is displayed on a plane, and the camera parameters are used for reverse projection to map all pixels to a unified plane. After the projection transformation, image fusion is performed. In order to fully retain the image details and eliminate blur and ghosting, an image fusion method based on Laplace pyramid transform is adopted. The image is decomposed into different scales using Laplace pyramid transform, and then the texture, edge and corner features of the image at different scales are extracted. For overlapping areas, the same layers of their pyramids are merged according to the weighted average method. Finally, the merged pyramids are inversely Laplace transformed to obtain the fused image.

[0014] A further improvement of the present invention is that in step 1), image feature extraction and matching include three aspects: feature point detection, feature point descriptor vector generation and feature point matching.

[0015] A further improvement of the present invention is that in step 1), during the camera image registration stage, an offline calibration operation is added during the initialization process, with the purpose of outputting the internal and external parameters of each camera; the SIFT feature description algorithm is used to extract image features, and a mask is added to each image according to the camera relative position information to shorten the feature extraction time, and then the nearest neighbor search based on the KD tree is used for feature matching, and finally the image homography matrix is ​​calculated according to the feature matching results to restore the camera parameters.

[0016] A further improvement of the present invention is that the camera parameter matrices K and R are respectively:

[0017]

[0018]

[0019] Where f is the focal length, c is the optical center position, subscript x represents the horizontal direction, subscript y represents the vertical direction, H is the homography matrix, i represents the matrix row index, and j represents the matrix column index.

[0020] A further improvement of the present invention is that in step 2), the iteration formula of the Levenberg-Marquardt algorithm is:

[0021]

[0022] Where f is the function to be optimized, J is the Jacobian matrix of f, μ is the damping factor, and I is the identity matrix.

[0023] A further improvement of the present invention is that, in step 2), firstly, offline calibration parameters are used as default parameters, and the offline calibration selects a scene with a parallax less than 20 pixels for calibration, and the reprojection error of the camera parameters is obtained to be less than 1 pixel. In subsequent use, this set of parameters is used as a benchmark parameter to compare with the result of global optimization, and the L2 norm of the globally optimized camera extrinsic parameters and intrinsic parameters of the benchmark parameters are calculated respectively. If the result differs from the benchmark parameters by more than 15%, it is considered that the online calibration has failed in the current environment, and the benchmark parameters are used in the subsequent stitching process to ensure the stitching quality; if the deviation between the global optimization result and the benchmark parameter is less than 15%, the global optimization result is used, because the global optimization takes into account the influence of the current environmental factors, and a better stitching effect is obtained subsequently.

[0024] A further improvement of the present invention is that in step 3), in the image fusion stage, a step of updating the stitching seam online is added, and the parameters of the image fusion are determined by the results of the image registration and global optimization during initialization, and are related to the scene at the time of initialization. When the average parallax between the current scene and the scene at the time of initialization exceeds 10%, the position of the optimal stitching seam will also change accordingly; in the image fusion stage, the position of the optimal stitching seam is recalculated according to the current scene at a set time interval, and the image fusion parameters are updated.

[0025] A further improvement of the present invention is that, when the image parallax variation is greater than 20 pixels, using the original optimal stitching seam for stitching may cause a ghosting problem, thus affecting the stitching effect.

[0026] Compared with the prior art, the present invention has at least the following beneficial technical effects:

[0027] The present invention provides a panoramic video real-time stitching method based on an embedded platform, which uses scale-invariant feature transformation to detect features of input images. The present invention is improved in that, in the feature extraction stage, a separated Gaussian convolution is used to replace the ordinary two-dimensional Gaussian convolution. Compared with the ordinary convolution, the separated Gaussian convolution can greatly reduce the amount of calculation and reduce the image edge information loss caused by the ordinary convolution. By calibrating the system offline and combining the known camera position information, accurate camera internal and external parameters are obtained as reference parameters to provide accurate prior information for subsequent video stitching.

[0028] Furthermore, the verification method for global optimization of camera parameters proposed in the present invention adds verification of optimization results on the basis of traditional global optimization, thereby avoiding the problem of inaccurate optimization results caused by external environmental interference or single scene texture features, and improving the overall robustness and environmental adaptability of the system.

[0029] Furthermore, the present invention adds a process of real-time updating of the optimal stitching seam in the image fusion stage. During the operation of the device, the optimal stitching seam is updated and adjusted in real time according to the different parallax sizes of the scene in which the device is located, thereby eliminating ghosting caused by inaccurate stitching seams, allowing the system to work normally in multiple scenes with different parallaxes, thereby improving the video panoramic stitching quality and system robustness.

[0030] To solve this problem, a real-time feature matching technology combining online and offline is proposed. The relative position of the cameras is fixed by making a specific structure. Based on the relative position relationship of the cameras, the cameras are calibrated offline. According to the offline calibration results, in the real-time processing stage, the online calibration and global optimization results are verified according to prior information such as the relative position relationship of the cameras. This avoids calibration failures caused by external environmental factors such as the scene and excessive parameter deviations, and achieves fast and accurate image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Panoramic video stitching algorithm flow chart

[0032] Figure 2 Schematic diagram of Gaussian scale space extreme value detection.

[0033] Figure 3 (a) is a schematic diagram of a standard two-dimensional Gaussian convolution. Figure 3 (b) is a schematic diagram of separable Gaussian convolution.

[0034] Figure 4 Schematic diagram of camera imaging model

[0035] Figure 5 is a schematic diagram of cylindrical projection, where Figure 5 (a) is a schematic diagram of the image plane to be projected and the projected surface. Figure 5(b) is a schematic diagram of the projection of a point in space onto the cylindrical surface.

[0036] Figure 6 Schematic diagram of multi-band fusion. DETAILED DESCRIPTION

[0037] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to be able to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0038] The panoramic stitching equipment is composed of visible light cameras, video processing boards and structural parts. The visible light cameras are evenly distributed on the side facades of the structural parts. The video processing board simultaneously receives the visible light camera video, stitches the video in real time, and outputs real-time panoramic video. The algorithm flow is as follows: Figure 1 shown.

[0039] The present invention provides a panoramic video real-time stitching method based on an embedded platform, comprising the following steps:

[0040] 1) Image feature extraction and registration based on scale-invariant feature transformation

[0041] The scale-invariant feature transform is used to detect and describe local features of the input image. First, starting from the original input image, a blurred version of the image is generated by smoothing it with a Gaussian filter. The Gaussian filter is a linear smoothing filter that is used to remove high-frequency noise in the image and blur the image. Based on the first layer of blurred image, the Gaussian filter is applied again for smoothing to generate the second layer of blurred image. The above steps are repeated, and the degree of Gaussian blur is continuously increased to generate a series of blurred images to form a scale space pyramid. The standard deviation ratio between Gaussian filters at each scale level is 2, and image pyramids of different scales are generated to cover features at various scales. When constructing the scale space pyramid, the image is downsampled between different scale levels to ensure the consistency of image size and feature point positions between different scales. Figure 2As shown in the figure, in the scale space, the local minimum and maximum values ​​are determined by comparing the values ​​of each pixel with the surrounding pixels. At the same time, the pixel is accurately positioned by comparing the points in the adjacent scale space. Near the extreme points, Gaussian curvature calculation is used to determine the exact position of the key points. The key points are assigned directions to make them scale invariant and rotation invariant. The local image gradient histogram is used to generate a high-dimensional feature vector for the image area around each key point.

[0042] In order to reduce the computational complexity and number of parameters of the convolution operation, we use separable Gaussian convolution to replace the standard two-dimensional Gaussian convolution, such as Figure 3 As shown in the figure, ordinary Gaussian convolution uses N*N convolution kernel to calculate at each pixel position of the image. This method has a large amount of calculation and serious loss of image edge information. Using the separability of Gaussian function, separated Gaussian convolution decomposes a two-dimensional Gaussian convolution kernel into two one-dimensional Gaussian convolution kernels, filtering the image in the horizontal and vertical directions respectively. First, the image is convolved along the x direction with a 1*N convolution kernel, and then the image is convolved along the y direction with an N*1 convolution kernel. Compared with the standard two-dimensional Gaussian convolution, separated convolution can significantly reduce the amount of calculation while retaining the edge information of the image.

[0043] Image registration is to establish the geometric correspondence between images by extracting and matching features between adjacent images, and to calculate the homography matrix between images based on the feature points in the overlapping areas of the images. Image feature extraction and matching mainly involve three aspects: feature point detection, feature point descriptor vector generation, and feature point matching. Image feature point detection uses a general parallel computing architecture based on GPU to make full use of parallel computing resources and improve algorithm processing speed. Camera image registration includes offline calibration and online calibration. The camera imaging model is as follows: Figure 4 As shown in the figure, CXYZ is the camera coordinate system, point P is a point in space, cxy is the image coordinate system, point p is the projection of point P in space on the image coordinate system, and f is the focal length of the camera. The offline calibration estimates the extrinsic parameters of each camera through the actual relative position relationship of the cameras. The online calibration corrects the camera intrinsic and extrinsic parameters according to the images captured by the camera in real time, accurately estimates the homography matrix between images, and provides necessary information for subsequent image stitching.

[0044] 2) Global optimization of multi-camera parameters

[0045] Image registration can be used to obtain the relative position relationship between each camera and the internal and external parameters of each camera, but the parameters obtained in this way ignore the constraints between multiple images and will produce cumulative errors. Global optimization is needed to accurately calculate the camera parameters. The core problem of global optimization is to minimize the reprojection error function. The Levenberg-Marquardt algorithm is used to solve the problem. The iterative formula of the Levenberg-Marquardt algorithm is:

[0046]

[0047] Where f is the function to be optimized, J is the Jacobian matrix of f, μ is the damping factor, and I is the identity matrix. The Levenberg-Marquardt algorithm can be used to obtain accurate homography matrix and camera parameters through global optimization.

[0048] 3) Image Fusion

[0049] Since the images are taken at different angles and are not on the same plane, directly stitching the overlapping areas will destroy the consistency of the field of view. Therefore, the images need to be projected before stitching. The purpose is to project all images into a unified coordinate system (projection plane). The final stitched image needs to be displayed on a plane. It is necessary to use the camera parameters for reverse projection to map all pixels to a unified plane. After the projection transformation, image fusion is performed. In order to fully retain the image details and eliminate blur and ghosting, an image fusion method based on Laplace pyramid transform is adopted. The core idea of ​​the algorithm is to use Laplace pyramid transform to decompose the image into different scales, and then extract the obvious features of the image at different scales, such as texture and edge. Finally, these features are fused according to certain rules. The fused information is then inversely transformed to obtain the fused image.

[0050] In step 1), during the camera image registration phase, an offline calibration operation is added during the system initialization process. The purpose is to output the internal and external parameters of each camera as accurately as possible. Image stitching has not yet started during the initialization phase, so it is called offline calibration. The scale-invariant feature transformation feature description algorithm is used to extract image features. A mask is added to each image based on the camera relative position information to shorten the feature extraction time. After that, the nearest neighbor search based on the KD tree is used for feature matching. Finally, the image homography matrix is ​​calculated based on the feature matching results to restore the camera parameters. The camera parameter matrices K and R are respectively

[0051]

[0052]

[0053] Where f is the focal length, c is the optical center position, and H is the homography matrix. Offline calibration uses a robust and accurate feature extraction and matching algorithm. Compared with the fast feature matching method used in online calibration, it is not easily affected by the external environment and can provide an accurate priori benchmark to ensure that the results of subsequent steps do not have large deviations.

[0054] In step 2), the global optimization result of the camera parameters is verified using prior information. In actual use, due to the uncertainty of the external environment, when encountering scenes with small space, large parallax, and lack of obvious texture features, the results of the global optimization of the camera parameters are usually too different from the actual situation, resulting in the inability to output a complete panoramic stitching video, affecting the stability of the entire system. In response to this problem, the present invention first uses offline calibration parameters as the system default parameters. Offline calibration selects scenes with smaller parallax for calibration, and more accurate camera parameters can be obtained. In subsequent use, this set of parameters is used as a benchmark parameter to compare with the results of global optimization, and the L2 norms of the globally optimized camera extrinsic parameters and intrinsic parameters of the benchmark parameters are calculated respectively. If the difference between the two is too large, it is considered that the current external environment is not suitable for online calibration, and the benchmark parameters are used in the subsequent stitching process to ensure the stitching quality. If the deviation between the global optimization result and the benchmark parameter is within a certain range, the global optimization result is adopted. Because the global optimization takes into account the influence of current environmental factors, a better stitching effect can be obtained later.

[0055] In step 3), in the image fusion stage, an online update step of the stitching seam is added. The parameters of image fusion are determined by the results of image registration and global optimization when the system is initialized, and are related to the scene in which the system is located when the system is initialized. When the scene in which the system is located is greatly different from the scene when the system is initialized, the position of the optimal stitching seam will also change accordingly, especially when the image parallax changes greatly. At this time, using the original optimal stitching seam for stitching will cause ghosting problems, affecting the stitching effect. To address this problem, in the image fusion stage, the position of the optimal stitching seam is recalculated based on the current scene at regular time intervals, and the image fusion parameters are updated.

[0056] Example

[0057] 1) Overall process

[0058] The real-time panoramic video stitching algorithm based on an embedded platform proposed in the present invention mainly includes three stages, namely system initialization, system calibration and panoramic stitching. In the initialization stage, the system preset parameters are mainly loaded through the configuration file, including the input and output of the system, camera parameters, structural parameters and stitching algorithm parameters. The system calibration calibrates the system in different ways according to the preset parameters, and the available calibration methods include online calibration, offline calibration and hybrid calibration. After the system calibration, an accurate stitching template can be obtained, and the subsequent stitching algorithm uses the stitching template to perform real-time panoramic stitching of the video.

[0059] 2) System calibration

[0060] During the system calibration phase, the algorithm recalculates a new set of stitching templates for subsequent use based on the current environment through feature matching, image registration, and global optimization. Figure 4 As shown in the figure, by confirming the transformation relationship between a point P in space and its projection p on the image, each camera is calibrated to obtain the camera parameter matrices K and R. Then, the SIFT algorithm is used to extract and match features of each image, establish the geometric correspondence between images, and calculate the homography matrix between images based on the feature points in the overlapping areas of the images.

[0061]

[0062]

[0063] Where f is the focal length, c is the optical center position, and H is the homography matrix. The homography matrix describes the projection relationship between the two image planes and can be expressed as

[0064] Hp=p ′

[0065] Right now:

[0066]

[0067] p(u,v,1) and p ′ (u ′ ,v ′ ,1) are a set of corresponding points on the two image planes.

[0068] The reprojection error of the i-th set of matching point pairs can be written as:

[0069]

[0070] The sum of squares of the reprojection errors of all matching point pairs is:

[0071]

[0072] The purpose of global optimization is to minimize the reprojection error function, which is a nonlinear least squares optimization problem. It is solved using the Levenberg-Marquardt method. The iterative formula of the Levenberg-Marquardt algorithm is:

[0073]

[0074] Where f is the function to be optimized, J is the Jacobian matrix of f, μ is the damping factor, and I is the identity matrix. The Levenberg-Marquardt algorithm can be used to obtain accurate homography matrix and camera parameters through global optimization.

[0075] 3) Panoramic stitching

[0076] Panoramic stitching is to display the images captured by all cameras on a plane. Since the images are taken at different angles, they are not on the same plane. If the overlapping areas are stitched directly, the consistency of the field of view will be destroyed. Therefore, the images need to be projected before stitching. The purpose is to project all images into a unified coordinate system. The cylindrical projection method is used. The cylindrical projection uses the coordinate origin as the center point of the cylinder and the camera focal length as the radius of a cylinder as the projection surface. The schematic diagram of the cylindrical projection is shown in the figure below. Figure 5 As shown. The cylindrical panoramic image can meet 360-degree viewing in the horizontal direction, has a good visual effect, and the cylindrical projection is consistent with the actual placement of the camera. The pixel coordinates after cylindrical projection are:

[0077]

[0078]

[0079] After the projection transformation is completed, image fusion can be performed. In order to retain the high-frequency components of the image (that is, the details of the image), a multi-band fusion method is used to establish a Laplacian pyramid so that the information on each frequency band is retained and fused together. Figure 6 As shown in the figure, multi-band fusion first calculates the Gaussian pyramid and Laplacian pyramid of the input image, then fuses the Laplacian pyramids at the same level by weighted averaging for the overlapping areas, and finally performs an inverse Laplace transform on the merged pyramids to obtain the final fused image.

[0080] Although the present invention has been described in detail above with general descriptions and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements may be made thereto based on the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all fall within the scope of protection claimed by the present invention.

Claims

1. A panoramic video real-time stitching method based on an embedded platform, characterized in that: The panoramic stitching device based on the method includes a visible light camera, a video processing board and a shell. The visible light cameras are evenly distributed on the side facade of the shell. The video processing board is used to receive the visible light camera video, stitch the video in real time, and output the real-time panoramic video. The method comprises the following steps: 1) Camera Image Registration Camera image registration includes offline calibration and online calibration. Offline calibration collects visible light camera images at the same time, extracts features from each image, and establishes geometric correspondence between images by matching feature points between adjacent images. The homography matrix between images is calculated based on the feature points in the overlapping area of ​​the images. Image feature extraction and matching include feature point detection, feature point descriptor vector generation and feature point matching. In the camera image registration stage, offline calibration operations are added during the initialization process to output the internal and external parameters of each camera. The SIFT feature description algorithm is used to extract image features, and masks are added to each image based on the relative position information of the camera to shorten the feature extraction time. Then, the nearest neighbor search based on the KD tree is used for feature matching. Finally, the image homography matrix is ​​calculated based on the feature matching results to restore the camera parameters. 2) Global optimization of multi-camera parameters The homography matrix between every two adjacent images is obtained from the previous step. The homography matrix reflects the transformation relationship between images and the relative position relationship between cameras. The Levenberg-Marquardt algorithm is used for global optimization to minimize the reprojection error function. The Levenberg-Marquardt algorithm is used for global optimization to obtain accurate homography matrix and camera parameters. First, the offline calibration parameters are used as the default parameters. The offline calibration selects scenes with a parallax of less than 20 pixels for calibration, and the reprojection error of the camera parameters is less than 1 pixel. In subsequent use, this set of parameters is used as the benchmark parameters to compare with the results of global optimization. The L2 norms of the globally optimized camera extrinsic parameters and intrinsic parameters of the benchmark parameters are calculated respectively. If the result differs from the benchmark parameters by more than 15%, it is considered that the online calibration has failed in the current environment, and the benchmark parameters are used in the subsequent stitching process to ensure the stitching quality. If the deviation between the global optimization result and the benchmark parameter is less than 15%, the global optimization result is used. Because the global optimization takes into account the influence of current environmental factors, a better stitching effect is obtained later. 3) Image Fusion Using the homography matrix and camera parameters obtained in the previous step, the images are projected before stitching, and all images are projected into a unified coordinate system. The final stitched image is displayed on a plane, and the camera parameters are used for reverse projection to map all pixels to a unified plane; image fusion is performed after the projection transformation. In order to fully retain the image details and eliminate blur and ghosting, an image fusion method based on Laplace pyramid transform is adopted; the image is decomposed into different scales using Laplace pyramid transform, and then the texture, edge and corner features of the image at different scales are extracted. For overlapping areas, the same layers of their pyramids are merged according to the weighted average method, and finally the merged pyramid is inversely Laplace transformed to obtain the fused image; in the image fusion stage, the step of updating the stitching seam online is added, and the position of the optimal stitching seam is recalculated according to the current scene at a set time interval, and the image fusion parameters are updated.

2. The method for real-time panoramic video stitching based on an embedded platform according to claim 1, characterized in that: The camera parameter matrices K and R are: Where f is the focal length, c is the optical center position, subscript x represents the horizontal direction, subscript y represents the vertical direction, H is the homography matrix, i represents the matrix row index, and j represents the matrix column index.

3. The method for real-time panoramic video stitching based on an embedded platform according to claim 2, characterized in that: In step 2), the iteration formula of the Levenberg-Marquardt algorithm is: Where f is the function to be optimized, J is the Jacobian matrix of f, μ is the damping factor, and I is the identity matrix.

4. The method for real-time panoramic video stitching based on an embedded platform according to claim 3, characterized in that: In step 3), in the image fusion stage, an online update step of the stitching seam is added. The parameters of the image fusion are determined by the results of the image registration and global optimization during initialization, and are related to the scene at the time of initialization. When the average parallax between the current scene and the scene at the time of initialization exceeds 10%, the position of the optimal stitching seam will also change accordingly. In the image fusion stage, the position of the optimal stitching seam is recalculated according to the current scene at a set time interval, and the image fusion parameters are updated.

5. The method for real-time panoramic video stitching based on an embedded platform according to claim 4, characterized in that: When the image parallax change is greater than 20 pixels, using the original optimal stitching seam for stitching will cause ghosting problems, affecting the stitching effect.

Images