Multi-channel video stitching method and device based on embedded technology development platform

Abstract

The invention provides a multi-channel video stitching method and device based on an embedded technology development platform, and belongs to the field of video stitching. The core technology problems of low efficiency, high energy consumption, high delay, difficulty in modeling and the like in video stitching are solved; the method specifically comprises the following steps: acquiring and calculating an external parameter matrix of each camera, and performing distortion correction on a video frame; dividing data acquired by all the cameras into a group A and a group B, and extracting independent cameras; defining an energy function, searching splicing lines of different video frames based on a BFS algorithm, and sequentially splicing the video frames in the group A and the group B to complete preliminary splicing; performing illumination consistency compensation on the preliminarily spliced video frames by using a lightweight depth compensation network technology, and performing optimization processing on the compensated video frames by using a pyramid multi-scale fusion technology to obtain a spliced video; according to the invention, distortion correction, geometric alignment and video stitching are carried out on the videos shot by the cameras, so that a high-quality stitched video is obtained, and the video stitching efficiency is improved.

Description

Technical Field

[0001] This invention relates to a multi-channel video splicing method and apparatus based on an embedded technology development platform, and pertains to the field of video splicing.

[0002] This invention belongs to the field of computer vision and embedded systems technology, specifically relating to a method and system for real-time stitching of multi-channel video based on an embedded hardware platform, which is suitable for wide-field video synthesis of large-scale camera arrays. Background Technology

[0003] Existing multi-channel video stitching methods have the following shortcomings:

[0004] Multi-path lighting modeling is highly complex: as the number of cameras increases, the dimensionality of the mathematical model grows exponentially, and the parameter space becomes extremely complex. In addition, since the sensor response curves of different cameras are different, if the initial lighting model is not established accurately, the subsequent global constraint equations may underfit or overfit, resulting in the brightness of some stitching areas being distorted.

[0005] The curse of dimensionality and model complexity: Essentially, the parameters that the lighting model needs to estimate (such as the response function of each camera, exposure parameters, and ambient light components) increase nonlinearly with the number of cameras. This not only leads to an explosion in computational load, but also makes the parameter space high-dimensional and non-convex, making it difficult to find the global optimal solution. From the perspective of optimization theory, this problem becomes an ill-conditioned optimization problem, which is extremely difficult to solve.

[0006] Systematic errors introduced by sensor differences: The sensor response curves of different cameras have inherent differences; even when shooting the same scene, the pixel values ​​output by each camera have systematic deviations; if the initial calibration (such as calibration) is inaccurate, this error will be carried into the model, which makes it very difficult to build a unified imaging model. The initial error will affect the entire optimization process as a priori bias.

[0007] Failure risks in the optimization process: Due to the complexity of the model, an oversimplified model is used to control the amount of computation, which makes it unable to capture real lighting changes. The result is discontinuous brightness or obvious "blocking effect" at the stitching area. The model is over-adapted to the training data (such as a few frames) and is too sensitive to noise and local changes, which leads to brightness compensation distortion and may produce unnatural color spots or texture smoothness. In ill-posed problems, traditional global optimization algorithms (such as gradient descent-based methods) are very likely to converge to local extrema, which will result in unacceptable brightness distortion in the stitching area.

[0008] Color compensation may introduce artifacts: In scenes with strong lighting contrast (such as when part of the camera's shooting area is in direct sunlight and another part is in shadow), the brightness compensation model may need to be adjusted significantly to balance the overall image; in addition, when existing systems use gradient domain compensation, if the local lighting difference exceeds the threshold set by the algorithm, the brightness gradient is over-smoothed, resulting in a significant loss of image details and a decrease in the clarity of the stitched texture.

[0009] Compensation distortion under extreme lighting: When the light ratio in a scene (such as the direct sunlight area and the shadow area) far exceeds the dynamic range of the sensor, forced brightness mapping requires a large, non-linear adjustment range; this adjustment may cause pixel values ​​to be "stretched" or "compressed" to an unreasonable range, resulting in color cast, abnormal color saturation or loss of contrast, thus producing unnatural artifacts and destroying the physical authenticity of the image.

[0010] Limitations and loss of detail in gradient domain fusion: Gradient domain methods are a classic technique for solving stitching seams, but their core lies in smoothing gradients. When local lighting differences are too large, the algorithm may over-smooth the lighting gradient in that area to eliminate obvious seams. The gradient information of an image not only contains lighting transitions but also carries key texture details. Excessive gradient smoothing will directly erase these high-frequency information, resulting in blurred textures and lost details in the stitching area, i.e., "over-smoothing," making the stitching area look like it has been "smeared," with a significant decrease in clarity.

[0011] The dilemma of algorithm robustness and threshold setting: The algorithm relies on a preset threshold to determine when to compensate and the strength of compensation; in complex real-world environments, a fixed threshold is difficult to handle all scenarios; if the threshold is set too high, it may lead to insufficient compensation in medium-contrast scenarios, and the seams may still be visible; if the threshold is set too low, it will inevitably cause the above-mentioned overcompensation and loss of detail problems in high-contrast scenarios; this reflects the shortcomings of traditional methods in generalization and adaptability.

[0012] Embedded platform resources are limited: For low-power ARM or FPGA platforms, computing resources are limited and memory bandwidth is restricted, making it difficult to fully realize high-precision real-time stitching; if existing systems choose low-resolution or low-precision floating-point calculations to improve speed, the accuracy of illumination compensation will decrease, and slight brightness changes may reappear at the stitching boundary; if a high-precision mode is used, the frame rate will decrease, making it difficult to meet real-time requirements. Summary of the Invention

[0013] In view of the shortcomings of existing technologies, the purpose of this invention is to provide a multi-channel video splicing method and apparatus based on an embedded technology development platform, aiming to solve the problem of low video splicing efficiency.

[0014] To achieve the above objectives, the present invention provides a multi-channel video stitching method based on an embedded technology development platform, comprising:

[0015] Connect various cameras in the target area to the embedded platform and initialize the cameras; calculate the extrinsic parameter matrix of each camera and perform distortion correction on the video frames captured by each camera; divide all cameras into Group A, which overlaps with the video frames captured by the reference camera, and assign the remaining cameras to Group B; perform geometric alignment on the video frames in Group A and Group B, and extract the independent cameras.

[0016] Define an energy function, use the BFS algorithm to find the stitching lines of different video frames, and stitch the video frames in Group A and Group B in turn to complete the initial stitching; use a lightweight depth compensation network to perform illumination consistency compensation on the initially stitched video frames, and then use pyramid multi-scale fusion to optimize the compensated video frames to obtain the stitched video.

[0017] The embedded platform outputs the stitched video and provides feedback on the camera IDs of the individual cameras and the video frames they captured.

[0018] Furthermore, the steps for calculating the extrinsic parameter matrix are as follows:

[0019] Perform camera calibration for each camera and obtain the distortion coefficients corresponding to each camera;

[0020] Obtain the real-world coordinates of the reference object and its pixel coordinates in the video, and calculate the extrinsic parameter matrix for each camera;

[0021] Obtain the intrinsic parameter matrix Kc of the camera, and let the rotation matrix be Rc and the translation vector be Tc;

[0022] Combine matrix Rc and column vector Tc into extrinsic matrix Em;

[0023] Let the i-th real-world coordinate of the reference object be (Xs) (i) Ys (i) Zs (i) ), pixel coordinates are (uc (i) VC (i) The coordinates are then converted to homogeneous coordinates to obtain column vectors. and

[0024] Calculate column vectors normalized vector u (nor) ;

[0025] Establish a linear equation:

[0026] Where × represents cross product;

[0027] Substituting all the world coordinates and pixel coordinates of the reference object into the linear equations, we obtain a system of linear equations;

[0028] Extract (Kc*Em) as the projection matrix Pr, and expand the matrix Pr into a (12×1) column vector pl;

[0029] Expand the linear equations and extract the equations corresponding to the horizontal and vertical axes to obtain Equation 1 and Equation 2.

[0030] Expand equations 1 and 2 to construct the coefficient matrix Ac of column vector pl;

[0031] Perform singular value decomposition on matrix Ac to obtain orthogonal matrix Vc;

[0032] Take the last column elements of matrix Vc and reconstruct it into a new projection matrix Pr. (new) ;

[0033] Calculate the new extrinsic parameter matrix Em (new) Em (new) =(Kc) -1 *Pr (new) ;

[0034] Extracting matrix Em (new) The first three columns are used as the rotation matrix Rc (con) The last column is the translation vector Tc(con).

[0035] Furthermore, the specific steps for deformity correction are as follows:

[0036] Record the timestamp corresponding to each video frame captured by each camera, and synchronize the video frames captured by each camera in time;

[0037] Perform distortion correction on video frames captured by each camera;

[0038] Detect straight line segments in video frames, extract the pixel coordinates of the straight line segments, and calculate the average curvature of all straight line segments in the video frame.

[0039] If the average curvature is less than 0.1%, no action is taken.

[0040] If the average curvature is ≥0.1%, then distortion removal is performed.

[0041] Convert each pixel in the video frame into normalized coordinates with the center pixel coordinates as the origin;

[0042] Let the normalized coordinates that have not yet undergone distortion removal be the ideal coordinates. Calculate the offset of these coordinates under the distortion model. Use Newton's iteration method to gradually adjust the normalized coordinates so that the offset of the normalized coordinates is reduced until the offset is less than a preset threshold.

[0043] Using the first camera as a reference camera, a feature matching algorithm is used to identify and determine whether there are overlapping parts in the video frames captured by the first camera and the remaining cameras.

[0044] Cameras whose video frames overlap with those captured by the reference camera are grouped into Group A, and the remaining cameras are grouped into Group B.

[0045] Furthermore, the specific steps for geometric alignment are as follows:

[0046] Map the coordinates of all video frames captured by cameras in Group A to the pixel coordinate system of the first camera in Group A; extract the intrinsic parameter matrix Kj of the first camera. (1) Extract the first two columns rj of the rotation matrix of the first camera. (1) and rj (2) The translation vector tj of the first camera (1) ;

[0047] Extract the intrinsic parameter matrix Ki of the second camera (2) Extract the first two columns ri of the rotation matrix of the second camera. (1) and ri (2) The translation vector ti of the second camera (2) ;

[0048] Calculate the homography matrix H that maps the pixel coordinates in the second camera video frame to the pixel coordinate system in the first camera video frame. (1←2) :

[0049] H (1←2) =Kj (1) *[rj (1) ,rj (2) ,tj (1) ]*[ri (1) ,ri (2) ,ti (2) ] -1 *(Ki (2) ) -1 ;

[0050] Let the pixel coordinates in the second camera video frame be (ui, vi), and the pixel coordinates after mapping to the first camera pixel coordinate system be (uj, vj). Define formula A1:

[0051]

[0052] According to formula A1, the pixel coordinates in the second camera video frame are mapped to the pixel coordinate system of the first camera;

[0053] Map the pixel coordinates of the video frames captured by the other cameras in Group A to the pixel coordinate system of the first camera to complete the geometric alignment of Group A.

[0054] If, after clustering, there is still a camera in Group B that does not overlap with any other camera in the video frames captured by any other camera, then that camera is an independent camera.

[0055] Define the energy function.

[0056] Furthermore, the steps for defining the energy function are as follows:

[0057] The video frame captured by the first camera is taken as frame image a, and the video frame captured by the second camera is taken as frame image b;

[0058] Let the coordinates of a pixel in the overlapping region be (xu, yv);

[0059] In frame image a, the pixel brightness at (xu, yv) is Ia. (xu，yv) In frame image b, the pixel brightness at (xu, yv) is Ib. (xu，yv) ;

[0060] The brightness gradient G is calculated using finite difference calculations (xu, yv). (xu，yv) :

[0061] Define the energy function E (xu，yv) :

[0062] E (xu,yv) =(α (u) ×|Ia (xu,yv) -Ib (xu,yv) |)+(β (v) ×G (xu,yv) ); where α (u) Represents the brightness weight, β (v) Indicates gradient weights;

[0063] For each pixel in the overlapping region, establish connections to frame image a and frame image b:

[0064] For each pixel vertex in the overlapping region, two edges are drawn, one connecting to the source vertex and the other connecting to the sink vertex; the connection weights of these two edges are the confidence function of the pixel belonging to the camera.

[0065] Create a graph structure of pixels within overlapping regions.

[0066] Furthermore, the steps for creating a graph structure are as follows:

[0067] Determine the source point at the starting position of the overlapping region belonging to frame image a, and determine the sink point at the ending position of the overlapping region belonging to frame image b.

[0068] Create a set of points connecting pixels and a set of edges constructed from multiple pixels;

[0069] The weight of each edge is determined based on the connection weights of all pixels that make up each edge.

[0070] Let the pixels that make up the edge A—B—C be: point A, point B, and point C;

[0071] If the connection weights of point A connected to frame image a and point A connected to frame image b are similar, then the smaller connection weight is selected as the point weight of point A; otherwise, the larger connection weight is selected as the point weight of point A.

[0072] Determine the point weights of points B and C, and calculate the average of the point weights of the three points A, B, and C as the edge weights of edge A—B—C.

[0073] Initialize the flow of all edges to 0, and find all augmenting paths in the overlapping regions based on the BFS algorithm;

[0074] According to the energy function E (xu，yv) Calculate the energy function value of all pixels on each augmentation path; select the augmentation path that causes the largest decrease in energy function value from the source point to the sink point as the stitching line, and perform preliminary stitching of frame image a and frame image b;

[0075] A lightweight depth compensation network is used to perform illumination consistency compensation on the initially stitched video frames, and then pyramid multi-scale fusion is used to optimize the compensated video frames.

[0076] Furthermore, the steps for finding the splicing line are as follows:

[0077] Add the source point to the queue and mark an array to record the access status of the starting point to other pixels;

[0078] Mark the starting point as visited and mark other pixels as unvisited; define a predecessor array to record the predecessor pixels of each pixel in the starting point's access path;

[0079] Take the source point from the queue and use the BFS algorithm to traverse all the neighboring edges of the point; for each neighboring edge, check whether the pixels that make up it have not been visited and whether the residual capacity of the edge is greater than 0.

[0080] If so, mark the pixel as visited and add it to the queue; use the predecessor vertex of the current pixel as the current pixel and continue traversing;

[0081] If not, then do not process it;

[0082] During the traversal, if the current pixel encounters the sink, it means that a path from the source to the sink has been found. By backtracking the predecessor array, the complete path is obtained, thus obtaining the path from the source to the sink.

[0083] After finding the path from the source to the sink, record each edge on the path and its corresponding residual capacity;

[0084] Determine if any edge along the path has a residual capacity less than or equal to 0. If it does, mark the path as a non-enlarging path; otherwise, mark it as an augmenting path.

[0085] Choose a new source point as the starting point and find all augmenting paths within the overlapping region.

[0086] Furthermore, the optimization steps for video frames are as follows:

[0087] For the original images corresponding to frame image a and frame image b, perform continuous Gaussian blur and downsampling operations to obtain the Gaussian pyramid Gik;

[0088] The first layer of the Laplacian pyramid is obtained by subtracting the upsampled image of the second layer from the image of the first layer of the Gaussian pyramid.

[0089] The second layer of the Laplacian pyramid is obtained by subtracting the upsampled image of the third layer from the image of the second layer of the Gaussian pyramid.

[0090] By analogy, construct the Laplace pyramid;

[0091] The fusion formula, which uses suture masks to represent the images of each layer in the Laplacian pyramid, is as follows:

[0092] Lf (k) =M (k) La (k) +(1-M (k) )Lb (k) ;

[0093] Among them, Lf (k) This represents the merged image of the k-th layer of the Laplacian pyramid.

[0094] M (k) Indicates a suture mask;

[0095] La (k) This indicates that the k-th layer of the Laplacian pyramid image is constructed from the original image corresponding to frame image a;

[0096] Lb (k) This indicates that the k-th layer of the Laplacian pyramid image is constructed from the original image corresponding to frame image b;

[0097] The information from each layer of the fused Laplacian pyramid is upsampled and accumulated to obtain a stitched video frame of frame image a and frame image b;

[0098] The video frames captured by the remaining cameras in Group A and Group B are spliced ​​together.

[0099] Multi-channel video splicing devices based on embedded technology development platforms include:

[0100] A processor and a memory, the memory storing computer-readable instructions, the steps of the above method being executed when the computer-readable instructions are executed by the processor.

[0101] Compared with the prior art, the beneficial effects of the present invention are:

[0102] Enhancing the visual continuity of the splicing area: This invention calculates the brightness gradient of each frame of video in real time on an embedded platform and assigns weights to each pixel in the overlapping area using a weighted multiplication method, so that the color transition of the splicing boundary is natural and the gradient is smooth, reducing the glaring visual experience, improving the viewing quality of the video, and making the spliced ​​picture more realistic and immersive.

[0103] An innovative optimization objective: focusing on gradient smoothness. Unlike traditional methods that only focus on color value matching, this invention uses brightness gradient as the core optimization metric. Gradient information contains image edge and texture details, and ensuring its smooth transition is key to eliminating the perception of "seams." This makes the optimization process more consistent with the human visual system's judgment of continuity.

[0104] Precise fusion operator: Adaptive weight allocation uses weighted multiplication rather than simple averaging or other fixed rules to assign weights to each pixel in overlapping regions; this approach allows the algorithm to perform dynamic and adaptive fusion based on local image content (such as gradient intensity); in edge or textured regions, it can intelligently select the best pixel from a particular camera, thereby avoiding blurring and ghosting.

[0105] Effective smoothing of different brightness levels: The pyramid multi-scale fusion method of the present invention constructs a Gaussian pyramid and a Laplacian pyramid, and uses a stitch mask to perform weighted fusion on each layer of the Laplacian pyramid, so as to smoothly transition the brightness at different levels, so that the stitched image can maintain naturalness in both overall brightness and local details.

[0106] Frequency domain decomposition: Separating brightness information, a sequence of images from high resolution (detail layer) to low resolution (contour layer) is constructed through successive Gaussian blurring and downsampling; the bottom layer retains high-frequency details (such as texture and edges), while the top layer captures the low-frequency global brightness distribution; obtained by calculating the difference between adjacent Gaussian pyramid levels, it is essentially the output of a series of bandpass filters, each layer capturing image information and details at a specific scale.

[0107] Layered fusion: Precise weight control, the core innovation lies in applying the stitch mask obtained from the stitching line optimization to each layer of the Laplacian pyramid; this means that fusion is not performed at a single pixel level, but simultaneously at multiple scales; at the top layer (low resolution), the mask guides the fusion process to smoothly handle large-scale overall brightness differences, ensuring the macroscopic lighting continuity of the stitched image and avoiding obvious blocky color differences; at the bottom layer (high resolution), the mask is used to fuse high-frequency details and textures. Since the brightness tone has been unified at the top layer, the fusion here can retain the clear details from each source image to the maximum extent, avoiding texture blurring caused by over-smoothing.

[0108] Multi-scale advantages: It solves the fundamental contradiction of traditional methods. Traditional single-scale fusion methods are difficult to simultaneously achieve large-scale brightness smoothing and detail preservation, often falling into the dilemma of "chaos when expanded and dead when contracted". The multi-scale method of this invention decouples the two tasks of "adjusting the overall brightness" and "preserving local details", with different pyramid levels responsible for different tasks at different scales, thereby systematically solving the conflict between the global and local.

[0109] Compatible with embedded hardware platforms: This invention optimizes the computational structure by linearizing and vectorizing the brightness balancing and color mapping processes, respectively. By utilizing the parallel computing characteristics of embedded GPUs or NPUs, the complex pixel-level mapping function is decomposed into multi-dimensional matrix operations, thereby reducing algorithm latency. Attached Figure Description

[0110] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0111] Figure 1 This is a schematic diagram of the method of the present invention;

[0112] Figure 2 This is a schematic diagram of video frame overlap according to the present invention;

[0113] Figure 3 This is a schematic diagram of the device of the present invention;

[0114] Figure 4 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0115] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0116] Example 1

[0117] Please see Figure 1 and Figure 3 Multi-channel video stitching methods based on embedded technology development platforms include:

[0118] Step S1: Connect various cameras in the target area to the embedded platform and initialize the cameras; calculate the extrinsic parameter matrix of each camera and perform distortion correction on the video frames captured by each camera; (after completing distortion correction) divide all cameras into Group A, which overlaps with the video frames captured by the reference camera, and assign the remaining cameras to Group B; perform geometric alignment on the video frames in Group A and Group B, and extract the independent cameras.

[0119] The specific steps of step S1 are as follows:

[0120] Camera access: Connect various cameras (such as MIPI CSI cameras, USB cameras, etc.) to the embedded platform. The embedded platform detects the number and type of connected cameras, loads the corresponding drivers for each type of camera, and completes the initialization of various camera devices.

[0121] Configure the interface: Create a corresponding device node for each camera in the embedded platform; for MIPICSI cameras, define their virtual channel ID and number of lanes; for USB cameras, configure their relevant parameters (such as resolution, pixel format, frame rate parameters, etc.).

[0122] Perform camera calibration for each camera and obtain the distortion coefficients corresponding to each camera;

[0123] Geometric model of camera imaging:

[0124] Let the object's real-world coordinates be (X... (r) Y (r) Z (r) The object's coordinates within the camera are (x, y). (c) y (c) , z (c) );

[0125] The mathematical relationship that maps objects from the real world to the view within the camera is:

[0126]

[0127] Among them, R (ca)T represents the rotation matrix, which describes the rotation of the object's coordinate system within the camera view relative to its real-world coordinates; (ca) This represents the translation vector, which is the translation of the "coordinates of the object within the camera" relative to the "real-world coordinates".

[0128] Project the coordinates within the camera image onto the image, and let the two-dimensional coordinates of the object on the image be (x, y). (p) y (p) The mathematical relationship for projecting the "coordinates of the object within the camera" onto the image is:

[0129] Among them, f (ca) Indicates the focal length of the camera;

[0130] The pixel coordinates of the object in the image are (u (x) v (y) The mathematical relationship between projecting the "two-dimensional coordinates of an object on an image" onto the pixel coordinates of the image is as follows:

[0131] Among them, K (ca) K represents the intrinsic parameter matrix of the camera. (ca) The mathematical expression is:

[0132]

[0133] Where, γ (ca) The axis skew parameter represents the degree of tilt between the horizontal and vertical axes in an image; in most modern cameras, the axis skew parameter is equal to zero.

[0134] (ce (x) ce (y) () represents the coordinates of the center pixel of the image;

[0135] fc (x) fc represents the horizontal focal length of the camera. (x) The formula for calculation is:

[0136]

[0137] fc (y) fc represents the vertical focal length of the camera. (y) The formula for calculation is:

[0138]

[0139] Among them, sr (x) and sr (y) pi represents the horizontal and vertical light-sensing distances of the camera, respectively, which is the camera's light-sensing size (unit: millimeters); (x) and pi(y) , representing the number of pixels in the horizontal and vertical directions of the image, respectively;

[0140] Overall, the mathematical expression for projecting an object from real-world coordinates to image pixel coordinates is:

[0141]

[0142] Select a reference object within the shooting area of ​​each camera (the reference object must be a definite and fixed object; a landmark building in a certain area can be selected as the reference object for the first camera), obtain the real-world coordinates of the reference object and the pixel coordinates of the reference object in the video, and calculate the extrinsic parameter matrix for each camera;

[0143] The real-world coordinates of the reference object are designated as (Xs). (1) Ys (1) Zs (1) )~(Xs (re) Ys (re) Zs (re) ), pixel coordinates (uc (1) VC (1) )~(uc (re) VC (re) ); where re represents the number of real-world coordinates and pixel coordinates of the reference object (the value of re is not less than 6);

[0144] Obtain the intrinsic parameter matrix Kc of the camera (the matrix Kc is in the form of a (3×3) matrix), and let the rotation matrix of the camera be Rc:

[0145] (3×3) matrix;

[0146] Let the translation vector of the camera be Tc:

[0147] (3×1) column vector;

[0148] Combine matrix Rc and column vector Tc to form extrinsic matrix Em:

[0149] (3×4) matrix;

[0150] Let the i-th real-world coordinate of the reference object be (Xs) (i) Ys (i) Zs (i) ), pixel coordinates are (uc (i) VC (i) The value range of i is 1 to re;

[0151] (Xs) (i) Ys(i) Zs (i) ) and (uc (i) VC (i) Transforming the coordinates into homogeneous coordinates yields the column vector. and

[0152] Calculate column vectors normalized vector u (nor) : Where -1 represents the inverse of the matrix, and * represents matrix multiplication;

[0153] Establish a linear equation:

[0154] Where × represents the cross product (i.e., vector product);

[0155] Substituting all the world coordinates and pixel coordinates of the reference object into the linear equations, we obtain a system of linear equations;

[0156] It should be noted that in the above linear equation, matrix Kc, the real-world coordinates and pixel coordinates of the reference object are known quantities, while matrix Em has 12 unknown quantities;

[0157] Simultaneously expanding the linear equations corresponding to "each world coordinate point and pixel coordinate point" allows us to sequentially obtain the horizontal axis (i.e., uc) corresponding to "that world coordinate point and pixel coordinate point". (i) (corresponding axis) and vertical axis (i.e., vc) (i) The equations in the two directions corresponding to the axes (after expanding the above linear equations, we can actually obtain the equations in the three directions; but the vectors) The axis corresponding to "1" in the third row, i.e. the homogeneous factor corresponding to the third axis, can be ignored due to the redundancy of the homogeneous coordinate scale; therefore, only two axes are taken in actual calculations. Since there are 12 unknowns in the matrix Em, at least 6 points (forming 12 equations) are needed to solve the above linear equation system.

[0158] Solve the system of linear equations:

[0159] Extract (Kc*Em) as the projection matrix Pr (i.e., Pr = Kc*Em, matrix Pr is a (3×4) matrix), and expand matrix Pr into a (12×1) column vector pl;

[0160] Expand the linear equation and extract the horizontal axis (i.e., uc). (i) (corresponding axis) and vertical axis (i.e., vc) (i) By finding the equations corresponding to the axes, we obtain equations 1 and 2.

[0161] Equation 1:

[0162]

[0163] Equation 2:

[0164]

[0165] Among them, Pr (1,k) ~Pr (3,k) , representing the elements in matrix Pr from the 1st row and kth column to the 3rd row and kth column respectively;

[0166] Ss (i,k) , representing the k-th component of the homogeneous coordinates corresponding to the i-th real-world coordinate of the reference object;

[0167] When k is 1, Ss (i,k) Xs (i) When k is 2, Ss (i,k) Ys (i) ;

[0168] When k is 3, Ss (i,k) Zs (i) When k is 4, Ss (i,k) Indicates 1;

[0169] Expand equations 1 and 2 to construct the coefficient matrix Ac of column vector pl; matrix Ac is a ([2×re]×12) matrix;

[0170] Singular value decomposition of matrix Ac yields an orthogonal matrix Uc ([2×re]×[2×re]), an orthogonal matrix Vc (12×12), and a diagonal matrix Σc ([2×re]×12), i.e.:

[0171] Ac=Uc*Σc*(Vc) T ;

[0172] The column vectors in matrix Uc are "matrix eigenvectors of ".

[0173] The column vectors in matrix Vc are "matrix eigenvectors of ".

[0174] When [2×re]≥12, matrix Σc has non-zero elements only on the diagonal of the 12×12 matrix region, and all other positions are zero; the non-zero elements of matrix Σc are: the matrix in descending order. "The square root of the corresponding eigenvalue;

[0175] example:

[0176] (2×re)≥12, the first 12 rows contain non-zero elements;

[0177]

[0178] Σ (1) To Σ (12) Sort in descending order;

[0179] When [2×re] < 12, matrix Σc has non-zero elements only on the diagonal of the region corresponding to ([2×re]×[2×re]), and all other positions are zero; the non-zero elements of matrix Σc are: the matrix arranged in descending order. "The square root of the corresponding eigenvalue;

[0180] example:

[0181] (2×re) < 12, the first (2×re) columns contain non-zero elements;

[0182]

[0183] Σ (1) To Σ (2×re) Sort in descending order;

[0184] Take the last column elements of matrix Vc and reconstruct it into a new projection matrix Pr. (new) ;

[0185] Calculate the new extrinsic parameter matrix Em (new) Em (new) =(Kc) -1 *Pr (new) ;

[0186] Extracting matrix Em (new) The first three columns are used as the rotation matrix Rc (con) The last column is the translation vector Tc(con);

[0187] Determine Rc (con) Does it satisfy the orthogonality constraint (i.e., (Rc)? (con) ) T *Rc (con) =1 ( All-one matrix ) );

[0188] If the condition is met, no action is taken;

[0189] If not satisfied, then for Rc (con) SVD correction is performed, followed by Tc processing using nonlinear optimization (such as the Levenberg-Marquardt algorithm). (con) and the corrected Rc (con) ;

[0190] Record the timestamp corresponding to each video frame captured by each camera, and synchronize the video frames captured by each camera in time;

[0191] Perform distortion correction on video frames captured by each camera;

[0192] Use Hough transform or LSD (Line Segment Detector) algorithm to detect line segments in video frames and extract the pixel coordinates of the line segments;

[0193] Using the least squares method, with the horizontal pixel coordinates as the independent variable and the vertical pixel coordinates as the dependent variable, the pixel coordinates of the line segment are fitted into a quadratic function.

[0194] Let the pixel coordinates of line segment Li be: (u (1) v (1) )~(u (fr) v (fr) ); fr represents the number of pixels in the line segment Li;

[0195] Let the pixel coordinates of the j-th pixel in line segment Li be (u (j) v (j) The quadratic function corresponding to this line is:

[0196] Where a, b, and c represent the coefficients of the quadratic function in one variable; the range of j is 1 to fr.

[0197] (u (j) v (j) The corresponding curvature is k. (cur) :

[0198]

[0199] According to k (cur) The calculation formula, calculate (u) (1) v (1) )~(u (fr) v (fr) The curvature kc at the corresponding point (1) ~kc (fr) The calculated average value is used as the curvature of the straight line segment Li.

[0200] Repeat the same steps as calculating the curvature of line segment Li to calculate the average curvature of all line segments in the video frame.

[0201] If the average curvature (of all straight segments) is less than 0.1%, then no action is taken.

[0202] If the average curvature (of all straight line segments) is ≥0.1%, it indicates that the image in the video frame is distorted. Distortion removal should be performed on the image in the video frame.

[0203] Convert each pixel in the video frame (i.e., the image corresponding to the video frame with distortion) into normalized coordinates with the center pixel coordinates as the origin;

[0204] Let the current normalized coordinates without distortion removal be the ideal coordinates. Calculate the offset of these coordinates under the distortion model. Use Newton's iteration method to gradually adjust the normalized coordinates, reducing the offset until the offset is less than a preset threshold (e.g., 10). -3 (pixels);

[0205] Assume (u (di) v (di) ) represents the pixel coordinates in a video frame (i.e., distortion occurs in the video frame);

[0206] (u (ex) v (ex) The ) represents the pixel coordinates after distortion removal (i.e., the ideal coordinates);

[0207] The distortion model is as follows:

[0208]

[0209] After distortion removal is completed, select one camera from all cameras as the reference camera;

[0210] The first camera is used as the reference camera (users or relevant technicians may also choose other cameras as the reference camera);

[0211] Use feature matching algorithms (such as SURF or SIFT) to identify and determine whether there are overlapping parts in the video frames captured by the first camera and the remaining cameras;

[0212] Cameras whose video frames overlap with those captured by the reference camera are grouped into Group A, and the remaining cameras are grouped into Group B.

[0213] Map the coordinates of all video frames captured by cameras in Group A to the pixel coordinate system of the first camera in Group A; (Here, "the first camera in Group A" refers to the reference camera mentioned above, i.e., the first camera).

[0214] Extract the intrinsic parameter matrix Kj of the first camera (1) Extract the first two columns rj of the rotation matrix of the first camera. (1) and rj (2) The translation vector tj of the first camera (1) ;

[0215] Extract the intrinsic parameter matrix Ki of the second camera (2) Extract the first two columns ri of the rotation matrix of the second camera. (1) and ri (2)The translation vector ti of the second camera (2) ;

[0216] Among them, Kj (1) with Ki (2) They are all (3×3) matrices;

[0217] rj (1) and rj (2) and tj (1) and ri (1) and ri (2) and ti (2) They are all (3×1) column vectors;

[0218] Calculate the homography matrix H that maps the pixel coordinates in the second camera video frame to the pixel coordinate system in the first camera video frame. (1←2) :

[0219] H (1←2) =Kj (1) *[rj (1) ,rj (2) ,tj (1) ]*[ri (1) ,ri (2) ,ti (2) ] -1 *(Ki (2) ) -1 ;

[0220] Let the pixel coordinates in the second camera video frame be (ui, vi), and the pixel coordinates after mapping to the first camera pixel coordinate system be (uj, vj). Define formula A1:

[0221]

[0222] According to formula A1, the pixel coordinates in the second camera video frame are mapped to the pixel coordinate system of the first camera;

[0223] Repeat the same steps of "mapping the pixel coordinates in the video frame of the second camera to the pixel coordinate system of the first camera" to map the pixel coordinates of the video frames captured by the other cameras in Group A to the pixel coordinate system of the first camera, thus completing the geometric alignment of Group A.

[0224] Feature matching algorithms (such as SURF or SIFT) and clustering algorithms are used to cluster the video frames captured by each camera in Group B; a reference camera is selected in each cluster, and the pixel coordinates of the video frames captured by other cameras in each cluster are mapped to the pixel coordinate system of the corresponding reference camera in each cluster to complete the geometric alignment of Group B.

[0225] If, after clustering, there is still a camera in Group B that does not overlap with any video frame captured by any other camera, then this camera is an independent camera.

[0226] Summarize the data from Group A and Group B, and complete the geometric alignment.

[0227] Step S2: Define the energy function, find the stitching lines of different video frames based on the BFS algorithm, and stitch the video frames in Group A and Group B in turn to complete the initial stitching; use the Lightweight Depth Compensation Network (LightCompNet) to perform illumination consistency compensation on the initially stitched video frames, and then use "Pyramid Multi-Scale Fusion" to optimize the compensated video frames to obtain the stitched video;

[0228] The specific steps of step S2 are as follows:

[0229] In the overlapping area between the video frames captured by the first camera and the second camera in Group A, select the splicing line;

[0230] The video frame captured by the first camera is taken as frame image a, and the video frame captured by the second camera is taken as frame image b;

[0231] The overlapping area between frame image a and frame image b is taken as region D. (area) ;

[0232] Let the pixel coordinates in the overlapping region be (xu, yv);

[0233] In frame image a, the pixel brightness at (xu, yv) is Ia. (xu，yv) In frame image b, the pixel brightness at (xu, yv) is Ib. (xu，yv) ;

[0234] The brightness gradient G of (xu, yv) is calculated using finite difference (approximation). (xu，yv) :

[0235]

[0236] E (xu,yv) =(α (u) ×|Ia (xu,yv) -Ib (xu,yv) |)+(β (v) ×G (xu,yv) ); where α (u) Represents the brightness weight, β (v) Represents gradient weights; α (u) and β (v) The simplified determination process:

[0237] The overlapping area of ​​frame image a and frame image b is divided into blocks; the mean brightness and texture complexity of each image block in its respective frame image (i.e., frame image a and frame image b) are calculated (e.g., the texture complexity of each image block is calculated by calculating the variance of the gradient magnitude).

[0238] If the average brightness of an image patch differs significantly between corresponding regions in two frames (i.e., the average brightness of an image patch in frame a (or frame b) is ≥ 1.5 times the average brightness of the same image patch in frame b (or frame a), then brightness has a significant impact on the stitching quality of the corresponding region of that image patch.

[0239] Let α (u) The initial value is α (ini) The average brightness of the corresponding image patch in frame image a and frame image b is lu. (a) and lu (b) ;

[0240] Calculate the relative difference d between the mean brightness values. (lum) :

[0241] Calculate α (u) Value:

[0242]

[0243] Where, k (lum) T represents the adjustment coefficient (for adjusting brightness weights). (lum) This represents the reference threshold (for adjusting brightness weights); (k (lum) and T (lum) (Set by the user or relevant technical personnel);

[0244] If the texture complexity of an image patch differs significantly in corresponding regions between two frames, repeat α. (u) The process of obtaining β is based on... (v) The initial values ​​determine the gradient weights;

[0245] When the pixels on the splicing line are selected from different image blocks, the energy of each pixel on the splicing line is calculated using the brightness weight and gradient weight corresponding to each image block.

[0246] For each pixel in the overlapping region, establish connections to frame image a and frame image b:

[0247] For each pixel vertex in the overlapping region, two edges are drawn, one connecting to the source vertex and the other connecting to the sink vertex; the connection weights of these two edges are: the confidence function that the pixel belongs to the camera; (meaning of connection);

[0248] Create a graph structure for pixels within overlapping regions:

[0249] Determine the source point at the starting position of the overlapping region belonging to frame image a, and determine the sink point at the ending position of the overlapping region belonging to frame image b.

[0250] Create a set of points connecting pixels and a set of edges constructed from multiple pixels (at least two pixels); (all edges constructed from pixels within overlapping regions must pass through a source and a sink; there can be multiple source and sink points);

[0251] The weight of each edge is determined based on the connection weights of all pixels that make up each edge.

[0252] Let the pixels that make up the edge A—B—C be: point A, point B, and point C;

[0253] If the connection weights of point A connected to frame image a and point A connected to frame image b are close (i.e., the absolute value of the difference between “connection weight of point A connected to frame image a” and “connection weight of point A connected to frame image b” is less than 0.1), then the smaller connection weight is selected as the point weight of point A from the connection weights of point A connected to frame image a and point A connected to frame image b.

[0254] Conversely, if there is a difference in the connection weights between point A and frame a and frame b (i.e., the absolute value of the difference between the connection weights of point A and frame a and the connection weights of point A and frame b is greater than 0.1), then the larger connection weight is selected as the point weight of point A from the connection weights of point A and frame a and frame b.

[0255] Repeat the steps to determine the weight of point A, determine the weights of points B and C, and calculate the average of the corresponding weights of points A, B, and C as the edge weight of edge A—B—C.

[0256] Initialize the flow of all edges to 0, indicating that in the initial state there is no flow through any edge;

[0257] Finding splicing lines using the BFS algorithm:

[0258] (Starting from the source point) Starting from the source point, add the source point to the queue and mark an array to record the access status of the starting point to other pixels;

[0259] Mark the starting point as visited (this can be represented by a specific marker value), and mark other pixels as unvisited; define a predecessor array (e.g., prev) to record the predecessor pixels of each pixel in the starting point access path;

[0260] Take the source point from the queue and use the BFS algorithm to traverse all the neighboring edges of the point; for each neighboring edge, check whether the pixels that make up it have not been visited and whether the residual capacity of the edge is greater than 0.

[0261] If the above two conditions are met, mark the pixel as visited and add it to the queue; use the predecessor vertex of the pixel as the current pixel and continue traversing.

[0262] If not, then do not process it;

[0263] During the traversal, if the current pixel encounters the sink, it means that a path from the source to the sink has been found. By backtracking the predecessor array (e.g., prev), the complete path is obtained, thus obtaining the path from the source to the sink.

[0264] After finding the path from the source to the sink, record each edge on the path and its corresponding residual capacity; ("Residual capacity" is an object defined in the BFS algorithm, used to find augmenting paths);

[0265] Determine if any edge along the path has a residual capacity less than or equal to 0. If it does, mark the path as a non-enlarging path; otherwise, mark it as an augmenting path.

[0266] Choose a new source point as the starting point and find all augmenting paths within the overlapping area;

[0267] According to the energy function E (xu，yv) Calculate the energy function value of all pixels on each augmentation path; select the augmentation path that causes the largest decrease in energy function value from the source point to the sink point as the stitching line, and perform preliminary stitching of frame image a and frame image b;

[0268] Lightweight depth compensation network (LightCompNet) is used to perform illumination consistency compensation on the initially stitched video frames, and then "pyramid multi-scale fusion" is used to optimize the compensated video frames:

[0269] Constructing the Gaussian pyramid: Perform continuous Gaussian blur and downsampling operations on the original images corresponding to frame images a and b (i.e., frame images a and b without any processing, which are the video frames directly captured by the first and second cameras in Group A) to obtain the Gaussian pyramid Gik.

[0270] Each layer of the Gaussian pyramid Gik is the result of Gaussian filtering and reducing the resolution of the image in the next layer; the lower pyramids contain image details, while the higher pyramids preserve the image's outline and general structure.

[0271] Constructing the Pyramid of Laplace:

[0272] The first layer of the Laplace pyramid: The image of the first layer of the Gaussian pyramid minus the upsampled image of the second layer (i.e., the layer above it);

[0273] The second layer of the Laplace pyramid: The image of the second layer of the Gaussian pyramid minus the image of the third layer after upsampling;

[0274] By analogy, construct the Laplace pyramid;

[0275] The fusion formula, which uses suture masks to represent the images of each layer in the Laplacian pyramid, is as follows:

[0276] Lf (k) =M (k) La (k) +(1-M (k) )Lb (k) ;

[0277] Among them, Lf (k) This represents the merged image of the k-th layer of the Laplacian pyramid.

[0278] M (k) This represents the suture mask (the suture mask is a two-dimensional matrix corresponding to each layer of the Laplace pyramid);

[0279] La (k) This indicates that the k-th layer of the Laplacian pyramid image is constructed from the original image corresponding to frame image a;

[0280] Lb (k) This indicates that the k-th layer of the Laplacian pyramid image is constructed from the original image corresponding to frame image b;

[0281] The information from each layer of the fused Laplacian pyramid is upsampled and accumulated to obtain the (final) stitched video frame of frame image a and frame image b;

[0282] Repeat the processing steps for stitching the video frames from the first and second cameras in Group A, and stitch together the video frames captured by the remaining cameras in Group A and Group B.

[0283] Step S3: Use the embedded platform to output the stitched video and provide feedback (to the user) on the camera number of the individual camera and the video frames it captured;

[0284] Key points for implementation:

[0285] GPU parallel stitching rendering:

[0286] Use OpenGL ES or CUDA to implement image projection and perspective transformation, and map the stitched video images onto a unified output plane;

[0287] Multi-threaded pipeline architecture:

[0288] Thread 1: Acquisition, input, and buffering of video frames captured by each camera;

[0289] Thread 2: Video frame processing and seam calculation;

[0290] Thread 3: GPU splicing rendering and output encoding.

[0291] Please see Figure 4 The overall effect of the present invention is as follows:

[0292] Phase 1: Preprocessing and Intelligent Grouping (laying the foundation for stitching, i.e., step S1):

[0293] Precise geometric calibration: First, the extrinsic parameter matrix of each camera is calculated and distortion correction is performed to provide a precise mathematical basis for subsequent geometric transformations.

[0294] Dynamic topology grouping: Innovatively, all cameras are divided into Group A (overlapping areas) and Group B (non-overlapping areas) and individual cameras are identified; this intelligent grouping strategy allows the system to use different strategies to process video streams with different spatial relationships and optimize the allocation of computing resources.

[0295] Phase Two: Core Stitching and Optimization (Achieving Visual Seamlessness, i.e., Step S2):

[0296] Graph theory-based stitching line search: By defining an energy function (considering pixel differences and gradient information) and using the BFS (breadth-first search) algorithm to find the optimal stitching line in the overlapping area, the method effectively avoids crossing key objects and achieves initial seamless stitching.

[0297] Deep learning and multi-scale fusion: After the initial stitching, a lightweight depth compensation network is introduced to specifically address the lighting differences between different cameras; subsequently, pyramid multi-scale fusion technology is used to smoothly transition across multiple resolution levels, ensuring that the stitching results are natural and consistent in both overall and detailed aspects.

[0298] Phase 3: Output and Feedback (forming a closed system loop, i.e., step S3):

[0299] Finally, a unified, high-quality panoramic stitched video stream is output through an embedded platform. A unique feedback mechanism is designed to synchronously output the video streams and their numbers from individual cameras; this provides the system with redundant information and detailed close-up capabilities, enhancing the overall system reliability and application flexibility.

[0300] Example 2

[0301] Please see Figure 2 A multi-channel video splicing device based on an embedded technology development platform includes: a memory, a processor, and a display device;

[0302] The memory stores computer-readable instructions. When these instructions are executed by the processor, the steps in the multi-channel video splicing method based on the embedded technology development platform described above are performed to complete the splicing of multiple video channels and display them on the display device. Through the above technical solution, the processor and memory are interconnected and communicate with each other via a communication bus and / or other forms of connection mechanisms (not shown). The memory stores a processor-executable computer program. When the electronic device is running, the processor executes this computer program to perform the corresponding method steps in Embodiment 1 above, thereby achieving the following functions:

[0303] Connect various cameras in the target area to the embedded platform and initialize the cameras; calculate the extrinsic parameter matrix of each camera and perform distortion correction on the video frames captured by each camera; (after completing distortion correction) divide all cameras into Group A, which overlaps with the video frames captured by the reference camera, and assign the remaining cameras to Group B; perform geometric alignment on the video frames in Group A and Group B, and extract the independent cameras;

[0304] Define an energy function, use the BFS algorithm to find the stitching lines of different video frames, and stitch the video frames in Group A and Group B in turn to complete the initial stitching; use a lightweight depth compensation network (LightCompNet) to compensate for the illumination consistency of the initially stitched video frames, and then use "pyramid multi-scale fusion" to optimize the compensated video frames to obtain the stitched video;

[0305] The embedded platform outputs the stitched video and provides feedback (to the user) on the camera number of each individual camera and the video frames it captured.

[0306] The above formulas are all dimensionless calculations. The formulas are derived from software simulations using a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation. For example, there are weighting coefficients and proportional coefficients. The values ​​set are to quantify each parameter to obtain a specific value, which is convenient for subsequent comparison. The values ​​of the weighting coefficients and proportional coefficients are only required to not affect the proportional relationship between the parameters and the quantified values.

[0307] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-channel video stitching method based on an embedded technology development platform, characterized in that, The method includes: Connect various cameras in the target area to the embedded platform and initialize the cameras; calculate the extrinsic parameter matrix of each camera and perform distortion correction on the video frames captured by each camera; divide all cameras into Group A, which overlaps with the video frames captured by the reference camera, and assign the remaining cameras to Group B; perform geometric alignment on the video frames in Group A and Group B, and extract the independent cameras. Define an energy function, use the BFS algorithm to find the stitching lines of different video frames, and stitch the video frames in Group A and Group B in turn to complete the initial stitching; use a lightweight depth compensation network to perform illumination consistency compensation on the initially stitched video frames, and then use pyramid multi-scale fusion to optimize the compensated video frames to obtain the stitched video. The embedded platform outputs the stitched video and provides feedback on the camera IDs of the individual cameras and the video frames they captured.

2. The multi-channel video stitching method based on an embedded technology development platform according to claim 1, characterized in that, The steps for calculating the extrinsic parameter matrix are as follows: Perform camera calibration for each camera and obtain the distortion coefficients corresponding to each camera; Obtain the real-world coordinates of the reference object and its pixel coordinates in the video, and calculate the extrinsic parameter matrix for each camera; Obtain the intrinsic parameter matrix Kc of the camera, and let the rotation matrix be Rc and the translation vector be Tc; Combine matrix Rc and column vector Tc into extrinsic matrix Em; Let the i-th real-world coordinate of the reference object be (Xs) (i) Ys (i) Zs (i) ), pixel coordinates are (uc (i) VC (i) The coordinates are then converted to homogeneous coordinates to obtain column vectors. and Calculate column vectors normalized vector u (nor) ; Establish a linear equation: Where × represents cross product; Substituting all the world coordinates and pixel coordinates of the reference object into the linear equations, we obtain a system of linear equations; Extract (Kc*Em) as the projection matrix Pr, and expand the matrix Pr into a (12×1) column vector pl; Expand the linear equations and extract the equations corresponding to the horizontal and vertical axes to obtain Equation 1 and Equation 2. Expand equations 1 and 2 to construct the coefficient matrix Ac of column vector pl; Perform singular value decomposition on matrix Ac to obtain orthogonal matrix Vc; Take the last column elements of matrix Vc and reconstruct it into a new projection matrix Pr. (new) ; Calculate the new extrinsic parameter matrix Em (new) Em (new) =(Kc) -1 *Pr (new) ; Extracting matrix Em (new) The first three columns are used as the rotation matrix Rc (con) The last column is the translation vector Tc(con).

3. The multi-channel video stitching method based on an embedded technology development platform according to claim 1, characterized in that, The specific steps for deformity correction are as follows: Record the timestamp corresponding to each video frame captured by each camera, and synchronize the video frames captured by each camera in time; Perform distortion correction on video frames captured by each camera; Detect straight line segments in video frames, extract the pixel coordinates of the straight line segments, and calculate the average curvature of all straight line segments in the video frame. If the average curvature is less than 0.1%, no action is taken. If the average curvature is ≥0.1%, then distortion removal is performed. Convert each pixel in the video frame into normalized coordinates with the center pixel coordinates as the origin; Let the normalized coordinates that have not yet undergone distortion removal be the ideal coordinates. Calculate the offset of these coordinates under the distortion model. Use Newton's iteration method to gradually adjust the normalized coordinates so that the offset of the normalized coordinates is reduced until the offset is less than a preset threshold. Using the first camera as a reference camera, identify and determine whether there are overlapping parts in the video frames captured by the first camera and the remaining cameras; Cameras whose video frames overlap with those captured by the reference camera are grouped into Group A, and the remaining cameras are grouped into Group B.

4. The multi-channel video stitching method based on an embedded technology development platform according to claim 1, characterized in that, The specific steps for geometric alignment are as follows: Map the coordinates of all video frames captured by cameras in Group A to the pixel coordinate system of the first camera in Group A; extract the intrinsic parameter matrix Kj of the first camera. (1) Extract the first two columns rj of the rotation matrix of the first camera. (1) and rj (2) The translation vector tj of the first camera (1) ; Extract the intrinsic parameter matrix Ki of the second camera (2) Extract the first two columns ri of the rotation matrix of the second camera. (1) and ri (2) The translation vector ti of the second camera (2) ; Calculate the homography matrix H that maps the pixel coordinates in the second camera video frame to the pixel coordinate system in the first camera video frame. (1←2) : H (1←2) =Kj (1) *[rj (1) ,rj (2) ,tj (1) ]*[ri (1) ,ri (2) ,the (2) ] -1 *(Sweet (2) ) -1 ; Let the pixel coordinates in the second camera video frame be (ui, vi), and the pixel coordinates after mapping to the first camera pixel coordinate system be (uj, vj). Define formula A1: According to formula A1, the pixel coordinates in the second camera video frame are mapped to the pixel coordinate system of the first camera; Map the pixel coordinates of the video frames captured by the other cameras in Group A to the pixel coordinate system of the first camera to complete the geometric alignment of Group A. If, after clustering, there is still a camera in Group B that does not overlap with any other camera in the video frames captured by any other camera, then that camera is an independent camera. Define the energy function.

5. The multi-channel video stitching method based on an embedded technology development platform according to claim 4, characterized in that, The steps for defining the energy function are as follows: The video frame captured by the first camera is taken as frame image a, and the video frame captured by the second camera is taken as frame image b; Let the coordinates of a pixel in the overlapping region be (xu, yv); In frame image a, the pixel brightness at (xu, yv) is Ia. (xu，yv) In frame image b, the pixel brightness at (xu, yv) is Ib. (xu，yv) ; The brightness gradient G is calculated using finite difference calculations (xu, yv). (xu，yv) : Define the energy function E (xu，yv) : E (xu,yv) =(α (u) ×|Ia (xu,yv) -Ib (xu,yv) |)+(β (v) ×G (xu,yv) ); where α (u) Represents the brightness weight, β (v) Indicates gradient weights; For each pixel in the overlapping region, establish connections to frame image a and frame image b: For each pixel vertex in the overlapping region, two edges are drawn, one connecting to the source vertex and the other connecting to the sink vertex; the connection weights of these two edges are the confidence function of the pixel belonging to the camera. Create a graph structure of pixels within overlapping regions.

6. The multi-channel video stitching method based on an embedded technology development platform according to claim 5, characterized in that, The steps to create a graph structure are as follows: Determine the source point at the starting position of the overlapping region belonging to frame image a, and determine the sink point at the ending position of the overlapping region belonging to frame image b. Create a set of points connecting pixels and a set of edges constructed from multiple pixels; The weight of each edge is determined based on the connection weights of all pixels that make up each edge. Let the pixels that make up the edge A—B—C be: point A, point B, and point C; If the connection weights of point A connected to frame image a and point A connected to frame image b are close, then the smaller connection weight is selected as the point weight of point A from the connection weights of point A connected to frame image a and point A connected to frame image b. Conversely, choose the larger connection weight as the point weight of point A; Determine the point weights of points B and C, and calculate the average of the point weights of the three points A, B, and C as the edge weights of edge A—B—C. Initialize the flow of all edges to 0, and find all augmenting paths within the overlapping regions; According to the energy function E (xu，yv) Calculate the energy function value of all pixels on each augmentation path; select the augmentation path that causes the largest decrease in energy function value from the source point to the sink point as the stitching line, and perform preliminary stitching of frame image a and frame image b; Illumination consistency compensation is performed on the initially stitched video frames, and then pyramid multi-scale fusion is used to optimize the compensated video frames.

7. The multi-channel video stitching method based on an embedded technology development platform according to claim 6, characterized in that, The steps to find the splicing line are as follows: Add the source point to the queue and mark an array to record the access status of the starting point to other pixels; Mark the starting point as visited and mark other pixels as unvisited; define a predecessor array to record the predecessor pixels of each pixel in the starting point's access path; Take the source point from the queue and use the BFS algorithm to traverse all the neighboring edges of the point; for each neighboring edge, check whether the pixels that make up it have not been visited and whether the residual capacity of the edge is greater than 0. If so, mark the pixel as visited and add it to the queue; use the predecessor vertex of the current pixel as the current pixel and continue traversing; If not, then do not process it; During the traversal, if the current pixel encounters the sink, it means that a path from the source to the sink has been found. By backtracking the predecessor array, the complete path is obtained, thus obtaining the path from the source to the sink. After finding the path from the source to the sink, record each edge on the path and its corresponding residual capacity; Determine if any edge along the path has a residual capacity less than or equal to 0. If it does, mark the path as a non-enlarging path; otherwise, mark it as an augmenting path. Choose a new source point as the starting point and find all augmenting paths within the overlapping region.

8. The multi-channel video stitching method based on an embedded technology development platform according to claim 6, characterized in that, The optimization steps for video frames are as follows: For the original images corresponding to frame image a and frame image b, perform continuous Gaussian blur and downsampling operations to obtain the Gaussian pyramid Gik; The first layer of the Laplacian pyramid is obtained by subtracting the upsampled image of the second layer from the image of the first layer of the Gaussian pyramid. The second layer of the Laplacian pyramid is obtained by subtracting the upsampled image of the third layer from the image of the second layer of the Gaussian pyramid. By analogy, construct the Laplace pyramid; The fusion formula, which uses suture masks to represent the images of each layer in the Laplacian pyramid, is as follows: Lf (k) =M (k) La (k) +(1-M (k) )Lb (k) ; Among them, Lf (k) This represents the merged image of the k-th layer of the Laplacian pyramid. M (k) Indicates a suture mask; La (k) This indicates that the k-th layer of the Laplacian pyramid image is constructed from the original image corresponding to frame image a; Lb (k) This indicates that the k-th layer of the Laplacian pyramid image is constructed from the original image corresponding to frame image b; The information from each layer of the fused Laplacian pyramid is upsampled and accumulated to obtain a stitched video frame of frame image a and frame image b; The video frames captured by the remaining cameras in Group A and Group B are spliced ​​together.

9. A multi-channel video splicing device based on an embedded technology development platform, characterized in that, The splicing device includes a processor and a memory, the memory storing computer-readable instructions, which, when executed by the processor, perform the steps of the method corresponding to any one of claims 1-8.

Images