Multi-channel real-time video stitching method and system based on mesh deformation and GPU acceleration

Abstract

The application discloses a kind of multi-path real-time video splicing method and system based on grid deformation and GPU acceleration, belong to civil airport security monitoring and computer vision video image processing technical field.For civil airport large parallax, multiple dynamic target, high-resolution real-time processing and the ghosting, ghost, low efficiency and picture inconsistency problems existing in the prior art under complex illumination scene splicing technology, the application adopts APAP local grid deformation to generate static remapping table to realize fast alignment;Exposure compensation gain coefficient is solved by constructing global illumination error function, to ensure illumination consistency;Using dense optical flow to detect motion area and alternately injecting opposite pure color pixels to guide joint to go around, eliminate dynamic ghost;Combined with GPU acceleration and laplace pyramid multi-band fusion, panoramic video is generated efficiently.The application can stably output clear, ghost-free, natural illumination panoramic monitoring picture, meet the demand of airport 180 degree real-time panoramic monitoring.

Description

Technical Field

[0001] This invention relates to the fields of security monitoring and computer vision video image processing technology for civil airports, and in particular to a method and system for multi-channel real-time video stitching based on mesh deformation and GPU acceleration. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Civil airports are the core area for airport security control. This area can extend over 200 meters in depth and cover more than 150 meters in width, requiring 180° horizontal panoramic monitoring without blind spots. Simultaneously, it must also capture detailed identification of ground staff and special vehicles near the aircraft gates, and dynamically track passenger aircraft on distant taxiways. This scenario involves three types of dynamic targets: large taxiing aircraft, medium-speed moving special vehicles, and low-speed walking personnel, all within a complex, unobstructed lighting environment. This places stringent industry compliance requirements on the accuracy, real-time performance, and robustness of video stitching.

[0004] The existing multi-channel video stitching technology faces the following main problems in its application to this airport surveillance scenario: (1) Image ghosting and misalignment in scenes with large parallax. Traditional methods generally adopt global homography matrix and spherical projection rigid transformation scheme. These methods can only be adapted to planar scenes. However, in airport scenes, the depth difference between the near-view aircraft position and the far-view taxiway exceeds 200 meters. In practical applications, the problem of "the near view being aligned and the far view being ghosted, and the far view being aligned and the near view being misaligned" is very likely to occur. This cannot meet the image clarity requirements for security evidence collection, resulting in the failure of key event tracing.

[0005] (2) Ghosting in splicing under multiple dynamic targets. In airport scenes, there are three types of dynamic targets with great differences in speed and size: aircraft, vehicles and personnel. Existing fixed stitching lines and static gradient seam solutions cannot simultaneously avoid multiple moving targets. It is very easy for problems such as broken aircraft wings, ghosting of vehicles and missing personnel to occur at the splicing seams, resulting in monitoring blind spots and missed events.

[0006] (3) There are bottlenecks in real-time processing of high-resolution video. Airport security requires real-time processing of three 4-megapixel 25fps video streams. The existing APAP mesh deformation stitching algorithm has high computational complexity, and the pure CPU solution can only achieve a frame rate of 8fps, which cannot meet the real-time requirement of 25fps in civil aviation standards. Conventional CUDA acceleration solutions are only adapted to global transformations and cannot efficiently support parallel computing of local mesh deformations, so they cannot be implemented in airport edge data center servers.

[0007] (4) Poor image consistency under complex lighting conditions. Airport open scenes have extreme lighting environments such as strong direct sunlight during the day, backlighting when aircraft taxi, and interference from multiple light sources at night. The existing gamma correction scheme based on the average value of overlapping areas has insufficient correction accuracy, which easily leads to abrupt changes in brightness at the splicing boundary and overexposure distortion in the night lighting area, resulting in the failure of the continuity of the monitoring image. Summary of the Invention

[0008] To address the problems of image ghosting in high parallax scenes, stitching ghosting caused by multiple dynamic targets, low real-time processing efficiency of high-resolution video, and poor image consistency under complex lighting conditions in existing technologies, this invention provides a multi-channel real-time video stitching method and system based on mesh deformation and GPU acceleration. By employing a projection-as-it-is method for local mesh deformation and generating a static remapping table, the alignment difficulties caused by depth differences are resolved. By detecting moving regions with dense optical flow and injecting opposite solid-color pixels to guide seam detours, multi-target ghosting is eliminated. By constructing a global illumination error function to optimize exposure compensation gain, illumination consistency correction is achieved. Combined with GPU acceleration and multi-band pyramid fusion, the real-time stitching requirements of high-resolution video are met.

[0009] On the one hand, a multi-channel real-time video stitching method based on mesh deformation and GPU acceleration is provided, including: Acquire multiple video data streams and obtain a synchronized video frame sequence by aligning them with timestamps; Based on the first frame image, feature points are extracted and matched in the overlapping area of ​​adjacent images. The first frame image is divided into a two-dimensional grid. The local homography matrix of each grid vertex is calculated using the projection method. A static pixel remapping table is generated by interpolation. Remapping is then performed on subsequent video frame images to obtain deformed images. The effective pixels of the overlapping area are extracted from the deformed image, a global illumination error function is constructed, the exposure compensation gain coefficient that minimizes the global illumination error function is solved and applied to the whole image to obtain the illumination-corrected image. The dense optical flow field of the overlapping area between the illumination-corrected image and the adjacent image is calculated to generate a motion mask. When the area ratio of moving pixels exceeds a preset threshold, opposite solid color pixels are alternately injected into the mask area. By finding the seam path with the minimum color difference, a fusion mask is generated. Based on the illumination-corrected image and the fusion mask, a Laplacian pyramid and a Gaussian pyramid are constructed. The Laplacian band data is then weighted and mixed layer by layer using the Gaussian mask weights. The resulting panoramic video frames are then reconstructed using the Laplacian pyramid.

[0010] Furthermore, the step of extracting and matching feature points in the overlapping area of ​​adjacent images based on the first frame image specifically involves: extracting feature points using the SIFT algorithm, constructing a Gaussian difference pyramid and generating feature descriptors; performing feature matching using the fast nearest neighbor matching algorithm, filtering high-quality matching pairs by the ratio of the nearest neighbor to the second nearest neighbor distance, and eliminating mismatched pairs using the RANSAC algorithm.

[0011] Furthermore, the step of calculating the local homography matrix of each grid vertex using the projection-as-it-is method and generating a static pixel remapping table through interpolation specifically involves: dividing the first frame image into a uniform two-dimensional grid of a preset size; assigning a Gaussian decay weight to each grid vertex based on its spatial Euclidean distance to the matching feature point set; solving the weighted direct linear transformation equation to obtain the local homography matrix of each grid cell; and calculating the coordinate mapping relationship of each pixel in each input image in the stitched panoramic image through bilinear interpolation based on the local homography matrices of all grid vertices, thereby generating static pixel remapping tables in the horizontal and vertical directions.

[0012] Furthermore, the global illumination error function is specifically as follows: ; in, and Let be the exposure compensation gain coefficients of the adjacent images i and j, respectively. The set of valid pixels in the overlapping region. These are the spatial coordinates of specific pixels within the overlapping region; and These represent the original pixel intensities read from the boundary pixel point p of images i and j, respectively.

[0013] Furthermore, the dense optical flow field of the overlapping region between the illumination-corrected image and adjacent images is calculated to generate a motion mask. Specifically, the dense optical flow algorithm is used to calculate the dense optical flow field of the overlapping region to obtain the horizontal and vertical displacement of each pixel; the motion vector amplitude of each pixel is calculated, and by comparing it with a preset motion threshold, pixels with motion amplitude greater than the threshold are marked as dynamic pixels, and the rest are marked as static pixels, thereby generating a binary motion mask.

[0014] Furthermore, when the proportion of moving pixel area exceeds a preset threshold, the opposite solid color pixels are alternately injected into the mask area. Specifically, according to the parity of the current frame number, the opposite solid color pixels are injected into the dynamic area marked by the moving mask. The first solid color is injected in even frames and the second solid color is injected in odd frames. The first solid color and the second solid color are opposite colors to each other.

[0015] Furthermore, the step of generating a fusion mask by finding the seam path with the minimum color difference specifically involves: using a dynamic programming algorithm to find a vertical seam path from top to bottom in the overlapping region with the goal of minimizing the sum of color differences of all pixels on the path; using the found optimal seam path as the boundary line to generate non-overlapping binary fusion masks for adjacent images.

[0016] Furthermore, based on the illumination-corrected image and fusion mask, a Laplacian pyramid and a Gaussian pyramid are constructed. Gaussian mask weights are used to perform layer-by-layer weighted mixing of the Laplacian band data. The resulting panoramic video frame is then reconstructed using the Laplacian pyramid. Specifically, this involves: constructing multiple layers of Gaussian pyramids in parallel based on the corresponding fusion masks for each channel, serving as spatial transition fusion weights; constructing Laplacian pyramids of the same number of layers based on the spatially aligned and illumination-corrected deformed images of each channel; at each pyramid level, using a pairwise fusion method, layer-by-layer weighted mixing of the Laplacian band data of adjacent channels using Gaussian mask weights to generate a panoramic fusion Laplacian pyramid; and performing reverse reconstruction layer by layer from the top of the panoramic fusion Laplacian pyramid downwards, using layer-by-layer upsampling and frequency band residual superposition to restore the mixed panoramic fusion Laplacian pyramid into a high-definition panoramic video frame image.

[0017] On the other hand, a multi-channel real-time video stitching system based on mesh deformation and GPU acceleration is also provided, including: The video acquisition and synchronization module is configured to: acquire multiple video data streams and obtain a synchronized video frame sequence by aligning the timestamps; The mesh deformation module is configured to: extract and match feature points in the overlapping area of ​​adjacent images based on the first frame image, divide the first frame image into a two-dimensional mesh, calculate the local homography matrix of each mesh vertex using the projection method, generate a static pixel remapping table through interpolation, and perform remapping on subsequent video frame images to obtain deformed images; The exposure compensation module is configured to: extract effective pixels of the overlapping area from the deformed image, construct a global illumination error function, solve for the exposure compensation gain coefficient that minimizes the global illumination error function and apply it to the entire image to obtain the illumination-corrected image; The seam calculation module is configured to: calculate the dense optical flow field in the overlapping area of ​​the illumination-corrected image and the adjacent image to generate a motion mask; when the proportion of the moving pixel area exceeds a preset threshold, alternately inject opposite solid color pixels into the mask area, and generate a fusion mask by finding the seam path with the smallest color difference. The fusion module is configured to: construct a Laplacian pyramid and a Gaussian pyramid based on the illumination-corrected image and the fusion mask; use the Gaussian mask weights to perform layer-by-layer weighted mixing of the Laplacian band data; and obtain stitched panoramic video frames through Laplacian pyramid reconstruction.

[0018] Furthermore, a computer-readable storage medium is provided on which a computer program is stored, which, when executed by a processor, performs the method of the first aspect.

[0019] The above technical solution has the following advantages or beneficial effects: (1) By deforming the local projection grid of APAP, the near and far views were accurately aligned at the same time under the condition of a 200-meter depth difference at the airport, which effectively avoided the ghosting and tearing of near and far views and met the clarity requirements of security evidence collection.

[0020] (2) Through GPU-accelerated dense optical flow and dynamic color penalty seam calculation, ghosting of three types of dynamic targets in airport scenes, namely aircraft, vehicles and personnel, can be effectively eliminated, ensuring the integrity of event tracing.

[0021] (3) Based on the heterogeneous collaborative acceleration of Xeon Gold 5318H CPU and Tesla T4 GPU, the processing speed is improved by more than 200% compared with the pure CPU solution when processing three 4-megapixel 25 frames / second video streams.

[0022] (4) By using a global exposure compensation algorithm with smoothing constraints, the brightness difference in airport strong light, backlight and nighttime multi-source environments is effectively controlled, and stable operation is achieved 24 hours a day. Attached Figure Description

[0023] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0024] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings. Those skilled in the art should understand that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0026] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0027] Example 1 In this embodiment, for the 180° panoramic monitoring scenario of the flight area of ​​a civil airport, in order to solve the problems of large parallax ghosting and misalignment, fragmentation of ghost images of multiple dynamic targets, bottleneck of real-time processing of high-resolution video, failure of consistency of complex lighting images, and lack of industry-compliant streaming media distribution, a multi-channel real-time video stitching method based on mesh deformation and GPU acceleration is provided to realize the end-to-end processing from the original video stream of the airport front-end camera to the compliant panoramic video stream.

[0028] The hardware foundation of this embodiment is a dedicated server with an Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz, an NVIDIA Tesla T4 GPU, and 32G of memory, deployed in the airport security server room. The front-end acquisition equipment consists of three DS-2CD7T48DWD-XS / JM 4-megapixel bullet cameras, which are arranged horizontally and equidistantly to form a 180° panoramic coverage, with an overlap of ≥32% between adjacent camera images.

[0029] The present invention will be further described in detail below with reference to specific implementation steps.

[0030] S101: Acquire multiple video data streams and obtain a synchronized video frame sequence by aligning the timestamps; S102: Based on the first frame image, feature points are extracted and matched in the overlapping area of ​​adjacent images. The first frame image is divided into a two-dimensional grid. The local homography matrix of each grid vertex is calculated using the projection method. A static pixel remapping table is generated by interpolation. The remapping of subsequent video frame images is performed in the GPU to obtain the deformed image. S103: Extract the effective pixels of the overlapping area from the deformed image, construct the global illumination error function, solve for the exposure compensation gain coefficient that minimizes the global illumination error function and apply it to the whole image to obtain the illumination-corrected image; S104: Calculate the dense optical flow field in the overlapping area of ​​the illumination-corrected image and adjacent images to generate a motion mask; when the proportion of the moving pixel area exceeds a preset threshold, alternately inject opposite pure color pixels into the mask area, and generate a fusion mask by finding the seam path with the smallest color difference.

[0031] S105: Based on the illumination-corrected image and fusion mask, construct the Laplacian pyramid and Gaussian pyramid, and use the Gaussian mask weights to perform layer-by-layer weighted mixing of the Laplacian band data. The stitched panoramic video frames are obtained by reconstructing the Laplacian pyramid.

[0032] Step S101 aims to solve the problems of concurrent acquisition, hardware decoding and frame synchronization of multiple surveillance video streams at the airport, avoid misalignment of moving targets caused by asynchronous timing of multiple video frames, and provide time-synchronized and format-uniform video frame data for subsequent splicing.

[0033] Specifically, three airport surveillance cameras transmit 4-megapixel (1920×1080) raw video streams at 25fps in H.264 format via the RTSP protocol. Leveraging the 18 cores and 36 threads of the Xeon Gold 5318H CPU on the server, an independent acquisition thread is allocated to each camera to concurrently pull the raw video streams. Simultaneously, a lock-free concurrent read / write queue based on a lightweight semaphore mechanism is created for each stream, with a queue depth of 15 frames. A single producer-single consumer model is employed to eliminate lock contention bottlenecks in multi-threaded concurrent scenarios.

[0034] Next, the CPU software decoder based on FFmpeg is invoked to perform parallel hardware decoding on each bitstream, and the decoded video frames are converted from the YUV420P color space to BGR format.

[0035] Subsequently, a frame timestamp alignment mechanism is used to synchronize the three video frames, ensuring that the time difference between the acquisition of video frames from adjacent cameras does not exceed 5ms. When the queue is full, expired frames at the head of the queue are automatically discarded; when the queue is empty, the consumer thread is blocked and waits, thus guaranteeing strict frame synchronization of the three video streams.

[0036] After the above processing, three time-synchronized, uniformly formatted RGB video frame sequences of 1920×1080 resolution are obtained, denoted as left frame I1, middle frame I2, and right frame I3, respectively, collectively covering the airport's 180° panoramic field of view. These synchronized video frames directly serve as the sole input source for feature point extraction and mesh deformation calculation in step S102. Their frame synchronization accuracy determines the alignment accuracy of subsequent mesh deformation, effectively preventing misalignment of moving targets.

[0037] In step S102, feature point extraction and local projection mesh deformation are used to specifically solve the problem of near and far image ghosting caused by the global homography matrix in airport scenes with a large parallax difference of 200 meters depth. The goal is to accurately align the near-view boarding bridge area and the far-view taxiway area at the same time, and to significantly reduce real-time computing overhead by using a static mapping table for solidification.

[0038] Specifically, based on the first frame images (I1, I2, I3) in the three-channel synchronous video frame sequence output in step S101, the following steps are taken: First, utilizing the multi-core parallel capability of the Xeon CPU, feature points are extracted in parallel using the SIFT algorithm in the overlapping areas of adjacent images (I1 and I2, I2 and I3), a 5-layer Gaussian difference pyramid is constructed, and extreme points are extracted through non-maximum suppression, completing the main direction allocation of feature points and the generation of 128-dimensional descriptors. Next, the FLANN fast nearest neighbor matching algorithm is used for feature matching, and high-quality matching pairs are selected with a nearest neighbor / second nearest neighbor distance ratio of 0.65. Then, the RANSAC algorithm is used to remove mismatched pairs, finally obtaining a set of high-quality matching points with no less than 300 pairs of adjacent images, evenly distributed in near and far views.

[0039] Then, parallel computation of APAP local projection mesh deformation is performed. Each input image is divided into a uniform 64×64 pixel two-dimensional mesh, and an independent computation thread is assigned to each mesh vertex to solve the local homography matrix specific to each mesh vertex in parallel. For each mesh vertex x*, x* represents each feature point x* in the matching point set. i Assign Gaussian decay weights, with the following weighting function: ; Here, σ is set to 120 pixels to adapt to the weight attenuation requirements of large airport scenes; then, based on the weighted matching point set, the weighted direct linear transformation equation is solved to obtain the optimal local homography matrix H* that minimizes the local projection error of the mesh cell. The solution formula is as follows: ; in, These are the weights calculated in the previous step; Representing feature points The coordinates of the corresponding matching point in the target projection plane; This indicates that the feature points on the input image will be... Predicted coordinates after transformation to the target plane using the homography matrix H; express With predicted coordinates The Euclidean distance between them, i.e., geometric misalignment deviation; This represents the entire weighted energy function. It squares the reprojection errors of all feature points, multiplies them by their respective distance weights, and then sums them up; argmin represents finding an optimal matrix H such that the total weighted error accumulated after the summation reaches its minimum value.

[0040] After traversing all mesh vertices, the deformation calculation of the entire non-rigid mesh is completed, which effectively solves the problem of poor global transformation adaptability in large parallax scenes.

[0041] After traversing all mesh vertices, the deformation calculation of the entire non-rigid mesh is completed, which effectively solves the problem of poor global transformation adaptability in large parallax scenes.

[0042] Based on the local homography matrix of all grid vertices, bilinear interpolation is performed in parallel using the CUDA cores of the Tesla T4 to calculate the coordinate mapping relationship of each pixel in the stitched panoramic image, generating a horizontal mapping table xmap and a vertical mapping table ymap. This mapping table is then stored in the Tesla T4 video memory and server memory, allowing subsequent video frame processing to directly call the mapping table to complete pixel remapping, thus skipping the time-consuming process of feature extraction and matrix solving.

[0043] The final output includes a static pixel remapping table corresponding to the three images, as well as the three images I1', I2', and I3' after spatial alignment and mesh deformation. These outputs are the sole input source for the exposure compensation calculation in step S103. The remapping table generated in this step based on the mesh deformation calculation completed synchronously in the first frame in step S101 also becomes the core basis for spatial alignment of all subsequent frames; the accuracy of mesh deformation directly determines the alignment effect of the final stitched image.

[0044] It is understandable that for images that are not the first frame, the feature extraction and matrix solving steps are skipped, and the generated and stored static pixel remapping table is directly called to perform pixel remapping on the current frame, outputting a deformed image.

[0045] Step S103 aims to solve the problems of large brightness and color differences between adjacent images and abrupt changes in splicing boundaries in open airport scenes with strong light, backlight, and multiple light sources at night, so as to achieve consistency in lighting and color of multiple images and thus ensure the continuity of the monitoring images.

[0046] Specifically, based on the spatially aligned three-way deformed images I1', I2', and I3' output in step S102, the effective pixel set Ω of the overlapping region is first extracted for each pair of adjacent images (I1' and I2', I2' and I3'). ij Specifically, the geometric overlap region of adjacent images is calculated based on the static pixel remapping table, and abnormal pixels with overexposed highlights and flickering highlights are directly removed within this region by setting a healthy brightness range, thereby extracting the effective pixel set Ω of the static background. ij .

[0047] Subsequently, a global illumination error function with smoothing constraints is constructed to avoid abrupt gain changes caused by local light sources at night. The error function is as follows: ; in, and Let be the exposure compensation gain coefficients of the adjacent images i and j, respectively. The set of valid pixels in the overlapping region. These are the spatial coordinates of specific pixels within the overlapping region; and These represent the original pixel intensity (brightness value) read from the boundary pixel point p of images i and j, respectively.

[0048] By solving for the exposure compensation gain coefficient that minimizes the error function, and then smoothly applying the obtained exposure compensation gain coefficient to the RGB three channels of the entire image, the energy error function achieves global residual minimization convergence during the solution process. Experimental observation and statistical verification show that the mean brightness difference in the overlapping areas of adjacent images after correction can be stably controlled within four gray levels, meeting the human eye's requirement for seamless transition concealment. Simultaneously, the system re-solves the gain coefficient every preset time interval (600 seconds) to adapt to slow changes in illumination, and immediately triggers a re-solution when a sudden change in image brightness exceeds a preset threshold (20%), thus ensuring the spatiotemporal consistency of image illumination and resulting in a natural transition in illumination and color between adjacent images after stitching.

[0049] After the above processing, three images I1'', I2'', and I3'' after illumination and color consistency correction are obtained. These corrected images become the sole input source for seam calculation in step S104. This step completes exposure compensation based on the spatially aligned image from step S102, and its correction effect directly determines the naturalness of the boundary transition in the final stitched image.

[0050] Step S104 is based on optical flow splicing seam calculation, which specifically solves the splicing ghosting, breakage and loss problems caused by multiple dynamic targets such as aircraft, vehicles and people in airport scenes. Dynamic ghosting is effectively eliminated by avoiding all moving targets through the seam path.

[0051] Specifically, the three images I1'', I2'', and I3'' obtained in step S103 after illumination correction, as well as the corresponding images of adjacent frames, are used as the basis. First, for the overlapping region of each pair of adjacent images, the Farneback dense optical flow algorithm accelerated by Tesla T4 CUDA is called to calculate the dense optical flow field V(p)=(u(p),v(p)) in the overlapping region between two adjacent frames, where u(p) is the horizontal (X-axis) displacement component of pixel p between two adjacent frames, and v(p) is the vertical (Y-axis) displacement component of pixel p between two adjacent frames. Before calculation, the overlapping region is downsampled and scaled by a preset ratio (0.25 times), which reduces the size of the pixels to be processed to 1 / 16 of the original size. Based on this, relying on the inherent hardware acceleration advantage of Tesla T4 graphics card, the dense matrix parallel computing capability of its 2560 CUDA cores and hardware stream (cuda Stream) is used to parallelize and distribute the motion vectors of tens of thousands of pixels in the downsampled overlapping region to each CUDA thread for concurrent solution, so that the overall calculation and generation time of the optical flow field of a single frame is stably controlled within 3 milliseconds.

[0052] Next, the proportion of moving pixels in the overlapping area of ​​adjacent cameras is calculated, a motion sensitivity threshold τ is set, and a binary motion mask is generated based on the motion vector amplitude through binarization. : ; The percentage of moving pixels in the overlapping region is calculated. If the percentage is lower than a preset threshold, the seam result of the previous frame is directly reused; if the percentage is higher than the threshold, subsequent dynamic seam calculations are performed.

[0053] The dynamic region of the motion mask marking is characterized and defined by a specific grayscale feature value (128) assigned after the optical flow field binarization decision. This system adopts a stepped multi-valued mask design: the initial effective area of ​​the entire image is uniformly marked with the conventional maximum grayscale value of 255, while the corresponding matrix element values ​​of pixels determined by the dense optical flow algorithm to have dynamic motion are forcibly overwritten and rewritten to the intermediate grayscale value of 128. The set of coordinates of all pixels with a grayscale value equal to 128 is the dynamic region of the motion mask marking.

[0054] Based on the parity of the current frame number, the dynamic region marked by the motion mask is injected with opposite solid color pixels: even frames are injected with pure black pixels, and odd frames are injected with pure white pixels, generating an image after injection penalty, thereby forcing the seam path to bypass the motion region.

[0055] A dynamic programming algorithm is used to minimize the sum of color differences among all pixels along the path. To achieve the objective, we find the optimal longitudinal seam path S in the overlapping region, i.e.: ; Where S represents the optimal longitudinal seam path to be solved; p represents the coordinates of any pixel on the seam path S; and These represent the RGB three-channel color vector values ​​at the same pixel coordinate p of adjacent images after spatial alignment and illumination correction.

[0056] Based on the found optimal seam path S, generate non-overlapping binary fusion masks Mask1 and Mask2 for adjacent images.

[0057] Finally, the optimal seam path and fusion mask for each pair of adjacent images are output. These masks become the core weight basis for multi-band fusion in step S105. This step relies on the illumination-corrected image from step S103 to complete the seam calculation. The rationality of the seam path directly determines the splicing integrity of dynamic targets, thereby effectively avoiding ghosting and breaks.

[0058] Step S105 aims to solve the problems of unnatural transitions and loss of details in the stitching of large-size panoramic images of airports, thereby achieving seamless stitching of 180° panoramic images, while using GPU parallel acceleration to ensure real-time processing efficiency.

[0059] Specifically, it is based on the corrected images I1'', I2'', and I3'' output in step S103, and the corresponding fusion mask output in step S104.

[0060] First, for the fusion mask of each image, a 6-layer Gaussian pyramid Gmask is constructed in parallel using the CUDA core of Tesla T4 to obtain spatial transition fusion weights at different spatial scales. At the same time, based on the deformed images of each image after spatial alignment and illumination correction, a corresponding 6-layer Laplacian pyramid Lim9 is constructed in parallel to decompose the image into low-frequency contour components and high-frequency detail components, thereby achieving separation processing of information at different frequencies.

[0061] Then, a pairwise fusion method was adopted to first fuse I1'' and I2'', and then the fusion result was fused with I3'' a second time.

[0062] Phase 1: Fusing images I1'' and I2'' to generate an intermediate panoramic feature pyramid. The fusion formula is: ; Second stage: Fusing intermediate results with image I3'' to generate the final panoramic fused Laplacian pyramid. The fusion formula is: ; Where k represents the pyramid level number (in this embodiment, the value ranges from 0 to 5, and a total of 6 pyramid levels are constructed); (x, y) represents the coordinates of the two-dimensional pixel matrix at the current pyramid level k. , and These represent the pixel matrix data of the k-th layer of the Laplacian pyramid corresponding to the three corrected images I1'', I2'', and I3'', respectively; Represents the initial fusion mask The corresponding weight values ​​of the kth layer of a 6-layer Gaussian pyramid; Represents a secondary fusion mask The corresponding weight values ​​of the kth layer of a 6-layer Gaussian pyramid; This represents the k-th layer of the intermediate panoramic feature pyramid generated after the first stage of fusion; 7) This represents the k-th layer of the panoramic fused Laplacian pyramid data generated by the two-stage iterative fusion process.

[0063] The high-frequency layer uses a narrow transition band to preserve edge details, while the low-frequency layer uses a wide transition band to achieve a smooth transition of illumination.

[0064] Finally, through a top-down Laplacian pyramid reconstruction operation, the fused Laplacian pyramid is upsampled and overlaid layer by layer to reconstruct a 180° panoramic fused image with a resolution of 4530×1080. The entire process is completed in the Tesla T4 video memory, with a single frame fusion time of no more than 2 milliseconds. The panoramic image is then transferred from the GPU video memory to the CPU memory via cudaMemcpy.

[0065] The seamlessly stitched panoramic video frames output in this step become the sole input source for image cropping and streaming media encoding. The fusion effect directly determines the visual quality and evidence validity of the final panoramic surveillance image.

[0066] After completing the panoramic fusion, this embodiment also includes image cropping and panoramic streaming media encoding steps, which aim to solve the problem of low-latency distribution of panoramic video when multiple terminals access it concurrently at the airport, realize compliant video encoding and streaming media services, and meet the compliance requirements of airport security.

[0067] Specifically, based on the seamlessly stitched panoramic video frames output in step S105, the merged panoramic video frames are first cropped, mapped back to regular rectangles and black borders are removed, and then the massive panoramic BGR image is efficiently converted to YUV420P format through color space conversion.

[0068] Next, the converted image is sent to an FFmpeg-based software encoder to compress and package the H.264 video stream according to the set target bitrate, and then a network callback is triggered to complete the push stream distribution.

[0069] Based on the multi-core network scheduling capabilities of Xeon CPUs, a streaming media server conforming to the GB / T28181 national standard protocol is built, with built-in national standard authentication and device registration mechanisms, enabling it to directly connect to the airport's existing video security platform, while also being compatible with RTSP protocol streaming.

[0070] The final output is an H.264 format panoramic video stream that conforms to civil aviation industry standards, which can be retrieved in real time by various airport monitoring terminals via GB / T28181 or RTSP protocols. This step, as the final output of the entire technical solution, achieves a complete closed-loop processing from the front-end raw video stream to the back-end compliant panoramic video stream.

[0071] Example 2 This embodiment provides a multi-channel real-time video stitching system based on mesh deformation and GPU acceleration, including: The video acquisition and synchronization module is configured to: acquire multiple video data streams and obtain a synchronized video frame sequence by aligning the timestamps; The mesh deformation module is configured to: extract and match feature points in the overlapping area of ​​adjacent images based on the first frame image, divide the first frame image into a two-dimensional mesh, calculate the local homography matrix of each mesh vertex using the projection method, generate a static pixel remapping table through interpolation, and perform remapping on subsequent video frame images to obtain deformed images; The exposure compensation module is configured to: extract effective pixels from the overlapping area of ​​the deformed image, construct a global illumination error function, solve for the exposure compensation gain coefficient that minimizes the global illumination error function and apply it to the entire image to obtain the illumination-corrected image; The seam calculation module is configured to: calculate the dense optical flow field in the overlapping area of ​​the illumination-corrected image and the adjacent image to generate a motion mask; when the proportion of the moving pixel area exceeds a preset threshold, alternately inject opposite solid color pixels into the mask area, and generate a fusion mask by finding the seam path with the smallest color difference. The fusion module is configured to: construct a Laplacian pyramid and a Gaussian pyramid based on the illumination-corrected image and the fusion mask; use the Gaussian mask weights to perform layer-by-layer weighted mixing of the Laplacian band data; and obtain stitched panoramic video frames through Laplacian pyramid reconstruction.

[0072] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation process is the same, so it will not be repeated here.

[0073] Example 3 This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method of Embodiment 1.

[0074] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-channel real-time video stitching method based on mesh deformation and GPU acceleration, characterized in that, include: Acquire multiple video data streams and obtain a synchronized video frame sequence by aligning them with timestamps; Based on the first frame image, feature points are extracted and matched in the overlapping area of ​​adjacent images. The first frame image is divided into a two-dimensional grid. The local homography matrix of each grid vertex is calculated using the projection method. A static pixel remapping table is generated by interpolation. Remapping is then performed on subsequent video frame images to obtain deformed images. The effective pixels of the overlapping area are extracted from the deformed image, a global illumination error function is constructed, the exposure compensation gain coefficient that minimizes the global illumination error function is solved and applied to the whole image to obtain the illumination-corrected image. The dense optical flow field of the overlapping area between the illumination-corrected image and the adjacent image is calculated to generate a motion mask. When the area ratio of moving pixels exceeds a preset threshold, opposite solid color pixels are alternately injected into the mask area. By finding the seam path with the minimum color difference, a fusion mask is generated. Based on the illumination-corrected image and the fusion mask, a Laplacian pyramid and a Gaussian pyramid are constructed. The Laplacian band data is then weighted and mixed layer by layer using the Gaussian mask weights. The resulting panoramic video frames are then reconstructed using the Laplacian pyramid.

2. The method according to claim 1, characterized in that, The step of extracting and matching feature points in the overlapping area of ​​adjacent images based on the first frame image is as follows: the SIFT algorithm is used to extract feature points, a Gaussian difference pyramid is constructed and feature descriptors are generated; the fast nearest neighbor matching algorithm is used for feature matching, high-quality matching pairs are selected by the ratio of the nearest neighbor to the second nearest neighbor distance, and the RANSAC algorithm is used to remove mismatched pairs.

3. The method according to claim 1, characterized in that, The method of calculating the local homography matrix of each grid vertex using the projection-as-it-is method and generating a static pixel remapping table through interpolation is as follows: the first frame image is divided into uniform two-dimensional grids of a preset size; for each grid vertex, a Gaussian decay weight is assigned based on its spatial Euclidean distance to the matching feature point set; the weighted direct linear transformation equation is solved to obtain the local homography matrix of each grid cell; based on the local homography matrices of all grid vertices, the coordinate mapping relationship of each pixel in each input image in the stitched panoramic image is calculated through bilinear interpolation to generate static pixel remapping tables in the horizontal and vertical directions.

4. The method according to claim 1, characterized in that, The global illumination error function is specifically as follows: ； in, and Let be the exposure compensation gain coefficients of the adjacent images i and j, respectively. The set of valid pixels in the overlapping region. These are the spatial coordinates of specific pixels within the overlapping region; and These represent the original pixel intensities read from the boundary pixel point p of images i and j, respectively.

5. The method according to claim 1, characterized in that, The dense optical flow field of the overlapping region between the illumination-corrected image and adjacent images is calculated to generate a motion mask. Specifically, the dense optical flow algorithm is used to calculate the dense optical flow field of the overlapping region to obtain the horizontal and vertical displacement of each pixel. The motion vector amplitude of each pixel is calculated and compared with a preset motion threshold. Pixels with motion amplitude greater than the threshold are marked as dynamic pixels, and the rest are marked as static pixels, thereby generating a binary motion mask.

6. The method according to claim 1, characterized in that, When the proportion of moving pixel area exceeds a preset threshold, the opposite solid color pixels are alternately injected into the mask area. Specifically, according to the parity of the current frame number, the opposite solid color pixels are injected into the dynamic area marked by the moving mask. The first solid color is injected in even frames and the second solid color is injected in odd frames. The first solid color and the second solid color are opposite colors to each other.

7. The method according to claim 1, characterized in that, The process of generating a fusion mask by finding the seam path with the minimum color difference involves: using a dynamic programming algorithm to find a vertical seam path from top to bottom in the overlapping area with the goal of minimizing the sum of color differences of all pixels on the path; and using the found optimal seam path as the boundary line to generate non-overlapping binary fusion masks for adjacent images.

8. The method according to claim 1, characterized in that, Based on the illumination-corrected image and fusion mask, a Laplacian pyramid and a Gaussian pyramid are constructed. The Laplacian band data is then weighted and mixed layer by layer using Gaussian mask weights. The resulting panoramic video frames are then reconstructed using the Laplacian pyramid. Specifically, multiple Gaussian pyramids are constructed in parallel based on the corresponding fusion masks for each channel, serving as spatial transition fusion weights. Based on the deformed images of each channel after spatial alignment and illumination correction, Laplacian pyramids with the same number of layers are constructed respectively. At each pyramid level, a pairwise fusion method is adopted, and Gaussian mask weights are used to perform layer-by-layer weighted aliasing of the Laplacian frequency band data of adjacent channels to generate a panoramic fused Laplacian pyramid. Reverse reconstruction is performed layer by layer from the top of the panoramic fused Laplacian pyramid downwards. Through layer-by-layer upsampling and frequency band residual superposition, the mixed panoramic fused Laplacian pyramid is restored to a high-definition panoramic video frame image.

9. A multi-channel real-time video stitching system based on mesh deformation and GPU acceleration, characterized in that, include: The video acquisition and synchronization module is configured to: acquire multiple video data streams and obtain a synchronized video frame sequence by aligning the timestamps; The mesh deformation module is configured to: extract and match feature points in the overlapping area of ​​adjacent images based on the first frame image, divide the first frame image into a two-dimensional mesh, calculate the local homography matrix of each mesh vertex using the projection method, generate a static pixel remapping table through interpolation, and perform remapping on subsequent video frame images to obtain deformed images; The exposure compensation module is configured to: extract effective pixels of the overlapping area from the deformed image, construct a global illumination error function, solve for the exposure compensation gain coefficient that minimizes the global illumination error function and apply it to the entire image to obtain the illumination-corrected image; The seam calculation module is configured to: calculate the dense optical flow field in the overlapping area of ​​the illumination-corrected image and the adjacent image to generate a motion mask; when the proportion of the moving pixel area exceeds a preset threshold, alternately inject opposite solid color pixels into the mask area, and generate a fusion mask by finding the seam path with the smallest color difference. The fusion module is configured to: construct a Laplacian pyramid and a Gaussian pyramid based on the illumination-corrected image and the fusion mask; use the Gaussian mask weights to perform layer-by-layer weighted mixing of the Laplacian band data; and obtain stitched panoramic video frames through Laplacian pyramid reconstruction.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the multi-channel real-time video stitching method based on mesh deformation and GPU acceleration as described in any one of claims 1-7.

Images