Multi-channel video low-delay processing and transmission method and system for real scene fusion

Abstract

The application discloses a multi-channel video low-delay processing and transmission method and system for real scene fusion, and belongs to the technical field of video splicing and transmission. The method comprises the following steps: multi-channel video synchronous acquisition and creation of a hardware acceleration processing channel for each channel; reading distortion correction parameters and clipping frame parameters in a configuration file, and performing distortion correction and invalid edge clipping on the acquired multi-channel video images; performing down-sampling processing on the multi-channel video images after the distortion correction and invalid edge clipping, to generate multi-channel low-resolution thumbnails; performing feature point extraction and matching on the low-resolution thumbnails, to calculate a transformation matrix; performing full-resolution sub-pixel level registration and seamless fusion by using the transformation matrix; and transmitting and displaying the fused image data. The method decouples the real-time processing flow into two links of "parameter calculation based on low-resolution thumbnails" and "accurate execution of full-resolution images based on parameters", so that the calculation complexity and time cost are significantly reduced.

Description

Technical Field

[0001] This invention relates to the field of video stitching and transmission technology, and in particular to a method and system for low-latency processing and transmission of multi-channel video for real-scene fusion. Background Technology

[0002] With the rapid development of computer vision and communication technologies, the application of multi-channel video in real-time stitching and transmission is becoming increasingly widespread. The security monitoring field has a growing demand for panoramic high-definition video to achieve blind-spot-free coverage and refined monitoring; the medical imaging field requires low-latency, high-quality video transmission to support remote diagnosis and surgical guidance; and virtual reality (VR) relies on ultra-high-resolution video to provide an immersive experience. However, these application scenarios all place extremely high demands on the real-time performance, image quality, and bandwidth utilization of video transmission. Currently, ultra-high-resolution video faces many challenges in real-time stitching and transmission, such as high latency leading to desynchronized images, degraded image quality affecting the viewing experience, and bandwidth bottlenecks limiting data transmission efficiency. These problems not only restrict the practical application of related technologies but also become key challenges that urgently need to be solved by academia and industry. Summary of the Invention

[0003] To address the problems of existing technologies, this invention provides a method and system for low-latency processing and transmission of multi-channel video for real-scene fusion.

[0004] On the one hand, a method for low-latency processing and transmission of multi-channel video for real-scene fusion is provided, the method comprising:

[0005] S1: Simultaneous acquisition of multiple video streams and creation of hardware-accelerated processing channels for each stream;

[0006] S2: Acquire multiple video images from different channels and store them in the video processing resource buffer;

[0007] S3: Read the distortion correction parameters and cropping box parameters from the configuration file, and perform distortion correction and invalid edge cropping on the acquired multi-channel video images;

[0008] S4: Perform downsampling processing on the multi-channel video images after distortion correction and invalid edge cropping to generate multi-channel low-resolution thumbnails;

[0009] S5: Extract and match feature points on the low-resolution thumbnail, and calculate the transformation matrix;

[0010] S6: Full-resolution sub-pixel level registration and seamless fusion are performed using a transformation matrix;

[0011] S7: Transmit and display the fused image data.

[0012] Furthermore, S1 specifically includes:

[0013] Each camera in the array is given an independent hardware-accelerated processing channel that is hardware isolated from each other, and each channel is bound to the corresponding camera physical interface. Each channel is configured with a fixed-size Triple Buffer, which corresponds to the acquisition buffer, processing buffer, and output buffer respectively.

[0014] The system monitors the camera's working status in real time. When the camera goes offline, it automatically releases the hardware acceleration resources of the corresponding channel and reclaims them to the system resource pool for dynamic reuse by other channels or video fusion modules.

[0015] Furthermore, in S1, a hardware-triggered unified clock synchronization signal is used to achieve microsecond-level frame synchronization of multiple cameras and video stream timestamp alignment, thereby completing synchronous high-definition acquisition and parallel hardware acceleration processing of multiple industrial camera arrays.

[0016] Furthermore, S3 specifically includes:

[0017] Before system deployment, the checkerboard calibration method is used to calibrate each camera offline. By acquiring multi-pose checkerboard images, the camera intrinsic parameter matrix and distortion coefficient vector are calculated using the calibration toolkit. The parameters are saved in the configuration file of the file system. The pre-generated distortion correction lookup table is used to map the distorted image coordinates to the corrected image coordinates for the acquired camera images.

[0018] After distortion correction, the cropping is performed using a pre-defined cropping frame, and the information of the cropping frame parameters is stored in the configuration file.

[0019] Furthermore, S4 specifically includes:

[0020] The video image is divided into N×N pixel blocks using the regional average downsampling method. The arithmetic mean of all pixels in each pixel block is calculated as the corresponding pixel value of the thumbnail, thus generating a low-resolution thumbnail.

[0021] Furthermore, S5 specifically includes:

[0022] Feature points are extracted from low-resolution thumbnails using the ORB algorithm, and matching is performed using Hamming distance.

[0023] Using the selected matching point set, the homography transformation matrix applicable to full-resolution images is obtained through SVD decomposition to determine the perspective transformation relationship between two images. The key parameters include the eight degrees of freedom parameters of the homography transformation matrix and the interior point threshold.

[0024] Furthermore, S6 specifically includes:

[0025] By using the calculated homography transformation matrix to perform registration operations on the full-resolution image, sub-pixel level alignment accuracy is achieved, and a multi-band fusion algorithm is used to eliminate stitching seams.

[0026] Furthermore, in S7, the fused image data is directly transmitted from the frame buffer of the hardware acceleration channel to the display terminal through the PCIe channel, without needing to be copied to the CPU memory, thus reducing image latency.

[0027] On the other hand, a multi-channel video low-latency processing and transmission system for real-scene fusion is provided to implement the aforementioned multi-channel video low-latency processing and transmission method for real-scene fusion. The system includes:

[0028] Multi-channel industrial camera array, used to acquire multiple video images;

[0029] The channel creation module is used to create a hardware-accelerated processing channel for each camera.

[0030] The video acquisition module is used to acquire multiple video images and store them in the video processing resource buffer.

[0031] The distortion correction module is used to read the distortion correction parameters and cropping box parameters in the configuration file, and to perform distortion correction and invalid edge cropping on the acquired multi-channel video images.

[0032] The downsampling processing module is used to downsample the multi-channel video images after distortion correction and invalid edge cropping to generate multiple low-resolution thumbnails.

[0033] The feature point extraction and matching module is used to extract and match feature points on low-resolution thumbnails and calculate the transformation matrix.

[0034] The stitching and fusion module is used to perform full-resolution sub-pixel level registration and seamless fusion using a transformation matrix;

[0035] The video output module is used to transmit and display the fused image data.

[0036] Furthermore, the video output module is a frame buffer processing module with an integrated hardware acceleration channel, and the image data of the frame buffer in the module is directly transmitted to the display terminal via the PCIe channel.

[0037] The beneficial effects of the technical solution provided by the embodiments of the present invention are as follows: The present invention performs distortion correction and cropping processing on the acquired video stream, and then decomposes the traditional whole frame processing mode into thumbnail fast calculation through the "resolution decoupling processing" mechanism. The transformation matrix is ​​quickly calculated using the thumbnail and then mapped to the full resolution image for registration. This reduces the computational load and achieves seamless stitching through the multi-band fusion algorithm.

[0038] In addition, each camera has an independent hardware channel for fault isolation; the dynamic resource reuse strategy improves hardware utilization, avoids single-point bottlenecks, and significantly reduces the overall processing time cost, thereby reducing the overall video latency. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of a low-latency processing and transmission method for multi-channel video fusion based on real-scene fusion provided by the present invention;

[0041] Figure 2 This is an overall block diagram of a multi-channel video low-latency processing and transmission system for real-scene fusion provided by the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0044] Example 1

[0045] A method for low-latency processing and transmission of multi-channel video for real-scene fusion includes the following steps: Step (1): System startup, camera initialization, and video processing system initialization.

[0046] Step (2): Synchronous acquisition of multiple video streams and channel creation.

[0047] The system synchronously acquires live video streams using a multi-channel high-resolution industrial camera array. Upon system startup, an independent hardware-accelerated processing channel is created for each camera, with each channel bound to a specific camera physical interface.

[0048] Each camera has an independent hardware processing channel (including an independent DMA buffer, distortion correction engine, and cropping module), and the channels operate in parallel without blocking each other. A fixed-size circular buffer (TripleBuffer mechanism, including one set each of acquisition buffer, processing buffer, and output buffer) is reserved for each channel to prevent memory fragmentation; when a camera temporarily goes offline, its hardware acceleration unit resources are automatically released to the resource pool for dynamic reuse by other channels or fusion modules. Microsecond-level frame synchronization of multiple cameras is achieved through hardware-triggered clock synchronization signals to ensure timestamp alignment of each video stream.

[0049] Step (3): Acquire the captured video images and store them in the video processing resource buffer.

[0050] Step (4): Read the distortion correction parameters and cropping box parameters in the configuration file, perform distortion correction and invalid edge cropping on the acquired multi-channel video images, and obtain high-resolution video.

[0051] Before system deployment, checkerboard calibration was used to calibrate each camera offline. Multi-pose checkerboard images were acquired, and a calibration toolkit was used to calculate the camera intrinsic parameter matrix (focal length, principal point, etc.) and distortion coefficient vector, which were then stored in a configuration file in the file system. The hardware-accelerated processing channel used a pre-generated distortion correction lookup table (LUT) to map the distorted image coordinates to the corrected image coordinates from the acquired camera images.

[0052] After distortion correction, black borders remain at the image edges due to missing pixel mapping. A pre-defined cropping box is then used for cropping. The cropping box parameters (coordinates, length, width) are stored in a configuration file on the file system.

[0053] Step (5): The input high-resolution video is downsampled and the area average downsampling method is used to generate resolution thumbnails. Specifically, the full-resolution image is divided into N×N pixel blocks (usually N=4 or N=8), and the arithmetic mean of all pixels in each block is calculated as the corresponding pixel value of the thumbnail. A series of low-resolution thumbnails are generated. These thumbnails retain the key structural information of the original video while significantly reducing the amount of data.

[0054] Step (6): Feature point extraction and matching on the low-resolution thumbnail: Feature points are extracted using the ORB (Oriented Fast and Rotated BRIEF) algorithm and matched using Hamming distance. Using the filtered set of matching points, the homography transformation matrix is ​​solved by SVD decomposition to determine the perspective transformation relationship between the two images. Key parameters include the eight degrees of freedom of the homography transformation matrix H and the interior point threshold.

[0055] Step (7): Full-resolution subpixel level registration and seamless fusion: The full-resolution image is registered using the transformation matrix calculated from the thumbnail, and a multi-band fusion algorithm is used to eliminate stitching seams; the homography transformation matrix H obtained from the thumbnail is directly applied to the coordinate transformation of the full-resolution image. Since the thumbnail and the full-resolution image have a linear scale relationship, the transformation matrix does not need to be recalculated and can be directly mapped. The full-resolution image is resampled using bilinear interpolation to achieve subpixel level alignment accuracy.

[0056] Step (8): The fused image data is directly transmitted from the frame buffer of the hardware acceleration module to the display terminal through the PCIe channel without going through the CPU memory copy, which further reduces the image latency.

[0057] It is worth noting that, through the combined use of the above processes, this invention achieves video stream acquisition, distortion correction, and cropping in a multi-channel video processing system. By employing a "resolution decoupling processing" mechanism, the traditional full-frame processing mode is broken down into thumbnail-based rapid calculations. The transformation matrix is ​​then quickly calculated using the thumbnails and mapped to the full-resolution image for registration. This reduces the computational load and achieves seamless stitching through a multi-band fusion algorithm. Furthermore, each camera has an independent hardware channel, ensuring fault isolation; a dynamic resource reuse strategy improves hardware utilization, avoids single-point bottlenecks, and significantly reduces overall processing time costs, thereby reducing overall video latency.

[0058] Example 2

[0059] A low-latency multi-channel video processing and transmission system for real-scene fusion is provided to implement the low-latency multi-channel video processing and transmission method for real-scene fusion in Embodiment 1. The system includes: a multi-channel industrial camera array, a channel creation module, a video acquisition module, a distortion correction module, a downsampling processing module, a feature point extraction and matching module, a stitching and fusion module, and a video output module.

[0060] A multi-channel industrial camera array is used to synchronously acquire real-world video streams. A channel creation module creates a hardware-accelerated processing channel for each camera, with each channel bound to a specific camera physical interface. A video acquisition module acquires multiple video images and stores them in a video processing resource buffer. A distortion correction module reads distortion correction and cropping parameters from a configuration file and performs distortion correction and invalid edge cropping on the acquired video images. A downsampling module downsamples the distorted and cropped video images to generate multiple low-resolution thumbnails. A feature point extraction and matching module extracts and matches feature points on the low-resolution thumbnails and calculates the transformation matrix. A stitching and fusion module uses the transformation matrix for full-resolution sub-pixel level registration and seamless fusion. The video output module is a frame buffer processing module with an integrated hardware-accelerated channel; image data from the frame buffer within this module is directly transmitted to the display terminal via a PCIe channel.

[0061] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0062] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for low-latency processing and transmission of multi-channel video for real-scene fusion, characterized in that, The method includes: S1: Simultaneous acquisition of multiple video streams and creation of hardware-accelerated processing channels for each stream; S2: Acquire multiple video images from different channels and store them in the video processing resource buffer; S3: Read the distortion correction parameters and cropping box parameters from the configuration file, and perform distortion correction and invalid edge cropping on the acquired multi-channel video images; S4: Perform downsampling processing on the multi-channel video images after distortion correction and invalid edge cropping to generate multi-channel low-resolution thumbnails; S5: Extract and match feature points on the low-resolution thumbnail, and calculate the transformation matrix; S6: Full-resolution sub-pixel level registration and seamless fusion are performed using a transformation matrix; S7: Transmit and display the fused image data.

2. The method for low-latency processing and transmission of multi-channel video for real-scene fusion according to claim 1, characterized in that, S1 specifically includes: Each camera in the array is given an independent hardware-accelerated processing channel that is hardware isolated from each other, and each channel is bound to the corresponding camera physical interface. Each channel is configured with a fixed-size Triple Buffer, which corresponds to the acquisition buffer, processing buffer, and output buffer respectively. The system monitors the camera's working status in real time. When the camera goes offline, it automatically releases the hardware acceleration resources of the corresponding channel and reclaims them to the system resource pool for dynamic reuse by other channels or video fusion modules.

3. The method for low-latency processing and transmission of multi-channel video for real-scene fusion according to claim 2, characterized in that, The S1 uses a hardware-triggered unified clock synchronization signal to achieve microsecond-level frame synchronization of multiple cameras and video stream timestamp alignment, thus completing synchronous high-definition acquisition and parallel hardware acceleration processing of multiple industrial camera arrays.

4. The method for low-latency processing and transmission of multi-channel video for real-scene fusion according to claim 1, characterized in that, S3 specifically includes: Before system deployment, the checkerboard calibration method is used to calibrate each camera offline. By acquiring multi-pose checkerboard images, the camera intrinsic parameter matrix and distortion coefficient vector are calculated using the calibration toolkit. The parameters are saved in the configuration file of the file system. The pre-generated distortion correction lookup table is used to map the distorted image coordinates to the corrected image coordinates for the acquired camera images. After distortion correction, the cropping is performed using a pre-defined cropping frame, and the information of the cropping frame parameters is stored in the configuration file.

5. The method for low-latency processing and transmission of multi-channel video for real-scene fusion according to claim 1, characterized in that, S4 specifically includes: The video image is divided into N×N pixel blocks using the regional average downsampling method. The arithmetic mean of all pixels in each pixel block is calculated as the corresponding pixel value of the thumbnail, thus generating a low-resolution thumbnail.

6. The method for low-latency processing and transmission of multi-channel video for real-scene fusion according to claim 1, characterized in that, S5 specifically includes: Feature points are extracted from low-resolution thumbnails using the ORB algorithm, and matching is performed using Hamming distance. Using the selected matching point set, the homography transformation matrix applicable to full-resolution images is obtained through SVD decomposition to determine the perspective transformation relationship between two images. The key parameters include the eight degrees of freedom parameters of the homography transformation matrix and the interior point threshold.

7. The method for low-latency processing and transmission of multi-channel video for real-scene fusion according to claim 6, characterized in that, S6 specifically includes: By using the calculated homography transformation matrix to perform registration operations on the full-resolution image, sub-pixel level alignment accuracy is achieved, and a multi-band fusion algorithm is used to eliminate stitching seams.

8. The method for low-latency processing and transmission of multi-channel video for real-scene fusion according to claim 1, characterized in that, In S7, the fused image data is directly transmitted from the frame buffer of the hardware acceleration channel to the display terminal through the PCIe channel, without needing to be copied to the CPU memory, thus reducing image latency.

9. A multi-channel video low-latency processing and transmission system for real-scene fusion, characterized in that, The system is used to implement the multi-channel video low-latency processing and transmission method for real-scene fusion as described in any one of claims 1 to 8, the system comprising: Multi-channel industrial camera array, used to acquire multiple video images; The channel creation module is used to create a hardware-accelerated processing channel for each camera. The video acquisition module is used to acquire multiple video images and store them in the video processing resource buffer. The distortion correction module is used to read the distortion correction parameters and cropping box parameters in the configuration file, and to perform distortion correction and invalid edge cropping on the acquired multi-channel video images. The downsampling processing module is used to downsample the multi-channel video images after distortion correction and invalid edge cropping to generate multiple low-resolution thumbnails. The feature point extraction and matching module is used to extract and match feature points on low-resolution thumbnails and calculate the transformation matrix. The stitching and fusion module is used to perform full-resolution sub-pixel level registration and seamless fusion using a transformation matrix; The video output module is used to transmit and display the fused image data.

10. The multi-channel video low-latency processing and transmission system for real-scene fusion according to claim 9, characterized in that, The video output module is a frame buffer processing module with an integrated hardware acceleration channel. The image data of the frame buffer in the module is directly transmitted to the display terminal via the PCIe channel.

Images