Low-illumination enhancement method for panoramic video

Through parallel optical axis binocular array imaging and frequency domain decomposition enhancement technology, combined with short- and long exposure fusion and adaptive motion compensation, the problems of uneven brightness and loss of details in low-illumination panoramic monitoring are solved, and high-quality panoramic video acquisition is achieved , suitable for complex environments.

CN119067899APending Publication Date: 2024-12-03DRAGONFLY INTELLIGENT VISION TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411015579.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing low-illumination video enhancement technology is difficult to obtain stable and clear and rich in details surveillance video in environments with extremely weak lighting and complex and changeable scenes. Especially in panoramic monitoring applications, there are problems of uneven brightness, loss of details and blurred motion. Difficult to meet actual needs.

Method used

Using parallel optical axis binocular array imaging, short- and long-exposure fusion, frequency domain decomposition enhancement and area adaptive motion compensation technology, short-exposure and long-exposure images are collected through binocular parallel optical axis lens modules, image preprocessing, front and back scenes Separation and motion compensation, combined with frequency domain fusion strategy, generate high-quality low-illumination enhanced panoramic video.

Benefits of technology

It realizes stable acquisition of high-quality, large field of view video at extremely low illumination, significantly improves the signal-to-noise ratio in dark areas, avoids excessively enhanced artifacts, is suitable for dynamic and static mixed scenes, and provides 360-degree panoramic clear and uniform brightness image, supports large scene monitoring.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067899A_ABST
    Figure CN119067899A_ABST
Patent Text Reader

Abstract

The invention discloses a low-illumination enhancement method for a panoramic video. The method comprises the following steps: step 1, carrying out optical axis parallel calibration on at least four groups of binocular parallel optical axis lens modules; step 2, two lenses in each group of binocular parallel optical axis lens module respectively collect a short exposure image and a long exposure image of the same scene; 3, carrying out image preprocessing, foreground and background separation and motion compensation estimation on the short-exposure image and the long-exposure image collected by each group of binocular parallel optical axis lens modules; step 4, performing image registration and fusion on a high-frequency foreground image in the short-exposure image after foreground separation and the long-exposure background image after preprocessing to obtain a low-illumination enhanced image of each group of binocular parallel optical axis lens modules; and step 5, carrying out coding compression on the low-illumination enhanced images of all the binocular parallel optical axis lens modules to obtain multiple paths of low-illumination enhanced videos, and carrying out panoramic stitching on the multiple paths of low-illumination enhanced videos to generate a final low-illumination enhanced panoramic video.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of panoramic video enhancement methods, and more specifically, to a low-illumination enhancement method for panoramic videos. Background Art

[0002] With the continuous development and widespread application of video surveillance technology, people have put forward higher and higher requirements for the quality of surveillance videos at night and in low-light environments. However, due to factors such as insufficient light, high noise, and narrow dynamic range, traditional video surveillance systems often find it difficult to obtain clear and stable images under low-light conditions, which seriously affects the reliability and effectiveness of surveillance.

[0003] Currently, the following technical solutions are mainly used in the field of low-light video enhancement:

[0004] 1. Hardware-based enhancement methods. Representative solutions include increasing the lens aperture, using ultra-large image sensors, and external infrared fill lights. Although these methods have improved the low-light imaging quality to a certain extent, they also have problems such as limited improvement in imaging quality, complex and bulky systems, difficulty in integration, and high costs, and are subject to many restrictions in practical applications.

[0005] 2. Enhancement methods based on image post-processing. Representative solutions include histogram equalization, Retinex enhancement, wavelet transform denoising, bilateral filtering, etc. These methods are simple to operate and low in cost, but the performance improvement is not obvious, especially in complex scenes with extremely low illumination and severe noise. They often fail to achieve the ideal enhancement effect and are prone to over-enhancement artifacts.

[0006] 3. Enhancement methods based on multi-frame synthesis. Representative solutions include multi-frame noise reduction, super-resolution reconstruction, high dynamic range imaging, etc. This type of method uses image redundant information in the time domain to improve the imaging quality to a certain extent. However, most existing solutions are aimed at static scenes, and are prone to motion blur and smearing for moving targets. They also have high requirements on hardware computing power and are not ideal in real time.

[0007] 4. Enhancement method based on deep learning. This method uses the powerful feature extraction and mapping capabilities of convolutional neural networks to achieve relatively ideal enhancement effects. However, it has limitations such as difficulty in obtaining training samples, insufficient model generalization ability, large amount of calculation, and strong hardware dependence, which hinders its application on resource-constrained platforms such as embedded systems.

[0008] Although the above-mentioned solutions have achieved certain results in specific situations, they are still difficult to meet the diverse needs of actual monitoring applications, especially in harsh environments with extremely weak lighting and complex and changeable scenes. It is still a huge challenge to obtain stable, clear and detailed monitoring videos.

[0009] In addition, with the urgent need for large-scale panoramic monitoring in the fields of security, transportation, and industry, traditional monocular monitoring systems can no longer meet actual needs. However, most existing multi-camera array splicing solutions only consider the expansion of the field of view and ignore the imaging quality under low illumination, resulting in prominent problems such as uneven brightness, loss of details, and motion blur in panoramic videos, making it difficult to fully play the role of panoramic monitoring.

[0010] In summary, the existing low-light video enhancement technology cannot well take into account the actual needs of image quality, real-time performance, hardware adaptability, integration convenience and economy, especially the lack of an effective technical solution suitable for low-light panoramic monitoring. This greatly restricts the reliability and effectiveness of video monitoring at night and in low-light environments. It is urgent to research and develop a more systematic and excellent low-light enhancement method to meet the increasingly diverse monitoring application needs. Summary of the invention

[0011] In view of this, the present invention proposes a low-light enhancement method for panoramic videos. This method innovatively introduces key technologies such as parallel optical axis binocular array imaging, short-long exposure fusion, frequency domain decomposition enhancement, and regional adaptive motion compensation. Under the premise of controllable hardware costs, it achieves stable acquisition of high-quality, large-field-of-view videos under extremely low illumination.

[0012] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0013] A method for enhancing low illumination of panoramic video, the method comprising the following steps:

[0014] Step 1: perform optical axis parallel calibration on at least four groups of binocular parallel optical axis lens modules, wherein the horizontal field angle of each group of binocular parallel optical axis lens modules is greater than 90°, and the optical axis directions of the four groups of binocular parallel optical axis lens modules are spaced 90° apart, covering a total of 360° panoramic view;

[0015] Step 2: The two lenses in each group of the binocular parallel optical axis lens modules respectively capture a short exposure image and a long exposure image of the same scene;

[0016] Step three, performing image preprocessing, foreground and background separation, and motion compensation estimation on the short exposure images and long exposure images collected by each group of binocular parallel optical axis lens modules;

[0017] Step 4, performing image registration and fusion on the high-frequency foreground image in the short-exposure image after foreground separation and the long-exposure background image after preprocessing to obtain a low-illumination enhanced image of each set of binocular parallel optical axis lens modules;

[0018] Step 5: Encode and compress the low-light enhanced images of all binocular parallel optical axis lens modules to obtain multiple low-light enhanced videos, and perform panoramic stitching on the multiple low-light enhanced videos to generate a final low-light enhanced panoramic video.

[0019] Specifically, the binocular parallel optical axis lens module includes two lenses with strictly parallel optical axes and an image sensor. The focal lengths of the two lenses are the same, and the relative positions of the two lenses are fixed and as close as possible.

[0020] Specifically, the preprocessing of the long-exposure image in Step 3 includes: image scaling, overexposure removal, hole filling, and image magnification and restoration.

[0021] Specifically, the scaling ratio of the long-exposure image in Step 3 is 1 / 2, and the magnification and restoration ratio is 2 times.

[0022] Specifically, the preprocessing of the short-exposure image in Step 3 includes: image scaling, Gaussian filtering, background modeling, foreground and background separation, hole filling and denoising, and image magnification and restoration.

[0023] Specifically, the Gaussian kernel function used for the Gaussian filtering is:

[0024] where G(x, y) is the Gaussian function, (x, y) is the image coordinate, σ is the standard deviation of the Gaussian distribution, and its value determines the smoothness.

[0025] Specifically, the background modeling is based on the per-pixel Gaussian mixture model, and its probability density function is:

[0026]

[0027] where p(X t ) is the probability density of the pixel value being X at time t, K is the number of Gaussian distributions, N(X t ; μ t , ∑ i ) is the i-th Gaussian distribution, μ i is the mean value, ∑ i is the covariance, ω i is the weight of the i-th Gaussian distribution, i If the probability density p(X ) of the pixel point < T, it is judged as the foreground, otherwise it is the background, where T is the set threshold. t ) < T is judged as the foreground, otherwise it is the background, where T is the set threshold.

[0028] Specifically, the motion compensation estimation of the short-exposure image in Step 3 is based on the block matching algorithm. The image is divided into blocks and the best matching block is searched on the reference frame. The displacement between the matching block and the current block is the motion vector, that is:

[0029]

[0030] Among them, C t is the block image of the current frame, R t-1 is the reference frame image, N x ,N y is the block size, (u,v) is the motion vector, and the MAD is minimized.

[0031] Specifically, the fusion of the short-exposure foreground image and the long-exposure background image in step 4 is achieved by weighting according to the frequency band, that is:

[0032] F=W H ·F H +W M ·F M +W L ·B L +W0

[0033] Among them, F is the fusion result, F H ,F M are the high-frequency and medium-frequency components of the short-exposure foreground image, respectively. L is the low-frequency component of the long-exposure background image, W H ,W M ,W L is the weight of the corresponding component, and W0 is the brightness compensation.

[0034] Specifically, another implementation of the short exposure image preprocessing in step 3 is: after enhancing the brightness and contrast of the short exposure image, IIR filtering is used to separate the foreground and background, and the IIR filtering is:

[0035] I(xy)=αI′(x,y)+(1-α)I′ p (x,y)

[0036] Among them, I(x,y) is the filter output, I′(x,y) is the current short exposure image, and I′ p (x, y) is the filtering result of the previous frame, and 0<α<1 is the smoothing factor.

[0037] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0038] 1. Make full use of the advantages of short and long exposure complementary imaging, while retaining the details in the bright area, significantly improve the signal-to-noise ratio in the dark area, make the image more layered and greatly improve the visual quality.

[0039] 2. The frequency domain fusion strategy can adaptively adjust the weight of each frequency band according to the scene content. While enhancing the detail texture, it avoids artifacts such as over-enhancement and gradient reversal, making the processed image more natural.

[0040] 3. The adaptive motion estimation and compensation algorithm can accurately obtain pixel-level motion information and effectively eliminate the afterimage blur of moving targets. It is suitable for mixed dynamic and static scenes.

[0041] 4. The perfect panoramic stitching solution can obtain 360-degree panoramic images with a wide field of view, clear details and uniform brightness, providing strong support for large-scene monitoring.

[0042] 5. Modular processing flow, simple and compact hardware structure, low algorithm complexity, easy for engineering implementation and industrial promotion.

[0043] In summary, the panoramic video low-light enhancement method proposed in the present invention can effectively overcome the shortcomings of the existing technology and realize high-quality, panoramic video acquisition in extremely weak light environments. It provides strong technical support for night security monitoring, emergency rescue, archaeological detection and other fields, and promotes the development and application of computer vision technology in the field of low-light imaging. It has important theoretical research value and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A complete flow chart of a low-light enhancement method for panoramic video of the present invention;

[0045] Figure 2 is another flow chart of a low-light enhancement method for panoramic video of the present invention;

[0046] Figure 3 A stitching flow chart of a low-light enhancement method for panoramic video of the present invention;

[0047] Figure 4 A calibration flow chart of a low-light enhancement method for panoramic video of the present invention;

[0048] Figure 5 The figure is a flow chart of a method for enhancing low illumination of panoramic video according to the present invention. DETAILED DESCRIPTION

[0049] The scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. The embodiments in the present invention and all other embodiments obtained by ordinary persons in the art without making creative work are within the scope of protection of the present invention.

[0050] In the description of the present invention, it should be noted that the terms "upper", "lower", "inner", "outer", and "top / bottom" indicate the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0051] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "provided with", "set / connected", and "connected" should be understood in a broad sense. For example, "connected" means fixed connection, detachable connection, or integral connection; mechanical connection, electrical connection; direct connection, indirect connection through an intermediate medium, and internal communication between two components. For ordinary people in the field, the specific meanings of the above terms in the present invention are understood according to specific circumstances.

[0052] Please refer to Figure 1-5 The present invention provides a low-light enhancement method for panoramic video, comprising the following steps:

[0053] Step 1: calibrate at least four groups of binocular parallel optical axis lens modules for optical axis parallelism, wherein the horizontal field angle of each group of binocular parallel optical axis lens modules is greater than 90°, and the optical axis directions of the four groups of binocular parallel optical axis lens modules are spaced 90° apart, covering a total of 360° panoramic view;

[0054] Four or more binocular lens modules are used here, arranged at 90-degree intervals horizontally, and each binocular lens has a field of view greater than 90 degrees. This layout can seamlessly cover a 360-degree panoramic imaging range. The key is that each binocular lens must be precisely calibrated with parallel optical axes to ensure that only horizontal parallax exists in the captured image, laying the foundation for subsequent image registration and fusion.

[0055] The binocular parallel optical axis lens module comprises two lenses with strictly parallel optical axes and an image sensor. The two lenses have the same focal length, and the relative positions of the two lenses are fixed and as close as possible.

[0056] A feature of the present invention is that the binocular lens structure is miniaturized and the optical axes are precisely parallel. The two lenses have the same parameters, are arranged along the same horizontal plane and as close as possible to obtain a stable binocular parallax relationship. In practice, the lens focal length can be selected in the range of 2.8mm to 8mm, and the baseline distance is controlled within 3cm. The binocular calibration can adopt the Zhang Zhengyou calibration method or the automatic calibration method based on point features.

[0057] Step 2: The two lenses in each group of the binocular parallel optical axis lens modules respectively capture a short exposure image and a long exposure image of the same scene;

[0058] One of the binocular lenses uses a shorter exposure time to capture images with clear moving scenes but darker backgrounds; the other lens uses a longer exposure time to capture images with brighter backgrounds but blurred moving scenes. The two exposure methods complement each other and provide the hardware foundation for obtaining high dynamic range images with rich light and dark levels.

[0059] Regarding the setting of exposure time, you can refer to the following empirical values: short exposure time can be set to about 1 / 2 of normal exposure time, such as 1 / 30s, 1 / 60s, etc.; long exposure time can be set to about 2 times of normal exposure, such as 1 / 8s, 1 / 4s, etc. When the light changes, the short and long exposure times can be dynamically adjusted through the automatic gain control circuit or the mapping relationship obtained by the lookup table. In short, it is necessary to strike a balance between obtaining sufficient environmental information and avoiding excessive blur.

[0060] Step three, performing image preprocessing, foreground and background separation, and motion compensation estimation on the short exposure images and long exposure images collected by each group of binocular parallel optical axis lens modules;

[0061] Image preprocessing mainly includes operations such as scaling, denoising, and overexposure removal, among which:

[0062] The preprocessing process for short exposure images is: image scaling (such as 1 / 2) → Gaussian filtering → foreground segmentation → foreground image enhancement → image magnification and restoration (such as 2 times).

[0063] The preprocessing process for long exposure images is: image scaling (such as 1 / 2) → overexposed area detection and threshold removal → overexposed point repair based on morphological or frequency domain filtering → image denoising and enhancement → image magnification and restoration (such as 2 times).

[0064] The purpose of image scaling is to reduce the amount of computation and to facilitate multi-scale analysis. The scaling factor in the present invention can be flexibly set, preferably a power of 2.

[0065] Gaussian filtering is used to smooth the image in preparation for foreground segmentation. The Gaussian kernel used is:

[0066]

[0067] Where (x, y) is the pixel coordinates, and σ is the Gaussian scale parameter. The larger σ is, the more obvious the smoothing effect is. It is usually an odd number, and its size is related to the image size and noise level. 3×3 or 5×5 kernels are more commonly used.

[0068] The foreground segmentation uses a mixed Gaussian background modeling algorithm. The basic principle is to use a weighted mixture of multiple Gaussian distributions to approximate the background pixel grayscale distribution. Where p(X t ) is the pixel grayscale at time t is X t The probability density of K is the number of Gaussian components (usually 3 to 5), N(X t ;μ i ,∑ i ) is the mean μ i , covariance matrix∑ i The Gaussian component of i is its weight coefficient, and

[0069] Background updates are done recursively:

[0070] ω it =(1-α)ω i,t-1 +αM i,t

[0071] μ it =(1-ρ)μ i,t-1 +ρX t

[0072] ∑ it =(1-ρ)∑ i,t-1 +ρ(X t -μ i,t )X t -μ i,t ) T

[0073] Where α is the learning rate, ρ = αN(X t ;μ i ,∑ i ) is the update rate, M i , t is whether the i-th component matches pixel X t The indicator function of the pixel is judged as the foreground when: |(X t -μ i )|>λ∑ i . Or it does not belong to the first B background Gaussian components. Generally, K = 3-5, B = 1-3, α = 0.005, λ = 2.5-3.5. The specific parameters need to be set in combination with the actual scene. Perform morphological closing operation on the segmented foreground to remove small internal holes and edge depressions.

[0074] Motion compensated estimation is based on a block matching algorithm:

[0075]

[0076] That is, the current frame image is divided into blocks, and the block most similar to the current block is searched in the previous reference frame. The offset is the motion vector (u, v). The similarity criterion uses the mean absolute difference (MAD) of the grayscale between blocks. To speed up the matching, strategies such as pyramid layering and motion vector prediction can be used.

[0077] The selection of block size needs to take into account both motion resolution and computational efficiency. If the block size is too large, the motion matching will be distorted, and if it is too small, the computational effort will increase. In practice, the block size is generally 8×8 to 32×32. For scenes with small motion, full-frame search can be used, or a fixed search window, such as 16×16, can be set. For large motion, the window can be adaptively enlarged according to the motion amplitude, or a hierarchical coarse and fine search strategy can be used.

[0078] Step 4, performing image registration and fusion on the high-frequency foreground image in the short-exposure image after foreground separation and the long-exposure background image after preprocessing to obtain a low-illumination enhanced image of each set of binocular parallel optical axis lens modules;

[0079] First, use the motion vector obtained in step 3 to perform motion compensation on the foreground and background respectively to eliminate the position offset caused by parallax and motion. The compensation formula is:

[0080] I c (ij) = I p (iu,jv)

[0081] Among them I c is the compensated image, I p is the previous frame image, (u,v) is the motion vector of pixel (i,j). For half-integer vector values, bilinear interpolation can generally be used for approximation.

[0082] After alignment, a multi-band fusion algorithm based on the Laplacian pyramid is used to integrate the high-frequency detail information of the short-exposure image and the low-frequency background information of the long-exposure image. The steps are as follows:

[0083] (1) For the short exposure foreground image F H and long exposure background image B L Gaussian pyramid decomposition is performed separately to obtain their respective low-frequency pyramids {G F 0,G F 1,…,G F N} and {G B 0,G B 1,…,G B N}, where N is the number of decomposition levels.

[0084] (2) Calculate the Laplacian pyramid from the Gaussian pyramid to obtain the respective high-frequency passband subbands {L F 0,L F 1,…,L FN-1} and {L B 0,L B 1,…,L BN-1},in

[0085] L F 0=G F 0,L B 0=G B 0,

[0086] L Fk =G Fk -Expand(G Fk+1 ),k=1,2,…,N-1

[0087] Among them, Expand is an interpolation and magnification operation.

[0088] (3) Fusion of high and low frequencies:

[0089] L k =W F ×L Fk +W B ×L Bk ,k=1,2,…,N-1

[0090] L0=W F ×G FN +W B ×G BN

[0091] By weighted fusion of high-frequency details and low-frequency background, a new Laplacian pyramid {L0,L1,…,L N-1}. Weight W F and W B Satisfy W F +W B = 1. Empirically, the foreground weight W F Take 0.6-0.8,W B Take 0.2-0.4.

[0092] (4) The fused Laplacian pyramid is restored layer by layer, interpolated and enlarged until the original resolution is reached, thus obtaining the final low-light enhanced image R

[0093] R=L0+ExpandL1+Expand(…+Epand(LN -1 ))

[0094] It should be pointed out that when constructing the Laplacian pyramid, the scaling ratio between adjacent layers should match the initial Gaussian smoothing parameter σ, which is generally 1.4 to 2 times. In the common 5-layer decomposition pyramid, the scaling ratios of adjacent layers are 1 / 2, 1 / 4, 1 / 8, and 1 / 16, corresponding to the selected σ = 1.0, 1.6, 2.0, 3.2, 5.12, etc.

[0095] Step 5: Encode and compress the low-light enhanced images of all binocular parallel optical axis lens modules to obtain multiple low-light enhanced videos, and perform panoramic stitching on the multiple low-light enhanced videos to generate the final low-light enhanced panoramic video. Perform H.264 encoding compression on the enhanced multiple videos, and the compression ratio can be flexibly set between 30 and 60. Use the image stitching algorithm based on SIFT feature point matching to perform 360-degree sequential stitching on the multiple videos.

[0096] The details are as follows: (1) Extract SIFT features of two adjacent videos, perform feature point matching, and remove incorrect matching point pairs. (2) Calculate the affine transformation matrix or homography matrix between the videos based on the coordinates of the matching points, and use algorithms such as RANSAC to remove outliers. (3) Perform frame-by-frame registration and splicing of the videos based on the transformation matrix, and use weighted averaging or multi-band fusion methods to achieve smooth transition in the splicing area. (4) Perform post-processing such as color correction and brightness balance on the panoramic video to output the final low-light enhanced panoramic video.

[0097] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0098] Example 1

[0099] Step 1: Build 4 sets of parallel optical axis binocular lens arrays. Each array contains 2 5-megapixel image sensors and 8mm wide-angle fixed-focus lenses. The lens focal length deviation is less than 196, and the binocular baseline distance is 6cm. The four arrays are arranged in a 90-degree plum blossom shape, and the optical axis is precisely parallelized through chessboard calibration. The field of view range is about 400° (horizontal) × 120° (vertical).

[0100] Step 2: The binocular lens exposure parameters are set as: short exposure time 1 / 60s, long exposure time 1 / 8s, 0dB analog gain.

[0101] Step 3: The original left and right images of 640×480 are collected and first downsampled to 1 / 2. Then, the short-exposure image is subjected to Gaussian filtering with σ=1.0, and the foreground is extracted according to the mixed Gaussian model, with the number of Gaussian components K=3, the matching coefficient λ=2.8, the learning rate α=0.005, and the background judgment threshold B=2. The extracted foreground image is subjected to opening operation for denoising and closing operation for filling holes. Finally, it is interpolated and enlarged to the original resolution. Overexposure detection is performed on the long-exposure image, with the threshold set to 250, and the overexposed area is repaired by median filtering, and then the overall histogram equalization is enhanced, and finally it is interpolated and enlarged to the original resolution.

[0102] Step 4: Using the long exposure image as a reference, a three-step search algorithm is used to estimate the motion of the short exposure image, with a search range of ±16 pixels, a block size of 16×16, and a step size of 8 pixels. After obtaining the motion vector field, motion compensation is performed in units of blocks.

[0103] Then, the compensated short-exposure foreground image and long-exposure background image are decomposed into 5-level Laplacian pyramids, with the scale coefficients σ set to 1.0, 1.6, 2.0, 3.2, and 5.12 respectively. When fused, the foreground and background weights of each layer are 0.8 and 0.2 respectively, the second-to-last layer is 0.6 and 0.4, and the top layer is 0.5 and 0.5. The fused pyramid is interpolated and restored layer by layer to obtain a 320×240 low-light enhanced image.

[0104] Step 5: Encode the 4 enhanced image sequences in H.264, set the GOP length to 50 frames, and the quantization parameter QP = 35, and output 4 640×480 low-light enhanced videos. When splicing the videos, extract SIFT features, use the RANSAC algorithm to estimate the homography transformation matrix between adjacent videos, and the splicing order is: top → right → bottom → left, the splicing seam is 80 pixels, the transition area is smoothed by multi-band fusion, and the color correction uses the histogram matching algorithm. Finally, the low-light panoramic video of 1920×960 is output.

[0105] The measured results show that the low-light enhanced panoramic video system of this embodiment can obtain clear, bright, and coherent 360-degree panoramic video images at an extremely low illumination of 0.05 Lux, with a signal-to-noise ratio increased by more than 8 dB and a dynamic range expanded by 2.1 times, which is 3 to 4 times that of the traditional monocular panoramic system. The subjective visual effect is far better than simple multi-frame noise reduction, histogram equalization and other post-processing methods. The technical performance advantages of the present invention are fully verified.

[0106] Example 2

[0107] To further improve the imaging quality under harsh lighting conditions, this embodiment makes the following improvements and optimizations to the solution of Embodiment 1:

[0108] (1) The binocular lens group was replaced with a fixed-focus lens with a larger photosensitive area (1 / 1.8") and an all-glass lens. The lens aperture was increased from F2.0 to F1.6, effectively increasing the amount of light entering by 1.2 times.

[0109] (2) Apply gamma correction of Gamma = 0.6 to the long exposure image to compress the highlight area and reduce the overexposure tendency.

[0110] (3) In the mixed Gaussian background modeling, the number of Gaussian components K is increased from 3 to 5, and the background judgment threshold B is reduced from 2 to 1 to adapt to more complex backgrounds.

[0111] (4) In the Laplacian pyramid fusion, the low-frequency part of the top layer is filled with a long-exposure image to retain more background details. At the same time, the foreground weight is increased to 0.85, and the background weight is reduced accordingly to highlight the foreground.

[0112] (5) When stitching videos, a brightness smoothing step based on regional energy is added after color correction to further improve the brightness consistency of the panoramic video.

[0113] After testing, the video image of this embodiment is still clearly discernible under ultra-low illumination of 0.001 Lux, the sharpness is improved by more than 20% compared with embodiment 1, the background texture is richer, and the visual effect is close to that of normal illumination video, which proves that the optimization measures of the present invention are practical and effective.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A low-light enhancement method for panoramic video, characterized in that: The method comprises the following steps: Step 1: perform optical axis parallel calibration on at least four groups of binocular parallel optical axis lens modules, wherein the horizontal field angle of each group of binocular parallel optical axis lens modules is greater than 90°, and the optical axis directions of the four groups of binocular parallel optical axis lens modules are spaced 90° apart, covering a total of 360° panoramic view; Step 2: The two lenses in each group of the binocular parallel optical axis lens modules respectively capture a short exposure image and a long exposure image of the same scene; Step three, performing image preprocessing, foreground and background separation, and motion compensation estimation on the short exposure images and long exposure images collected by each group of binocular parallel optical axis lens modules; Step 4, performing image registration and fusion on the high-frequency foreground image in the short-exposure image after foreground separation and the long-exposure background image after preprocessing to obtain a low-illumination enhanced image of each set of binocular parallel optical axis lens modules; Step 5: Encode and compress the low-light enhanced images of all binocular parallel optical axis lens modules to obtain multiple low-light enhanced videos, and perform panoramic stitching on the multiple low-light enhanced videos to generate a final low-light enhanced panoramic video.

2. The panoramic video low illumination enhancement method according to claim 1, characterized in that: The binocular parallel optical axis lens module comprises two lenses with strictly parallel optical axes and an image sensor. The two lenses have the same focal length, and the relative positions of the two lenses are fixed and as close as possible.

3. The panoramic video low illumination enhancement method according to claim 1, characterized in that: The preprocessing of the long exposure image in step three includes: image scaling, overexposure removal, hole filling and image magnification restoration.

4. The panoramic video low illumination enhancement method according to claim 1 or 3, characterized in that: In the step 3, the scaling ratio of the long exposure image is 1 / 2, and the magnification restoration ratio is 2 times.

5. The panoramic video low illumination enhancement method according to claim 1, characterized in that: The preprocessing of the short exposure image in step three includes: image scaling, Gaussian filtering, background modeling, foreground and background separation, hole filling and denoising, and image magnification and restoration.

6. The method for low-light enhancement of panoramic video according to claim 5, characterized in that: The Gaussian kernel function used in the Gaussian filtering is: Among them, G(x, y) is the Gaussian function, (x, y) is the image coordinate, and σ is the standard deviation of the Gaussian distribution, and its value determines the degree of smoothing.

7. The method for enhancing low illumination of panoramic video according to claim 5, characterized in that: The background modeling is based on a pixel-by-pixel Gaussian mixture model, whose probability density function is: where p(X t ) is the probability density that the pixel value is X at time t, K is the number of Gaussian distributions, N(X t ; μ i , ∑ i ) is the i-th Gaussian distribution, μ i is the mean, ∑ i is the covariance, ω i is the weight of the i-th Gaussian distribution, if the probability density p(X t ) of the pixel point < T, it is judged as the foreground, otherwise it is the background, where T is the set threshold.​​ 8. The method for low-light enhancement of panoramic video according to claim 1, characterized in that: The motion compensation estimation of the short exposure image in step 3 is based on a block matching algorithm, which divides the image into blocks and searches for the best matching block on the reference frame. The displacement between the matching block and the current block is the motion vector, that is: Among them, C t is the block image of the current frame, R t-1 is the reference frame image, N x , N y is the block size, (u, v) is the motion vector, and the MAD is minimized.

9. The panoramic video low illumination enhancement method according to claim 1 is characterized in that: In step 4, the fusion of the short exposure foreground image and the long exposure background image is achieved by weighting the frequency band, that is: F=W H ·F H +W M ·F M +W L ·B L +W0 Among them, F is the fusion result, F H , F M are the high-frequency and medium-frequency components of the short-exposure foreground image, respectively. L is the low-frequency component of the long-exposure background image, W H , W M , W L is the weight of the corresponding component, and W0 is the brightness compensation.

10. The method for low-light enhancement of panoramic video according to claim 1, characterized in that: Another implementation method of the short exposure image preprocessing in step 3 is: after enhancing the brightness and contrast of the short exposure image, IIR filtering is used to separate the foreground and background. The IIR filtering is: I(xy)=αI′(x,y)+(1-α)I′ p (x,y) Among them, I(x, y) is the filter output, I′(x, y) is the current short exposure image, and I′ p (x, y) is the filtering result of the previous frame, and 0<α<1 is the smoothing factor.