Ultrahigh-definition long-distance full-color imaging method under extremely low illumination and module thereof

Abstract

The invention provides an ultra-high-definition long-distance full-color imaging method under extremely low illumination and a module thereof, and relates to the technical field of image processing, and the method comprises the steps: carrying out the short-exposure shooting of the same scene, synchronously obtaining a plurality of visible light frames and a plurality of infrared frames, and carrying out the image registration; performing brightness enhancement on the plurality of corresponding visible light frames based on the plurality of infrared frames, and synthesizing the plurality of visible light frames after brightness enhancement by using an adaptive weighted fusion algorithm to generate a long exposure equivalent single-frame image; performing noise reduction processing on the long exposure equivalent single-frame image and the plurality of infrared frames; inputting the long-exposure equivalent single-frame image after noise reduction processing and the plurality of infrared frames into a pre-trained color derivation model for color reconstruction, and generating a color information matrix for constructing a full-color image; and based on the color information matrix, performing color mapping and coloring on the long-exposure equivalent single-frame image to generate a final full-color image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an ultra-high-definition long-distance full-color imaging method under extremely low illumination and a module thereof. Background Art

[0002] In the prior art, in low-light conditions, traditional visible light camera equipment suffers from severe image quality degradation due to the decrease in signal-to-noise ratio, resulting in a large amount of noise, loss of details, and color distortion. To overcome this problem, some existing solutions include using high-sensitivity sensors, increasing artificial lighting, or using long exposure times to increase image brightness. However, these methods all have their own limitations: although high-sensitivity sensors can increase sensitivity, they usually introduce more noise; increasing artificial lighting may change the natural state of the scene and is not feasible in some application scenarios (such as wildlife observation, military reconnaissance, etc.); long exposure times can easily cause blurring of moving objects and are not suitable for shooting dynamic scenes.

[0003] Therefore, it is necessary to provide an ultra-high-definition long-distance full-color imaging method and module under extremely low illumination to solve the above technical problems. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides an ultra-high-definition long-distance full-color imaging method and module under extremely low illumination, which solves the problem that traditional cameras are difficult to capture clear and color-accurate images at night or under extremely low light conditions. By combining visible light and infrared imaging technologies, high-quality color image reconstruction can be achieved.

[0005] The present invention provides an ultra-high-definition long-distance full-color imaging method under extremely low illumination, the method comprising the following steps:

[0006] Perform short exposure shooting for the same scene, synchronously acquire multiple visible light frames and multiple infrared frames and perform image registration, wherein the multiple visible light frames are captured by a configured visible light camera, and the multiple infrared frames are captured by a configured infrared sensor;

[0007] Based on the multiple infrared frames, brightness enhancement is performed on the corresponding multiple visible light frames, and the multiple visible light frames after brightness enhancement are synthesized by using an adaptive weighted fusion algorithm to generate a long exposure equivalent single frame image;

[0008] Performing noise reduction processing on the long exposure equivalent single frame image and the multiple infrared frames;

[0009] Inputting the long-exposure equivalent single-frame image and the multiple infrared frames after noise reduction processing into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image, wherein the color derivation model is a deep learning model trained with sample data including visible light images and infrared image pairs in extremely low illumination scenes;

[0010] Based on the color information matrix, the long-exposure equivalent single-frame image is color mapped and colored to generate a final full-color image.

[0011] Preferably, the image registration includes:

[0012] Extracting feature points from the multiple visible light frames and the multiple infrared frames;

[0013] Matching the feature points in the visible light frame with the feature points in the infrared frame using a feature point matching algorithm;

[0014] According to the matched feature points, the transformation parameters between the images are calculated;

[0015] The transformation parameters are applied to spatially align the visible light frame with the infrared frame to complete image registration.

[0016] Preferably, the brightness enhancement of the corresponding multiple visible light frames based on the multiple infrared frames, and synthesis of the multiple visible light frames after brightness enhancement using an adaptive weighted fusion algorithm to generate a long exposure equivalent single frame image includes:

[0017] Based on the brightness distribution in the multiple infrared frames, identify the high brightness area and the low brightness area in the scene, and enhance the brightness of the low brightness area in the corresponding visible light frame according to the thermal radiation intensity provided by the multiple infrared frames;

[0018] For each set of registered and brightness-enhanced visible light frames, calculate the local contrast and brightness values ​​around each pixel;

[0019] Dynamically adjust the weights between different frames based on the local contrast and brightness values ​​of each pixel;

[0020] Using the calculated weights, all visible light frames are weighted averaged to generate a long-exposure equivalent single-frame image.

[0021] Preferably, the weight adjustment is performed according to a preset dynamic weight function, wherein the dynamic weight function is used to determine the weight contributed by each pixel in the long exposure equivalent single frame image according to the local contrast and brightness value of each pixel. Specifically, the mathematical expression of the dynamic weight function is:

[0022]

[0023] Where f represents the frame index, N represents the number of visible light frames, and g f (C, L) represents the comprehensive function combining the local contrast C and the brightness value L;

[0024] Among them, the mathematical expression of the comprehensive function is:

[0025]

[0026] Among them, ∈ represents a small positive number, α represents a parameter for controlling the degree of influence of contrast, β represents a parameter for controlling the degree of influence of brightness value, and μ represents the expected average brightness value.

[0027] Preferably, the performing noise reduction processing on the long exposure equivalent single frame image and the multiple infrared frames includes:

[0028] Applying a multi-scale transformation algorithm to decompose the long-exposure equivalent single-frame image and the multiple infrared frames respectively to obtain image components of different frequencies;

[0029] Performing noise estimation on each frequency component based on a statistical model, and adjusting the threshold of each component according to the estimation result to remove the noise, wherein the statistical model is pre-trained with data including noisy and noise-free image pairs;

[0030] The frequency components that have been denoised are recombined through inverse multi-scale transformation to restore the denoised long-exposure equivalent single-frame image and infrared frame.

[0031] Preferably, the step of inputting the long-exposure equivalent single-frame image and the multiple infrared frames after the noise reduction process into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image includes:

[0032] In the color reconstruction process, the color derivation model receives the long-exposure equivalent single-frame image and the infrared frame after noise reduction as input, and extracts the features of the image through multi-layer convolution, activation, pooling and full connection layer operations;

[0033] The extracted features are mapped to the color space using the encoding and decoding structure within the color derivation model to reconstruct a color information matrix that matches the original scene.

[0034] Preferably, the color mapping and coloring of the long-exposure equivalent single-frame image based on the color information matrix to generate a final full-color image includes:

[0035] Using the color distribution information in the color information matrix, the grayscale value of the long-exposure equivalent single-frame image is converted into a corresponding RGB color value through a lookup table;

[0036] For each pixel, combining its original brightness information and the color information obtained from the color information matrix, applying the Lab color space model to adjust the color attribute of the corresponding pixel;

[0037] Perform color correction on the adjusted color properties to obtain the final full-color image.

[0038] The present invention also provides an ultra-high-definition long-distance full-color imaging module under extremely low illumination, comprising a visible light camera, an infrared sensor and an image processing unit, wherein the image processing unit is used to perform the following steps:

[0039] Take short-exposure photos of the same scene, acquire multiple visible light frames and multiple infrared frames simultaneously, and perform image registration;

[0040] Based on the multiple infrared frames, brightness enhancement is performed on the corresponding multiple visible light frames, and the multiple visible light frames after brightness enhancement are synthesized by using an adaptive weighted fusion algorithm to generate a long exposure equivalent single frame image;

[0041] Performing noise reduction processing on the long exposure equivalent single frame image and the multiple infrared frames;

[0042] Inputting the long-exposure equivalent single-frame image and the multiple infrared frames after noise reduction processing into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image, wherein the color derivation model is a deep learning model trained with sample data including visible light images and infrared image pairs in extremely low illumination scenes;

[0043] Based on the color information matrix, the long-exposure equivalent single-frame image is color mapped and colored to generate a final full-color image.

[0044] Compared with the related art, the ultra-high-definition long-distance full-color imaging method and module under extremely low illumination provided by the present invention have the following beneficial effects:

[0045] The present invention synchronously acquires multiple visible light frames and infrared frames of the same scene, and performs precise image registration to ensure spatial alignment between the two modal images, thereby laying the foundation for subsequent processing. Next, the brightness distribution information in the infrared frame is used to enhance the brightness of the visible light frame, especially for low-brightness areas. Then, an adaptive weighted fusion algorithm is used to dynamically adjust the weights between different frames according to the local contrast and brightness value of each pixel point, and a single-frame image equivalent to a long exposure effect is synthesized to simulate the high-brightness image that can be obtained by long-term exposure, while avoiding the problem of motion blur.

[0046] In order to further improve the image quality, the method also includes a denoising step, which applies a multi-scale transformation algorithm and a noise estimation technique based on a statistical model to effectively remove noise from the image while retaining important details. Subsequently, the denoised long-exposure equivalent single-frame image and infrared frame are input into a pre-trained color inference model, which is a deep learning model trained with sample data of visible light and infrared image pairs in extremely low-light scenes. It is used for color reconstruction to generate a color information matrix that matches the original scene. Finally, based on this color information matrix, the Lab color space model is combined to perform color mapping and coloring on the long-exposure equivalent single-frame image, including the conversion of grayscale values ​​to RGB color values, color attribute adjustment and color correction, and finally generate a full-color image with accurate colors and rich details. This method can not only provide high-quality color images under extreme lighting conditions, but also has good adaptability and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flow chart of an ultra-high-definition, long-distance, full-color imaging method under extremely low illumination provided by the present invention;

[0048] Figure 2 A schematic diagram of the structure of an ultra-high-definition, long-distance, full-color imaging module under extremely low illumination provided by the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0050] It should also be noted that, for ease of description, only the part relevant to the present invention but not all content is shown in the accompanying drawings. It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processing or methods depicted as flow charts. Although the flow chart describes each operation (or step) as sequential processing, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0051] Embodiment 1

[0052] The present invention provides an ultra-high-definition long-distance full-color imaging method under extremely low illumination, referring to Figure 1As shown, the method comprises the following steps:

[0053] S1: Perform short exposure shooting for the same scene, synchronously acquire multiple visible light frames and multiple infrared frames and perform image registration, wherein the multiple visible light frames are captured by a configured visible light camera, and the multiple infrared frames are captured by a configured infrared sensor.

[0054] Specifically, image registration includes the following steps:

[0055] S11: extracting feature points from the multiple visible light frames and the multiple infrared frames.

[0056] In this embodiment, under extremely low illumination conditions, visible light images usually have lower contrast and higher noise, while infrared images provide thermal radiation information but lack color information. In order to ensure that the two modal images can be accurately aligned, stable feature points must be extracted from each image. These feature points can be corners, edges or other significant structures, which have good robustness under different lighting conditions. The feature point detection algorithm includes but is not limited to Harris corner detection, SIFT (scale invariant feature transform), and SURF (speeded up robust features). The feature point detection algorithm can extract a set of feature points from the visible light frame and the infrared frame respectively, providing a basis for subsequent matching and registration.

[0057] S12: Matching the feature points in the visible light frame with the feature points in the infrared frame using a feature point matching algorithm.

[0058] In this embodiment, due to the large difference between visible light and infrared images, direct matching may lead to mismatching. Therefore, it is necessary to use a robust matching algorithm, including but not limited to RANSAC (Random Sample Consensus) and FLANN (Fast Nearest Neighbor Search Library), to screen out reliable matching pairs. The matching algorithm can find the matching point pairs that best meet the geometric constraints in the presence of a large number of outliers. Through the matching process, the relative position relationship between the two images can be preliminarily determined to prepare for further calculation of transformation parameters.

[0059] S13: Calculate the transformation parameters between the images based on the matched feature points.

[0060] In this embodiment, after completing the feature point matching, the next step is to calculate the geometric transformation parameters required to convert the visible light frame to the infrared frame coordinate system. The transformation model includes translation, rotation, scaling, and affine transformation. By minimizing the error between the matching point pairs, the least square method can be used to solve the optimal transformation parameters. This step ensures the spatial consistency of the two images, so that the subsequent fusion processing can be operated based on the same spatial coordinates.

[0061] S14: Apply the transformation parameters to spatially align the visible light frame with the infrared frame to complete image registration.

[0062] In this embodiment, once the transformation parameters are obtained, they can be applied to the entire visible light frame so that all pixels are adjusted to corresponding positions according to the calculated transformation rules, thereby achieving spatial alignment of the two images. In order to improve the registration accuracy, a bilinear interpolation algorithm is introduced in the transformation process to ensure the quality of the transformed image. After the registration is completed, the visible light frame and the infrared frame are completely aligned in space, providing a solid foundation for subsequent steps such as brightness enhancement, fusion, noise reduction, and color reconstruction.

[0063] S2: Based on the multiple infrared frames, the corresponding multiple visible light frames are brightness enhanced, and the multiple visible light frames after brightness enhancement are synthesized by using an adaptive weighted fusion algorithm to generate a long exposure equivalent single frame image.

[0064] Specifically, step S2 includes the following steps:

[0065] S21: Based on the brightness distribution in the multiple infrared frames, identify the high brightness area and the low brightness area in the scene, and enhance the brightness of the low brightness area in the corresponding visible light frame according to the thermal radiation intensity provided by the multiple infrared frames.

[0066] In this embodiment, in the process of identifying high brightness areas and low brightness areas in the scene based on the brightness distribution in the multiple infrared frames, and performing brightness enhancement on the low brightness areas in the corresponding visible light frames according to the thermal radiation intensity provided by the multiple infrared frames, the infrared image can capture the thermal radiation emitted by the object due to its characteristics, so that even in the absence of a visible light source, the temperature distribution information of different objects in the scene can be obtained. This information is used to identify which areas are relatively bright (i.e., areas with higher heat) and which areas are relatively dim (i.e., areas with lower heat). Then, based on this thermal radiation intensity information, brightness enhancement processing is performed on the areas in the visible light frame corresponding to the low brightness (low temperature) in the infrared image.

[0067] S22: For each set of registered and brightness-enhanced visible light frames, calculate the local contrast and brightness value around each pixel.

[0068] In this embodiment, the work at this stage is to prepare for the next step of weight adjustment. By calculating the local contrast and brightness values ​​of each pixel and its neighborhood, information about the local features of the image can be obtained. The local contrast reflects the presence of image edges and other significant structures, while the brightness value represents the overall brightness of the image. These two parameters work together to determine the importance of each frame in the subsequent fusion process.

[0069] The specific calculation is to define a window around each pixel through the sliding window method, calculate the standard deviation of the pixel values ​​in the window to obtain the local contrast, and the average value to obtain the brightness value. Such local analysis helps to retain important features in the image, such as edges and textures, while avoiding excessive enhancement of noise.

[0070] S23: Dynamically adjust the weights between different frames according to the local contrast and brightness value of each pixel.

[0071] The weight adjustment here is a crucial step in the entire brightness enhancement and fusion process. It involves applying a preset dynamic weight function that takes into account the local contrast and brightness value of each pixel to determine the weight of the pixel's contribution to the final long-exposure equivalent single-frame image.

[0072] The weight adjustment is performed according to a preset dynamic weight function, wherein the dynamic weight function is used to determine the weight contributed by each pixel in the long exposure equivalent single frame image according to the local contrast and brightness value of each pixel. Specifically, the mathematical expression of the dynamic weight function is:

[0073]

[0074] Where f represents the frame index, N represents the number of visible light frames, and g f (C, L) represents a comprehensive function combining the local contrast C and the brightness value L.

[0075] Among them, the mathematical expression of the comprehensive function is:

[0076]

[0077] Among them, ∈ represents a small positive number, α represents a parameter for controlling the degree of influence of contrast, β represents a parameter for controlling the degree of influence of brightness value, and μ represents the desired average brightness value.

[0078] S24: Using the calculated weights, weighted averaging is performed on all visible light frames to generate a long exposure equivalent single frame image.

[0079] In this embodiment, by weighted averaging all visible light frames, the generated long exposure equivalent single frame image not only effectively improves the brightness, but also maintains good image quality, including detail preservation and noise suppression. This method effectively simulates the effect of long exposure, enhances the visibility and visual appeal of the image, and reduces the risk of motion blur because it is synthesized based on multiple frames of short exposure images.

[0080] S3: performing noise reduction processing on the long exposure equivalent single frame image and the multiple infrared frames.

[0081] Specifically, step S3 includes the following steps:

[0082] S31: applying a multi-scale transformation algorithm to decompose the long-exposure equivalent single-frame image and the multiple infrared frames respectively to obtain image components of different frequencies.

[0083] In this embodiment, multi-scale transformation is a technology that can decompose image signals into different frequency components, including but not limited to wavelet transformation, contourlet transformation and non-subsampled contourlet transformation (NSCT). These transformation methods can separate images into multiple frequency bands representing different levels of detail without losing image information. Low-frequency components mainly contain the overall structural information of the image, while high-frequency components carry more texture and edge details. Through multi-scale transformation, noise can be separated from the main features of the image because noise is often concentrated in high-frequency components. This process facilitates subsequent noise estimation and removal.

[0084] S32: performing noise estimation on each frequency component based on a statistical model, and adjusting the threshold of each component according to the estimation result to remove noise, wherein the statistical model is pre-trained by data containing noisy and noise-free image pairs.

[0085] In this embodiment, due to the complex characteristics of image noise in extremely low illumination environments, it is difficult for traditional fixed threshold denoising methods to achieve ideal results. Therefore, the present invention adopts a method based on a statistical model to dynamically determine the optimal threshold for each frequency component. This statistical model is obtained in advance through a large amount of data training containing noisy and noise-free image pairs, and it can accurately predict the noise level in each frequency component. For each frequency component, the statistical model calculates its noise distribution parameters, such as mean and variance. Then, the threshold is set according to these parameters, so that the part below the threshold is considered to be noise and suppressed, while the part above the threshold is considered to be a valid signal and retained. This method not only ensures the effective removal of noise, but also protects the original details of the image as much as possible.

[0086] S33: Recombining the frequency components that have been subjected to noise reduction processing through inverse multi-scale transformation to restore the noise-reduced long-exposure equivalent single-frame image and infrared frame.

[0087] In this embodiment, after the noise estimation and removal of each frequency component are completed, they need to be recombined to reconstruct the complete image. Inverse multi-scale transformation is the inverse process of multi-scale transformation, which can synthesize different frequency components back to the original size image according to the coefficients during decomposition.

[0088] In this process, the phase relationship between all components is ensured to be correct so that a natural and undistorted image can be restored. In addition, considering the possible boundary effects or inconsistencies in overlapping areas, appropriate smoothing techniques can be used in the synthesis stage to further improve the image quality. Ultimately, the long-exposure equivalent single-frame images and infrared frames that have been denoised will have lower noise levels, higher signal-to-noise ratios, and clearer visual effects, providing better quality input data for the subsequent color reconstruction step.

[0089] S4: Inputting the long-exposure equivalent single-frame image and the multiple infrared frames after noise reduction processing into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image, wherein the color derivation model is a deep learning model trained with sample data containing visible light images and infrared image pairs in extremely low illumination scenes.

[0090] Specifically, step S4 includes the following steps:

[0091] S41: During the color reconstruction process, the color derivation model receives the long exposure equivalent single frame image and infrared frame after noise reduction as input, and extracts the features of the image through multi-layer convolution, activation, pooling and fully connected layer operations.

[0092] During the color reconstruction process, the color derivation model receives the long exposure equivalent single frame image and infrared frame after noise reduction as input, and extracts the image features through multi-layer convolution, activation, pooling and fully connected layer operations. Deep learning models, especially convolutional neural networks (CNNs), have demonstrated powerful performance in the field of image processing, especially in feature extraction. These models can automatically learn multi-level abstract feature representations from data, which is particularly important for ultra-high-definition long-distance full-color imaging under extremely low illumination. In this embodiment, the color derivation model is a specifically designed and trained deep learning model, the purpose of which is to learn how to complete the color information of the infrared image from the visible light and infrared image pairs.

[0093] First, the color inference model receives a long-exposure equivalent single-frame image and multiple infrared frames after noise reduction. These input images are the results of the previous steps S1 to S3, and have been image registration, brightness enhancement, fusion, and noise reduction, providing a high-quality basis for color reconstruction. The input layer of the model directly receives the data of these images, usually organized into a matrix or tensor structure in the form of pixel values. For each input image, it is fed to the initial layers of the model, which is a series of convolutional layers.

[0094] The filters (or kernels) in the convolutional layer slide over the input image and perform local weighted sum operations to detect various features in the image, such as edges, textures, and other patterns. The output of each layer is passed to the next layer, and as the number of layers increases, the learned features become more complex and abstract. After the convolutional layer, activation functions (including but not limited to ReLU) are used to introduce nonlinearity so that the model can capture more complex patterns; there are also pooling layers to reduce the spatial dimension of the features, reduce the amount of calculations, and control overfitting.

[0095] S42: Mapping the extracted features to the color space using the encoding and decoding structure within the color derivation model to reconstruct a color information matrix that matches the original scene.

[0096] In this embodiment, at this stage, the encoding part of the model has completed the compression and abstract representation of the input image features, while the task of the decoding part is to gradually restore these high-level features back to the original color space and finally generate a color information matrix.

[0097] The decoding process generally starts from a lower-dimensional feature space and gradually increases the resolution until it reaches the same size as the input image. This process involves a deconvolution operation, which is equivalent to the inverse process of convolution and is used to enlarge the feature map. In some cases, the decoder also combines jump connections to reintroduce fine-grained information preserved during the encoding process to help recover more details.

[0098] In addition, the last layer of the decoder usually has an output layer, which is responsible for converting the feature map into specific values ​​in the color space. In this process, the model also uses a softmax activation function to ensure that the output value falls within a reasonable color range.

[0099] S5: Based on the color information matrix, color mapping and coloring are performed on the long exposure equivalent single frame image to generate a final full-color image.

[0100] In the process of color mapping and coloring, we first need to use the color distribution information in the color information matrix to convert the grayscale value of the long exposure equivalent single frame image into the corresponding RGB color value through a lookup table. The work at this stage is based on the color information matrix obtained in the previous step. The color information matrix contains the color information extracted and reconstructed from the infrared image and the visible light image, and it provides the color attributes that each pixel should have. To achieve this, a lookup table (LUT) is usually created. This lookup table maps the grayscale value of each pixel in the long exposure equivalent single frame image to the corresponding RGB color value according to the color distribution in the color information matrix.

[0101] Specifically, a lookup table is a pre-calculated data structure that establishes a one-to-one correspondence between the input grayscale value and the output color value. In this process, for each pixel in the long-exposure equivalent single-frame image, the corresponding RGB color value is looked up based on its grayscale value. The advantage of the lookup table is that it can quickly complete the color conversion while ensuring the consistency and accuracy of the color conversion. In this way, the original grayscale image is given color, thus forming a preliminary color image.

[0102] Next, for each pixel, the color attribute of the corresponding pixel is adjusted by applying the Lab color space model in combination with its original brightness information and the color information obtained from the color information matrix. The Lab color space is a device-independent color model that divides color into three components: L represents brightness, and a and b represent the change from green to red and from blue to yellow, respectively. The advantage of using the Lab color space is that brightness and chromaticity information can be processed independently, which is very important for color adjustment, especially when changing the color while maintaining the original brightness characteristics.

[0103] In this step, for each pixel, it will be converted from the RGB color space to the Lab color space. Then, according to the color information provided by the color information matrix, the a and b components are adjusted to ensure the accuracy and naturalness of the final color. At the same time, the L component (brightness) is retained or fine-tuned according to the brightness information of the original image to ensure that the overall contrast and brightness characteristics of the image do not change significantly. In this way, the color restoration and authenticity can be effectively improved without affecting the brightness of the image. In addition, the use of the Lab color space also helps to reduce color distortion and improve the smoothness of color transitions, making the generated color images more natural and realistic.

[0104] Finally, the adjusted color properties are color corrected to obtain the final full-color image. The purpose of color correction is to ensure that the generated color image is not only visually pleasing, but also conforms to the reality of the physical world. This step involves gamma correction, white balance adjustment, color saturation enhancement, etc. Gamma correction is used to compensate for the nonlinear response characteristics of the display so that the image looks closer to the effect perceived by the human eye; white balance adjustment is to correct the color deviation caused by different color temperatures of the light source to ensure that white objects appear as true white in the image; and color saturation enhancement can make the colors in the image more vivid and lively, but care should also be taken to avoid color distortion caused by over-enhancement.

[0105] Embodiment 2

[0106] The present invention also provides an ultra-high-definition long-distance full-color imaging module under extremely low illumination, referring to Figure 2As shown, it includes a visible light camera, an infrared sensor and an image processing unit, wherein the image processing unit is used to perform the following steps:

[0107] Short exposure shooting is performed for the same scene, multiple visible light frames and multiple infrared frames are acquired synchronously and image registration is performed.

[0108] The corresponding multiple visible light frames are brightness enhanced based on the multiple infrared frames, and the multiple visible light frames after brightness enhancement are synthesized using an adaptive weighted fusion algorithm to generate a long exposure equivalent single frame image.

[0109] Noise reduction processing is performed on the long-exposure equivalent single-frame image and the multiple infrared frames.

[0110] The long-exposure equivalent single-frame image and the multiple infrared frames after noise reduction are input into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image, wherein the color derivation model is a deep learning model trained with sample data containing visible light images and infrared image pairs in extremely low illumination scenes.

[0111] Based on the color information matrix, the long-exposure equivalent single-frame image is color mapped and colored to generate a final full-color image.

[0112] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0114] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A method for ultra-high-definition, long-distance, full-color imaging under extremely low illumination, characterized in that: The method comprises the following steps: Perform short exposure shooting for the same scene, synchronously acquire multiple visible light frames and multiple infrared frames and perform image registration, wherein the multiple visible light frames are captured by a configured visible light camera, and the multiple infrared frames are captured by a configured infrared sensor; Based on the multiple infrared frames, brightness enhancement is performed on the corresponding multiple visible light frames, and the multiple visible light frames after brightness enhancement are synthesized by using an adaptive weighted fusion algorithm to generate a long exposure equivalent single frame image; Performing noise reduction processing on the long exposure equivalent single frame image and the multiple infrared frames; Inputting the long-exposure equivalent single-frame image and the multiple infrared frames after noise reduction processing into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image, wherein the color derivation model is a deep learning model trained with sample data including visible light images and infrared image pairs in extremely low illumination scenes; Based on the color information matrix, the long-exposure equivalent single-frame image is color mapped and colored to generate a final full-color image.

2. The method for ultra-high-definition, long-distance, full-color imaging under extremely low illumination according to claim 1, characterized in that: The image registration comprises: Extracting feature points from the multiple visible light frames and the multiple infrared frames; Matching the feature points in the visible light frame with the feature points in the infrared frame using a feature point matching algorithm; According to the matched feature points, the transformation parameters between the images are calculated; The transformation parameters are applied to spatially align the visible light frame with the infrared frame to complete image registration.

3. The method for ultra-high-definition, long-distance, full-color imaging under extremely low illumination according to claim 2, characterized in that: The method of enhancing the brightness of the corresponding multiple visible light frames based on the multiple infrared frames, and synthesizing the multiple visible light frames after the brightness enhancement by using an adaptive weighted fusion algorithm to generate a long exposure equivalent single frame image includes: Based on the brightness distribution in the multiple infrared frames, identify the high brightness area and the low brightness area in the scene, and enhance the brightness of the low brightness area in the corresponding visible light frame according to the thermal radiation intensity provided by the multiple infrared frames; For each set of registered and brightness-enhanced visible light frames, calculate the local contrast and brightness values ​​around each pixel; Dynamically adjust the weights between different frames based on the local contrast and brightness values ​​of each pixel; Using the calculated weights, all visible light frames are weighted averaged to generate a long-exposure equivalent single-frame image.

4. The method for ultra-high-definition, long-distance, full-color imaging under extremely low illumination according to claim 3, characterized in that: The weight adjustment is performed according to a preset dynamic weight function, wherein the dynamic weight function is used to determine the weight contributed by each pixel in the long exposure equivalent single frame image according to the local contrast and brightness value of each pixel. Specifically, the mathematical expression of the dynamic weight function is: Where f represents the frame index, N represents the number of visible light frames, and g f (C, L) represents the comprehensive function combining the local contrast C and the brightness value L; Among them, the mathematical expression of the comprehensive function is: Among them, ∈ represents a small positive number, α represents a parameter for controlling the degree of influence of contrast, β represents a parameter for controlling the degree of influence of brightness value, and μ represents the expected average brightness value.

5. The method for ultra-high-definition, long-distance, full-color imaging under extremely low illumination according to claim 4, characterized in that: The performing noise reduction processing on the long exposure equivalent single frame image and the multiple infrared frames comprises: Applying a multi-scale transformation algorithm to decompose the long-exposure equivalent single-frame image and the multiple infrared frames respectively to obtain image components of different frequencies; Performing noise estimation on each frequency component based on a statistical model, and adjusting the threshold of each component according to the estimation result to remove the noise, wherein the statistical model is pre-trained with data including noisy and noise-free image pairs; The frequency components that have been denoised are recombined through inverse multi-scale transformation to restore the denoised long-exposure equivalent single-frame image and infrared frame.

6. The method for ultra-high-definition, long-distance, full-color imaging under extremely low illumination according to claim 5, characterized in that: The step of inputting the long-exposure equivalent single-frame image and the plurality of infrared frames after the noise reduction process into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image includes: In the color reconstruction process, the color derivation model receives the long-exposure equivalent single-frame image and the infrared frame after noise reduction as input, and extracts the features of the image through the operations of multiple layers of convolution, activation, pooling and fully connected layers; The extracted features are mapped to the color space using the encoding and decoding structure within the color derivation model to reconstruct a color information matrix that matches the original scene.

7. The method for ultra-high-definition, long-distance, full-color imaging under extremely low illumination according to claim 6, characterized in that: The color mapping and coloring of the long-exposure equivalent single-frame image based on the color information matrix to generate a final full-color image includes: Using the color distribution information in the color information matrix, the grayscale value of the long-exposure equivalent single-frame image is converted into a corresponding RGB color value through a lookup table; For each pixel, combining its original brightness information and the color information obtained from the color information matrix, applying the Lab color space model to adjust the color attribute of the corresponding pixel; Perform color correction on the adjusted color properties to obtain the final full-color image.

8. An ultra-high-definition, long-distance, full-color imaging module under extremely low illumination, characterized in that: The system comprises a visible light camera, an infrared sensor and an image processing unit, wherein the image processing unit is used to perform the following steps: Take short-exposure photos of the same scene, acquire multiple visible light frames and multiple infrared frames simultaneously, and perform image registration; Based on the multiple infrared frames, brightness enhancement is performed on the corresponding multiple visible light frames, and the multiple visible light frames after brightness enhancement are synthesized by using an adaptive weighted fusion algorithm to generate a long exposure equivalent single frame image; Performing noise reduction processing on the long exposure equivalent single frame image and the multiple infrared frames; Inputting the long-exposure equivalent single-frame image and the multiple infrared frames after noise reduction processing into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image, wherein the color derivation model is a deep learning model trained with sample data including visible light images and infrared image pairs in extremely low illumination scenes; Based on the color information matrix, the long-exposure equivalent single-frame image is color mapped and colored to generate a final full-color image.

Images