Ultra-high-definition long-distance full-color imaging method and module under extremely low illuminance

Abstract

The present invention provides an ultra-high definition long-distance full-color imaging method and module under extremely low illuminance, which relates to the technical field of image processing. It includes performing short-exposure shooting on the same scene, synchronously obtaining multiple visible light frames and multiple infrared frames and performing image registration; enhancing the brightness of the corresponding multiple visible light frames based on the multiple infrared frames, and synthesizing the multiple brightness-enhanced visible light frames by using an adaptive weighted fusion algorithm to generate a long-exposure equivalent single-frame image; performing noise reduction processing on the long-exposure equivalent single-frame image and the multiple infrared frames; inputting the noise-reduced long-exposure equivalent single-frame image and the multiple infrared frames into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image; and performing color mapping and coloring on the long-exposure equivalent single-frame image based on the color information matrix to generate a final full-color image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to an ultra-high definition long-distance full-color imaging method and its module under extremely low illuminance. Background Art

[0002] In the prior art, in the case of insufficient light, traditional visible light imaging devices suffer from a serious decline in signal-to-noise ratio, resulting in severely damaged image quality, such as a large amount of noise, loss of details, and color distortion. To overcome this problem, some existing solutions include using high-sensitivity sensors, increasing artificial lighting, or adopting long exposure times to improve image brightness. However, these methods have their respective limitations: Although high-sensitivity sensors can improve sensitivity, they usually introduce more noise; increasing artificial lighting may change the natural state of the scene and is not feasible in some application scenarios (such as wildlife observation, military reconnaissance, etc.); long exposure times are prone to blurring of moving objects and are not suitable for shooting dynamic scenes.

[0003] Therefore, it is necessary to provide an ultra-high definition long-distance full-color imaging method and its module under extremely low illuminance to solve the above technical problems. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides an ultra-high definition long-distance full-color imaging method and its module under extremely low illuminance, which solves the problem that traditional cameras are difficult to capture clear and color-accurate images at night or under extremely low light conditions, and realizes high-quality color image reconstruction by combining visible light and infrared imaging technologies.

[0005] The present invention provides an ultra-high definition long-distance full-color imaging method under extremely low illuminance, and the method includes the following steps:

[0006] Performing short exposure shooting on the same scene, synchronously acquiring a plurality of visible light frames and a plurality of infrared frames and performing image registration, wherein the plurality of visible light frames are captured by a configured visible light camera, and the plurality of infrared frames are captured by a configured infrared sensor;

[0007] Enhancing the brightness of the corresponding plurality of visible light frames based on the plurality of infrared frames, and synthesizing the plurality of visible light frames after brightness enhancement by using an adaptive weighted fusion algorithm to generate a long exposure equivalent single frame image;

[0008] Performing noise reduction processing on the long exposure equivalent single frame image and the plurality of infrared frames;

[0009] Input the denoised long-exposure equivalent single-frame image and the multiple infrared frames into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image, where the color derivation model is a deep learning model trained with sample data including visible light images and infrared image pairs in extremely low illumination scenarios;

[0010] Based on the color information matrix, perform color mapping and coloring on the long-exposure equivalent single-frame image to generate the final full-color image.

[0011] Preferably, the image registration includes:

[0012] Extract feature points from the multiple visible light frames and the multiple infrared frames;

[0013] Use a feature point matching algorithm to match the feature points in the visible light frames with the feature points in the infrared frames;

[0014] According to the matched feature points, calculate the transformation parameters between the images;

[0015] Apply the transformation parameters to spatially align the visible light frames and the infrared frames to complete image registration.

[0016] Preferably, the method of enhancing the brightness of the corresponding multiple visible light frames based on the multiple infrared frames and synthesizing the brightness-enhanced multiple visible light frames using an adaptive weighted fusion algorithm to generate a long-exposure equivalent single-frame image includes:

[0017] Based on the brightness distribution in the multiple infrared frames, identify the high-brightness regions and low-brightness regions in the scene, and enhance the brightness of the low-brightness regions in the corresponding visible light frames according to the thermal radiation intensity provided by the multiple infrared frames;

[0018] For each group of registered and brightness-enhanced visible light frames, calculate the local contrast and brightness value around each pixel point;

[0019] According to the local contrast and brightness value of each pixel point, dynamically adjust the weights between different frames;

[0020] Use the calculated weights to perform weighted averaging on all visible light frames to generate a long-exposure equivalent single-frame image.

[0021] Preferably, the adjustment of the weights is completed according to a preset dynamic weight function, where the dynamic weight function is used to determine the weight contributed by each pixel point in the long-exposure equivalent single-frame image according to its local contrast and brightness value. Specifically, the mathematical expression of the dynamic weight function is:

[0022]

[0023] Among them, represents the index of the frame, represents the number of visible light frames, represents the combined local contrast and the luminance value of the combined function;

[0024] Among them, the mathematical expression of the combined function is:

[0025]

[0026] Among them, represents a small positive number, represents a parameter for controlling the influence degree of the contrast, represents a parameter for controlling the influence degree of the luminance value, represents the desired average luminance value.

[0027] Preferably, the denoising process for the long-exposure equivalent single-frame image and the multiple infrared frames includes:

[0028] Applying a multi-scale transform algorithm to decompose the long-exposure equivalent single-frame image and the multiple infrared frames respectively to obtain image components of different frequencies;

[0029] Based on a statistical model, estimating the noise of each frequency component and adjusting the threshold of each component according to the estimation result to remove the noise, and the statistical model is pre-trained with data including pairs of noisy and noise-free images;

[0030] Recombining the denoised frequency components through an inverse multi-scale transform to restore the denoised long-exposure equivalent single-frame image and infrared frames.

[0031] Preferably, inputting the denoised long-exposure equivalent single-frame image and the multiple infrared frames into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image includes:

[0032] During the color reconstruction process, the color derivation model receives the denoised long-exposure equivalent single-frame image and infrared frames as inputs and extracts the features of the images through operations of multiple convolutional, activation, pooling, and fully connected layers;

[0033] Using the encoding-decoding structure inside the color derivation model, mapping the extracted features to the color space to reconstruct a color information matrix that matches the original scene.

[0034] Preferably, performing color mapping and coloring on the long-exposure equivalent single-frame image based on the color information matrix to generate a final full-color image includes:

[0035] Using the color distribution information in the color information matrix, converting the grayscale value of the long-exposure equivalent single-frame image into the corresponding RGB color value through a look-up table;

[0036] For each pixel point, combining its original brightness information and the color information obtained from the color information matrix, and applying the Lab color space model to adjust the color attributes of the corresponding pixel;

[0037] Performing color correction on the adjusted color attributes to obtain the final full-color image.

[0038] The present invention also provides an ultra-high-definition long-distance full-color imaging module under extremely low illuminance, including a visible light camera, an infrared sensor, and an image processing unit, wherein the image processing unit is used to perform the following steps:

[0039] Performing short-exposure shooting for the same scene, synchronously acquiring multiple visible light frames and multiple infrared frames and performing image registration;

[0040] Based on the multiple infrared frames, enhancing the brightness of the corresponding multiple visible light frames, and synthesizing the brightness-enhanced multiple visible light frames by using an adaptive weighted fusion algorithm to generate a long-exposure equivalent single-frame image;

[0041] Performing noise reduction processing on the long-exposure equivalent single-frame image and the multiple infrared frames;

[0042] Inputting the long-exposure equivalent single-frame image and the multiple infrared frames after noise reduction processing into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image, wherein the color derivation model is a deep learning model trained with sample data including visible light images and infrared image pairs under extremely low illuminance scenarios;

[0043] Based on the color information matrix, performing color mapping and coloring on the long-exposure equivalent single-frame image to generate a final full-color image.

[0044] Compared with the related technology, an ultra-high-definition long-distance full-color imaging method and its module provided by the present invention have the following beneficial effects:

[0045] The present invention lays a foundation for subsequent processing by synchronously acquiring multiple visible light frames and infrared frames of the same scene and performing precise image registration to ensure spatial alignment between the two modal images. Then, the brightness of the visible light frames is enhanced using the brightness distribution information in the infrared frames, especially for low-brightness regions. Subsequently, an adaptive weighted fusion algorithm is employed to dynamically adjust the weights between different frames according to the local contrast and brightness value of each pixel point, synthesizing a single-frame image equivalent to the long-exposure effect to simulate the high-brightness image that can be obtained by long-time exposure while avoiding the problem of motion blur.

[0046] To further improve the image quality, the method also includes a noise reduction processing step, applying a multi-scale transform algorithm and a noise estimation technique based on a statistical model to effectively remove the noise in the image while retaining important details. Subsequently, the denoised long-exposure equivalent single-frame image and the infrared frame are input into a pre-trained color derivation model, which is a deep learning model trained with sample data including visible light and infrared image pairs in extremely low illumination scenarios for color reconstruction, generating a color information matrix that matches the original scene. Finally, based on this color information matrix and in combination with the Lab color space model, color mapping and coloring are performed on the long-exposure equivalent single-frame image, including the conversion from grayscale values to RGB color values, color attribute adjustment, and color correction, ultimately generating a full-color image with accurate colors and rich details. This method can not only provide high-quality color images under extreme lighting conditions but also has good adaptability and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flowchart of an ultra-high-definition long-distance full-color imaging method under extremely low illumination provided by the present invention;

[0048] Figure 2 It is a schematic structural diagram of an ultra-high-definition long-distance full-color imaging module provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] The present invention will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention and not to limit the present invention. Additionally, it should be noted that for the sake of description, only parts related to the present invention are shown in the drawings rather than all structures. Furthermore, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0050] It should also be noted that, for the convenience of description, only the parts related to the present invention rather than all the content are shown in the drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operations are completed, but there can also be additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, and so on.

[0051] Embodiment 1

[0052] The present invention provides a method for ultra-high-definition long-distance full-color imaging under extremely low illuminance. Referring to Figure 1 as shown, the method includes the following steps:

[0053] S1: Perform short-exposure shooting for the same scene, synchronously obtain a plurality of visible light frames and a plurality of infrared frames and perform image registration, wherein the plurality of visible light frames are captured by a configured visible light camera, and the plurality of infrared frames are captured by a configured infrared sensor.

[0054] Specifically, the image registration includes the following steps:

[0055] S11: Extract feature points from the plurality of visible light frames and the plurality of infrared frames.

[0056] In this embodiment, under extremely low illuminance conditions, visible light images usually have low contrast and high noise, while infrared images provide thermal radiation information but lack color information. To ensure accurate alignment of the two modal images, stable feature points must be extracted from each image. These feature points can be corner points, edges, or other significant structures, which have good robustness under different lighting conditions. The feature point detection algorithms include but are not limited to Harris corner detection, SIFT (Scale-Invariant Feature Transform), and SURF (Speeded-Up Robust Features). By using the feature point detection algorithms, a set of feature points can be extracted from the visible light frames and the infrared frames respectively, providing a basis for subsequent matching and registration.

[0057] S12: Use a feature point matching algorithm to match the feature points in the visible light frames with the feature points in the infrared frames.

[0058] In this embodiment, due to the significant differences between visible light and infrared images, direct matching may lead to false matches. Therefore, robust matching algorithms, including but not limited to RANSAC (Random Sample Consensus) and FLANN (Fast Library for Approximate Nearest Neighbors), need to be adopted to filter out reliable matching pairs. The matching algorithm can find the matching point pairs that best conform to the geometric constraints in the presence of a large number of outliers. Through the matching process, the relative position relationship between the two images can be preliminarily determined, preparing for further calculation of the transformation parameters.

[0059] S13: Calculate the transformation parameters between the images based on the matched feature points.

[0060] In this embodiment, after completing the feature point matching, the next step is to calculate the geometric transformation parameters required to convert the visible light frame to the infrared frame coordinate system. The transformation models include translation, rotation, scaling, and affine transformation, etc. By minimizing the error between the matched point pairs, the least squares method can be used to solve the optimal transformation parameters. This step ensures the spatial consistency of the two images, enabling subsequent fusion processing to be performed based on the same spatial coordinates.

[0061] S14: Apply the transformation parameters to spatially align the visible light frame and the infrared frame to complete image registration.

[0062] In this embodiment, once the transformation parameters are obtained, they can be applied to the entire visible light frame, so that all pixel points are adjusted to the corresponding positions according to the calculated transformation rules, thereby achieving the spatial alignment of the two images. To improve the registration accuracy, a bilinear interpolation algorithm is introduced during the transformation process to ensure the quality of the transformed image. After completing the registration, the visible light frame and the infrared frame are completely spatially aligned, providing a solid foundation for subsequent steps such as brightness enhancement, fusion, noise reduction, and color reconstruction.

[0063] S2: Based on the multiple infrared frames, perform brightness enhancement on the corresponding multiple visible light frames, and use the adaptive weighted fusion algorithm to synthesize the brightness-enhanced multiple visible light frames to generate a long-exposure equivalent single-frame image.

[0064] Specifically, step S2 includes the following steps:

[0065] S21: Based on the brightness distribution in the multiple infrared frames, identify the high-brightness regions and low-brightness regions in the scene, and enhance the brightness of the low-brightness regions in the corresponding visible light frames according to the thermal radiation intensity provided by the multiple infrared frames.

[0066] In this embodiment, during the process of identifying the high-brightness regions and low-brightness regions in the scene based on the brightness distribution in the multiple infrared frames and enhancing the brightness of the low-brightness regions in the corresponding visible-light frames according to the thermal radiation intensity provided by the multiple infrared frames, the infrared image can capture the thermal radiation emitted by objects due to its characteristics. This enables obtaining the temperature distribution information of different objects in the scene even in the absence of visible light sources. This information is used to identify which regions are relatively bright (i.e., regions with higher heat) and which regions are relatively dim (i.e., regions with lower heat). Then, based on this thermal radiation intensity information, brightness enhancement processing is performed on the regions in the visible-light frame corresponding to the low brightness (low temperature) in the infrared image.

[0067] S22: For each group of registered and brightness-enhanced visible-light frames, calculate the local contrast and brightness value around each pixel.

[0068] In this embodiment, the work in this stage is to prepare for the next weight adjustment. By calculating the local contrast and brightness value of each pixel and its neighborhood, information about the local features of the image can be obtained. The local contrast reflects the presence of image edges and other significant structures, while the brightness value represents the overall light and dark degree of the image. These two parameters work together to determine the importance of each frame in the subsequent fusion process.

[0069] The specific calculation is through the sliding window method. Define a window around each pixel, calculate the standard deviation of the pixel values within the window to obtain the local contrast, and the average value to obtain the brightness value. Such local analysis helps to retain important features in the image, such as edges and textures, while avoiding over-enhancing noise.

[0070] S23: Dynamically adjust the weights between different frames according to the local contrast and brightness value of each pixel.

[0071] The weight adjustment here is a crucial step in the entire brightness enhancement and fusion process. It involves applying a preset dynamic weight function, which takes into account the local contrast and brightness value of each pixel to determine the weight contributed by this pixel in the finally generated long-exposure equivalent single-frame image.

[0072] Among them, the adjustment of the weight is completed according to a preset dynamic weight function, where the dynamic weight function is used to determine the weight contributed by each pixel in the long-exposure equivalent single-frame image according to the local contrast and brightness value of each pixel. Specifically, the mathematical expression of the dynamic weight function is:

[0073]

[0074] Among them, Indicates the index of the frame, Indicates the number of visible light frames, Indicates the combined local contrast and the luminance value of the comprehensive function.

[0075] Among them, the mathematical expression of the comprehensive function is:

[0076]

[0077] Among them, represents a small positive number, represents a parameter that controls the influence degree of contrast, represents a parameter that controls the influence degree of luminance value, represents the desired average luminance value.

[0078] S24: Use the calculated weights to perform weighted averaging on all visible light frames to generate a long-exposure equivalent single-frame image.

[0079] In this embodiment, by performing weighted averaging on all visible light frames, the generated long-exposure equivalent single-frame image will not only have effectively improved luminance, but also maintain good image quality, including detail retention and noise suppression. This method effectively simulates the effect of long-time exposure, enhances the visibility and visual attractiveness of the image, and at the same time reduces the risk of motion blur because it is based on the synthesis of multiple short-exposure images.

[0080] S3: Perform noise reduction processing on the long-exposure equivalent single-frame image and the multiple infrared frames.

[0081] Specifically, step S3 includes the following steps:

[0082] S31: Apply a multi-scale transform algorithm to decompose the long-exposure equivalent single-frame image and the multiple infrared frames respectively to obtain image components of different frequencies.

[0083] In this embodiment, multi-scale transform is a technique that can decompose an image signal into different frequency components, including but not limited to wavelet transform, Contourlet transform, and non-subsampled contourlet transform (NSCT). These transform methods can separate an image into multiple frequency bands representing different levels of detail without losing image information. The low-frequency components mainly contain the overall structural information of the image, while the high-frequency components carry more texture and edge details. By multi-scale transform, noise can be separated from the main features of the image because noise tends to be concentrated in the high-frequency components. This process facilitates subsequent noise estimation and removal.

[0084] S32: Estimate the noise for each frequency component based on a statistical model, and adjust the threshold of each component according to the estimation result to remove the noise. The statistical model is pre-trained with data including pairs of noisy and noise-free images.

[0085] In this embodiment, due to the complex noise characteristics of images in extremely low illumination environments, traditional fixed-threshold denoising methods are difficult to achieve ideal results. Therefore, the present invention adopts a method based on a statistical model to dynamically determine the optimal threshold for each frequency component. This statistical model is pre-trained with a large amount of data including pairs of noisy and noise-free images, and it can accurately predict the noise level in each frequency component. For each frequency component, the statistical model calculates its noise distribution parameters, such as the mean and variance. Then, according to these parameters, a threshold is set so that the part below the threshold is considered noise and is suppressed, while the part above the threshold is considered valid signal and is retained. This method not only ensures the effective removal of noise but also protects the original details of the image as much as possible.

[0086] S33: Recombine the denoised frequency components through inverse multi-scale transformation to restore the denoised long-exposure equivalent single-frame image and infrared frame.

[0087] In this embodiment, after the noise estimation and removal of each frequency component are completed, they need to be recombined to reconstruct the complete image. Inverse multi-scale transformation is the inverse process of multi-scale transformation, which can synthesize different frequency components back into an image of the original size according to the coefficients during decomposition.

[0088] During this process, ensure that the phase relationship between all components is correct to restore a natural and distortion-free image. In addition, considering possible boundary effects or inconsistencies in overlapping regions, appropriate smoothing techniques can be adopted in the synthesis stage to further improve the image quality. Finally, the denoised long-exposure equivalent single-frame image and infrared frame will have a lower noise level, a higher signal-to-noise ratio, and a clearer visual effect, providing higher-quality input data for the subsequent color reconstruction step.

[0089] S4: Input the denoised long-exposure equivalent single-frame image and the multiple infrared frames into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image, where the color derivation model is a deep learning model trained with sample data including pairs of visible light images and infrared images in extremely low illumination scenarios.

[0090] Specifically, step S4 includes the following steps:

[0091] S41: During the color reconstruction process, the color derivation model receives the denoised long-exposure equivalent single-frame image and the infrared frame as inputs, and extracts the features of the images through operations of multiple convolutional, activation, pooling, and fully connected layers.

[0092] During the color reconstruction process, the color derivation model receives the denoised long-exposure equivalent single-frame image and the infrared frame as inputs, and extracts the features of the images through operations of multiple convolutional, activation, pooling, and fully connected layers. Deep learning models, especially convolutional neural networks (CNNs), have demonstrated powerful performance in the field of image processing, especially in feature extraction. These models can automatically learn multi-level abstract feature representations from data, which is particularly important for ultra-high-definition long-distance full-color imaging under extremely low illuminance. In this embodiment, the color derivation model is a specifically designed and trained deep learning model, whose purpose is to learn how to complete the color information of the infrared image from visible light and infrared image pairs.

[0093] First of all, the color derivation model receives the denoised long-exposure equivalent single-frame image and multiple infrared frames. These input images are the results of the previous steps S1 to S3, and have undergone image registration, brightness enhancement, fusion, and denoising processing, providing a high-quality basis for color reconstruction. The input layer of the model directly receives the data of these images, usually organized in the form of pixel values into a matrix or tensor structure. For each input image, it is fed into the initial several layers of the model, namely a series of convolutional layers.

[0094] The filters (or kernels) in the convolutional layer slide on the input image, performing local weighted summation operations to detect various features in the image, such as edges, textures, and other patterns. The output of each layer is passed to the next layer, and as the number of layers increases, the learned features become more complex and abstract. After the convolutional layer, activation functions (including but not limited to ReLU) are used to introduce non-linearity, enabling the model to capture more complex patterns; and pooling layers are used to reduce the spatial dimension of the features, reducing the computational amount and controlling overfitting.

[0095] S42: Using the encoding-decoding structure inside the color derivation model, map the extracted features to the color space to reconstruct a color information matrix that matches the original scene.

[0096] In this embodiment, at this stage, the encoding part of the model has completed the compression and abstract representation of the features of the input image, and the task of the decoding part is to gradually restore these high-level features back to the original color space, and finally generate a color information matrix.

[0097] The decoding process generally starts from a lower-dimensional feature space and gradually increases the resolution until it reaches the same size as the input image. This process involves deconvolution operations, which are the inverse of convolution and are used to enlarge the feature maps. In some cases, the decoder also incorporates skip connections to reintroduce the fine-grained information retained during the encoding process to help recover more details.

[0098] In addition, the last layer of the decoder usually has an output layer that is responsible for converting the feature map into specific values in the color space. During this process, the model also uses the softmax activation function to ensure that the output values fall within a reasonable color range.

[0099] S5: Based on the color information matrix, perform color mapping and coloring on the long-exposure equivalent single-frame image to generate the final full-color image.

[0100] During the color mapping and coloring process, first, it is necessary to utilize the color distribution information in the color information matrix to convert the grayscale values of the long-exposure equivalent single-frame image into corresponding RGB color values through a lookup table. The work in this stage is based on the color information matrix obtained in the previous steps. The color information matrix contains the color information extracted and reconstructed from the infrared image and the visible light image, and it provides the color attributes that each pixel point should possess. To achieve this, a lookup table (LUT) is usually created, and this lookup table maps the grayscale value of each pixel in the long-exposure equivalent single-frame image to the corresponding RGB color value according to the color distribution in the color information matrix.

[0101] Specifically, the lookup table is a pre-computed data structure that establishes a one-to-one correspondence between the input grayscale values and the output color values. During this process, for each pixel in the long-exposure equivalent single-frame image, the corresponding RGB color value is looked up according to its grayscale value. The advantage of the lookup table is that it can quickly complete the color conversion while ensuring the consistency and accuracy of the color conversion. In this way, the original grayscale image is given color, thus forming a preliminary color image.

[0102] Next, for each pixel point, combine its original brightness information and the color information obtained from the color information matrix, and apply the Lab color space model to adjust the color attributes of the corresponding pixel. The Lab color space is a device-independent color model that divides colors into three components: L represents brightness, and a and b represent the changes from green to red and from blue to yellow, respectively. The advantage of using the Lab color space is that it can independently process brightness and chromaticity information, which is very important for color adjustment, especially when changing colors while maintaining the original brightness characteristics.

[0103] In this step, for each pixel, it will be converted from the RGB color space to the Lab color space. Then, according to the color information provided by the color information matrix, the a and b components are adjusted to ensure the accuracy and naturalness of the final color. At the same time, the L component (luminance) is retained or slightly adjusted according to the luminance information of the original image to ensure that the overall contrast and luminance characteristics of the image do not change significantly. In this way, without affecting the image luminance, the color restoration and authenticity can be effectively improved. In addition, the use of the Lab color space helps to reduce color distortion and improve the smoothness of color transition, making the generated color image more natural and realistic.

[0104] Finally, color correction is performed on the adjusted color attributes to obtain the final full-color image. The purpose of color correction is to ensure that the generated color image is not only visually satisfactory but also conforms to the real situation in the physical world. This step involves gamma correction, white balance adjustment, color saturation enhancement, etc. Gamma correction is used to compensate for the non-linear response characteristics of the display, making the image look closer to the effect perceived by the human eye; white balance adjustment is to correct the color deviation caused by different color temperatures of the light source to ensure that white objects appear as true white in the image; and color saturation enhancement can make the colors in the image more vivid and lively, but attention should also be paid to avoiding color distortion caused by excessive enhancement.

[0105] Embodiment 2

[0106] The present invention also provides an ultra-high-definition long-distance full-color imaging module under extremely low illuminance. Referring to Figure 2 as shown, it includes a visible light camera, an infrared sensor, and an image processing unit, wherein the image processing unit is used to perform the following steps:

[0107] Perform short-exposure shooting for the same scene, synchronously obtain multiple visible light frames and multiple infrared frames and perform image registration.

[0108] Based on the multiple infrared frames, perform luminance enhancement on the corresponding multiple visible light frames, and use the adaptive weighted fusion algorithm to synthesize the luminance-enhanced multiple visible light frames to generate a long-exposure equivalent single-frame image.

[0109] Perform noise reduction processing on the long-exposure equivalent single-frame image and the multiple infrared frames.

[0110] Input the noise-reduced long-exposure equivalent single-frame image and the multiple infrared frames into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image, wherein the color derivation model is a deep learning model trained with sample data including visible light images and infrared image pairs under extremely low illuminance scenarios.

[0111] Based on the color information matrix, perform color mapping and coloring on the long-exposure equivalent single-frame image to generate a final full-color image.

[0112] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 one or more flows and / or blocks Figure 1 or multiple blocks.

[0113] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and this program can be stored in a computer-readable storage medium. The storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically-erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc memories, magnetic disk memories, tape memories, or any other computer-readable medium capable of carrying or storing data.

[0114] It should also be noted that the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, commodity or device comprising the element.

Claims

1. A method for ultra-high-definition long-distance full-color imaging under extremely low illuminance, characterized in that, The method includes the following steps: Perform short-exposure shooting for the same scene, synchronously obtain a plurality of visible light frames and a plurality of infrared frames and perform image registration, wherein the plurality of visible light frames are captured by a configured visible light camera, and the plurality of infrared frames are captured by a configured infrared sensor; Enhance the brightness of the corresponding plurality of visible light frames based on the plurality of infrared frames, and synthesize the plurality of visible light frames after brightness enhancement by using an adaptive weighted fusion algorithm to generate a long-exposure equivalent single-frame image, specifically including: Based on the brightness distribution in the plurality of infrared frames, identify the high-brightness areas and low-brightness areas in the scene, and enhance the brightness of the low-brightness areas in the corresponding visible light frames according to the thermal radiation intensity provided by the plurality of infrared frames; For each group of registered and brightness-enhanced visible light frames, calculate the local contrast and brightness value around each pixel; Dynamically adjust the weights between different frames according to the local contrast and brightness value of each pixel, wherein the adjustment of the weights is completed according to a preset dynamic weight function, and the dynamic weight function is used to determine the weight contributed by each pixel in the long-exposure equivalent single-frame image according to the local contrast and brightness value of each pixel. Specifically, the mathematical expression of the dynamic weight function is: Among them, represents the index of the frame, represents the number of visible light frames, represents the combined local contrast and brightness value of the comprehensive function; Wherein, the mathematical expression of the comprehensive function is: wherein, represents a small positive number, represents a parameter for controlling the influence degree of contrast, represents a parameter for controlling the influence degree of brightness value, represents the desired average brightness value; Use the calculated weights to perform weighted averaging on all visible light frames to generate a long-exposure equivalent single-frame image; Perform noise reduction processing on the long-exposure equivalent single-frame image and the plurality of infrared frames; Input the long-exposure equivalent single-frame image and the plurality of infrared frames after noise reduction processing into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image, wherein the color derivation model is a deep learning model trained with sample data including visible light images and infrared image pairs under extremely low illumination scenarios; Based on the color information matrix, perform color mapping and coloring on the long-exposure equivalent single-frame image to generate a final full-color image.

2. The ultra-high-definition long-distance full-color imaging method under extremely low illuminance according to claim 1, characterized in that, The image registration includes: Extract feature points in the plurality of visible light frames and the plurality of infrared frames; Use a feature point matching algorithm to match the feature points in the visible light frames with the feature points in the infrared frames; Calculate the transformation parameters between the images according to the matched feature points; Apply the transformation parameters to spatially align the visible light frames and the infrared frames to complete image registration.

3. The ultra-high definition long-distance full-color imaging method under extremely low illumination according to claim 2, wherein The noise reduction processing on the long-exposure equivalent single-frame image and the plurality of infrared frames includes: Apply a multi-scale transform algorithm to decompose the long-exposure equivalent single-frame image and the plurality of infrared frames respectively to obtain image components of different frequencies; Perform noise estimation on each frequency component based on a statistical model, and adjust the threshold of each component according to the estimation result to remove noise. The statistical model is pre-trained with data including pairs of noisy and noise-free images; Recombine the frequency components after noise reduction processing through inverse multi-scale transform to restore the long-exposure equivalent single-frame image and infrared frames after noise reduction.

4. The ultra-high definition long-distance full-color imaging method under extremely low illuminance according to claim 3, characterized in that, Feeding the denoised long-exposure equivalent single-frame image and the multiple infrared frames into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image, including: During the color reconstruction process, the color derivation model receives the denoised long-exposure equivalent single-frame image and infrared frames as inputs, and extracts features of the images through operations of multiple convolutional layers, activation layers, pooling layers, and fully connected layers; Using the encoding-decoding structure inside the color derivation model, the extracted features are mapped to the color space to reconstruct a color information matrix that matches the original scene.

5. A method for ultra-high-definition long-distance full-color imaging under extremely low illuminance according to claim 4, characterized in that, Based on the color information matrix, performing color mapping and coloring on the long-exposure equivalent single-frame image to generate a final full-color image, including: Using the color distribution information in the color information matrix, converting the grayscale values of the long-exposure equivalent single-frame image into corresponding RGB color values through a look-up table; For each pixel, combining its original brightness information and the color information obtained from the color information matrix, and applying the Lab color space model to adjust the color attributes of the corresponding pixel; Performing color correction on the adjusted color attributes to obtain the final full-color image.

6. An ultra-high-definition long-distance full-color imaging module under extremely low illuminance, which is used to execute an ultra-high-definition long-distance full-color imaging method under extremely low illuminance as described in any one of claims 1 to 5, and is characterized in that, Including a visible light camera, an infrared sensor, and an image processing unit, where the image processing unit is used to perform the following steps: Performing short-exposure shooting for the same scene, synchronously acquiring multiple visible light frames and multiple infrared frames and performing image registration; Enhancing the brightness of the corresponding multiple visible light frames based on the multiple infrared frames, and synthesizing the brightness-enhanced multiple visible light frames using an adaptive weighted fusion algorithm to generate a long-exposure equivalent single-frame image; Performing denoising processing on the long-exposure equivalent single-frame image and the multiple infrared frames; Feeding the denoised long-exposure equivalent single-frame image and the multiple infrared frames into a pre-trained color derivation model for color reconstruction to generate a color information matrix for constructing a full-color image, where the color derivation model is a deep learning model trained with sample data including pairs of visible light images and infrared images in extremely low-light scenarios; Based on the color information matrix, performing color mapping and coloring on the long-exposure equivalent single-frame image to generate a final full-color image.

Images