An adaptive imaging method and system under complex illumination

Abstract

The application provides a kind of self-adaptive imaging method and system under complex illumination, it is related to computer vision and image processing technical field.The method first acquires multiple sets of exposure image data under multiple preset light intensity conditions respectively, constructs multiple exposure image data set, and expands multiple exposure image data set;For the multiple exposure images collected under each light intensity condition, two-by-two fusion is carried out respectively, multiple candidate fusion images are obtained, and quality evaluation is carried out on each candidate fusion image, and the source image exposure time combination corresponding to the best fusion image is recorded;Then, according to the image quality evaluation results under multiple light intensity conditions, the relationship curve between the ambient light intensity and the best exposure time combination is fitted;Finally, according to the actual returned ambient light intensity value and the relationship curve between the ambient light intensity and the best exposure time combination, the exposure duration is automatically selected to shoot the image, and the shot image is subjected to multiple exposure image fusion to obtain the final high-quality image.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, and in particular to an adaptive imaging method and system under complex lighting conditions. Background Technology

[0002] Multi-exposure image fusion technology belongs to the field of computer vision and image processing. Its basic principle is to fuse multiple images of the same scene taken under different exposure conditions to overcome the limitations of the dynamic range of a single image, thereby obtaining a result image with balanced detail and brightness. This technology can improve the visibility of dark areas while preserving details in highlight areas and is widely used in fields such as intelligent photography, medical imaging, industrial inspection, and autonomous driving. For example, in security monitoring, fusing low-exposure and high-exposure images can improve the recognition of targets at night; in medical microscopy, it can expand the dynamic range of fluorescence signals to clearly reveal cell structures; and in autonomous driving, it can address the imaging needs of scenes with sudden changes in lighting, such as tunnel entrances and backlighting.

[0003] Existing multi-exposure image fusion methods are mainly divided into two categories: one is based on traditional fusion methods, which assign pixel-level weights based on manually designed fusion rules (such as brightness, contrast, saturation, sharpness, etc.) and often combine multi-scale pyramid decomposition algorithms to reduce seams and blur. These methods have low computational cost and are suitable for deployment on resource-constrained devices, but they have limited detail preservation and insufficient fusion robustness in dynamic objects and complex texture scenes. The other category is based on deep learning fusion methods, which automatically extract and fuse features from images with different exposures through models such as convolutional neural networks, Transformer architectures, or generative adversarial networks, achieving better results than traditional methods in terms of detail restoration and visual consistency. However, these methods typically require a large amount of labeled data and suffer from difficulties in image registration and inaccurate optical flow estimation in dynamic scenes. They also require high computing power from edge devices, limiting real-time applications.

[0004] In domestic research, lightweight model design and physical model data synthesis (such as the decomposition of illumination and reflection components based on Retinex theory) are often used to alleviate the problem of insufficient data, and fusion strategies are optimized in specific scenarios such as industrial inspection and security monitoring. International research, on the other hand, focuses more on the global modeling capabilities of end-to-end deep learning models in dynamic scenes and applies them to high-precision scenarios such as medical imaging and autonomous driving. However, existing technologies generally suffer from the following shortcomings: Insufficient data diversity: It is difficult to collect multi-exposure data in low light (especially at night) and dynamic scenes, which limits the model's ability to generalize in complex environments.

[0005] Poor robustness of dynamic scene fusion: Traditional methods have difficulty handling situations such as moving objects and partial occlusion. Although deep learning methods have made improvements, they are still limited by feature alignment accuracy and optical flow estimation error.

[0006] Poor adaptability to edge devices: High-performance models are difficult to run in real time on devices with limited computing power. Although traditional methods can be deployed, the quality of integration is limited, making it difficult to balance real-time performance and effectiveness.

[0007] In summary, how to improve both fusion effect and operating efficiency under limited hardware resources, and achieve high-quality multi-exposure image fusion in low-light and dynamic complex scenes, are key issues that existing technologies still need to address. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing an adaptive imaging method and system under complex lighting conditions. This method automatically adjusts imaging parameters according to the lighting conditions, realizes multi-exposure image fusion, improves the ability to retain details and expand dynamic range under different exposure conditions, and ensures efficient operation on resource-constrained hardware platforms.

[0009] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: On one hand, the present invention provides an adaptive imaging method under complex lighting conditions, comprising: Multiple sets of exposure image data were collected under various preset light intensity conditions to construct a multi-exposure image dataset. The image is decomposed based on a physical model and the multi-exposure image dataset is expanded. For the multi-exposure images acquired under each light intensity condition, pairwise fusion is performed to obtain multiple candidate fused images. An objective image quality evaluation index is introduced to evaluate the quality of each candidate fused image, and the source image exposure time combination corresponding to the best fused image is recorded. Based on the image quality evaluation results under multiple light intensity conditions, the relationship curve between ambient light intensity and the optimal exposure time combination was obtained by fitting. Based on the actual returned ambient light intensity value, and according to the fitted relationship curve between ambient light intensity and the optimal exposure time combination, the exposure time is automatically selected for image capture, and the captured images are then fused using multiple exposure image fusion to obtain the final high-quality image.

[0010] Furthermore, when acquiring multiple sets of exposure image data under various preset light intensity conditions, the multi-exposure images are acquired according to the following formula for increasing exposure time: ; in: Let be the exposure time for the i-th acquisition step; β is the initial exposure time; i is the exposure time increment factor; i is the number of acquisition steps; ensuring the acquisition of a complete sequence from low exposure to high exposure; During the incremental exposure acquisition process, if the current average grayscale value of the image has reached the set maximum allowable grayscale value... G max If the image acquisition stops, then image acquisition will cease; simultaneously, if the difference in the average grayscale values ​​of two consecutive images is less than the set convergence threshold... Image acquisition will also stop.

[0011] Furthermore, the specific method for decomposing images based on physical models and expanding the multi-exposure image dataset is as follows: Original lighting image Perform orthogonal decomposition to obtain the illumination invariant of the image. Compared with the original light component For any pixel in the image, a new illumination component can be generated using the following formula: ; in, The original light component, Represents the newly generated light component. This is an increase in illumination; achieved by controlling... The size of the light components results in light components with different exposure durations; The newly generated illumination components are then converted using the following formula. Invariant to the original illumination Composite images to generate images with different exposure levels. : ; in, Represents the basis vector of the illumination direction; Finally, the image A pixel image transformed to the corresponding RGB space.

[0012] Furthermore, the specific method for fusing the multi-exposure images acquired under each light intensity condition to obtain multiple candidate fused images, and for introducing objective image quality evaluation indicators to assess the quality of each candidate fused image, is as follows: Step A1: Calculate the multi-dimensional feature weights of each pixel in each image, including exposure weight, contrast weight, saturation weight, and sharpness weight. The four feature weights are integrated into a pixel-level comprehensive fusion weight and normalized. A two-dimensional map is formed by the contribution of each source image to the final fusion result at each pixel position, which serves as the weight map of the source image. Step A2: An improved fusion method based on the Enfuse multi-resolution fusion framework is adopted. Sharpness weight is introduced on the basis of exposure, contrast, and saturation weights to perform pairwise fusion of images in the expanded multi-exposure image dataset. Pyramids are constructed for the images and corresponding weight maps in the expanded multi-exposure image dataset, and fused layer by layer to obtain fusion pyramids at each scale. The fusion pyramids are then inversely reconstructed to generate the final fused image. Step A3: Evaluate the quality of the candidate fused image obtained by fusing each group of candidate exposure images.

[0013] Furthermore, the comprehensive fusion weights in step A1 are shown in the following formula: ; in, For the first i The image is in pixels Location-based integrated weighting, The first i The image is in pixels The exposure weight, contrast weight, saturation weight, and sharpness weight of the location. , , , These are the adjustment coefficients for the weights of the four features, representing the importance of the fusion weights for different features; The integrated weights are normalized to obtain the normalized integrated weight values, as shown in the following formula: ; in, For the first i The image is in pixels Normalized fusion weight values ​​for location; For the first i The overall fusion weight of the images; For the first j The overall fusion weight of the images; To prevent extremely small constants with a denominator of 0; The contribution of each source image to the final fusion result at each pixel location is compiled into a two-dimensional map, which serves as the normalized weight map of that source image. i After Gaussian pyramid decomposition, the weight values ​​of the normalized weighted map corresponding to the image at the k-th layer are shown in the following formula: ; in, For the first i Normalized weighted map of the image; This is the k-th level decomposition operation of the Gaussian pyramid; For the firsti The weight values ​​of Zhang Yuan's image in the k-th layer of the normalized weighted graph.

[0014] Furthermore, the specific method for step A2 is as follows: After decomposing the image into a Laplacian pyramid and the corresponding weight map into a Gaussian pyramid, the fusion pyramids at each scale are obtained based on the Laplacian pyramids of the two candidate source images participating in the current fusion and the Gaussian pyramids of the corresponding normalized weight maps: ; in, For the first k Layer fusion results; For the first i The image in the first k A layered decomposition diagram of the Laplace pyramid; Subsequently, the fusion pyramid at each scale is subjected to layer-by-layer inverse transformation and reconstruction to obtain the final fused image. The final fused image output is represented as: ; in, This represents the reconstruction process of the Pyramid of Laplace, where K represents the maximum number of layers in the pyramid.

[0015] Furthermore, the specific method for evaluating the quality of the candidate fused image obtained by fusing each group of candidate exposure images in step A3 is as follows: Step A31: Calculate the color card brightness index for each candidate fused image; Let the color swatch region in the r-th candidate fused image be . The total number of pixels in the color chart area is The average brightness of the color card region in the candidate fused image is defined as: ; in, This represents the average brightness of the color card region in the r-th candidate fused image. This represents the brightness value of pixel (x, y) within the color chart region in the r-th candidate fused image; according to The deviation from the standard brightness value of the color chart is used to calculate the brightness index of the color chart. Step A32: Calculate the MEF-SSIM index for each candidate fused image; For the For the candidate fusion image, two images to be fused that actually participated in generating the candidate fusion image are selected as reference inputs; the two images to be fused and the candidate fusion image have the same size and have been spatially registered, so that the corresponding local windows of the three represent the same scene area; and the two images to be fused and the candidate fusion image are converted into brightness images according to the same rules; The original resolution brightness image is used as the first scale, i.e., the finest scale. From the second scale onwards, the two images to be fused and the candidate images to be fused from the previous scale are filtered using the same low-pass filter, and downsampled by a factor of two in both the horizontal and vertical directions, until the image is constructed. The image sequence is divided into several scales; then the quality value within the local window of each scale is calculated and spatially averaged, and then the MEF-SSIM index is calculated by the weighted geometric mean of the quality values ​​of each scale. Step A33: Select the color chart brightness index and MEF-SSIM index as evaluation indicators, and unify the dimensions of different indicators by normalization. Then, perform a weighted average of the two normalized indicators to obtain the weighted comprehensive score of each candidate fused image.

[0016] Furthermore, the specific method for fitting the relationship curve between ambient light intensity and the optimal exposure time combination based on image quality evaluation results under multiple light intensity conditions is as follows: Based on the weighted comprehensive score of each candidate fused image, the candidate fused image with the highest comprehensive score is selected as the best fused image under the current light intensity condition, and the exposure time combination of the source images corresponding to the best fused image is recorded; experimental data under different light intensity conditions are compiled, including ambient light intensity value, exposure time combination of source images corresponding to each candidate fused image, color chart brightness index, MEF-SSIM index, weighted comprehensive score, and the finally selected best exposure time combination; based on the above experimental data, the relationship curve between ambient light intensity and the best exposure time combination is obtained by fitting with MATLAB.

[0017] On the other hand, the present invention provides an adaptive imaging system under complex lighting conditions, including a multi-exposure image acquisition module, an image augmentation module, an image fusion module, a quality assessment module, a relationship curve construction module, and a high-quality image generation module; The multi-exposure image acquisition module acquires multiple sets of exposure image data under multiple preset light intensity conditions to construct a multi-exposure image dataset. The image augmentation module decomposes images based on a physical model and expands the multi-exposure image dataset; The image fusion module performs pairwise fusion on the multi-exposure images acquired under each light intensity condition to obtain multiple candidate fused images; The quality assessment module introduces objective image quality evaluation indicators to assess the quality of each candidate fused image and records the source image exposure time combination corresponding to the best fused image. The relationship curve construction module fits the relationship curve between ambient light intensity and the optimal exposure time combination based on the image quality evaluation results under multiple light intensity conditions. The high-quality image generation module automatically selects the exposure time to capture images based on the actual returned ambient light intensity value and the fitted relationship curve between ambient light intensity and the optimal exposure time combination. The captured images are then fused using multiple exposure images to obtain the final high-quality image.

[0018] Compared with existing technologies, the adaptive imaging method and system under complex lighting conditions provided by this invention have the following advantages: (1) In view of the problem that existing multi-exposure image fusion methods are prone to overexposure, underexposure, loss of dark details or saturation of highlight areas in low light, strong light, backlight and large contrast scenes, the present invention collects multi-exposure images with different exposure times under the same light intensity and uses a multi-exposure image fusion algorithm to fuse them. This can comprehensively retain the effective details in different exposure images and improve the brightness balance, clarity and dynamic range of the fused image. (2) In view of the problems of setting exposure parameters and difficulty in automatically selecting appropriate exposure time according to changes in ambient light intensity in existing methods, this invention obtains the ambient light intensity RAW value in real time through a photoresistor and combines the light intensity-optimal exposure time relationship curve obtained by fitting under different light intensity conditions. It can automatically determine the exposure time combination for image acquisition and fusion according to the current ambient light intensity, thereby reducing manual parameter adjustment and improving the adaptability and stability of the imaging process under complex lighting conditions. (3) In view of the difficulties in acquiring real multi-exposure images and the lack of data diversity in low light and dynamic complex scenes, this invention decomposes the illumination components and illumination invariants of the image based on the physical model, and generates images with different exposure levels by adjusting the illumination components, thereby expanding the multi-exposure image dataset, reducing the cost of real data acquisition, enhancing the data coverage, and providing richer input samples for subsequent fusion algorithms. (4) In view of the problem that the evaluation of existing multi-exposure fusion results relies on manual observation and the evaluation standards are not uniform, this invention introduces color card brightness deviation and MEF-SSIM index to evaluate the quality of candidate fusion images, and normalizes and weights the evaluation index to achieve a comprehensive score. This can objectively screen out the fusion images with better quality under the current light intensity conditions and their corresponding exposure time combinations, thereby improving the accuracy and repeatability of the process of determining the best exposure time. (5) In view of the problem that some deep learning fusion methods have high requirements for computing resources and are difficult to deploy on edge devices such as Raspberry Pi, this invention adopts a combination of lightweight multi-exposure fusion algorithm and light intensity-exposure time curve fitting, which reduces the computational complexity while ensuring the quality of fused images, and is conducive to realizing adaptive multi-exposure imaging on low-cost, resource-constrained edge imaging devices.

[0019] In summary, the method of the present invention can improve image fusion quality under complex lighting conditions, reduce overexposure, underexposure and loss of detail, improve the automation of exposure parameter selection and imaging stability, and take into account data expansion capabilities and edge device deployment requirements. Attached Figure Description

[0020] Figure 1 This is a flowchart of an adaptive imaging method under complex lighting conditions provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the process of an adaptive imaging method under complex lighting conditions provided in Embodiment 1 of the present invention; Figure 3 The image provided in Embodiment 1 of the present invention is a multi-exposure fusion image, wherein (a) is an underexposed image, (b) is an overexposed image, (c) is a real reference image, and (d) is a multi-exposure fusion image. Detailed Implementation

[0021] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0022] Example 1: In this embodiment, an adaptive imaging method under complex lighting conditions is described, such as... Figure 1 , Figure 2 As shown, this involves image acquisition, physical modeling to expand the dataset, exposure curve fitting, and multi-exposure image fusion, specifically including the following steps: Step S1: Acquire multi-exposure images and construct a multi-exposure image dataset; In this embodiment, an experimental imaging device is built using a Raspberry Pi platform to collect multiple sets of exposure image data under various preset light intensity conditions, forming a multi-exposure image dataset. For each preset light intensity condition, the ambient light intensity is kept relatively stable during the acquisition of a single set of multi-exposure images, and the current ambient light intensity is recorded by the output value of a photosensor. At the same time, an image sequence from low exposure to high exposure is obtained according to a preset exposure time increment rule.

[0023] During the shooting process, it is crucial to strictly control the stability of ambient light intensity to avoid data consistency loss due to light intensity fluctuations. Simultaneously, special attention must be paid to occlusion issues at the shooting angle, ensuring there are no dynamic obstructions such as leaves or debris in front of the lens to prevent pixel misalignment or information loss between multiple exposure images, which could affect the subsequent image fusion effect. These measures ensure the validity, repeatability, and reliability of the experimental data and the fusion algorithm.

[0024] In this embodiment, multi-exposure images are acquired according to the following formula for increasing exposure time: ; in: Let be the exposure time (milliseconds) for the i-th acquisition step; β is the initial exposure time (milliseconds); β is the exposure time increment factor (>1); i is the number of acquisition steps. This ensures the acquisition of a complete sequence from low exposure to high exposure.

[0025] In this embodiment, the image acquisition termination condition is used to control the termination time of the multi-exposure image acquisition process, which is a dynamic stopping strategy. During the incremental exposure acquisition process, if the current average grayscale value of the image has reached the set maximum allowable grayscale value... G max This indicates that the image is already close to saturation; further increasing the exposure will only lead to overexposure and loss of detail in highlight areas, so capturing should be stopped immediately. Additionally, if the difference in the average grayscale values ​​of two consecutive images... Less than the set convergence threshold This indicates that increasing the exposure time can no longer significantly improve the image brightness (the brightness curve tends to flatten out). At this point, continuing to collect data will result in redundant data, and it should be stopped.

[0026] This ensures data coverage of the full exposure range while reducing invalid data collection and saving storage and processing resources. ; in: G This represents the average grayscale value of the current image. G max Maximum allowed grayscale value; This is the convergence threshold for grayscale changes.

[0027] S2: A complex illumination image fusion algorithm is used to process the images in the multi-exposure image dataset to determine the optimal exposure time for the camera under different light intensity conditions; the images are decomposed based on a physical model and the multi-exposure image dataset is expanded; the multi-exposure image groups acquired under each light intensity condition are fused separately, and an objective image quality evaluation index is introduced to evaluate the quality of each candidate fused image; based on the image quality evaluation results under multiple light intensity conditions, the relationship curve between ambient light intensity and the optimal exposure time combination is obtained by fitting with MATLAB to reflect the optimal exposure time for the camera under different light intensity conditions.

[0028] S21: The multi-exposure image dataset is expanded using a physical model to obtain an enhanced multi-exposure image dataset; The original illumination image is orthogonally decomposed using a physical model to obtain an illumination component and an illumination invariant. By changing the size of the illumination component and combining it with the original illumination invariant, a multi-exposure image is obtained. Finally, the multi-exposure images are fused to obtain the final enhanced image. In the original lighting image After orthogonal decomposition, the illumination invariants of the image can be obtained. With light component By enhancing or weakening the illumination components of an image, illumination components with different exposure times can be obtained.

[0029] For any pixel in the image, the generation of a new illumination component can be expressed as follows: ; in, The original light component, Represents the newly generated light component. This is an increase in illumination. It is achieved by controlling... The size can be used to obtain light components with different exposure times.

[0030] The newly generated illumination components are then converted using the following formula. Invariant to the original illumination By compositing, images with different exposure levels can be generated. : ; in, Represents the basis vector of the illumination direction; Finally, the image A pixel image transformed to the corresponding RGB space.

[0031] S22: To find the optimal combination of exposure times under different light intensities, N images of the same scene captured at different exposure times are fused pairwise to obtain candidate fused images. The fusion method uses a lightweight Enfuse: each pixel of each input image is assigned a weight related to brightness, contrast, saturation, and sharpness, and after normalization, it participates in pixel-level and multi-scale fusion, thereby obtaining seamless, clear candidate fused images with a wide dynamic range. Then, the brightness deviation of the color chart and the MEF-SSIM (a quality evaluation index that does not require an ideal fusion reference image but uses the two images to be fused that actually participate in the generation of the candidate fused image as reference inputs) are used to score each candidate fused image, select the best combination of exposure images, and fit the light intensity-optimal exposure time relationship curve.

[0032] S22-1: Define an enhanced multi-exposure image set containing N images with different exposure times, denoted as: ; in, This represents the Nth exposed image, ( x , y () represents the pixel coordinates in the image.

[0033] S22-2: A pixel-level weighting model is designed to calculate multi-dimensional feature weights for each pixel of each input image, automatically determining the degree to which each pixel is preserved in the final fusion result. The multi-dimensional feature weights comprehensively consider features such as exposure, contrast, saturation, and sharpness, ensuring that the fused image is both rich in detail and color-accurate, while avoiding seams and blurring. This is more precise than traditional methods that select pixels solely based on brightness.

[0034] For the i Image, pixels ( x , y The multidimensional feature weights include exposure weight, contrast weight, saturation weight, and sharpness weight; S22-2a: Calculate the exposure weight of each pixel in each image; the exposure weight encourages pixels with brightness close to the median value (0.5), suppresses overly dark and overly bright areas, ensures that the fusion result is neither overexposed nor underexposed, and preserves details under different exposures in a balanced manner. The calculation formula is as follows: ; in, For the first i The normalized pixel brightness values ​​of the image are in the range of [0,1]. To control the standard deviation of the Gaussian distribution for brightness selectivity, it is generally set to 0.2 to 0.3; For the first i The image is in pixels The exposure weight value of a point is maximum when the brightness is 0.5, and approaches 0 when it is 0 or 1.

[0035] S22-2b: Calculate the contrast weight of each pixel in each image; the contrast weight uses the Laplacian operator to detect local edges and details in the image, and prioritizes the preservation of areas rich in detail during image fusion to prevent details from being overwhelmed by low-contrast areas; the contrast weight calculation of this invention uses a 3×3 Laplacian kernel to calculate the image brightness / grayscale value relative to the pixel ( x , y The second derivative of is calculated by taking its absolute value to ensure non-negativity. The formula is shown below: ; in, The Laplacian operator (edge ​​detection) is applied to the image. Used to extract pixels ( x , y The second-order brightness change; For the first i The pixels of the image ( x , y The edge strength at a given point is the contrast weight; the richer the local texture and edges of a pixel, the higher this weight.

[0036] S22-2c: Calculate the saturation weight of each pixel in each image; the saturation weight preserves the most vibrant pixels, resulting in a more realistic and natural color reproduction in the fused image, avoiding a grayish appearance caused by average fusion. The calculation formula is shown below: ; in, For the pixel at the th i The first image c Pixel values ​​for the channels (red, green, blue); For the first i Average brightness of the RGB three channels of the image; Indicates the first i The image is in pixels ( x , y The saturation weight at the position is calculated by the dispersion of the RGB three-channel pixel values ​​relative to the average brightness. The larger the value, the higher the color saturation and the richer the color of the pixel, and the higher the retention weight in the fusion.

[0037] S22-2d: Calculates the sharpness weight for each pixel in each image. The sharpness weight emphasizes the clarity of edge regions, encouraging sharp, well-focused areas in the image to dominate the fusion process, enhancing local image sharpness, and strengthening the contribution of sharp, focused areas during fusion to make the output image sharper. The calculation formula is shown below: ; in, For the first i The pixels of the image ( x , y The gradient value at a given point (such as those calculated by Sobel or Prewitt) represents the local brightness change; For the first i The pixels of the image ( x , y The sharpness weight at point () indicates that the larger the value, the more obvious the edge.

[0038] S22-2e: By unifying four features into a pixel-level comprehensive fusion weight, and using exponential amplification coefficients, the influence of each factor can be flexibly controlled to adapt to different application needs. The comprehensive fusion weight is shown in the following formula: ; in, For the first i The overall fusion weight of the images, The larger the number, the more likely it is to be the first. i The image is in pixels The more valuable the location, the better; , , , These are the adjustment coefficients for the weights of the four features, representing the importance of the fusion weights of different features; the influence of important features is strengthened through an exponential amplification mechanism to improve the fusion quality.

[0039] S22-2f: Each input image in pixels Each position will receive a comprehensive fusion weight. It is calculated based on features such as pixel brightness, contrast, saturation, and sharpness. However, the calculated weight values ​​usually lack a unified dimension, and the numerical distribution between different pixels and different images can vary greatly. Directly using these raw weights will lead to distortion of the fusion result. Therefore, it is necessary to normalize the comprehensive fusion weights of each pixel in the input image to obtain normalized fusion weight values, as shown in the following formula: ; in, For the first i The image is in pixels Normalized fusion weight values ​​for location; For the first i The overall fusion weight of the images; For the first j The overall fusion weight of the images; To prevent extremely small constants with a denominator of 0; The contribution of each source image to the final fusion result at each pixel location is compiled into a two-dimensional map, which serves as the normalized weight map of that source image; i Normalized weight map corresponding to Zhang image After Gaussian pyramid decomposition, the weights at the k-th level are: ; in, For the first i Normalized weighted map of the image; This is the k-th level decomposition operation of the Gaussian pyramid; For the first i The weight values ​​of Zhang Yuan's image normalized weight map in the k-th level Gaussian pyramid.

[0040] S22-2g: Construct pyramids for the images and corresponding weight maps in the expanded multi-exposure image dataset, and fuse them layer by layer to obtain fused pyramids at each scale; To avoid image seams or blurring caused by simple weighted fusion, this invention employs a multi-scale strategy using the Enfuse algorithm to perform pairwise fusion of images in an expanded multi-exposure image dataset: images are decomposed into Laplacian pyramids, and the corresponding weight maps are decomposed into Gaussian pyramids, with fusion performed layer by layer at each scale. The aim is to fuse the detailed information of each image while eliminating fusion boundary artifacts at large scales, thereby improving image naturalness and clarity.

[0041] After decomposing the image into a Laplacian pyramid and the corresponding weight map into a Gaussian pyramid, the two candidate source images participating in the current fusion are used as the basis for the fusion. , The Laplace pyramid and the corresponding Gaussian pyramid with normalized weights are used to obtain the fused pyramids at various scales: ; in, For the first k Layer fusion results; For the first i The image in the first k Layered Laplace pyramid decomposition diagram; multi-scale k It is usually set to 4–6 layers. and These represent two candidate source images selected from the expanded multi-exposure image dataset; and Representing the source images respectively and The corresponding normalized fusion weights; This represents the k-th level Laplacian pyramid decomposition result of the image; This represents the Gaussian pyramid decomposition result of the k-th layer of the weighted graph; S22-2h: Perform inverse reconstruction of the fused pyramid to generate the final fused image; In S22-2g, the two candidate source images involved in the current fusion have been used. , The Laplace pyramid and the corresponding Gaussian pyramid with normalized weights are used to obtain the fused pyramids at various scales. Subsequently, the fusion pyramid at each scale was subjected to layer-by-layer inverse transformation and reconstruction to obtain the final fused image. The final fused image can be represented as: ; in, This represents the Laplacian pyramid reconstruction process, where K represents the maximum number of pyramid layers. The fused image combines the best details from different exposure images, possessing good sharpness and dynamic range, making it suitable for subsequent optimal exposure time estimation.

[0042] S22-3: Image Fusion Quality Evaluation; The quality of the candidate fused image obtained by fusing each set of candidate exposure images is evaluated. Evaluation metrics include the color chart brightness index and the MEF-SSIM index. The color chart brightness index measures the closeness between the average brightness of the color chart region in the candidate fused image and the standard brightness value. The MEF-SSIM index measures the ability of the candidate fused image to retain structural information, local contrast, and visual naturalness relative to the source image. By normalizing and weighting the above two metrics, the candidate fused image with the highest comprehensive score is selected as the optimal fused image under the current lighting conditions.

[0043] S22-3a: Calculate the color card brightness index for each candidate fused image; For each candidate fused image, the brightness of a single pixel in the image. The definition is as follows: ; in, r , g , b These represent the pixel values ​​of the RGB channels for a single pixel. .

[0044] Let the color swatch region in the r-th candidate fused image be . The total number of pixels in the color chart area is The average brightness of the color card region in the candidate fused image is defined as: ; in, This represents the average brightness of the color card region in the r-th candidate fused image. This represents the brightness value of pixel (x, y) within the color chart region in the r-th candidate fused image. Subsequently, based on... The brightness index of the color card is calculated based on the deviation from the standard brightness value of the color card.

[0045] S22-3b: Calculate the MEF-SSIM index for each candidate fused image; MEF-SSIM is a quality evaluation metric that does not require an ideal fusion reference image, but uses two images to be fused that actually participate in generating the candidate fusion image as reference inputs. It utilizes the perceptual mechanism of the human visual system, taking structural information as the primary basis, to compare the consistency between the fused image and the source image in terms of local structure, brightness, and contrast. A higher MEF-SSIM index value indicates richer structural information and a more natural fusion.

[0046] For the For each candidate fusion image, two images to be fused that actually participated in generating the candidate fusion image are selected as reference inputs. The two images to be fused and the candidate fusion image have the same size and have been spatially registered, so that the corresponding local windows of the three represent the same scene area. The two images to be fused and the candidate fusion image are then converted into brightness images according to the same rules.

[0047] The original resolution brightness image is used as the first scale, i.e., the finest scale. From the second scale onwards, the two images to be fused and the candidate images to be fused from the previous scale are filtered using the same low-pass filter, and downsampled by a factor of two in both the horizontal and vertical directions, until the image is constructed. A sequence of images at various scales. The image at scale n is generated by the first... Image generation at various scales, and Indicates the finest scale. This indicates the coarsest scale.

[0048] In the The first scale Within a local window, corresponding brightness blocks from the two images to be merged are expanded in the same pixel order. column vectors in and The corresponding brightness blocks of the candidate fused images are then expanded into column vectors. ; In the The first scale Within a local window, the corresponding brightness blocks of the two images to be merged are expanded in the same pixel order. N 3D column vector and and will the The corresponding brightness blocks of the candidate fused images are expanded in the same pixel order. N 3D column vector .

[0049] First, calculate the average brightness of the local brightness blocks in each image to be fused, and then perform mean removal processing: ; ; ; in, For the first The average brightness of local brightness blocks in the image to be fused. This refers to the local brightness block after mean removal processing, i.e., the zero-mean local brightness vector. Indicates local contrast. Represents local structure; when When equal to 0, let the corresponding ; for A column vector of all 1s. For the sake of simplicity, some calculations above are omitted. , and Subscript.

[0050] Calculate the structural consistency between local brightness blocks in two images to be fused: ; ; in, For structural consistency, For exponential parameters; when and At that time, the normalized weights of the local structures of each image to be fused are determined based on structural consistency: ; in, For the first Zhang's normalized weights for the local structure of the image to be fused. and Each represents the image number to be merged. It is the dummy index for summation in the normalized denominator.

[0051] The desired contrast and desired structure are constructed from the local contrast and local structure of the two images to be fused: ; ; in, To determine the desired contrast of the current local window, take the local contrast of the two images to be fused. and The larger value in; For the first Local structure vectors of the images to be fused; For the first Zhang's normalized weights for the local structure of the image to be fused; This is the weighted composite vector of the local structure vectors of the two images to be fused; This is the desired structure vector obtained by normalizing the weighted composite vector; The desired zero-mean structure-contrast image block is obtained by multiplying the desired contrast with the desired structure vector.

[0052] Degenerate windows are handled according to the following rules: when At that time, directly ordered No longer calculated , and ;when and At that time, according to The limit determines the weight of the local structure of each image to be fused, that is, the weight is assigned to the image to be fused with the largest local contrast. When the two images to be fused have the same maximum local contrast in the current local window, the weight is evenly distributed.

[0053] when season and The aforementioned degradation window processing rules are only applicable to processing local structure vectors. Structural consistency Exponential parameters Normalized weights and expected structure vector The calculation formulas are used for undefined boundary points that appear under zero contrast, perfect uniformity, or structural cancellation conditions; for local windows that do not meet the above degradation conditions, the MEF-SSIM local quality value is still calculated according to the aforementioned calculation formulas.

[0054] The desired zero-mean structure-contrast image block With the Corresponding brightness blocks of the candidate fused image Structural comparison was performed to obtain the first The first scale Quality value of a local window : ; in, Expected zero-mean structure—contrast image patch The local variance, For the corresponding brightness blocks of the candidate fused image The local variance, for and Local covariance, A positive stability constant is set to prevent the denominator from approaching zero.

[0055] In this embodiment, an 11×11 pixel sliding window with a step size of 1 is used for each scale, and only windows completely within the image area are counted. The mean and contrast of the image blocks to be fused are calculated with equal weights for pixels within the window, and the local variance and covariance are calculated using 11×11 normalized Gaussian weights with a standard deviation of 1.5. A 2×2 mean low-pass filter is used before downsampling, and the boundaries are symmetrically extended.

[0056] For the Spatially average the quality values ​​of all local windows at each scale to obtain the first... Quality value at each scale : ; No. MEF-SSIM index of candidate fused images Defined as: ; ; in, For the first The total number of local windows at each scale. These are the non-negative weights corresponding to the scale. The preferred parameters are: ; The above scale weights are arranged in order from fine to coarse. This refers to the dynamic range of the luminance values; for an 8-bit luminance image, For normalized brightness images, . The larger the value, the better the candidate fused image preserves the local structural information and cross-scale relative brightness structure of the two images to be fused.

[0057] S22-3c: Select the color chart brightness index and MEF-SSIM index, unify the dimensions of different indices, obtain the overall quality ranking of candidate fused images, and automatically select the exposure combination corresponding to the best fused image through the score. The formula is expressed as: Color chart brightness deviation normalization: ; ; in, This indicates the standard brightness value of the color chart. Indicates the first The deviation between the average brightness of the color card region and the standard brightness value in the candidate fused image. This represents the largest color card brightness deviation among all candidate fused images. Indicates the first The brightness evaluation score after normalizing the brightness deviation of the candidate fused image. The larger the value, the closer the brightness of the color chart of the candidate fused image is to the standard brightness value. To prevent extremely small constants with a denominator of 0.

[0058] MEF-SSIM index normalization: ; in, For the normalized MEF-SSIM index, Indicates the first r The raw MEF-SSIM evaluation value of the candidate fused image is as follows: the larger the value, the better the structural information of the candidate fused image is preserved and the more natural the fusion effect is. and The minimum and maximum values ​​of MEF-SSIM for all candidate fused images under the same ambient light intensity conditions; when When this is done, the normalized MEF-SSIM index of all candidate fused images in the same group is uniformly set to the same preset value, preferably 0, to avoid the normalization denominator being 0.

[0059] The weighted average of the two normalized metrics is used to obtain the weighted comprehensive score for each candidate fused image: ; in, For the first The weighted comprehensive score of the candidate fused images. Assuming the weights of the two indicators, in this embodiment, Set the value to 0.5; S23: Based on the weighted comprehensive score of each candidate fused image, select the candidate fused image with the highest comprehensive score as the best fused image under the current light intensity condition, and record the source image exposure time combination corresponding to the best fused image; organize the experimental data under different light intensity conditions, including ambient light intensity value, source image exposure time combination corresponding to each candidate fused image, color chart brightness index, MEF-SSIM index, weighted comprehensive score, and the finally selected best exposure time combination; based on the above experimental data, obtain the relationship curve between ambient light intensity and the best exposure time combination by fitting with MATLAB; S3: Iterative processing: Based on the actual returned ambient light intensity value, according to the fitted relationship curve between ambient light intensity and the optimal exposure time combination, the exposure time is automatically selected to capture images, and the captured images are fused into multiple exposure images to obtain the final high-quality image.

[0060] like Figure 3 As shown, the results are obtained by processing images taken under complex lighting conditions using a designed multi-exposure image fusion algorithm.

[0061] Example 2: In this embodiment, an adaptive imaging system under complex lighting conditions includes a multi-exposure image acquisition module, an image augmentation module, an image fusion module, a quality assessment module, a relationship curve construction module, and a high-quality image generation module. The multi-exposure image acquisition module acquires multiple sets of exposure image data under multiple preset light intensity conditions to construct a multi-exposure image dataset. The image augmentation module decomposes images based on a physical model and expands the multi-exposure image dataset; The image fusion module performs pairwise fusion on the multi-exposure images acquired under each light intensity condition to obtain multiple candidate fused images; The quality assessment module introduces objective image quality evaluation indicators to assess the quality of each candidate fused image and records the source image exposure time combination corresponding to the best fused image. The relationship curve construction module fits the relationship curve between ambient light intensity and the optimal exposure time combination based on the image quality evaluation results under multiple light intensity conditions. The high-quality image generation module automatically selects the exposure time to capture images based on the actual returned ambient light intensity value and the fitted relationship curve between ambient light intensity and the optimal exposure time combination. The captured images are then fused using multiple exposure image fusion to obtain the final high-quality image.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the present invention.

Claims

1. An adaptive imaging method under complex illumination, characterized in that, include: Multiple sets of exposure image data were collected under various preset light intensity conditions to construct a multi-exposure image dataset. Image decomposition based on physical models and expansion of multi-exposure image datasets; For each light intensity condition, the multi-exposure images acquired are fused in pairs to obtain multiple candidate fused images. An objective image quality evaluation index is introduced to evaluate the quality of each candidate fused image, and the source image exposure time combination corresponding to the best fused image is recorded. Based on the image quality evaluation results under multiple light intensity conditions, the relationship curve between ambient light intensity and the optimal exposure time combination was obtained by fitting. Based on the actual returned ambient light intensity value, and according to the fitted relationship curve between ambient light intensity and the optimal exposure time combination, the exposure time is automatically selected for image capture, and the captured images are then fused using multiple exposure image fusion to obtain the final high-quality image.

2. The adaptive imaging method under complex illumination according to claim 1, characterized in that, When acquiring multiple sets of exposure image data under multiple preset light intensity conditions, the method acquires multiple exposure images according to the following exposure time increment formula: ； in: Let be the exposure time for the i-th acquisition step; β is the initial exposure time; i is the exposure time increment factor; i is the number of acquisition steps; ensuring the acquisition of a complete sequence from low exposure to high exposure; During the incremental exposure acquisition process, if the current average grayscale value of the image has reached the set maximum allowable grayscale value... G max If the image acquisition stops, then image acquisition will cease; simultaneously, if the difference in the average grayscale values ​​of two consecutive images is less than the set convergence threshold... Image acquisition will also stop.

3. The adaptive imaging method under complex illumination according to claim 2, characterized in that, The specific method for decomposing images based on physical models and expanding multi-exposure image datasets is as follows: Original lighting image Perform orthogonal decomposition to obtain the illumination invariant of the image. Compared with the original light component For any pixel in the image, a new illumination component can be generated using the following formula: ； in, The original light component, Represents the newly generated light component. This is an increase in illumination; achieved by controlling... The size of the light components results in light components with different exposure durations; The newly generated illumination components are then converted using the following formula. Invariant to the original illumination Composite images to generate images with different exposure levels. : ； in, Represents the basis vector of the illumination direction; Finally, the image A pixel image transformed to the corresponding RGB space.

4. The adaptive imaging method under complex illumination according to claim 3, characterized in that, The specific method for fusing the multi-exposure images acquired under each light intensity condition to obtain multiple candidate fused images, and for introducing objective image quality evaluation indicators to assess the quality of each candidate fused image, is as follows: Step A1: Calculate the multi-dimensional feature weights of each pixel in each image, including exposure weight, contrast weight, saturation weight, and sharpness weight. The four feature weights are integrated into a pixel-level comprehensive fusion weight and then normalized. A two-dimensional map is formed by calculating the contribution of each source image to the final fusion result at each pixel location, which serves as the weight map for that source image. Step A2: An improved fusion method based on the Enfuse multi-resolution fusion framework is adopted. In addition to the exposure, contrast and saturation weights, a sharpness weight is introduced. Pyramids are constructed for the images and corresponding weight maps in the expanded multi-exposure image dataset, and then fused layer by layer to obtain the fusion pyramids at each scale. The fusion pyramid is then inversely reconstructed to generate the final fused image. Step A3: Evaluate the quality of the candidate fused image obtained by fusing each group of candidate exposure images.

5. The adaptive imaging method under complex illumination according to claim 4, characterized in that, The comprehensive fusion weights in step A1 are shown in the following formula: ； in, For the first i The image is in pixels Location-based integrated weighting, The first i The image is in pixels The exposure weight, contrast weight, saturation weight, and sharpness weight of the location. , , , These are the adjustment coefficients for the weights of the four features, representing the importance of the fusion weights for different features; The integrated weights are normalized to obtain the normalized integrated weight values, as shown in the following formula: ； in, For the first i The image is in pixels Normalized fusion weight values ​​for location; For the first i The overall fusion weight of the images; For the first j The overall fusion weight of the images; To prevent extremely small constants with a denominator of 0, a two-dimensional graph is constructed from the contribution of each source image at each pixel location to the final fusion result, serving as the normalized weight graph of that source image. i After Gaussian pyramid decomposition, the weight values ​​of the normalized weighted map corresponding to the image at the k-th layer are shown in the following formula: ； in, For the first i Normalized weighted map of the image; This is the k-th level decomposition operation of the Gaussian pyramid; For the first i The weight values ​​of Zhang Yuan's image in the k-th layer of the normalized weighted graph.

6. The adaptive imaging method under complex illumination according to claim 5, characterized in that, The specific method for step A2 is as follows: After decomposing the image into a Laplacian pyramid and the corresponding weight map into a Gaussian pyramid, the fusion pyramids at each scale are obtained based on the Laplacian pyramids of the two candidate source images participating in the current fusion and the Gaussian pyramids of the corresponding normalized weight maps: ； in, For the first k Layer fusion results; For the first i The image in the first k A layered decomposition diagram of the Laplace pyramid; Subsequently, the fusion pyramid at each scale is subjected to layer-by-layer inverse transformation and reconstruction to obtain the final fused image. The final fused image output is represented as: ； in, This represents the reconstruction process of the Pyramid of Laplace, where K represents the maximum number of layers in the pyramid.

7. The adaptive imaging method under complex illumination according to claim 6, characterized in that, The specific method for evaluating the quality of the candidate fused image obtained by fusing each group of candidate exposure images in step A3 is as follows: Step A31: Calculate the color card brightness index for each candidate fused image; Let the color swatch region in the r-th candidate fused image be . The total number of pixels in the color chart area is The average brightness of the color chart region in the candidate fused image is defined as: ； in, This represents the average brightness of the color card region in the r-th candidate fused image. This represents the brightness value of pixel (x, y) within the color chart region in the r-th candidate fused image; according to The deviation from the standard brightness value of the color chart is used to calculate the brightness index of the color chart. Step A32: Calculate the MEF-SSIM index for each candidate fused image; For the For the candidate fusion image, two images to be fused that actually participated in generating the candidate fusion image are selected as reference inputs; the two images to be fused and the candidate fusion image have the same size and have been spatially registered, so that the corresponding local windows of the three represent the same scene area; and the two images to be fused and the candidate fusion image are converted into brightness images according to the same rules; The original resolution brightness image is used as the first scale, i.e., the finest scale. From the second scale onwards, the two images to be fused and the candidate images to be fused from the previous scale are filtered using the same low-pass filter, and downsampled by a factor of two in both the horizontal and vertical directions, until the image is constructed. The image sequence is divided into several scales; then the quality value within the local window of each scale is calculated and spatially averaged, and then the MEF-SSIM index is calculated by the weighted geometric mean of the quality values ​​of each scale. Step A33: Select the color chart brightness index and MEF-SSIM index as evaluation indicators, and unify the dimensions of different indicators by normalization. Then, perform a weighted average of the two normalized indicators to obtain the weighted comprehensive score of each candidate fused image.

8. The adaptive imaging method under complex illumination according to claim 7, characterized in that, The specific method for fitting the relationship curve between ambient light intensity and the optimal exposure time combination based on image quality evaluation results under multiple light intensity conditions is as follows: Based on the weighted comprehensive score of each candidate fused image, the candidate fused image with the highest comprehensive score is selected as the best fused image under the current light intensity condition, and the exposure time combination of the source images corresponding to the best fused image is recorded; experimental data under different light intensity conditions are compiled, including ambient light intensity value, exposure time combination of source images corresponding to each candidate fused image, color chart brightness index, MEF-SSIM index, weighted comprehensive score, and the finally selected best exposure time combination; based on the above experimental data, the relationship curve between ambient light intensity and the best exposure time combination is obtained by fitting with MATLAB.

9. An adaptive imaging system under complex lighting conditions, implemented based on the adaptive imaging method under complex lighting conditions as described in claim 1, characterized in that, It includes a multi-exposure image acquisition module, an image augmentation module, an image fusion module, a quality assessment module, a relationship curve construction module, and a high-quality image generation module; The multi-exposure image acquisition module acquires multiple sets of exposure image data under multiple preset light intensity conditions to construct a multi-exposure image dataset. The image augmentation module decomposes images based on a physical model and expands the multi-exposure image dataset; The image fusion module performs pairwise fusion on the multi-exposure images acquired under each light intensity condition to obtain multiple candidate fused images; The quality assessment module introduces objective image quality evaluation indicators to assess the quality of each candidate fused image and records the source image exposure time combination corresponding to the best fused image. The relationship curve construction module fits the relationship curve between ambient light intensity and the optimal exposure time combination based on the image quality evaluation results under multiple light intensity conditions. The high-quality image generation module automatically selects the exposure time to capture images based on the actual returned ambient light intensity value and the fitted relationship curve between ambient light intensity and the optimal exposure time combination. The captured images are then fused using multiple exposure image fusion to obtain the final high-quality image.

Images