Smooth mask guided high dynamic range remote sensing image data synthesis method and system
By using a smooth mask-guided method to generate high dynamic range remote sensing image data, the problem of insufficient synthesis of dark and overexposed areas in existing technologies is solved, thereby improving the applicability and image quality of remote sensing image enhancement models.
Patent Information
- Application Number
- CN202411430013.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing technologies cannot effectively generate high dynamic range remote sensing image data that simultaneously includes dark and overexposed areas, resulting in insufficient generalization performance of image enhancement models on different imaging devices, and existing synthesis methods cannot accurately match the imaging range of remote sensing images under real lighting conditions.
A smoothing mask-guided method is adopted to generate overexposed and underexposed images, and a smoothing mask is generated using sensor calibration parameters and a noise model. Combined with weighted fusion technology, high dynamic range remote sensing image data is generated.
It provides a high dynamic range remote sensing image data synthesis scheme that is closer to reality, improves the applicability and generalization ability of image enhancement models, and enhances the quality of remote sensing images.
Smart Images

Figure CN119006305A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a remote sensing image data synthesis method and system, in particular to a high dynamic range remote sensing image data synthesis method and system guided by a smoothing mask, and belongs to the technical field of image information processing. Background Art
[0002] When a high-brightness area under strong light sources (sunlight, lamps or reflections, etc.) and a relatively low-brightness area such as shadows and backlight exist in the image at the same time, the image output by the camera will show that the bright area becomes white due to overexposure, while the dark area becomes black due to underexposure. In other words, the camera has limitations in its performance of the brightest area and the darkest area in the same scene, and this limitation is the "dynamic range".
[0003] Due to the large sensing range of the sensor, remote sensing images may contain both overexposed and dark areas, which seriously affects the image quality. To solve the problem of limited sensor dynamic range, an effective method is to use high dynamic range imaging technology with deep learning-assisted image restoration. To this end, it is necessary to provide a dataset consisting of paired low-quality and high-quality remote sensing images in high dynamic range application scenarios for deep learning-based image enhancement models. Unlike other remote sensing data, high dynamic range remote sensing data refers to remote sensing image data that is taken in a simulated high dynamic range imaging environment for the application of high dynamic range technology and is in urgent need of enhancement, including different degrees of dark light and overexposed areas.
[0004] Image enhancement technology based on deep learning requires a large amount of data to train the model. However, the collection of real paired images is difficult and requires high time and labor costs. In addition, there are two problems with real datasets: 1. The shooting scenes of real data are limited. For extreme illumination images, the diversity of shooting scenes is limited by the need to ensure that there is no displacement between the extreme illumination image and its corresponding normal illumination image during imaging, so it is necessary to ensure that the imaging device and the shooting scene are relatively still; for images containing shadow areas caused by the solar altitude angle, the diversity of shooting scenes is limited by the above factors and the need to select scenes with large terrain changes during imaging. Due to the combined influence of the solar altitude angle and the height of the terrain, it is difficult to obtain remote sensing images containing extreme illumination areas and comparison images without extreme illumination areas in the same scene. This greatly increases the cost of capturing image data and limits the choice of shooting scenes.
[0005] 2. The captured image data is affected by the characteristics of the imaging device and is difficult to migrate to other imaging devices. Different imaging devices have different hardware characteristics, and the captured data also has different noises. This makes it difficult to achieve consistent performance and effects when the algorithm model learned from the image data captured by one imaging device is applied to the images captured by another imaging device. In other words, the algorithm model cannot be adaptively generalized for the currently deployed imaging devices, resulting in failure to achieve optimal performance. The neural network model based on deep learning requires a large amount of data for training to ensure its performance, so it is necessary to produce simulated data that is as close as possible to the extreme illumination images captured by real imaging devices.
[0006] Current realistic data synthesis methods usually only process a single task. For example, the original image is converted into a dark image or an overexposed image. However, for remote sensing image data, its size can usually reach hundreds of millions of pixels, and its image may contain both dark and overexposed areas. If only pure dark-overexposed images are used as training sets, it may not be close to the imaging range of remote sensing images under real lighting conditions, resulting in insufficient model processing capabilities for high dynamic range data.
[0007] In order to generate synthetic data that is closer to the actual remote sensing imaging situation, there is an urgent need for a method that can generate synthetic data that simultaneously contains regular, dark, and overexposed scenes, with smooth and natural pixel values in the transition area. Summary of the invention
[0008] The purpose of the present invention is to address the defects and shortcomings of the prior art. In order to solve the defect that the existing method only generates synthetic data for a single extreme illumination situation and cannot fit the situation where shadows, overexposure and normal areas exist in actual large-scale remote sensing images, a high dynamic range remote sensing image data synthesis method and system guided by a smoothing mask are creatively proposed.
[0009] The present invention provides a high dynamic range training data synthesis scheme that is more in line with the actual data illumination conditions for the image enhancement task of large-scale remote sensing images, improves the quality of the data set, and makes the image enhancement model more suitable for processing real high dynamic range remote sensing imaging data.
[0010] In order to achieve the above object, the present invention adopts the following technical solutions.
[0011] A smooth mask-guided high dynamic range remote sensing image data synthesis method comprises the following steps: Step 1: Generate basic data by inputting the original image to generate the corresponding overexposed image and dark light image.
[0012] Among them, for normal remote sensing image data, an overexposed image is generated by multiplying the brightness coefficient parameter.
[0013] For normal remote sensing image data, the inverse process of image signal processing is simulated through the full-element physical quantity noise model and the parameters obtained by sensor calibration. The RGB remote sensing image is inversely processed into a RAW image, and then the brightness is lowered and noise is added using the noise model to synthesize low-light noise data.
[0014] Among them, the parameter calibration of the sensor can be obtained by shooting flat field frames and bias frames.
[0015] In addition, the calibrated sensor parameters can be used to perform noise addition operations on normal image data.
[0016] Step 2: Generate a random seed matrix and upsample it to a preset size to obtain a smoothing mask.
[0017] Among them, the preset upsampling result size is the input data size of subsequent other computer vision tasks.
[0018] Step 3: Remap the generated mask result to the value range space of [0,1], and use the mapped mask to perform weighted fusion on the image to generate the final high dynamic range remote sensing image synthesis result.
[0019] At the same time, the present invention proposes a smooth mask guided high dynamic range remote sensing image data generation system, including an overexposure data generation module, a dark light data generation module, a smooth mask generation module and a basic data fusion module. Among them, the output ends of the overexposure data generation module, the dark light data generation module and the smooth mask generation module are all connected to the input end of the basic data fusion module. Beneficial Effects
[0020] The present invention effectively solves the deficiency of existing data synthesis methods that only generate synthetic remote sensing data for a single lighting scene, provides a high dynamic range remote sensing image data synthesis solution that is closer to the actual situation for large-scale remote sensing image enhancement tasks, and helps to train image enhancement models for high dynamic range real remote sensing data. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 It is the overall flow chart of the method of the present invention; Figure 2 It is a composition diagram of the system of the present invention. DETAILED DESCRIPTION
[0022] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.
[0023] Implementation example Figure 1 As shown, a smooth mask-guided high dynamic range remote sensing image data synthesis method comprises the following steps: Step S10: basic data generation, generating corresponding over-exposure images and dark-light images through the input original image.
[0024] Step S10 further includes the following two steps: Step S11: for normal remote sensing image data, an overexposed image is generated by multiplying the brightness coefficient parameter.
[0025] Specifically, the overexposed image is synthesized using random parameters. and The two parameters generate an overexposure map, and the generation of overexposure data is shown in the following formula: (1) in, is the generated overexposed image; is the original image after cutting; is the brightness coefficient, which represents the scaling factor of the image pixel value; is the offset, the default value is 0.
[0026] Step S12: For normal remote sensing image data, the inverse process of the image signal processing module is simulated through the full-element physical quantity noise model and the parameters obtained by sensor calibration, the RGB remote sensing image is inversely processed into a RAW image, and then the brightness is lowered and noise is added using the noise model to synthesize low-light noise data.
[0027] Specifically, in this embodiment, a noise calibration and modeling scheme based on a physical model is adopted. For noise model establishment, in the process of using a camera to shoot an image, the RAW data obtained by shooting Proportional to the number of photons incident on the sensor , and the ratio is affected by the camera's analog gain and digital gain.
[0028] Assume that the imaging system gain is , considering various noises, the process can be expressed as: (2) in, Represents the sum of all noise during the capture process.
[0029] The present invention takes into account the photon shot noise in the imaging process , read noise , with line noise The main sensor stripe noise and quantization noise generated during digital-to-analog conversion . Shot Noise is the noise at the photon level, , as well as is considered as noise at the digital signal level. Therefore, the noise is expressed as: (3) Furthermore, for noise parameter calibration, in order to accurately model different sensors of various remote sensing shooting equipment, it is necessary to calibrate noise parameters for different sensors. For each sensor, two special calibration frames need to be shot to complete parameter calibration, including flat field frame and bias frame.
[0030] Imaging system gain It reflects the relationship between the grayscale value of the captured image and the number of photons captured by the lens. For a given sensor, this method takes a series of flat-field frames with different exposure times at a fixed ISO.
[0031] Among them, the flat field frame is a picture taken when the light is evenly shining on the sensor. In actual shooting, the camera is fixed on a tripod, white paper is pasted on a wall evenly illuminated by natural light (or DC light source) as the shooting target, and the mobile phone is remotely controlled to adjust the exposure time and perform shooting to avoid lens shaking. In addition, in order to reduce the impact of uneven lighting, the focus distance of the lens is adjusted to infinity.
[0032] The frame is a picture taken in complete darkness, which can be used to estimate other noise parameters of the sensor that are independent of the lighting conditions, namely and Corresponding parameters. In the actual acquisition process, it is necessary to cover the lens and shoot in a dark room.
[0033] Furthermore, according to the noise model assumption of this method, the flat-field frame RAW image Variance Theoretical truth value There is a linear relationship between the variances of Therefore, The variance and Perform a linear fit, and the slope is the system gain .
[0034] In the actual shooting process, since it is impossible to ensure that the paper surface and lighting are completely uniform, in order to obtain a more accurate variance , by taking two identical frames at each exposure time and subtracting the two frames to reduce the impact of non-uniformity, and only intercepting the middle 200x200 area of each flat field frame. Considering the nature of variance, the residual map obtained by subtracting the two frames needs to be divided by To ensure the correct variance, the median of the pixels is taken as the true value after averaging the two frames.
[0035] To estimate the read noise (including and ) and use the minimum exposure time to shoot bias frames with ISO 50~6400. In order to estimate the parameters related to row noise, the mean of each row of the bias frame can be calculated, and the mean of all rows of the same frame can be used to fit the zero-mean Gaussian distribution using maximum likelihood estimation to obtain the standard deviation of the row noise. Since for a certain row, except for the row noise, the remaining noise is zero-mean, the row noise can be estimated by the pixel mean of each row in the RAW data. After completing the row noise calibration, it is necessary to subtract this component and then continue to calibrate other noise parameters.
[0036] Furthermore, the remaining read noise Estimation and noise parameter calibration. In order to estimate the readout noise more accurately, Tukey Lambda distribution is used to fit the distribution and size of the noise. Tukey Lambda distribution is a distribution family that can be used to fit a series of common distributions. The parameter that controls the distribution category is the shape parameter , by changing The specific shape of the distribution can be determined in a way that
[0037] After determining the shape parameters, the standard deviation parameters of the read noise are determined by maximum likelihood estimation, so that the probability distribution model can fit the distribution of the sample points. At this point, the corresponding system gain can be calculated for the determined ISO value. and other noise parameters.
[0038] Furthermore, in order to determine the various noise parameters at any ISO, it is necessary to further estimate the joint distribution of system gain and noise parameters. After analysis, it can be found that ISO and system gain There is an accurate positive proportional relationship. Therefore, taking ISO=800 as the benchmark, we can get ISO= The corresponding system gain is : (4) For other noise parameters, the shape parameter of Tukey Lambda distribution is found to be Basically unaffected by ISO, so only the line noise standard deviation can be considered and readout noise standard deviation With system gain Relationship between system gain , and Take the logarithm respectively, perform linear least squares fitting on the logarithmic scale, and use a Gaussian distribution with the least squares result as the mean to calibrate the parameters and Perform estimation and sampling.
[0039] Step S20: Generate a random 4x4 seed matrix, upsample it to a preset size, and normalize the generated result to obtain a smoothing mask; Specifically, in the upsampling process, the bicubic interpolation method is used for interpolation operation, which is used to perform cubic spline interpolation on pixels within a 4x4 pixel area when the image is enlarged. This interpolation method can effectively avoid the occurrence of aliasing during the upsampling process and has good stability, convergence and smoothness.
[0040] Step S30: re-map the generated mask result to the value range space of [0,1], and use the mapped mask to perform weighted fusion on the image to generate the final high dynamic range remote sensing image synthesis result; Specifically, for the generated smoothing mask, since the adjacent pixel values are not actually continuous in a strict sense, if the mask is used directly to weight the image, the difference between overexposure, dark light, and original image pixels is too large, which will lead to obvious image stratification and fail to generate a more natural image. Therefore, in order to synthesize a more natural high dynamic range image, two quadratic functions are introduced to map the value of the smoothing mask as the final weight.
[0041] The specific synthesis method is: when the mask strength is 0.5, the corresponding pixels of the synthesized image use the original image. When the mask strength is greater than 0.5, the overexposed image is mixed with the original image. In particular, when the mask strength is 1, the overexposed image is used completely. When the mask strength is less than 0.5, the dark light image is mixed with the original image. In particular, when the mask strength is 0, the dark light image is used completely. In order to reduce the difference in pixel values between the extreme illumination conditions and the original image, when mixing images, the proportion of extreme illumination images is appropriately increased through quadratic function mapping weights. The process can be expressed as: in, is the value of the mask, is the weight after mapping, is the original remote sensing image that has not been processed. To generate an overexposed image, To generate a dark light image, This is the final mixed result image.
[0042] Figure 2The system composition diagram of the device of the present invention is as follows: The high dynamic range remote sensing image data generation system guided by a smooth mask comprises an overexposure data generation module M10, a dark light data generation module M20, a smooth mask generation module M30 and a basic data fusion module M40.
[0043] in: The overexposure data generating module M10 is used to generate overexposure image data through input normal remote sensing image data.
[0044] The dark-light data generating module M20 is used to generate dark-light image data by inputting normal remote sensing image data.
[0045] The smoothing mask generating module M30 is used to generate a smoothing mask for guiding image fusion.
[0046] The basic data fusion module M40 is used to remap the smoothing mask and guide the fusion of normal, dark and overexposed images into the final high dynamic range remote sensing image data.
[0047] The connection relationship between the above modules is: The output ends of the overexposure data generating module M10 , the dark light data generating module M20 , and the smoothing mask generating module M30 are all connected to the input end of the basic data fusion module M40 .
[0048] The present invention can solve the problem that existing data synthesis methods can only generate synthetic remote sensing data for a single lighting scene, and provide a high dynamic range remote sensing image data synthesis solution that is closer to the actual situation for large-scale remote sensing image enhancement tasks, which is helpful for training image enhancement models of high dynamic range real remote sensing data.
Claims
1. A smoothing mask-guided high dynamic range remote sensing image data synthesis method, characterized in that: The following steps are involved: Step 1: Basic data generation, generate corresponding over-exposure images and dark light images through the input original image; for normal remote sensing image data, the over-exposure image is generated by multiplying the brightness coefficient parameter; the sensor parameter calibration is obtained by shooting flat field frames and bias frames; for normal remote sensing image data, the inverse process of image signal processing is simulated through the full-element physical quantity noise model and the parameters obtained for sensor calibration, the RGB remote sensing image is inversely processed into a RAW image, and then the brightness is lowered and the noise model is used to add noise to synthesize low-light noisy data; Step 2: Generate a random seed matrix, upsample it to a preset size, and obtain a smoothing mask; the preset upsampling result size is the input data size of subsequent other computer vision tasks; Step 3: Remap the generated mask result to the value domain space of [0,1], and use the mapped mask to perform weighted fusion on the image to generate the final high dynamic range remote sensing image synthesis result.
2. The smooth mask guided high dynamic range remote sensing image data synthesis method according to claim 1, characterized in that: In step 1, the overexposed image is synthesized using random parameters as guidance; use and The two parameters generate an overexposure map, and the generation of overexposure data is shown in the following formula: in, is the generated overexposed image; is the original image after cutting; is the brightness coefficient, which represents the scaling factor of the image pixel value; is the offset, the default value is 0.
3. The smooth mask guided high dynamic range remote sensing image data synthesis method according to claim 1, characterized in that: In step 1, for noise model establishment, in the process of using the camera to shoot images, the RAW data obtained is Proportional to the number of photons incident on the sensor , and the ratio is affected by the camera analog gain and digital gain; let the imaging system gain be , considering various noises, the process can be expressed as: in, Represents the sum of all noise during the capture process; considers the photon shot noise during the imaging process , read noise , with line noise The main sensor stripe noise and quantization noise generated during digital-to-analog conversion ; Shot noise is the noise at the photon level, , as well as Considered as noise at the digital signal level, the noise is expressed as: Noise parameter calibration is performed for different sensors. For each sensor, two special calibration frames are taken to complete parameter calibration, including flat field frame and bias frame. Imaging system gain Reflects the relationship between the grayscale value of the captured image and the number of photons captured by the lens; The value of the flat-field frame is taken with different exposure times for a known sensor under a fixed ISO. According to the noise model assumption, the flat-field frame RAW image Variance Theoretical truth value There is a linear relationship between the variances of ;right The variance and Perform a linear fit, and the slope is the system gain ; After determining the shape parameters, the standard deviation parameters of the readout noise are determined by maximum likelihood estimation, so that the probability distribution model can fit the distribution of the sample points; at this point, the corresponding system gain can be calculated for the determined ISO value And other noise parameters; To determine the various noise parameters at any ISO, it is necessary to further estimate the joint distribution of system gain and noise parameters, ISO and system gain It is an accurate positive proportional relationship. Taking ISO=800 as the benchmark, we get ISO= The corresponding system gain is : For other noise parameters, only the row noise standard deviation is considered and readout noise standard deviation With system gain Relationship between system gain , and Take the logarithm respectively, perform linear least squares fitting on the logarithmic scale, and use a Gaussian distribution with the least squares result as the mean to calibrate the parameters and Perform estimation and sampling.
4. The method for synthesizing high dynamic range remote sensing image data guided by smooth mask according to claim 3, characterized in that: A flat-field frame is a picture taken when the light hits the sensor evenly; The bias frame is a picture taken in complete darkness and is used to estimate other noise parameters of the sensor that are independent of the lighting conditions, namely and The corresponding parameters.
5. The smooth mask guided high dynamic range remote sensing image data synthesis method according to claim 3, characterized in that: To estimate the relevant parameters of the readout noise, bias frames with ISO range of 50 to 6400 were captured using the minimum exposure time; To estimate the relevant parameters of the row noise, the mean of each row of the bias frame is calculated, and the mean of all rows of the same frame is used to fit the zero-mean Gaussian distribution using maximum likelihood estimation to obtain the standard deviation of the row noise; In RAW data, the row noise is estimated using the pixel mean of each row. After completing the row noise calibration, the calibrated row noise needs to be subtracted to continue calibrating other noise parameters.
6. The smooth mask guided high dynamic range remote sensing image data synthesis method according to claim 3, characterized in that: Tukey Lambda distribution is used to fit the distribution and size of noise; Tukey Lambda distribution is a distribution family used to fit a series of common distributions, in which the parameter that controls the distribution category is the shape parameter , by changing The specific shape of the distribution can be determined in a way that 7. The smooth mask guided high dynamic range remote sensing image data synthesis method according to claim 3, characterized in that: To obtain the variance , by taking two identical frames at each exposure time and subtracting the two frames to reduce the impact of non-uniformity, and only intercepting the middle 200x200 area of each flat field frame; the residual map obtained by subtracting the two frames needs to be divided by , to ensure the correct variance; average the two frames and take the median of the pixels as the true value.
8. The smooth mask guided high dynamic range remote sensing image data synthesis method according to claim 1, characterized in that: In step 2, a random 4x4 seed matrix is generated, upsampled to a preset size, and the generated result is normalized to obtain a smoothing mask; during the upsampling process, the bicubic interpolation method is used for interpolation, which is used to perform cubic spline interpolation on pixels within a 4x4 pixel neighborhood when the image is enlarged.
9. The smooth mask guided high dynamic range remote sensing image data synthesis method according to claim 1, characterized in that: In step 3, two quadratic functions are introduced to map the values of the smoothing mask as the final weights; The specific synthesis method is: when the mask strength is 0.5, the corresponding pixels of the synthesized image use the original image. When the mask strength is greater than 0.5, the overexposed image is mixed with the original image. When the mask strength is 1, the overexposed image is used completely. When the mask strength is less than 0.5, the dark light image is mixed with the original image. When the mask strength is 0, the dark light image is used completely. In order to reduce the difference in pixel values between the extreme illumination case and the original image, when mixing images, the proportion of the extreme illumination image is appropriately increased through the quadratic function mapping weight. The process is expressed as: in, is the value of the mask, is the weight after mapping, is the original remote sensing image that has not been processed. To generate an overexposed image, To generate a dark light image, This is the final mixed result image.
10. A smooth mask-guided high dynamic range remote sensing image data synthesis system, characterized in that: The invention comprises an overexposure data generation module (M10), a dark light data generation module (M20), a smoothing mask generation module (M30) and a basic data fusion module (M40); wherein: the overexposure data generation module (M10) is used to generate overexposure image data by inputting normal remote sensing image data; the dark light data generation module (M20) is used to generate dark light image data by inputting normal remote sensing image data; the smoothing mask generation module (M30) is used to generate a smoothing mask for guiding image fusion; the basic data fusion module (M40) is used to remap the smoothing mask and guide the fusion of normal, dark light and overexposure images into final high dynamic range remote sensing image data; the connection relationship between the above modules is: the output ends of the overexposure data generation module (M10), the dark light data generation module (M20) and the smoothing mask generation module (M30) are all connected to the input end of the basic data fusion module (M40).
Citation Information
Patent Citations
HDR (High Dynamic Range) video generation method applicable to LDR (Low Dynamic Range) video
CN106506983A
HDR image fusion method based on a plurality of LDR images with different exposure degrees
CN107220956A
Single-frame high-dynamic range image generation method
CN108305232A
Near-real-time remote sensing estimation method and device for crop agronomy phenological period
CN115761508A
Method and device for generating high dynamic range image, storage medium and electronic equipment
CN115797200A
