Wide-spectrum image simulation method

Abstract

The invention relates to a wide-spectrum image simulation method. The method comprises the following specific steps: shooting wide-spectrum Raw data, and fixing shooting parameters; shooting visible light Raw data; calculating the quotient of the visible light Raw data and the wide spectrum Raw data to obtain a simulation parameter eta; calculating the difference between the visible light Raw data and the wide-spectrum Raw data to obtain near-infrared Raw data; calculating the average value of the sub-channels of the near-infrared Raw data, and comparing the average value with the overall average value to obtain three weighted simulation parameters; performing inverse ISP transformation on the picture in the visible light near-infrared data set to a Raw domain; coupling the Raw domain data to a wide spectrum Raw domain; and performing forward ISP conversion on the wide-spectrum Raw domain data to obtain a simulation result. According to the method, an existing visible light-near infrared data set can be simulated into wide-spectrum data, the method can adapt to different wide-spectrum sensors and illumination environments, good expansibility and applicability are achieved, and the problem that wide-spectrum image training data are insufficient is effectively solved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and relates to a wide-spectrum image simulation method. Background Art

[0002] Weak-light imaging technology plays a key role in many important fields, such as environmental monitoring, biological research, national defense and military, and security monitoring. However, the main challenge it faces is that under extremely weak light conditions, the number of photons is scarce, resulting in a low signal-to-noise ratio, which affects the imaging quality. At present, there are two main methods to solve this problem: the first method is to study the imaging noise model under weak light conditions and design corresponding algorithms to reduce imaging noise. The second method is to increase the amount of light during imaging by expanding the spectral acquisition range, that is, using wide-spectrum imaging technology, thereby improving the signal-to-noise ratio.

[0003] Broadspectral imaging technology allows the sensor to image over a wide spectral range (400-1100nm), which typically includes visible light (400-780nm) and near-infrared light (780-1100nm). In traditional imaging systems, in order to collect only visible light, a near-infrared cutoff filter is placed in front of the sensor to filter out additional near-infrared components. However, if this filter is removed, the sensor can operate over a wide spectral range, significantly increasing the amount of light entering.

[0004] Although wide-spectrum imaging technology can improve the imaging quality under low-light conditions, it also brings about the problem of imaging color degradation. The presence of near-infrared light makes the imaging color significantly different from visible light, and color is a very important aspect of human visual perception. Therefore, the key to whether wide-spectrum imaging technology can be widely used lies in whether the problem of color degradation can be solved. Acquiring wide-spectrum images to increase the light throughput of low-light imaging and using algorithms to solve the problem of color degradation in wide-spectrum imaging is an effective way to improve the signal-to-noise ratio of low-light imaging.

[0005] In recent years, artificial intelligence technology, especially deep learning, has made significant progress in the field of image processing and has been used to solve various image degradation problems. Deep learning is a data-driven algorithm, and its model training relies on a large amount of training data. However, there is currently a lack of sufficient wide-spectrum image datasets, as well as a lack of effective methods for modeling and simulating wide-spectrum images. These are issues that need further research and resolution. Summary of the invention

[0006] In order to solve the above technical problems existing in the prior art, the present invention proposes an innovative wide-spectrum image simulation method.

[0007] This method is mainly divided into two parts: the first part is to calibrate the simulation parameters using real wide-spectrum data, and the second part is to simulate the wide-spectrum data using the calibrated simulation parameters.

[0008] The specific technical solution adopted by the present invention is as follows:

[0009] A wide spectrum image simulation method, the method comprising the following steps:

[0010] Step 1: Use a wide spectrum camera to capture a wide spectrum image and save it as data R wide ;

[0011] Step 2: Under the same shooting parameters and position as step 1, a near infrared cutoff filter is added to the front end of the wide spectrum camera to shoot a visible light image and save it as data R vis ;

[0012] Step 3: Calculate data R vis With data R wide The average value ratio is used to calibrate the simulation coefficient η of the wide spectrum mixing ratio;

[0013] Step 4: Calculate data R wide Subtract the data R vis The difference is obtained to obtain the near-infrared Raw domain data R nir ;

[0014] Step 5: Calculate the data R nir The three channels and data R nir The overall average ratio is used to calibrate the three simulation coefficients w r ,、w g , and w b ;

[0015] Step 6: Obtain paired visible light images I vis and near infrared image I nir image;

[0016] Step 7: transform the image in step 6 into the Raw domain through inverse image signal processing, and record them as images

[0017] Step 8: Near infrared image Perform energy normalization to ensure that its energy is consistent with the visible light image The average energy of

[0018] Step 9, using the simulation coefficient w r ,、w g and w b The near-infrared component is weighted to simulate the spectral response in wide-spectrum imaging and obtain a weighted near-infrared image. c∈(r,g,b);

[0019] Step 10: Spectral domain coupled visible light image And weighted near infrared image Get wide spectrum Raw domain data

[0020] Step 11: The simulation results of wide spectrum data are obtained by transforming the image signal into sRGB domain through forward image signal processing. wide .

[0021] In actual operation, the simulation coefficients may vary due to differences in ambient lighting and sensor response curves, so multiple calibrations may be required to obtain the optimal parameters. The present invention can simulate the existing visible light-near infrared data set into wide-spectrum data through the above four simulation parameters. In addition, the calibration method of the simulation coefficients provided by the present invention can adapt to different wide-spectrum sensors and lighting environments, has good scalability and applicability, and effectively solves the problem of insufficient wide-spectrum image training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of the wide spectrum image simulation method of the present invention;

[0023] Figure 2 It is the simulation parameter calibration process of the present invention;

[0024] Figure 3 In the embodiment of the present invention, (a) the visible light image I vis Reverse ISP transformation flow chart, (b) Near infrared image I nir Reverse ISP transformation flow chart, (c) Raw domain coupling image Forward ISP transformation flow chart;

[0025] Figure 4 A flow chart for generating a wide spectrum image according to the present invention. DETAILED DESCRIPTION

[0026] The method of the present invention is described in detail below in conjunction with the accompanying drawings and actual operation process.

[0027] This embodiment provides a wide spectrum image simulation method, which specifically includes the following steps:

[0028] Step 1: Calibrate simulation parameters by shooting wide-spectrum data. The four simulation parameters are obtained by removing or adding near-infrared cutoff filter (IRCF) during shooting. wide and R vis The calibration process is shown as follows Figure 2 shown.

[0029] Step 11: First, capture the wide spectrum Raw data R wide: Remove the IRCF on the front of the camera to ensure that there is no overexposure during shooting, use the default white balance settings, fix the exposure time, sensitivity, aperture, and gain parameters, and shoot wide-spectrum Raw data R wide .

[0030] Step 12: Then capture the visible light Raw data R vis : Keep the parameters and camera angle used in step 11 unchanged, place IRCF at the front of the camera, and shoot visible light Raw data R vis .

[0031] Step 13, by calculating R wide With R vis The simulation parameter η is obtained by taking the mean ratio of η∈(0,1). mean(·) is the image mean operation, and η is the wide spectrum mixing ratio simulation coefficient, which indicates the proportion of visible light components in the wide spectrum Raw domain data.

[0032] Step 14, by calculating R wide With R vis The difference is the data R nir : R nir =R wide -R vis , calculate R nir The average of each channel is divided into the overall average value (w r ,w g ,w b )Three simulation parameters: c∈(r,g,b). nir ) c Represents R nir Different channels of mean((R nir ) c ) is the data R nir Calculate the mean of different channels, mean(R nir ) is the data R nir The overall mean of the three channels is calculated.

[0033] Step 2: simulate the wide spectrum data using the calibrated simulation parameters.

[0034] like Figure 1 Although paired visible-light near-infrared datasets are easy to obtain, there may be registration problems during the data capture process. Therefore, the paired images in the dataset (such as the public dataset on the Internet) are first registered and cropped, and the image pairs that are difficult to register in the dataset are filtered out. The images after registration and screening can be used as the initial simulation data.

[0035] The paired near-infrared and visible light images in the dataset have gone through the camera’s in-camera signal processing program ISP (image signal process). In step 11, during the wide-spectrum image acquisition process, the near-infrared and visible light components are mixed and acquired to become wide-spectrum data without going through the ISP program. Therefore, it is necessary to perform an inverse ISP transformation on the visible light and near-infrared data in the dataset to return the data to the Raw domain.

[0036] For visible light image I vis ,like Figure 3 of(a)I vis As shown in the inverse ISP transformation process, this process includes inverse tone mapping, inverse gamma transformation, inverse color correction matrix (CCM) transformation, inverse white balance and inverse gain transformation. This embodiment implements this process through a series of mathematical formulas:

[0037] Inverse Tone Mapping:

[0038] Inverse Gamma Transform:

[0039] Inverse CCM transform:

[0040] Inverse WB transform:

[0041] Inverse gain adjustment:

[0042] For near infrared image I nir ,like Figure 3 (b)I nir As shown in the inverse ISP transformation flow chart, the inverse ISP transformation only includes brightness transformations such as tone mapping, inverse gamma transformation, and inverse gain transformation, and does not include color transformations such as inverse CCM and inverse WB. The mathematical formula is similar to the above and will not be repeated. Finally, the data is obtained

[0043] The coefficients of each inverse transformation are related to the calibration of specific camera parameters, and their parameter calibration is beyond the scope of this method.

[0044] According to the above inverse ISP transformation, the Raw domain data is obtained like Figure 4 As shown, After energy normalization and RGB channel weighting operations and Perform spectral domain coupling to obtain wide spectrum Raw domain data The specific implementation process is as follows.

[0045] Energy normalization: Where mean(·) is the operation of calculating the mean of the image, specifically:

[0046] By energy normalization, tend to be consistent to facilitate subsequent spectral domain fusion.

[0047] RGB channel weighting: c∈(r,g,b), where concat(·) is the channel concatenation operation. The near-infrared component of the wide-spectrum data is obtained by weighting and concatenating the three channels. Weighting coefficient (w r ,w g ,w b ) is the simulation coefficient obtained in step 14.

[0048] Spectral domain coupling operation:

[0049] In the Raw domain coupling image After that, the wide spectrum data is transformed to the appropriate sRGB through forward ISP transformation to adapt to subsequent image processing. The forward ISP transformation order is opposite to the reverse ISP, such as Figure 3 (c) As shown in the forward ISP transformation flow chart,

[0050] Gain Adjustment:

[0051] WB Transformation:

[0052] CCM transformation:

[0053] Gamma transformation:

[0054] Tone Mapping:

[0055] After the forward ISP process, the final wide spectrum simulation result I is obtained wide . Figure 1 The effectiveness of the simulation method of the present invention is demonstrated in the figure. The wide spectrum simulation results are highly consistent with the actual wide spectrum images, and the effect is good.

Claims

1. A wide spectrum image simulation method, characterized in that: The method comprises the following steps: Step 1: Use a wide spectrum camera to capture a wide spectrum image and save it as data R wide ; Step 2: Under the same shooting parameters and position as step 1, a near infrared cutoff filter is added to the front end of the wide spectrum camera to shoot a visible light image and save it as data R vis ; Step 3: Calculate data R vis With data R wide The average value ratio is used to calibrate the simulation coefficient η of the wide spectrum mixing ratio; Step 4: Calculate data R wide Subtract the data R vis The difference is obtained to obtain the near-infrared Raw domain data R nir ; Step 5: Calculate the data R nir The three channels and data R nir The overall average ratio is used to calibrate the three simulation coefficients w r ,,w g , and w b ; Step 6: Obtain paired visible light images I vis and near infrared image I nir image; Step 7: transform the image in step 6 into the Raw domain through inverse image signal processing, and record them as images and Step 8: Near infrared image Perform energy normalization to ensure that its energy is consistent with the visible light image The average energy of Step 9, using the simulation coefficient w r ,,w g and w b The near-infrared component is weighted to simulate the spectral response in wide-spectrum imaging and obtain a weighted near-infrared image. c∈(r,g,b); Step 10: Spectral domain coupled visible light image And weighted near infrared image Get wide spectrum Raw domain data Step 11: The simulation results of wide spectrum data are obtained by transforming the image signal into sRGB domain through forward image signal processing. wide .

2. A wide spectrum image simulation method according to claim 1, characterized in that: In step 3, the calculation method of the wide spectrum mixing ratio simulation coefficient η is as follows: Where mean(·) is the operation to calculate the mean of the image.

3. A wide spectrum image simulation method according to claim 1, characterized in that: In step 5, the three simulation coefficients w r 、w g and w b The calibration method is as follows: Among them (R nir ) c Representative data R nir Different channels of mean((R nir ) c ) is the data R nir Calculate the mean of different channels, mean(R nir ) is the data R nir The three channels are averaged as a whole.

4. A wide spectrum image simulation method according to claim 1, characterized in that: In step 7, for the visible light image I vis The inverse image signal processing includes inverse tone mapping, inverse Gamma transformation, inverse color correction matrix transformation, inverse white balance and inverse gain transformation; for the near infrared image I nir , the inverse image signal processing includes tone mapping, inverse gamma transformation and inverse gain transformation.

5. The method for simulating a wide spectrum image according to claim 1, characterized in that: In step 8, the near infrared image The method for energy normalization is as follows: Where mean(·) is the operation to calculate the mean of the image.

6. A wide spectrum image simulation method according to claim 1, characterized in that: In step 9, the simulation coefficient w is used r 、w g and w b The method for weighting the near infrared component is as follows: Where concat(·) is a channel concatenation operation.

7. A wide spectrum image simulation method according to claim 1, characterized in that: In step 10, the spectral domain coupling method is as follows:

Images