A wide spectrum image simulation method

Abstract

The present application relates to a kind of wide spectrum image simulation method.The specific steps of the method are as follows: wide spectrum Raw data is shot, and shooting parameter is fixed;Visible light Raw data is shot;The quotient of visible light Raw data and wide spectrum Raw data is calculated to obtain simulation parameter η;The difference of visible light Raw data and wide spectrum Raw data is calculated to obtain near-infrared Raw data;The mean ratio of near-infrared Raw data is calculated to obtain three weighted simulation parameters;The picture in visible light near-infrared data set is transformed to Raw domain by inverse ISP;Raw domain data is coupled to wide spectrum Raw domain;Wide spectrum Raw domain data is transformed to obtain simulation result by forward ISP.The method of the present application can simulate existing visible light-near-infrared data set to wide spectrum data, and can adapt to different wide spectrum sensors and illumination environment, with good expansibility and applicability, effectively solve the problem of insufficient wide spectrum image training data.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and relates to a wide spectrum image simulation method. BACKGROUND

[0002] Weak light imaging technology plays a key role in many important fields such as environmental monitoring, biological scientific research, national defense and military, and security monitoring. However, it faces the main challenge of low signal-to-noise ratio under extremely weak light conditions due to the scarcity of photons, which affects the imaging quality. Currently, there are two main methods to solve this problem: the first method is to study the imaging noise model under weak light conditions and design corresponding algorithms to reduce imaging noise. The second method is to expand the spectral acquisition range, that is, to use wide spectrum imaging technology to increase the light quantity during imaging, thereby improving the signal-to-noise ratio.

[0003] Wide spectrum imaging technology allows the sensor to image in a wider spectrum range (400-1100nm), which usually includes visible light (400-780nm) and near-infrared light (780-1100nm). In traditional imaging systems, in order to only collect visible light, a near-infrared cutoff filter is placed in front of the sensor to filter out the additional near-infrared component. However, if this filter is removed, the sensor can work in the wide spectrum range, thereby significantly increasing the light quantity.

[0004] Although wide spectrum imaging technology can improve the imaging quality under weak light conditions, it also brings the problem of color degradation in imaging. The presence of near-infrared light makes the imaging color significantly different from visible light, and color is a very important aspect of human visual perception. Therefore, the key to the wide application of wide spectrum imaging technology lies in solving the problem of color degradation. By collecting wide spectrum images to increase the light quantity of weak light imaging, and using algorithms to solve the problem of color degradation in wide spectrum imaging, it is an effective way to improve the signal-to-noise ratio of weak light imaging.

[0005] In recent years, artificial intelligence technology, especially deep learning, has made significant progress in the field of image processing and has been used to solve various image degradation problems. Deep learning is a data-driven algorithm, and its model training relies on a large amount of training data. However, there is currently a lack of sufficient wide spectrum image datasets, and there is also a lack of effective methods for modeling and simulating wide spectrum images, which are problems that need to be further researched and solved. SUMMARY

[0006] In order to solve the above technical problems existing in the prior art, the present application proposes an innovative wide spectrum image simulation method.

[0007] This method mainly consists of two parts: the first part is to shoot wide spectrum data to calibrate simulation parameters, and the second part is to simulate wide spectrum data using the calibrated simulation parameters.

[0008] The specific technical solutions adopted by the present application are as follows:

[0009] A wide-spectrum image simulation method, comprising the following steps:

[0010] Step 1, a wide-spectrum image is taken using a wide-spectrum camera and saved as data R wide ;

[0011] Step 2, a visible light image is taken under the same shooting parameters and position as step 1, with a near-infrared cut-off filter added in front of the wide-spectrum camera, and saved as data R vis ;

[0012] Step 3, the average ratio of data R vis to data R wide is calculated to calibrate the wide-spectrum mixing ratio simulation coefficient η;

[0013] Step 4, the difference between data R wide and data R vis is calculated to obtain near-infrared Raw domain data R nir ;

[0014] Step 5, the average ratio of the three channels of data R nir to the overall average of data R nir is calculated to calibrate the three simulation coefficients w r , w g , and w b ;

[0015] Step 6, paired visible light image I vis and near-infrared image I nir images are obtained;

[0016] Step 7, the images of step 6 are transformed to the Raw domain through inverse image signal processing, and are respectively denoted as images

[0017] Step 8, the near-infrared image is energy-normalized to ensure that its energy matches the average energy of the visible light image ;

[0018] Step 9, the near-infrared component is weighted using the simulation coefficients w r , w g , and w b to simulate the spectral response in wide-spectrum imaging, and a weighted near-infrared image c∈(r,g,b) is obtained;

[0019] Step 10, the spectral domain coupled visible light image and the weighted near-infrared image Obtain wide-spectrum Raw domain data

[0020] Step 11, data Transformed to sRGB domain by forward image signal processing, obtain simulation results I of wide-spectrum data wide .

[0021] In actual operation, the simulation coefficients may vary due to different ambient light and sensor response curves, so multiple calibrations may be required to obtain the best parameters. The present application can simulate the existing visible-near-infrared data set into wide-spectrum data through the above four simulation parameters. In addition, the calibration method of the simulation coefficient provided by the present application can adapt to different wide-spectrum sensors and lighting environments, has good expansibility and applicability, and effectively solves the problem of insufficient wide-spectrum image training data. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 Flowchart of the wide-spectrum image simulation method of the present application;

[0023] Figure 2 Simulation parameter calibration process of the present application;

[0024] Figure 3 (a) visible light image I vis (b) near-infrared image I nir (c) Raw domain coupled image Forward ISP transformation flowchart;

[0025] Figure 4 Flowchart of the wide-spectrum image generation method of the present application. DETAILED DESCRIPTION

[0026] The method of the present application is described in detail below in combination with the drawings and actual operation process.

[0027] The present embodiment provides a wide-spectrum image simulation method, which specifically comprises the following steps:

[0028] Step 1, calibrate simulation parameters with actual wide-spectrum data. The four simulation parameters are obtained by removing or adding near-infrared cutoff filter (IRCF) during shooting to obtain data R wide and R vis Data. The calibration process is shown in Figure 2 .

[0029] Step 11, first shoot wide-spectrum Raw data R wide: Remove the IRCF on the front of the camera to ensure no overexposure during the shooting process, use the default white balance settings, fix the shooting exposure time, sensitivity, aperture, gain parameters, and shoot wide spectrum Raw data R wide .

[0030] Step 12, then shoot visible light Raw data R vis : Keep the parameters and camera angle used in step 11 unchanged, place IRCF at the front of the camera, and shoot visible light Raw data R vis .

[0031] Step 13, by calculating R wide With R vis The simulation parameter η is obtained by taking the mean ratio of η∈(0,1). mean(·) is the image mean operation, and η is the wide-spectrum mixing ratio simulation coefficient, which represents the proportion of visible light components in the wide-spectrum raw domain data.

[0032] Step 14, by calculating R wide With R vis The difference is the data R nir :R nir =R wide -R vis , calculate R nir The average value of each channel is greater than the overall average value (w r ,w g ,w b )Three simulation parameters: c∈(r,g,b). nir ) c Represents R nir Different channels of mean((R nir ) c ) is the data R nir Calculate the mean of different channels, mean(R nir ) is the data R nir The overall mean of the three channels is calculated.

[0033] Step 2: simulate the broadband data using the calibrated simulation parameters.

[0034] like Figure 1 While paired visible-light and near-infrared datasets are readily available, they can present registration issues during data capture. Therefore, we first register and crop paired images from a dataset (such as publicly available datasets online) and filter out image pairs that are difficult to register. These registered and filtered images can be used as initial simulation data.

[0035] The paired near-infrared and visible light pictures in the data set have undergone an in-camera signal processing program ISP (image signal process). In the step 11 wide spectrum image acquisition process, the near-infrared and visible light components are mixed to be acquired as wide spectrum data without ISP program, so it is necessary to perform inverse ISP transformation on the visible light and near-infrared data in the data set to return the data to the Raw domain.

[0036] For the visible light image I vis As shown in Figure 3 (a) I vis As shown in the inverse ISP transformation flowchart, this process includes inverse tone mapping, inverse Gamma transformation, inverse color correction matrix (CCM) transformation, inverse white balance and inverse gain transformation. This embodiment realizes this process through a series of mathematical formulas:

[0037] Inverse tone mapping:

[0038] Inverse Gamma transformation:

[0039] Inverse CCM transformation:

[0040] Inverse WB transformation:

[0041] Inverse gain adjustment:

[0042] For the near-infrared image I nir As shown in Figure 3 (b) I nir As shown in the inverse ISP transformation flowchart, the inverse ISP transformation only includes tone mapping, inverse Gamma transformation, inverse gain transformation, which are luminance transformations, and does not include inverse CCM and inverse WB and other color transformations. The mathematical formulas are similar to the above and are not repeated, and finally the data

[0043] The coefficients of each inverse transformation are related to the calibration of specific camera parameters, and the parameter calibration is not within the scope of discussion of the present method.

[0044] According to the above inverse ISP transformation, the Raw domain data As shown in Figure 4 , After energy normalization and RGB channel weighting operation and spectrum domain coupling to obtain wide spectrum Raw domain data The specific implementation process is as follows.

[0045] Energy normalization: Wherein mean(·) is the mean value operation of the calculation image, and the specific operation is

[0046] By energy normalization, tends to be consistent so as to subsequent spectral domain fusion.

[0047] RGB channel weighting: c∈(r,g,b), where concat(·) is the channel concatenation operation. By weighting and concatenating three channels to get the near-infrared component of the wide spectrum data The weighting coefficient (w r ,w g ,w b ) is the simulation coefficient obtained in step 14.

[0048] Spectral domain coupling operation:

[0049] After obtaining the Raw domain coupled image , the wide spectrum data is converted to the appropriate sRGB through the forward ISP transformation to adapt to the subsequent image processing. The sequence of the forward ISP transformation is opposite to the reverse ISP, as shown in the (c) Figure 3 forward ISP transformation flowchart, wherein,

[0050] Gain adjustment:

[0051] WB transformation:

[0052] CCM transformation:

[0053] Gamma transformation:

[0054] Tone mapping:

[0055] After the forward ISP process, the final wide spectrum simulation result I wide is obtained. Figure 1 The effectiveness of the simulation method of the application is shown in the figure, and the wide spectrum simulation result has high agreement with the wide spectrum picture actually taken, and the effect is good.​

Claims

1. A method of wide spectrum image simulation, characterized in that, The method comprises the following steps: Step 1, take a wide spectrum image using a wide spectrum camera and save as data R wide ; Step 2. Take a visible light image with a near infrared cut-off filter in front of the wide spectrum camera at the same shooting parameters and location as step 1 and save as data R vis ; Step 3, compute data R vis with the average value of data R wide calibrate the wide spectrum mixing ratio simulation coefficient η; Step 4, calculating data R wide Subtracting the difference from data R vis , the near-infrared Raw domain data R nir ; Step 5, calculate data R nir for each of the three channels and the average value of the data R nir for the whole; calibrate three simulation coefficients w r , w g and w b ; Step 6, obtaining a pair of visible light images I vis and near infrared images I nir images; Step 7, the image of step 6 is transformed to the Raw domain by inverse image signal processing, denoted as image and Step 8, Energy normalization of near infrared image ensuring that its energy matches the average energy of the visible light image ; Step 9, the near infrared component is weighted using the simulation coefficients w r , w g , and w b to simulate the spectral response in wide spectrum imaging to obtain a weighted near infrared image Step 10, Spectral Domain Coupled Visible Image and weighted near infrared image Obtain wide spectrum Raw domain data Step 11, data is transferred to the sRGB domain by forward image signal processing The simulation result I of the wide spectrum data is obtained by transforming to the sRGB domain through forward image signal processing wide .

2. A wide-spectrum image simulation method according to claim 1, characterized in that, In step 3, the calculation method of the wide-spectrum mixed ratio simulation coefficient η is as follows: Wherein mean(·) is the mean image operation.

3. The wide-spectrum image simulation method of claim 1, wherein, In step 5, the three simulation coefficients w r , w g and w b are calibrated as follows: where (R nir ) c represent the data R nir for different channels, mean((R nir ) c ) is the average of the data R nir for different channels, and mean(R nir ) is the average of the data R nir for all three channels.

4. The wide-spectrum image simulation method of claim 1, wherein, In step 7, for the visible light image I vis , the inverse image signal processing comprises inverse tone mapping, inverse Gamma transform, inverse color correction matrix transform, inverse white balance and inverse gain transform; for the near infrared image I nir , the inverse image signal processing comprises tone mapping, inverse Gamma transform and inverse gain transform.

5. The wide-spectrum image simulation method of claim 1, wherein, In step 8, the pair of near infrared images The method of energy normalization is as follows: Wherein mean(·) is the mean image operation.

6. A wide-spectrum image simulation method according to claim 5, characterized in that, In step 9, the use of the simulation coefficients w r , w g , and w b to weight the near infrared component is as follows: Wherein concat(·) is the channel splicing operation.

7. The wide-spectrum image simulation method of claim 1, wherein In step 10, the method of spectral domain coupling is as follows:

Images