A Temperature Noise Correction Method for CMOS Space Cameras Based on Attention Mechanism and LSTM

Abstract

A temperature noise correction method for CMOS space cameras based on attention mechanisms and LSTM is presented. This method relates to the field of optical remote sensing technology, specifically to the field of temperature noise correction for CMOS space cameras. The core of the method is the use of a multi-level Long Short-Term Memory (LSTM) network with an attention mechanism to explore how temperature changes affect the noise performance of CMOS space cameras under two different operating conditions: dark and bright fields. This deep learning model possesses powerful autonomous learning capabilities, capable of mining and understanding complex noise patterns hidden within massive amounts of data, thereby improving the accuracy of noise identification and calibration. Its key advantage lies in its end-to-end learning approach, automatically learning and establishing a deep, nonlinear relationship between temperature and noise directly from the raw data, without requiring manual intervention for feature extraction or designing complex correction algorithms. This significantly enhances the model's generalization ability and adaptability to unseen data scenarios.

Description

Technical Field

[0001] This invention relates to the field of optical remote sensing technology applications, specifically to the field of temperature noise correction technology for CMOS space cameras. Background Technology

[0002] In recent years, with the continuous advancement of semiconductor technology and the increasing maturity of microelectronics manufacturing processes, CMOS image sensors have achieved significant performance improvements, and their numerous advantages have become increasingly prominent, making them a strong competitor to traditional CCD image sensors in the field of aerospace camera applications. Compared to CCDs, CMOS sensors exhibit superior characteristics in terms of integration, anti-interference capability, power consumption control, cost-effectiveness, size, and weight. These advantages have greatly promoted the widespread application of CMOS in aerospace imaging systems. However, despite the significant progress made in CMOS technology, due to its inherent material structure characteristics and processing limitations, CMOS aerospace cameras are susceptible to temperature-related image signal drift in aerospace environments, especially when facing extreme temperature changes. This leads to increased image noise, severely affecting image quality and limiting the overall performance and reliability of CMOS aerospace cameras, thus posing a challenge to their further promotion and use in the aerospace field.

[0003] Therefore, research on temperature-induced noise correction in CMOS space cameras has extremely high engineering application value and scientific significance. This research direction has attracted the attention and investment of many scholars, aiming to develop more accurate and efficient temperature compensation algorithms and techniques to improve the image quality and stability of CMOS space cameras under various temperature conditions. However, current traditional temperature noise correction methods rely on manually designed rules or models, which cannot capture complex noise patterns and suffer from the problem of complex computational correction methods. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a temperature noise correction method for CMOS space cameras based on attention mechanisms and LSTM, the method comprising the following steps:

[0005] S1. Acquire dark field and light field images using a CMOS space camera;

[0006] S2. Select the temperature and average DN of the dark field image from step S1 to construct a sample set. The constructed sample set is called the dark field single pixel module sample set.

[0007] S3. Construct a multi-level LSTM network model;

[0008] S4. Generate the mean DN of the image through a fully connected layer;

[0009] S5. Select the column mean values ​​of temperature and DN values ​​of the dark field image from step S1 to construct a sample set. The constructed sample set is called the dark field column difference module sample set.

[0010] S6. Construct a single-layer LSTM network model;

[0011] S7. Use the attention mechanism to obtain the degree of influence of inter-column difference features and individual pixel DN values ​​on the dark field DN value of the image.

[0012] S8. Use a fully connected layer to generate pixel-wise DN values ​​for the dark field image;

[0013] S9. Select the column mean values ​​of temperature and DN values ​​of the light field image from step S1 to construct a sample set. The constructed sample set is called the light field module sample set.

[0014] S10. Use a fully connected layer to obtain the embedded vector representation of the column mean of the temperature and light field image;

[0015] S11. Use the attention mechanism to obtain the degree of influence of temperature and light field image DN values ​​on image light field noise, respectively.

[0016] S12. Use a fully connected layer to generate pixel-wise DN values ​​for light field image noise;

[0017] S13. Calculate the light field DN value of the image based on the acquired dark field noise and light field noise.

[0018] The beneficial effects of the method described in this invention are as follows: Addressing the noise problem generated by CMOS space cameras under varying temperature environments, this invention innovatively introduces deep learning technology for temperature noise correction, proposing a temperature noise correction method for CMOS space cameras based on attention mechanisms and LSTM. The core of this method utilizes a multi-level long short-term memory (LSTM) network with an attention mechanism to explore how temperature changes affect the noise impact on CMOS space camera performance under both dark and bright operating conditions. This deep learning model possesses powerful autonomous learning capabilities, capable of mining and understanding complex noise patterns hidden within massive amounts of data, thereby improving the accuracy of noise identification and calibration. Its key advantage lies in its end-to-end learning approach, automatically learning and establishing a deep, non-linear relationship between temperature and noise directly from raw data, without requiring manual intervention for feature extraction or designing complex correction algorithms. This significantly enhances the model's generalization ability and adaptability to unseen data scenarios, providing strong technical support for the stable operation of CMOS space cameras in complex space environments. Attached Figure Description

[0019] Figure 1A flowchart illustrating the temperature noise correction process for a CMOS space camera based on attention mechanisms and LSTM.

[0020] Figure 2 This is a comparative analysis of the predicted values ​​and experimental test values ​​of the algorithm model before and after correction. Figure 2 In the figure, (a)-(f) represent the fitting comparison between the predicted and actual values ​​of the dark field module at different temperatures;

[0021] Figure 3 This is a comparative analysis of the predicted values ​​and experimental test values ​​of the algorithm model before and after correction. Figure 3 In the diagram, (a)-(f) represent the noise reduction capabilities of the light field module for images under different radiance and temperature conditions. Detailed Implementation

[0022] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0023] like Figure 1 The diagram shown illustrates the temperature noise correction flowchart for a CMOS space camera based on attention mechanisms and LSTM. The implementation steps of the method described in this embodiment are described below:

[0024] Step 1: Acquire dark field and light field images.

[0025] In the radiation calibration laboratory, an integrating sphere with an inner diameter of 4 meters was used to precisely control the output of a uniform and adjustable light field environment to simulate the illumination conditions of the Jilin-1 Wide-Switch 02A satellite camera in space. The camera's operating temperature continuously varies between 25°C and 40°C, simulating its operating environment temperature. During the experiment, a total of 46 sets of light field images and 78 sets of dark field images were acquired from the Jilin-1 Wide-Switch 02A satellite camera. These image data comprehensively cover the camera's performance under different temperature conditions.

[0026] Step 2: Construct a sample set of single-pixel modules for dark fields.

[0027] To study the overall DN value of the image as a function of temperature, the temperature of the dark field image in step 1 and the mean DN value of the image were selected to construct a sample set.

[0028] Step 3: Construct a multi-layered LSTM network model.

[0029] A 3-layer LSTM network model is used as the dark field single-pixel module. The temperature of the image is used as the input to the network model in the form of an embedding vector, as shown in Equation 1:

[0030]

[0031] In the formula, This represents the temperature of the nth dark-field image. Represents the set of real numbers. yes The embedding vector representation input into the deep learning model, where ds is the dimension of the hidden vector of a single pixel module. It is a weight matrix. It is the bias vector. Then, The input is fed into a multi-layer LSTM network to obtain the hidden state of each pixel in the nth image. As shown in Equation 2-4, where LSTM1, LSTM2, and LSTM3 represent three-layer LSTM networks, i 1 i 2 i 3 These represent the initial state of each LSTM network layer.

[0032]

[0033]

[0034]

[0035] Step 4: Generate the mean DN of the image through a fully connected layer.

[0036] The hidden state of a single pixel in the nth image is input into the fully connected layer to obtain the embedding vector representation of the predicted DN value, as shown in Equation 5:

[0037]

[0038] In the formula, It is the weight matrix of the fully connected layer. That is, the embedding vector representation of the DN value predicted for a single pixel in the nth image.

[0039] Step 5: Construct the sample set for the dark field inter-column difference module.

[0040] To study the variation of inter-column differences between individual pixels in an image with temperature, the temperature of the dark field image in step 1 and the column mean of the image DN value were selected to construct a sample set.

[0041] Step 6: Build the network using LSTM.

[0042] A single-layer LSTM network model is used as the inter-column difference module to predict the inter-column differences of pixels in the same row of an image at the same temperature. The temperature of the image is used as the input to the network model in the form of an embedding vector, as shown in Equation 6-7:

[0043]

[0044]

[0045] dc is the dimension of the hidden vector of the inter-column difference module. express The embedding vector representation, This represents the hidden state of the LSTM network. The bias vector is represented; then the temperature embedding vector is input into the LSTM network to obtain the hidden state of the inter-column difference module.

[0046] This is the initial state of the LSTM network in the inter-column difference module.

[0047] Step 7: Use the attention mechanism to obtain the influence of inter-column difference features and individual pixel DN values ​​on the dark field DN values ​​of the image.

[0048] An attention mechanism is used to adaptively assign weights to the hidden states of inter-column differences and the embedding vector representations of single pixels that vary with temperature, as shown in Equation 8-10:

[0049]

[0050]

[0051]

[0052] In the formula, It is the embedding vector of a single pixel module, representing p n Input to the attention mechanism in the inter-column difference module; It is a weight matrix; It is the bias vector; It is the weight of the effect of a single pixel's temperature change on the DN value of the entire image. It is the pixel index, that is, the weight of the influence of inter-column differences on the DN value of the whole image; the attention function g(x,y) is shown in Equation 11:

[0053]

[0054] therefore,

[0055] In the formula, This refers to the query information for the attention mechanism in the inter-column difference module, specifically the dark field DN value of the nth image; after passing through the attention mechanism, the embedding vector representation of the entire image is obtained. As shown in Formula 12:

[0056]

[0057] In the formula, It is the embedding vector representation of the entire image.

[0058] Step 8: Use a fully connected layer to generate pixel-by-pixel DN values ​​for the dark field image.

[0059] The embedding vector representation of the dark field image is passed through a fully connected layer to obtain the dark field DN value of the image to be predicted, as shown in Equation 13:

[0060]

[0061] In the formula, Here, dy is the DN value to be predicted for the image, and dy is the image dimension. It is a weight matrix. It is the bias vector.

[0062] Step 9: Construct a sample set for the light field module.

[0063] To study the variation of image noise in a light field with temperature and radiance, the column mean of temperature and DN value of the light field image in step 1 was selected to construct a sample set, where the radiance factor is represented by the DN value of the light field image.

[0064] Step 10: Use a fully connected layer to obtain the embedded vector representation of the column mean of the temperature and light field images.

[0065] Under light field conditions, the DN value of the output image of a CMOS space camera is affected by factors such as temperature, dark current, and entrance pupil photon number, resulting in noise. To correct this noise under light field conditions, the light field module of this method employs a feedforward neural network based on an attention mechanism to correct the noise in the output image of the CMOS space camera under light field conditions based on the image DN value under both temperature and light field conditions. The temperature and light field DN values ​​are respectively processed through fully connected layers to obtain their embedding vector representations, as shown in Equations 14-15:

[0066]

[0067]

[0068] In the formula, This represents the temperature of the nth light field image; It is a weight matrix; It is temperature The embedding vector representation, where db is the dimension of the hidden vector of the light field module; d is the DN value of the nth image; dy is the dimension of the image column mean; It is a weight matrix. It is the DN value of the light field. The embedding vector representation.

[0069] Step 11: Use the attention mechanism to obtain the degree of influence of temperature and radiance, i.e., the DN value of the light field image, on the light field noise of the image.

[0070] An attention mechanism is used to adaptively acquire different weights for temperature and DN values ​​in the light field image, such as...

[0071] As shown in Equation 16-17:

[0072]

[0073]

[0074] In the formula, This refers to the query information for the attention mechanism in the light field module, specifically the light field noise of the nth image. It is the weight of the effect of temperature on the light field noise. This represents the weight of the light field DN value on the light field noise. After an attention mechanism, the light field embedding vector representation of the nth image is obtained. As shown in Formula 18:

[0075]

[0076] By inputting the embedding vector representation of the light field into a feedforward neural network, the hidden state of the image unaffected by the dark field is obtained. As shown in Formula 19:

[0077]

[0078] In the formula, It is the weight matrix of the feedforward neural network. is the bias vector of the feedforward neural network, and softplus(·) is the activation function of the feedforward neural network, as shown in Equation 20:

[0079] softplus(x) = log(1 + e) x );

[0080] Step 12: Use a fully connected layer to generate pixel-wise DN values ​​for light field image noise.

[0081] Hidden state The input is fed into the fully connected layer to obtain the DN value of the optical field noise. As shown in Formula 21:

[0082]

[0083] In the formula, It is a weight matrix.

[0084] Step 13: Calculate the true effective light field DN value of the image based on the acquired dark field noise and light field noise.

[0085] Based on the dark field DN value predicted by the dark field module and the light field noise predicted by the light field module, the light field DN value of the nth image is obtained, as shown in Formula 22:

[0086]

[0087] in It is the DN value of the nth light field image. This is the dark field noise calculated by the dark field module. It is the light field noise calculated by the light field module. It is the DN value of the light field after denoising and correction of the nth image obtained from the final model.

[0088] This method addresses the pixel drift and image noise issues encountered by CMOS space cameras under temperature-varying environments. A temperature noise correction model based on a deep learning framework was designed and implemented. The temperature noise under both dark and bright operating conditions was thoroughly investigated, and the effects of temperature changes on the noise characteristics of the CMOS space camera under these conditions were precisely analyzed. Through detailed pixel-by-pixel analysis, the specific impact of temperature changes on pixel values ​​was studied and verified. The potential impact of temperature changes on key performance indicators such as internal camera non-uniformity was also considered. The effectiveness of this deep learning method in correcting temperature-induced noise in CMOS space cameras was systematically evaluated and verified from multiple dimensions. The deep learning-based temperature noise correction method proposed in this patent not only has significant theoretical research value but also demonstrates great potential for application in engineering fields, and is of great significance for improving the imaging quality and stability of CMOS space cameras in complex space environments.

[0089] Under dark field conditions, the comparison and analysis of the predicted values ​​(test values) and experimental test values ​​(predicted values) of the algorithm model before and after correction are shown in the figure below. Figure 2 As shown, where T represents temperature. Figure 2In the figure, (a)-(f) represent the fitting comparison between the predicted value and the actual value of the dark field module at different temperatures. Through analysis, the fitting variance (RMES) between the algorithm's predicted value and the experimental test value ranges from 0.29585 to 5.798307, which can prove the accuracy and effectiveness of the proposed model algorithm under dark field conditions.

[0090] Under light field conditions, the comparison and analysis of the predicted values ​​(test values) and experimental test values ​​(predicted values) of the algorithm model before and after correction using this method are shown in the figure below. Figure 3 As shown, where L represents radiance and T represents temperature. Figure 3 In the diagram, (a)-(f) represent the denoising capabilities of the light field module for images under different radiance and temperature conditions; by Figure 3 The excellent performance of the light field module in improving inter-column differences can be seen.

[0091] Under light field conditions, the standard deviation and signal-to-noise ratio before and after correction by this method are shown in Table 1. As can be seen from Table 1, this method effectively corrects the temperature noise of CMOS space cameras.

[0092] Table 1:

[0093]

[0094]

Claims

1. A method for temperature noise correction in a CMOS space camera based on attention mechanism and LSTM, characterized in that, The method includes the following steps: S1. Acquire dark field and light field images using a CMOS space camera; S2. Select the temperature and average DN of the dark field image from step S1 to construct a sample set. The constructed sample set is called the dark field single pixel module sample set. S3. Construct a multi-level LSTM network model; S4. Generate the mean DN of the image through a fully connected layer; S5. Select the column mean values ​​of temperature and DN values ​​of the dark field image from step S1 to construct a sample set. The constructed sample set is called the dark field column difference module sample set. S6. Construct a single-layer LSTM network model; S7. Use the attention mechanism to obtain the degree of influence of inter-column difference features and individual pixel DN values ​​on the dark field DN value of the image. S8. Use a fully connected layer to generate pixel-wise DN values ​​for the dark field image; S9. Select the column mean values ​​of temperature and DN values ​​of the light field image from step S1 to construct a sample set. The constructed sample set is called the light field module sample set. S10. Use a fully connected layer to obtain the embedded vector representation of the column mean of the temperature and light field image; S11. Use the attention mechanism to obtain the degree of influence of temperature and light field image DN values ​​on image light field noise, respectively. S12. Use a fully connected layer to generate pixel-wise DN values ​​for light field image noise; S13. Calculate the light field DN value of the image based on the acquired dark field noise and light field noise.

2. The temperature noise correction method for CMOS space camera based on attention mechanism and LSTM according to claim 1, characterized in that, Step S3 specifically involves: A 3-layer LSTM network model is used as the dark field single pixel module. The temperature of the image is represented as an embedding vector and input into the network model to obtain the hidden state of a single pixel in the nth image.

3. The temperature noise correction method for CMOS space camera based on attention mechanism and LSTM according to claim 1, characterized in that, Step S4 specifically involves inputting the hidden state of a single pixel in the nth image into a fully connected layer to obtain the embedding vector representation of the predicted DN value.

4. The method for temperature noise correction of CMOS space camera based on attention mechanism and LSTM according to claim 1, characterized in that, Step S6 specifically involves: A single-layer LSTM network model is used as the inter-column difference module. The temperature of the image is represented as an embedding vector and input into the network model to obtain the inter-column difference of pixels in the same row of the image at the same temperature.

5. The temperature noise correction method for CMOS space camera based on attention mechanism and LSTM according to claim 1, characterized in that, Step S7 specifically involves: An attention mechanism is employed to adaptively assign weights to the hidden state representing inter-column differences and the temperature-dependent embedding vector representation of a single pixel, through: ； ； ； get, It is the embedding vector representation of the DN value predicted for the nth image as a single pixel. It is the embedding vector of a single pixel module, representing Input to the attention mechanism in the inter-column difference module; It is a weight matrix. It is the bias vector; It is the weight of the effect of a single pixel on the DN value of the entire image as a function of temperature; It is the pixel index, that is, the weight of the influence of the inter-column difference on the DN value of the whole image; Attention function Specifically: ; In the formula, It is the dimension of the hidden vector of the inter-column difference module. This refers to the query information for the attention mechanism in the inter-column difference module, specifically the dark field DN value of the nth image. After passing through the attention mechanism, the embedding vector representation of the entire image is obtained. for: ; In the formula, This represents the hidden state of the LSTM network. It is the embedding vector representation of the entire image, representing the degree of influence of inter-column difference features and individual pixel DN values ​​on the dark field DN values ​​of the image.

6. The method for temperature noise correction of CMOS space camera based on attention mechanism and LSTM according to claim 1, characterized in that, Step S10 specifically involves: A feedforward neural network based on an attention mechanism is used to correct the noise in the output image of the CMOS space camera under light field conditions based on the image DN value under temperature and light field conditions. The temperature and light field DN values ​​are respectively obtained by obtaining their embedding vector representation through a fully connected layer.

7. The method for temperature noise correction of CMOS space camera based on attention mechanism and LSTM according to claim 1, characterized in that, Step S11 specifically involves: An attention mechanism is used to adaptively acquire different weights for temperature and DN value of light field image. The embedding vector of light field is input into feedforward neural network to obtain the hidden state of image without dark field influence. The hidden state can represent the degree of influence of temperature and DN value of light field image on image light field noise.

8. The method for temperature noise correction of CMOS space camera based on attention mechanism and LSTM according to claim 1, characterized in that, Step S13 specifically involves: Based on the dark field DN value predicted by the dark field module and the light field noise predicted by the light field module, the light field DN value of the nth image is obtained. ； in It is the DN value of the nth light field image. This is the dark field noise calculated by the dark field module. It is the light field noise calculated by the light field module. It is the DN value of the light field after denoising and correction of the nth image obtained from the final model.

Images