A nonlinear response correction method based on a fully connected neural network

Abstract

The application discloses a nonlinear response correction method based on a full connection neural network, and comprises the following steps: (1) performing stripe removal processing on original image data obtained by different detector imaging modules in an infrared detector; (2) extracting and pre-processing the overlapping pixel responses between different detector imaging modules according to the obtained stripe-removed images; (3) establishing a full connection neural network model, and training the pre-processed overlapping pixel responses between the modules two by two; and (4) using the trained full connection network model to perform correction processing on image data obtained by a detector module to be corrected. By using the application, the nonlinear response problem between different detection modules of the infrared detector can be solved, the response error between the modules is reduced to below 0.5%, and meanwhile, the image quality is almost not damaged.

Description

Technical Field

[0001] This invention relates to the field of photoelectric detection technology, and in particular to a nonlinear response correction method based on a fully connected neural network. Background Technology

[0002] With technological advancements, higher demands are being placed on the swath width and spatial resolution of space infrared cameras. Due to limitations in the manufacturing process of large-area infrared detectors, multi-module stitching imaging has become a common solution to improve spatial resolution and swath width. Differences in the manufacturing and optical deviations of different detector modules lead to variations in their responses. This problem, known as the striping phenomenon, has been observed in remote sensing cameras employing multi-module stitching structures and requires targeted research and solutions.

[0003] The Sustainable Development Science Satellite-1 (SDGSAT-1) is the world's first scientific satellite dedicated to serving the UN 2030 Agenda for Sustainable Development and the first Earth science satellite of the Chinese Academy of Sciences. Thermal infrared data acquired from the Thermal Infrared Imager (TIS) onboard SDGSAT-1 has been widely applied in various scientific fields, such as spacecraft monitoring, surface ship detection, and lunar surface temperature and emissivity retrieval. The infrared focal plane array of the TIS payload on SDGSAT-1 consists of four detector modules, each with 512×4 pixels, and utilizes time-delay integration to improve detector sensitivity. In the application of TIS thermal infrared data, the relative response differences among the four modules affect image quality and retrieval accuracy. Research has found that due to the positional differences of the four modules along the scanning direction, their scanning angles during imaging differ, and the effect of different scanning angles on the detector response is nonlinear. This explains the nonlinear differences in the response of overlapping pixels among the four modules. Furthermore, this nonlinear response is difficult to eliminate using on-orbit two-point correction methods.

[0004] Existing technical solutions: For the correction of response differences between Landsat 8 detector modules, Begeman, C. and his team calculated the relative gain and bias of detector pixels between modules using their linear flight data in uniform scenes, and then constructed a linear correction equation based on this after response linearization to complete the correction. In RapidEye image data, Anderson, C. and his team discovered significant response differences between the left and right modules of the detector, and calculated detector pixel bias using dark scene data to reduce the response differences between modules. During the relative radiometric calibration of the ZY-P satellite, VAZenin and his team calculated and corrected the response differences between different detector modules using piecewise linear equations. M. Wang and his team used histograms of satellite linear flight data to correct the response differences between different detector modules. It can be seen that most existing technical solutions correct the relative differences between different detector modules using linear methods. The nonlinear components used are mostly used to correct the nonlinear response of infrared detectors to radiation of different intensities, and there is no research on the response nonlinearity caused by detector module splicing, especially the development of a nonlinear response correction method for the TIS payload of SDGSAT-1.

[0005] Based on the above description, with the development of large-area infrared detectors, it is necessary to propose a method for correcting the nonlinear response of detectors caused by multi-module splicing. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a nonlinear response correction method based on a fully connected neural network, which can solve the nonlinear response problem between different detection modules of an infrared detector.

[0007] A nonlinear response correction method based on a fully connected neural network is used to correct the nonlinear response differences between raw image data obtained from different detector imaging modules in an infrared detector, comprising the following steps:

[0008] (1) Perform stripe removal processing on the raw image data obtained by different detector imaging modules in the infrared detector.

[0009] (2) Based on the obtained destriped image, extract the overlapping pixel responses between different detector imaging modules and perform preprocessing;

[0010] (3) Establish a fully connected neural network model and train the preprocessed pairs of overlapping pixel responses between modules in pairs;

[0011] (4) Use the trained fully connected network model to perform correction processing on the image data obtained by the detector module to be corrected.

[0012] Furthermore, in step (1), the specific process of stripe removal is as follows:

[0013] First, reflective boundary conditions are applied to the original images of each detector imaging module, and additional image blocks are added outside the image edges. Then, stripes are removed in both the horizontal and vertical directions of the images with boundary conditions applied. Finally, the additional image blocks added at the edges are removed from the stripe-removed images to obtain stripe-removed images of the same size as the input images.

[0014] Furthermore, in step (2), the preprocessing specifically involves discarding edge pixels in the overlapping pixels and selecting only the central region pixels to calculate the column mean.

[0015] Furthermore, after selecting the column mean of the pixel response in the central region, the obtained column mean is subjected to mean filtering, and the shape of the filter kernel of the mean filtering is 1×17.

[0016] Further, in step (2), the extracted overlapping pixel responses are three sets of overlapping pixel responses extracted from the four imaging modules respectively. The response vector shape is M×N, where M is the number of pixels in the overlapping area between the two modules, and N is the number of pixels in a row after matching. The size of the pixel response vector in the central region is selected as αM×N, where α is 50% to 55%.

[0017] Further, in step (3), the fully connected neural network model includes two branches G(·) and B(·), which respectively calculate the gain correction coefficient and bias correction coefficient of the input vector;

[0018] The two branches share a three-layer encoder and a two-layer decoder; each encoder and decoder consists of a fully connected layer + linearly modified activation function ReLU + random dropout to encode and decode the input features; finally, the two branches are split through a different layer. G(·) is constrained to the range of 0-1 through a fully connected layer + sigmoid activation function, and will be transformed to -0.015 to +0.015 in the subsequent linear transformation to form the gain coefficient; B(·) is directly output through a fully connected layer to form the bias coefficient.

[0019] The input features are corrected using G(·) and B(·), as shown in the following formula:

[0020] F out =G(N(F) in ))*F in +B(N(F in ))

[0021] Where N(·) refers to the normalization operation, and F in For network input, F out For network output.

[0022] As one application, the infrared detector mentioned is the Thermal Infrared Imager (TIS) carried by the Sustainable Development Science Satellite 1 (SDGSAT-1).

[0023] For the thermal infrared imager TIS, in steps (3) and (4), the goal of the fully connected neural network model during training and calibration is to correct the responses of the first, second, and fourth modules to the level of the third module with the best linearity. The specific order is: second module, fourth module, first module, and the corresponding target response is: third module, third module, and the corrected second module.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] This invention effectively corrects the inter-module nonlinear response in thermal infrared data, effectively suppressing distortion of module edge images and improving the response consistency between modules. According to the Landsat 8 inter-module correction evaluation index, the processed inter-module response error is reduced to below 0.5%. Furthermore, based on the edge slope index in the edge diffusion equation, the image quality after inter-module processing is almost unaffected (decreased by 0.1%), demonstrating good engineering practicality. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the steps of a nonlinear response correction method based on a fully connected neural network proposed in an embodiment of this application.

[0027] Figure 2 This is a schematic diagram of satellite data where there are differences in response between modules.

[0028] Figure 3 The graph shows the response difference curves of overlapping pixels between modules obtained statistically from the TIS load data of SDGSAT-1.

[0029] Figure 4 The graph shows the linearity curve of the inter-module overlapping pixel response obtained from the TIS load data of SDGSAT-1, where the nonlinear region is circled in red.

[0030] Figure 5 This is a schematic diagram of the infrared focal plane of the TIS payload of SDGSAT-1, on which the method described in this embodiment is based.

[0031] Figure 6 This is an example diagram of the TIS load from SDGSAT-1 used in this embodiment after position rematching.

[0032] Figure 7 This is a schematic diagram illustrating the role of boundary conditions in stripe removal processing.

[0033] Figure 8 This is a schematic diagram of the quantitative column mean after stripe removal processing.

[0034] Figure 9 This is a schematic diagram of the quantitative row mean after inter-module correction.

[0035] Figure 10 This is a structural diagram of the fully connected neural network model used for inter-module correction.

[0036] Figure 11 This is a flowchart illustrating the training and correction processes during the correction of nonlinear errors between modules using overlapping pixel responses.

[0037] Figure 12 This is a schematic diagram illustrating the correction effect of the inter-module nonlinear response correction method proposed in this invention.

[0038] Figure 13 This is a schematic diagram of the quantitative column mean of the correction effect of the inter-module nonlinear response correction method extracted in this invention.

[0039] Figure 14 This is a statistical graph showing the relative error of the radiance of the overlapping area between modules after correction by the method of this invention.

[0040] Figure 15 This is a comparison of image quality based on edge slope in the edge diffusion equation after correction by the method of this invention. Detailed Implementation

[0041] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.

[0042] like Figure 1 As shown, a nonlinear response correction method based on a fully connected neural network includes the following steps:

[0043] Step (1): Perform stripe removal processing on the original image data.

[0044] In this embodiment, the original image is first divided into modules, separating the image blocks obtained from different detector modules and processing them separately. Second, reflection boundary conditions are constrained for the image of each module, and additional image blocks (20 rows / columns on each side) are added outside the image edges. Then, the LRSID destriating method guided by the frequency domain is used to remove stripes in both the horizontal and vertical directions of the image with the added boundary conditions. Finally, the image blocks added at the edges are removed from the striped image to obtain a destriated image of the same size as the input image.

[0045] Step (2): Extract and preprocess overlapping pixels from the striped image.

[0046] In this embodiment, firstly, the positions of four images from four detector imaging modules are matched to eliminate positional differences in the scanning direction; secondly, overlapping pixels are extracted from the four images after position matching; thirdly, pixels with less noise among the overlapping pixels are selected, that is, several rows of pixels close to the image edge are removed; then, the row mean is calculated for these selected pixels; finally, mean filtering is applied to the obtained overlapping pixel mean vector.

[0047] Step (3): Train a fully connected network model based on the obtained overlapping pixel responses.

[0048] In this embodiment, the obtained overlapping pixel response vectors are input into the established fully connected neural network model to fit the response relationship mapping from module A to module B. During the process, three models are trained successively, corresponding to module two to module three, module four to module three, and module one to the corrected module two, respectively.

[0049] Step (4): Use the trained model to correct the nonlinear response of the corresponding module.

[0050] In this embodiment, the trained network model is used to correct the corresponding module images. During the correction process, the image to be corrected is input into the model pixel by pixel to obtain the corrected pixels. The correction is performed sequentially on three modules: Module 2, Module 4, and Module 1. The corrected Module 2 is used as the input vector source for the Module 1 correction model.

[0051] Step (5): Quantitatively evaluate the correction effect.

[0052] In this embodiment, the corrected image is quantitatively evaluated in two aspects. When evaluating the effect of nonlinear response correction between modules, a large number of overlapping pixels from the four module images are extracted in batches, and their relative errors are calculated. When evaluating the impact of this method on image quality, a small land-sea boundary region with high edge signal-to-noise ratio is selected, and the edge diffusion equation (ESF) curve of the same region in the de-striated image and the inter-module corrected image is measured using the knife-edge method. The edge slope (ES) is compared, and the image with a higher edge slope has higher image quality.

[0053] Combination Figure 2 , Figure 3 and Figure 4 The research objective of the nonlinear response correction method of this invention will be explained:

[0054] like Figure 2As shown, the striping phenomenon in large-area infrared focal plane arrays with multiple modules, which exists due to the response differences between detector modules, as proposed in this application, is not only present in the SDGSAT-1 satellite addressed by this invention, but also in satellites such as Landsat 8 and Resource-P.

[0055] like Figure 3 As shown, the inter-module response difference in the TIS load of SDGSAT-1 proposed in this application is obtained by statistically analyzing the response of overlapping pixels. The curve of inter-module overlapping pixel response difference is obtained by statistically analyzing the response of overlapping pixels. The lines of different colors represent the response differences of different pixels. It can be seen that the response differences of different pixels have a certain consistency and are not an occasional phenomenon of individual pixels.

[0056] like Figure 4 As shown, the nonlinearity of the inter-module response in the TIS load of SDGSAT-1 proposed in this application is obtained by statistically analyzing the inter-module overlapping pixel response linearity curve, where the part selected by the red circle is the nonlinear part.

[0057] To verify the effectiveness of the present invention, in conjunction with the flowchart (e.g.) Figure 1 (As shown) The technical solution is described in detail. The infrared focal plane of the TIS payload of SDGSAT-1 involved in the embodiment is as follows: Figure 5 As shown, it is composed of four 512×3 detector imaging modules stitched together. An example image after position rematching is shown below. Figure 6 As shown, a 10000×2048 image is transformed into a 9836×2048 image by translating and stitching between modules. The purpose of rematching is to improve the visual appeal of the example image.

[0058] First, the example image undergoes stripe removal processing in modules. During this stripe removal process, the design and effects of boundary conditions are as follows: Figure 7 As shown. Figure 7 Image (a) is the original image to be processed, and the sampling area is a 512×512 sub-image at the boundary between the two modules. Figure 8 (b) and Figure 7 (c) shows the striped subplots after removing boundary conditions and after adding boundary conditions, respectively (the corresponding residuals compared to the original image are...). Figure 7 As shown in Figures 7(e) and 7(f)), without boundary conditions, the boundaries between modules are amplified, resulting in strong distortion. This distortion is suppressed after adding boundary conditions, making the response differences of overlapping pixels between modules consistent with the response differences of the numerous other pixels between modules. A schematic diagram of adding boundary conditions is shown below. Figure 7 As shown in (d), the selected area is a small region in the upper left corner of the image. The column mean plot after stripe removal is shown below. Figure 8As shown, the area near the boundary between modules is magnified for display. The more jagged the column mean curve, the more stripe components there are. It can be seen that after stripe removal processing, the column means of the image become smoother. However, without adding boundary conditions, abnormal peaks appear in the boundary areas between modules, which are eliminated after adding boundary condition constraints.

[0059] Next, overlapping pixels are extracted, filtered, and averaged in the de-striped image. Specifically, 7 rows of the central region are retained from the 13 rows of overlapping pixels between modules one and two, and between modules three and four. From the 37 rows of overlapping pixels between modules two and three, 19 rows of the central region are retained. After averaging, a 1×17 mean filter is used for preprocessing. The significance of the mean filter preprocessing can be demonstrated by the quantitative row mean image (e.g., ...). Figure 9 As shown in the figure, the change in the row mean curve in the normal processing flow is reflected in the correction of the nonlinear response of the scanning angle of different modules by the inter-module correction model. However, the image after neural network fitting without mean filter processing produces new noise stripes, which are reflected as new jagged features (blue lines) on the row mean curve.

[0060] Then, the processed overlapping pixel response vectors are input into the model for fitting. The fully connected neural network model proposed in this application is as follows: Figure 10 As shown, the network has two branches (G(·) and B(·)) that calculate the relative gain coefficient and bias coefficient for the current pixel, respectively. These two branches share most of the network structure, including a three-layer encoder and a two-layer decoder. Each encoder and decoder consists of a fully connected layer + ReLU activation function + dropout to encode and decode the input features. Finally, the two branches are split through a different layer. G(·) uses a fully connected layer + sigmoid activation function to constrain the response to the range of 0-1, which will then be transformed to -0.015 to +0.015 in the subsequent linear transformation, forming the gain coefficient. B(·) outputs directly through a fully connected layer, forming the bias coefficient. The input features are corrected using G(·) and B(·). The correction model is as follows:

[0061] F out =G(N(F) in ))*F in +B(N(F in ))

[0062] Where N(·) refers to the normalization operation, and F in For network input, F out For network output.

[0063] The training and processing flow of the network model for the image to be corrected is as follows: Figure 11As shown. Figure 11 (a) describes the training process using overlapping cell responses between modules. Figure 11 (b) describes the calibration process using the trained model. During calibration, each module has a separate calibration network trained using the overlapping pixel responses of the target module (e.g., the calibration models for modules two and four are trained using their overlapping pixel responses with module three, while the calibration model for module one is trained using its overlapping pixel responses with the calibrated module two). During calibration, each of the 512 pixels of the module to be calibrated is calibrated one by one. It is worth noting that the input pixel responses during calibration no longer need to undergo mean filtering.

[0064] The resulting corrected image is as follows Figure 12 As shown, Figure (ac) is the original image of the boundary region between the three modules, Figure (df) is the image of Figure (ac) after stripe removal processing, and Figure (gi) is the image of Figure (df) after inter-module correction. It can be seen that the response differences between modules are bridged, and the significant differences at the boundaries are corrected and alleviated. The quantitative column mean plot is shown below. Figure 13 (The boundary regions between modules are magnified). It can be seen that after stripe removal and inter-module correction, the response curves of each module become smoother and are shifted as a whole, making the response of the entire imaging plane more continuous and reducing the non-uniformity of the response between modules.

[0065] Finally, the correction results are quantitatively evaluated. This invention evaluates the consistency of responses of adjacent modules by assessing the overlapping pixel responses between image modules. The evaluation index is shown in the following formula, where L... Ma and L Mb The corresponding radiance values ​​of the overlapping cell responses of two adjacent modules that are not yet to be evaluated.

[0066]

[0067] During the evaluation process, we collected the overlapping response of over 600 uniform scenes from more than 180 images. The calculation results are as follows: Figure 14 As shown in Figures (ac), the relative errors between modules 1 and 2, between modules 2 and 3, and between modules 3 and 4 are respectively. More than 99.8% of the pixels had relative errors between modules controlled below 0.5% after correction. The remaining approximately 0.2% of pixels were found to exist in some cloud-covered images, and the cloud-covered areas caused pixel mismatch between modules.

[0068] In evaluating the impact of the correction model on image quality, the edge slope index from the edge diffusion equation was used. A 50×50 region with a high edge signal-to-noise ratio and a straight coastline was selected. The ESF curve was fitted using the Fermi function to obtain the edge slope index. The calculation results are as follows: Figure 15 As shown, compared to the image before inter-module correction (ES=1.7124), the corrected image (ES=1.7102) only showed a decrease of about 0.1%, and the image quality was almost unaffected.

[0069] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A nonlinear response correction method based on a fully connected neural network, used to correct the nonlinear response differences between raw image data obtained from different detector imaging modules in an infrared detector, characterized in that, Includes the following steps: (1) Perform stripe removal processing on the raw image data obtained from different detector imaging modules in the infrared detector; the specific process of stripe removal processing is as follows: First, reflective boundary conditions are applied to the original images of each detector imaging module, and additional image blocks are added outside the image edges. Then, stripes in both the horizontal and vertical directions are removed from the images with boundary conditions applied. Finally, the additional image blocks added at the edges are removed from the stripe-removed images to obtain stripe-removed images of the same size as the input images. (2) Based on the obtained destriped image, extract the overlapping pixel responses between different detector imaging modules and perform preprocessing; the preprocessing specifically involves discarding edge pixels in the overlapping pixels and selecting only the central region pixel responses to calculate the column mean. (3) Establish a fully connected neural network model and train the preprocessed pairs of overlapping pixel responses between modules in pairs; the fully connected neural network model contains two branches. and Calculate the gain correction coefficient and bias correction coefficient of the input vector respectively; The two branches share a three-layer encoder and a two-layer decoder; each encoder and decoder consists of a fully connected layer + linearly modified activation function ReLU + random dropout, which realizes the encoding and decoding of input features; Ultimately, the two branches are separated by a different layer. By using a fully connected layer and a sigmoid activation function, the response is constrained to the range of 0-1. In the subsequent linear transformation, it will be transformed to -0.015 to +0.015, forming the gain coefficient. The bias coefficient is formed by directly outputting through a fully connected layer; (4) Use the trained fully connected network model to perform correction processing on the image data obtained by the detector module to be corrected.

2. The nonlinear response correction method based on a fully connected neural network according to claim 1, characterized in that, After selecting the pixel response of the central region and calculating the column mean, the obtained column mean is subjected to mean filtering.

3. The nonlinear response correction method based on a fully connected neural network according to claim 2, characterized in that, The shape of the filter kernel for the mean filter is: .

4. The nonlinear response correction method based on a fully connected neural network according to claim 1, characterized in that, In step (2), the extracted overlapping pixel responses are three sets of overlapping pixel responses extracted from the four imaging modules respectively. The response vector shape is M×N, where M is the number of pixels in the overlapping area between the two modules, and N is the number of pixels in a row after matching. The size of the pixel response vector in the central region is selected as... , It is 50% to 55%.

5. The nonlinear response correction method based on a fully connected neural network according to claim 1, characterized in that, pass and The input features are corrected using the following formula: in, Refers to the normalization operation, where For network input, For network output.

6. The nonlinear response correction method based on a fully connected neural network according to claim 1, characterized in that, The infrared detector mentioned is the Thermal Infrared Imager (TIS) carried by the Sustainable Development Science Satellite 1 (SDGSAT-1).

7. The nonlinear response correction method based on a fully connected neural network according to claim 6, characterized in that, In steps (3) and (4), the goal of the fully connected neural network model during training and calibration is to calibrate the responses of the first, second, and fourth modules to the level of the linearly optimal third module. The specific order is: second module, fourth module, first module, and the corresponding target response is: third module, third module, and the calibrated second module.

Images