Infrared image vertical stripe removal method based on dynamic adjustment of vertical stripe distribution

Abstract

The application discloses an infrared image vertical stripe removing method based on vertical stripe distribution dynamic adjustment, which comprises the following steps: 1) a high-dimensional feature vector matrix extraction step; 2) a vector parameter extraction step, which comprises inputting the high-dimensional feature vector matrix and each frame of infrared image to be processed into a vertical stripe removing parameter generation network together to generate two column vector parameters and two row vector parameters; and 3) an improved traditional vertical stripe removing algorithm processing step, which comprises performing Gaussian filtering on the infrared image to be processed in the row direction, calculating vertical stripe noise column by column, judging whether the vertical stripe noise is real vertical stripe noise column by column, obtaining a vertical stripe noise image of the infrared image to be processed, and then performing difference to obtain an infrared image after vertical stripe removing. The application designs suitable loss functions and training processes for a vertical stripe distribution information extraction network and a vertical stripe removing parameter generation network, so that the model reaches good performance, can effectively analyze the distribution of vertical stripes and generate robust vertical stripe removing parameters.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a method for removing vertical stripes from infrared images based on dynamic adjustment of vertical stripe distribution. Background Technology

[0002] Infrared detectors convert the thermal radiation of a target into electrical signals, which are then processed by image processing algorithms to obtain an infrared image. Since all objects above absolute zero emit thermal radiation, infrared images can clearly capture target information even in darkness, making them widely used in military, exploration, and hunting fields. However, infrared detectors are extremely sensitive to temperature changes, and the internal temperature of the thermal imager gradually changes over time, resulting in non-uniformity in the infrared image due to internal temperature variations. Typically, thermal imagers correct this temperature-dependent non-uniformity by lowering a baffle plate at fixed intervals, but some fixed columns of vertical stripe noise still remain. Therefore, to address this fixed column of vertical stripe noise in infrared images, thermal imager manufacturers usually add de-striping algorithms to remove this fixed vertical stripe noise, eliminating this abnormal fixed column of non-uniformity.

[0003] Invention patent CN201510119228.7 discloses a method for eliminating stripe noise in infrared images. First, the average value of N columns in the infrared image is calculated; then, the stripe noise is estimated. Invention patent CN201910051808.5 discloses a method for eliminating stripe noise in infrared images. First, a deviation function between column stripe noise in adjacent columns of the original single-frame infrared image is established; then, the deviation function is minimized recursively to estimate the column noise deviation value; finally, the column noise deviation value is subtracted from the original image to remove the column noise.

[0004] However, the vertical stripe noise in infrared images is not always in one form. As the internal temperature of the thermal imager changes, the width and amplitude of the vertical stripe noise fluctuate. However, the destripping algorithms used by manufacturers typically apply the same set of parameters to the entire image across all time periods, thus addressing vertical stripe noise in most cases. But for vertical stripes that these parameters cannot handle, thermal imager manufacturers leave them unprocessed, resulting in residual noise in the infrared image and a poor user experience. Summary of the Invention

[0005] Therefore, the technical problem to be solved by this invention is to overcome the defects in the prior art. It proposes an infrared image vertical ridge removal method based on dynamically adjusting parameters according to vertical ridge distribution information. This method addresses the inability of traditional vertical ridge removal algorithms to handle dynamically changing vertical ridges and to simultaneously process vertical ridges with large differences in width and amplitude. The method utilizes a vertical ridge distribution information extraction network to analyze the vertical ridge distribution once per row within a certain time interval. Then, within this time interval, based on the obtained vertical ridge distribution, a vertical ridge removal parameter generation network generates the row and column vector parameters required by the improved traditional vertical ridge removal algorithm. The generated vector parameters are then input into the improved traditional vertical ridge removal algorithm for vertical ridge removal. This method can solve the problem of vertical ridge removal with dynamically changing ridge shapes and large differences in width and amplitude, which traditional vertical ridge removal algorithms cannot handle.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] A method for removing vertical stripes from infrared images based on dynamic adjustment of vertical stripe distribution includes the following steps:

[0008] 1) The high-dimensional feature vector matrix extraction step includes selecting multiple frames of infrared images with vertical stripe information within a time interval set according to the noise degradation state of the thermal imager and inputting them into the vertical stripe distribution information extraction network to extract the high-dimensional feature vector matrix representing the vertical stripe distribution;

[0009] 2) Vector parameter extraction step, which includes inputting the high-dimensional feature vector matrix and each frame of the infrared image to be processed into the destigmatization parameter generation network to generate two column vector parameters and two row vector parameters;

[0010] 3) The improved traditional vertical stripe removal algorithm processing steps include performing Gaussian filtering on the infrared image to be processed in the row direction to extract high-frequency information, calculating vertical stripe noise column by column based on the threshold parameter of the number of pixels summed, and then judging column by column whether the vertical stripe noise is real vertical stripe noise based on the intensity threshold parameter to obtain the vertical stripe noise image of the infrared image to be processed, and subtracting the infrared image to be processed from the vertical stripe noise image to obtain the infrared image after vertical stripe removal.

[0011] Among them, the window size parameter and standard deviation parameter of Gaussian filtering in the row direction have a corresponding relationship with the two column vector parameters, and the pixel summation number limit parameter and intensity limit threshold parameter have a corresponding relationship with the two row vector parameters.

[0012] As one preferred approach, a classic visual feature extraction network is used as the vertical stripe distribution information extraction network to extract the high-dimensional feature vector matrix.

[0013] As one preferred embodiment, the desquamation parameter generation network consists of four subnets that generate column vector parameters and row vector parameters respectively. These subnets have identical network structures, differing only in the parameters used in the average pooling layer at the end. The implementation of the desquamation parameter generation network includes the following steps:

[0014] The high-dimensional feature vector matrix and the infrared image to be processed are concatenated along the feature dimension and then subjected to multiple dimensionality reductions to reduce the feature dimension to 1.

[0015] It is then fed into an average pooling layer to aggregate in the spatial dimension to generate the corresponding column vector parameters or row vector parameters.

[0016] As one preferred embodiment, the extraction of high-frequency information includes the following steps:

[0017] a. Traverse the neighboring pixels one by one and calculate the pixel value difference.

[0018] ,

[0019] in, This represents the pixel value in the i-th row and j-th column of the infrared image to be processed. This represents the pixel value in the i-th row and k-th column of the infrared image to be processed. Let be the difference between the pixel value at position j in row i and the pixel value at position k in column i. This is for taking the absolute value;

[0020] b. Calculate Gaussian weights based on pixel value differences.

[0021] ,

[0022] in, Let be the Gaussian weight of the difference between the pixel values ​​at the j-th column and the k-th column in the i-th row. The standard deviation parameter used for performing one-dimensional Gaussian filtering row by row is...

[0023] c. Calculate the weighted average of the pixels in the neighborhood along the row direction to obtain the one-dimensional Gaussian filtered pixel value.

[0024] ,

[0025] in, The pixel value after one-dimensional Gaussian filtering is obtained by taking the weighted average of the pixels in the neighborhood of the pixel position in the i-th row and j-th column along the row direction. The window size parameter for the Gaussian filter in the row direction corresponding to the i-th row;

[0026] d. Low-frequency information in the vertical direction was obtained by Gaussian filtering in the upper direction. Then it is compared with the infrared image to be processed. By subtracting from the high-frequency information, we can obtain high-frequency information.

[0027] As one preferred embodiment, the implementation of column-by-column calculation of vertical ridge noise includes the following steps:

[0028] Calculate the mean of all Gaussian weights in the neighborhood of each pixel. Does it meet the threshold parameter for the sum of pixels exceeding the limit set for filtering vertical stripe noise in this column? If this condition is met, the high-frequency information at that pixel location will be retrieved. Accumulate; if the condition is not met, then accumulate the high-frequency information at that pixel location. No accumulation operation is performed, where, , To find the position of the pixel in the j-th column of the i-th row, take the position of the pixel in the row direction with the pixel as the center and take the position of the pixel to the left and right. The number of pixels is used as the neighborhood to calculate the average of the corresponding Gaussian weights. The window size parameter for the Gaussian filter in the row direction corresponding to the i-th row;

[0029] The accumulated values ​​of the high-frequency information in this column are aligned and averaged to obtain the vertical ripple noise for that column. .

[0030] As one preferred embodiment, the step of determining whether the vertical ridge noise is genuine vertical ridge noise column by column includes the following steps:

[0031] Set an intensity limit threshold parameter for each column. And the vertical ripple noise calculated above Perform column-by-column checks: if If the calculated vertical ridge noise is valid, it should be retained for the next step; otherwise, the calculated vertical ridge noise should be invalid, and the vertical ridge noise in that column should be set to 0.

[0032] As one preferred embodiment, the correspondence is a linear mapping between the parameter adjustment range and the numerical range of the row vector parameter or column vector parameter.

[0033] As one preferred approach, the pre-training process for the vertical stripe distribution information extraction network includes:

[0034] A small number of consecutive frames of infrared images with vertical stripes are acquired. Several frames are randomly selected from these frames twice and input into the vertical stripe distribution information extraction network to extract a high-dimensional feature vector matrix.

[0035] A loss function is established for backpropagation, and the specific loss function is shown below:

[0036] ;

[0037] in, These are the high-dimensional feature vector matrices extracted in the first and second extractions, respectively. The loss function established during the pre-training process of the vertical stripe distribution information extraction network. This is an L2 norm operation.

[0038] As one preferred approach, a loss function is constructed using the infrared image without vertical stripe noise and the output infrared image without vertical stripe noise for backpropagation. This allows the row vector parameters and column vector parameters to gradually learn the parameters best suited to the distribution of vertical stripe noise. The specific loss function is shown below:

[0039] ,

[0040] in, The two images are infrared images with vertical stripes removed and infrared images with true vertical stripe noise-free, respectively, obtained by inputting the row vector parameters and column vector parameters to the improved traditional vertical stripe removal algorithm during the training process. The loss function established during the training of the parameter generation network for destigmatization This is an L1 norm operation.

[0041] As one preferred option, the time interval is either fixed or dynamically set based on the noise degradation state of the thermal imager.

[0042] The technical solution of this invention has the following advantages:

[0043] The present invention provides an infrared image vertical stripe removal method based on dynamically adjusting parameters according to vertical stripe distribution information. This method utilizes a vertical stripe distribution information extraction network to analyze the vertical stripe distribution once every certain time interval. Then, within this time interval, a vertical stripe removal parameter generation network is used to generate the parameters required by the improved traditional vertical stripe removal algorithm based on the obtained vertical stripe distribution. The generated parameters are then input into the improved traditional vertical stripe removal algorithm to remove the vertical stripes. This method can solve the problem of vertical stripe removal for dynamically changing vertical stripe shapes with different widths and large amplitude differences, which cannot be solved by traditional vertical stripe removal algorithms.

[0044] The improved traditional vertical stripe removal algorithm of this invention changes the setting of the same vertical stripe removal parameters globally to setting different vertical stripe removal parameters row by row and column by column. It makes full use of the ability of the vertical stripe distribution information extraction network to dynamically analyze the vertical stripe distribution, and adjusts the most suitable parameters for vertical stripes of different widths and amplitudes, thereby effectively removing vertical stripes of different conditions and dynamic changes.

[0045] Finally, this invention designs appropriate loss functions and training processes for the vertical stripe distribution information extraction network and the vertical stripe removal parameter generation network, enabling the model to achieve good performance and effectively analyze the distribution of vertical stripes and generate robust vertical stripe removal parameters. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the infrared image vertical stripe removal method of the present invention.

[0047] Figure 2 This is a schematic diagram of the network structure for extracting vertical stripe distribution information.

[0048] Figure 3 A schematic diagram of the network structure generated for removing vertical stripes.

[0049] Figure 4 A schematic diagram of an improved traditional vertical stripe removal algorithm.

[0050] Figure 5 The image shows a comparison of the results of the method for removing vertical stripes from infrared images. The left side is the input infrared image with vertical stripe noise, and the right side is the infrared image after the vertical stripe noise has been removed using the method of this invention.

[0051] Figure 6 The image shows a comparison of the results of the method for removing vertical stripes from infrared images. The left side is the input infrared image with vertical stripe noise, and the right side is the infrared image after the vertical stripe noise has been removed using the method of this invention.

[0052] Figure 7 The image shows a comparison of the results of the method for removing vertical stripes from infrared images. The left side is the input infrared image with vertical stripe noise, and the right side is the infrared image after the vertical stripe noise has been removed using the method of this invention. Detailed Implementation

[0053] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] like Figure 1-4 As shown, the present invention provides a method for removing vertical stripes from infrared images based on dynamic adjustment of vertical stripe distribution, comprising the following steps:

[0055] 1) The high-dimensional feature vector matrix extraction step includes selecting multiple frames of infrared images with vertical stripe information within a certain time interval and inputting them into the vertical stripe distribution information extraction network to extract the high-dimensional feature vector matrix representing the vertical stripe distribution; the certain time interval can be specifically set according to the noise degradation state of the specific thermal imager, and can be dynamically set. For example, when the internal temperature of the machine rises rapidly at the beginning of the power-on, it can be once every 2 minutes; after 10 minutes of power-on, once every 5 minutes; after 1 hour of power-on, once every 30 minutes.

[0056] This method inputs multiple frames of infrared images containing vertical stripe information acquired within a certain time interval (e.g., three frames in this embodiment) into a vertical stripe distribution information extraction network to extract a high-dimensional feature vector matrix representing the vertical stripe distribution within that time interval. This provides support for the vertical stripe morphology distribution within that time interval to the vertical stripe parameter generation network. The vertical stripe distribution information extraction network is constructed using a foundation network and a fully connected network. The foundation network provides the visual features contained in the infrared images, and the fully connected network simplifies and refines them into the high-dimensional feature vector matrix required by the vertical stripe parameter generation network. The foundation network can be any backbone of a classic visual feature extraction network, such as ResNet, NAFNet, VGG, or ConvNet. In this invention, the backbone of NAFNet is used as the foundation network. The number of layers in the fully connected network can be set according to the performance of the running platform and experimental results. Here, this invention uses a three-layer fully connected network, and its network structure diagram is shown below. Figure 2 As shown. Specifically, this invention performs feature stitching on three frames of infrared images with vertical stripe information and then inputs them into a convolutional layer for preliminary feature extraction. In this embodiment, three frames are selected only to extract the vertical stripe variation information in the time domain. Selecting three frames is only for the purpose of minimizing computational and storage requirements. If the application platform has sufficient performance, five, ten, or more frames can be selected, as detailed below:

[0057] (1)

[0058] in, The input consists of 3 frames of infrared images, each with a size of H×W; To perform vector concatenation along the feature dimension, conv1 is a 3×3 convolution operation with 3 input channels and 64 output channels; The feature vector is the result of preliminary feature extraction from three infrared images, and its size is 64×H×W.

[0059] Next, the present invention will extract feature vectors from the preliminary features of the three infrared images. The input is fed into five consecutive NAFNet modules to extract high-dimensional feature vectors that characterize the vertical stripes in the infrared image at that moment, as shown below:

[0060] (2)

[0061] in, The high-dimensional feature vector extracted is 64×H×W, which can characterize the vertical stripes in the infrared image at that moment. This is a NAFNet module with the same structure, and both the input and output channels are 64. This module is a submodule in the classic NAFNet network, and its structure will not be described in detail here.

[0062] The extracted high-dimensional feature vector can characterize the vertical stripes in the infrared image at that moment. This invention uses three fully connected layers and activation functions to simplify and refine the visual features it expresses, laying a solid foundation for the generation of robust and effective parameters for the vertical stripe parameter generation network, as detailed below:

[0063] (3)

[0064] (4)

[0065] (5)

[0066] in, These are the feature vectors purified in the first step, the feature vectors purified in the second step, and the feature vectors output by this vertical stripe distribution information extraction network, with a size of 64×H×W. Both are fully connected layers with the same structure, each with 64 input and output channels, and ReLU is the torch.nn.relu() activation function in Pythorch.

[0067] 2) Vector parameter extraction step, which includes inputting the high-dimensional feature vector matrix and each frame of the infrared image to be processed into the destigmatization parameter generation network to generate two column vector parameters and two row vector parameters;

[0068] By using a vertical stripe distribution information extraction network, this invention obtains a high-dimensional feature vector matrix that can accurately characterize the vertical stripe situation in the infrared image at that moment. Next, this invention inputs the high-dimensional feature vector matrix representing the vertical stripe distribution state and the infrared image with vertical stripes to be processed into a stripe removal parameter generation network. This network dynamically generates different parameters for each row and column based on the stripe distribution state—namely, row vector parameters and column vector parameters. This provides the next step for the improved traditional stripe removal algorithm to dynamically remove vertical stripes of varying widths and amplitudes. Specifically, this invention uses four identical network structures to generate two column vector parameters and two row vector parameters, respectively. A schematic diagram of this network structure is shown below. Figure 3 As shown, the network structures that generate column vector parameters and row vector parameters differ only in the parameters used in the average pooling layer at the end; the rest of the network is the same, as shown in the following equation:

[0069] (6)

[0070] (7)

[0071] (8)

[0072] (9)

[0073] in, For generating destigmatization parameters for column vector parameters, The column vector parameters generated for the corresponding network structure are all of size 1×H; A desquamation parameter generation network for generating row vector parameters. The row vector parameters generated for the corresponding network structure are all of size 1×W; This is a high-dimensional feature vector matrix extracted by the vertical stripe distribution information extraction network, representing the vertical stripe situation in the infrared image at this moment. This is the infrared image to be processed, which has vertical stripes in the current frame. Because... Since the network structures are the same, this invention will not describe them in detail one by one, but only focuses on... The present invention will be described in detail and will provide different parameters regarding the average pooling layer used therein. Although The network structures are the same, but this invention uses the same network structure to generate different parameters; during the training process, as the model gradually backpropagates based on the loss function, these networks with the same structure will have different network parameters, that is, the network parameters will change during the training process to adapt to the production of specific corresponding row vector parameters or column vector parameters.

[0074] Specifically, for the input high-dimensional feature vector matrix representing the vertical stripe distribution state... Infrared image with vertical stripes in the current frame This invention concatenates the feature vectors along their feature dimensions and then inputs the concatenation into a convolutional layer and activation function to gradually reduce the dimensionality of the feature vectors, as shown below:

[0075] (10)

[0076] in, The feature vector of this invention is the feature vector after the feature dimension is reduced, and its size is 32×H×W; This is for vector concatenation operations along the feature dimension. This is the torch.nn.sigmod() activation function in Pythorch; conv1 is a 5×5 convolution operation with 65 input channels and 32 output channels.

[0077] Then, the present invention reduces the feature vector after simplifying the feature dimension. The input is fed into a convolutional layer and activation function to further reduce its feature dimension from 32 to 16, as shown below:

[0078] (11)

[0079] in, This is the feature vector of the present invention after further simplification of the feature dimensions, and its size is 16×H×W; This is the torch.nn.sigmod() activation function in Pythorch, and conv2 is a 7×7 convolution operation with 32 input channels and 16 output channels.

[0080] Next, the present invention reduces the feature vector after simplifying the feature dimension. Further simplification to 1, integrating information from all channels to lay the foundation for the final parameter generation, as shown below:

[0081] (12)

[0082] in, The feature vector is the last feature vector reduced in the feature dimension, and its size is 1×H×W; This is the torch.nn.sigmod() activation function in Pythorch, and conv3 is a 3×3 convolution operation with 16 input channels and 1 output channel.

[0083] Finally, this invention reduces the feature vector with a feature dimension of 1. Aggregation is performed along the spatial dimension to generate the corresponding column vector parameters, as shown below:

[0084] (13)

[0085] in, Generate a network for removing vertical stripes. The output is a column vector of size 1×H, and AvgPool2d is the average pooling layer of torch.nn.AvgPool2d() in Pythorch. For the above-mentioned... The differential parameters used in the average pooling layer are described in detail in this invention: The parameters of AvgPool2d used are kernel_size=(1,W), stride=(1,W), and padding=0; The parameters of AvgPool2d used are kernel_size=(H,1), stride=(H,1), and padding=0.

[0086] 3) Improved steps of the traditional vertical stripe removal algorithm. The traditional vertical stripe removal algorithm is based on the following analysis: the position and intensity of vertical stripe noise in infrared images are relatively fixed, and vertical stripe noise belongs to high-frequency noise in the frequency domain. The traditional vertical stripe removal algorithm is designed based on this characteristic: (a) after extracting low-frequency information through Gaussian filtering in the horizontal direction, the difference is made with the original image to extract high-frequency information above a specific cutoff frequency in the vertical direction; (b) the extracted high-frequency information is subjected to thresholding to filter out as much detail information as possible, so that only vertical stripe noise is retained in the extracted high-frequency information; (c) the vertical stripe noise is subtracted from the original infrared image containing vertical stripes to obtain the infrared image after vertical stripe removal. Therefore, in the above process, four parameters are set reasonably to remove the vertical stripes that appear most of the time, namely: the window size parameter, the standard deviation parameter, the pixel summation limit parameter, and the intensity limit threshold parameter for filtering vertical stripe noise. Traditional algorithms cannot analyze the distribution of vertical stripes in real time. They can only rely on past vertical stripe patterns observed with the thermal imager, requiring professional image algorithm engineers to set four fixed parameters for the entire image to accommodate all possible vertical stripe patterns. However, the vertical stripe noise in infrared images is not always uniform; its width and amplitude fluctuate with changes in the internal temperature of the thermal imager. Therefore, traditional vertical stripe removal algorithms have an inherent drawback: relying solely on parameters set once by professionals cannot handle dynamically changing vertical stripe patterns with significant differences in amplitude and width. To address this, this invention utilizes the powerful learning and fitting capabilities of deep learning methods. It improves the traditional vertical stripe removal process by dynamically setting different parameters row by row and column by column. This allows for adjustments to obtain the most suitable parameters for vertical stripes of varying widths and amplitudes, effectively removing vertical stripes of different types and dynamic variations. The method flow is as follows: Figure 4 As shown.

[0087] The improved traditional vertical stripe removal algorithm proposed in this invention first processes the input infrared image containing vertical stripes. One-dimensional Gaussian filtering is performed row by row. For the i-th row, the present invention iterates through the pixel values ​​in its neighborhood, calculates the corresponding weights, performs a weighted average summation, and then uses this summation as the one-dimensional Gaussian filtered pixel value for that pixel. Specifically:

[0088] a. Calculate the pixel value difference by traversing the neighboring pixels one by one.

[0089] (14)

[0090] in, This represents the pixel value in the i-th row and j-th column of the current frame's infrared image with vertical stripes. This represents the pixel value in the i-th row and k-th column of the current frame's infrared image with vertical stripes. This represents the difference between the pixel values ​​at position j in row i and position k in column i. This is for taking the absolute value.

[0091] b. Calculate Gaussian weights based on pixel value differences.

[0092] (15)

[0093] in, Let be the Gaussian weight of the difference between the pixel values ​​at the j-th column and the k-th column in the i-th row. The standard deviation parameter is used to perform a one-dimensional Gaussian filter row by row for the i-th row.

[0094] The mean of all Gaussian weights in the neighborhood is used as a reference for the threshold limit of the number of pixels to sum in subsequent steps for that pixel.

[0095] (16)

[0096] in, For the position of the pixel in the j-th column of the i-th row, take the left and right sides of it along its row direction. The number of pixels is used as the neighborhood to calculate the average of the corresponding Gaussian weights. The parameter is the window size for Gaussian filtering in the row direction corresponding to the i-th row; it is used to calculate the number of pixels to sum for filtering vertical noise in subsequent steps, and is calculated in advance here.

[0097] c. The pixel value after one-dimensional Gaussian filtering is obtained by weighting and averaging the pixels in the neighborhood along the row direction.

[0098] (17)

[0099] in, The pixel value after one-dimensional Gaussian filtering is obtained by taking the weighted average of the pixels in the neighborhood of the pixel position in the i-th row and j-th column along the row direction. is the window size parameter for Gaussian filtering in the row direction corresponding to the i-th row.

[0100] d. By using Gaussian filtering in the upper row direction, this invention obtains low-frequency information in the vertical direction. , It includes all positions (i, j traversed). The whole, and then the present invention combines it with the original image. The difference is used to extract high-frequency information above a specific cutoff frequency in the vertical direction, as shown below:

[0101] (18)

[0102] in, This invention extracts high-frequency information in the vertical direction. The different Gaussian weights and window sizes used in the preceding Gaussian filtering process generate different low-frequency images, which, when subtracted from the original image, produce high-frequency information with different cutoff frequencies.

[0103] Next, the present invention extracts high-frequency information. Thresholding is applied column by column to filter out as much detailed information as possible, ensuring that only vertical ridge noise is retained in the extracted high-frequency information. Specifically:

[0104] a. Calculate vertical ridge noise column by column.

[0105] For the above extraction of high-frequency information in the vertical direction In the j-th column, this invention determines the mean of all Gaussian weights in the neighborhood of each pixel. Does it meet the threshold parameter for the sum of pixels exceeding the limit set for filtering vertical stripe noise in this column? If this condition is met, the high-frequency information at that pixel location will be retrieved. Accumulate; if the condition is not met, then accumulate the high-frequency information at that pixel location. No accumulation operation is performed. For the accumulated value of the high-frequency information in this column, the present invention aligns and averages the values ​​to obtain the vertical ridge noise of that column, as shown below:

[0106] (19)

[0107] in, Let the j-th column of the vertical ripple noise two-dimensional matrix NS be an example of this. To perform the mean operation; This is the mean of all Gaussian weights in the neighborhood of the pixel position in the i-th row and j-th column, calculated in advance during the one-dimensional Gaussian filtering process of this invention. This is the threshold parameter set in this invention to limit the number of pixels summed for filtering vertical stripe noise in the j-th column.

[0108] b. Determine column by column whether the vertical ridge noise is a true vertical ridge noise.

[0109] This invention uses intensity limiting threshold parameters set for each column. Let's take the j-th column of the vertical striated noise image NS calculated above. Perform column-by-column checks: if If the calculated vertical ridge noise is valid, it should be retained for the next step; otherwise, the calculated vertical ridge noise should be invalid, and the vertical ridge noise in that column should be set to 0.

[0110] Finally, this invention uses the difference between the original image and the extracted vertical stripe noise to obtain the stripe-free infrared image, as shown below:

[0111] (20)

[0112] in, The image shown is the infrared image after removing the vertical stripes according to the present invention, and NS is the vertical stripe noise image extracted above according to the present invention.

[0113] This invention, in conjunction with the above description, explains how to use the vector parameters generated by the desquamation parameter generation network in the previous step. The improved traditional desquamation algorithm of this invention includes the following four parameters: the window size parameter for Gaussian filtering in the row direction. Standard deviation parameter of Gaussian filter in the row direction Threshold parameter for limiting the number of pixels to sum when filtering vertical stripe noise. 1. Intensity limit threshold parameter for filtering vertical noise. .in, The parameter vector has a size of 1×H, and these parameters, along with the parameter vector generated by the desquamation parameter generation network, are... There is a corresponding relationship; The parameter vector has a size of 1×W, which is the sum of the parameter vectors generated by the desquamation parameter generation network. There is a corresponding relationship. However, the row and column vector parameters generated by the desquamation parameter generation network cannot be directly applied to the improved traditional desquamation algorithm according to the above correspondence, because the output vector parameter values ​​of the desquamation parameter generation network are in the range of 0-1, which differs from the parameters actually used. In order to establish a relationship between the output vector parameter values ​​and the actual parameters used, this invention uses a linear mapping between the actual parameter adjustment range of the traditional desquamation algorithm and the output vector parameter value range of the desquamation parameter generation network to achieve both practicality and ensure that the network output values ​​fall within an abnormal range. The mapping relationship is shown in Table 1 below. and The mapping between numerical values, this invention provides The possible values ​​are enumerated to include 10 values ​​used in practical applications. ,Will Divide the range of 0-1 into 10 equal intervals and establish a one-to-one correspondence with the above 10 enumerated values; for and The mapping relationship between numerical values, this invention will The actual value range is 1-100 and A linear mapping is performed on the numerical range of 0-1; for and The mapping relationship between numerical values, this invention will The actual value range is 2000-4000 and A linear mapping is performed on the numerical range of 0-1; for and The mapping relationship between numerical values, this invention will The actual value range is 2-60 and The values ​​are linearly mapped from 0 to 1. Table 1 shows the numerical mapping relationship between the vector parameters generated by the network and the vector parameters set by the desquamation algorithm.

[0114] Table 1

[0115]

[0116] The loss function and training process are as follows:

[0117] For the vertical stripe distribution information extraction network in this method, the present invention pre-trains it. After pre-training, the parameters are frozen and not used in the backpropagation process of the vertical stripe removal parameter generation network. The training of the vertical stripe distribution information extraction network is based on the following understanding: the vertical stripe noise in infrared images does not change its shape within a small time interval; only when the internal temperature of the thermal imager changes to a certain extent will the shape of the vertical stripe noise change. Therefore, the present invention acquires a small number of frames, such as 15 frames, of infrared images with vertical stripes generated by an actual thermal imager. During the training process, three frames are randomly selected from these 15 frames and input into the vertical stripe distribution information extraction network. The high-dimensional feature vector matrix representing the vertical stripe distribution within this time interval, which is output by the vertical stripe distribution information extraction network in two random processes, is used to establish a loss function for backpropagation. The specific loss function is as follows:

[0118] (twenty one)

[0119] in, The high-dimensional feature vector matrices representing the vertical stripe distribution within that time interval are obtained by randomly selecting 3 frames from 15 infrared images in the first and second iterations and inputting them into the vertical stripe distribution information extraction network. The loss function established during the pre-training process of the vertical stripe distribution information extraction network. This is an L2 norm operation.

[0120] Through the above process, this invention completes the pre-training process of the vertical stripe distribution information extraction network. In the following process, the parameters of the vertical stripe distribution information extraction network are frozen and not backpropagated. Instead, the infrared image of the current frame with vertical stripes is input into the vertical stripe distribution information extraction network at fixed time intervals to obtain a high-dimensional feature vector matrix representing the vertical stripe distribution within that time interval. This matrix is ​​then input into the destripping parameter generation network for use. For the training process of the destripping parameter generation network, this invention does not directly supervise the training of its four output vector parameters and the actual network parameters. Instead, it transforms the output parameters according to the mapping relationship in Table 1 to the parameter range required by the improved traditional destripping algorithm, and then inputs them into the improved traditional destripping algorithm to obtain the destripped infrared image. A loss function is constructed using the infrared image without vertical stripe noise and the output destripped infrared image for backpropagation, so that the four vector parameters output by the network gradually learn the parameters most suitable for the vertical stripe noise distribution. The specific loss function is as follows:

[0121] (twenty two)

[0122] in, The two methods are as follows: inputting the vector parameters output by the network during training into the improved traditional destigmatization algorithm to obtain a destigmatized infrared image and a true infrared image without vertical stripe noise. The loss function established during the training of the parameter generation network for destigmatization This is an L1 norm operation.

[0123] This invention utilizes multiple frames of infrared images, such as three discontinuous frames, to extract the dynamic vertical stripe information over time. This information is then represented by a high-dimensional two-dimensional matrix, and a deep learning network is used to generate specific column and row vector parameters. The improved vertical stripe removal algorithm sets different parameters row by row and column by column. The aforementioned high-dimensional feature vector matrix representing the vertical stripe distribution, combined with the deep learning method, provides the ability to generate parameters row by row and column by column. The vertical stripe distribution information extraction network runs periodically to match the vertical stripe generation state at different stages, ensuring that the vertical stripe noise morphology does not change significantly within a certain time interval. Based on this noise morphology of the vertical stripe distribution, the current frame of the infrared image containing vertical stripes is added to generate some minor parameter changes for each frame of the infrared image to be processed.

[0124] like Figure 5-7 As shown, the processing method of this invention has the following significant effects: In the processed image, the original regular vertical stripes almost completely disappear, especially at the edges of the image and the outlines of buildings. The "tearing" effect caused by the vertical stripes is effectively weakened, and the overall texture is smoother and more natural. While eliminating the vertical stripes, the grayscale distribution of the processed image is more uniform, and the detail differentiation of dark areas (such as building shadows) and bright areas (such as smoke areas) is improved, avoiding the problem of local contrast imbalance caused by vertical stripe interference. Moreover, no obvious new noise is introduced during the removal of vertical stripes, and the overall purity of the image is improved, especially in solid color areas (such as the sky background), where the performance is more stable.

[0125] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for removing vertical stripes from infrared images based on dynamic adjustment of vertical stripe distribution, characterized in that, Includes the following steps, 1) The high-dimensional feature vector matrix extraction step includes selecting multiple frames of infrared images with vertical stripe information within a time interval set according to the noise degradation state of the thermal imager and inputting them into the vertical stripe distribution information extraction network to extract the high-dimensional feature vector matrix representing the vertical stripe distribution; 2) Vector parameter extraction step, which includes inputting the high-dimensional feature vector matrix and each frame of the infrared image to be processed into the destigmatization parameter generation network to generate two column vector parameters and two row vector parameters; 3) The improved traditional vertical stripe removal algorithm processing steps include performing Gaussian filtering on the infrared image to be processed in the row direction to extract high-frequency information, calculating vertical stripe noise column by column based on the threshold parameter of the number of pixels summed, and then judging column by column whether the vertical stripe noise is real vertical stripe noise based on the intensity threshold parameter to obtain the vertical stripe noise image of the infrared image to be processed, and subtracting the infrared image to be processed from the vertical stripe noise image to obtain the infrared image after vertical stripe removal. Among them, the window size parameter and standard deviation parameter of Gaussian filtering in the row direction have a corresponding relationship with the two column vector parameters, and the pixel summation number limit parameter and intensity limit threshold parameter have a corresponding relationship with the two row vector parameters.

2. The infrared image destripping method based on dynamic adjustment of vertical stripe distribution according to claim 1, characterized in that, The high-dimensional feature vector matrix is ​​extracted by using a classic visual feature extraction network as the vertical stripe distribution information extraction network.

3. The infrared image destripping method based on dynamic adjustment of vertical stripe distribution according to claim 1, characterized in that, The desquamation parameter generation network consists of four subnets that generate column vector parameters and row vector parameters respectively. They have identical network structures, differing only in the parameters used in the average pooling layer at the end. The implementation of the desquamation parameter generation network includes the following steps. The high-dimensional feature vector matrix and the infrared image to be processed are concatenated along the feature dimension and then subjected to multiple dimensionality reductions to reduce the feature dimension to 1. It is then fed into an average pooling layer to aggregate in the spatial dimension to generate the corresponding column vector parameters or row vector parameters.

4. The infrared image destripping method based on dynamic adjustment of vertical stripe distribution according to claim 1, characterized in that, The extraction of high-frequency information includes the following steps: a. Traverse the neighboring pixels one by one and calculate the pixel value difference. ， in, This represents the pixel value in the i-th row and j-th column of the infrared image to be processed. This represents the pixel value in the i-th row and k-th column of the infrared image to be processed. Let be the difference between the pixel value at position j in row i and the pixel value at position k in column i. This is for taking the absolute value; b. Calculate Gaussian weights based on pixel value differences. ， in, Let be the Gaussian weight of the difference between the pixel values ​​at the j-th column and the k-th column in the i-th row. The standard deviation parameter used for performing one-dimensional Gaussian filtering row by row is... c. Calculate the weighted average of the pixels in the neighborhood along the row direction to obtain the one-dimensional Gaussian filtered pixel value. ， in, The pixel value after one-dimensional Gaussian filtering is obtained by taking the weighted average of the pixels in the neighborhood of the pixel position in the i-th row and j-th column along the row direction. The window size parameter for the Gaussian filter in the row direction corresponding to the i-th row; d. Low-frequency information in the vertical direction was obtained by Gaussian filtering in the upper direction. Then it is compared with the infrared image to be processed. By subtracting from the high-frequency information, we can obtain high-frequency information.

5. The infrared image destripping method based on dynamic adjustment of vertical stripe distribution according to claim 1, characterized in that, The implementation of column-by-column calculation of vertical ridge noise includes the following steps: Calculate the mean of all Gaussian weights in the neighborhood of each pixel. Does it meet the threshold parameter for the sum of pixels exceeding the limit set for filtering vertical stripe noise in this column? If this condition is met, the high-frequency information at that pixel location will be retrieved. Accumulate; if the condition is not met, then accumulate the high-frequency information at that pixel location. No accumulation operation is performed, where, , To find the position of the pixel in the j-th column of the i-th row, take the position of the pixel in the row direction with the pixel as the center and take the position of the pixel to the left and right. The number of pixels is used as the neighborhood to calculate the average of the corresponding Gaussian weights. The window size parameter for the Gaussian filter in the row direction corresponding to the i-th row; The accumulated values ​​of the high-frequency information in this column are aligned and averaged to obtain the vertical ripple noise for that column. .

6. The infrared image destripping method based on dynamic adjustment of vertical stripe distribution according to claim 5, characterized in that, The method of determining whether the vertical striate noise is genuine vertical striate noise column by column includes the following steps: Set an intensity limit threshold parameter for each column. And the vertical ripple noise calculated above Perform column-by-column checks: if If the calculated vertical ridge noise is valid, it should be retained for the next step; otherwise, the calculated vertical ridge noise should be invalid, and the vertical ridge noise in that column should be set to 0.

7. The infrared image destripping method based on dynamic adjustment of vertical stripe distribution according to claim 1, characterized in that, The aforementioned correspondence is a linear mapping between the parameter adjustment range and the numerical range of the row vector parameter or column vector parameter.

8. The infrared image destripping method based on dynamic adjustment of vertical stripe distribution according to claim 6, characterized in that, The pre-training process for the vertical stripe distribution information extraction network includes: A small number of consecutive frames of infrared images with vertical stripes are acquired. Several frames are randomly selected from these frames twice and input into the vertical stripe distribution information extraction network to extract a high-dimensional feature vector matrix. A loss function is established for backpropagation, and the specific loss function is shown below: ； in, These are the high-dimensional feature vector matrices extracted in the first and second extractions, respectively. The loss function established during the pre-training process of the vertical stripe distribution information extraction network. This is an L2 norm operation.

9. The infrared image destripping method based on dynamic adjustment of vertical stripe distribution according to claim 8, characterized in that, A loss function is constructed using the infrared image without vertical stripe noise and the output infrared image without vertical stripe noise. This loss function is then used for backpropagation, allowing the row vector parameters and column vector parameters to gradually learn the parameters best suited to the distribution of vertical stripe noise. The specific loss function is shown below: ， in, The two images are infrared images with vertical stripes removed and infrared images with true vertical stripe noise-free, respectively, obtained by inputting the row vector parameters and column vector parameters to the improved traditional vertical stripe removal algorithm during the training process. The loss function established during the training of the parameter generation network for destigmatization This is an L1 norm operation.

10. The infrared image destigma method based on dynamic adjustment of vertical stripe distribution according to claim 1, characterized in that, The time interval is set either fixedly or dynamically based on the noise degradation status of the thermal imager.

Images