Method for generating thermal infrared background image based on principal component analysis and diffusion model

Abstract

The application discloses a thermal infrared background image generation method based on principal component analysis and diffusion model. The method aims to model the thermal infrared background image, thereby generating diversified thermal infrared background images, and providing effective training data pairs for a neural network algorithm for thermal infrared image background removal. The principal component analysis can extract the main components in the thermal infrared background image and obtain the main features in the thermal infrared background image. The diffusion model learns the features of the remaining components in the thermal infrared background image, thereby generating more remaining components of the thermal infrared background image in conformity with actual distribution. A large number of thermal infrared background images can be generated based on the main components and the generated remaining components. Experiments show that, under the same background removal network, compared with thermal infrared images constructed by using directly collected thermal infrared background images, the thermal infrared images constructed by using the thermal infrared background images generated by the application have better background removal effect of the network trained.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method for generating a thermal infrared background image based on principal component analysis and a diffusion model. Background Art

[0002] Due to manufacturing limitations, each pixel in a thermal infrared image sensor responds differently to the same intensity of infrared radiation. Therefore, to better represent the infrared radiation distribution of a scene, background removal is required to ensure that each pixel exhibits a consistent response to the same temperature. Background removal is the first step in thermal infrared image processing algorithms, and its effectiveness significantly impacts the quality of subsequent image processing. Furthermore, as the operating time of a thermal infrared camera increases and the operating environment changes, the background of the thermal infrared image also changes, rendering previously acquired background images inappropriate.

[0003] The traditional method of removing background from thermal infrared images is generally as follows: when the thermal infrared camera is turned on, the shutter is opened once and the image on the shutter is collected as the background image. Subsequently, during use, the background image is subtracted pixel by pixel from the detector output to obtain the image after background removal. The traditional background removal method has strong limitations. On the one hand, due to the influence of noise, the image after background subtraction is not clean, and a gauze-like background will still be covered on the image. On the other hand, as the camera body operates and the environment changes, the thermal infrared image will drift in temperature. At this time, the previously collected background image is no longer applicable, and the shutter needs to be opened frequently to re-collect the background. In some usage scenarios, the pause time of opening the shutter will have a great impact on the user.

[0004] Neural network-based background removal methods can calculate the background image to be subtracted based on local and global features in thermal infrared images. The background-removed image obtained through the neural network has less residual background and noise, and while subtracting noise and background, it can well preserve the object information without losing any details. In summary, the background removal method using a neural network can produce clean and clear thermal infrared images. When constructing the training data set for the neural network, thermal infrared background images are required. Because thermal infrared background images change over time and the environment, it is impossible to capture all background conditions, which can lead to poor performance of the trained background removal network. Summary of the Invention

[0005] The present invention discloses a method for generating thermal infrared background images based on principal component analysis and a diffusion model. The main goal of this method is to extract the main components of the thermal infrared background image through principal component analysis, and to learn the distribution of the remaining components of the thermal infrared background image through a diffusion model, thereby generating a more diverse thermal infrared background image. Principal component analysis mainly obtains some inherent components in the thermal infrared background image; the diffusion model mainly learns the distribution characteristics of the remaining components in the thermal infrared background image and generates remaining components that conform to this distribution. The background removal network trained on the thermal infrared background image generated by principal component analysis and the diffusion model has a better background removal effect. While removing the thermal infrared background image, it can also well preserve the image content information. The clean image provides a good foundation for subsequent image processing. The thermal infrared background image generation method based on principal component analysis and a diffusion model proposed in the present invention only needs to collect a small number of single thermal infrared background images to generate a large number of diverse thermal infrared background images.

[0006] The specific implementation steps of the method of the present invention are as follows:

[0007] A method for generating thermal infrared background images based on principal component analysis and a diffusion model is presented. This method uses principal component analysis and a diffusion model to model a small number of single thermal infrared background images, generating a large number of diverse thermal infrared background images. Within the same background removal network, the thermal infrared background images generated using this method achieve better background removal. The specific steps of this method are as follows:

[0008] S1: Thermal infrared background image acquisition. Use a thermal infrared camera to capture a small amount of raw thermal infrared background images of a black body with uniform temperature. This serves as the basis for subsequent analysis.

[0009] S2: Modeling thermal infrared background images using principal component analysis and diffusion model.

[0010] S21: Obtain the components of the thermal infrared background image based on the principal component analysis method, analyze the contribution of each component, and retain the main components and their corresponding weights.

[0011] S22: removing the main components of the thermal infrared background image to obtain the remaining components.

[0012] S23: Learning the features of the residual components based on the diffusion model to generate various residual components of the thermal infrared background image. Specifically, the diffusion model is based on the residual component features and obtains the residual components of any set of thermal infrared background images by inputting different Gaussian noise matrices.

[0013] S3: Based on the main components and the generated residual components, a variety of simulated thermal infrared background images are obtained.

[0014] In the above technical solution, further, step S1 is specifically: using a thermal infrared camera, aiming at a black body with uniform temperature, collecting the raw data output by the detector at a frequency of 2 frames per second, serving as the original thermal infrared background image.

[0015] Furthermore, the principal component analysis method can be used to obtain the main components of the thermal infrared background image, and the diffusion model can be used to learn the characteristics of the remaining components in the thermal infrared background image, thereby generating more diverse thermal infrared background images.

[0016] Furthermore, the step S21 is specifically as follows:

[0017] The collected n thermal infrared background images of size h*w are stitched into a matrix X n*(h*w) .

[0018] Calculate the average

[0019] Calculate the covariance matrix C = E{(Xu)(Xu) T}.

[0020] Calculate the eigenvalue matrix λ of the covariance matrix C n*n , eigenvector matrix V n*n , sort the eigenvalues ​​and eigenvectors.

[0021] Calculate the corresponding component as PC = X T *V, based on the cumulative contribution value, only retain the eigenvectors V of the first k components k .

[0022] An image that only retains the first k components of the thermal infrared background image

[0023] The main component image is subtracted from the original thermal infrared background image to obtain the residual component of the thermal infrared background image. This serves as the training set for the diffusion model. Based on the characteristic distribution of the training set, a more realistic residual component of the thermal infrared background image can be generated. The forward diffusion process of the diffusion model involves continuous noise addition. A certain number of diffusion steps are set for the residual component of the initial thermal infrared background image, and a certain amount of noise is added at each step, ultimately converting the initial data into a random Gaussian noise matrix. The backward diffusion process of the diffusion model involves continuous denoising. The random Gaussian noise matrix is ​​passed through the backward diffusion module of the diffusion model to predict and remove the noise. After a certain number of diffusion steps, a clear residual component of the thermal infrared background image is obtained. The backward diffusion module of the diffusion model is a U-shaped network. The U-shaped network consists of six layers. The left half of the network is the encoder. Each layer of the encoder performs downsampling to reduce the spatial size of the image and increase the feature dimensionality. The right half of the network is the decoder. Each layer of the decoder performs upsampling to increase the spatial size of the image and reduce the feature dimensionality. Skip links exist between the encoder and decoder. The six-layer structure has 128 dimensions for the first and second layers, 256 dimensions for the third and fourth layers, and 512 dimensions for the fifth and sixth layers. Each layer has a processing module, which can be either a convolutional module or an attention module. Except for the fifth layer, which has two attention modules, all other layers have two convolutional modules. The two processing modules in the sixth layer are connected by a processing block, which in turn includes a convolutional module, an attention module, and a convolutional module.

[0024] The convolution module includes a residual structure and a stride information processing module. The residual structure includes two sub-modules. Each sub-module consists of a group normalization layer, a SiLu activation function and a convolution with a convolution kernel size of 3×3 and a stride of 1. The two sub-modules are connected in series; the stride information processing module includes a SiLu activation function and a linear layer. The output of the first sub-module plus the output of the stride information processing module are used as the input of the second sub-module.

[0025] The attention module includes an attention mechanism, where the step information passes through two linear layers to obtain the key K and value V, and the image information passes through a linear layer to obtain the query Q. Get the attention weight, where d k It represents the dimension of K. Softmax is the soft maximum function that can convert the output of the network into a probability distribution. The final weight is then output through a linear layer.

[0026] The loss function of the diffusion model is defined as: where y irepresents the added noise, represents the noise predicted by the diffusion model, and m is the total number of image pixels.

[0027] In step S3, a variety of thermal infrared background images are obtained based on the principal components and the generated residual components. Specifically, the principal components and corresponding weights of the thermal infrared background images are obtained based on step S21. By randomly sampling the weight range, the principal components of any group of thermal infrared background images can be obtained. Furthermore, the residual components of any group of thermal infrared background images can be obtained based on step S23. The residual components of any group of thermal infrared background images are added together to obtain a variety of thermal infrared background images. This can subsequently be used to construct training data for the background removal network.

[0028] The background removal network's dataset is constructed from a high-quality dataset based on visible light. High-quality pseudo-thermal infrared images are generated through simulation and then combined with the background generated by the method of the present invention to form training data pairs for the background removal network. Specifically, the high-quality visible light image is first converted into a grayscale image. This grayscale image is then randomly linearly stretched to a 14-bit range. This grayscale image is then supplemented with the thermal infrared background image generated by the present invention using principal component analysis and diffusion modeling, resulting in training data pairs of a thermal infrared image containing background and a clean thermal infrared image. A state-of-the-art denoising network is then used to perform the background removal task.

[0029] Experimental data demonstrates that neural networks trained using thermal infrared images constructed from thermal infrared background images generated by the method of the present invention significantly outperform networks trained using directly acquired thermal infrared background images. Specifically, networks trained using thermal infrared images constructed from thermal infrared background images generated by the method of the present invention achieve a higher peak signal-to-noise ratio (PSNR), demonstrating that the network is more accurate in restoring image detail. Furthermore, when processing real thermal infrared images, the network generates cleaner image content and reduces residual background information. This advantage not only improves image quality but also enhances the robustness and reliability of the network in practical applications. Therefore, the method of the present invention demonstrates significant potential and advantages in the field of thermal infrared image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a structural diagram of the diffusion model in the method of the present invention.

[0031] Figure 2 Schematic diagram of background synthesis in the method of the present invention.

[0032] Figure 3 Schematic diagram of data synthesis in the method of the present invention.

[0033] Figure 4 This is a comparison chart of the final results obtained in the method of the present invention. DETAILED DESCRIPTION

[0034] The present invention is further described below with reference to specific embodiments and accompanying drawings.

[0035] Example

[0036] A method for generating thermal infrared background images based on principal component analysis and diffusion model is as follows:

[0037] A thermal infrared camera is used to collect the original thermal infrared background image. The specific operation is to use a thermal infrared camera, aim it at a black body with uniform temperature, collect the original data output by the detector at a frequency of 2 frames per second, and collect 2000 thermal infrared background images as the original thermal infrared background image.

[0038] The 2000 thermal infrared background images of size 640*512 collected are stitched into a matrix X ij , i=1,2,3…2000, j=1,2,3…(640*512).

[0039] Calculate the matrix X ij The average value of the pixel values ​​in the second dimension

[0040] Calculate the covariance matrix C=E{(X ij -u i )(X ij -u i ) T}.

[0041] Calculate the eigenvalue matrix λ of the covariance matrix C 2000*2000 , eigenvector matrix V 2000*2000 .

[0042] Calculate the matrix X ij The corresponding component PC = X ij T *V 2000*2000 , we get the contribution of each component based on the eigenvalues. We find that the first two components contain 99.9972% of the information. Therefore, we retain the eigenvectors V2 of the first two components, and the corresponding component weights are the first two eigenvalues ​​λ2.

[0043] Only the first two components of the thermal infrared background image are retained as the main component images

[0044] The main component image is subtracted from the original thermal infrared background image to obtain the residual component of the thermal infrared background image. The residual component is saved as the data set for diffusion model training. The network structure of the diffusion model is as follows: Figure 1 shown.

[0045] The forward diffusion process of the diffusion model is a process of continuously adding noise. For the remaining components of the initial thermal infrared background image, a certain number of diffusion steps is set, and a certain amount of noise is added in each step, and finally the data is converted into a random Gaussian noise matrix.

[0046] The reverse diffusion process of the diffusion model is a continuous denoising process. By passing the random Gaussian noise matrix through the reverse diffusion module of the diffusion model, the noise is predicted and removed. After a certain number of diffusion steps, a clear residual component of the thermal infrared background image can be obtained.

[0047] The main structure of the reverse diffusion module of the diffusion model is a U-shaped network. The U-shaped network includes a total of 6 layers. The left half of the network is the encoder. Downsampling operations are performed between each layer of the encoder to achieve the purpose of reducing the spatial size of the image and increasing the feature dimension. The right half of the network is the decoder. Upsampling operations are performed between each layer of the decoder to achieve the purpose of increasing the spatial size of the image and reducing the feature dimension. There is a jump link between the encoder and the decoder. The dimensions of the six-layer structure are 128 dimensions for the first and second layers, 256 dimensions for the third and fourth layers, and 512 dimensions for the fifth and sixth layers. Each layer is equipped with a processing module, which includes a convolution module or an attention module. Except for the fifth layer, which has two attention modules, the remaining layers have two convolution modules. The two processing modules of the sixth layer are connected by a processing block, which in turn includes a convolution module, an attention module, and a convolution module.

[0048] The convolution module includes a residual structure and a stride information processing module. The residual structure includes two submodules. Each submodule consists of a group normalization layer, a SiLu activation function, and a convolution kernel with a size of 3×3 and a stride of 1. The two submodules are connected in series. The stride information processing module includes a SiLu activation function and a linear layer. The output of the first submodule plus the output of the stride information processing module serve as the input of the second submodule. The attention module includes an attention mechanism. The stride information passes through two linear layers to obtain the key K and value V. The image information passes through a linear layer to obtain the query Q. The formula Get the attention weight, where d k represents the dimension of K. Softmax is the soft maximum function that converts the network output into a probability distribution. The final weight is then output through a linear layer. The specific processing process of the back diffusion module for the remaining components of the thermal infrared background image is shown in

[0048] to

[0051] :

[0049] First, a convolutional layer with a kernel size of 1×1, a stride of 1, and 128 channels is used to process each pixel in the image independently to obtain the image features.

[0050] The feature representation of the image is Ih*w*n, which is then passed through a group normalization layer, a SiLu activation function, and a convolution with a kernel size of 3×3 and a stride of 1 to obtain the result I′ h*w*n The encoding value of the time information passes through a SiLu activation function and a fully connected layer to obtain T h*w*n The obtained I′ h*w*n Add T h*w*n After that, it goes through a group normalization layer, a SiLu activation function and a convolution with a kernel size of 3×3 and a stride of 1 to get the final result O h*w*n The final result is passed through a 2x downsampling module to the next layer of the U-shaped network. The 2x downsampling module first reduces the dimension of the feature image by half through a convolution. The dimension of the feature image changes from (c, h, w) to Then rearrange the image to get the dimension The features are fed into the next layer of the U-shaped network.

[0051] Repeat

[0049] three times. In the fifth layer of the U-shaped network, the fifth layer structure consists of two attention modules. The step information passes through two linear layers to obtain the key K and value V. The image information passes through a linear layer to obtain the query Q. Through the formula Get the attention weight, where d k represents the dimension of K. Softmax is the soft maximum function that converts the network output into a probability distribution. The final weight is then passed through a linear layer output. The output then passes through a downsampling module identical to step

[0049] , and then repeats step

[0049] to complete the downsampling portion of the U-shaped network. After downsampling, it enters the processing block, passing through a convolution module, an attention module, and a convolution module in sequence, before entering the decoder.

[0052] The decoder part is symmetrical to the encoder part. The difference is that the last downsampling step of each layer is changed to upsampling first. The upsampling operation is to double the dimension of the feature image through a convolution. The dimension of the feature image is changed from become The image is then rearranged to obtain features of dimension (c, h, w) and fed into the next layer of the U-shaped network. During the decoding process, skip connections are also used to pass the output of each layer of the encoder directly to the corresponding layer of the decoder.

[0053] The loss function of the diffusion model is defined as: Where yi represents the added noise, represents the noise predicted by the diffusion model, and m is the total number of image pixels.

[0054] In step S21, the principal component analysis method can be used to obtain the main components and corresponding weights of the thermal infrared background image. By randomly sampling its weight range, the main components of any group of thermal infrared background images can be obtained. In step S23, the diffusion model can obtain the residual components of any group of generated thermal infrared background images by inputting different Gaussian noise matrices based on the characteristics of the residual components that have been learned. The addition of the two can obtain a variety of thermal infrared background images. The synthesis method is as follows: Figure 2 As shown, in order to make the shape of the residual component more obvious, the figure shows the residual component after stretching, but the unstretched residual component is actually used in the process of generating the background image.

[0055] The data pairs for background removal network training are constructed as follows: first, the high-quality visible light image is converted into a grayscale image, and then the grayscale image is randomly linearly stretched to a 14-bit range. On this basis, the thermal infrared background image generated by principal component analysis and diffusion model is added to obtain the thermal infrared image with background and the clean thermal infrared image as the training data pair. The generated training data is as follows: Figure 3 Then, the current advanced denoising network is used to achieve the task of removing the background.

[0056] Experimental data show that the network trained with thermal infrared images constructed using the thermal infrared background images generated by this method is significantly better than the network trained with thermal infrared images constructed using directly collected thermal infrared background images. Specifically, the network trained with thermal infrared images constructed using the thermal infrared background images generated by this method has a higher peak signal-to-noise ratio, and can obtain cleaner thermal infrared image content with less background residual information on real thermal infrared images. Results comparison Figure 4 shown.

Claims

1. A method for generating thermal infrared background images based on principal component analysis and diffusion model, characterized in that: A single thermal infrared background image is modeled by principal component analysis and diffusion model to generate various thermal infrared background images. The specific steps of the thermal infrared background image generation method are as follows: S1: Thermal infrared background image acquisition: Use a thermal infrared camera to capture a black body with uniform temperature to obtain the original thermal infrared background image; S2: Modeling thermal infrared background images using principal component analysis and diffusion model; S21: Obtain the components of the thermal infrared background image based on the principal component analysis method, analyze the contribution of each component, and retain the main components and their corresponding weights; The specific process of step S21 is as follows: S211 will collect The size of the The thermal infrared background image is stitched into a matrix ; S212 calculation matrix The average value of the pixel values ​​in the second dimension ; S213 Calculate the covariance matrix ; S214 calculates the covariance matrix The eigenvalue matrix of , the eigenvector matrix ; S215 calculation matrix Corresponding ingredients: , according to the characteristic value, the contribution of each component is obtained, and only the top components with larger contribution values ​​are retained. The eigenvector of the component , the corresponding component weight is eigenvalues ; S216 only retains the front of the thermal infrared background image components as the main component images ; S22: removing the main components of the thermal infrared background image to obtain the remaining components; S23: Learn the features of the remaining components based on the diffusion model to generate the remaining components of the thermal infrared background image; S3: Based on the main components and the generated residual components, a variety of thermal infrared background images are obtained.

2. The method for generating thermal infrared background images based on principal component analysis and diffusion model according to claim 1, characterized in that: The specific operation of collecting the thermal infrared background image in step S1 is to use a thermal infrared camera, aim it at a black body with uniform temperature, and collect the raw data output by the detector at a frequency of 2 frames per second to serve as the original thermal infrared background image.

3. The method for generating thermal infrared background images based on principal component analysis and diffusion model according to claim 1, characterized in that: The step S22 specifically includes: subtracting the main component image from the original thermal infrared background image to obtain the remaining component of the thermal infrared background image, and saving the remaining component as a training set for the diffusion model.

4. The method for generating thermal infrared background images based on principal component analysis and diffusion model according to claim 3, characterized in that: A diffusion model is used to learn the residual component features of the thermal infrared background image and generate output results similar to the distribution of the residual component features of the training set; the training process of the diffusion model includes a forward diffusion process and a backward diffusion process.

5. The method for generating thermal infrared background images based on principal component analysis and diffusion model according to claim 1, characterized in that: The reverse diffusion module of the diffusion model is a U-shaped network, which includes a total of 6 layers. The left half of the network is the encoder, and each layer of the encoder performs a downsampling operation, thereby reducing the spatial size of the image and increasing the feature dimension; the right half of the network is the decoder, and each layer of the decoder performs an upsampling operation, thereby increasing the spatial size of the image and reducing the feature dimension; there is a jump link between the encoder and the decoder; the dimensions of the six-layer structure are 128 dimensions for the first and second layers, 256 dimensions for the third and fourth layers, and 512 dimensions for the fifth and sixth layers; each layer is provided with a processing module, which includes a convolution module or an attention module, the fifth layer has two attention modules, and the remaining layers have two convolution modules; the two processing modules of the sixth layer are connected by a processing block, and the processing block includes a convolution module, an attention module, and a convolution module in sequence; The convolution module includes a residual structure and a step information processing module. The residual structure includes two submodules, each submodule consists of a group normalization layer, a SiLu activation function and a convolution, and the two submodules are connected in series; the step information processing module includes a SiLu activation function and a linear layer, and the output of the first submodule plus the output of the step information processing module are used as the input of the second submodule; The attention module includes an attention mechanism, which is obtained by passing the step information through two linear layers. Sum , the image information is queried through a linear layer , through the formula Get the attention weight, where It means Dimensions, It is a softmax function that converts the output of the network into a probability distribution; the final weight is output through a linear layer.

6. The method for generating thermal infrared background images based on principal component analysis and diffusion model according to claim 5, characterized in that: The loss function of the diffusion model is defined as: ,in represents the added noise, represents the noise predicted by the diffusion model, is the total number of image pixels.

7. The method for generating thermal infrared background images based on principal component analysis and diffusion model according to claim 1, characterized in that: The step S3 specifically comprises: obtaining the main components of any group of thermal infrared background images by randomly sampling the weight range based on the main components and corresponding weights of the thermal infrared background image obtained in step S21; outputting the remaining components of any group of thermal infrared background images based on the diffusion model in step S23; and adding the two together to obtain a variety of thermal infrared background images.

Images