Spacecraft image fusion method with multiple illumination angles based on unsupervised deep learning
By fusing spacecraft images from multiple illumination angles through an unsupervised deep learning encoder-decoder architecture, we resolve the shadow and dark area problems caused by illumination changes, generate high-quality image inputs, and support computer vision tasks.
Patent Information
- Application Number
- CN202211203401.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-29
- Publication Date
- 2025-09-12
- Estimated Expiration
- 2042-09-29
AI Technical Summary
Existing deep learning image fusion methods cannot effectively process spacecraft images under multiple lighting angles, resulting in limited accuracy of computer vision tasks, especially the shadows and dark areas caused by lighting changes in space environments.
Using unsupervised deep learning methods, a convolutional neural network with an encoder-decoder architecture is constructed. Through feature fusion weight calculation and loss function optimization, spacecraft images under multiple lighting angles are fused to generate a fused image with good lighting.
It effectively removes shadows and dark areas in spacecraft images without the need for manual data annotation, provides high-quality image input, and lays the foundation for computer vision tasks.
Smart Images
Figure CN115511763B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image fusion, and in particular relates to a multi-illumination angle spacecraft image fusion method based on unsupervised deep learning. Background Art
[0002] Modern life is highly dependent on a growing satellite infrastructure, which provides a variety of services, including communications, transportation, and weather forecasting. In addition to this satellite infrastructure, there are also large spacecraft used for space exploration, such as the International Space Station and the Chinese Space Station. Monitoring and maintaining this expensive space equipment is extremely important, but due to the unique characteristics of the space environment, sensors are generally used to remotely monitor the status of satellites.
[0003] Carrying cameras on spacecraft is a common monitoring method. By applying computer vision technology to process rich image data, a lot of useful information can be obtained. A typical application is to use target detection and segmentation technology to detect key parts of spacecraft (handles, antennas, docking holes, etc.), which is crucial for spacecraft docking, maintenance and other tasks.
[0004] However, in space, sunlight shines directly on the spacecraft, while reflected light from the ground has little impact. The data obtained by the camera is heavily dependent on the lighting conditions, often resulting in a large number of shadows and dark areas. As the lighting angle changes, the shadow areas also change. Therefore, even when captured from the same perspective, the image data obtained can vary significantly, severely limiting the accuracy of computer vision technology. By analyzing spacecraft image data, we found that while a single image is heavily dependent on lighting conditions and exhibits defects such as dark areas, as the lighting angle changes, a series of spacecraft images captured from the same perspective by a camera at multiple lighting angles show complementary information. Image fusion methods can be used to fuse this complementary information, thereby removing defects such as dark areas and shadows from a single image.
[0005] Most of the most advanced image fusion algorithms currently available are based on deep learning methods, such as underexposed and overexposed image fusion, multi-focus image fusion, and infrared and visible light image fusion. Deep learning methods offer greater robustness than traditional approaches, but their performance relies on the collection of training datasets and network architecture design tailored to the problem. Therefore, these image fusion methods cannot be directly applied to multi-illumination angle spacecraft image fusion. Summary of the Invention
[0006] This paper addresses the shortcomings of existing technologies by proposing a multi-illumination-angle spacecraft image fusion method based on unsupervised deep learning. By inputting a series of multi-illumination-angle spacecraft images, the method fuses them to generate a single, well-illuminated image with shadows removed, providing excellent input for other computer vision tasks, such as object detection.
[0007] The object of the present invention is achieved through the following technical solution: a multi-illumination angle spacecraft image fusion method based on unsupervised deep learning, comprising the following steps:
[0008] S1. Acquire spacecraft images under multiple illumination angles to construct a dataset. The dataset includes multiple sets of spacecraft image sequences under different viewing angles, where each set of image sequences is an image of the spacecraft under the same viewing angle but different illumination angles.
[0009] S2, converting two images at different illumination angles from the same viewing angle from the blue-green-red space to the optimized color video signal space, and performing weighted fusion on the blue chromaticity component channel and the red chromaticity component channel of the two images to obtain new blue chromaticity component channel and red chromaticity component channel;
[0010] S3. Construct a convolutional neural network based on an encoder-decoder architecture and train it on a natural image dataset to obtain an encoder and decoder with good generalization performance. The luminance component channels of the two images are used as input to the encoder. The encoder extracts features and calculates fusion weights based on the activity levels of the feature maps. The input features are fused according to the fusion weights. After the input features are fused, they are reconstructed into new luminance component channels using the decoder.
[0011] S4. Recombining the luminance component channel generated by the convolutional neural network with the blue chrominance component and red chrominance component channels obtained by weighted fusion to obtain a fused image and converting it back to the blue-green-red space.
[0012] Furthermore, the acquisition of spacecraft images under multiple illumination angles to construct a data set is specifically as follows:
[0013] Public spacecraft 3D models were collected from the Internet. Parallel light was set in the lighting rendering engine to simulate sunlight. The background was set to a dark, matte background. The angle of the parallel light was manually adjusted to construct a dataset of spacecraft simulation images with varying lighting conditions that simulated the space environment. The dataset included multiple image sequences, each of which was a spacecraft simulation image generated by the lighting rendering engine with varying lighting angles at the same viewing angle.
[0014] Furthermore, in step S3, the loss functions used in training the convolutional neural network include mean square error loss, structural similarity loss, and total variation loss. The calculation principles of these three loss functions are as follows:
[0015] First, the mean square error loss is used to evaluate the reconstruction effect between two images, which is represented as the input image y i and the output image y o The expected value of the square of the difference, the mean square error loss l mse The calculation formula is as follows:
[0016] l mse =||y o -y i ||2
[0017] Convolutional neural network input image y i and the output image y o The brightness difference l(y i ,y o ), contrast difference c(y i ,y o ) and structural differences s(y i ,y o ) is calculated by the following formula:
[0018]
[0019] in and is the input image y i The mean and standard deviation of and is the output image y o The mean and standard deviation of is the input image y i and the output image y o The covariance of the two images; c1, c2, c3 are positive constants used to prevent the formula from calculating the exception of division by 0; the structural similarity index SSIM (y i ,y o ) is calculated by the following formula:
[0020]
[0021] Structural similarity loss ssim Expressed as:
[0022] l ssim =1-SSIM(y i ,y o )
[0023] The total variation loss mainly focuses on preserving the image gradient information during image reconstruction while further eliminating noise. tv The calculation formula is as follows:
[0024] R(x,y)=y o (x,y)-y i (x,y)
[0025]
[0026] Where x and y represent the horizontal and vertical coordinates of the image, o (x,y) and y i(x, y) represents the pixel value of the input and output images at the horizontal and vertical coordinates, R(x, y) represents the difference in pixel value between the input and output images at (x, y), and ||·||2 represents the two-norm.
[0027] Furthermore, the fusion weight is determined according to the activity level of the feature map, and its specific meaning is:
[0028] The feature activity level measurement method is used to measure the feature strength between the same regions of different images, and the fusion weight is determined according to the feature strength;
[0029] In the encoder-decoder network architecture, the intermediate feature space is composed of Represents, where i∈{1,2} represents the fused i-th image, m represents the m-th channel of the feature space, Represents all feature spaces from 1 to M, x, y represent the horizontal and vertical coordinates of the image, and the initial feature intensity map C is calculated by the following formula:
[0030]
[0031] Among them C i (x,y) represents the feature intensity map of the i-th input image, and ||·||1 represents the norm;
[0032] Then the local area average is performed. Represents the averaged feature intensity map:
[0033]
[0034] The coefficient r is used to control the size of the averaged local area; r = 1 means that a pixel in the averaged feature intensity map is the average of the pixels in the 3×3 range centered on it in the initial feature intensity map; r = 2 means that the average is performed in a 5x5 local area, where r is 1.
[0035] The fusion weight calculation rules are as follows:
[0036]
[0037] where ω i (x,y) represents the fusion weight corresponding to the coordinate (x,y) of the i-th image. The weight of the image pixel with the largest feature intensity at the coordinate (x,y) is set to 1, and the rest are 0. This weight calculation method is more conducive to fusing the complementary information between different images. The calculation method of the fused image feature space is as follows:
[0038]
[0039] where f m(x,y) represents the feature space of the mth channel of the fused image.
[0040] Furthermore, the method is not only applicable to the fusion of two images, but can also be sequentially extended to fuse multiple images. Specifically, two images are first fused to obtain a fused image, and then the fused result is fused with a third image; in this way, a series of source images are fused.
[0041] The beneficial effects of the present invention are:
[0042] This paper constructs a spacecraft image dataset with multiple illumination angles, which can effectively simulate the space lighting environment. This dataset can be used not only for research on image fusion algorithms, but also for other computer vision tasks such as spacecraft target detection and segmentation.
[0043] This paper proposes an image fusion algorithm based on unsupervised deep learning. This algorithm requires no manually annotated ground-truth images and is trained solely on source image data. To address the problem of multi-illumination-angle spacecraft image fusion, the encoder and decoder are first trained on a large dataset of natural images. This avoids overfitting caused by direct training on the multi-illumination-angle spacecraft image dataset collected by this method. The resulting encoder and decoder possess excellent image feature extraction and reconstruction capabilities.
[0044] This paper achieves good image fusion effects through the design of special loss functions, including the use of structural similarity loss to maintain the structural similarity between the fused image and the source image, and total variation loss to maintain image gradient information.
[0045] The present invention removes illumination defects such as dark areas and shadows by fusing complementary information in spacecraft images at different illumination angles, thus providing good input for further applications of spacecraft images.
[0046] This method is an unsupervised deep learning method that trains a convolutional neural network based on an encoder-decoder architecture directly on a large natural image dataset without the need for labeling information, resulting in an encoder and decoder with good generalization performance.
[0047] This method is not only suitable for the fusion of two images, but can also be extended to fuse multiple images in turn. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 A flowchart of multi-illumination angle spacecraft image fusion based on unsupervised deep learning provided by an embodiment of the present invention;
[0049] Figure 2 A three-dimensional model diagram of a spacecraft provided in an embodiment of the present invention;
[0050] Figure 3 The multi-illumination angle image sequence provided by the embodiment of the present invention;
[0051] Figure 4 A diagram showing the structure of an encoder-decoder convolutional neural network according to an embodiment of the present invention;
[0052] Figure 5 This is a test effect diagram provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0053] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0054] The embodiment of the present invention provides a multi-illumination angle spacecraft image fusion method based on unsupervised deep learning, and its overall fusion process is as follows: Figure 1 As shown in the figure, the input is two images of the same spacecraft under different lighting angles. The two input images are converted from BGR space to YCbCr space. The Cb and Cr channels of the two images are weightedly fused to generate new Cb and Cr channels. The Y channel of the image is extracted by the encoder, and the features extracted from the Y channels of the two images are fused using the calculated feature fusion weights. The features are then reconstructed into a new Y channel by the encoder. The Y channel generated by the neural network is recombined with the Cb and Cr channels obtained by weighted fusion to form a fused image, which is then converted back to BGR space to obtain the fused image.
[0055] The detailed process of deep neural network training and experimentation includes the following steps:
[0056] S1. To simulate the space environment, we collected multiple three-dimensional models of spacecraft from the Internet, such as Figure 1 Then, in the Unreal 4 lighting rendering engine, parallel light was set to simulate sunlight, the background was set to a dark, matte background, and the angle of the parallel light was manually adjusted to obtain images and construct a dataset.
[0057] This dataset includes multiple spacecraft image sequences from different perspectives, and each image sequence is an image of the spacecraft from the same perspective with different illumination angles. Figure 2 The image sequence shown includes images from multiple lighting angles at the same viewing angle. Due to the different lighting angles, each image has varying shadows and occlusions. A shadowed area in one image may be well-lit in another, indicating that information between different images is complementary.
[0058] S2. Convert the two images with different illumination angles from BGR space to YCbCr space, and perform weighted fusion of the Cb and Cr channels of the two images to obtain new Cb and Cr channels. The fusion method of Cb and Cr channels is the same. Taking the Cr channel as an example, the weighted fusion formula used in this method is as follows:
[0059]
[0060] Cr1, Cr2 represent the Cr channel pixel values of the input image, respectively, f Cr Represents the fused Cr channel.
[0061] S3. Construct a convolutional neural network with an encoder-decoder architecture and train it on a common object dataset for image recognition. The COCO2014 image dataset is used for training, with a total of 82,783 images. The encoder and decoder with good generalization ability are obtained. The Y channels of the two images are used as input to the encoder. The encoder extracts features and calculates the fusion weights using the fusion decision rule. The fused features are then reconstructed into a new Y channel by the decoder. The encoder-decoder network structure is as follows: Figure 3 shown.
[0062] like Figure 4 As shown in the figure, the encoder includes a three-layer convolutional neural network, and the number of channels of its output feature maps are 16, 32, and 48 respectively. The gradually increasing number of feature channels is conducive to extracting deep image features. The decoder also includes a three-layer convolutional neural network to restore the intermediate feature vectors to images, and the number of its output channels are 32, 16, and 1 respectively.
[0063] During the training process, structural similarity loss, mean square error loss, and total variation loss are used to calculate the loss error between the fused image and the source image. The weights and biases of each layer of the convolutional neural network are calculated through back propagation. After the loss function converges, the optimal model is obtained. The calculation principles of these three loss functions are as follows:
[0064] First, the mean square error loss is used to evaluate the reconstruction effect between two images, which can be expressed as the input image y i and the output image y o The expected value of the square of the difference is calculated as follows:
[0065] l mse =||y o -y i ||2
[0066] Encoder-decoder network input image y i and the output image y o The brightness difference l(y i ,y o ), contrast difference c(y i ,y o ) and structural differences s(y i ,y o ) can be calculated by the following formula:
[0067]
[0068] in and and The input image y i and the output image y o The mean and standard deviation of is the input image y i and the output image y o The covariance of . c1, c2, c3 are positive constants used to prevent the formula from dividing by 0. The structural detail index (SSIM) of the two images is calculated as follows:
[0069]
[0070] The structural similarity loss is expressed as:
[0071] l ssim =1-SSIM(y i ,y o )
[0072] The total variation loss mainly focuses on preserving the image gradient information during image reconstruction while further eliminating noise. Its calculation principle is as follows:
[0073] R(x,y)=y o (x,y)-y i (x,y)
[0074]
[0075] Where x and y represent the horizontal and vertical coordinates of the image, o (x,y) and y i (x, y) represents the pixel value of the input and output images at the horizontal and vertical coordinates, R(x, y) represents the difference in pixel value between the input and output images at (x, y), and ||·||2 represents the two-norm.
[0076] The fusion weight is determined by the activity level of the feature map. Due to the characteristics of spacecraft images with multiple illumination angles, different images not only have differences in local illumination intensity, but also have shadow areas and dark areas with different distributions. In order to effectively fuse these complementary images and avoid local brightness reduction and blurring caused by fusion of shadow areas, a feature activity level metric is used to measure the feature intensity between the same areas of different images, and the fusion weight is determined based on the feature intensity.
[0077] In the encoder-decoder network architecture, the intermediate feature space can be composed of Represents, where i∈{1,2} represents the fused i-th image, m represents the m-th channel of the feature space, Representing all feature spaces from 1 to M, the initial feature intensity map C can be calculated as follows, where ||·||1 represents the norm:
[0078]
[0079] Then the local area average is performed. Represents the averaged feature intensity map:
[0080]
[0081] Where r = 1 represents the averaged feature strength Figure 1 The pixel is obtained by averaging the pixels in a 3×3 range centered on the initial feature intensity map.
[0082] The fusion weight calculation rules are as follows:
[0083]
[0084] ω i (x,y) represents the fusion weight corresponding to the coordinate (x,y) of the i-th image. We set the weight of the image pixel with the largest feature intensity at the coordinate (x,y) to 1, and the rest to 0. This weight calculation method is more conducive to fusing the complementary information between different images.
[0085] The calculation method of the fused image is as follows, f m (x,y) represents the feature space of the mth channel of the fused image.
[0086]
[0087] After completing feature fusion, the decoder reconstructs the fused features into the image Y channel.
[0088] S4. Recombining the Y channel generated by the neural network with the Cb and Cr channels obtained by weighted fusion to obtain a fused image and converting it back to BGR space.
[0089] After training the neural network, we tested it using spacecraft images from a real space environment. The test results are as follows: Figure 5 As shown in the figure, the two images on the left are the input source images, and the image on the far right is the fusion result. It can be seen that in real space environments, defects such as shadows and dark areas caused by lighting angles do exist. However, the image fusion network trained in the simulation environment can also be directly applied to real environments and achieve good results.
[0090] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.
Claims
1. A multi-illumination angle spacecraft image fusion method based on unsupervised deep learning, characterized in that: The steps include: S1. Acquire spacecraft images under multiple illumination angles to construct a dataset. The dataset includes multiple sets of spacecraft image sequences under different viewing angles, where each set of image sequences is an image of the spacecraft under the same viewing angle but different illumination angles. S2, converting two images at different illumination angles from the same viewing angle from the blue-green-red space to the optimized color video signal space, and performing weighted fusion on the blue chromaticity component channel and the red chromaticity component channel of the two images to obtain new blue chromaticity component channel and red chromaticity component channel; S3. Construct a convolutional neural network based on an encoder-decoder architecture and train it on a natural image dataset to obtain an encoder and decoder with good generalization performance. The luminance component channels of the two images are used as input to the encoder. The encoder extracts features and calculates fusion weights based on the activity levels of the feature maps. The input features are fused according to the fusion weights. After the input features are fused, they are reconstructed into new luminance component channels using the decoder. The loss functions used in training convolutional neural networks include mean square error loss, structural similarity loss, and total variation loss. The calculation principles of these three loss functions are as follows: First, the mean square error loss is used to evaluate the reconstruction effect between two images, which is represented as the input image y i and the output image y o The expected value of the square of the difference, the mean square error loss l mse The calculation formula is as follows: l mse =||and o -and i ||2 Convolutional neural network input image y i and the output image y o The brightness difference l(y i ,y o ), contrast difference c(y i ,y o ) and structural differences s(y i ,y o ) is calculated by the following formula: in and is the input image y i The mean and standard deviation of and is the output image y o The mean and standard deviation of is the input image y i and the output image y o The covariance of the two images; c1, c2, c3 are positive constants used to prevent the formula from calculating the exception of division by 0; the structural similarity index SSIM (y i ,y o ) is calculated by the following formula: Structural similarity loss ssim Expressed as: l ssim =1-SSIM(y i ,y o ) The total variation loss mainly focuses on preserving the image gradient information during image reconstruction while further eliminating noise. tv The calculation formula is as follows: R(x,y)=y o (x,y)-y i (x,y) Where x and y represent the horizontal and vertical coordinates of the image, o (x,y) and y i (x, y) represents the pixel value of the output and input images at the horizontal and vertical coordinates, R(x, y) represents the difference in pixel value between the input and output images at (x, y), and ||·||2 represents the two-norm; S4. Recombining the luminance component channel generated by the convolutional neural network with the blue chrominance component and red chrominance component channels obtained by weighted fusion to obtain a fused image and converting it back to the blue-green-red space.
2. The multi-illumination angle spacecraft image fusion method based on unsupervised deep learning according to claim 1, characterized in that: The acquisition of spacecraft images under multiple illumination angles to construct a data set is specifically as follows: Public spacecraft 3D models were collected from the Internet. Parallel light was set in the lighting rendering engine to simulate sunlight. The background was set to a dark, matte background. The angle of the parallel light was manually adjusted to construct a dataset of spacecraft simulation images with varying lighting conditions that simulated the space environment. The dataset included multiple image sequences, each of which was a spacecraft simulation image generated by the lighting rendering engine with varying lighting angles at the same viewing angle.
3. The multi-illumination angle spacecraft image fusion method based on unsupervised deep learning according to claim 1, characterized in that: The fusion weight is determined by the activity level of the feature map, and its specific meaning is: The feature activity level measurement method is used to measure the feature strength between the same regions of different images, and the fusion weight is determined according to the feature strength; In the encoder-decoder network architecture, the intermediate feature space is composed of Represents, where i∈{1,2} represents the fused i-th image, m represents the m-th channel of the feature space, represents all feature spaces from 1 to M, x and y represent the horizontal and vertical coordinates of the image, and the initial feature intensity map C is calculated by the following formula: Among them C i (x,y) represents the feature intensity map of the i-th input image, and ||·||1 represents the norm; Then the local area average is performed. Represents the averaged feature intensity map: The coefficient r is used to control the size of the average local area; r = 1 means that in the averaged feature intensity map, a pixel is obtained by averaging the pixels in the 3×3 range centered on it in the initial feature intensity map; r = 2 means that the local area of 5x5 is averaged, where r is 1. The fusion weight calculation rules are as follows: where ω i (x,y) represents the fusion weight corresponding to the coordinate (x,y) of the i-th image. The weight of the image pixel with the largest feature intensity at the coordinate (x,y) is set to 1, and the rest are 0. This weight calculation method is more conducive to fusing the complementary information between different images. The calculation method of the fused image feature space is as follows: where f m (x,y) represents the feature space of the mth channel of the fused image.
4. The multi-illumination angle spacecraft image fusion method based on unsupervised deep learning according to claim 1, characterized in that: The method can be sequentially extended to fuse multiple images. Specifically, two images are first fused to obtain a fused image, and then the fused result is fused with a third image; in this way, a series of source images are fused.