A DSM-constrained method for colorizing high-resolution, high-fidelity, high-resolution 7 panchromatic images

Abstract

This invention relates to a high-resolution, high-fidelity panchromatic image colorization method based on DSM constraints from the Gaofen-7 satellite. It involves acquiring forward and backward panchromatic and backward multispectral images from the Gaofen-7 satellite, preprocessing them to obtain registered panchromatic, DSM, and true-color images, and constructing a DSM-constrained dual-decoder panchromatic image colorization network. Multi-scale features of the DSM and panchromatic images are extracted using a dual-branch ConvNeXt encoder. An adaptive feature selection fusion module fuses and upsamples the multi-scale features of the DSM and panchromatic images, inputting them into a symmetrical luminance decoder and color decoder to reconstruct the luminance and color channels in CIELAB space. The network's internal parameters are optimized by incorporating perceptual and color losses to alleviate problems of low color fidelity, low saturation, and color distortion, ultimately reconstructing a high-resolution, high-fidelity true-color image.

Description

Technical Field

[0001] This invention relates to the field of image colorization and simulation technology, specifically to a method for colorizing high-resolution, high-fidelity, high-resolution 7 panchromatic images based on DSM constraints. Background Technology

[0002] Panchromatic image colorization offers significant benefits in fields such as scene 3D reconstruction, semantic segmentation, and object detection. However, fusion-based panchromatic sharpening methods rely on simultaneous input of panchromatic and multispectral images, failing to function properly if either image is missing, severely limiting their application scenarios. Viewing panchromatic image colorization as a source-to-target domain conversion task avoids this problem. Image conversion methods based on generative adversarial networks (GANs) have developed rapidly and are widely used in multimodal remote sensing image conversion tasks such as SAR to visible light and infrared to visible light. These methods enrich the details of the results through multi-scale feature extraction, increased receptive field information, and effective utilization of contextual information. However, these methods prioritize the realism of the generated results, paying insufficient attention to color saturation and accuracy, resulting in less natural and vibrant results. Unlike the aforementioned conversion methods that operate in the standard RGB color space, image colorization methods use the luminance channel in the CIELAB color space to predict missing color channels, significantly improving the vibrancy and saturation of the generated results. However, these methods only predict color channels, neglecting the spatial structure of the generated results, leading to low structural similarity between the results and the real image, and a tendency for color distortion. Summary of the Invention

[0003] The purpose of this invention is to provide a method for colorizing high-resolution, high-fidelity, high-resolution 7 panchromatic images based on DSM constraints, which can realize the colorization of remote sensing panchromatic images and improve the saturation, vividness, and color accuracy of the generated true-color images.

[0004] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: a method for colorizing high-resolution, high-fidelity, high-resolution 7 panchromatic images based on DSM constraints, comprising the following steps:

[0005] The system acquires front- and rear-view panchromatic and rear-view multispectral images from the Gaofen-7 satellite, and after preprocessing, obtains registered panchromatic, DSM, and true-color images.

[0006] Constructing a DSM-constrained dual-decoder panchromatic image colorization network, including:

[0007] A dual-branch ConvNeXt multi-source multi-scale feature encoder extracts multi-scale features from panchromatic and DSM images;

[0008] The adaptive feature selection fusion module fuses and upsamples the multi-scale features extracted from the panchromatic image and the DSM image;

[0009] The dual-decoder structure includes a symmetrical luminance decoder and a color decoder, which respectively reconstruct the luminance and color channels of the true-color image in CIELAB space from multi-scale fusion features.

[0010] Furthermore, a loss function is established to obtain vibrant true-color images by separating color and brightness in the CIELAB space.

[0011] Preferably, the preprocessing includes:

[0012] Radiometric and orthometric corrections are performed on the front and rear panchromatic stereo images, and a rear-view DSM image is generated and registered with the rear-view panchromatic image.

[0013] Radiometric, atmospheric, and orthorectified corrections were performed on the rear-view multispectral image, and it was geometrically registered with the rear-view panchromatic image. The two images were then fused using the Nearest Neighbor Diffusion Pan Sharpening fusion algorithm to generate a high-resolution true-color rear-view image, which was used as a reference image for training the colorization network.

[0014] Preferably, the pre-processed and registered panchromatic image, DSM image, and true color image are cropped into small blocks of size 256×256 according to the same coordinates without overlapping.

[0015] More preferably, the cropped and registered DSM image, panchromatic image, and true color image are divided into a training set, a validation set, and a test set in a ratio of 7:1:2.

[0016] Preferably, ConvNeXt-L is used to extract shallow texture structure and deep semantic features from panchromatic and DSM images; and the multi-source, multi-scale feature encoder extraction operation of the dual-branch ConvNeXt specifically involves:

[0017] ;

[0018] ;

[0019] in, and These represent the input panchromatic image and DSM image, respectively. , , and The dimensions extracted by ConvNeXt are respectively , , and Panchromatic image feature map, , , and The dimensions extracted by ConvNeXt are respectively , , and The DSM image feature map, where H and W are the height and width of the input image, respectively.

[0020] Preferably, the adaptive feature selection fusion module includes convolutional layers, activation layers, and a multi-scale spatial attention mechanism module, and its operation is as follows:

[0021] ;

[0022] ;

[0023] ;

[0024] ;

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] in, and These are feature maps of the same size extracted by the two encoders. The kernel size is Two-dimensional convolution, For activation function, Indicates will and To splice; , , and This represents a spatial attention mechanism module with convolution kernel sizes of c1, c2, c3, and c4.

[0030] Preferably, the spatial attention mechanism module dynamically adjusts the convolution kernel size during training, utilizing four lightweight sub-networks. , , and Extract separately Each feature constrains the size of the convolution kernel to the range [1,7] through the Sigmoid function and linear transformation. By calculating the average size of the current batch, the single convolution kernel size closest to the average size is found and used as the convolution kernel size of this spatial attention mechanism.

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] in, This represents the calculation of the average size of the current batch. This indicates that the kernel size in spatial attention is constrained to the range [1, 7]. As the activation function, this process uses land cover category information from the DSM image to constrain the panchromatic image, giving different degrees of attention to various land cover types in the panchromatic image, thereby ensuring the accuracy of the colorization results.

[0036] Preferably, four cascaded PixelShuffle layers are used to upsample the multi-scale fused features, with dimensions of 512, 512, 256, and 256, respectively, and the output feature map sizes are respectively... , , and Furthermore, a cross-connection structure is used to concatenate the feature maps of the same size output by the encoder and the upsampling layer. This residual structure helps to fully utilize semantic features, where H and W are the height and width of the input image, respectively.

[0037] Preferably, the luminance decoder and color decoder have the same structure, both including three stacked Transformer block structures. Each Transformer block structure controls three cascaded Transformer blocks, each receiving a sensor with a size of [size missing]. , and The characteristics of the brightness decoder and color decoder are as follows:

[0038] ;

[0039] ;

[0040] in, , These represent the outputs of the color decoder and the luminance decoder, respectively. , These represent the color memory sequence and the brightness memory sequence, respectively. , and These represent the sizes of the upsampling output, respectively. , and The characteristics; C represents the dimension of the output, and These represent the number of queries for the color sequence and the luminance sequence, respectively. ColorDecoder() represents the color decoder, and LumDecoder() represents the luminance decoder.

[0041] The color memory sequence and the brightness memory sequence are mapped to feature maps respectively. Then, through two-dimensional convolution, the output is used to obtain the predicted color. and brightness channel As shown below:

[0042] ;

[0043] ;

[0044] ;

[0045] ;

[0046] in, This is the output of the last PixelShuffle layer. Representing color memory sequences to feature maps The mapping, Represents the brightness memory sequence to feature map The mapping, Indicates the predicted brightness. Indicates the predicted color.

[0047] Preferably, the loss function includes color loss, color distance loss, luminance distance loss, and perceptual loss;

[0048] The color loss is measured using the cosine similarity between the generated image and the reference image, specifically:

[0049] ;

[0050] in, This represents the generated true-color image. Indicates a reference image;

[0051] The color distance loss and luminance distance loss are used The distance was calculated as follows:

[0052] ;

[0053] ;

[0054] in, and These represent the reference color channel and luminance channel, respectively;

[0055] The perceptual loss is calculated by extracting depth features from the colorized result and the reference image using a pre-trained VGG16 model. Distance loss.

[0056] Preferably, the color and luminance channels generated by the luminance decoder and color decoder are stitched together, and then converted from CIELAB space to standard RGB space to obtain a high-resolution, high-fidelity true-color image. The specific conversion process is as follows:

[0057] Convert CIELAB color space to XYZ color space:

[0058] ;

[0059] ;

[0060] ;

[0061] in, , , These represent the predicted luminance channel and two color channels, respectively. , , These represent the values ​​of the three channels when converting from the CIELAB color space to the XYZ color space;

[0062] right , , Perform the same nonlinear transformation, to For example:

[0063] ;

[0064] Multiply by the reference white point value:

[0065] ;

[0066] ;

[0067] ;

[0068] Convert the XYZ color space to the RGB color space:

[0069] ;

[0070] Perform the same gamma correction on R, G, and B, taking R as an example:

[0071] ;

[0072] The RGB result is restricted to [0,1], and multiplied by 255 to complete the generation of the true color image.

[0073] Beneficial Effects: The DSM-constrained dual-decoder panchromatic image colorization network constructed in this invention utilizes the source-to-target domain image transformation approach, generating high-resolution, high-fidelity true-color images from only panchromatic images as input. This avoids the problem of common panchromatic image colorization methods based on panchromatic sharpening requiring both panchromatic and multispectral images as input; it is applicable to scenarios with only panchromatic images available. Furthermore, it innovatively integrates DSM image features into the extracted panchromatic image features, utilizing the land cover categories and edge constraint information at different elevations in the DSM image to impose prior information constraints on the panchromatic image, thereby improving the accuracy of the reconstruction results.

[0074] The adaptive feature selection and fusion module established in this invention can adaptively and dynamically adjust the kernel size in multiple spatial attention mechanism modules, capture and selectively fuse effective features of panchromatic images and DSM images. This process uses land cover category information in DSM images to constrain the panchromatic images, giving different degrees of attention to various land covers in the panchromatic images, thereby ensuring the correctness of the colorization results.

[0075] This invention improves the quality of remote sensing panchromatic image colorization results and solves the problem that conventional panchromatic sharpening methods cannot generate high-resolution, high-fidelity images when multispectral images are missing. It has broad application potential in the field of remote sensing panchromatic image colorization and has broad application prospects in downstream tasks such as real-scene 3D visualization interaction and semantic segmentation. Attached Figure Description

[0076] Figure 1 This is a flowchart of the high-resolution, high-fidelity, high-resolution 7 panchromatic image colorization method based on DSM constraints of the present invention.

[0077] Figure 2 This is a diagram of the dual-decoder panchromatic image colorization network structure based on DSM constraints of the present invention.

[0078] Figure 3 This is a schematic diagram of the adaptive feature selection fusion module of the present invention.

[0079] Figure 4This is a schematic diagram of the structure of the brightness decoder and color decoder of the present invention.

[0080] Figure 5 These are images showing the results of colorizing the Gaofen-7 panchromatic image before and after this invention. Detailed Implementation

[0081] To make the objectives and advantages of this invention clearer, the invention will be specifically described below with reference to embodiments. It should be understood that the following text is merely used to describe one or more specific embodiments of the invention and does not strictly limit the scope of protection specifically claimed by the invention.

[0082] Example: Figure 1 As shown, a method for colorizing a high-resolution, high-fidelity, high-resolution 7 panchromatic image based on DSM constraints includes the following steps:

[0083] Step 1: Acquire front- and rear-looking panchromatic images and rear-looking multispectral images from the Gaofen-7 satellite, and obtain registered panchromatic images, DSM images and true-color images after preprocessing;

[0084] Preprocessing includes: performing radiometric and orthorectification on the fore- and rear-view panchromatic stereo images and generating a rear-view DSM image, which is then registered with the rear-view panchromatic image; performing radiometric, atmospheric, and orthorectification on the rear-view multispectral image and geometrically registering it with the rear-view panchromatic image; and using the Nearest Neighbor Diffusion Pan Sharpening fusion algorithm to fuse the two images to generate a rear-view high-resolution true-color image, which serves as a reference image for training the colorization network.

[0085] The pre-processed and registered rear-view panchromatic image, rear-view true-color image, and rear-view DSM image are cropped into small blocks of 256×256 size according to the same coordinates without overlap.

[0086] The cropped and registered DSM images, panchromatic images, and true-color images were divided into training, validation, and test sets in a ratio of 7:1:2.

[0087] Step 2: Construct a DSM-constrained dual-decoder panchromatic image colorization network, such as... Figure 2As shown, a multi-source, multi-scale feature encoder using a dual-branch ConvNeXt architecture extracts multi-scale features from DSM and panchromatic images. An adaptive feature selection fusion module then fuses the multi-scale features of the DSM and panchromatic images separately. The fused results are upsampled and input into symmetrical luminance and color decoders. The luminance and color channels in the CIELAB space are reconstructed from the two decoders respectively. Luminance and color distance losses are used to separate luminance and color, avoiding their coupling. Perceptual and color losses are combined to optimize the network's internal parameters, alleviating problems such as low color fidelity, low saturation, and color distortion, thus reconstructing a high-resolution, high-fidelity, true-color image. The specific steps are as follows:

[0088] 2.1 A multi-source, multi-scale feature encoder based on dual-branch ConvNeXt is constructed to leverage ConvNeXt's advantages in multi-scale feature modeling, detail enhancement, and context fusion, enabling the encoder to adapt to various land features with significant size differences in Gaofen-7 panchromatic and DSM images. A symmetrical ConvNeXt encoder extracts multi-scale features from both panchromatic and DSM images. ConvNeXt consists of standard convolutional neural network modules and has four variants, arranged from shallowest to deepest: ConvNeXt-T, ConvNeXt-S, ConvNeXt-B, and ConvNeXt-L. Considering the complexity of land feature categories in remote sensing images, ConvNeXt-L is used to extract shallow texture structures and deep semantic features from both panchromatic and DSM images, which will then be incorporated into the multi-scale feature fusion module. The specific extraction operation of the dual-branch ConvNeXt multi-source, multi-scale feature encoder is as follows:

[0089] ;

[0090] ;

[0091] in, and These represent the input panchromatic image and DSM image, respectively. , , and The dimensions extracted by ConvNeXt are respectively , , and Panchromatic image feature map, , , and The dimensions extracted by ConvNeXt are respectively , , and The DSM image feature map, where H and W are the height and width of the input image, respectively.

[0092] 2.2 An adaptive feature selection and fusion module is constructed. This module, based on a spatial attention mechanism, adaptively and dynamically adjusts the kernel size of multiple spatial attention mechanism modules to capture and selectively fuse effective features from panchromatic and DSM images. The fusion result is then upsampled to form a multi-scale feature pyramid for luminance and color channel reconstruction. For the multi-scale features extracted from the symmetrical ConvNeXt panchromatic and DSM images, the adaptive feature selection and fusion module adaptively selects and fuses feature maps of the same scale, such as... Figure 3 As shown, the obtained multi-scale features will be fed into the decoder to reconstruct high-resolution, high-fidelity, true-color images; this module consists of convolutional layers, activation layers, and a multi-scale spatial attention module, and its working process is as follows:

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] ;

[0098] ;

[0099] ;

[0100] ;

[0101] in, and These are feature maps of the same size extracted by the two encoders. The kernel size is Two-dimensional convolution, For activation function, Indicates will and To splice; , , and This represents a spatial attention mechanism module with convolutional kernel sizes c1, c2, c3, and c4. c1, c2, c3, and c4 will dynamically change during training to adapt to features at different scales and selectively fuse effective features. Finally, the feature maps processed by the multi-scale attention mechanism are summed to obtain the selected fusion result. .

[0102] Unlike the basic spatial attention mechanism module, the spatial attention mechanism module established in this embodiment dynamically adjusts the convolution kernel size during training, utilizing four lightweight sub-networks. , , and Extract separately Each feature constrains the size of the convolution kernel to the range [1,7] through the Sigmoid function and linear transformation. By calculating the average size of the current batch, the single convolution kernel size closest to the average size is found and used as the convolution kernel size of this spatial attention mechanism.

[0103] ;

[0104] ;

[0105] ;

[0106] ;

[0107] in, This represents the calculation of the average size of the current batch. This indicates that the kernel size in spatial attention is constrained to the range [1, 7]. This is the activation function.

[0108] This process uses land cover category information from DSM imagery to constrain panchromatic imagery, giving different degrees of attention to various land cover types in the panchromatic imagery, thereby ensuring the accuracy of the colorization results.

[0109] 2.3. Leveraging the advantages of Transformer in global modeling, construct a brightness and color dual decoder based on a multi-scale feature pyramid, referencing... Figure 4 As shown, the symmetrical luminance decoder and color decoder reconstruct the luminance and color information of the true color image from the multi-scale feature pyramid output in step 2.2, respectively. This operation is performed in the CIELAB color space to avoid the strong coupling problem between luminance and color in the standard RGB color space and enhance the color fidelity of the true color image.

[0110] 2.31. Four cascaded PixelShuffle layers are used to upsample the multi-scale fused features obtained in step 2.3, with dimensions of 512, 512, 256, and 256, respectively. The output feature map sizes are respectively... , , and Furthermore, a cross-connection structure is employed to concatenate feature maps of the same size output from the encoder and upsampling layer. This residual structure facilitates the full utilization of semantic features. , and The size features will then serve as input to the symmetrical luminance decoder and color decoder for predicting luminance and color information.

[0111] 2.32. The luminance decoder and color decoder have the same structure, consisting of three cascaded Transformer blocks as a group; they each receive signals with a size of... , and The characteristics of the color decoder are shown below. To fully utilize these characteristics, three sets of Transformer block structures are designed and stacked.

[0112] ;

[0113] ;

[0114] in, , These represent the outputs of the color decoder and the luminance decoder, respectively. , These represent the color memory sequence and the brightness memory sequence, respectively. , and These represent the sizes of the upsampling output, respectively. , and The characteristics; C represents the dimension of the output, and These represent the number of queries for the color sequence and the luminance sequence, respectively. ColorDecoder() represents the color decoder, and LumDecoder() represents the luminance decoder.

[0115] 2.33. Map the color memory sequence and brightness memory sequence to feature maps respectively. Then, through two-dimensional convolution, the output is used to obtain the predicted color. and brightness channel As shown below:

[0116] ;

[0117] ;

[0118] ;

[0119] ;

[0120] in, This is the output of the last PixelShuffle layer. Representing color memory sequences to feature maps The mapping, Represents the brightness memory sequence to feature map The mapping, Indicates the predicted brightness. Indicates the predicted color. Figure 4 In the Transformer module, V, K, and Q represent the three core components of the attention mechanism: value, key, and query.

[0121] 2.34. The generated color and luminance channels are stitched together, and then converted from CIELAB space to standard RGB space to obtain a high-resolution, high-fidelity true-color image.

[0122] The conversion process includes:

[0123] First, convert the CIELAB color space to the XYZ color space:

[0124] ;

[0125] ;

[0126] ;

[0127] in, , , These represent the predicted luminance channel and two color channels, respectively. , , These represent the values ​​of the three channels when converting from the CIELAB color space to the XYZ color space;

[0128] Again , , Perform the same nonlinear transformation, to For example:

[0129] ;

[0130] Multiply by the reference white point value:

[0131] ;

[0132] ;

[0133] ;

[0134] Convert the XYZ color space to the RGB color space:

[0135] ;

[0136] Perform the same gamma correction on R, G, and B, taking R as an example:

[0137] ;

[0138] The RGB result is restricted to [0,1], and multiplied by 255 to complete the generation of the true color image.

[0139] 2.4. Design a comprehensive loss function that includes color loss, color distance loss, luminance distance loss and perceptual loss to optimize the parameters of the colorization network and reconstruct high-fidelity, high-resolution true-color images;

[0140] Color loss is measured using the cosine similarity between the generated image and the reference image, specifically:

[0141] ;

[0142] in, This represents the generated true-color image. Indicates a reference image;

[0143] Color distance loss and luminance distance loss are used The distance was calculated as follows:

[0144] ;

[0145] ;

[0146] in, and These represent the reference color channel and luminance channel, respectively;

[0147] The perceptual loss is calculated by extracting depth features from the colorized result and the reference image using a pre-trained VGG16 model. Distance loss.

[0148] Step 3: Use the Gaofen-7 panchromatic image colorization network trained in Step 2 to complete the colorization task of the Gaofen-7 panchromatic image. Example of the result is shown below. Figure 5 As shown, the colorization results of the panchromatic image have the characteristics of high resolution, high color saturation, and high similarity to the spatial structure and color of the reference image, indicating that the colorization results have high reliability.

[0149] The embodiments of the present invention have been described in detail above with reference to the examples. However, the present invention is not limited to the above embodiments. For those skilled in the art, after learning the contents described in the present invention, several equivalent changes and substitutions can be made without departing from the principle of the present invention. These equivalent changes and substitutions should also be considered to fall within the protection scope of the present invention.

Claims

1. A DSM constraint-based high-resolution high-fidelity panchromatic image colorization method, characterized in that: Comprising the following steps: Step one: collect the front and rear view panchromatic images from Gao Fen 7 satellite, and obtain registered panchromatic images, DSM images and true color images after preprocessing; Step two: build a dual-decoder panchromatic image colorization network based on DSM constraints, including: 2.1, build a multi-source multi-scale feature encoder based on dual-branch ConvNeXt, extract multi-scale features of panchromatic images and DSM images; 2.2, build an adaptive feature selection fusion module, which includes convolutional layers, activation layers and multi-scale spatial attention mechanism modules; based on spatial attention mechanism, the convolution kernel size in the multi-scale spatial attention mechanism module is adaptively adjusted, and the multi-scale features extracted from the panchromatic images and the DSM images are adaptively selected and fused to obtain multi-scale fusion features; 2.3, build a brightness and color dual-decoder based on multi-scale feature pyramid, including symmetrical brightness decoder and color decoder; four PixelShuffle layers are used to upsample the multi-scale fusion features; Reconstruct the brightness and color channels of CIELAB space from the brightness decoder and the color decoder respectively, splice the generated color and brightness channels, and then convert from CIELAB space to standard RGB space to obtain high-resolution and high-fidelity true color images; 2.4, establish a loss function, combine the true color images obtained after preprocessing to optimize the parameters of the dual-decoder panchromatic image colorization network based on DSM constraints, and reconstruct high-resolution and high-fidelity true color images; Step three, use the trained panchromatic image colorization network in step two to complete the colorization of Gao Fen 7 panchromatic images; The preprocessing includes: Radiometric correction and orthorectification are performed on the front and rear view panchromatic image pairs, and the rear view DSM image is generated and registered with the rear view panchromatic image; Radiometric correction, atmospheric correction and orthorectification are performed on the rear view multispectral image, and geometric registration is performed with the rear view panchromatic image. Nearest Neighbor Diffusion Pan Sharpening fusion algorithm is used to fuse the two to generate high-resolution true color images as reference images for training colorization network.

2. The DSM constraint based high resolution high fidelity panchromatic image colorization method according to claim 1, wherein: The registered panchromatic images, true color images and DSM images after preprocessing are cropped into small blocks of 256x256 size according to the same coordinates without overlapping.

3. The DSM constraint based high resolution high fidelity panchromatic image colorization method according to claim 1, wherein: In the multi-source multi-scale feature encoder of dual-branch ConvNeXt, ConvNeXt-L is used to extract shallow texture structure and deep semantic features of panchromatic images and DSM images; and the multi-source multi-scale feature encoder of dual-branch ConvNeXt extraction operation is as follows: ； ； wherein, and respectively represent the input panchromatic image and DSM image, , , and respectively represent the panchromatic image feature maps extracted by ConvNeXt with the size of , , and respectively, , , and respectively represent the DSM image feature maps extracted by ConvNeXt with the size of , , and respectively, and H, W represent the height and width of the input image respectively.

4. The DSM constraint based high resolution high fidelity panchromatic image colorization method according to claim 1, wherein: The working process of the adaptive feature selection fusion module is as follows: ； ； ； ； ； ； ； ； wherein, and are two feature maps of the same size extracted by two encoders respectively, is a two-dimensional convolution with a kernel size of is an activation function, represents splicing and ; , , and represent spatial attention mechanism modules with kernel sizes of c1, c2, c3 and c4 respectively.​ 5. The DSM constraint based high resolution high fidelity panchromatic image colorization method according to claim 1, wherein: The multi-scale spatial attention mechanism module dynamically adjusts the convolution kernel size during the training process, uses four lightweight sub-networks 、 、 and extract the features of , respectively, each through a Sigmoid function and a linear transformation, the size of the convolution kernel is constrained between [1, 7], the single convolution kernel size closest to the average size is found as the convolution kernel size of the spatial attention mechanism by calculating the average size of the current batch.

6. The DSM constraint-based high-resolution high-fidelity panchromatic image colorization method of claim 1, wherein: When the four PixelShuffle layers in series are used to up-sample the multi-scale fusion features, the dimensions are 512, 512, 256, and 256 respectively, and the sizes of the output feature maps are , , and respectively, and the same size feature maps output by the encoder and the up-sampling layer are spliced using a cross connection structure, wherein H and W are the height and width of the input image respectively.

7. The DSM constraint based high resolution high fidelity panchromatic image colorization method according to claim 1, wherein: The luminance decoder and the color decoder have the same structure, each including three groups of stacked Transformer block structures, each group of Transformer block structures receiving features with a size of , and respectively, and controlling three serially connected Transformer blocks; the luminance decoder and the color decoder work as follows: ； ； wherein, , respectively represent the output of the color decoder and the luminance decoder; , respectively represent the color memory sequence and the luminance memory sequence; , and respectively represent the features of the up-sampled output of size , and ; C represents the dimension of the output, and respectively represent the number of lookups of the color sequence and the luminance sequence, ColorDecoder() denotes the color decoder, and LumDecoder() denotes the luminance decoder; mapping the color memory sequence and the luminance memory sequence to feature maps, respectively In the middle, via two-dimensional convolution output, get the predicted color and luminance channels As follows: ； ； ； ； in, This is the output of the last PixelShuffle layer. Representing color memory sequences to feature maps The mapping, Represents the brightness memory sequence to feature map The mapping, Indicates the predicted brightness. Indicates the predicted color.

8. The DSM constraint based high resolution high fidelity panchromatic image colorization method according to claim 1, wherein: The loss function includes color loss, color distance loss, brightness distance loss and perception loss; The color loss is measured by the cosine similarity between the generated image and the reference image, which is specifically: ； wherein, denotes the generated true color image, denotes the reference image; The color distance loss and the brightness distance loss use The distance is calculated as follows: ； ； wherein, and respectively represent the reference color channel and luminance channel. The perception loss extracts deep features of the colorized result and the reference image using a pre-trained VGG16 model and calculates Distance loss.

9. The DSM constraint based high resolution high fidelity panchromatic image colorization method according to claim 1, wherein: The color and luminance channels generated by the luminance decoder and the color decoder are spliced, and are converted from the CIELAB space to the standard RGB space to obtain a high-resolution high-fidelity true-color image, and the conversion is specifically as follows: The CIELAB color space is converted into the XYZ color space: ； ； ； wherein, , , respectively represent the predicted luminance channel and two color channels, , , respectively represent the values of the three channels converted to XYZ color space from CIELAB color space; The same non-linear transformation is performed on , , to as an example: ； The reference white point value is multiplied: ； ； ； The XYZ color space is converted into the RGB color space: ； The same gamma correction is performed on R, G and B, and R is taken as an example: ； The result of RGB is limited within [0, 1], and after being multiplied by 255, the generation of the true-color image is completed.

Images