Remote sensing image colorization method and system based on multi-scale stage feature step-by-step fusion

Through the pre-fusion, semantic information embedding and fusion attention module of the multi-scale generation network, the problems of unreasonable fusion of dark and shallow features and lack of spatial continuity in the coloring of remote sensing images are solved, and better image continuity and context embedding are achieved. , improves the generation effect of remote sensing images.

CN114782568BActive Publication Date: 2024-11-22OCEAN UNIV OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210240490.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-10
Publication Date
2024-11-22
Estimated Expiration
2042-03-10

AI Technical Summary

Technical Problem

The existing remote sensing image coloring methods have problems such as unreasonable fusion of deep and shallow features, lack of spatial continuity, and inability to balance spatial consistency and local detailed information generation. Especially when dealing with remote sensing images with unbalanced category distribution, it is difficult to adequately Utilize image context information.

Method used

A multi-scale generation network is adopted to fuse features of different scales step by step through pre-fusion modules, semantic information embedding modules and fusion attention modules to ensure the spatial continuity of the generated image and the retention of detailed information, and use macroscopic scales to guide microscopic scale generation.

Benefits of technology

The continuity and context embedding of generated images are improved, the discontinuity of the color space caused by image distribution imbalance is smoothed, the correlation between regions is enhanced, and the effective fusion of depth and shallow information is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114782568B_ABST
    Figure CN114782568B_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of image processing, and discloses a remote sensing image coloring method and system for step-by-step fusion of multi-scale stage features. The system comprises an image graying processing module, a generator and a discriminator. The image graying processing module processes the original image into grayscale images of different scales. The generator comprises three generating networks N1, N2 and N3, each comprising a semantic information embedding module. A pre-fusion module is provided between the generating networks N1 and N2, and between N2 and N3. A fusion attention module is provided between the generating networks N1 and N2, and between N2 and N3. The pre-fusion module is used to fuse the grayscale image of the current level network with the shallow features of the previous level network, and serves as the input of the current level network. The semantic information embedding module is used to fuse the network features of the current level and transmit them to the previous level network. The fusion attention module is used to feed back the generated features of networks of different scales to the previous level network step by step, so that the top level network fully contains the generated information of each network, and outputs the generated image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a remote sensing image coloring method and system for step-by-step fusion of multi-scale stage features. Background Art

[0002] Remote sensing image colorization is the process of generating color remote sensing images from grayscale remote sensing images. The colorization of grayscale remote sensing images can increase the ability to interpret and analyze remote sensing images. Traditional colorization methods use a single-scale generative network to implement the colorization process, which makes the coloring lack of spatial continuity and prone to skipped pixels. The cutting-edge colorization method uses a generative adversarial network to achieve colorization, and generates color images through the game between the generator and the discriminator. However, this method has the following problems:

[0003] First, the fusion of deep features and shallow features is unreasonable. The shallow information cannot be fully and reasonably utilized in the generation process. There are significant differences in semantic information and spatial distribution between deep and shallow features. Simple stacking cannot produce satisfactory generation results.

[0004] Second, the spatial continuity of the generated image cannot be guaranteed. It lacks a global perspective and cannot fully incorporate image context information for coloring.

[0005] Third, it is impossible to balance the spatial consistency of colorization and the generation of local detail information. Existing methods focus on learning the local pixel distribution rules in the study of remote sensing images with extremely unbalanced category distribution, which aggravates the instability of colorization; and there is a lack of information interaction between networks of different scales, ignoring the guidance of the macro-scale on the micro-scale network in the generation process. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a remote sensing image colorization method and system with step-by-step fusion of multi-scale stage features, which adopts a multi-scale generation network to fully extract microscopic detail information and macroscopic global information. When fusing multi-scale features, a pre-fusion module and a semantic information embedding module are respectively used in the network input stage, generation stage, and output stage, and a fusion attention module is used to fuse the upper / lower layer features step by step.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] On the one hand, the present invention provides a remote sensing image colorization system with multi-scale stage feature step-by-step fusion, including an image graying processing module, a generator and a discriminator, wherein the image graying processing module is used to gray the original image into grayscale images of different scales; the generator includes three generation networks N1, N2, and N3 with a U-net architecture, and grayscale images of different scales are respectively input into the three generation networks, and the color image generated by N1 is input into the discriminator, and the discriminator determines whether it is the original image or the generated image, and the output images of N2 and N3 are only used for loss calculation;

[0009] Among them, the generating networks N1, N2, and N3 each include a semantic information embedding module. At the front end of the generating network, a pre-fusion module is provided between the generating networks N1 and N2, and between the generating networks N2 and N3; at the back end of the generating network, a fusion attention module is provided between the generating networks N1 and N2, and between the generating networks N2 and N3;

[0010] The pre-fusion module is used to fuse the grayscale image of the current level network with the shallow features of the previous level network, and serve as the input of the current level network.

[0011] The semantic information embedding module is used to fuse the network features of this level and transmit them to the previous level network, wherein the network features of this level fused by the semantic information embedding module include the shallow coding features F extracted from the early network layer i1 , intermediate feature F ie And the deep decoding feature F extracted by the later network layer of U-net in , i = 1, 2, 3, are the numbers of the three generated networks,

[0012] The fusion attention module is used to feed back the generated features of networks of different scales to the upper-level network step by step, so that the upper-level network fully contains the generated information of each network and outputs the generated image.

[0013] Furthermore, the output feature F of the semantic embedding module of the generated network N3 is 3o Respectively with the intermediate features F of the generative network N2 2e And the deep decoding feature F extracted by the U-net late network layer in the generative network N3 3n Fusion, composed of the output features F of the semantic embedding module 3o Compared with the deep decoding feature F extracted by the later network layer of U-net 3n The fused feature F 3,n+1 Generate a color image of the corresponding size of network N3 for calculating the generation loss; process feature F through the fusion attention module 3,n+1 , and the processed features are compared with F 2,n+1 Fusion;

[0014] Generate the output feature F of the semantic embedding module of network N2 2oRespectively with the intermediate features F of the generative network N1 1e And the deep decoding feature F extracted by the U-net late network layer in the generative network N2 2n Fusion, embedding the output features F of the semantic embedding module 2o Compared with the deep decoding feature F extracted by the later network layer of U-net 2n Fusion, generating feature F 2,n+1 , the feature F 2,n+1 With F 3,n+1 The feature F obtained by fusing the attention module (A) 3,n+1 Fusion, get feature F' 2,n+1 , thereby generating a color image of the corresponding size of network N2 for calculating the generation loss;

[0015] Generate the output feature F of the semantic embedding module of network N1 1o The deep decoding feature F extracted from the U-net late network layer in the generative network N1 1n Fusion, get feature F 1,n+1 , the feature F 1,n+1 With F' 2,n+1 The feature F obtained by fusing the attention module (A) 2,n+1 Fusion, get feature F' 1,n+1 , thereby generating a color image of the corresponding size of network N1, which is the final generated color image.

[0016] On the other hand, the present invention provides a remote sensing image colorization method with multi-scale stage feature step-by-step fusion, which is implemented using the system as described above and includes the following steps:

[0017] S1, graying the original image into grayscale images X1, X2, and X3 of different scales, where X1 is the original image size, X2 is the image size of the downsampled image by one time, and X3 is the image size of the downsampled image by two times;

[0018] S2, inputting the grayscale images of different scales in step S1 into the three generating networks N1, N2, and N3 of the generator for data processing, including three stages: in the input processing stage, integrating the features of the upper network as the input of the current network, and increasing the detailed information of the current network; in the generating network stage, integrating the features of the current network through the semantic information embedding module and transmitting them to the upper network step by step, so that the macro-scale network is used as the prior of the micro-scale network; in the network output stage, feeding back the generated features of the networks of different scales to the upper network step by step, so that the uppermost network fully contains the generated information of each network, and outputs the generated image;

[0019] S3, calculate the loss function;

[0020] S4. Optimize the network through the game between the generator and the discriminator.

[0021] Furthermore, the process of generating the network N3 in step S2 is as follows:

[0022] S21, first perform a convolution on X1 to obtain the feature map F 11 , feature map F ij :i represents the network number, j represents the feature number, and F 11 And X2 are fused through the pre-fusion module to obtain the feature map F 21 Similarly, F 21 And X3 are fused through the pre-fusion module to obtain the feature map F 31 ; F 31 Input network N3 for encoding and decoding operations;

[0023] S22, the semantic information embedding module extracts shallow coding features F from the early network layer 31 , intermediate feature F 3e And the deep decoding feature F extracted by the later network layer of U-net 3n Fusion, output feature F 3o ; Feature F 31 To F 3n The features between them are connected by jump connection with the corresponding features of the encoder and decoder in the same way as U-net;

[0024] S23, embed the semantic information into the output feature F of the module 3o Respectively with the embedding feature F of network N2 2e And the deep decoding feature F extracted by the U-net later network layer in network N3 3n Fusion, F 3o With F 3n The fused feature is defined as F 3,n+1 , by F 3,n+1 Generate a color image of the corresponding size of network N3; this image is used to calculate the generation loss and is not used as the output of the network;

[0025] S24. Process feature F by fusion attention module 3,n+1 , and get the feature F (A) 3,n+1, , the feature F (A) 3,n+1, With the feature F of the previous network N2 2,n+1 Fusion, get feature F' 2,n+1 ;

[0026] The above describes the interaction process between the generated networks N3 and N2. The interaction process between the generated networks N2 and N1 is the same as the above process.

[0027] Furthermore, the output feature F of the semantic embedding module of the generated network N3 is 3o The deep decoding feature F extracted from the U-net late network layer in the generative network N3 3n Fusion, get feature F 3,n+1 , feature F 3,n+1 After being processed by the fusion attention module, feature F is obtained (A) 3,n+1, , F (A) 3,n+1 With F 2,n+1 Fusion, generating feature F' 2,n+1 , then generate a color image of the corresponding size of network N2 and output it; at the same time, the feature F' 2,n+1 After being processed by the fusion attention module, feature F is obtained (A) 2,n+1 , the feature F (A) 2,n+1 With feature F 1,n+1 Fusion, generating feature F' 1,n+1 , thereby generating a color image of the corresponding size of network N1 and outputting it.

[0028] Furthermore, the process of processing data by the pre-fusion module is as follows: i+1 After convolution and activation operations, with F i1 The corresponding feature image of is added, and the obtained feature is used as the input feature of the next residual block; in the residual block, the input feature is first convolved, activated, and convolved, and the result is added to the corresponding element image of the input feature of the residual block, and finally the result is activated.

[0029] Furthermore, the process of processing data by the semantic embedding module is as follows: decoding feature F in Perform average pooling on each channel to obtain a feature vector; use the Sigmoid function to activate the obtained vector and obtain the importance ratio of the corresponding channel in all channels; then perform upsampling to restore the corresponding encoding feature F in The ratio is multiplied by the corresponding element of the decoding feature to obtain the fusion result of the decoding feature and the encoding feature; the intermediate feature performs the same operation; the corresponding elements of the fusion result of the decoding feature and the encoding feature, and the corresponding elements of the fusion result of the intermediate feature and the encoding feature are added, and the obtained features are added to the corresponding elements of the decoding feature to obtain the output of this module.

[0030] Furthermore, when the fusion attention module processes data, it performs i,n+1 Perform channel attention and spatial attention operations separately: For channel attention, perform maximum pooling and average pooling on the input features by channel, and obtain feature vectors, which are input into the multi-layer perceptron respectively. After adding the corresponding elements of the results, perform sigmoid activation to obtain the feature Fic ; For spatial attention, the input features are subjected to maximum pooling and average pooling respectively according to the element position, the results are stacked, and then convolution and sigmoid activation operations are performed to obtain the feature map F is ; F ic 、F is Reset to the same size as the input feature, multiply the corresponding elements of the input feature respectively, and add the corresponding elements of the results to get the output of this module.

[0031] Furthermore, the loss function of the network is as follows:

[0032]

[0033]

[0034]

[0035]

[0036]

[0037]

[0038] (1) In the formula, x1 is the sample of the first line, x2 is the sample of the second line, x3 is the sample of the third line, G(x1,x2,x3) is the generated image sample, y is the original image, D(G(x1,x2,x3)) represents the discriminator D's discrimination of the generated image; f1 is the probability that the discriminator judges the generated image to be real; when training the generator G, we hope that the generated images are as real as possible, so the smaller the f1, the better;

[0039] (2) where f2 is the L1 loss function, which minimizes the sum of the absolute differences between the corresponding pixel values ​​of the generated image and the real image. It is the sum of the L1 losses of the three networks. i is a hyperparameter;

[0040] Thus, we get an objective function (3), where is the loss of the generator G. The objective function represents maximizing the probability that the discriminator D identifies the generated image as real and minimizing the corresponding pixel difference between the generated image and the real image;

[0041] (4) In the formula, D(y1|(x1,x2,x3)) represents the discriminator D’s discrimination of the real image y1, and f3 is the probability that the discriminator D discriminates the real image y1 as true, so the larger the better;

[0042] (6) In the formula, D(G(x1,x2,x3)|(x1,x2,x3)) represents the discriminator's discrimination of the generated image, and f4 is the probability that the discriminator D discriminates the generated image as true. Therefore, the smaller it is, the better, and the larger f4 is, the better. Thus, an objective function (6) is obtained; is the loss of the discriminator D. The objective function represents maximizing the probability that the discriminator identifies the real image as real and minimizing the probability that the discriminator identifies the generated image as real.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] 1. The present invention designs multiple generative networks and uses a multi-scale network method to fully extract microscopic detail information and macroscopic global information to improve the continuity of the generated image. Through the larger receptive field of the macro-scale network, more contextual information is captured, the embedding of context is enhanced, and while maintaining the detail information, spatial information is supplemented to improve the correlation between regions.

[0045] 2. The entire network of the present invention is a step-by-step conduction fusion, with downward conduction at the beginning stage and upward feedback fusion at the middle and final stages. That is to say, in the input processing stage, the features of the upper-level network are fused as the input of the network at this level to increase the detailed information of the network at this level; in the generation network stage, the features of the network at this level are fused and transmitted to the upper-level network step by step through the semantic information embedding module, so that the macro-scale network is used as the prior of the micro-scale network, thereby smoothing the discontinuity of the coloring space caused by the imbalanced distribution of remote sensing images and reducing the occurrence of ill-posed problems; in the network output stage, the generated features of the macro-scale network are fed back to the upper-level network step by step, so that the upper-level network fully contains the generated information of each network and outputs the generated image.

[0046] 3. Fully integrate the deep and shallow information in the generated network to achieve effective integration of detail information and spatial information. There are significant differences in semantic information and spatial distribution between low-level and high-level features, so simply superimposing them cannot achieve significant improvement. Through the semantic information embedding module, the semantic and intermediate layer features from high-level features are embedded in low-level features with emphasis, which greatly improves the semantic features of the underlying features without losing the spatial details of the underlying features. Then, they are superimposed on the corresponding decoding features, which greatly improves the effectiveness of the fusion of high-level and low-level features. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0048] Figure 1 It is a schematic diagram of the network structure of the present invention;

[0049] Figure 2 It is a schematic diagram of the generator structure of the present invention;

[0050] Figure 3 It is a schematic diagram of the structure of the pre-fusion module of the present invention;

[0051] Figure 4 It is a schematic diagram of the structure of the semantic information embedding module of the present invention;

[0052] Figure 5 Schematic diagram of the structure of the fusion attention module of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0054] Example 1

[0055] Combination Figure 1 As shown, a remote sensing image colorization system with multi-scale stage feature step-by-step fusion includes an image graying processing module, a generator and a discriminator. The image graying processing module is used to gray the original image into grayscale images of different scales; the generator includes three generating networks N1, N2, and N3 with a U-net architecture, and grayscale images of different scales are respectively input into the three generating networks. In this embodiment, the original image with a size of 256*256 is first grayed to generate a grayscale image, and then the grayscale image is subjected to double average pooling and quadruple average pooling to generate a 128*128 grayscale image and a 64*64 grayscale image respectively. The 256*256 grayscale image is input into N1, the 128*128 grayscale image is input into N2, and the 64*64 grayscale image is input into N3. N1 generates a color image and inputs it into the discriminator. The discriminator determines whether it is an original image or a generated image. The output images of N2 and N3 are only used for loss calculation.

[0056] Recombination Figure 2 As shown, the generative networks N1, N2, and N3 respectively include semantic information embedding modules. At the front end of the generative network, a pre-fusion module is provided between the generative networks N1 and N2, and between the generative networks N2 and N3; at the back end of the generative network, a fusion attention module is provided between the generative networks N1 and N2, and between the generative networks N2 and N3.

[0057] The pre-fusion module is used to fuse the grayscale image of the current level network with the shallow features of the previous level network, and serve as the input of the current level network.

[0058] The semantic information embedding module is used to fuse the network features of this level and transmit them to the previous level network, wherein the network features of this level fused by the semantic information embedding module include the shallow coding features F extracted from the early network layer i1 , intermediate feature F ie And the deep decoding feature F extracted by the later network layer of U-net in , i = 1, 2, 3, are the numbers of the three generated networks,

[0059] The fusion attention module is used to feed back the generated features of networks of different scales to the upper-level network step by step, so that the upper-level network fully contains the generated information of each network and outputs the generated image.

[0060] Combined with the attached drawings, each module is introduced in detail:

[0061] Pre-fusion module combination Figure 3 As shown, the process of pre-fusion module processing data is as follows: i+1 After convolution and activation operations, with F i1 The corresponding feature image of is added, and the obtained feature is used as the input feature of the next residual block; in the residual block, the input feature is first convolved, activated, and convolved, and the result is added to the corresponding element image of the input feature of the residual block, and finally the result is activated.

[0062] Semantic information embedding module combined Figure 4 As shown in the figure, the process of semantic information embedding module processing data is as follows: decoding feature F in Perform average pooling on each channel to obtain a feature vector; use the Sigmoid function to activate the obtained vector and obtain the importance ratio of the corresponding channel in all channels; then perform upsampling to restore the corresponding encoding feature F in The ratio is multiplied by the corresponding element of the decoding feature to obtain the fusion result of the decoding feature and the encoding feature; the intermediate feature performs the same operation; the corresponding elements of the decoding feature and the encoding feature fusion result, and the intermediate feature and the encoding feature fusion result are added, and the obtained features are added to the corresponding elements of the decoding feature to obtain the output of this module. Through the fusion operation of encoding and decoding features and intermediate features (embedded features), the problem that the high and low features are difficult to bridge due to differences in physical information and spatial distribution is solved.

[0063] Fusion attention module combination Figure 5 As shown, the spatial attention and channel attention are combined, and the fusion attention module processes the data for feature F i,n+1 Perform channel attention and spatial attention operations separately: For channel attention, perform maximum pooling and average pooling on the input features by channel, and obtain feature vectors, which are input into the multi-layer perceptron respectively. After adding the corresponding elements of the results, perform sigmoid activation to obtain the feature Fic ; For spatial attention, the input features are subjected to maximum pooling and average pooling respectively according to the element position, the results are stacked, and then convolution and sigmoid activation operations are performed to obtain the feature map F is ; F ic 、F is Reset to the same size as the input feature, multiply the corresponding elements of the input feature respectively, and add the corresponding elements of the results to get the output of this module.

[0064] The following describes the data processing process of each generation network and the interaction process between different generation networks:

[0065] Generate the output feature F of the semantic embedding module of network N3 3o Respectively with the intermediate features F of the generative network N2 2e And the deep decoding feature F extracted by the U-net late network layer in the generative network N3 3n Fusion, composed of the output features F of the semantic embedding module 3o The deep decoding feature F extracted from the U-net late network layer in the generative network N3 3n Fusion, get feature F 3,n+1 , feature F 3,n+1 Generate a color image of the corresponding size of network N3 to calculate the generation loss; at the same time, feature F 3,n+1 After being processed by the fusion attention module, feature F is obtained (A) 3,n+1, ;

[0066] Generate the output feature F of the semantic embedding module of network N2 2o Respectively with the intermediate features F of the generative network N1 1e And the deep decoding feature F extracted by the U-net late network layer in the generative network N2 2n Fusion, embedding the output features F of the semantic embedding module 2o The deep decoding feature F extracted from the U-net late network layer in the generative network N2 2n Fusion, get feature F 2,n+1 , the feature F 2,n+1 With F 3,n+1 The feature F obtained by fusing the attention module (A) 3,n+1 Fusion, get feature F' 2,n+1 , thereby generating a color image of the corresponding size of the network N2 and outputting it for calculating the generation loss; at the same time, the feature F' 2,n+1 After being processed by the fusion attention module, feature F is obtained (A) 2,n+1 ;

[0067] Generate the output feature F of the semantic embedding module of network N1 1o The deep decoding feature F extracted from the U-net late network layer in the generative network N1 1n Fusion, get feature F 1,n+1 , the feature F 1,n+1 With feature F (A) 2,n+1 Fusion, generating feature F' 1,n+1 , thereby generating a color image of the corresponding size of network N1 and outputting it, which is the final generated color image.

[0068] In this embodiment, three scale grayscale images are defined as X1, X2, and X3 respectively. First, a convolution is performed on x1 to obtain a feature map F. 11 (F ij :i represents the network number; j represents the feature number), and F 11 And X2 are fused through the pre-fusion module to obtain the feature map F 21 Similarly, F 21 And X3 are fused through the pre-fusion module to obtain the feature map F 31 . 31 The input generation network N3 performs encoding and decoding operations. Unlike the conventional U-net, this network changes the simple operation of stacking encoder features into the corresponding layer of the decoder in U-net. The semantic information embedding module added by the present invention embeds the shallow coding features F extracted by the early network layer. 31 , embedding feature F 3e And the later network layers extract deep decoding features F with rich semantic information 3n Fusion, output feature F 3o , which solves the problem that high and low features have differences in physical information and spatial distribution and are difficult to bridge. Feature F 31 To F 3n The features between them are connected by jump connection with the encoder-decoder corresponding features in the same way as U-net. The semantic information is embedded in the output feature F of the module. 3o Respectively with the embedding features of the generative network N2 and the deep decoding features F in the generative network N3 3n Fusion, F 3o With F 3n The fused feature is defined as F 3,n+1 , by F 3,n+1 Generate a color image of the corresponding size of the generative network. This image is used to calculate the generation loss and is not used as the output of the network. The feature F is processed by the fusion attention module 3n+1 , and the processed features are compared with F 2,n+1 Fusion.

[0069] The above describes the generation of network N3 and the interaction between network N3 and N2. The structure of the generation network N1 and N2 and the interaction between N2 and N1 are the same as the above process. Finally, the generation network N1 fuses the information of different scale networks to generate a colored image and output it.

[0070] Example 2

[0071] The remote sensing image colorization method of multi-scale stage feature step-by-step fusion is implemented using the system described in Example 1, combined with Figure 1 , including the following steps:

[0072] S1. Gray the original image into grayscale images X1, X2, and X3 of different scales, where X1 is the original image size, X2 is the image size of the downsampled image by one time, and X3 is the image size of the downsampled image by two times.

[0073] S2. Input the grayscale images of different scales in step S1 into the three generating networks N1, N2, and N3 of the generator for data processing, which includes three stages: in the input processing stage, the features of the upper-level network are fused as the input of the current-level network to increase the detailed information of the current-level network; in the generating network stage, the features of the current-level network are fused through the semantic information embedding module and transmitted to the upper-level network step by step, so that the macro-scale network is used as the prior of the micro-scale network; in the network output stage, the generated features of networks of different scales are fed back to the upper-level network step by step, so that the upper-level network fully contains the generated information of each network and outputs the generated image.

[0074] The process of generating network N3 in step S2 is as follows:

[0075] S21, first perform a convolution on X1 to obtain the feature map F 11 , feature map F ij :i represents the network number, j represents the feature number, and F 11 And X2 are fused through the pre-fusion module to obtain the feature map F 21 Similarly, F 21 And X3 are fused through the pre-fusion module to obtain the feature map F 31 ; F 31 Input network N3 for encoding and decoding operations;

[0076] S22, the semantic information embedding module extracts shallow coding features F from the early network layer 31 , intermediate feature F 3e And the deep decoding feature F extracted by the later network layer of U-net 3n Fusion, output feature F 3o ; Feature F 31 To F 3nThe features between them are connected by jump connection with the corresponding features of the encoder and decoder in the same way as U-net;

[0077] S23, embed the semantic information into the output feature F of the module 3o Respectively with the embedding feature F of network N2 2e And the deep decoding feature F extracted by the U-net later network layer in network N3 3n Fusion, F 3o With F 3n The fused feature is defined as F 3,n+1 , by F 3,n+1 Generate a color image of the corresponding size of network N3; this image is used to calculate the generation loss and is not used as the output of the network;

[0078] S24. Process feature F by fusion attention module 3,n+1 , and get the feature F (A) 3,n+1, , the feature F (A) 3,n+1, With the feature F of the previous network N2 2,n+1 Fusion, get feature F' 2,n+1 ;

[0079] The above describes the interaction process between the generated networks N3 and N2. The interaction process between the generated networks N2 and N1 is the same as the above process.

[0080] Generate the output feature F of the semantic embedding module of network N3 3o The deep decoding feature F extracted from the U-net late network layer in the generative network N3 3n Fusion, get feature F 3,n+1 , feature F 3,n+1 After being processed by the fusion attention module, feature F is obtained (A) 3,n+1, , F (A) 3,n+1 With F 2,n+1 Fusion, generating feature F' 2,n+1 , then generate a color image of the corresponding size of network N2 and output it; at the same time, the feature F' 2,n+1 After being processed by the fusion attention module, feature F is obtained (A) 2,n+1 , the feature F (A) 2,n+1 With feature F 1,n+1 Fusion, generating feature F' 1,n+1 , thereby generating a color image of the corresponding size of network N1 and outputting it.

[0081] The data processing process of each module has been described above:

[0082] Pre-fusion module combination Figure 3 As shown in the figure, the input of the fusion attention is the grayscale image of the current level and the upper level feature. The process of pre-fusion module processing data is as follows: i+1 After convolution and activation operations, with F i1 The corresponding feature image of is added, and the obtained feature is used as the input feature of the next residual block; in the residual block, the input feature is first convolved, activated, and convolved, and the result is added to the corresponding element image of the input feature of the residual block, and finally the result is activated.

[0083] Semantic information embedding module combined Figure 4 As shown in the figure, the process of semantic information embedding module processing data is as follows: decoding feature F in Perform average pooling on each channel to obtain a feature vector; use the Sigmoid function to activate the obtained vector and obtain the importance ratio of the corresponding channel in all channels; then perform upsampling to restore the corresponding encoding feature F in The ratio is multiplied by the corresponding element of the decoding feature to obtain the fusion result of the decoding feature and the encoding feature; the intermediate feature performs the same operation; the corresponding elements of the decoding feature and the encoding feature fusion result, and the intermediate feature and the encoding feature fusion result are added, and the obtained features are added to the corresponding elements of the decoding feature to obtain the output of this module. Through the fusion operation of encoding and decoding features and intermediate features (embedded features), the problem that the high and low features are difficult to bridge due to differences in physical information and spatial distribution is solved.

[0084] Fusion attention module combination Figure 5 As shown, when the fusion attention module processes data, the feature F i,n+1 Channel attention and spatial attention operations are performed separately. The upper part of the figure is the processing process of channel attention, and the lower part is the processing process of spatial attention: For channel attention, the input features are subjected to maximum pooling and average pooling by channel, and the obtained feature vectors are input into the multi-layer perceptron respectively. After adding the corresponding elements of the results, sigmoid activation is performed to obtain the feature F ic ; For spatial attention, the input features are subjected to maximum pooling and average pooling respectively according to the element position, the results are stacked, and then convolution and sigmoid activation operations are performed to obtain the feature map F is ; F ic 、F is Reset to the same size as the input feature, multiply the corresponding elements of the input feature respectively, and add the corresponding elements of the results to get the output of this module.

[0085] S3, calculate the loss function;

[0086] The following describes the loss calculation of the network model of the present invention during training:

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] (1) In the formula, x1 is the sample of the first line, x2 is the sample of the second line, x3 is the sample of the third line, G(x1,x2,x3) is the generated image sample, y is the original image, D(G(x1,x2,x3)) represents the discriminator D's discrimination of the generated image; f1 is the probability that the discriminator judges the generated image to be real; when training the generator G, we hope that the generated images are as real as possible, so the smaller the f1, the better;

[0094] (2) where f2 is the L1 loss function, which minimizes the sum of the absolute differences between the corresponding pixel values ​​of the generated image and the real image. It is the sum of the L1 losses of the three networks. i is a hyperparameter;

[0095] Thus, we get an objective function (3), where is the loss of the generator G. The objective function represents maximizing the probability that the discriminator D identifies the generated image as real and minimizing the corresponding pixel difference between the generated image and the real image;

[0096] (4) In the formula, D(y1|(x1,x2,x3)) represents the discriminator D’s discrimination of the real image y1, and f3 is the probability that the discriminator D discriminates the real image y1 as true, so the larger the better;

[0097] (7) In the formula, D(G(x1,x2,x3)|(x1,x2,x3)) represents the discriminator's discrimination of the generated image, and f4 is the probability that the discriminator D discriminates the generated image as true. Therefore, the smaller it is, the better, and the larger f4 is, the better. Thus, we get an objective function (6); is the loss of the discriminator D. The objective function represents maximizing the probability that the discriminator identifies the real image as real and minimizing the probability that the discriminator identifies the generated image as real.

[0098] S4. Optimize the network through the game between the generator and the discriminator;

[0099] When training the generator, the discriminator is fixed, and when training the discriminator, the generator is fixed. As an example, the NWPU-reric45 dataset is used, which consists of 45 scene categories, each category consists of 700 images, and a total of 31,500 images of size 256*256. The network uses 500, 100, and 100 images in each category as training sets, test sets, and validation sets, respectively. The Adam optimizer is used. The hyperparameter λ is set to 100. This network adopts an end-to-end batch training method, and each batch is set to 8.

[0100] S5. Input the remote sensing image into the network optimized by S4 and output the generated image.

[0101] In addition, it should be noted that in the embodiment part, three generation networks are used as an example. The generation network of the present invention can also be more, such as four. The input image of the generation network N4 is 32*32, but the processing effect is not as good as the processing effect of the three generation networks.

[0102] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Any changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A remote sensing image colorization system with multi-scale stage feature fusion, characterized by: It includes an image graying processing module, a generator and a discriminator. The image graying processing module is used to gray the original image into grayscale images of different scales. The generator includes three generating networks N1, N2 and N3 with a U-net architecture. Grayscale images of different scales are respectively input into the three generating networks. N1 generates a color image and inputs it into the discriminator. The discriminator determines whether it is an original image or a generated image. The output images of N2 and N3 are only used for loss calculation. Among them, the generating networks N1, N2, and N3 each include a semantic information embedding module. At the front end of the generating network, a pre-fusion module is provided between the generating networks N1 and N2, and between the generating networks N2 and N3; at the back end of the generating network, a fusion attention module is provided between the generating networks N1 and N2, and between the generating networks N2 and N3; The pre-fusion module is used to fuse the grayscale image of the current level network with the shallow features of the previous level network, and serve as the input of the current level network. The semantic information embedding module is used to fuse the network features of this level and transmit them to the previous level network, wherein the network features of this level fused by the semantic information embedding module include the shallow coding features F extracted from the early network layer i1 , intermediate feature F ie And the deep decoding feature F extracted by the later network layer of U-net in , i = 1, 2, 3, are the numbers of the three generated networks, The fusion attention module is used to feed back the generated features of networks of different scales to the upper-level network step by step, so that the upper-level network fully contains the generated information of each network and outputs the generated image.

2. The remote sensing image colorization system with multi-scale stage feature step-by-step fusion according to claim 1 is characterized in that: Generate the output feature F of the semantic embedding module of network N3 3o Respectively with the intermediate features F of the generative network N2 2e And the deep decoding feature F extracted by the U-net late network layer in the generative network N3 3n Fusion, composed of the output features F of the semantic embedding module 3o Compared with the deep decoding feature F extracted by the later network layer of U-net 3n The fused feature F 3,n+1 Generate a color image of the corresponding size of network N3 for calculating the generation loss; process feature F through the fusion attention module 3,n+1 , and the processed features are compared with F 2,n+1 Fusion; Generate the output feature F of the semantic embedding module of network N2 2o Respectively with the intermediate features F of the generative network N1 1e And the deep decoding feature F extracted by the U-net late network layer in the generative network N2 2n Fusion, embedding the output features F of the semantic embedding module 2o Compared with the deep decoding feature F extracted by the later network layer of U-net 2n Fusion, generating feature F 2,n+1 , the feature F 2,n+1 With F 3,n+1 The feature F obtained by fusing the attention module (A) 3,n+1 Fusion, get feature F' 2,n+1 , thereby generating a color image of the corresponding size of network N2 for calculating the generation loss; Generate the output feature F of the semantic embedding module of network N1 1o The deep decoding feature F extracted from the U-net late network layer in the generative network N1 1n Fusion, get feature F 1,n+1 , the feature F 1,n+1 With F' 2,n+1 The feature F obtained by fusing the attention module (A) 2,n+1 Fusion, get feature F' 1,n+1 , thereby generating a color image of the corresponding size of network N1, which is the final generated color image.

3. A remote sensing image colorization method with step-by-step fusion of multi-scale stage features, characterized in that: The method is implemented by the system according to claim 1 or 2, comprising the following steps: S1, graying the original image into grayscale images X1, X2, and X3 of different scales, where X1 is the original image size, X2 is the image size of the downsampled image by one time, and X3 is the image size of the downsampled image by two times; S2, inputting the grayscale images of different scales in step S1 into the three generating networks N1, N2, and N3 of the generator for data processing, including three stages: in the input processing stage, integrating the features of the upper network as the input of the current network, and increasing the detailed information of the current network; in the generating network stage, integrating the features of the current network through the semantic information embedding module and transmitting them to the upper network step by step, so that the macro-scale network is used as the prior of the micro-scale network; in the network output stage, feeding back the generated features of the networks of different scales to the upper network step by step, so that the uppermost network fully contains the generated information of each network, and outputs the generated image; S3, calculate the loss function; S4. Optimize the network through the game between the generator and the discriminator.

4. The remote sensing image colorization method of multi-scale stage feature step-by-step fusion according to claim 3 is characterized in that: The process of generating network N3 in step S2 is as follows: S21, first perform a convolution on X1 to obtain the feature map F 11 , feature map F ij :i represents the network number, j represents the feature number, and F 11 And X2 are fused through the pre-fusion module to obtain the feature map F 21 Similarly, F 21 And X3 are fused through the pre-fusion module to obtain the feature map F 31 ; F 31 Input network N3 for encoding and decoding operations; S22, the semantic information embedding module extracts shallow coding features F from the early network layer 31 , intermediate feature F 3e And the deep decoding feature F extracted by the later network layer of U-net 3n Fusion, output feature F 3o ; Feature F 31 To F 3n The features between them are connected by jump connection with the corresponding features of the encoder and decoder in the same way as U-net; S23, embed the semantic information into the output feature F of the module 3o Respectively with the embedding feature F of network N2 2e And the deep decoding feature F extracted by the U-net later network layer in network N3 3n Fusion, F 3o With F 3n The fused feature is defined as F 3,n+1 , by F 3,n+1 Generate a color image of the corresponding size of network N3; this image is used to calculate the generation loss and is not used as the output of the network; S24. Process feature F by fusion attention module 3,n+1 , and get the feature F (A) 3,n+1 , the feature F (A) 3,n+1 With the feature F of the previous network N2 2,n+1 Fusion, get feature F' 2,n+1 ; The above describes the interaction process between the generated networks N3 and N2. The interaction process between the generated networks N2 and N1 is the same as the above process.

5. The remote sensing image colorization method of multi-scale stage feature step-by-step fusion according to claim 4 is characterized in that: Generate the output feature F of the semantic embedding module of network N3 3o The deep decoding feature F extracted from the U-net late network layer in the generative network N3 3n Fusion, get feature F 3,n+1 , feature F 3,n+1 After being processed by the fusion attention module, feature F is obtained (A) 3,n+1 , F (A) 3,n+1 With F 2,n+1 Fusion, generating feature F' 2,n+1 , then generate a color image of the corresponding size of network N2 and output it; at the same time, the feature F' 2,n+1 After being processed by the fusion attention module, feature F is obtained (A) 2,n+1 , the feature F (A) 2,n+1 With feature F 1,n+1 Fusion, generating feature F' 1,n+1 , thereby generating a color image of the corresponding size of network N1 and outputting it.

6. The remote sensing image colorization method of multi-scale stage feature step-by-step fusion according to claim 4 is characterized in that: The process of processing data in the pre-fusion module is as follows: i+1 After convolution and activation operations, with F i1 The corresponding feature image of is added, and the obtained feature is used as the input feature of the next residual block; in the residual block, the input feature is first convolved, activated, and convolved, and the result is added to the corresponding element image of the input feature of the residual block, and finally the result is activated.

7. The remote sensing image colorization method of multi-scale stage feature step-by-step fusion according to claim 4 is characterized in that: The process of processing data in the semantic embedding module is as follows: Decoding feature F in Perform average pooling on each channel to obtain a feature vector; use the Sigmoid function to activate the obtained vector and obtain the importance ratio of the corresponding channel in all channels; then perform upsampling to restore the corresponding encoding feature F in The ratio is multiplied by the corresponding element of the decoding feature to obtain the fusion result of the decoding feature and the encoding feature; The same operation is performed on the intermediate features; the corresponding elements of the decoding feature and the encoding feature fusion result, the intermediate feature and the encoding feature fusion result are added, and the obtained features are then added to the corresponding elements of the decoding feature to obtain the output of this module.

8. The remote sensing image colorization method of multi-scale stage feature step-by-step fusion according to claim 4 is characterized in that: When the fusion attention module processes data, the feature F i,n+1 Perform channel attention and spatial attention operations separately: For channel attention, perform maximum pooling and average pooling on the input features by channel, and obtain feature vectors, which are input into the multi-layer perceptron respectively. After adding the corresponding elements of the results, perform sigmoid activation to obtain the feature F ic ; For spatial attention, the input features are subjected to maximum pooling and average pooling respectively according to the element position, the results are stacked, and then convolution and sigmoid activation operations are performed to obtain the feature map F is ; F ic 、F is Reset to the same size as the input feature, multiply the corresponding elements of the input feature respectively, and add the corresponding elements of the results to get the output of this module.

9. The remote sensing image colorization method of multi-scale stage feature step-by-step fusion according to claim 4 is characterized in that: The loss function of the network is as follows: (1) In the formula, x1 is the sample of the first line, x2 is the sample of the second line, x3 is the sample of the third line, G(x1,x2,x3) is the generated image sample, y is the original image, D(G(x1,x2,x3)) represents the discriminator D's discrimination of the generated image; f1 is the probability that the discriminator judges the generated image to be real; when training the generator G, we hope that the generated images are as real as possible, so the smaller the f1, the better; (2) where f2 is the L1 loss function, which minimizes the sum of the absolute differences between the corresponding pixel values ​​of the generated image and the real image. It is the sum of the L1 losses of the three networks. i is a hyperparameter; Thus, we get an objective function (3), where is the loss of the generator G. The objective function represents maximizing the probability that the discriminator D identifies the generated image as real and minimizing the corresponding pixel difference between the generated image and the real image; (4) In the formula, D(y1|(x1,x2,x3)) represents the discriminator D’s discrimination of the real image y1, and f3 is the probability that the discriminator D discriminates the real image y1 as true, so the larger the better; (5) In the formula, D(G(x1,x2,x3)|(x1,x2,x3)) represents the discriminator's discrimination of the generated image, and f4 is the probability that the discriminator D discriminates the generated image as true. Therefore, the smaller it is, the better, and the larger f4 is, the better. Thus, we get an objective function (6); is the loss of the discriminator D. The objective function represents maximizing the probability that the discriminator identifies the real image as real and minimizing the probability that the discriminator identifies the generated image as real.