A forestry resource change monitoring method based on multimodal image generation

Abstract

The invention discloses a forestry resource change monitoring method based on multimodal image generation, which belongs to the technical field of image data processing, including constructing a data set; selecting a cycleGAN network, improving its loss function, and using the data set to train an improved generator; constructing a SAR auxiliary encoder and an improved SETR network; constructing a complete data set G all ; Use G all The improved SETR network is trained to obtain an improved SETR model for change detection. The present invention realizes the effective use of multimodal information, which can not only enhance the training stability and improve the generalization ability, but also make adaptive adjustments according to the image quality, thereby achieving more accurate semantic segmentation of forest images, and combining semantic segmentation maps at different times to obtain more accurate change areas. The present invention can monitor the dynamic changes of forests, and timely identify events such as forest degradation, fire occurrence and disease spread, thereby providing key information for forestry management and disaster response.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a forestry resource change monitoring method based on multimodal image generation. Background Art

[0002] In modern forestry management, remote sensing technology has become a key tool for monitoring forest cover, assessing natural resources and managing disaster response. In particular, in the fields of forest area planning and resource change monitoring, accurate and high-quality image data are crucial. Accurate and reliable remote sensing images are essential for the effective implementation of environmental monitoring and disaster response strategies. However, single-modality remote sensing images (such as optical images only) are often difficult to meet the needs of real-time and comprehensive monitoring due to weather conditions and time constraints. Specifically: (1) Single-modality remote sensing images, especially systems that rely on optical images, are greatly reduced in cloudy, foggy or nighttime conditions, because these conditions severely limit the effectiveness of optical sensors. (2) The processing and update speed of single-modality remote sensing technology may not be sufficient to support the monitoring of rapidly changing environments, especially in disaster management and emergency response scenarios. (3) Single-modality images may not provide sufficient detail resolution in some application scenarios, especially in complex forestry and geospatial environments. (4) Single-modality remote sensing data may be limited in the breadth and depth of application, especially in scientific research and commercial applications that require complex environmental interpretation.

[0003] SAR images are images acquired through synthetic aperture radar (SAR). Synthetic aperture radar uses the principle of synthetic aperture to achieve high-resolution microwave imaging, and has many characteristics such as all-day, all-weather, high resolution, and wide bandwidth.

[0004] SETR: SEgementation TRansformer (SETR) is a semantic segmentation model with a new architecture based on ViT. It is still the commonly used Encoder-Decoder architecture for semantic segmentation. The core idea is to use Transformer instead of CNN as the Encoder. Specifically, the encoder of SETR is a Transformer layer (multi-layer Transformer encoder stack). It first divides the input image into blocks, then embeds each image block and adds position encoding, converts the image into a vector sequence, and then extracts features through the Transformer layer. In the decoder part, SETR uses different upsampling methods to convert the encoder's output vector into a segmentation result.

[0005] The Transformer model is an encoder-decoder architecture, which is divided into two parts: the encoding component and the decoding component. The encoding component is composed of multiple layers of the same structure of the Transformer encoder stacked together, and the Transformer encoder is responsible for mapping the natural language sequence to a hidden layer (containing the mathematical expression of the natural language sequence). The Transformer encoder is composed of multiple identical layers stacked together, and each layer has two sublayers (sublayers are represented as sublayers). The Transformer encoder is composed of multiple identical layers stacked together, and each layer contains two main sublayers: a multi-head self-attention (Multi-Head Self-Attention, multi-head attention module) mechanism and a fully connected feedforward neural network (multi-layer perceptron MLP). The feature sequence entering the Transformer encoder first passes through three linear layers, and the feature sequence is transformed through different linear transformations to obtain multiple sets of different query matrices Q, key matrices K, and value matrices V. The three matrices are then sent to the multi-head attention module for subsequent processing. Summary of the invention

[0006] The purpose of the present invention is to provide a method for monitoring changes in forestry resources based on multimodal image generation to solve the above-mentioned problem, which is difficult to meet the real-time and comprehensive monitoring needs due to weather conditions and time constraints.

[0007] In order to achieve the above-mentioned object, the technical solution adopted by the present invention is as follows: a forestry resource change monitoring method based on multimodal image generation comprises the following steps;

[0008] S1, construct the dataset;

[0009] Obtain optical remote sensing images in different forest scenes to form an optical image dataset G X For each optical remote sensing image, obtain the real SAR image under the same scene to form the SAR image dataset G Y ; The optical remote sensing image and the real SAR image under the same scene correspond one to one, G X The inner image belongs to domain X, G Y The inner image belongs to domain Y;

[0010] S2, select a cycleGAN network, including generator G, generator F, discriminator D X and the discriminator D Y , the generator G is used to convert the image from domain X to domain Y, the generator F is used to convert the image from domain Y to domain X, and the discriminator D X The discriminator D is used to input the image of domain X and output the probability value of it being a real optical remote sensing image. YIt is used to input an image in domain Y and output the probability value of it being a real SAR image;

[0011] S3, improve the loss function of the cycleGAN network, using G X , G Y Train the cycleGAN network, update the generator G, and get the improved generator ;

[0012] S4, construct a SAR auxiliary encoder, including two Transformer encoders, marked as optical encoder TF1 and SAR encoder TF2 respectively. The Q value matrix of TF1 and TF2 is weighted and summed to obtain the fused Q value matrix , and then sent to the multi-head attention module of TF1;

[0013] S5, constructing an improved SETR network, including steps S51 to S53;

[0014] S51, selecting a SETR network, including an image block embedding unit, an encoding layer and a decoding layer, wherein the encoding layer is a 24-layer Transformer encoder;

[0015] S52, there are two image block embedding units, marked as Emb1 and Emb2, which are used to input images of domain X and domain Y respectively, and perform block division and position encoding on the images, and output vector sequences corresponding to the images;

[0016] S53, replace the first 12 layers of Transformer encoders in the coding layer with SAR auxiliary encoders, and TF1 and TF2 in the first layer of SAR auxiliary encoder are marked as , , 1≤l≤12, , The outputs are , , Output Input, Output Input, The input is connected to the output of Emb1, The input of is connected to the output of Emb2, The output is connected to the 13th layer Transformer encoder;

[0017] S6, construct the complete dataset G all ;

[0018] For G X Optical remote sensing images, part of which are from G Y The corresponding real SAR image is found in the Convert to domain Y, output false SAR images, and form a dataset G with real SAR images and false SAR images b ,generate , where a is G X where a is an optical remote sensing image, b is a real SAR image or a false SAR image corresponding to a, and (a, b) is a set of image pairs;

[0019] S7, use G all Train the improved SETR network until convergence to obtain the improved SETR model. During training, the fusion Q value matrix of the l-th layer SAR auxiliary encoder According to steps S71-S72,

[0020] S71, for G b For each image in Y Output the probability value of it being a real SAR image ;

[0021] S72, the fusion Q value matrix of the l-th layer SAR auxiliary encoder is obtained according to the following formula: ;

[0022] ,

[0023] In the formula, for The Q value matrix, for The Q value matrix of

[0024] S8, change detection using improved SETR model;

[0025] Obtain the optical remote sensing image a1 and SAR image b1 at time t1, and the optical remote sensing image a2 and SAR image b2 at time t2, to form image pairs (a1, b1) and (a2, b2), respectively send (a1, b1) and (a2, b2) to the improved SETR model, output the corresponding semantic segmentation maps M1 and M2, and obtain the change area ΔM based on M1 and M2;

[0026] Where b1 is the real SAR image corresponding to a1 or the image generated by the improved generator The false SAR image obtained by converting b1 to domain Y, b2 is the real SAR image corresponding to a2 or the improved generator The false SAR image obtained by converting b2 to domain Y.

[0027] As a preferred embodiment: S3 is specifically:

[0028] S31, construct improved adversarial loss, including generator G and discriminator D YImproved adversarial loss , generator F and discriminator D X Improved adversarial loss ;

[0029] ,

[0030] ,

[0031] In the formula, x is G X Inner optical remote sensing image, y is G Y In the figure, there is a SAR image corresponding to x, G(x) is the output of generator G converting x to domain Y, and F(y) is the output of generator F converting y to domain X. is the interpolation sample between x and G(x), , is the interpolation sample between y and F(y), , E * [·] is to find the expectation of a batch of sample variables, , D Y (y), D Y (G(x)) are the discriminators D Y For the output of y and G(x), D X (x), D X (F(y)) are the discriminators D X For the output of x and F(y), λ1 and λ2 are respectively Y and the discriminator D X The gradient penalty weight represents the calculated gradient;

[0032] S32, construct improved cycle consistency loss L cyc (G,F);

[0033] ,

[0034] Where F(G(x)) is the output of generator F converting G(x) to domain X, G(F(y)) is the output of generator G converting F(y) to domain Y, λ3 is the weight decay coefficient, θ G and θ F are the network parameters of generator G and generator F respectively, is the L1 norm, is the L2 norm;

[0035] S33, construct the self-consistency loss L of the generator id ;

[0036] ,

[0037] Where G(y) is the output of generator G when input y, and F(x) is the output of generator F when input x;

[0038] S34, the total loss L of the cycleGAN network constructed based on improved adversarial loss, improved cycle consistency loss and self-consistency loss total , and use G X , G Y To minimize L total Train the cycleGAN network, update the generator G, and get the improved generator .

[0039] As a preferred embodiment: in S34, L total According to the following formula:

[0040] ,

[0041] In the formula, α and β are L cyc (G,F), The weight of .

[0042] As a preferred embodiment: in S8, , To calculate the absolute value of the pixel at the corresponding position in the image.

[0043] As a preferred embodiment: further comprising step S9;

[0044] S9, threshold processing and classification statistics are performed on the change area ΔM according to the following formula:

[0045] ,

[0046] N is the total number of pixels in ΔM, ΔM(j) is the value of the jth pixel in ΔM, 1≤j≤N, is the screening threshold of the category c to be predicted, is the indicator function, when 1 if yes, 0 otherwise. For more than The number of pixels.

[0047] The improvements of the present invention mainly include the following three points:

[0048] (1) The improved cycleGAN network contains three loss functions, namely improved adversarial loss, improved cycle consistency loss, and generator self-consistency loss. , the improved cycleGAN network is trained according to the new loss function to obtain the trained generator G and discriminator D Y .

[0049] Improved adversarial loss: The adversarial loss trains the generator to generate more realistic images. For each discriminator, the generator tries to generate images that are sufficient to deceive the discriminator, while the discriminator tries to distinguish between real images and generated images. The adversarial loss makes the generated images more visually realistic and more difficult for the discriminator to distinguish. However, mode collapse usually occurs during the training process. In order to alleviate this problem, the present invention introduces a gradient penalty in the original adversarial loss. For example, in Introduced , Introduced The gradient penalty is introduced into the adversarial loss to ensure that the discriminator maintains appropriate smoothness throughout its entire operating range, effectively control the gradient explosion or vanishing problem of the discriminator during training, and enhance the stability of training.

[0050] Improved cycle consistency loss: In order to solve the problem of no paired training data, the present invention introduces an improved cycle consistency loss in the cycleGAN network. This loss function ensures that an image can be reconstructed back to the original image after two conversions (from domain X to domain Y, and then from domain Y back to domain X). The present invention introduces the improved cycle consistency loss in the original cycle consistency loss. ,This method penalizes larger weight values ​​and pushes the model to learn smaller weight values, thereby helping the model learn simpler and more general internal representations and increasing its generalization ability on unseen data.

[0051] For self-consistency loss: A new self-consistency loss is added to ensure that generators G and F can not only convert images from one domain to another, but also keep these images unchanged when encountering images in their own domain. This helps to maintain the consistency of color and other basic information, improve the stability of the model and the quality of generated images.

[0052] Mark the trained generator G as the improved generator , is used to generate false SAR images for optical remote sensing images without real SAR images, and the discriminator D Y For output value, as the fusion Q value matrix in step S72 The weight of As This method will discriminate the weight of D Y The output of is used as a dynamic weight to achieve adaptive adjustment of the fusion process.

[0053] (2) A new SAR auxiliary encoder is proposed. The SAR auxiliary encoder is based on two Transformer encoders, one for inputting optical remote sensing images, so it is marked as optical encoder TF1, and the other for inputting real SAR images or false SAR images, so it is marked as SAR encoder TF2. The Q value matrix of TF1 and TF2 is based on the discriminator D Y The weights are fused to obtain the fused Q value matrix , so the discriminator D Y The output probability determines the weight of the SAR image relative to the influence of the model.

[0054] (3) Construct an improved SETR network, mainly improving the backbone network by changing the 24-layer Transformer encoder to a 12-layer SAR auxiliary encoder and a 12-layer Transformer encoder. For the 12-layer SAR auxiliary encoder, the optical encoder can use SAR information at each layer, improving the model's ability to process complex remote sensing data. At the same time, the 12 layers of the SAR encoder remain relatively independent, ensuring that the unique information of the SAR image will not be diluted, thereby achieving a good balance and fusion of information in the structure of the entire model.

[0055] Compared with the prior art, the advantages of the present invention are:

[0056] (1) Enhanced training stability and improved generalization ability: This paper improves the loss function of the cycleGAN network and controls the gradient of the discriminator through gradient penalty to ensure that the behavior of the discriminator in its input space is smoother and more stable, thereby preventing the gradient explosion or vanishing problem during training. By penalizing large weights in the model, the model is helped to avoid overfitting, enabling the model to better adapt and respond to new, unseen regions or environmental conditions in remote sensing image analysis.

[0057] (2) Effective use of multi-modal information: Optical images excel in visible light and color details, while SAR images can provide effective information in cloud cover or at night. The combination of the two can greatly enhance the accuracy and completeness of analysis and provide more comprehensive forest coverage, especially in extreme weather conditions or low-light environments, ensuring the continuity and integrity of remote sensing monitoring. For optical remote sensing images, if there is a real SAR image, the present invention directly fuses the optical remote sensing image with the real SAR image. If there is no real SAR image, the improved generator is used. Generate a fake SAR image and fuse the optical remote sensing image with the fake SAR image. By generating fake SAR images and fusing images, the model can increase the use of complementary data sources.

[0058] (3) Adaptive adjustment based on image quality: using the discriminator D YThe output of is used as a dynamic weight to adjust its influence in the model according to the authenticity of the SAR image. The model can automatically adjust the processing strategy according to the specific quality of each image, improving the flexibility of the model and its adaptability to complex data.

[0059] (4) Based on the semantic segmentation maps obtained at different times, more accurate change areas can be obtained. By regularly analyzing images at different time points through the change detection algorithm to monitor the dynamic changes of the forest, events such as forest degradation, fire occurrence, and disease spread can be identified in a timely manner, thereby providing key information for forestry management and disaster response.

[0060] (5) The application of the present invention is not limited to monitoring and analysis, but can also accurately reconstruct the required land features at a specific geographical location, such as tree species, water distribution, and other natural landforms. Through change detection algorithms, these features can be continuously tracked over time to timely detect forest degradation, water pollution, or other ecological changes, greatly enhancing the depth and breadth of forestry management and environmental assessment, making decisions in all aspects from forest health monitoring to resource planning more accurate and efficient. Multimodal image segmentation technology not only improves the quality and reliability of remote sensing data, but also brings innovative application potential to the fields of forestry and environmental science, significantly improving the actual effects of forestry management and ecological protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a schematic diagram of the cycleGAN network structure;

[0062] Figure 2 It is a schematic diagram of the SAR auxiliary encoder structure;

[0063] Figure 3 Schematic diagram for generating image pairs;

[0064] Figure 4 Schematic diagram of the improved SETR network structure. DETAILED DESCRIPTION

[0065] The present invention will be further described below in conjunction with the accompanying drawings.

[0066] Example 1: See Figures 1 to 4 ,A forestry resource change monitoring method based on multimodal image generation includes the following steps;

[0067] S1, construct the data set;

[0068] Obtain optical remote sensing images in different forest scenes to form an optical image dataset G X For each optical remote sensing image, obtain the real SAR image under the same scene to form the SAR image dataset G Y; The optical remote sensing image and the real SAR image under the same scene correspond one to one, G X The inner image belongs to domain X, G Y The inner image belongs to domain Y;

[0069] S2, select a cycleGAN network, including generator G, generator F, discriminator D X and the discriminator D Y , the generator G is used to convert the image from domain X to domain Y, the generator F is used to convert the image from domain Y to domain X, and the discriminator D X The discriminator D is used to input the image of domain X and output the probability value of it being a real optical remote sensing image. Y It is used to input an image in domain Y and output the probability value of it being a real SAR image;

[0070] S3, improve the loss function of the cycleGAN network, using G X , G Y Train the cycleGAN network, update the generator G, and get the improved generator ;

[0071] S4, construct a SAR auxiliary encoder, including two Transformer encoders, marked as optical encoder TF1 and SAR encoder TF2 respectively. The Q value matrix of TF1 and TF2 is weighted and summed to obtain the fused Q value matrix , and then sent to the multi-head attention module of TF1;

[0072] S5, constructing an improved SETR network, including steps S51 to S53;

[0073] S51, selecting a SETR network, including an image block embedding unit, an encoding layer and a decoding layer, wherein the encoding layer is a 24-layer Transformer encoder;

[0074] S52, there are two image block embedding units, marked as Emb1 and Emb2, which are used to input images of domain X and domain Y respectively, and perform block division and position encoding on the images, and output vector sequences corresponding to the images;

[0075] S53, replace the first 12 layers of Transformer encoders in the coding layer with SAR auxiliary encoders, and TF1 and TF2 in the first layer of SAR auxiliary encoder are marked as , , 1≤l≤12, , The outputs are , , Output Input, Output Input, The input is connected to the output of Emb1, The input of is connected to the output of Emb2, The output is connected to the 13th layer Transformer encoder;

[0076] S6, construct the complete dataset G all ;

[0077] For G X Optical remote sensing images, part of which are from G Y The corresponding real SAR image is found in the Convert to domain Y, output false SAR images, and form a dataset G with real SAR images and false SAR images b ,generate , where a is G X where a is an optical remote sensing image, b is a real SAR image or a false SAR image corresponding to a, and (a, b) is a set of image pairs;

[0078] S7, use G all Train the improved SETR network until convergence to obtain the improved SETR model. During training, the fusion Q value matrix of the l-th layer SAR auxiliary encoder According to steps S71-S72,

[0079] S71, for G b For each image in Y Output the probability value of it being a real SAR image ;

[0080] S72, the fusion Q value matrix of the l-th layer SAR auxiliary encoder is obtained according to the following formula: ;

[0081] ,

[0082] In the formula, for The Q value matrix, for The Q value matrix of

[0083] S8, change detection using improved SETR model;

[0084] Obtain the optical remote sensing image a1 and SAR image b1 at time t1, and the optical remote sensing image a2 and SAR image b2 at time t2, to form image pairs (a1, b1) and (a2, b2), respectively send (a1, b1) and (a2, b2) to the improved SETR model, output the corresponding semantic segmentation maps M1 and M2, and obtain the change area ΔM based on M1 and M2;

[0085] Where b1 is the real SAR image corresponding to a1 or the image generated by the improved generator The false SAR image obtained by converting b1 to domain Y, b2 is the real SAR image corresponding to a2 or the improved generator The false SAR image obtained by converting b2 to domain Y.

[0086] Other, Figure 2 middle, for The Q value matrix, for The Q value matrix, for The V value matrix, for The K value matrix, for The V value matrix, for The K value matrix of .

[0087] Example 2: See Figures 1 to 4 , based on Example 1, S3 is specifically:

[0088] S31, construct improved adversarial loss, including generator G and discriminator D Y Improved adversarial loss , generator F and discriminator D X Improved adversarial loss ;

[0089] ,

[0090] ,

[0091] In the formula, x is G X Inner optical remote sensing image, y is G Y In the figure, there is a SAR image corresponding to x, G(x) is the output of generator G converting x to domain Y, and F(y) is the output of generator F converting y to domain X. is the interpolation sample between x and G(x), , is the interpolation sample between y and F(y), , E* [·] is to find the expectation of a batch of sample variables, , D Y (y), D Y (G(x)) are the discriminators D Y For the output of y and G(x), D X (x), D X (F(y)) are the discriminators D X For the output of x and F(y), λ1 and λ2 are respectively Y and the discriminator D X The gradient penalty weight represents the calculated gradient;

[0092] S32, construct improved cycle consistency loss L cyc (G,F);

[0093] ,

[0094] Where F(G(x)) is the output of generator F converting G(x) to domain X, G(F(y)) is the output of generator G converting F(y) to domain Y, λ3 is the weight decay coefficient, θ G and θ F are the network parameters of generator G and generator F respectively, is the L1 norm, is the L2 norm;

[0095] S33, construct the self-consistency loss L of the generator id ;

[0096] ,

[0097] Where G(y) is the output of generator G when input y, and F(x) is the output of generator F when input x;

[0098] S34, the total loss L of the cycleGAN network constructed based on improved adversarial loss, improved cycle consistency loss and self-consistency loss total , and use G X , G Y To minimize L total Train the cycleGAN network, update the generator G, and get the improved generator .

[0099] For self-consistency loss, see Figure 1 , for the generator G, its goal is to convert images from domain X to domain Y, but try to keep the images from domain Y unchanged. YThe SAR image y is a real SAR image. When it is input into the generator G and the generator G outputs the false SAR image G(y), G(y)≈y should be satisfied. Similarly, for the generator F, if the input is an image of domain X, it should also remain unchanged as much as possible. That is to say, when the real optical remote sensing image x is input into the generator F, the false optical remote sensing image F(x) it outputs should be satisfied. Therefore, based on this idea, the present invention designs the self-consistency loss L id .

[0100] Example 3, see Figures 1 to 4 Based on Example 2, in S34, L total According to the following formula:

[0101] ,

[0102] In the formula, α and β are L cyc (G,F), The weight of .

[0103] Example 4: See Figures 1 to 4 , based on Examples 1 to 3, in S8, , To calculate the absolute value of the pixel at the corresponding position in the image.

[0104] Example 5: See Figures 1 to 4 , based on embodiments 1 to 4, further comprising step S9;

[0105] S9, threshold processing and classification statistics are performed on the change area ΔM according to the following formula:

[0106] ,

[0107] N is the total number of pixels in ΔM, ΔM(j) is the value of the jth pixel in ΔM, 1≤j≤N, is the screening threshold of the category c to be predicted, is the indicator function, when 1 if yes, 0 otherwise. For more than The number of pixels.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A forestry resource change monitoring method based on multimodal image generation, characterized in that: The steps include: S1, construct the dataset; Obtain optical remote sensing images in different forest scenes to form an optical image dataset G X For each optical remote sensing image, obtain the real SAR image under the same scene to form the SAR image dataset G Y ; The optical remote sensing image and the real SAR image under the same scene correspond one to one, G X The inner image belongs to domain X, G Y The inner image belongs to domain Y; S2, select a cycleGAN network, including generator G, generator F, discriminator D X and the discriminator D Y , the generator G is used to convert the image from domain X to domain Y, the generator F is used to convert the image from domain Y to domain X, and the discriminator D X The discriminator D is used to input the image of domain X and output the probability value of it being a real optical remote sensing image. Y It is used to input an image in domain Y and output the probability value of it being a real SAR image; S3, improve the loss function of the cycleGAN network, using G X , G Y Train the cycleGAN network, update the generator G, and get the improved generator ; S4, construct a SAR auxiliary encoder, including two Transformer encoders, marked as optical encoder TF1 and SAR encoder TF2 respectively. The Q value matrix of TF1 and TF2 is weighted and summed to obtain the fused Q value matrix , and then sent to the multi-head attention module of TF1; S5, constructing an improved SETR network, including steps S51 to S53; S51, selecting a SETR network, including an image block embedding unit, an encoding layer and a decoding layer, wherein the encoding layer is a 24-layer Transformer encoder; S52, there are two image block embedding units, marked as Emb1 and Emb2, which are used to input images of domain X and domain Y respectively, and perform block division and position encoding on the images, and output vector sequences corresponding to the images; S53, replace the first 12 layers of Transformer encoders in the coding layer with SAR auxiliary encoders, and TF1 and TF2 in the first layer of SAR auxiliary encoder are marked as , , 1≤l≤12, , The outputs are , , Output Input, Output Input, The input is connected to the output of Emb1, The input of is connected to the output of Emb2, The output is connected to the 13th layer Transformer encoder; S6, construct the complete dataset G all ; For G X Optical remote sensing images, part of which are from G Y The corresponding real SAR image is found in the Convert to domain Y, output false SAR images, and form a dataset G with real SAR images and false SAR images b ,generate , where a is G X where a is an optical remote sensing image, b is a real SAR image or a false SAR image corresponding to a, and (a, b) is a set of image pairs; S7, use G all Train the improved SETR network until convergence to obtain the improved SETR model. During training, the fusion Q value matrix of the l-th layer SAR auxiliary encoder According to steps S71-S72, S71, for G b For each image in Y Output the probability value of it being a real SAR image ; S72, the fusion Q value matrix of the SAR auxiliary encoder of the first layer is obtained according to the following formula: ; ， In the formula, for The Q value matrix, for The Q value matrix of S8, change detection using improved SETR model; Obtain the optical remote sensing image a1 and SAR image b1 at time t1, and the optical remote sensing image a2 and SAR image b2 at time t2, to form image pairs (a1, b1) and (a2, b2), respectively send (a1, b1) and (a2, b2) to the improved SETR model, output the corresponding semantic segmentation maps M1 and M2, and obtain the change area ΔM based on M1 and M2; Where b1 is the real SAR image corresponding to a1 or the image generated by the improved generator The false SAR image obtained by converting a1 to domain Y, b2 is the real SAR image corresponding to a2 or the improved generator The false SAR image obtained by converting a2 to domain Y; S3 includes S31~S34; 31. Construct an improved adversarial loss, including the generator G and the discriminator D Y Improved adversarial loss , generator F and discriminator D X Improved adversarial loss ; ， ， In the formula, x is G X Inner optical remote sensing image, y is G Y In the figure, there is a SAR image corresponding to x, G(x) is the output of generator G converting x to domain Y, and F(y) is the output of generator F converting y to domain X. is the interpolation sample between x and G(x), , is the interpolation sample between y and F(y), , E * [·] is a variable for a batch of samples Seek hope, , D Y (y), D Y (G(x)) are the discriminators D Y For the output of y and G(x), D X (x), D X (F(y)) are the discriminators D X For the output of x and F(y), λ1 and λ2 are respectively Y and the discriminator D X The gradient penalty weight is Express Calculate the gradient; S32, construct improved cycle consistency loss L cyc (G,F); ， Where F(G(x)) is the output of generator F converting G(x) to domain X, G(F(y)) is the output of generator G converting F(y) to domain Y, λ3 is the weight decay coefficient, θ G and θ F are the network parameters of generator G and generator F respectively, is the L1 norm, is the L2 norm; S33, construct the self-consistency loss L of the generator id ; ， Where G(y) is the output of generator G when input y, and F(x) is the output of generator F when input x; S34, the total loss L of the cycleGAN network constructed based on improved adversarial loss, improved cycle consistency loss and self-consistency loss total , and use G X , G Y To minimize L total Train the cycleGAN network, update the generator G, and get the improved generator .

2. The forestry resource change monitoring method based on multimodal image generation according to claim 1 is characterized in that: In S34, L total According to the following formula: ， In the formula, α and β are L cyc (G,F), The weight of .

3. The forestry resource change monitoring method based on multimodal image generation according to claim 1 is characterized in that: In S8, , To calculate the absolute value of the pixel at the corresponding position in the image.

4. The forestry resource change monitoring method based on multimodal image generation according to claim 1 is characterized in that: Also includes step S9; S9, threshold processing and classification statistics are performed on the change area ΔM according to the following formula: ， N is the total number of pixels in ΔM, ΔM(j) is the value of the jth pixel in ΔM, 1≤j≤N, is the screening threshold of the category c to be predicted, is the indicator function, when 1 if yes, 0 otherwise. For more than The number of pixels.

Images