Remote sensing image vegetation crown breadth extraction method and system based on improved U-Net model

Abstract

The invention discloses a remote sensing image vegetation crown breadth extraction method and system based on an improved U-Net model, and the method comprises the steps: S1, collecting an unmanned plane image, marking a vegetation crown breadth label, building an image sample data set, and generating a high-fidelity image through a CycleGAN network model; s2, the U-Net vegetation crown breadth extraction model comprises an encoder and a decoder, the encoder comprises N1 contraction layers, each contraction layer is subjected to convolution and down-sampling operation, the decoder comprises N1 expansion layers, each contraction layer is subjected to convolution and up-sampling operation, and a multi-scale enhancement module is horizontally connected between the corresponding layers of the encoder and the decoder; and S3, collecting an unmanned aerial vehicle image of a research area, inputting the unmanned aerial vehicle image into the U-Net vegetation crown breadth extraction model, and outputting a vegetation crown breadth prediction mask after crown breadth segmentation. According to the method, the unmanned aerial vehicle image of the research area is shot through the unmanned aerial vehicle, the vegetation crown breadth can be accurately extracted and segmented, large-area rapid detection of the research area is achieved, and the method is of great significance to ecological and environmental monitoring.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing, and in particular to a remote sensing image vegetation crown extraction method and system based on an improved U-Net model. Background Art

[0002] Vegetation crown width refers to the average width of the crown of trees or seedlings in the north-south and east-west directions. It is used to indicate the specifications of trees and seedlings and to measure their growth. It represents the width of the crown of trees or seedlings and is an important reference standard for measuring plant growth and health. The measurement of crown width usually needs to consider factors such as the morphology of the tree, the shape of the crown, and the growth status of the tree. Common measurement methods include direct measurement and indirect measurement; direct measurement method: use measuring tools (such as tape measures, laser rangefinders, etc.) to directly measure the width of the crown in the north-south and east-west directions, and then take the average value; indirect measurement method: calculate the crown width based on the projection area of ​​the crown or other relevant parameters; for example, the crown width can be estimated by measuring the circumference and height of the crown, and then using mathematical formulas or models. The above two methods have technical problems such as difficulty in obtaining plant crown width parameters and large workload.

[0003] Extraction of vegetation crown width and tree crown parameters based on satellite remote sensing images is gradually emerging. At present, the research directions of extracting tree crown parameters based on satellite remote sensing images are mainly divided into the following two categories: the first method directly obtains tree crown information from the image by identifying the spectral and texture differences between tree crowns and other objects; this method has poor universality because the spectral and texture characteristics of tree crowns and other objects may differ significantly due to environmental changes or different characteristics of the objects themselves. The second method relies on geometric optical features and image features (such as local ray method, watershed segmentation method and regional growing method, etc.) to extract the edge of the tree crown; this method is easily affected by the complexity and uncertainty of image texture, thereby reducing stability. Vegetation crown width extraction is an important task in remote sensing image analysis. Researchers need to obtain information such as tree growth status, biomass and wood properties by studying tree crowns, which is of great value and significance to environmental monitoring, agricultural management and ecological research. Summary of the invention

[0004] The purpose of the present invention is to solve the technical problems pointed out by the background technology, and provide a remote sensing image vegetation crown extraction method based on an improved U-Net model. The image sample data set is subjected to generative adversarial processing according to paired image groups through the CycleGAN network model to obtain high-fidelity images to enrich the image sample data set. Multi-scale feature extraction and crown segmentation and recognition training are performed by improving the U-Net vegetation crown extraction model. The multi-scale enhancement module effectively solves the technical problem that the instance boundaries are difficult to distinguish due to the easy overlap of plant crowns. At the same time, the multi-scale enhancement module enhances the U-Net vegetation crown extraction model's mining of multi-scale contexts, thereby improving the crown extraction accuracy.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A method for extracting vegetation crown width from remote sensing images based on an improved U-Net model, the method comprising:

[0007] S1. Collect drone images and mark vegetation canopy labels to construct an image sample dataset; construct a CycleGAN network model, input the image sample dataset into the CycleGAN network model in pairs of image groups, and use the image sample dataset for model training. Each pair of image groups contains two visually similar images. The CycleGAN network model generates high-fidelity images and stores them in the image sample dataset;

[0008] S2. Construct an improved U-Net vegetation crown extraction model. The U-Net vegetation crown extraction model includes an encoder and a decoder. The encoder includes N1 contraction layers that are output sequentially. Each contraction layer performs convolution and downsampling operations. The decoder includes N1 expansion layers that are output sequentially. Each contraction layer performs convolution and upsampling operations. The N1 contraction layers correspond to the N1 expansion layers by layer and are horizontally connected between the corresponding layers. There is a multi-scale enhancement module. The multi-scale enhancement module includes N2 sequential and parallel branches. The multi-scale enhancement module takes the features of the contraction layer as input and obtains the feature RMFE i , the feature RMFE i Add splicing to the input of the same-level expansion layer; the last expansion layer of the U-Net vegetation crown extraction model outputs a pixel-level crown segmentation mask;

[0009] S3. Collect UAV images of the study area and input them into the U-Net vegetation crown extraction model to output the vegetation crown prediction mask after crown segmentation.

[0010] In order to better implement the present invention, the CycleGAN network model of the present invention includes two generators G and F and two discriminators D X , D Y, the CycleGAN network model performs bidirectional cycle processing on the real images x and y in the paired image group according to the following method:

[0011] Take the real image x as the input image, and the generator G generates a fake image By the discriminator D x To judge whether the image is true or false, Input the generator and generate fake images The CycleGAN network model uses loss function supervision The degree of fidelity with x;

[0012] Take the real image y as the input image, and the generator G generates a fake image By the discriminator D x To judge whether the image is true or false, Input the generator and generate fake images The CycleGAN network model uses loss function supervision The degree of fidelity between y and y.

[0013] Preferably, the encoder and decoder of the U-Net vegetation crown extraction model are constructed according to the U-Net convolutional neural network structure, each contraction layer includes two convolutions and one downsampling operation, each convolution process is composed of a 3×3 convolution kernel and a rectified linear unit activation function, and the downsampling operation adopts 2×2 maximum pooling; each expansion layer includes two convolutions and one upsampling operation, each convolution process is composed of a 3×3 convolution kernel and a rectified linear unit activation function.

[0014] Preferably, the U-Net vegetation crown extraction model has an output layer after the last expansion layer, and the output layer is processed by 1×1 convolution and Softmax activation function. The expression of Softmax activation function is as follows:

[0015] where a k (x) is the activation value of each pixel point x corresponding to category k on the feature map, Pk(x) is the probability that the pixel point x corresponds to category k, and K is the number of categories.

[0016] Preferably, the encoder of the U-Net vegetation crown width extraction model includes four layers of contraction layers, the decoder includes four layers of expansion layers, the first layer of contraction layer and the first layer of expansion layer are horizontally connected with a first layer of multi-scale enhancement module, the second layer of contraction layer and the second layer of expansion layer are horizontally connected with a second layer of multi-scale enhancement module, the third layer of contraction layer and the third layer of expansion layer are horizontally connected with a third layer of multi-scale enhancement module, and the fourth layer of contraction layer and the fourth layer of expansion layer are horizontally connected with a fourth layer of multi-scale enhancement module; the first to fourth layers of contraction layers are sequentially subjected to different scale feature encoding processing, and the fourth to first layers of expansion layers are sequentially subjected to different scale feature encoding processing. Decoding processing of features of different scales: the feature output of the fourth contraction layer is concatenated with the feature output of the fourth multi-scale enhancement module as the feature input of the fourth expansion layer; the feature output of the fourth expansion layer is concatenated with the feature output of the third multi-scale enhancement module as the feature input of the third expansion layer; the feature output of the third expansion layer is concatenated with the feature output of the second multi-scale enhancement module as the feature input of the second expansion layer; the feature output of the second expansion layer is concatenated with the feature output of the first multi-scale enhancement module as the feature input of the first expansion layer; the first expansion layer outputs the crown segmentation mask.

[0017] Preferably, the multi-scale enhancement module of the present invention includes four sequential and parallel branches, each branch configuration includes 1×1 convolution, 1×3 and 3×1 asymmetric convolution, and the output of each branch is added to the input of the next branch, and the expression is as follows:

[0018] where F i is the i-th feature map of the input, k is the number of branches, is the output of the k-th branch and the i-th feature map, For element-by-element addition, Conv r () is the convolutional layer;

[0019] The four branch outputs are concatenated, followed by a 1×1 convolution, the channel is adjusted to 64, and added to the input feature F i Finally, the output feature RMFE is obtained i , i∈{1, 2, 3, 4}

[0020] Where Conv0 represents 1×1 convolution, Indicates that all four branches are spliced.

[0021] Preferably, the overall loss function Loss expression of the U-Net vegetation crown width extraction model of the present invention is as follows:

[0022] Loss = L BCE +L IOU , where L BCErepresents the cross entropy loss, L IOU represents the intersection-and-combination loss;

[0023] L BCE =-∑(P(x,y)*ln(G(x,y))+(1-P(x,y))*ln(1-G(x,y)));

[0024]

[0025] P(x, y) and G(x, y) represent the crown segmentation mask and crown true value mask of the pixel at the original image coordinate position (x, y), respectively. x is an integer in [0, W-1], y is an integer in [0, H-1], and H and W represent the length and width of the image, respectively.

[0026] Preferably, the CycleGAN network model of the present invention is built by using the Pytorch deep learning framework, and the CycleGAN network model uses cycle consistency loss to optimize model parameters. cyc The expression of (G, F) is as follows:

[0027]

[0028] in represents expectation, F(G(x)) represents the output of the CycleGAN network model after processing the real image x as input, G(F(y)) represents the output of the CycleGAN network model after processing the real image y as input, and || || represents the L1 norm.

[0029] A remote sensing image vegetation crown width extraction system based on an improved U-Net model comprises a data acquisition module, an image sample data set, a CycleGAN network model and a U-Net vegetation crown width extraction model. The image sample data set is constructed by the data acquisition module collecting drone images and annotating vegetation crown width labels; the CycleGAN network model inputs the image sample data set into paired image groups and uses the image sample data set to perform model training. Each paired image group contains two visually similar images. The CycleGAN network model generates The high-fidelity image is formed and stored in the image sample data set; the U-Net vegetation crown extraction model includes an encoder and a decoder, the encoder includes N1 contraction layers output in sequence, each contraction layer performs convolution and downsampling operations, the decoder includes N1 expansion layers output in sequence, each contraction layer performs convolution and upsampling operations, the N1 contraction layers correspond to the N1 expansion layers in layers, and there is a multi-scale enhancement module horizontally connected between the corresponding layers, the multi-scale enhancement module includes N2 sequential and parallel branches, the multi-scale enhancement module uses the features of the contraction layer as input and obtains the feature RMFE i, the feature RMFE i Add splicing to the input of the expansion layer of the same layer; the last expansion layer of the U-Net vegetation crown extraction model outputs a pixel-level crown segmentation mask; the data acquisition module also collects drone images of the study area and inputs them into the trained U-Net vegetation crown extraction model, and then outputs the vegetation crown prediction mask after crown segmentation as the vegetation crown extraction result.

[0030] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0031] (1) The present invention uses the CycleGAN network model to generate adversarial processing on the image sample dataset according to paired image groups to obtain high-fidelity images to enrich the image sample dataset. By improving the U-Net vegetation crown extraction model, multi-scale feature extraction and crown segmentation and recognition training are performed. The multi-scale enhancement module effectively solves the technical problem that the instance boundaries are difficult to distinguish due to the easy overlap of plant crowns. At the same time, the multi-scale enhancement module enhances the U-Net vegetation crown extraction model's mining of multi-scale contexts, thereby improving the crown extraction accuracy.

[0032] (2) The present invention uses drone images of the study area to accurately extract the vegetation crown width and segment it to obtain the vegetation crown width prediction mask based on the drone images of the study area. The vegetation crown width data of each location point in the study area can be quickly obtained, thereby realizing rapid detection and analysis of a large area in the study area, which is of great significance to ecological and environmental monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the method of the present invention;

[0034] Figure 2 Schematic diagram of the principle of bidirectional cycle processing of the CycleGAN model in the embodiment;

[0035] Figure 3 It is a structural schematic diagram of the improved U-Net vegetation crown width extraction model in the embodiment;

[0036] Figure 4 Schematic diagram of the structure of the multi-scale enhancement module in the embodiment;

[0037] Figure 5 To illustrate the mapping schematic diagram of the Daliuta coal mine subsidence area as a study area;

[0038] Figure 6 To give an example, the crown width extraction and segmentation effects of several small blocks were taken from the Daliuta coal mine subsidence area as the study area. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below in conjunction with embodiments:

[0040] Example

[0041] like Figure 1 As shown, a method for extracting vegetation crown width from remote sensing images based on an improved U-Net model includes:

[0042] S1. Collect drone images and annotate vegetation canopy labels to construct an image sample dataset (use drones to collect actual data and annotate the data). Construct a CycleGAN network model, input the image sample dataset into the CycleGAN network model in paired image groups, and use the image sample dataset for model training (to train the U-Net vegetation canopy extraction model, divide the dataset into training set, validation set, and test set. To train the CycleGAN network model, screen similar paired images in the dataset and divide the paired images into training set, validation set, and test set). Each paired image group contains two visually similar images. The CycleGAN network model generates high-fidelity images and stores them in the image sample dataset.

[0043] This paper takes the Daliuta coal mine subsidence area as an example of the research area. The survey area of ​​the research area is as follows: Figure 5 As shown in the figure, the present invention uses the CycleGAN model to generate high-fidelity training data, thereby alleviating the problem of data scarcity; uses the improved U-Net vegetation crown extraction model to extract the features of UAV remote sensing images, and uses the U-Net vegetation crown extraction model to predict the plant crown width, thereby improving the accuracy and automation of crown width extraction; uses the residual multi-scale feature extraction module RMFE to enhance the perception of subtle crown width edges by mining multi-scale clue features, thereby further improving the detection accuracy.

[0044] In the embodiment in which the Daliuta coal mine subsidence area is used as the study area, image data are collected and made into a data set, and the data set is divided into a training set, a test set and a validation set.

[0045] a. Image acquisition: The present invention uses the M350 RTK drone produced by DJI Innovations Technology Co., Ltd. as a data acquisition tool. The image shooting is selected in a clear or cloudy environment with sufficient light. The relative flight altitude is set to 50m, and the heading overlap and lateral overlap are both set to 85%. 5694 original drone images are collected in the subsidence area of ​​the Daliuta coal mine;

[0046] b, Data annotation and data set division; After obtaining the original drone images, the 3D modeling software Agisoft Metashape Professional was used to stitch the images and correct the camera distortion to generate orthophotos. The orthophotos were cropped to a suitable size and the Labelme dataset annotation tool was used to annotate the plant crown images; 2041 images were selected as the training set, 287 images as the validation set, and the remaining 3366 images as the test set.

[0047] In some embodiments, the CycleGAN network model includes two generators G, F and two discriminators D X , D Y , the CycleGAN network model performs bidirectional cycle processing on the real images x and y in the paired image group according to the following method:

[0048] like Figure 2 As shown in the left figure, the real image x is used as the input image, and the generator G generates a false image By the discriminator D X To judge whether the image is true or false, Input generator F and generate fake images The CycleGAN network model uses loss function supervision The degree of fidelity with x;

[0049] like Figure 2 As shown in the right figure of , the real image y is used as the input image, and the generator G generates a false image By the discriminator D X To judge whether the image is true or false, Input generator F and generate fake images The CycleGAN network model uses loss function supervision The degree of fidelity between y and y.

[0050] When the CycleGAN network model is trained, the generator migrates another dataset to a new dataset under the semantic guidance of one of the training sets. Specifically, for two real datasets x and y, with x as the input image, the generator G generates a fake image. By the discriminator D Y To judge whether the image is true or false, Input the generator and generate fake images Supervised by the loss function Fidelity with x, see Figure 2In this process, the generator aims to generate realistic images to confuse the discriminator, and the discriminator aims to distinguish the generated images from the real images. At the same time, the generator G migrates the input image from the distribution domain of the dataset x to the distribution domain of the dataset y, and the generator F migrates the input image from the distribution domain of the dataset y to the distribution domain of the dataset x. Similarly, see Figure 2 The right figure takes y as the input image, and the fidelity between y and the image is supervised by the loss function; Figure 2 The two processes in the left and right figures constitute a bidirectional loop. By inputting the generated image into the generator to form a bidirectional loop, the authenticity of the model-generated image is constrained, thereby generating an image that is closer to the real thing. In the embodiment of the Daliuta coal mine subsidence area as the study area, the image sample data set of the CycleGAN network model is divided and processed: after scaling the data sample to a size of 336*336, it is input into the CLIP model visual encoder to obtain feature encoding and category tokens, calculate the cosine similarity of the category tokens between images, and filter similar images according to the threshold to form image pairs, and the threshold is set to 0.75. The filtered image pairs are manually inspected to form a data set for training the CycleGAN model. The training set, validation set, and test set have 760, 113, and 1085 sets of image pairs, respectively. The CycleGAN network model is built using the Pytorch deep learning framework, using GPU for calculation, using the Adam optimizer for optimization, with the batch size set to 8, the learning rate set to 0.0001, and 120 epochs of optimization; the specific software and hardware parameters used are as follows:

[0051]

[0052]

[0053] During the training process, after each round of training, the CycleGAN network model is verified on the validation set, the mean of the loss function of all samples on the validation set is calculated, and the current test result is compared with the previous test result. If it is better than the previous performance, the model is saved. Use the trained CycleGAN network model to generate new image data; scale the images in the original training data set to 336*336 size, input the trained CycleGAN network model, generate new image data, and you can get 2041 generated samples. Manual screening excludes images with poor generation quality, and finally obtain 1839 generated images and add them to the original training set.

[0054] In some embodiments, the CycleGAN network model is built using the Pytorch deep learning framework. The CycleGAN network model uses a cycle consistency loss to optimize model parameters. The cycle consistency loss L cycThe expression of (G, F) is as follows:

[0055]

[0056] in represents expectation, F(G(x)) represents the output of the CycleGAN network model after processing the real image x as input, G(F(y)) represents the output of the CycleGAN network model after processing the real image y as input, and || || represents the L1 norm.

[0057] S2. Construct an improved U-Net vegetation crown extraction model (use the deep learning U-Net architecture to construct the U-Net vegetation crown extraction model, separate the plant crown from the background, and determine the crown boundary of the fir). The U-Net vegetation crown extraction model includes an encoder and a decoder. The encoder (also known as the contraction path) includes N1 contraction layers that are output sequentially, and each contraction layer performs convolution and downsampling operations. The decoder (also known as the expansion path) includes N1 expansion layers that are output sequentially, and each contraction layer performs convolution and upsampling operations. The N1 contraction layers correspond to the N1 expansion layers by layer and are horizontally connected between the corresponding layers. There is a multi-scale enhancement module (the multi-scale enhancement module is preferably a residual multi-scale feature extraction module RMFE; in the two-dimensional remote sensing images of the drone, the plant crowns often overlap with each other, making boundary extraction more difficult. The multi-scale enhancement module can search for subtle boundary changes from receptive fields of different scales, thereby improving the accuracy of crown extraction). The multi-scale enhancement module includes N2 sequential and parallel branches. The multi-scale enhancement module takes the features of the contraction layer as input and obtains the feature RMFE i , the feature RMFE i Add splicing to the input of the same-layer expansion layer; the last expansion layer of the U-Net vegetation crown width extraction model outputs a pixel-level crown width segmentation mask. The present invention improves the U-Net vegetation crown width extraction model by copying and cropping to fuse the upsampled feature map with the high-resolution feature map downsampled by the corresponding layer, so that the convolution layer can learn more accurate deep features and improve the segmentation accuracy.

[0058] In some embodiments, the encoder and decoder of the U-Net vegetation crown extraction model are constructed according to the U-Net convolutional neural network structure. Each contraction layer includes two convolutions and one downsampling operation. Each convolution process consists of a 3×3 convolution kernel and a rectified linear unit activation function (full name in English: rectified linear unit, abbreviated as: ReLU), and the downsampling operation adopts 2×2 maximum pooling; each expansion layer includes two convolutions and one upsampling operation. Each convolution process consists of a 3×3 convolution kernel and a rectified linear unit activation function. The encoder ensures the model accuracy and reasoning speed. Preferably, the model parameters of the encoder are initialized using a pre-trained model based on ImageNet to obtain a higher training starting point. The decoder is a combination of multiple groups of upsampling, convolution, and relu blocks. The encoder and decoder are jump-connected for feature transmission; the decoder finally generates a pixel-level vegetation crown extraction mask, and the mask value is predicted by the mean absolute error loss constraint to ensure that the model prediction results gradually approach the true value during the training process.

[0059] In some embodiments, the U-Net vegetation crown extraction model has an output layer after the last expansion layer, and the output layer performs 1×1 convolution (stride 1) and Softmax activation function processing. The Softmax activation function expression is as follows:

[0060] where a k (x) is the activation value of each pixel x corresponding to category k on the feature map, P k (x) is the probability that pixel x corresponds to category k, and K is the number of categories.

[0061] In some embodiments, Figure 3As shown in the figure, the encoder of the U-Net vegetation crown extraction model includes four contraction layers, and the decoder includes four expansion layers. The first contraction layer and the first expansion layer are horizontally connected with the first multi-scale enhancement module, the second contraction layer and the second expansion layer are horizontally connected with the second multi-scale enhancement module, the third contraction layer and the third expansion layer are horizontally connected with the third multi-scale enhancement module, and the fourth contraction layer and the fourth expansion layer are horizontally connected with the fourth multi-scale enhancement module; the first to fourth contraction layers perform different scale feature encoding processing in turn, and the fourth to first expansion layers perform different scale feature encoding processing in turn. The same-scale feature decoding process is performed. The feature output of the fourth contraction layer is concatenated with the feature output of the fourth multi-scale enhancement module as the feature input of the fourth expansion layer. The feature output of the fourth expansion layer is concatenated with the feature output of the third multi-scale enhancement module as the feature input of the third expansion layer. The feature output of the third expansion layer is concatenated with the feature output of the second multi-scale enhancement module as the feature input of the second expansion layer. The feature output of the second expansion layer is concatenated with the feature output of the first multi-scale enhancement module as the feature input of the first expansion layer. The first expansion layer outputs the crown segmentation mask. The multi-scale enhancement module of each layer can enhance the model's ability to represent multi-scale features and improve the accuracy of plant crown segmentation.

[0062] In some embodiments, Figure 4 As shown in the figure, the multi-scale enhancement module (by mining multi-scale clue features, enhancing the perception of subtle crown edges, thereby improving the accuracy of crown extraction) includes four sequential and parallel branches (using parallel 3×3 convolutions, using residual blocks to successively enlarge the receptive field, and using 4 branches to capture different features). Each branch configuration includes 1×1 convolution, 1×3 and 3×1 asymmetric convolutions, and the output of each branch is added to the input of the next branch. The expression is as follows:

[0063] where F i is the i-th feature map of the input, k is the number of branches, is the output of the k-th branch and the i-th feature map, For element-by-element addition, conv r () is the convolutional layer;

[0064] The four branch outputs are concatenated, followed by a 1×1 convolution, the channel is adjusted to 64, and added to the input feature F i Finally, the output feature RMFE is obtained i , i∈{1,2,3,4}

[0065] Where Conv() represents 1×1 convolution, It means that all four branches are concatenated, and the multi-scale features are added to the original features to enhance the multi-scale representation ability of the features.

[0066] During the training process of the U-Net vegetation crown extraction model, in order to increase data diversity, the input samples of the image sample data set are randomly cropped, randomly rotated and other steps for data enhancement; the specific methods include: random flipping to flip the image horizontally with a probability of 0.5; random cropping of no more than 15 pixel units in length and width; random sampling of integers from [H-15, H] as the length of the cropped image, and random sampling of integers from [W-15, W] as the width of the cropped image, H and W represent the length and width of the original image; cropping with the center of the original image as the center; random rotation with a probability of 0.3, and rotating the cropped image by randomly sampling integers from -10° to 10°.

[0067] The enhanced image is input into the constructed U-Net vegetation crown extraction model to obtain the vegetation crown prediction mask of plant crown extraction, calculate the loss through the loss function, and perform back propagation to update the parameters. After each round of training, the model effect is tested on the validation set, and the best performing model on the validation set is saved, that is, the model with the smallest mean absolute error between all predicted masks and the true value mask is retained.

[0068] S3, collect UAV images of the study area and input them into the U-Net vegetation crown extraction model to output the vegetation crown prediction mask after crown segmentation. Figure 6 As shown in the example, in the Daliuta coal mine subsidence area as the study area, multiple small areas of the study area are intercepted (see Figure 6 , 6 small areas are selected to respectively demonstrate the extraction and segmentation effects of the plant crown width). The present invention can accurately extract the plant crown width and has excellent extraction quality and efficiency.

[0069] In some embodiments, the overall loss function Loss of the U-Net vegetation crown extraction model is expressed as follows:

[0070] Loss = L BCE +L IOU , where L BCE represents the cross entropy loss, L IOU represents the intersection-and-combination loss;

[0071] L BCE =-∑(P(x,y)*ln(G(x,y))+(1-P(x,y))*ln(1-G(x,y)));

[0072]

[0073] P(x, y) and G(x, y) represent the crown segmentation mask and crown true value mask of the pixel at the original image coordinate position (x, y), respectively. x is an integer in [0, W-1], y is an integer in [0, H-1], and H and W represent the length and width of the image, respectively.

[0074] The improved U-Net vegetation crown extraction model of the present invention is optimized using the Adam optimizer, with the batch size set to 32, the learning rate set to 0.0001, and the optimization is performed for 120 epochs.

[0075]

[0076] During the training process of the improved U-Net vegetation crown width extraction model, after each round of training, the improved U-Net vegetation crown width extraction model is verified on the validation set, the mean of the loss function of all samples of the model on the validation set is calculated, and the current test result is compared with the previous test result. If it is better than the previous performance, the improved U-Net vegetation crown width extraction model is saved. After the improved U-Net vegetation crown width extraction model is trained, the model performance is tested on the test set. During the test, the sample is scaled to 336*336 size. The test includes objective evaluation and subjective evaluation; the objective evaluation is the indicator calculation: the average absolute error between the crown width extraction mask predicted by the model and the true value of the crown width is calculated. The lower the average absolute error, the better the extraction effect. The subjective evaluation is evaluated by visually observing the extraction results.

[0077] The present invention conducts experiments on UAV remote sensing data sets by improving the U-Net vegetation crown width extraction model. The accuracy (Precision) and recall rate (Recall) of vegetation crown width extraction of the improved U-Net vegetation crown width extraction model of the present invention are significantly improved, as shown in the following table:

[0078]

[0079] A remote sensing image vegetation crown width extraction system based on an improved U-Net model comprises a data acquisition module, an image sample data set, a CycleGAN network model and a U-Net vegetation crown width extraction model. The image sample data set is constructed by the data acquisition module collecting drone images and annotating vegetation crown width labels; the CycleGAN network model inputs the image sample data set into paired image groups and uses the image sample data set to perform model training. Each paired image group contains two visually similar images. The CycleGAN network model generates The high-fidelity image is formed and stored in the image sample data set; the U-Net vegetation crown extraction model includes an encoder and a decoder, the encoder includes N1 contraction layers output in sequence, each contraction layer performs convolution and downsampling operations, the decoder includes N1 expansion layers output in sequence, each contraction layer performs convolution and upsampling operations, the N1 contraction layers correspond to the N1 expansion layers in layers, and there is a multi-scale enhancement module horizontally connected between the corresponding layers, the multi-scale enhancement module includes N2 sequential and parallel branches, the multi-scale enhancement module uses the features of the contraction layer as input and obtains the feature RMFE i , the feature RMFE i Add splicing to the input of the expansion layer of the same layer; the last expansion layer of the U-Net vegetation crown extraction model outputs a pixel-level crown segmentation mask; the data acquisition module also collects drone images of the study area and inputs them into the trained U-Net vegetation crown extraction model, and then outputs the vegetation crown prediction mask after crown segmentation as the vegetation crown extraction result.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for extracting vegetation crown width from remote sensing images based on an improved U-Net model, characterized in that: The methods include: S1. Collect drone images and mark vegetation canopy labels to construct an image sample dataset; construct a CycleGAN network model, input the image sample dataset into the CycleGAN network model in pairs of image groups, and use the image sample dataset for model training. Each pair of image groups contains two visually similar images. The CycleGAN network model generates high-fidelity images and stores them in the image sample dataset; S2. Construct an improved U-Net vegetation crown extraction model. The U-Net vegetation crown extraction model includes an encoder and a decoder. The encoder includes N1 contraction layers that are output sequentially. Each contraction layer performs convolution and downsampling operations. The decoder includes N1 expansion layers that are output sequentially. Each contraction layer performs convolution and upsampling operations. The N1 contraction layers correspond to the N1 expansion layers by layer and are horizontally connected between the corresponding layers. There is a multi-scale enhancement module. The multi-scale enhancement module includes N2 sequential and parallel branches. The multi-scale enhancement module takes the features of the contraction layer as input and obtains the feature RMFE i , the feature RMFE i Add splicing to the input of the same-level expansion layer; the last expansion layer of the U-Net vegetation crown extraction model outputs a pixel-level crown segmentation mask; S3. Collect UAV images of the study area and input them into the U-Net vegetation crown extraction model to output the vegetation crown prediction mask after crown segmentation.

2. The method for extracting vegetation crown width from remote sensing images based on the improved U-Net model according to claim 1, characterized in that: The CycleGAN network model consists of two generators G, F and two discriminators D X , D Y , the CycleGAN network model performs bidirectional cycle processing on the real images x and y in the paired image group according to the following method: Take the real image x as the input image, and the generator G generates a fake image By the discriminator D X To judge whether the image is true or false, Input the generator and generate fake images The CycleGAN network model uses loss function supervision The degree of fidelity with x; Take the real image y as the input image, and the generator G generates a fake image By the discriminator D X To judge whether the image is true or false, Input the generator and generate fake images The CycleGAN network model uses loss function supervision The degree of fidelity between y and y.

3. The method for extracting vegetation crown width from remote sensing images based on the improved U-Net model according to claim 1, characterized in that: The encoder and decoder of the U-Net vegetation crown extraction model are constructed according to the U-Net convolutional neural network structure. Each contraction layer includes two convolutions and one downsampling operation. Each convolution process consists of a 3×3 convolution kernel and a rectified linear unit activation function. The downsampling operation adopts 2×2 maximum pooling; each expansion layer includes two convolutions and one upsampling operation. Each convolution process consists of a 3×3 convolution kernel and a rectified linear unit activation function.

4. The method for extracting vegetation crown width from remote sensing images based on the improved U-Net model according to claim 1, characterized in that: The U-Net vegetation crown extraction model has an output layer after the last expansion layer. The output layer performs 1×1 convolution and Softmax activation function processing. The expression of Softmax activation function is as follows: where a k (x) is the activation value of each pixel x corresponding to category k on the feature map, P k (x) is the probability that pixel x corresponds to category k, and K is the number of categories.

5. The method for extracting vegetation crown width from remote sensing images based on the improved U-Net model according to claim 1, characterized in that: The encoder of the U-Net vegetation crown extraction model includes four contraction layers, and the decoder includes four expansion layers. The first contraction layer and the first expansion layer are horizontally connected with the first multi-scale enhancement module, the second contraction layer and the second expansion layer are horizontally connected with the second multi-scale enhancement module, the third contraction layer and the third expansion layer are horizontally connected with the third multi-scale enhancement module, and the fourth contraction layer and the fourth expansion layer are horizontally connected with the fourth multi-scale enhancement module; the first to fourth contraction layers perform different scale feature encoding processing in turn, and the fourth to first expansion layers perform different scale feature encoding processing in turn. Scale feature decoding processing: the feature output of the fourth contraction layer is concatenated with the feature output of the fourth multi-scale enhancement module as the feature input of the fourth expansion layer; the feature output of the fourth expansion layer is concatenated with the feature output of the third multi-scale enhancement module as the feature input of the third expansion layer; the feature output of the third expansion layer is concatenated with the feature output of the second multi-scale enhancement module as the feature input of the second expansion layer; the feature output of the second expansion layer is concatenated with the feature output of the first multi-scale enhancement module as the feature input of the first expansion layer; the first expansion layer outputs the crown segmentation mask.

6. The method for extracting vegetation crown width from remote sensing images based on the improved U-Net model according to claim 5, characterized in that: The multi-scale enhancement module consists of four sequential and parallel branches. Each branch configuration includes 1×1 convolution, 1×3 and 3×1 asymmetric convolution. The output of each branch is added to the input of the next branch. The expression is as follows: where F i is the i-th feature map of the input, k is the number of branches, is the output of the k-th branch and the i-th feature map, For element-by-element addition, Conv r () is the convolutional layer; The four branch outputs are concatenated, followed by a 1×1 convolution, the channel is adjusted to 64, and added to the input feature F i Finally, the output feature RMFE is obtained i , i∈{1, 2, 3, 4} Where Conv() represents 1×1 convolution, Indicates that all four branches are spliced.

7. The method for extracting vegetation crown width from remote sensing images based on the improved U-Net model according to claim 1, characterized in that: The overall loss function Loss expression of the U-Net vegetation crown extraction model is as follows: Loss = L BCE +L IOU , where L BCE represents the cross entropy loss, L IOU represents the intersection-and-combination loss; L BCE =-∑(P(x,y)*ln(G(x,y))+(1-P(x,y))*ln(1-G(x,y))); P(x, y) and G(x, y) represent the crown segmentation mask and crown true value mask of the pixel at the original image coordinate position (x, y), respectively. x is an integer in [0, W-1], y is an integer in [0, H-1], and H and W represent the length and width of the image, respectively.

8. The method for extracting vegetation crown width from remote sensing images based on the improved U-Net model according to claim 1, characterized in that: The CycleGAN network model is built using the Pytorch deep learning framework. The CycleGAN network model uses cycle consistency loss to optimize model parameters. cyc The expression of (G, F) is as follows: in represents expectation, F(G(x)) represents the output of the CycleGAN network model after processing the real image x as input, G(F(y)) represents the output of the CycleGAN network model after processing the real image y as input, and || || represents the L1 norm.

9. A remote sensing image vegetation crown extraction system based on an improved U-Net model, characterized by: The invention comprises a data acquisition module, an image sample data set, a CycleGAN network model and a U-Net vegetation crown width extraction model. The image sample data set is constructed by the data acquisition module collecting drone images and marking vegetation crown width labels. The CycleGAN network model inputs the image sample data set into paired image groups into the CycleGAN network model and uses the image sample data set for model training. Each paired image group contains two visually similar images. The CycleGAN network model generates high-fidelity images and stores them in the image sample data set. The U-Net vegetation crown width extraction model comprises an encoder and a decoder. The encoder comprises N1 contraction layers output in sequence, each contraction layer performs convolution and downsampling operations, the decoder comprises N1 expansion layers output in sequence, each contraction layer performs convolution and upsampling operations, the N1 contraction layers correspond to the N1 expansion layers in layers respectively and there is a multi-scale enhancement module horizontally connected between the corresponding layers. The multi-scale enhancement module comprises N2 sequential and parallel branches. The multi-scale enhancement module uses the features of the contraction layer as input and obtains the feature RMFE i , the feature RMFE i Add splicing to the input of the expansion layer of the same layer; the last expansion layer of the U-Net vegetation crown width extraction model outputs a pixel-level crown width segmentation mask; the data acquisition module also collects drone images of the study area and inputs them into the trained U-Net vegetation crown width extraction model, and then outputs the vegetation crown width prediction mask after crown width segmentation as the vegetation crown width extraction result.

Images