Semantic segmentation method for economic forest canopies for ginkgo leaves in real scene

Abstract

A semantic segmentation method for a ginkgo leaf economic forest canopy in a real scene adopts a semantic segmentation model to process a leaf ginkgo leaf canopy orthoimage collected by a low-altitude unmanned aerial vehicle, and a leaf ginkgo leaf canopy is distinguished from a complex background. The semantic segmentation model is a HaarUNet model; haarUNet is an improvement on a UNet segmentation network, and the improvement method comprises the following steps: under a network framework of reserving a code and a decoder of the UNet segmentation network, replacing a maximum pooling down-sampling layer of a coding layer with a Haar wavelet down-sampling module, decomposing original image information into a low-frequency sub-band and a high-frequency sub-band, subsequent feature extraction and reconstruction are carried out by utilizing the characteristic that the high-frequency information can be restored in a lossless manner; a SENet attention mechanism module is embedded after dimension raising and dimension reducing operation of the feature channel; and a Multi grid cavity convolution module is embedded before down-sampling of each layer of encoder. A test result shows that the HaarUNet model has relatively high segmentation identification precision.

Description

Technical Field

[0001] The present invention is an application of artificial intelligence in forestry technology, specifically a semantic segmentation method for the canopy of an economic forest for ginkgo leaves in a real scenario. Background Art

[0002] In order to harvest high-quality ginkgo leaves, farmers usually remove the top buds of ginkgo trees (also known as "castration"). [1,2] This method can promote the growth of more side branches in ginkgo trees, thereby increasing the yield of leaves and facilitating picking. However, due to the growth of side branches and the influence of complex terrain such as weeds and soil, [3] , it becomes more difficult to monitor the health and growth of the ginkgo canopy. In complex natural environments, accurate segmentation of target areas has always been a difficult problem in the field of image processing. The purpose of this study is to solve this problem through the method of deep semantic segmentation. [4] .

[0003] In recent years, with the application of advanced technologies in modern agriculture and forestry [5] . Image segmentation technology based on machine learning is applied to various plant leaf and canopy segmentation tasks. Such as edge detection, threshold segmentation [6] , Region Growing Algorithm [7] and watershed algorithm, etc. Lang Chunbo et al. [8] A method combining simulated annealing algorithm and hybrid particle swarm algorithm to optimize multi-threshold image segmentation is proposed, which optimizes the problem that the standard particle swarm algorithm is prone to fall into local optimality. Experimental results show that this method effectively handles the problem of complex plant canopy image segmentation. Zou Qiuxia et al. [9] It is proposed to convert the image RGB space into Lab space and use the K-Means clustering algorithm to segment plant leaves. The experimental results show that this method can not only eliminate the shadow noise generated during the shooting process but also ensure that the segmented image is a color image. Zhang Miaomiao et al.

[10] A new leaf image segmentation method is proposed. The Canny operator is used to extract the leaf contour and add it to the original grayscale image, and then add it to the inverted binary image. Experiments show that this method can retain the leaf feature information and achieve complete separation of leaves from the background. In order to deal with the problems of uneven lighting and complex background in real scenes of greenhouse vegetable leaves, Ma et al.

[11] An interactive region growing algorithm based on comprehensive color feature images was proposed to achieve the lesion segmentation of cucumber downy mildew images under complex backgrounds. In order to solve the problem of smooth edges but ineffective leaf segmentation, Jin et al.

[12] A texture hierarchical segmentation method based on region growing is proposed. Experiments show that the segmentation completeness of this method in the sweet cherry leaf dataset with jagged edges reaches 86.58%, and the algorithm performs well.

[0004] However, traditional image segmentation algorithms usually perform segmentation based on pixel-level features, which has low accuracy, makes it difficult to process complex scenes and objects with blurred boundaries, and has high computational complexity and is sensitive to parameters. In contrast, semantic segmentation algorithms based on deep learning can more accurately understand the semantic information in images, thereby improving segmentation accuracy, especially when processing complex scenes and tasks that require high segmentation accuracy, because they have a high degree of automation and are more suitable for processing large-scale images. Semantic segmentation technology based on deep learning is not limited to the industrial and medical fields, such as real-time road monitoring, medical image analysis, etc. [13,14] Semantic segmentation models are also gradually being widely used in agriculture and forestry.

[0005] Semantic segmentation models such as FCN

[15] , UNet, DeepLab series [16,17] , HRNet, etc., can effectively capture image features at different scales through deep network structures, thereby achieving more accurate and detailed target segmentation in the field of agriculture and forestry. These models can accurately distinguish crops, trees, pests and diseases from complex backgrounds, providing strong technical support for agriculture and forestry management and decision-making. In order to more accurately analyze the phenotypic conditions of various plants, Shrikrishna Kolhar et al.

[18] An improved residual U-Net model based on convolutional neural network was designed, and experiments showed that the dice coefficient of the model was as high as 0.9709 in the leaf segmentation challenge (LSC) dataset. Zou et al.

[19] In order to solve the problem of accurate weed segmentation in the field, an image enhancement method is proposed and a two-stage training method consisting of pre-training and fine-tuning parameters based on an improved U-Net is adopted. The experimental results show that the MIOU for weed segmentation reaches 92.91% and weeds can be accurately segmented in complex field environments. Srinivas Talasila et al.

[20] In order to solve the problem of difficult segmentation of plant leaves under complex background, a semantic segmentation model of PLRSNet was proposed for extracting leaf areas in farmland images. Experimental results show that the evaluation index DICE of this method reaches 96.9% and MIOU reaches 94.2%. Li Zhengkai et al.

[21] In order to solve the problem that kiwifruit trees overlap and it is inconvenient to segment the canopy, a PSP-Net semantic segmentation model integrating skeleton information is proposed, and mathematical morphology is introduced to achieve canopy segmentation. Experimental results show that the accuracy of this method is improved by about 15.71% compared with traditional methods.

[22] In order to solve the problem of complex terrain affecting winter wheat recognition, an improved semantic segmentation model CAHRNet (changeattention high-resolution Net) based on HRNet is proposed. The experimental results show that the MIOU and PA indicators of this method are as high as 81.72% and 97.08% respectively on the self-made dataset.

[0006] However, the traditional model still has the problem of partial loss of effective information during downsampling or upsampling, which affects the accuracy of feature extraction from the original image information; at the same time, during the network encoding and decoding process, the change in the number of feature channels will also lead to insufficient correlation representation between feature dimensions, making it impossible for the model to effectively use the information between these channels to improve segmentation accuracy. In addition, the series superposition of ordinary convolutional layers will cause the network complexity to be too high, and the repeated extraction of the same features of the image will also lead to a waste of computing resources. Summary of the invention

[0007] In order to solve many problems existing in the prior art, the present invention proposes a semantic segmentation method for the canopy of economic forests for ginkgo leaves in real scenes. This method designs a semantic segmentation model of Haar_UNet (Haar wavelet enhanced semantic attention UNet) to process the collected ginkgo canopy image. The ginkgo canopy image can be an orthophoto of the leaf ginkgo canopy collected by low-altitude remote sensing of a small unmanned aerial vehicle.

[0008] The present invention abandons the two downsampling layers of the maximum pooling layer and the average pooling layer of the traditional semantic segmentation model in the backbone network of Haar_UNet, and replaces them with a HWD downsampling module based on Haar wavelet transform.

[0009] In the maximum pooling layer, only the maximum value of the local area is retained, while in the average pooling, the average value is taken, which may lead to the loss of some information.

[0010] Since wavelet transform is based on the analysis of local features of the signal, the subtle features of the image can be more completely preserved during the downsampling process;

[0011] The encoder part serves as the main feature extraction layer. Embedding a Multi grid dilated convolution module before downsampling each layer enhances the nonlinear ability of the network, enabling the network to better adapt to features of different scales and complexities.

[0012] Moreover, compared with ordinary convolutional layers, the atrous convolutional layer can expand the receptive field, avoid repeated feature extraction, and improve feature extraction efficiency.

[0013] In addition, the present invention embeds the SENet attention mechanism module after the feature channel dimensionality increase and decrease operations to enhance the correlation information between different channels, so that the model can effectively utilize the information between these channels to improve the segmentation accuracy.

[0014] Taking the high-resolution leaf ginkgo canopy photographed by a low-altitude drone as the basic data source, the feature expression ability of the Haar_UNet model of the present invention was verified by comparing with traditional semantic segmentation models such as U-Net, DeepLabv3, and HRNet. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Represents the geographical location of the study area, where: the dashed lines of different colors in the figure represent the trajectory and direction of the UAV.

[0016] Figure 2 It is the network structure of the Haar_UNet model.

[0017] Figure 3 Represents the HWD module structure; the structure mainly consists of two modules: lossless feature encoding block and feature learning module, where the number of channels C of the feature map can be adjusted by the feature learning module. In the figure, H0 represents a low-pass decomposition filter; H1 represents a high-pass decomposition filter; ↓2 represents downsampling applied to the approximate component (low frequency) and the detail component (high frequency).

[0018] Figure 4 Represents the SENet network structure, where: Feature maps of various colors represent different attention weights assigned to different channels after the "compression" and "excitation" process of SENet. The difference in color indicates the different importance of each channel. The darker the color, the greater the importance or activation weight of the corresponding feature channel.

[0019] Figure 5 Represents the multi-scale dilated convolution module structure;

[0020] Figure 6 A comparison chart of downsampling layer visualization results, where the light-colored areas in the image represent the effects of feature areas extracted by various downsampling layers.

[0021] Figure 7 Represents the visualization results of the ablation experiment, where the solid red line represents the difference with the ground truth, and the dotted white line represents the enlarged view within the red box.

[0022] Figure 8a and Figure 8b They are line graphs of the MIOU value and loss of each model after 100 iterations.

[0023] Fig. 9Represents the visualization results of model comparison, where: the dashed boxes of different colors represent the differences between the visualization results of different models on the same image and the actual situation. DETAILED DESCRIPTION

[0024] After removing the terminal buds, it is easier for leaf-bearing ginkgo trees to extract lateral branches, thereby growing more leaves and increasing leaf yield. However, affected by the growth of lateral branches and complex objects such as weeds and soil, it has become increasingly difficult to monitor the health and growth potential of the canopy of a single ginkgo tree. In order to promote the needs of precision forestry, the present invention uses the orthophoto of the canopy of leaf-bearing ginkgo taken by a low-altitude drone as the data source, and proposes a semantic segmentation model of Haar_UNet (Haar wavelet enhanced semantic attention UNet). The SENet (Squeeze-and-Excitation Network) attention mechanism is embedded in the network to dynamically adjust the weights of the feature map by learning the relationship between feature channels, thereby enhancing the network's perception of important features; the Haar wavelet downsampling module is introduced to replace the maximum pooling downsampling layer, which effectively reduces the resolution of the feature map while retaining the feature information; the encoder part combines the Multi Grid technology to better capture the multi-scale information in the image by superimposing a multi-scale convolution module with multiple hole convolution layers, and distinguish the edge pixels of the leaf canopy interest area from the background.

[0025] The experimental results show that on the self-made high-resolution ginkgo canopy dataset, the mean intersection over union (MIOU) and pixel accuracy (PA) of the Haar_UNet model reached 93.5% and 97.9%, respectively. MIOU was 5%, 3.7%, 8.1%, 6.6%, 7.2%, 3.4% and 3% higher than that of FCN, Lraspp, DeepLabV3_mobilenet, DeepLabV3_ResNet50, DeepLabV3_ResNet101, HRNet and UNet, respectively. This shows that the Haar_UNet model has a high segmentation and recognition accuracy, which can provide technical support for further accurate acquisition of the growth phenotype information of the leaf ginkgo canopy.

[0026] The present invention is further described below in conjunction with specific embodiments.

[0027] 1 Study area and data sources

[0028] 1.1 Overview of the study area and collection equipment

[0029] The study area of ​​this case is located in the leaf-bearing ginkgo base of Changrong Agricultural Technology Co., Ltd. in Xinglou Town, Pizhou City, Xuzhou City, Jiangsu Province. The dashed lines of different colors in the figure indicate the trajectory of the drone's automatic timed shooting of some areas, so that the drone can always effectively collect ginkgo canopy images. The image data acquisition device used in this case is DJI's small, low-cost drone DJIMini2SE, which has a high spatial resolution RGB sensor, a total weight of only 249g, and a theoretical battery life of 31 minutes. Mini2SE is small in size, with the longest part of only 15.3cm in the folded state, making it very convenient to carry. At the same time, the price of the drone is only 2999RMB, which can greatly reduce the cost of collecting agricultural and forestry images. In addition, during use, the airflow caused by small drones is much smaller than that of medium and large drones, so it can complete the image collection at a lower flight altitude without basically affecting the original distribution of the crop canopy leaves, and can better match the collection requirements of high-precision crop images at multiple scales. In this case, the data collection is completed at a flight altitude of 3 meters.

[0030] 1.2 Data preprocessing and dataset construction

[0031] The growth cycle of leaf-bearing ginkgo trees is closely related to seasonal changes. In early spring, ginkgo trees enter the budding period from March to April, and the leaf-expanding period from April to May. At this time, the leaves rapidly grow larger and thicker and continue to grow until July, and enter the picking period in August and September. In order to monitor the growth status of leaf-bearing ginkgo trees and obtain the phenotype of plant canopy leaves, this example uses drones to collect high-resolution images of the ginkgo canopy at low altitude in May and July. The image contains complex objects such as soil and weeds. These noises cause great interference to the monitoring of the leaf area. Therefore, it is particularly important to remove these complex backgrounds and retain the leaf area of ​​interest. First, it is necessary to preprocess these high-resolution full-color images and segment them into images of 480 pixels × 480 pixels. After segmentation, 774 image sample data sets are obtained. Then, the ginkgo canopy image data set is annotated one by one using Labelme annotation software to generate a sample label data set corresponding to the original image in order, quantity and size. In order to enhance the robustness of the model, data augmentation was performed on the image and its label datasets, including random scaling, horizontal and vertical flipping, and random shuffling. Finally, 2,322 sets of data samples were obtained, which were divided into training sets, validation sets, and test sets in a ratio of 8:1:1, which were used for model training, parameter adjustment, and performance evaluation, respectively.

[0032] 2 Model network architecture design

[0033] 2.1 Improve UNet network structure

[0034] The Haar_UNet semantic segmentation model proposed in this invention redesigns the UNet segmentation network on the network framework of the "U"-shaped structure encoder and decoder. Figure 2 As shown. In order to enable the model to better extract the boundary features of the ginkgo canopy, the present invention respectively adds a SENet channel attention mechanism for enhancing the expression of deep channel features and a multi-scale dilated convolution module for capturing multi-scale region of interest ROI (ginkgo canopy) features. At the same time, the present invention replaces the maximum pooling downsampling layer of the coding layer with a Haar wavelet downsampling module, decomposes the original image information into low-frequency sub-bands and high-frequency sub-bands, and uses the characteristic of lossless restoration of high-frequency information for subsequent feature extraction and reconstruction. This ensures that important edge and detail information will not be lost when performing multi-scale feature extraction, thereby enhancing the extraction effect of detailed features inside the canopy.

[0035] 2.2Haar Wavelet Downsampling, HWD Module

[0036] like Figure 3 As shown, HWD module

[23] It includes two sub-modules: lossless feature encoding block and feature learning module.

[0037] in:

[0038] The lossless feature encoding block is responsible for converting features and reducing spatial resolution. To achieve this process, this study uses the Haar wavelet transform, which can reduce the resolution of feature maps while retaining all feature information. Compared with the downsampling layer (max-pooling or average pooling) of traditional semantic segmentation, the downsampling module based on the Haar wavelet transform does not simply select the local maximum or average value, but is a multi-scale signal analysis method that can decompose and reconstruct images at different scales, thereby better capturing the subtle features in the image.

[0039] In addition, in order to enable flexible adjustment of the number of channels, a feature learning module is introduced.

[0040] 2.2.1 Lossless Feature Coding Block

[0041] The lossless feature coding block uses a combination of Haar wavelet function and scaling function to decompose and reconstruct the input image feature signal, effectively reducing the spatial resolution of the feature map while retaining all information. Haar wavelet transform is a widely recognized, compact standard orthogonal transform that is widely used in image coding and edge extraction. Through wavelet transform, a low-frequency component (approximate) and high-frequency detail components in three directions (vertical, horizontal, and diagonal) corresponding to the characteristic pixels of the ginkgo canopy are established. These four components are then concat spliced ​​as parameters input to the feature learning module. The definition formulas of the first-level one-dimensional scaling function and the wavelet function of the Haar transform are as follows:

[0042]

[0043] Where: φ j,k (x) can be defined as:

[0044]

[0045] Where: parameter j represents the Haar basis function series (in this case, 1 means the scale of image processing is reduced by 1 / 2); in wavelet transform, k usually takes an integer value to determine the position of the wavelet function on the signal or image. is the Haar wavelet basis function, usually expressed as the difference of a unit step function. Here, Represents the original wavelet function To perform horizontal translation and scaling. Specifically, 2 j x scales the wavelet function horizontally, while -k shifts it to the right by k units. is a normalization coefficient, which ensures that the energy of the wavelet function remains consistent at different scales. In addition, is defined as:

[0046]

[0047] Therefore, the first-level one-dimensional using the 0-level Haar basis function can also be expressed as:

[0048]

[0049] This means that a signal of length L can be divided into two parts of length L / 2, which can be interpreted as low-pass and high-pass decomposition filters, respectively. When the Haar wavelet transform is applied to a two-dimensional signal, such as a grayscale image, four components are produced, each with half the spatial resolution of the original image.

[0050] 2.2.2 Feature Learning Module

[0051] This submodule consists of a 1×1 standard convolution layer, a batch normalization layer, and a ReLU activation function layer.

[0052] In the present invention, the Max-pooling downsampling layer is abandoned, and the HWD module is used to more completely retain the feature information of the image while reducing the resolution.

[0053] 2.3 SENet Attention Mechanism Module

[0054] Since the model undergoes four times of Haar wavelet downsampling and bilinear interpolation upsampling, the continuous change in the number of feature channels will cause the model to be unable to effectively distinguish features at different scales. This may cause the model to be unable to accurately capture important details and structural information of the image, thereby affecting the performance and accuracy of the model. Therefore, the present invention embeds the SENet attention mechanism before and after the channel change to enhance the inter-channel representation ability.

[24] , whose structure is as follows Figure 4 shown.

[0055] 2.4 Multi-scale dilated convolution module

[0056] The dilated convolution was originally developed to efficiently calculate the wavelet transform. Mathematically, the dilated convolution under a two-dimensional signal is expressed as follows:

[0057] y[i]=∑ k x[i+rk]ω[k] (5)

[0058] Where i represents the position of the current pixel, x represents the input image feature map, ω represents the convolution kernel weight, y refers to the dilated convolution output, r corresponds to the input signal sampling stride, and k represents the position coefficient of the convolution kernel. Standard convolution is a special case of r = 1, and dilated convolution is to adaptively modify the receptive field of the convolution layer by defining different values ​​of r.

[0059] The multi-scale dilated convolution module is an effective convolutional neural network module, which is used to extract multi-scale features in images in segmentation and classification tasks. [25,26] . The purpose of using dilated convolution after the Haar wavelet downsampling layer in the encoder of the present invention is to use a larger receptive field to obtain more effective information. The module is composed of three superimposed dilated convolutions, in which the dilation coefficient of each convolution layer is set to 1, 2 and 4 respectively, to capture feature information at different scales. Continuously superimposed standard convolution layers will extract feature information of the same pixel multiple times, which will result in the failure to extract effective information from the edge area of ​​the ginkgo canopy image with a smaller resolution and a larger area of ​​interest. This multi-scale dilated convolution operation helps to increase the receptive field of the network, thereby better understanding the global structure and local details in the ginkgo canopy image.

[0060] In the multi-scale dilated convolution module, the present invention first applies a convolution layer with a dilation coefficient of 1 to extract the original features of the image. Then, by adding convolution layers with dilation coefficients of 2 and 4, the receptive field can be effectively expanded to capture semantic information in a wider range. Finally, by adding a batch normalization (BN) layer to normalize the features, it helps to accelerate model convergence and improve the generalization ability of the model. The specific structure is as follows Figure 5 shown.

[0061] 2.5 Loss Function

[0062] In order to achieve the best training effect of the model, the loss function is used. Since the binary cross entropy function will cause a large impact on the loss value of pixels with unclear classification, the present invention adopts the Dice (Dice coefficient) loss function

[27] . In the ginkgo canopy segmentation, there is a class imbalance between the foreground (ginkgo canopy) and the background (surrounding environment). Dice Loss is a loss function suitable for processing class imbalance data because it focuses on the overlap between categories and helps to better handle the imbalance problem between the foreground and background. In addition, the calculation method of Dice Loss can intuitively evaluate the similarity between the segmentation results generated by the model and the true segmentation results, so it is more suitable for the evaluation and comparison of ginkgo canopy segmentation results.

[0063]

[0064] Where: y i and They represent the label value and predicted value of pixel i respectively. N is the total number of pixels, which is equal to the number of pixels in a single image multiplied by the batch size.

[0065] 3 Model testing and training

[0066] 3.1 Experimental environment and configuration

[0067] The experiment used a desktop computer with a 3.00GHz Intel(R)Core(TM)i9-13900KF CPU, 64GB RAM, 24GB video memory, and NVIDIA GeForce RTX 4090 GPU. The Python 3.9 version language and the deep learning framework based on Pytorch 1.11.1 were used to complete the network model construction, training, and parameter adjustment. The Adam algorithm was selected as the model training optimizer, with an initial learning rate of 0.001, a weight decay of 0.0001, a momentum of 0.9, a batch size of 8, and 100 iterations.

[0068] 3.2 Model Evaluation Indicators

[0069] The present invention adopts mean intersection over union (MIOU) and pixel accuracy (PA) as evaluation indicators to evaluate the accuracy of calculation results of different models.

[0070] The mean intersection-over-union ratio (MIOU) represents the ratio of the intersection and union between the true value and the predicted value of the image. The value range is between 0 and 1, and the calculation is shown in the formula.

[0071]

[0072] Pixel accuracy (PA) is the ratio of correctly marked pixels to the total number of pixels, and its calculation formula is as follows:

[0073]

[0074] Where k+1 is the number of categories, including the background category; is the number of pixels of class i that are correctly predicted as class i; is the number of pixels of class i that are predicted as class j; are the positive examples with incorrect predictions and the negative examples with incorrect predictions.

[0075] 4 Results Analysis

[0076] 4.1 Analysis of downsampling results based on Haar wavelet transform

[0077] Figure 6 It represents an application example of Haar wavelet transform downsampling layer, maximum pooling downsampling layer, strided convolution downsampling layer and average pooling downsampling layer in Haar_UNet model. Compared with the traditional downsampling method, the present invention is more accurate in extracting features such as the boundary, texture and detail information of the ginkgo canopy after using the Haar wavelet transform downsampling layer. In addition, the method is more complete in extracting features of foreground information (leaf canopy) and has less background noise. By designing the downsampling module of Haar wavelet transform, the integrity of the canopy is more retained, which is more suitable for accurately monitoring the growth status of plants and accurately extracting the phenotype of the region of interest than other traditional downsampling methods.

[0078] 4.2 Analysis of ablation experiment results

[0079] In order to clarify the effectiveness of the algorithm of the present invention in improving accuracy, an ablation test was designed to compare the UNet model, the SE_UNet model with the SENet module, and the SE_Multi_UNet model with both SENet and Multi-Grid modules with the Haar_UNet model of the present invention in the same environment. The test results on the test set are shown in Table 1.

[0080] Table 1 Ablation experiment results

[0081] Tab.1 Ablation Study Results

[0082]

[0083] From the perspective of model segmentation accuracy, the average intersection-over-combination and pixel accuracy of the UNet model reached 90.5% and 97.4%. After integrating the SENet attention mechanism module, the backbone network UNet can compress and excite the features of each channel, and can weight each channel according to the importance of the features, thereby enhancing the network's ability to represent important features, so that the average intersection-over-combination and pixel accuracy are increased by 1.5% and 0.2% respectively; on the basis of adding the SENet module, this study also added a multi-scale convolution module to better capture the multi-scale information in the image. The results show that the average intersection-over-combination and pixel accuracy of the SE_Multi_UNet model are 2.1% and 0.4% higher than those of UNet, respectively. Finally, a Haar wavelet transform downsampling module is added to the network framework of the SE_Multi_UNet model. This module can more accurately extract the boundary, texture and detail information of the ginkgo canopy, so that the average intersection-over-combination and pixel accuracy of the model are further improved to 93.5% and 97.9%. The results show that the improved network model Haar_UNet based on UNet achieves the optimal foreground (canopy area) segmentation IoU, average IoU and loss in the task of ginkgo canopy segmentation.

[0084] In order to more intuitively compare the visualization results of the ablation test, this experiment post-processed the output Mask format images obtained by different algorithms, and the results are shown in the figure above. Compared with the ground truth, the leaf disease phenotypes of Image 1 and Image 2 of UNet, SE_UNet and SE_Multi_UNet are lost; the edge details of Image 3 and Image 5 are lost. The red solid box indicates the difference between the output images of each algorithm and the true value, and the yellow virtual box indicates the enlarged image of the red solid box. The experimental results show that the prediction results of Haar_UNet are closer to the ground truth, accurately highlighting the detailed features of the canopy of leaf-using Ginkgo biloba, avoiding the loss of detailed feature information, and improving the quality of boundary segmentation.

[0085] In order to visually compare the performance differences of each model during the training process, the present invention creates a line graph of MIOU value and loss for each model after 100 iterations, as shown in Figure 8a and Figure 8b shown.

[0086] From Figure 8, it can be concluded that Haar_UNet has the best MIOU accuracy and loss value. In terms of segmentation accuracy, Haar_UNet converges faster than other traditional models and begins to converge after 10 iterations. In addition, the Haar_UNet model shows the smallest fluctuation in MIOU. In terms of loss size, Haar_UNet has the lowest and most stable loss. Therefore, overall, Haar_UNet performs best in the segmentation performance of ginkgo tree crowns.

[0087] 4.3 Comparative Analysis of Semantic Segmentation Models

[0088] Table 2 Comparison model results

[0089]

[0090] In order to more intuitively demonstrate the extraction ability of the Haar_UNet model for the leaf ginkgo canopy, typical sample pictures highlighting the characteristics of the ginkgo canopy under complex objects in the test set were selected, and different models were used to predict the leaf ginkgo canopy, and the segmentation performance of the Haar_UNet model for the ginkgo canopy was further evaluated. The traditional semantic segmentation model FCN model, DeepLabv3 model with different backbone network frameworks, Lraspp model and HRNet model were used for comparative experiments, and the results are shown in Table 2. From the perspective of the segmentation accuracy and efficiency of the model, compared with the traditional semantic segmentation network model, the total prediction time of the Haar_UNet model proposed in the present invention for 232 test pictures is 8.34s, 13.75s, 0.74s and 3.27s more than the four models of FCN, Lraspp, DeepLabV3_mobilenet and HRNet respectively; and 3.15s and 11.53s less than the prediction time of DeepLabV3_ResNet50 and DeepLabV3_ResNet101 models respectively. However, it has obvious advantages in segmentation accuracy. In the process of downsampling in traditional semantic segmentation models, the amount of calculation is reduced by reducing the spatial dimension of the feature map, but it will lead to the loss of image details and boundary information, thereby reducing the accuracy of segmentation. The Haar_UNet model introduces the Haar wavelet transform downsampling module to replace the maximum pooling downsampling layer, and retains the detailed features of the image through high-frequency information reconstruction. At the same time, in order to enhance the correlation between the channels of the convolution layer and better capture the multi-scale characteristics of the ginkgo canopy, the SENet attention mechanism and the multi-scale dilated convolution module are introduced, so that the average intersection-over-union ratio of the Haar_Unet model is 5%, 3.7%, 8.1%, 6.6%, 7.2% and 3.4% higher than that of the six models of FCN, Lraspp, DeepLabV3_mobilenet, DeepLabV3_ResNet50, DeepLabV3_ResNet101 and HRNet respectively; the pixel accuracy is increased by 1.5%, 1%, 2.9%, 2.1%, 2.7% and 1% respectively. In addition, the loss of the Haar_UNet model is the smallest, which makes the model prediction value closer to the true value. Therefore, the Haar_UNet model has a better effect on the feature extraction of the leaf-using Ginkgo canopy.

[0091] The post-processing effects of different algorithms are shown in the figure below. Fig. 9As shown, virtual boxes of different colors represent the feature extraction effects of different models for the same image. Among them, the three models of FCN, DeepLabV3_ResNet50 and DeepLabV3_ResNet101 have poor boundary segmentation effects for 5 images. This is mainly because the FCN model relies on local convolution operations and lacks the capture of global context information, resulting in insufficient recognition of large-scale features. Although the deeper network of the DeepLabV3 series model has stronger feature extraction capabilities, the nonlinear expression ability between channels is poor. HRNet, Lraspp and DeepLabV3_mobilenet have good segmentation effects for the overall information of the canopy, but poor extraction effects for detail information, resulting in partial loss of disease features in the ginkgo canopy. This is because traditional convolution and downsampling operations can cause the loss of high-resolution detail information, affecting the accuracy of detail segmentation. The Haar_UNet model proposed in the present invention is the best for the segmentation effect of the leaf ginkgo canopy area in terms of both boundaries and internal details. While fully considering the correlation between pixels and category areas, the integrity of the ginkgo canopy feature information and the accuracy of edge detail information are retained.

[0092] 5 Conclusion

[0093] This paper takes the high-resolution image of the canopy of leaf-bearing ginkgo in real scenes as the research object, and proposes a multi-scale feature adaptive semantic segmentation model Haar_UNet based on the UNet model. After a large number of experiments, the following conclusions are obtained:

[0094] (1) The Haar_UNet model proposed in this paper abandons the traditional downsampling method. The embedding of the downsampling module based on Haar wavelet transform solves the problem of inaccurate segmentation of ginkgo canopy detail features. Compared with the feature extraction effect of the traditional downsampling method, the HWD module better filters the image noise and more completely retains the canopy feature information.

[0095] (2) In order to enhance the ability of inter-channel feature expression and accurately capture the multi-scale features of the leaf-bearing ginkgo canopy, the Haar_UNet model introduced the SENet channel attention mechanism and the multi-scale dilated convolution module. The ablation test results show that the average intersection over union (MIOU) and pixel accuracy (PA) of the model increased by 1.5% and 0.2% respectively after the SENet attention mechanism was introduced; the average intersection over union (MIOU) and pixel accuracy (PA) increased by 2.1% and 0.4% respectively after adding the two modules at the same time.

[0096] (3) The segmentation effect of the Haar_UNet model proposed in the present invention is better than that of other segmentation models, with MIOU and PA of 93.5% and 97.9% respectively, which are 5%, 3.7%, 8.1%, 6.6%, 7.2% and 3.4% higher than those of FCN, Lraspp, DeepLabV3_mobilenet, DeepLabV3_ResNet50, DeepLabV3_ResNet101 and HRNet respectively; the pixel accuracy is improved by 1.5%, 1%, 2.9%, 2.1%, 2.7% and 1% respectively.

[0097] In summary, for the semantic segmentation task of leaf-bearing ginkgo canopy images in near-ground remote sensing and natural complex scenes, the Haar_UNet model proposed in this paper can overcome the interference of complex ground objects and multiple lighting conditions, and quickly and accurately segment the ginkgo canopy. Through model migration, the canopy segmentation of other types of agricultural and forestry crops in real scenes can also be achieved, providing theoretical and technical support for plant phenotypic information detection.

[0098] References

[0099] [1] HAN Qicheng. Ginkgo biloba Seedling Cultivation and Afforestation Techniques in the Bailongjiang Forest Region[J]. New Agriculture, 2023, (08): 40-1. (in Chinese)

[0100] [2] ZHU H. Ginkgo Planting and Maintenance Management Techniques in Beijing[J]. South China Agriculture, 2024, 18(04): 253-5. (in Chinese)

[0101] [3]WEN Q,JIANG K,WANG W,et al.Automatic building extraction fromGoogle Earth images under complex backgrounds based on deep instancesegmentation network[J].Sensors,2019,19(2):333.

[0102] [4]HAO S,ZHOU Y,GUO YA brief survey on semantic segmentation with deep learning[J]. Neurocomputing, 2020,406:302-21.

[0103] [5] Cao Lin, Zhou Kai, Shen Xin, et al. Current status and prospects of smart forestry development[J]. Journal of Nanjing Forestry University (Natural Science Edition), 2022, 46(06): 83-95.

[0104] CAO L,ZHOU K,SHEN X,et al.The status and prospects of smart forestry[J].Journal of Nanjing Forestry University(Natural Sciences Edition),2022,46(06):83-95.(in Chinese)

[0105] [6]CUEVAS E,SOSSA HA comparison of nature inspired algorithms for multi-threshold image segmentation[J].Expert Systems with Applications, 2013,40(4):1213-9.

[0106] [7]REDDY AS,REDDY P C.Novel algorithm based on region growing method for better image segmentation; proceedings of the 2018 3rd International Conference on Communication and Electronics Systems(ICCES),F,2018[C].IEEE.

[0107] [8] Lang Chunbo, Jia Heming, Xing Zhikai, et al. Plant canopy image segmentation based on improved particle swarm algorithm[J]. Forest Engineering, 2019, 35(01): 47-52.

[0108] LANG Chunbo,JIA Heming,XING Zhikai,et al.Plant Canopy ImageSegmentation Based on an Improved Particle Swarm Optimization Algorithm[J].FOREST ENGINEERING,2019,35(01):47-52.(in Chinese)

[0109] [9] Zou Qiuxia, Yang Linnan, Peng Lin, et al. Research on leaf segmentation algorithm based on Lab space and K-Means clustering[J]. Agricultural Mechanization Research, 2015, 37(09): 222-6.

[0110] ZOU Qiuxia,YANG Linnan,PENG Lin,et al.Research on Leaf SegmentationAlgorithm Based on Lab Color Space and K-Means Clustering[J].Research onAgricultural Mechanization,2015,37(09):222-6.(in Chinese)

[0111]

[10] Zhang Miaomiao, Lü Jialuo, Ni Haiming, et al. Research on leaf image segmentation based on a new summation algorithm[J]. Forest Engineering, 2019, 35(04): 65-9+75.

[0112] ZHANG Miaomiao,LV Jialuo,NI Haiming,et al.Blade Image SegmentationBased on New Addition Algorithm[J].FOREST ENGINEERING,2019,35(04):65-9+75.(inChinese)

[0113]

[11] MA J,DU K,ZHANG L,et al.A segmentation method for greenhousevegetable foliar disease spots images using color information and regiongrowing[J].Computers and Electronics in Agriculture,2017,142:110-7.

[0114]

[12] JIN Z,HAN F.An Automatic Hierarchical Leaf Venation SegmentationBased on Region Growing;proceedings of the 2020IEEE 5th InternationalConference on Signal and Image Processing(ICSIP),F,2020[C].IEEE.

[0115]

[13] SUETENS P,BELLON E,VANDERMEULEN D,et al.Image segmentation:methods and applications in diagnostic radiology and nuclear medicine[J].European journal of radiology,1993,17(1):14-21.

[0116]

[14] ZHU Z,XU G,YANG B,et al.VISATRAM:A real-time vision system forautomatic traffic monitoring[J].Image and Vision Computing,2000,18(10):781-94.

[0117]

[15] ZHANG Y,CHI M.Mask-R-FCN:A deep fusion network for semanticsegmentation[J].IEEE Access,2020,8:155753-65.

[0118]

[16] LIN Y,XU D,WANG N,et al.Road extraction from very-high-resolutionremote sensing images via a nested SE-Deeplab model[J].Remote sensing,2020,12(18):2985.

[0119]

[17] WANG Y,GAO L,HONG D,et al.Mask DeepLab:End-to-end imagesegmentation for change detection in high-resolution remote sensing images[J].International Journal of Applied Earth Observation and Geoinformation,2021,104:102582.

[0120]

[18] KOLHAR S,JAGTAP J.Convolutional neural network based encoder-decoder architectures for semantic segmentation of plants[J].EcologicalInformatics,2021,64:101373.

[0121]

[19] ZOU K,CHEN X,WANG Y,et al.A modified U-Net with a specific dataargumentation method for semantic segmentation of weed images in the field[J].Computers and Electronics in Agriculture,2021,187:106242.

[0122]

[20] TALASILA S,RAWAL K,SETHI G.PLRSNet: a semantic segmentation network for segmenting plant leaf region under complex background[J].International Journal of Intelligent Unmanned Systems,2023,11(1):132-50.

[0123]

[21] Li Zhengkai, Yu Jiahui, Pan Shijia, et al. Single tree skeleton extraction and canopy growth prediction method for kiwifruit trees in winter[J]. Smart Agriculture (Chinese and English), 2023, 5(04): 92-104.

[0124] LI Zhengkai,YU Jiahui,PAN Shijia,et al.Individual tree skeleton extraction and crown prediction meth od of winter kiwifruit trees[J].SmartAgriculture,2023,5(4):92-104.(in Chinese)

[0125]

[22] Li Xuqing, Wu Dongxue, Wang Yubo, et al. Semantic segmentation method of winter wheat from remote sensing images based on improved HRNet[J]. Transactions of the Chinese Society of Agricultural Engineering, 2024, 40(03): 193-200.

[0126] LI Xuqing,WU Dongxue,WANG Yubo,et al.Impoved HRNet semanticsegmentation model for remote sensing images of winter wheat[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2024,40(03):193-200.(in Chinese)

[0127]

[23] XU G,LIAO W,ZHANG X,et al.Haar wavelet downsampling:A simple buteffective downsampling module for semantic segmentation[J].PatternRecognition,2023,143:109819.

[0128]

[24] JI Z,TAO W,REN J.Boiler furnace temperature and oxygen contentprediction based on hybrid CNN,biLSTM,and SE-Net models[J].AppliedIntelligence,2024:1-21.

[0129]

[25] LIU Y,ZHU M,WANG J,et al.Multi-scaledeep neural network based ondilated convolution for spacecraft image segmentation[J].Sensors,2022,22(11):4222.

[0130]

[26] XIA H,SUN W,SONG S,et al.Md-net:multi-scaledilated convolutionnetwork for CT images segmentation[J].Neural Processing Letters,2020,51:2915-27.

[0131]

[27] ZHAO R,QIAN B,ZHANG X,et al.Rethinking dice loss for medicalimage segmentation;proceedings of the 2020 IEEE International Conference onData Mining(ICDM),F,2020[C].IEEE。

Claims

1. A semantic segmentation method for the canopy of economic ginkgo leaf forest in real scenes. This method uses a semantic segmentation model to process the orthophoto of the leaf ginkgo canopy collected by a small drone at low altitude remote sensing, and distinguishes the leaf ginkgo canopy from the complex background. Its characteristics are The semantic segmentation model is a Haar wavelet enhanced semantic attention UNet model, namely Haar_UNet; Haar_UNet is an improvement of the UNet segmentation network, and the improvement method is to perform the following design under the network framework of the "U"-shaped structure encoder and decoder of the UNet segmentation network:

1. Replace the maximum pooling downsampling layer of the coding layer with the Haar wavelet downsampling module. After decomposing the original image information into low-frequency sub-bands and high-frequency sub-bands, the high-frequency information can be losslessly restored for subsequent feature extraction and reconstruction. Second, embed the SENet attention mechanism module after the feature channel dimension increase and dimension reduction; 3. Embed a Multi grid hole convolution module before downsampling each encoder layer; The Haar Wavelet Downsampling module, HWD, includes a lossless feature coding block and a feature learning module; in, The lossless feature encoding block is responsible for transforming features and reducing spatial resolution; The lossless feature coding block uses the Haar wavelet function and the scaling function to decompose and reconstruct the input image feature signal; First, a low-frequency approximate component of the characteristic pixel of the ginkgo canopy and high-frequency detail components in the vertical, horizontal and diagonal directions are established through wavelet transform; then these four components are conca-concatenated as parameters input to the feature learning module; among them, The definition formula of the first-level one-dimensional scaling function and the wavelet function of the Haar transform is as follows: Where: φ j，k (x) is defined as: Where: parameter j represents the Haar basis function series, and taking 1 here means that the scale of the image processing field is reduced by 1 / 2); k takes an integer value and is used to determine the position of the wavelet function on the signal or image; is the Haar wavelet basis function, expressed as the difference of the unit step function, Represents the original wavelet function To perform horizontal translation and scaling, specifically, 2 j x scales the wavelet function horizontally, while -k translates it to the right by k units; is a normalization coefficient, which ensures that the energy of the wavelet function remains consistent at different scales; Defined as: Then, the first-level one-dimensional is expressed using the 0-level Haar basis function as: This means that a signal of length L is divided into two parts of length L / 2, which are interpreted as low-pass and high-pass decomposition filters respectively; when the Haar wavelet transform is applied to a two-dimensional signal, four components are produced, each with half the spatial resolution of the original image; The feature learning module consists of a 1×1 standard convolution layer, a batch normalization BN, and a ReLU activation function layer.

2. The method for semantic segmentation of the canopy of economic ginkgo leaves in a real scene according to claim 1 is characterized in that The Multi grid hole convolution module is a multi-scale dilated convolution module, which is composed of three dilated convolution layers stacked together; the dilation coefficients of the three convolution layers are set to 1, 2, and 4, respectively, to capture feature information at different scales; in the multi-scale dilated convolution module, First, a convolutional layer with a dilation factor of 1 is used to extract the original features of the image; Next, by adding convolutional layers with expansion factors of 2 and 4, the receptive field is expanded to capture semantic information in a wider range; Finally, a batch normalization layer BN is added to normalize the features.

3. The method for semantic segmentation of the canopy of economic ginkgo leaves in real scenes according to claim 1 is characterized by Haar _During the training process of the UNet model, the loss function is the Dice coefficient loss function Dice Loss, Where: y i and They represent the label value and predicted value of the i-th pixel respectively; N is the total number of pixels, which is equal to the number of pixels in a single image multiplied by the batch size.

4. The method for semantic segmentation of the canopy of economic ginkgo leaves in a real scene according to claim 1 is characterized in that The ginkgo canopy image is an image of the leaf ginkgo tree canopy taken by a low-altitude drone.

5. The method for semantic segmentation of the canopy of economic ginkgo leaves in a real scene according to claim 1, characterized in that The collected ginkgo canopy images are first preprocessed and the data set is constructed, and then input into Haar_UNet; the methods of data preprocessing and data set construction are as follows: Firstly, the collected ginkgo canopy orthophotos were uniformly segmented into images of the same size to obtain a sample data set; Then, the sample data sets are annotated one by one to generate a sample label data set corresponding to the original image in order, quantity and size; Next, data augmentation was performed on the images and their sample label datasets; Data augmentation includes random scaling, horizontal and vertical flipping, and random shuffling. Finally, multiple groups of sample data sets were obtained; these sample data sets were divided into training sets, validation sets, and test sets, which were used for training, parameter adjustment, and performance evaluation of the Haar_UNet model, respectively.

Images