Wide-angle single tree image segmentation method and system based on improved YOLOv8-Seg understory

Abstract

The invention discloses a method and system for segmenting wide-angle single-tree images under forest based on improved YOLOv8-Seg, the method comprising using a binocular fisheye camera to collect wide-angle depth image data of a target forest plot, obtaining a wide-angle RGB image of a forest tree, and dividing it into a training set and a test set; constructing a YOLOv8-Seg forest tree target detection model based on the wide-angle RGB image of the forest tree; the YOLOv8-Seg forest tree target detection model comprises a block convolution module, an adaptive weight module and a cross fusion module; training the YOLOv8-Seg forest tree target detection model based on the training set and the test set of the wide-angle RGB image of the forest tree; using the trained YOLOv8-Seg forest tree target detection model, extracting the depth information of the single-tree image, and calculating the actual tree height and breast diameter parameters of the single tree based on the depth information of the single-tree image. The invention enhances the segmentation ability of the model and improves the accuracy of the segmentation of single-tree images under forest under complex forest background.

Description

Technical Field

[0001] The present invention belongs to the field of image segmentation, and in particular to a method and system for segmenting wide-angle single-tree images under forest based on improved YOLOv8-Seg. Background Art

[0002] The study of individual tree segmentation (ITS) is one of the basic tasks for the construction of precision forestry and smart forestry in the future. In the forestry field, accurate extraction of forest structure parameters is an important basis for key tasks such as forest monitoring, forest stock and carbon storage assessment, an important indicator for revealing the growth status and laws of forests, and an important reference for judging the effectiveness of forest resource management.

[0003] For complex natural environments such as forests, the main method for obtaining forest structure parameters at this stage is to use drones or remote sensing image processing. However, the accuracy of images obtained by satellite remote sensing images and drone photography is low, and the operation and maintenance costs are relatively high. At the same time, due to factors such as canopy obstruction and shooting distance, the image data collected is difficult to meet the needs of obtaining deep information under the forest, and it is impossible to complete the extraction of more detailed growth structure information of the forest, such as breast diameter and tree height. Summary of the invention

[0004] The object of the present invention is to provide a method and system for segmenting wide-angle single tree images under the forest based on improved YOLOv8-Seg, so as to solve the above-mentioned problems existing in the prior art.

[0005] The technical solution to achieve the purpose of the present invention is: a method for segmenting a wide-angle single tree image under the forest based on improved YOLOv8-Seg, comprising the following steps:

[0006] S1. Use a binocular fisheye camera to collect wide-angle depth image data of the target forest plot, obtain wide-angle RGB images of trees, and divide them into training sets and test sets;

[0007] S2. Based on the wide-angle RGB image of trees, a YOLOv8-Seg tree target detection model is constructed; the YOLOv8-Seg tree target detection model includes a block convolution module, an adaptive weight module, and a cross fusion module;

[0008] S3, based on the training set and test set of the wide-angle RGB images of the trees, train the YOLOv8-Seg tree target detection model;

[0009] S4. Use the trained YOLOv8-Seg tree target detection model to extract the depth information of the single tree image, and calculate the actual tree height and breast diameter parameters of the single tree based on the depth information of the single tree image.

[0010] According to one aspect of the present application, step S1 further comprises:

[0011] S11, installing a binocular fisheye camera on a mobile platform, and capturing wide-angle depth image data of the target forest plot at a predetermined interval and angle, including two wide-angle RGB images on the left and right and corresponding depth images;

[0012] S12, using a fisheye distortion correction algorithm to perform distortion correction on the wide-angle depth image data to obtain a wide-angle RGB image of the forest;

[0013] S13, preprocessing the forest wide-angle RGB image, including removing noise, enhancing brightness, adjusting contrast and scaling;

[0014] S14. Using a data partitioning algorithm, the preprocessed forest wide-angle RGB image is divided into a training set and a test set according to a predetermined ratio.

[0015] According to one aspect of the present application, step S2 is further:

[0016] S21. Build the YOLOv8-Seg tree target detection model based on the wide-angle RGB image of the tree and the YOLOv8 Ultralytics open source framework.

[0017] S22, using a block convolution module to extract features from the preprocessed forest wide-angle RGB image to obtain a multi-scale feature map;

[0018] S23, using an adaptive weight module to adaptively adjust multi-scale feature maps through weight adaptation and attention mechanism;

[0019] S24, using a cross fusion module to fuse the adaptively adjusted feature maps;

[0020] S25, input the fused feature map into the head network to predict the boundary and category of the feature map;

[0021] S26, inputting the fused feature map and the predicted boundaries and categories into the segmentation network to predict the segmentation mask and contour of the single tree image;

[0022] S27. Construct the loss function of YOLOv8-Seg based on the segmentation mask and contour of the single tree image.

[0023] According to one aspect of the present application, step S3 further comprises:

[0024] S31. Use data enhancement algorithm to preprocess the wide-angle RGB images of trees in the training set, including rotation, translation, scaling, cropping, flipping and color transformation;

[0025] S32. Based on the preprocessed training set, the optimization algorithm is used to update the parameters of the YOLOv8-Seg forest target detection model to minimize the loss function.

[0026] S33. Use the YOLOv8-Seg tree target detection model with updated parameters to infer the wide-angle RGB image of the trees in the test set to obtain the segmentation results and detection box coordinates of each single tree.

[0027] S34, based on the segmentation results and detection box coordinates of each single tree, the YOLOv8-Seg tree target detection model with updated parameters is evaluated to obtain an evaluation index;

[0028] S35. Based on the evaluation index, use the early stopping method to determine whether to continue updating the parameters of the YOLOv8-Seg forest target detection model. If yes, return to step S31; otherwise, output the trained YOLOv8-Seg forest target detection model.

[0029] According to one aspect of the present application, step S4 further comprises:

[0030] S41. Use the trained YOLOv8-Seg forest target detection model to determine the pixel area of ​​each tree, extract the depth values ​​of all pixels in the pixel area, convert the depth values ​​into grayscale values, and obtain a grayscale image of the single tree area;

[0031] S42, using a Gaussian filter to smooth and reduce noise on the grayscale image of a single tree area;

[0032] S43, using a correction method to perform depth value correction on the single tree region grayscale image after smoothing and noise reduction;

[0033] S44, based on the grayscale value, converting the single tree region grayscale image after the depth value correction into a real single tree depth image;

[0034] S45. Based on the real single tree depth map, calculate the actual tree height and breast diameter parameters of the single tree.

[0035] According to one aspect of the present application, the method further includes labeling the wide-angle RGB image of the forest, and the specific steps are as follows:

[0036] S12a, obtaining spectral information of the wide-angle RGB image of the forest, and clustering the spectral information of the wide-angle RGB image of the forest using a spectral clustering algorithm to obtain a spectral category of each pixel;

[0037] S12b, using the region growing algorithm to merge the pixels with the same spectral category to obtain the spectral region of each single tree;

[0038] S12c, using a spectral feature extraction algorithm to extract features from the spectral region of each single tree, and obtaining a spectral feature vector of each single tree;

[0039] S12d, using a spectral classification algorithm to classify the spectral feature vector of each single tree to obtain a tree species label of each single tree;

[0040] S12e, annotate the tree species label of each single tree in its corresponding wide-angle RGB image of the forest.

[0041] According to one aspect of the present application, step S1 may also be:

[0042] S1a, using binocular fisheye camera and lidar camera, collect wide-angle RGBD image data of the target forest plot, including two wide-angle RGB images, depth image and point cloud data;

[0043] S1b, using the wide-angle camera calibration plate and the point cloud calibration plate, respectively obtain the internal and external parameters of the binocular fisheye camera and the internal and external parameters of the laser radar;

[0044] S1c, based on the internal and external parameters of the binocular fisheye camera and the internal and external parameters of the laser radar, the wide-angle RGBD image data is corrected by using the joint distortion correction and registration equation;

[0045] S1d. Use labelme annotation software to manually annotate the corrected wide-angle RGBD image data. Based on the manually annotated wide-angle RGBD image data, generate a self-supervised pre-training dataset and a supervised fine-tuning dataset.

[0046] According to one aspect of the present application, step S2 further includes:

[0047] S2a, build the YOLOv8-Seg forest target detection model based on the forest wide-angle RGB image and the YOLOv8 Ultralytics open source framework;

[0048] S2b, replace the backbone network of the YOLOv8-Seg forest target detection model with a hybrid feature pyramid HFP structure, and perform multi-scale feature extraction on the preprocessed forest wide-angle RGB image; wherein the hybrid feature pyramid HFP structure includes a shallow feature branch, a deep feature branch, and a point cloud feature branch;

[0049] S2c, based on the extracted multi-scale features, generate hard-to-distinguish samples, introduce adversarial generation mechanism into the detection head of the YOLOv8-Seg forest target detection model, and perform enhanced training on hard-to-distinguish samples;

[0050] S2d, adopts the multi-view fusion detection mechanism MFDet, constructs a gated recurrent unit and a bidirectional attention mechanism, performs spatiotemporal fusion on the hard-to-separate samples after enhanced training, and generates a three-dimensional detection frame.

[0051] According to one aspect of the present application, step S3 may also be:

[0052] S3a, based on the self-supervised pre-training dataset and the supervised fine-tuning dataset, a two-stage training strategy is adopted to train the YOLOv8-Seg forest target detection model;

[0053] S3b. Use contrastive learning loss and depth estimation loss to optimize the trained YOLOv8-Seg forest target detection model.

[0054] The present invention also provides a forest wide-angle single tree image segmentation system based on improved YOLOv8-Seg, using the forest wide-angle single tree image segmentation method based on improved YOLOv8-Seg described in any of the above technical solutions, including a data acquisition module, a model construction module, a model training module and an image segmentation module.

[0055] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention completes feature extraction through a block convolution module and an adaptive weight module, and uses a cross fusion module to complete feature fusion, thereby enhancing the segmentation ability of the model and improving the accuracy of image segmentation of single trees under the forest under a complex forest background, while solving the problem that drones or remote sensing technology cannot accurately obtain information under the forest. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flow chart of the present invention.

[0057] Figure 2 This is a flow chart of step S1 of the present invention.

[0058] Figure 3 This is a flow chart of step S2 of the present invention.

[0059] Figure 4 This is a flow chart of step S3 of the present invention.

[0060] Figure 5 This is a flow chart of step S4 of the present invention.

[0061] Figure 6 is a flow chart of an embodiment of the present invention.

[0062] Figure 7 It is an F1 score-confidence curve diagram after target detection in an embodiment of the present invention.

[0063] Figure 8Graph showing the accuracy-confidence after target detection in an embodiment of the present invention.

[0064] Fig. 9 Graph showing the precision-recall rate after target detection in an embodiment of the present invention.

[0065] Fig.10 This is a recall rate-confidence curve diagram after target detection in an embodiment of the present invention.

[0066] Fig.11 FIG. 4 is a graph showing an F1 score-confidence curve after mask processing in an embodiment of the present invention.

[0067] Fig.12 4 is a precision-confidence curve diagram after mask processing in an embodiment of the present invention.

[0068] Fig.13 4 is a precision-recall curve after mask processing in an embodiment of the present invention.

[0069] Fig.14 4 is a recall rate-confidence curve diagram after mask processing in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] like Figure 1 As shown, this application proposes a method for segmenting wide-angle single tree images under the forest based on improved YOLOv8-Seg, and the specific steps are:

[0071] S1. Use a binocular fisheye camera to collect wide-angle depth image data of the target forest plot, obtain wide-angle RGB images of trees, and divide them into training sets and test sets;

[0072] S2. Based on the wide-angle RGB image of trees, a YOLOv8-Seg tree target detection model is constructed; the YOLOv8-Seg tree target detection model includes a block convolution module, an adaptive weight module, and a cross fusion module;

[0073] S3, based on the training set and test set of the wide-angle RGB images of the trees, train the YOLOv8-Seg tree target detection model;

[0074] S4. Use the trained YOLOv8-Seg tree target detection model to extract the depth information of the single tree image, and calculate the actual tree height and breast diameter parameters of the single tree based on the depth information of the single tree image.

[0075] In this example, the YOLOv8-Seg model is an instance segmentation model based on deep learning, which can perform target detection and semantic segmentation at the same time, that is, identify and segment the category, position and outline of each object in the image. The YOLOv8-Seg model adds a segmentation branch on the basis of the YOLOv8 model to output the segmentation mask. The YOLOv8 model is a fast and accurate target detection model that uses multi-scale feature pyramids, attention mechanisms, path aggregation networks and other technologies to improve the model's feature extraction and target detection capabilities. This application improves the YOLOv8 model by introducing a block convolution module, an adaptive weight module and a cross fusion module, which are used to enhance the channel relationship of features, adjust the scale of features and fuse the spatial information of features, further improving the performance of the model.

[0076] In this embodiment, a binocular fisheye camera is used to collect wide-angle depth image data of the forest, which can effectively cover the all-round information of the trees and improve the integrity and accuracy of the data; the YOLOv8-Seg model is adopted, which combines the advantages of target detection and semantic segmentation, and can simultaneously output the category, position, boundary, mask and outline of the trees, thereby realizing the refined segmentation of the trees; the YOLOv8 model is improved by using a block convolution module, an adaptive weight module and a cross fusion module, which enhances the feature expression and fusion ability of the model and improves the performance and robustness of the model; based on the depth information of a single tree, the actual tree height and breast diameter parameters of the single tree are calculated, which can effectively reflect the growth status and resource value of the trees and provide useful information for forestry management and decision-making.

[0077] like Figure 2 As shown, according to one aspect of the present application, step S1 is further:

[0078] S11, installing a binocular fisheye camera on a mobile platform, and capturing wide-angle depth image data of the target forest plot at a predetermined interval and angle, including two wide-angle RGB images on the left and right and corresponding depth images;

[0079] S12, using a fisheye distortion correction algorithm to perform distortion correction on the wide-angle depth image data to obtain a wide-angle RGB image of the forest;

[0080] S13, preprocessing the forest wide-angle RGB image, including removing noise, enhancing brightness, adjusting contrast and scaling;

[0081] S14. Using a data partitioning algorithm, the preprocessed forest wide-angle RGB image is divided into a training set and a test set according to a predetermined ratio.

[0082] In a further embodiment, because the images currently acquired through satellite and UAV remote sensing have low accuracy, high operation and maintenance costs, and cannot collect more accurate forest information, a binocular fisheye camera is installed on a forest ground inspection robot to obtain depth information of forest tree images, and it has better adaptability in extracting information from tall trees.

[0083] To collect data of wide-angle depth images, a binocular fisheye camera is used to obtain the left and right wide-angle RGB images IL and IR and the corresponding depth image ID. Through distortion correction, the tree calibration images in the forest environment are obtained and made into training sets and test sets.

[0084] After obtaining the initial wide-angle image, the wide-angle image is corrected by the wide-angle lens distortion correction model. The model uses the checkerboard template method and combines the radial and tangential distortion correction equations to perform wide-angle lens correction. The formula is:

[0085] x c =x(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )+2p 1 xy+p 2 (r 2 +2x 2 );

[0086] y c =y(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )+2p 2 xy+p 1 (r 2 +2x 2 );

[0087] r 2 =x 2 +y 2 ;

[0088] Among them, (x c ,y c ) is the position of the pixel in the wide-angle image, r represents the radial distance from the pixel to the center of the image, and the position after correction by the distortion correction equation is (x, y). 1 , k 2 is the radial distortion coefficient of the wide-angle lens, p 1 、p 2 is the tangential distortion coefficient of the wide-angle lens. 2 and r4 Characterizes radial distortion and is related to radial distance; while xy, x 2 and 2 It represents the tangential distortion, which is related to the pixel coordinates. By properly setting the distortion coefficient k 1 , k 2 、p 1 and p 2 , different types and degrees of lens distortion can be corrected.

[0089] The tree calibration images are made into training sets and test sets. The labelme image annotation software is used to manually annotate the calibration images in the training set and test set respectively. The label file is in json format. The labelme_json_to_dataset command is used to convert the json format label file into a jpg format annotation image.

[0090] like Figure 3 As shown, according to one aspect of the present application, step S2 further comprises:

[0091] S21. Build the YOLOv8-Seg tree target detection model based on the wide-angle RGB image of the tree and the YOLOv8 Ultralytics open source framework.

[0092] S22, using a block convolution module to extract features from the preprocessed forest wide-angle RGB image to obtain a multi-scale feature map;

[0093] S23, using an adaptive weight module to adaptively adjust multi-scale feature maps through weight adaptation and attention mechanism;

[0094] S24, using a cross fusion module to fuse the adaptively adjusted feature maps;

[0095] S25, input the fused feature map into the head network to predict the boundary and category of the feature map;

[0096] S26, inputting the fused feature map and the predicted boundaries and categories into the segmentation network to predict the segmentation mask and contour of the single tree image;

[0097] S27. Construct the loss function of YOLOv8-Seg based on the segmentation mask and contour of the single tree image.

[0098] Since the image sources of the single tree segmentation task are often collected in outdoor places such as forests, the background has highly complex features, so the requirements for the model's feature extraction capabilities are higher than other segmentation tasks with clear backgrounds. YOLOv8-Seg is based on the YOLOv8 Ultralytics open source framework on github. The convolution module in the backbone network is replaced with a block convolution module, and the input is divided into blocks, y(0) and y(-1), where y(0) refers to the first position from left to right. The convolution block of the left channel y(-1) obtains the feature extraction and downsampling results, and the right channel y(0) uses a newly designed maximum pooling layer with a stride of 2 to complete the downsampling, and then uses a convolution block with a stride of 1 for feature extraction. The size of the feature map obtained is the same as that obtained from the left channel. Finally, the two feature maps are spliced ​​to obtain the feature map of the final result, which has the same size as after the convolution operation and contains richer feature information. In the process of feature extraction, the feature map completes two feature extraction operations after passing through the block convolution module, and the convolution receptive field of the left line is small, while the receptive field of the right line after passing through the 2×2 maximum pooling layer and the 1×1 convolution module is larger. This module extracts feature information of the image from different angles and splices and fuses the two. Compared with the existing convolution module, the block convolution module has stronger feature extraction capabilities.

[0099] The block convolution module consists of two parallel processing branches. The left branch contains a 3x3 convolution layer (Conv3x3), a batch normalization layer (BatchNorm) and a ReLU activation layer in sequence; the right branch contains a 2x2 maximum pooling layer (MaxPool2x2), a 1x1 convolution layer (Conv1x1), a batch normalization layer and a ReLU activation layer in sequence; finally, the outputs of the two branches are spliced ​​in the channel dimension to obtain the final output.

[0100] The input feature map of the block convolution module is x, with a shape of (B, C, H, W), i.e., batch size, number of channels, height, and width. First, x is equally divided into two parts in the channel dimension to obtain x1 and x2, both of which have a shape of (B, C / 2, H, W); in the left branch, x1 passes through Conv3x3, BatchNorm, and ReLU in turn to obtain a feature map f1 with a shape of (B, C / 2, H, W). Conv3x3 performs a normal convolution on the feature map to extract local features while maintaining resolution; in the right branch, x2 is first downsampled by MaxPool2x2 to obtain a feature map p with a shape of (B, C / 2, H / 2, W / 2). MaxPool2x2 uses a 2x2 sliding window to perform maximum pooling on the feature map, reducing the resolution and increasing the receptive field; then, the feature map p passes through Conv1x1, BatchNorm and ReLU to obtain the feature map f2, which has a shape of (B, C / 2, H / 2, W / 2). Conv1x1 performs point-by-point convolution on the downsampled feature map, adjusting the number of channels without changing the resolution; then, the feature map f2 is upsampled to the original size through bilinear interpolation to obtain the feature map f2', which has a shape of (B, C / 2, H, W). The upsampling operation restores the spatial detail information lost in the right branch; finally, the output feature map f1 of the left branch and the output feature map f2' of the right branch are spliced ​​in the channel dimension to obtain the final output y of the block convolution module, which has a shape of (B, C, H, W). The splicing operation integrates the feature information of different scales and different receptive fields extracted by the two branches.

[0101] This embodiment introduces a blocking mechanism, divides the input feature map into two parts in the channel dimension, and processes them separately, thereby increasing feature diversity; adopts different convolution and pooling operations to make the two branches have different receptive fields and feature scales, thereby improving the multi-scale feature extraction capability of the model; through cross-scale upsampling and splicing operations, the effective fusion of local and global features is achieved, and the feature representation capability is enhanced.

[0102] The C2f module is divided into blocks and improved into an adaptive weight module. Specifically, the input is divided into tensors after passing through the convolution block. The left and right channels are y(-1) and y(0) respectively. The attention mechanism is introduced after y(-1) and y(0). After the AVG module (AdaptivaAvgpooling) and the Sigmod function are normalized, the network can adaptively adjust the feature map, give a larger weight to the channel with a large contribution to the model, and vice versa, give a smaller weight to the channel with a small contribution. The weighted output of the left and right channels is obtained by the distribution method obtained by the weight learning of the feature map. The left channel multiplies the weight of y(-1) by the normalized weight to obtain a new weight. The right channel passes through n bottleneck layers (Bottleneck) after y(0), and the attention mechanism of the Sigmod function is applied to the bottleneck layer to obtain n outputs. All output results are spliced ​​according to the weight ratio learned by themselves, and finally the channel adjustment is completed through Conv2 to obtain the final output result.

[0103] The adaptive weight module consists of two parallel processing branches. The left branch contains an adaptive average pooling layer (AdaptiveAvgPool2d), a 1x1 convolution layer (Conv1x1), a batch normalization layer (BatchNorm), and a Sigmoid activation layer in sequence; the right branch contains n series-connected bottleneck blocks (Bottleneck), each of which contains a 1x1 convolution layer (Conv1x1), a batch normalization layer (BatchNorm), a ReLU activation layer, a 3x3 convolution layer (Conv3x3), a batch normalization layer, and a Sigmoid activation layer in sequence; finally, the outputs of the left and right branches are added according to the weights, and then the number of channels is adjusted through a 1x1 convolution layer to obtain the final output.

[0104] Suppose the input feature map of the adaptive weight module is x, with a shape of (B, C, H, W); in the left branch, x is first adaptively global average pooled by AdaptiveAvgPool2d to obtain a feature map p with a shape of (B, C, 1, 1). AdaptiveAvgPool2d adaptively adjusts the pooling window size according to the input size, so that the output shape is fixed to 1x1, which is equivalent to generating a global descriptor for each channel; then, the feature map p passes through Conv1x1, BatchNorm, and Sigmoid in turn to obtain the weight map w, which has a shape of (B, C, 1, 1), where Conv1x1 is used to adjust the number of channels, and Sigmoid maps the weight value to between 0-1 to obtain the normalized attention weight; then, the input x and the weight map w are multiplied element by element to obtain the weighted feature map f1, which has a shape of (B, C, H, W). This step applies the global attention weight to the original feature map, so that the model pays attention to important channels and regions; in the right branch, x passes through n bottleneck blocks in turn to obtain n intermediate feature maps. Each bottleneck block first uses Conv1x1 to reduce the number of channels, then uses Conv3x3 to extract features, and finally uses Sigmoid to generate spatial attention weights, which are multiplied with the intermediate feature map to obtain the weighted feature map. This process mines the multi-scale and multi-level information of the input feature map; the n weighted feature maps of the right branch are spliced ​​in the channel dimension to obtain the feature map f2, which has a shape of (B, nC, H, W); finally, the output feature map f1 of the left branch and the output feature map f2 of the right branch are added according to the weights α and 1-α, and the number of channels is adjusted through Conv1x1 to obtain the final output y of the adaptive weight module, which has a shape of (B, C, H, W). The weighted addition operation combines the global and local attention features extracted by the two branches.

[0105] Based on the traditional feature fusion module C2f, channel attention and spatial attention mechanisms are introduced, so that the model can adaptively adjust the importance of different regions and channels, and improve the feature representation ability; multiple serial bottleneck blocks are used to extract multi-scale features, which enhances the adaptability and robustness of the model to target scale changes; global and local attention features are fused by weighted addition, making full use of the complementarity of the two attention mechanisms to make the features more refined and discriminative.

[0106] The cross-fusion module is to splice the feature maps of the three layers output from the backbone network in different ways to maximize the efficiency of feature fusion, and finally send them to the head network for regression prediction. The cross-fusion module uses upsampling when fusing features in the x direction, and uses downsampling when fusing features in the y direction. The two are spliced, and then feature fusion is performed through C2f to obtain the output result. The cross-fusion module separates the upsampling and downsampling tasks of the model during feature fusion so that they do not interfere with each other. This method and feature fusion methods such as FCN require sampling, but separates the upsampling and downsampling tasks to make the feature fusion process clearer. The specific steps of the improvement method are as follows:

[0107] The input feature maps of the 4th layer (80×80), the 6th layer (40×40), and the 9th layer (20×20) are divided into (x0, y0), (x1, y1), and (x2, y2) respectively through the chunk() function;

[0108] After convolution and upsampling of x2, x40 is obtained after passing through the C2f module;

[0109] Concatenate x40 and x1, and perform convolution, upsampling, and C2f on the result to obtain x80;

[0110] Concatenate x80 and x0, and pass the result through Conv and C2f modules to get x80-1;

[0111] Concatenate x80-1 and y0, and get y40 through Conv and downsampling, C2f;

[0112] Concatenate y40 and y1, and after Conv and C2f, get y20, which is concatenated with y2 to get y20-final;

[0113] Out20: Concatenate y20-final with x20-1 (the result of Conv3 on x2), and pass through C2f to get Out20 (corresponding to a feature map of size 4x4).

[0114] Out40: Concatenate y40-final (the concatenation of y40 and y1) and x40-1 (the result of Conv after concatenating x40 and x1), and pass C2f to get Out40 (corresponding to a feature map of size 4x4).

[0115] Out80: Concatenate y80-final (the concatenation of x80-1 and y0) and x80-1, and pass C2f to get Out80;

[0116] Finally, the output (Out20, Out40, Out80) is sent to the segmentation module for segmentation.

[0117] The YOLOv8-Seg network model of this embodiment replaces the convolution (Conv) module in the backbone network with a block convolution module based on the idea of ​​task partitioning. The block convolution module partitions the feature extraction and downsampling tasks in the input x, and finally concatenates the two results to obtain the final result. The size of the resulting feature map is the same as that of the feature map obtained by the left channel, and contains richer feature information.

[0118] The cross-fusion module contains three parallel processing branches and two cross-fusion branches. The first parallel branch extracts features from the three feature maps x1, x2, and x3 of the backbone network in turn through a 3x3 convolution layer (Conv3x3); the second parallel branch downsamples the three feature maps in turn through a 2x2 maximum pooling layer (MaxPool2x2); the third parallel branch upsamples the three feature maps in turn through an upsampling layer (Upsample); the first cross-fusion branch concatenates the outputs of the first and third parallel branches layer by layer, and then adjusts the number of channels through a 1x1 convolution layer to obtain a fused feature map of three scales; the second cross-fusion branch concatenates the outputs of the second and third parallel branches layer by layer, and then adjusts the number of channels through a 1x1 convolution layer to obtain a fused feature map of three scales; finally, the outputs of the two cross-fusion branches are added element by element at the same scale to obtain the final output of three scales.

[0119] Assume that the three feature maps output by the backbone network are x1, x2, and x3, and their shapes are (B, C1, H1, W1), (B, C2, H2, W2), and (B, C3, H3, W3), where H1>H2>H3, W1>W2>W3;

[0120] In the first parallel branch, x1, x2, and x3 are respectively extracted with Conv3x3 to obtain feature maps f1, f2, and f3, with shapes of (B, C1', H1, W1), (B, C2', H2, W2), and (B, C3', H3, W3).

[0121] In the second parallel branch, x1, x2, and x3 are downsampled by MaxPool2x2 to obtain feature maps p1, p2, and p3, with shapes of (B, C1, H1 / 2, W1 / 2), (B, C2, H2 / 2, W2 / 2), and (B, C3, H3 / 2, W3 / 2), respectively.

[0122] In the third parallel branch, x1, x2, and x3 are upsampled by Upsample to obtain feature maps u1, u2, and u3, with shapes of (B, C1, H1*2, W1*2), (B, C2, H2*2, W2*2), and (B, C3, H3*2, W3*2), respectively.

[0123] In the first cross-fusion branch, the feature map f1 and the feature map u2 are spliced ​​in the spatial dimension to obtain the feature map c1, which has a shape of (B, C1'+C2, H1, W1); the feature map f2 and the feature map u3 are spliced ​​in the spatial dimension to obtain the feature map c2, which has a shape of (B, C2'+C3, H2, W2); the feature map f3 and the feature map u1 are spliced ​​in the spatial dimension to obtain the feature map c3, which has a shape of (B, C3'+C1, H3, W3). Then, the number of channels is adjusted through Conv1x1 to obtain the fused feature maps ff1, ff2, and ff3.

[0124] In the second cross-fusion branch, the feature map p1 and the feature map u3 are spliced ​​in the spatial dimension to obtain the feature map d1, which has a shape of (B, C1+C3, H1 / 2, W1 / 2); the feature map p2 and the feature map u1 are spliced ​​in the spatial dimension to obtain the feature map d2, which has a shape of (B, C2+C1, H2 / 2, W2 / 2); the feature map p3 and the feature map u2 are spliced ​​in the spatial dimension to obtain the feature map d3, which has a shape of (B, C3+C2, H3 / 2, W3 / 2). Then, the number of channels is adjusted through Conv1x1 to obtain the fused feature maps fp1, fp2, and fp3.

[0125] Finally, the outputs of the two cross-fusion branches are added element-wise at the same scale, i.e., ff1+fp1, ff2+fp2, ff3+fp3, to obtain the three-scale outputs o1, o2, and o3 of the cross-fusion module, with shapes of (B, C, H1, W1), (B, C, H2, W2), and (B, C, H3, W3), respectively.

[0126] This embodiment adopts parallel multi-scale feature extraction branches to perform convolution, downsampling and upsampling operations on the three-scale feature maps output by the backbone network, respectively, to obtain feature representations with different receptive fields and resolutions, thereby improving the model's ability to utilize multi-scale information; a cross-fusion mechanism is introduced to splice the features extracted by different parallel branches in the spatial dimension, and then perform channel adjustment, thereby achieving cross-scale and cross-level feature aggregation and complementarity, so that the final fused feature map contains more comprehensive and richer semantic information; on the basis of cross-fusion, the outputs of the two cross branches are further fused by element-by-element addition, making full use of the diverse features extracted from them at different scales, so that the discriminability and robustness of the output feature map are further improved; the entire fusion process achieves efficient and sufficient multi-scale feature fusion through clever splicing and addition operations without increasing the amount of calculation, providing high-quality feature input for subsequent target detection and instance segmentation tasks.

[0127] The adaptive weight module divides the input x into tensor blocks after convolution, and introduces the attention mechanism in the left and right channels. The left channel multiplies the weight of y(-1) by the normalized weight to obtain a new weight. The right channel passes through n bottleneck layers after y(0), and the attention mechanism of the Sigmod function is applied to the bottleneck layer to obtain n outputs. All output results are spliced ​​according to the weight ratio learned by themselves, and finally the channel adjustment is completed through Conv2 to obtain the final output result. The adaptive weight module is normalized by the (Adaptiva Avgpooling) AVG module and the Sigmod function, which can enable the network to adaptively adjust the feature map, giving larger weights to channels that contribute more to the model, and vice versa, giving smaller weights to channels that contribute less. The weighted output of the left and right channels is obtained by the distribution method obtained through the weight learning of the feature map.

[0128] The core idea of ​​the cross-fusion module improvement is to splice the three-layer feature maps output from the backbone network in different ways to maximize the efficiency of feature fusion, and finally send them to the head network for regression prediction. The improvement of the network can enhance the accuracy and efficiency of model feature acquisition and improve the model's ability to extract single trees in complex forest backgrounds. The vision-based single tree extraction solution also conforms to the current development of high-computing technology.

[0129] like Figure 4 As shown, according to one aspect of the present application, step S3 further comprises:

[0130] S31. Use data enhancement algorithm to preprocess the wide-angle RGB images of trees in the training set, including rotation, translation, scaling, cropping, flipping and color transformation;

[0131] S32. Based on the preprocessed training set, the optimization algorithm is used to update the parameters of the YOLOv8-Seg forest target detection model to minimize the loss function.

[0132] S33. Use the YOLOv8-Seg tree target detection model with updated parameters to infer the wide-angle RGB image of the trees in the test set to obtain the segmentation results and detection box coordinates of each single tree.

[0133] S34, based on the segmentation results and detection box coordinates of each single tree, the YOLOv8-Seg tree target detection model with updated parameters is evaluated to obtain an evaluation index;

[0134] S35. Based on the evaluation index, use the early stopping method to determine whether to continue updating the parameters of the YOLOv8-Seg forest target detection model. If yes, return to step S31; otherwise, output the trained YOLOv8-Seg forest target detection model.

[0135] In this embodiment, a data enhancement algorithm is used to rotate, translate, scale, crop, flip, and color-change the wide-angle RGB images of trees in the training set. The data enhancement algorithm is a method that increases the amount and diversity of data by performing some transformations on the original data, thereby improving the generalization and robustness of the model. This can simulate forest images under different viewing angles, scales, positions, and lighting conditions, increasing the coverage and complexity of the training data. A gradient descent-based optimization algorithm is used to update the parameters of the YOLOv8-Seg forest target detection model, so that it can better identify and locate forest targets and output the corresponding segmentation mask. The optimization algorithm is a method that minimizes the loss function by adjusting the parameters of the model, thereby improving the model's fitting ability and prediction performance. Based on the test set of wide-angle RGB images of trees, the YOLOv8-Seg tree target detection model after parameter update was evaluated, and some evaluation indicators, such as mean average precision (mAP), recall, precision, and segmentation accuracy (IoU), were obtained, reflecting the performance of the model in tree target detection and segmentation tasks. Evaluation indicators are a method to measure the accuracy and effectiveness of a model by comparing the model's prediction results with the true labels, thereby providing the optimization direction and stopping conditions of the model. Based on the evaluation indicators, an early stopping method based on the test set mAP was used to determine whether to continue training. If so, data augmentation and optimization will continue; otherwise, the trained YOLOv8-Seg tree target detection model is output as the final model. Early stopping is a method to prevent the model from overfitting or underfitting by monitoring the performance of the model on the test set, thereby improving the generalization ability and stability of the model.

[0136] like Figure 5 As shown, according to one aspect of the present application, step S4 further comprises:

[0137] S41. Use the trained YOLOv8-Seg forest target detection model to determine the pixel area of ​​each tree, extract the depth values ​​of all pixels in the pixel area, convert the depth values ​​into grayscale values, and obtain a grayscale image of the single tree area;

[0138] S42, using a Gaussian filter to smooth and reduce noise on the grayscale image of a single tree area;

[0139] S43, using a correction method to perform depth value correction on the single tree region grayscale image after smoothing and noise reduction;

[0140] S44, based on the grayscale value, converting the single tree region grayscale image after the depth value correction into a real single tree depth image;

[0141] S45. Based on the real single tree depth map, calculate the actual tree height and breast diameter parameters of the single tree.

[0142] In a further embodiment, the depth information of the target single tree after segmentation is extracted, the grayscale value of the single tree area is obtained, Gaussian filtering and correction are used to reduce errors and noise, and the result is converted into an actual depth value to obtain the actual tree height and breast diameter parameters of the target single tree. The region of interest (ROI) obtained after image segmentation is represented as (x, y, w, h), where x, y are pixel coordinates, w, h are the width and length of the region, and in the region of interest, the grayscale value generally reflects the brightness of each point in the image. The depth image is smoothed using the grayscale value of the single tree area using a Gaussian filter Gf to reduce noise. The filtered depth image is represented as D'=D×Gf, where D is the original depth image and × represents a convolution operation. Image correction uses Gaussian filtering to reduce errors and noise, and uses a correction function C to complete depth image correction, represented as D'=C(D'), where D' is the corrected depth image. The actual depth value of the image is obtained by introducing a scaling factor α, where α represents the ratio of the actual distance to the pixel height, which can be expressed as D real =α×D". The tree height parameter is calculated by calculating the depth difference between the highest point and the lowest point. The tree height h can be expressed as h=D high -D low The depth of the highest point is D high , the depth of the lowest point is D low .

[0143] Furthermore, it also includes labeling the wide-angle RGB image of the forest, and the specific steps are as follows:

[0144] S1a, obtaining spectral information of the wide-angle RGB image of the forest, clustering the spectral information of the wide-angle RGB image of the forest using a spectral clustering algorithm, and obtaining the spectral category of each pixel;

[0145] S1b, using the region growing algorithm, the pixels with the same spectral category are merged to obtain the spectral region of each single tree;

[0146] S1c, using a spectral feature extraction algorithm to extract features from the spectral region of each single tree, and obtaining a spectral feature vector of each single tree;

[0147] S1d, using spectral classification algorithm to classify the spectral feature vector of each single tree to obtain the tree species label of each single tree;

[0148] S1e, annotate the tree species label of each single tree in its corresponding wide-angle RGB image of the forest.

[0149] In this embodiment, first, the wide-angle RGB image of the forest is converted into a spectral image, that is, each pixel contains reflectance or radiance information of multiple wavelengths, and the spectral image is reconstructed from the RGB image by using a spectral reconstruction algorithm. Then, for each pixel of the spectral image, a spectral clustering algorithm is used to classify them into different spectral categories, such as vegetation, soil, water, etc., based on the spectral similarity of the pixels. This not only improves the spectral resolution of the image and the richness of spectral information, but also provides a basis for subsequent feature extraction and classification.

[0150] Then, the pixels with the same spectral category are combined into connected regions, that is, each region represents the spectral information of a single tree. By using the region growing algorithm, starting from one or more seed pixels, adjacent pixels of the same spectral category are merged into the same region according to the spectral category and spatial neighborhood of the pixels, until no new pixels can be added, forming a spectral region of a single tree. The spatial information of the spectral image is utilized to improve the spatial resolution of the image and the continuity of spatial information, providing a basis for subsequent feature extraction and classification.

[0151] The spectral region of each single tree is converted into a spectral feature vector, that is, each single tree is represented by a one-dimensional numerical vector. By using the spectral feature extraction algorithm, the statistical characteristics of the spectrum of each single tree are calculated, such as mean, variance, peak, valley, slope, curvature, etc., or the physical characteristics of the spectrum are calculated, such as vegetation index, moisture index, mineral index, etc., or the two are combined to form a spectral feature vector. In this way, the amount of information in the spectral region of each single tree can be compressed, reducing the dimension and complexity of the data, while retaining the important characteristics of the spectrum, providing an effective feature representation for subsequent classification.

[0152] The spectral feature vector of each single tree is assigned to a predefined tree species category, that is, each single tree is represented by a label to represent its tree species information. By using the spectral classification algorithm, the tree species category to which the spectral feature vector of each single tree belongs is determined based on its similarity or difference with the spectral feature vectors of different tree species, or based on its output in a pre-trained classification model, such as pine, willow, birch, etc. The information content of the spectral feature vector of each single tree is converted into tree species information, realizing the semantic interpretation and classification of the wide-angle RGB image of the forest, providing a basis for subsequent annotation.

[0153] Finally, the tree species label of each single tree is visualized on its corresponding wide-angle RGB image of the forest, that is, different colors or texts are used to represent the tree species information at the boundary of the spectral region of each single tree. By using the image annotation algorithm, for each spectral region of a single tree, according to its tree species label, a suitable color or text is selected and drawn on its boundary to form a wide-angle RGB image of the forest with tree species annotations. Presenting the tree species information of each single tree on the wide-angle RGB image of the forest enhances the readability and interpretability of the image, and provides intuitive results for subsequent analysis and application.

[0154] Furthermore, step S1 may also be:

[0155] S1a, using binocular fisheye camera and lidar camera, collect wide-angle RGBD image data of the target forest plot, including two wide-angle RGB images, depth image and point cloud data;

[0156] S1b, using the wide-angle camera calibration plate and the point cloud calibration plate, respectively obtain the internal and external parameters of the binocular fisheye camera and the internal and external parameters of the laser radar;

[0157] S1c, based on the internal and external parameters of the binocular fisheye camera and the internal and external parameters of the laser radar, the wide-angle RGBD image data is corrected by using the joint distortion correction and registration equation;

[0158] S1d. Use labelme annotation software to manually annotate the corrected wide-angle RGBD image data. Based on the manually annotated wide-angle RGBD image data, generate a self-supervised pre-training dataset and a supervised fine-tuning dataset.

[0159] In this implementation, a binocular fisheye camera is used to obtain two wide-angle RGB images IL and IR, and a lidar camera obtains the corresponding depth image ID and point cloud data PC. A wide-angle camera calibration plate and a point cloud calibration plate are used to obtain the internal and external parameters of the wide-angle lens and the internal and external parameters of the lidar, respectively. The wide-angle RGB image IL, the wide-angle RGB image IR, the depth image ID and the point cloud data PC are input into the joint distortion correction and registration equation to obtain the corrected images IL', IR', ID' and PC'. The joint correction equation is to unify the radial distortion, tangential distortion and depth distortion into an optimization framework for solution, and the registration equation is Use the ICP algorithm and feature matching algorithm to align the image and point cloud data to the same coordinate system; use the labelme annotation software to manually annotate the corrected images IL' and IR' of the training set and test set to obtain the label file in json format, and use the labelme_json_to_dataset command to convert the json file into the annotation result image ILlabel and the annotation result image IRlabel in jpg format. According to the corrected images IL', IR', ID' and PC', labels such as image transformation, spatial position relationship, and perspective consistency for self-supervised learning are automatically generated.

[0160] Furthermore, step S2 further includes:

[0161] S2a, build the YOLOv8-Seg forest target detection model based on the forest wide-angle RGB image and the YOLOv8 Ultralytics open source framework;

[0162] S2b, replace the backbone network of the YOLOv8-Seg forest target detection model with a hybrid feature pyramid HFP structure, and perform multi-scale feature extraction on the preprocessed forest wide-angle RGB image; wherein the hybrid feature pyramid HFP structure includes a shallow feature branch, a deep feature branch, and a point cloud feature branch;

[0163] S2c, based on the extracted multi-scale features, generate hard-to-distinguish samples, introduce adversarial generation mechanism into the detection head of the YOLOv8-Seg forest target detection model, and perform enhanced training on hard-to-distinguish samples;

[0164] S2d, adopts the multi-view fusion detection mechanism MFDet, constructs a gated recurrent unit and a bidirectional attention mechanism, performs spatiotemporal fusion on the hard-to-separate samples after enhanced training, and generates a three-dimensional detection frame.

[0165] In this embodiment, the YOLOv8-Seg model is first built based on the YOLOv8 Ultralytics open source framework, and the backbone network is replaced with a hybrid feature pyramid HFP structure to achieve multi-scale feature extraction, wherein the hybrid feature pyramid contains three branches: a shallow feature branch, a deep feature branch, and a point cloud feature branch. The shallow branch adopts a parallel convolution structure similar to HRNet to extract high-resolution feature maps; the deep branch adopts a feature pyramid structure similar to FPN to fuse deep features layer by layer from top to bottom; the point cloud branch inputs the voxelized point cloud into a 3D convolutional neural network to extract the point cloud feature map; the feature maps of the three branches are adaptively fused at each scale layer through the attention mechanism.

[0166] An adversarial generation mechanism is introduced in the detection head to synthesize difficult examples for enhanced training. Specifically, a discriminator D is connected in parallel to the detection head of YOLOv8 to distinguish the true or false of the original image and its detection results. A generator G is connected in parallel after the backbone network to generate realistic synthetic images based on the random noise vector z. The generator G learns to synthesize difficult samples based on the multi-scale features extracted by the backbone network. The discriminator D and the generator G iteratively optimize their respective objective functions through adversarial learning to improve the robustness of the detector.

[0167] The new multi-view fusion detection mechanism MFDet is adopted to make full use of the spatiotemporal information of the sequence frames. The multi-view fusion detection mechanism constructs a gated recurrent unit and a bidirectional attention mechanism in the temporal dimension and the spatial dimension respectively. In the temporal dimension, the detection results of the previous and next frames are fused through GRU to mine the motion information of the target; in the spatial dimension, the feature maps of the left and right views are complemented through bidirectional attention to obtain the stereoscopic information of the target; the feature maps after spatiotemporal fusion are sent to 3D convolution and 3D NMS for final detection to generate a stable and accurate three-dimensional detection frame.

[0168] Furthermore, step S3 may also be:

[0169] S3a, based on the self-supervised pre-training dataset and the supervised fine-tuning dataset, a two-stage training strategy is adopted to train the YOLOv8-Seg forest target detection model;

[0170] S3b. Use contrastive learning loss and depth estimation loss to optimize the trained YOLOv8-Seg forest target detection model.

[0171] In this embodiment, a two-stage training strategy is adopted, first self-supervised pre-training and then supervised fine-tuning. The first stage is to use automatically generated labels to perform self-supervised pre-training on the YOLOv8-Seg forest target detection model to learn general visual feature representations; the second stage is to use manually annotated data sets to supervise fine-tune the pre-trained model in the first stage to learn target detection and instance segmentation tasks. Contrastive learning loss and depth estimation loss are added during the training process to improve feature discrimination and depth perception. The contrastive learning loss minimizes the feature distance between samples of the same category and maximizes the feature distance between samples of different categories; the depth estimation loss minimizes the difference between the estimated depth map and the true depth map, and penalizes depth prediction errors. The trained model is used to perform joint inference on the test set images to obtain the three-dimensional segmentation results of each single tree.

[0172] Furthermore, it also includes extracting the three-dimensional point cloud of the target single tree after segmentation, obtaining the normal vector and curvature of the single tree area, dividing the point cloud into different single tree point clouds according to the single tree segmentation results, estimating the normal vector and calculating the curvature for each single tree point cloud, and obtaining the features reflecting the trunk morphology and texture. The single tree skeleton curve is fitted by point cloud matching and triangulation joint optimization, SIFT feature points are extracted from the single tree point cloud, and matched between the left and right views. The single tree skeleton curve is triangulated and solved according to the matching point pairs and camera parameters, and the triangulated skeleton point cloud is ICP matched with the original single tree point cloud, and the skeleton curve equation is jointly optimized. Parameters such as tree height, diameter at breast height, and height under branches are extracted on the fitted skeleton curve, and the three-dimensional coordinates of the top and bottom points of the trunk are extracted on the skeleton curve to calculate the tree height. The point set on the diameter at breast height circle at 1.3 meters on the skeleton curve is extracted to fit the diameter at breast height, and the three-dimensional coordinates are extracted at the intersection of the trunk curve and the first-level branches to calculate the height under branches. Finally, combined with the results of tree species identification, the estimated values ​​of tree height and breast diameter are optimized using the heterogeneous forest resource database. The tree species in the target area are predicted through image classification or point cloud semantic segmentation. The tree height-breast diameter allometric growth equation and tree height-volume equation are retrieved from the database according to the tree species. The estimated values ​​of tree height and breast diameter are substituted into the growth equation for recursive correction to obtain predicted values ​​that are more in line with biological laws.

[0173] Finally, the trained YOLOv8-Seg model is encapsulated as a gRPC microservice and deployed to a distributed cluster. High-precision GNSS and VR devices are used to collect panoramic data of the forest area in real time and transmitted to the edge server through the 5G network; the server calls the gRPC interface to process the panoramic data, extract the three-dimensional structural parameters of individual trees and visualize them; the parameters of individual trees are integrated with information such as tree species, location, and growth status into a database and applied to the smart forestry decision-making system.

[0174] Furthermore, an understory wide-angle single tree image segmentation system based on improved YOLOv8-Seg includes a data acquisition module, a model building module, a model training module and an image segmentation module.

[0175] The specific implementation methods of the data acquisition module, the model construction module, the model training module and the image segmentation module are the same as the contents of steps 1 to 4 in the signing method, and will not be repeated in the present invention.

[0176] In a further embodiment, Figure 6 As shown in the figure, the understory wide-angle single tree image segmentation method based on improved YOLOv8-Seg includes:

[0177] Step 1: Use a binocular fisheye camera to collect wide-angle depth image data of the target forest plot, and make the obtained forest wide-angle RGB images into training sets and test sets;

[0178] Step 2: Build a YOLOv8-Seg tree target detection model. Based on the idea of ​​task partitioning, replace the convolution (Conv) module in the backbone network with a block convolution module, improve the C2f module into an adaptive weight module to complete the feature extraction task, and use a new cross fusion module to complete the feature fusion task.

[0179] Step 3: Use the forest wide-angle RGB image training set to complete the weight parameter training of the improved model YOLOv8-Seg model, and use the model to complete the single tree image segmentation task;

[0180] Step 4: Extract the depth information of the target single tree after segmentation, obtain the grayscale value of the single tree area, use Gaussian filtering and correction to reduce errors and noise, convert the results into actual depth values, and obtain the actual tree height and breast diameter parameters of the target single tree;

[0181] Step 5: Deploy the trained YOLOv8-Seg wide-angle forest image segmentation model in the forest information management system. The system accepts the image data collected by the camera in real time to automatically complete the target detection task.

[0182] In this embodiment, the convolution module in the backbone network is replaced with an improved block convolution module, which blocks the input x, separates the feature extraction from the downsampling task, and expands the receptive field while retaining the downsampling function, thereby enhancing the feature extraction capability of the network. The convolution module is used to fuse high-level features with low-level features, adopts the idea of ​​weight adaptation in model improvement, and introduces an attention mechanism so that the network parameters can adaptively adjust the structure of the feature map, and allocate corresponding output weights according to the size of the model channel contribution, and finally obtain the weighted output of the left and right channels. This module can pay more attention to important feature areas and enrich the acquired semantic information. The cross-fusion module changes the splicing method of the three-layer feature maps output in the backbone network, separates the upsampling and downsampling tasks in the model, and makes the feature fusion process non-interfering and clearer, so as to maximize efficiency.

[0183] The present invention mainly aims at the problem of low accuracy of single tree image segmentation under complex forest background during forest inspection, and provides a wide-angle single tree image segmentation method under the forest based on improved YOLOv8-Seg. This method completes the feature extraction of the network by replacing the block convolution module and the adaptive weight module on the basis of the original model, and uses a new cross-fusion module to complete the feature fusion, thereby enhancing the segmentation ability of the model and improving the accuracy of single tree image segmentation under the forest under complex forest background, while solving the problem that drones or remote sensing technology cannot accurately obtain forest information. The wide-angle lens adopted in hardware has better adaptability in extracting information from tall trees in the forest, and provides data support for forest managers and researchers to obtain detailed structural information of trees, calculate forest volume, and reconstruct three-dimensional models of trees. The present invention can be applied to fields such as forestry monitoring and intelligent inspection robots.

[0184] like Figures 7 to 10 is the indicator result of target detection, Figures 11 to 14 is the index result after mask processing. Figure 7 In , the F1 score for all categories is 0.64 at a confidence level of 0.148; Figure 8 In , the F1 score of all classes is 1.00 at a confidence level of 0.866; Fig. 9 In , the recall-precision of the target tree is 0.632, and the average precision of all categories is 0.632 at a threshold of 0.5; Fig.10 In , the F1 score of all categories is 0.66 when the confidence level is 0.00; Fig.11 In , the F1 score for all categories is 0.66 at a confidence level of 0.154; Fig.12 In , the F1 score of all classes is 1.00 at a confidence level of 0.866; Fig.13In , the recall-precision of the target tree is 0.636, and the average precision of all categories is 0.636 at a threshold of 0.5; Fig.14 The F1 score for all classes at a confidence level of 0.00 is 0.65.

[0185] The image segmentation task based on YOLOv8-Seg goes through two stages: target detection and instance segmentation. In this paper, the precision, recall and average precision mAP of the target detection box and instance segmentation result (mask) are measured. 50 , mAP 50-95 The recognition accuracy of the model is evaluated by the index, where the subscript 50 refers to the average accuracy when the intersection over union (IOU) ratio is 0.5. The details are as follows:

[0186] Before improvement, the accuracy of the target detection box is 0.769, the recall rate is 0.433, the average precision is 0.572, and the recognition accuracy is 0.252; after improvement, the accuracy of the target detection box is 0.789, the recall rate is 0.546, the average precision is 0.632, and the recognition accuracy is 0.277; before improvement, the accuracy of the instance segmentation result is 0.774, the recall rate is 0.498, the average precision is 0.572, and the recognition accuracy is 0.361; after improvement, the accuracy of the instance segmentation result is 0.833, the recall rate is 0.550, the average precision is 0.636, and the recognition accuracy is 0.393.

[0187] The results show that the precision, recall and average precision of the improved model are all improved compared with those before the improvement. The precision of the model has increased from 0.774 to 0.833, an increase of 7.62%; the recall has increased from 0.498 to 0.550, an increase of 10.44%; the average precision mAP ...10.44%; the average precision mAP has increased from 0. 50 It increased from 0.572 before the improvement to 0.636, an increase of 11.19%.

[0188] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. A method for segmenting wide-angle single tree images under forest based on improved YOLOv8-Seg, characterized in that: The steps include: S1. Use a binocular fisheye camera to collect wide-angle depth image data of the target forest plot, obtain wide-angle RGB images of trees, and divide them into training sets and test sets; S2. Based on the wide-angle RGB image of trees, a YOLOv8-Seg tree target detection model is constructed; the YOLOv8-Seg tree target detection model includes a block convolution module, an adaptive weight module, and a cross fusion module; S3, based on the training set and test set of the wide-angle RGB images of the trees, train the YOLOv8-Seg tree target detection model; S4. Use the trained YOLOv8-Seg tree target detection model to extract the depth information of the single tree image. Based on the depth information of the single tree image, calculate the actual tree height and breast diameter parameters of the single tree, specifically: S41. Use the trained YOLOv8-Seg forest target detection model to determine the pixel area of ​​each tree, extract the depth values ​​of all pixels in the pixel area, convert the depth values ​​into grayscale values, and obtain a grayscale image of the single tree area; S42, using a Gaussian filter to smooth and reduce noise on the grayscale image of a single tree area; S43, using a correction method to perform depth value correction on the single tree region grayscale image after smoothing and noise reduction; S44, based on the grayscale value, converting the single tree region grayscale image after the depth value correction into a real single tree depth image; S45. Based on the real single tree depth map, calculate the actual tree height and breast diameter parameters of the single tree.

2. The method for segmenting wide-angle single tree images under forest based on improved YOLOv8-Seg according to claim 1, characterized in that: Step S1 is specifically as follows: S11, installing a binocular fisheye camera on a mobile platform, and capturing wide-angle depth image data of the target forest plot at a predetermined interval and angle, including two wide-angle RGB images on the left and right and corresponding depth images; S12, using a fisheye distortion correction algorithm to perform distortion correction on the wide-angle depth image data to obtain a wide-angle RGB image of the forest; S13, preprocessing the forest wide-angle RGB image, including removing noise, enhancing brightness, adjusting contrast and scaling; S14. Using a data partitioning algorithm, the preprocessed forest wide-angle RGB image is divided into a training set and a test set according to a predetermined ratio.

3. The understory wide-angle single tree image segmentation method based on improved YOLOv8-Seg according to claim 2, characterized in that: Step S2 is specifically as follows: S21. Build the YOLOv8-Seg tree target detection model based on the wide-angle RGB image of the tree and the YOLOv8 Ultralytics open source framework. S22, using a block convolution module to extract features from the preprocessed forest wide-angle RGB image to obtain a multi-scale feature map; S23, using an adaptive weight module to adaptively adjust multi-scale feature maps through weight adaptation and attention mechanism; S24, using a cross fusion module to fuse the adaptively adjusted feature maps; S25, input the fused feature map into the head network to predict the boundary and category of the feature map; S26, inputting the fused feature map and the predicted boundaries and categories into the segmentation network to predict the segmentation mask and contour of the single tree image; S27. Construct the loss function of YOLOv8-Seg based on the segmentation mask and contour of the single tree image.

4. The understory wide-angle single tree image segmentation method based on improved YOLOv8-Seg according to claim 3 is characterized in that: Step S3 is specifically as follows: S31. Use data enhancement algorithm to preprocess the wide-angle RGB images of trees in the training set, including rotation, translation, scaling, cropping, flipping and color transformation; S32. Based on the preprocessed training set, the optimization algorithm is used to update the parameters of the YOLOv8-Seg forest target detection model to minimize the loss function. S33. Use the YOLOv8-Seg tree target detection model with updated parameters to infer the wide-angle RGB image of the trees in the test set to obtain the segmentation results and detection box coordinates of each single tree. S34, based on the segmentation results and detection box coordinates of each single tree, the YOLOv8-Seg tree target detection model with updated parameters is evaluated to obtain an evaluation index; S35. Based on the evaluation index, use the early stopping method to determine whether to continue updating the parameters of the YOLOv8-Seg forest target detection model. If yes, return to step S31. Otherwise, output the trained YOLOv8-Seg forest target detection model.

5. The method for segmenting wide-angle single tree images under forest based on improved YOLOv8-Seg according to claim 1, characterized in that: It also includes labeling of wide-angle RGB images of forest trees. The specific steps are as follows: S12a, obtaining spectral information of the wide-angle RGB image of the forest, and clustering the spectral information of the wide-angle RGB image of the forest using a spectral clustering algorithm to obtain a spectral category of each pixel; S12b, using the region growing algorithm to merge the pixels with the same spectral category to obtain the spectral region of each single tree; S12c, using a spectral feature extraction algorithm to extract features from the spectral region of each single tree, and obtaining a spectral feature vector of each single tree; S12d, using a spectral classification algorithm to classify the spectral feature vector of each single tree to obtain a tree species label of each single tree; S12e, annotate the tree species label of each single tree in its corresponding wide-angle RGB image of the forest.

6. The method for segmenting wide-angle single tree images under forest based on improved YOLOv8-Seg according to claim 1, characterized in that: Step S1 is specifically as follows: S1a, using binocular fisheye camera and lidar camera, collect wide-angle RGBD image data of the target forest plot, including two wide-angle RGB images, depth image and point cloud data; S1b, using the wide-angle camera calibration plate and the point cloud calibration plate, respectively obtain the internal and external parameters of the binocular fisheye camera and the internal and external parameters of the laser radar; S1c, based on the internal and external parameters of the binocular fisheye camera and the internal and external parameters of the laser radar, the wide-angle RGBD image data is corrected by using the joint distortion correction and registration equation; S1d. Use labelme annotation software to manually annotate the corrected wide-angle RGBD image data. Based on the manually annotated wide-angle RGBD image data, generate a self-supervised pre-training dataset and a supervised fine-tuning dataset.

7. The method for segmenting a wide-angle single tree image under the forest based on improved YOLOv8-Seg according to claim 6, characterized in that: Step S2 is specifically as follows: S2a, build the YOLOv8-Seg forest target detection model based on the forest wide-angle RGB image and the YOLOv8 Ultralytics open source framework; S2b, replace the backbone network of the YOLOv8-Seg forest target detection model with the hybrid feature pyramid HFP structure, and perform multi-scale feature extraction on the preprocessed forest wide-angle RGB image; The hybrid feature pyramid HFP structure includes shallow feature branches, deep feature branches and point cloud feature branches; S2c, based on the extracted multi-scale features, generate hard-to-distinguish samples, introduce adversarial generation mechanism into the detection head of the YOLOv8-Seg forest target detection model, and perform enhanced training on hard-to-distinguish samples; S2d, adopts the multi-view fusion detection mechanism MFDet, constructs a gated recurrent unit and a bidirectional attention mechanism, performs spatiotemporal fusion on the hard-to-separate samples after enhanced training, and generates a three-dimensional detection frame.

8. The understory wide-angle single tree image segmentation method based on improved YOLOv8-Seg according to claim 7, characterized in that: Step S3 is specifically as follows: S3a, based on the self-supervised pre-training dataset and the supervised fine-tuning dataset, a two-stage training strategy is adopted to train the YOLOv8-Seg forest target detection model; S3b. Use contrastive learning loss and depth estimation loss to optimize the trained YOLOv8-Seg forest target detection model.

9. A forest wide-angle single tree image segmentation system based on improved YOLOv8-Seg, characterized in that: The method for segmenting a wide-angle single tree image under the forest based on the improved YOLOv8-Seg according to any one of claims 1 to 8 comprises: Data acquisition module: Use a binocular fisheye camera to collect wide-angle depth image data of the target forest plot, obtain wide-angle RGB images of trees, and divide them into training sets and test sets; Model building module: Based on the wide-angle RGB image of trees, a YOLOv8-Seg tree target detection model is built; the YOLOv8-Seg tree target detection model includes a block convolution module, an adaptive weight module, and a cross fusion module; Model training module: Train the YOLOv8-Seg forest target detection model based on the training set and test set of forest wide-angle RGB images; Image segmentation module: Use the trained YOLOv8-Seg tree target detection model to extract the depth information of a single tree image. Based on the depth information of the single tree image, calculate the actual tree height and breast diameter parameters of the single tree. Specifically: Use the trained YOLOv8-Seg tree target detection model to determine the pixel area of ​​each tree, extract the depth values ​​of all pixels in the pixel area, convert the depth values ​​to grayscale values, and obtain the grayscale image of the single tree area; A Gaussian filter was used to smooth and reduce noise in the grayscale image of a single tree area; The correction method is used to correct the depth value of the single tree area grayscale image after smoothing and noise reduction; Based on the grayscale value, the single tree area grayscale image after depth value correction is converted into a real single tree depth image; Based on the real single tree depth map, the actual tree height and breast diameter parameters of the single tree are calculated.

Images