Noisy Plant Point Cloud Semantic Segmentation Method and System Based on Self-Attention Feature Fusion

Abstract

The present invention relates to the technical field, and discloses a method and system for semantic segmentation of noisy plant point clouds based on self-attention feature fusion, which includes the following steps: S1: Obtain two-dimensional sequence images; S2: Perform three-dimensional reconstruction based on the two-dimensional sequence images to obtain plant three-dimensional point clouds; S3: Preprocess and manually annotate the plant three-dimensional point clouds to obtain a plant point cloud dataset; S4: Construct a semantic segmentation network for noisy plant point clouds based on self-attention feature fusion, and train the semantic segmentation network for noisy plant point clouds through the plant point cloud dataset; S5: Perform semantic segmentation on the plant point clouds through the trained semantic segmentation network for noisy plant point clouds to obtain the predicted semantic labels of each point. The present invention solves the problem that the prior art cannot segment plant scenes with noisy points, and has the characteristics of anti-noise, high throughput, and high precision.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional point cloud segmentation of plants, and more specifically, to a method and system for semantic segmentation of noisy plant point clouds based on self-attention feature fusion. Background Art

[0002] In recent years, the automated measurement and monitoring technology of plant phenotypic parameters has been widely applied to modern agriculture. Plant point cloud semantic segmentation is a key technology for measuring plant phenotypic data. By segmenting leaves, stems and other point cloud semantic labels, it can provide data support for subsequent acquisition of various plant parameters, monitoring of plant growth conditions and adjustment of subsequent cultivation plans. However, due to the influence of environmental light and the limitation of the camera's field of view, there are many noise points between the stems and leaves of the plant point cloud obtained by three-dimensional reconstruction, which are continuously distributed together, making the traditional stem-leaf segmentation method inapplicable. At present, the traditional methods for three-dimensional point cloud organ segmentation of plants include segmentation methods based on region growing and Euclidean distance clustering. The segmentation method based on region growing uses neighborhood information to classify nearby points with similar attributes to obtain segmentation regions and distinguish the differences between different regions. However, this method has problems in accurately determining the region boundary and is very sensitive to noise points. The method based on Euclidean distance clustering usually requires manually setting a segmentation threshold, and the accuracy of the segmentation threshold has a great impact on the segmentation result. This method is also very sensitive to noise points.

[0003] Deep learning technology has been widely applied in the agricultural field with its excellent data representation learning ability and strong generality. According to different processing methods, the 3D point cloud semantic segmentation methods based on deep learning can be divided into projection-based, discretization-based and directly point-based methods. The projection-based method projects the 3D point cloud onto multiple 2D views, and uses advanced image segmentation algorithms to segment and then fuse the scores of different views to draw a conclusion. However, this method is sensitive to view selection and occlusion, and the projection step is prone to information loss. The discretization-based method converts the unordered point cloud into a normalized discrete representation and uses 3D convolution for voxel segmentation. This method preserves the neighborhood structure of the 3D point cloud, but its complex data structure leads to a large memory calculation cost. The directly point-based method takes the original point cloud data as input and fully preserves the spatial position and multi-dimensional feature information of the point cloud. Although the development of point-based deep learning is rapid, such as PointNet, PointNet++, DGCNN, PointWeb, etc., there are few studies on plant segmentation.

[0004] There is currently a method and system for three-dimensional point cloud segmentation of plants. The point cloud used is a clean point cloud without noise points. Although it can achieve three-dimensional point cloud segmentation of organs such as stems, leaves, male spikes, and female spikes in plant plants, it can also perform high-precision segmentation even when the organs are very close or even touching each other.

[0005] However, the existing technology has the problem of being unable to segment plant scenes with noise points. Therefore, how to provide a method for segmenting plant point clouds in plant scenes with noise points has become an urgent problem for those skilled in the art to solve. Summary of the Invention

[0006] In order to solve the problem that the existing technology cannot segment plant scenes with noise points, the present invention provides a method and system for semantic segmentation of noisy plant point clouds based on self-attention feature fusion, which has the characteristics of anti-noise, high throughput, and high precision.

[0007] In order to achieve the above object of the present invention, the following technical solutions are adopted:

[0008] A method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion includes the following steps:

[0009] S1: Obtain two-dimensional sequence images;

[0010] S2: Perform three-dimensional reconstruction based on the two-dimensional sequence images to obtain plant three-dimensional point clouds;

[0011] S3: Preprocess and manually annotate the plant three-dimensional point clouds to obtain a plant point cloud dataset;

[0012] S4: Construct a semantic segmentation network for noisy plant point clouds based on self-attention feature fusion, and train the semantic segmentation network for noisy plant point clouds through the plant point cloud dataset;

[0013] S5. Perform semantic segmentation on the plant point clouds through the trained semantic segmentation network for noisy plant point clouds to obtain the predicted semantic labels of each point.

[0014] Preferably, in step S2, when performing three-dimensional reconstruction based on the two-dimensional sequence images to obtain plant three-dimensional point clouds, the specific steps are as follows:

[0015] S201. Use the scale-invariant feature transform algorithm to find extreme points in the image scale space of the two-dimensional sequence images, detect potential feature points, and select the most robust points as the final feature points;

[0016] S202. Introduce the k-dimensional space binary tree kd-tree model. Calculate the Euclidean distance between the final feature points of any pair of images in the two-dimensional sequence images through the kd-tree model to obtain their matching degree. Determine the matching points according to the level of the matching degree, and calculate the corresponding relationship between the final feature points of the image pair and the camera pose in the three-dimensional space;

[0017] S203. Solve the camera pose of the image pair and its final feature point pair through the triangulation principle to obtain the three-dimensional coordinates, thereby generating a three-dimensional sparse point cloud;

[0018] S204. Cluster the two-dimensional sequence images using the multi-view stereo clustering algorithm, and then use the multi-view stitching algorithm to expand the sparse point cloud to obtain a dense point cloud as the final three-dimensional point cloud of the plant.

[0019] Further, in step S3, preprocess the three-dimensional point cloud of the plant, specifically: perform point cloud downsampling, color threshold segmentation, and noise point removal on the three-dimensional point cloud of the plant in sequence:

[0020] When performing point cloud downsampling on the three-dimensional point cloud of the plant, use the three-dimensional voxel grid method to create a three-dimensional voxel grid through the input point cloud data; within each voxel of the three-dimensional voxel grid, approximate the other points in the voxel with the center of gravity of all the points in the voxel, represent all the points in the voxel with a center of gravity point, and after processing all the voxel blocks, obtain the filtered point cloud to complete the downsampling;

[0021] For the three-dimensional point cloud of the plant that has completed point cloud downsampling, manually adjust the appropriate parameter threshold according to the different RGB values of the point cloud colors, retain the plants in the three-dimensional point cloud of the plant, and filter the background and soil in the three-dimensional point cloud of the plant;

[0022] For the three-dimensional point cloud of the plant that has completed color threshold segmentation, use the statistical outlier removal algorithm to delete the points with too large a distance from the adjacent points in the three-dimensional point cloud of the plant to obtain the three-dimensional point cloud of the plant that has completed preprocessing.

[0023] Even further, use the statistical outlier removal algorithm to delete the points with too large a distance from the adjacent points in the three-dimensional point cloud of the plant, specifically:

[0024] Traverse all the points in the three-dimensional point cloud of the plant, and calculate the average distance d of each point from its nearest K neighbors ij ;

[0025] Calculate the mean μ and standard deviation σ of all these point distances to determine the distance threshold;

[0026] Manually set the number of neighbor points, specifying the number of adjacent elements to be considered when calculating the average distance of a given point; manually set the standard deviation, specifying the threshold level for the standard deviation of the average distance in the point cloud;

[0027] Define points with a distance between two points exceeding the threshold as noise points, and remove the noise points through the noise point removal formula:

[0028]

[0029]

[0030] Furthermore, in step S3, the manual annotation is specifically as follows:

[0031] For the noise points existing in the three-dimensional point cloud of the preprocessed plants and continuously distributed with the leaves and stems, manually segment different semantic segmentation label categories through cloudcompare software, label each point, and obtain the plant point cloud dataset.

[0032] Furthermore, the noisy plant point cloud semantic segmentation network adopts an encoder-decoder structure, including 4 self-attention feature fusion modules, 2 shared fully connected layers, and 3 upsampling layers;

[0033] The noisy plant point cloud semantic segmentation network obtains multi-level encoded features through 4 self-attention feature fusion modules; the obtained features are decoded layer by layer through 3 upsampling layers. Each upsampling layer upsamples the features in the input point cloud dataset points through the nearest neighbor interpolation algorithm, and uses skip connections to splice with the intermediate features of the corresponding dimension encoding layer; the 2 shared fully connected layers are the dropout layer and the Log-SoftMax layer respectively; the spliced point cloud passes through the dropout layer and the Log-SoftMax layer in sequence to obtain the predicted semantics of all points in the point cloud dataset.

[0034] Furthermore, the self-attention feature fusion module includes a downsampling module, a local feature fusion module, and a dual-branch attention pooling module; the downsampling module, the local feature fusion module, and the dual-branch attention pooling module are connected in sequence;

[0035] The downsampling module downsamples the point cloud in the input point cloud dataset points, reduces the number of point clouds while increasing the dimension of the features of each point. Specifically, it uses farthest point sampling to continuously iteratively select the farthest point from the existing sampling point set to obtain the sampled point cloud; for each point in it, use the KNN algorithm to find the K nearest points to the center point, generate a local neighborhood, and each local neighborhood passes through a multi-layer perceptron, and finally perform max pooling to obtain the output of downsampling;

[0036] The local feature fusion module aggregates the shallow features including the correlation between all point pairs within the neighborhood, the position and distance information of the points, and the high-level features including shape and semantic information to obtain local context features;

[0037] The first branch of the dual-branch attention pooling module uses a channel attention mechanism to adaptively filter out the locally low-correlation features of the local context features, which is used to guide the network to focus on the shape and structure information of the plant when extracting the semantic segmentation features of the plant point cloud; the second branch obtains the global context features by performing max pooling on the highest-level feature map of the local context features; finally, the local and global context features are fused to obtain multi-level encoded features.

[0038] Furthermore, the input scale of the local feature fusion module is N×(3 + d), where N represents the number of points, 3 represents the position coordinates in three-dimensional space, and d represents the feature attributes; the KNN algorithm is used once for each point to obtain the K points with the closest Euclidean distance; for the i-th point in the input, assuming its coordinate is p i , and the coordinates of the K neighbors are respectively The coordinates of p i and its neighbor points are concatenated with the center point coordinates, neighbor point coordinates, coordinate differences, and the distance between the center point and the neighbor points through local spatial encoding; a new feature that aggregates the Euclidean distance space information is obtained through adjustment by a shared perceptron

[0039]

[0040] In the formula, ⊕ represents the concatenation operation, and ‖·‖ calculates the distance from the center point to all neighbor points.

[0041] Furthermore, the dual-branch attention pooling module includes 4 inputs, namely geometric distance, adjacent point features, coordinate differences, and local aggregation features. Among them, the coordinate difference is the vector obtained by subtracting the adjacent point coordinates from the center point coordinates, and the local aggregation feature is the output of the local feature fusion module; assuming that g(i) and g(k) are two input feature vectors of this module, the feature distance between g(i) and g(k) is defined as:

[0042]

[0043] In the formula, |·| is to find the absolute value, and mean is the average value function; the parameter λ is used to control the weight of the feature distance, and λ is set to 0.1;

[0044]

[0045] In the formula, represents the geometric distance, represents the coordinate difference between the center point and the neighbor points; for the first branch of the dual-branch attention pooling module, the feature influence factor and the local aggregation feature Merge:

[0046]

[0047] f i k+ The attention weights are calculated through a linear layer and a Softmax function

[0048]

[0049] The local context features can be obtained by weighted summing the local aggregation features using the learned attention weights:

[0050]

[0051] For the second branch, the formula for the global context features is:

[0052]

[0053] A noisy plant point cloud semantic segmentation system based on self-attention feature fusion includes an image acquisition module, a 3D reconstruction module, a preprocessing module, a semantic segmentation network module, and a semantic segmentation module:

[0054] The described image acquisition module is used to obtain two-dimensional sequence images;

[0055] The described 3D reconstruction module is used to perform 3D reconstruction based on the two-dimensional sequence images to obtain the plant 3D point cloud;

[0056] The described preprocessing module is used to preprocess and manually annotate the plant 3D point cloud to obtain a plant point cloud dataset;

[0057] The described semantic segmentation network module is used to construct a noisy plant point cloud semantic segmentation network based on self-attention feature fusion, and train the described noisy plant point cloud semantic segmentation network through the plant point cloud dataset;

[0058] The described semantic segmentation module is used to perform semantic segmentation on the plant point cloud through the trained noisy plant point cloud semantic segmentation network to obtain the predicted semantic label for each point.

[0059] The beneficial effects of the present invention are as follows:

[0060] The present invention discloses a method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion. Specifically, it discloses three-dimensional reconstruction based on two-dimensional sequence images to obtain three-dimensional plant point clouds; preprocessing and manual annotation of the three-dimensional plant point clouds to obtain a plant point cloud dataset; and semantic segmentation of the plant point clouds through a trained semantic segmentation network for noisy plant point clouds to obtain the predicted semantic labels of each point. Thus, the present invention solves the problem in the prior art that plant scenes with noisy points cannot be segmented, and has the characteristics of anti-noise, high throughput, and high precision. Description of the Drawings

[0061] Figure 1 is a schematic flow chart of the method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion of the present invention.

[0062] Figure 2 is a specific schematic flow chart of step S2 in the method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion of the present invention.

[0063] Figure 3 is a diagram of the image acquisition platform built in step S1 in the method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion of the present invention.

[0064] Figure 4 is a schematic diagram of the manual annotation in step S3 in the method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion of the present invention.

[0065] Figure 5 is a schematic diagram of the structure of the semantic segmentation network for noisy plant point clouds in the method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion of the present invention.

[0066] Figure 6 is a diagram of the local feature aggregation module of the semantic segmentation network for noisy plant point clouds in the method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion of the present invention.

[0067] Figure 7 is a diagram of the dual-branch attention pooling module of the semantic segmentation network for noisy plant point clouds in the method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion of the present invention.

[0068] Figure 8 is a visualization diagram of the segmentation result of the three-dimensional plant point clouds in the method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion of the present invention. Detailed Embodiments

[0069] The following describes the present invention in detail with reference to the drawings and specific embodiments.

[0070] Embodiment 1

[0071] AsFigure 1 As shown in the figure, a method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion includes the following steps:

[0072] S1: Obtain two-dimensional sequence images;

[0073] S2: Perform three-dimensional reconstruction based on the two-dimensional sequence images to obtain plant three-dimensional point clouds;

[0074] S3: Preprocess and manually annotate the plant three-dimensional point clouds to obtain a plant point cloud dataset;

[0075] S4: Construct a semantic segmentation network for noisy plant point clouds based on self-attention feature fusion, and train the semantic segmentation network for noisy plant point clouds through the plant point cloud dataset;

[0076] S5. Perform semantic segmentation on the plant point clouds through the trained semantic segmentation network for noisy plant point clouds to obtain the predicted semantic labels of each point.

[0077] In this embodiment, as Figure 3 shown, build a multi-angle image acquisition platform for plants, and obtain high-precision two-dimensional sequence images through a high-definition camera; obtain plant images of two-dimensional sequence images through the plant image acquisition platform; the acquisition platform consists of a camera, a bracket and a circular turntable. The role of the bracket is to fix the camera. The circular turntable rotates so that the fixed camera can take photos of different angles of the plant. When the acquisition platform works, place the test plant at the center of the white turntable and let the plant rotate with the turntable. Select three identical cameras and aim them at the plant on the turntable and at different heights. One of them takes a top view photo of the plant, and the other two take 45° top view and side view photos. The bracket is placed within 0.5 m from the white circular turntable. Set the camera to the autofocus mode throughout the shooting process, and at the same time ensure that the camera parameters remain unchanged. Centered on the plant, take a two-dimensional image every 10 - 20 degrees of rotation. Each of the 3 cameras obtains 60 images from different perspectives, and a total of 180 images are collected for each plant sample. This solution builds a set of platforms for automatically collecting two-dimensional sequence images of plants, performs three-dimensional reconstruction with multi-angle sequence images, is less restricted by the environment and has self-calibration, has lower requirements for the camera. Different from a 3D camera like Kinect, only an ordinary RGB camera is needed, and it has strong robustness.

[0078] In this embodiment, the present invention constructs a semantic segmentation network, and thus proposes a data-driven end-to-end deep learning model in the agricultural field that directly processes 3D point clouds with a large amount of noise, and can automatically and efficiently perform organ-level segmentation on plant three-dimensional point clouds.

[0079] Embodiment 2

[0080] More specifically, as Figure 2 shown, in a specific embodiment, in step S2, three-dimensional reconstruction is performed based on the two-dimensional sequence images to obtain the three-dimensional point cloud of the plant. The specific steps are as follows:

[0081] S201. Use the Scale-Invariant Feature Transform (SIFT) algorithm to find the extreme points in the image scale space of the two-dimensional sequence images, detect the potential feature points, and select the most robust points as the final feature points;

[0082] S202. Introduce the k-dimensional space binary tree kd-tree model. Calculate the Euclidean distance between the final feature points of any two images in the two-dimensional sequence images through the kd-tree model to obtain their matching degree. Determine the matching points according to the high or low matching degree, and calculate the corresponding relationship between the final feature points of the image pair and the camera pose in the three-dimensional space;

[0083] S203. Solve the camera pose of the image pair and its final feature point pair through the triangulation principle to obtain the three-dimensional coordinates, thereby generating a three-dimensional sparse point cloud;

[0084] S204. Use the multi-view stereo clustering algorithm to cluster the two-dimensional sequence images, and then use the multi-view stitching algorithm to expand the sparse point cloud to obtain a dense point cloud as the final three-dimensional point cloud of the plant.

[0085] In this embodiment, when performing three-dimensional reconstruction on the plant point cloud, due to the influence of environmental light and the limitation of the camera's field of view, there are many noise points between the background, leaves, and stems of the plant point cloud. Especially under the leaves, which are the blind spots of the field of view, there are a large number of noise points continuously distributed together with the leaves and stems, causing a greater impact on the subsequent segmentation of the plant. Therefore, it is necessary to preprocess the original point cloud to improve the accuracy of point cloud segmentation.

[0086] In a specific embodiment, in step S3, the three-dimensional point cloud of the plant is preprocessed, specifically: performing point cloud downsampling, color threshold segmentation, and noise point removal on the three-dimensional point cloud of the plant in sequence:

[0087] When performing point cloud downsampling on the three-dimensional point cloud of the plant, use the three-dimensional voxel grid method to create a three-dimensional voxel grid through the input point cloud data; within each voxel of the three-dimensional voxel grid, approximate the other points in the voxel with the centroid of all the points in the voxel, represent all the points in the voxel with a centroid point, and after processing all the voxel blocks, obtain the filtered point cloud to complete the downsampling;

[0088] For the three-dimensional point cloud of the plant that has completed point cloud downsampling, manually adjust the appropriate parameter threshold according to the different RGB values of the point cloud colors, retain the plants in the three-dimensional point cloud of the plant, and filter the background and soil in the three-dimensional point cloud of the plant;

[0089] In this embodiment, according to the different RGB values of the point cloud colors, appropriate parameter thresholds are manually adjusted to retain green plants and filter out white backgrounds and black soils. The color threshold is specifically set as: G - B ≥ 45 and G - R ≥ 45. Color threshold segmentation can remove a large amount of background point clouds, but there are still many noise points similar to the plant colors, and there are also many outlier noise point clusters randomly distributed. Therefore, outlier removal is required.

[0090] For the three-dimensional point cloud of the plant after color threshold segmentation, the statistical outlier removal algorithm is used to delete the points with too large distances from the adjacent points in the three-dimensional point cloud of the plant, and the three-dimensional point cloud of the preprocessed plant is obtained.

[0091] In a specific embodiment, using the statistical outlier removal algorithm to delete the points with too large distances from the adjacent points in the three-dimensional point cloud of the plant is specifically as follows:

[0092] Traverse all points in the three-dimensional point cloud of the plant, and calculate the average distance d between each point and its nearest K neighbors ij ;

[0093] Calculate the mean μ and standard deviation σ of all these point distances to determine the distance threshold;

[0094] Manually set the number of neighbors, specifying the number of adjacent elements to be considered when calculating the average distance of a given point; manually set the standard deviation, specifying the threshold level of the standard deviation of the average distance in the point cloud;

[0095] Define that the distance between two points exceeding the threshold is a noise point, and remove the noise points through the noise point removal formula:

[0096]

[0097]

[0098] In a specific embodiment, in step S3, the manual annotation is specifically as follows:

[0099] For the noise points existing in the three-dimensional point cloud of the preprocessed plant and continuously distributed with the leaves and stems, different semantic segmentation label categories are manually segmented through cloudcompare software, and each point is labeled to obtain the plant point cloud dataset.

[0100] In this embodiment, as Figure 4 shown, the entire point cloud can be divided into leaf class, stem class, red paper class, and miscellaneous point class.

[0101] In this embodiment, since the subsequently constructed semantic segmentation network needs to be trained before it can be used, therefore, before actual segmentation, through preprocessing and manual annotation, a dataset for training the noisy plant point cloud semantic segmentation network based on self-attention feature fusion can be obtained. The dataset is divided into a training set and a test set according to a ratio of 5:1. The training set is used to iteratively train the semantic segmentation network, and the test set is used for accuracy testing to verify the semantic segmentation effect of the semantic segmentation network for point clouds with noise.

[0102] In a specific embodiment, the noisy plant point cloud semantic segmentation network adopts an encoder-decoder structure, including 4 self-attention feature fusion modules, 2 shared fully connected layers, and 3 upsampling layers;

[0103] The noisy plant point cloud semantic segmentation network obtains multi-level encoded features through 4 self-attention feature fusion modules; the obtained features are decoded layer by layer through 3 upsampling layers. Each upsampling layer upsamples the features in the input point cloud dataset points through the nearest neighbor interpolation algorithm, and uses skip connections to splice with the intermediate features of the corresponding dimension encoding layer; the 2 shared fully connected layers are a dropout layer and a Log-SoftMax layer respectively; the spliced point cloud passes through the dropout layer and the Log-SoftMax layer in sequence to obtain the predicted semantics of all points in the point cloud dataset.

[0104] In this embodiment, the input of the noisy plant point cloud semantic segmentation network is a point cloud of size N×d, where N represents the number of points and d represents the feature dimension of the input points. The number of points gradually decreases from N to N / 256, and the change process of the number of point cloud points is N→N / 4→N / 8→N / 64→N / 256. The feature dimension increases from d to 512, and the change process of the point cloud feature dimension is d→64→128→256→512.

[0105] In this embodiment, when upsampling is performed and skip connections are used to splice with the intermediate features of the corresponding dimension encoding layer, the number of points in the point cloud gradually returns to N.

[0106] In this embodiment, when passing through the 2 shared fully connected layers, the dimension change is (N,128)→(N,32)→(N,C). The random dropout rate of the dropout layer is set to 0.5, and the predicted semantics size of all points obtained by the Log-SoftMax layer is N×C, where C is the number of semantic label classes.

[0107] In a specific embodiment, the self-attention feature fusion module includes a downsampling module, a local feature fusion module, and a dual-branch attention pooling module; the downsampling module, the local feature fusion module, and the dual-branch attention pooling module are connected in sequence;

[0108] The downsampling module downsamples the point cloud in the input point cloud data set, reducing the number of points in the point cloud while dimensionality - increasing the features of each point. Specifically: Using farthest point sampling, iteratively select the farthest point from the existing set of sampled points to obtain the sampled point cloud; For each point among them, use the KNN algorithm to find the K points closest to the center point, generate a local neighborhood, and each local neighborhood passes through a multi - layer perceptron, and finally perform max - pooling to obtain the output of downsampling;

[0109] In this embodiment, as Figure 6 shown, in order to extract the local fine - grained features of the point cloud, a local feature fusion module that processes three - dimensional coordinates and point features separately is designed. The local feature fusion module aggregates the shallow features including the position and distance information of points and the high - level features including shape and semantic information according to the correlation between all point pairs within the neighborhood in the output of downsampling to obtain local context features;

[0110] The first branch of the dual - branch attention pooling module uses a channel attention mechanism to adaptively filter the locally low - correlated features of the local context features, which is used to guide the network to focus on the shape structure information of the plant when extracting the semantic segmentation features of the plant point cloud; The second branch obtains the global context features by performing max - pooling on the highest - level feature map of the local context features; Finally, the local and global context features are fused to obtain multi - level encoded features.

[0111] In a specific embodiment, from Figure 6 it can be seen that the input scale of the local feature fusion module is N×(3 + d), where N represents the number of points, 3 represents the position coordinates in three - dimensional space, and d represents the feature attribute; Use the KNN algorithm once for each point to obtain the K points with the closest Euclidean distance; For the i - th point in the input, assuming its coordinate is p i , the K neighbor coordinates are respectively Concatenate the coordinate information of p i and its neighbor points through local spatial encoding, including the center point coordinate, neighbor point coordinates, coordinate differences, and the distance between the center point and neighbor points; Adjust through a shared perceptron to obtain a new feature that aggregates the Euclidean distance spatial information

[0112]

[0113] In the formula, ⊕ represents the concatenation operation, and ‖·‖ calculates the distance from the center point to all neighbor points.

[0114] In this embodiment, such a local spatial encoding unit explicitly embeds the coordinates of all neighboring points, enabling the corresponding point features to perceive their relative spatial positions, which allows this encoding unit to learn shallow features such as the spatial positions and distances between the central point and the neighboring points. This endows the central point feature with the potential to represent the surrounding space, thereby enabling the entire network to effectively learn complex local structures.

[0115] In this embodiment, for the processing of point features, the present invention calculates the influence scores of all other points on each point within the neighborhood, and finally adjusts the features of each point according to the scores. The input of this component is a feature vector of size d×K, and the d×K-dimensional feature is first repeatedly expanded to d×K×K dimensions. For each neighborhood R determined by the central point and its feature set F = {F1, F2, …, Fm}, the feature difference map is obtained as follows:

[0116]

[0117] The feature difference map contains the information associations between all pairs of points. Using the self-attention mechanism, it passes through a multi-layer perceptron MLP and a Softmax function to obtain the attention scores of each point and other points within the neighborhood. Multiply the attention scores by the feature difference map to obtain the adaptively adjusted attention features. Then, through residual connection and weighted summation, the enhanced feature F containing information such as shape and semantics is obtained. i k Finally, and F i k are concatenated to obtain a local aggregation feature that fuses shallow spatial features and high-level semantic features, which endows each point feature with the potential to represent the surrounding space, thereby enabling the entire network to effectively learn complex local structures.

[0118] In a specific embodiment, as Figure 7 shown, the dual-branch attention pooling module has 4 inputs, namely geometric distance, neighboring point features, coordinate difference, and local aggregation feature. Among them, the coordinate difference is the vector obtained by subtracting the neighboring point coordinates from the central point coordinates, and the local aggregation feature is the output of the local feature fusion module; assuming that g(i) and g(k) are two input feature vectors of this module, the feature distance between g(i) and g(k) is defined as:

[0119]

[0120] In the formula, |·| is to find the absolute value, and mean is the average value function; the parameter λ is used to control the weight of the feature distance, and λ is set to 0.1.

[0121]

[0122] In the formula, represents the geometric distance, represents the coordinate difference between the center point and the neighbor point; for the first branch of the dual-branch attention pooling module, the feature influence factor and the local aggregation feature f i k are merged:

[0123]

[0124] f i k+ The attention weights are calculated through a linear layer and a Softmax function

[0125]

[0126] The local context features can be obtained by weighted summing the local aggregation features using the learned attention weights:

[0127]

[0128] For the second branch, the formula for the global context features is:

[0129] f iG = Max(MLP(f i k ))

[0130] In this embodiment, finally, the two parts of the features are added to obtain the discriminative features, which effectively fuse the local context features and the global context features, and improve the feature representation ability.

[0131] In this embodiment, for dense tasks such as semantic segmentation that classify each point in the point cloud, the point set needs to be upsampled to restore the original number of points. The upsampling module is used to upsample the point cloud to transition the features from the shape level to the point level. The KNN algorithm is used to find the K nearest neighbor points for each p point, and interpolation is performed based on the Euclidean distance between the point and its K nearest neighbor points in the three-dimensional space.

[0132]

[0133]

[0134] where p i is the neighbor point of p, and w i (p) represents the weight inversely proportional to the distance between the point p and p i Therefore, the farther p i is from p, the smaller the weight; dist(p, pi ) represents the distance function between point p and p i In this embodiment, the Euclidean distance is adopted.

[0135] Embodiment 3

[0136] More specifically, as Figure 2 shown, in a specific embodiment, in step S2, three-dimensional reconstruction is performed based on the two-dimensional sequence image to obtain the three-dimensional point cloud of the plant. The specific steps are as follows:

[0137] S201. Use the Scale-Invariant Feature Transform (SIFT) algorithm to find the extreme points in the image scale space of the two-dimensional sequence image, detect potential feature points, and select the most robust points as the final feature points;

[0138] S202. Introduce the k-dimensional space binary tree kd-tree model. Calculate the Euclidean distance between the final feature points of any two images in the two-dimensional sequence image to obtain their matching degree through the kd-tree model. Determine the matching points according to the level of the matching degree, and calculate the corresponding relationship between the final feature points of the image pair and the camera pose in the three-dimensional space;

[0139] S203. Solve the camera pose of the image pair and its final feature point pair through the triangulation principle to obtain the three-dimensional coordinates, thereby generating a three-dimensional sparse point cloud;

[0140] S204. Use the multi-view stereo clustering algorithm to cluster the two-dimensional sequence image, and then use the multi-view stitching algorithm to expand the sparse point cloud to obtain a dense point cloud as the final three-dimensional point cloud of the plant.

[0141] In a specific embodiment, in step S3, preprocessing is performed on the three-dimensional point cloud of the plant, specifically: performing point cloud downsampling, color threshold segmentation, and noise point removal on the three-dimensional point cloud of the plant in sequence:

[0142] When performing point cloud downsampling on the three-dimensional point cloud of the plant, the three-dimensional voxel grid method is adopted to create a three-dimensional voxel grid through the input point cloud data; within each voxel of the three-dimensional voxel grid, the center of gravity of all points in the voxel is used to approximately represent other points in the voxel, and all points in the voxel are represented by a center of gravity point. After processing all voxel blocks, the filtered point cloud is obtained to complete the downsampling;

[0143] For the three-dimensional point cloud of the plant that has completed point cloud downsampling, according to the different RGB values of different point clouds, manually adjust the appropriate parameter thresholds to retain the plants in the three-dimensional point cloud of the plant and filter the background and soil in the three-dimensional point cloud of the plant;

[0144] For the three-dimensional point cloud of the plant after color threshold segmentation, the statistical outlier removal algorithm is used to remove the points with too large a distance from the adjacent points in the three-dimensional point cloud of the plant, and the three-dimensional point cloud of the plant after preprocessing is obtained.

[0145] In a specific embodiment, using the statistical outlier removal algorithm to remove the points with too large a distance from the adjacent points in the three-dimensional point cloud of the plant is specifically as follows:

[0146] Traverse all the points in the three-dimensional point cloud of the plant, and calculate the average distance d between each point and its K nearest neighbors ij ;

[0147] Calculate the mean μ and standard deviation σ of all these point distances to determine the distance threshold;

[0148] Manually set the number of neighbor points, specifying the number of adjacent elements to be considered when calculating the average distance of a given point; manually set the standard deviation, specifying the threshold level of the standard deviation of the average distance in the point cloud;

[0149] Define the points with the distance between two points exceeding the threshold as noise points, and remove the noise points through the noise point removal formula:

[0150]

[0151]

[0152] In a specific embodiment, in the step S3, the manual annotation is specifically as follows:

[0153] For the noise points existing in the three-dimensional point cloud of the preprocessed plant and continuously distributed with the leaves and stems, manually segment different semantic segmentation label categories through the cloudcompare software, and label each point to obtain the plant point cloud data set.

[0154] In a specific embodiment, as Figure 5 shown, the noisy plant point cloud semantic segmentation network adopts an encoder-decoder structure, including 4 self-attention feature fusion modules, 2 shared fully connected layers, and 3 upsampling layers;

[0155] The noisy plant point cloud semantic segmentation network obtains multi-level encoded features through 4 self-attention feature fusion modules; the obtained features are decoded layer by layer through 3 upsampling layers. Each upsampling layer upsamples the features in the input point cloud data set points through the nearest neighbor interpolation algorithm, and uses skip connections to splice with the intermediate features of the corresponding dimension encoding layer; the 2 shared fully connected layers are the dropout layer and the Log-SoftMax layer respectively; the spliced point cloud passes through the dropout layer and the Log-SoftMax layer in turn to obtain the predicted semantics of all points in the point cloud data set.

[0156] In a specific embodiment, the self-attention feature fusion module includes a downsampling module, a local feature fusion module, and a dual-branch attention pooling module; the downsampling module, the local feature fusion module, and the dual-branch attention pooling module are connected in sequence;

[0157] The downsampling module downsamples the point cloud in the input point cloud data set points, reducing the number of point clouds while dimensionality-increasing the features of each point. Specifically: using farthest point sampling, iteratively selecting the farthest point from the existing sampling point set to obtain the sampled point cloud; for each point among them, using the KNN algorithm to find the K points closest to the center point, generating a local neighborhood, and each local neighborhood passes through a multi-layer perceptron, and finally max pooling is performed to obtain the output of the downsampling;

[0158] The first branch of the dual-branch attention pooling module adaptively filters the local low-correlation features of the local context features using the channel attention mechanism, which is used to guide the network to focus on the shape structure information of the plant when extracting the semantic segmentation features of the plant point cloud; the second branch obtains the global context features by performing max pooling on the highest-level feature map of the local context features; finally, the local and global context features are fused to obtain multi-level encoded features.

[0159] In a specific embodiment, from Figure 6 it can be known that the input scale of the local feature fusion module is N×(3 + d), where N represents the number of points, 3 represents the position coordinates in three-dimensional space, and d represents the feature attribute; using the KNN algorithm for each point once to obtain the K points with the closest Euclidean distance; for the i-th point in the input, assuming its coordinate is p i , and the coordinates of the K neighbors are respectively Concatenate the coordinate information of p i and its neighbor points through local spatial encoding, including the center point coordinate, neighbor point coordinates, coordinate differences, and the distance between the center point and neighbor points; adjust through a shared perceptron to obtain a new feature r i k :

[0160]

[0161] In the formula, ⊕ represents the concatenation operation, and ‖·‖ calculates the distance from the center point to all neighbor points.

[0162] In this embodiment, such a local spatial encoding unit explicitly embeds the coordinates of all neighboring points, enabling the corresponding point features to perceive their relative spatial positions, which allows this encoding unit to learn shallow features such as the spatial positions and distances between the central point and neighboring points. This endows the central point feature with the potential to represent the surrounding space, thereby enabling the entire network to effectively learn complex local structures.

[0163] In this embodiment, for the processing of point features, the present invention calculates the influence scores of all other points on each point in the neighborhood, and finally adjusts the features of each point according to the scores. The input of this component is a feature vector of size d×K, which is first repeatedly expanded from d×K dimensions to d×K×K dimensions. For each neighborhood R determined by the central point and its feature set F = {F1, F2, …, Fm}, the feature difference map is obtained according to the following:

[0164]

[0165] The feature difference map contains the information associations between all pairs of points. Using the self-attention mechanism, it passes through a multi-layer perceptron MLP and a Softmax function to obtain the attention scores of each point and other points in the neighborhood. Multiply the attention scores with the feature difference map to obtain the adaptively adjusted attention features. Then, through residual connection and weighted summation, enhanced features containing information such as shape and semantics are obtained. Finally, r i k and are concatenated to obtain a local aggregation feature that fuses shallow spatial features and high-level semantic features, which endows each point feature with the potential to represent the surrounding space, thereby enabling the entire network to effectively learn complex local structures.

[0166] In a specific embodiment, as Figure 7 shown, the dual-branch attention pooling module has 4 inputs, namely geometric distance, neighboring point features, coordinate difference, and local aggregation feature. Among them, the coordinate difference is the vector obtained by subtracting the neighboring point coordinates from the central point coordinates, and the local aggregation feature is the output of the local feature fusion module; assuming that g(i) and g(k) are two input feature vectors of this module, the feature distance between g(i) and g(k) is defined as:

[0167]

[0168] In the formula, |·| is to find the absolute value, and mean is the average value function; the parameter λ is used to control the weight of the feature distance, and λ is set to 0.1.

[0169]

[0170] In the formula, represents the geometric distance, represents the coordinate difference between the center point and the neighbor points; for the first branch of the dual-branch attention pooling module, the feature influence factor and the local aggregation feature f i k are merged:

[0171]

[0172] f i k+ The attention weights are calculated through a linear layer and the Softmax function

[0173]

[0174] The local context features can be obtained by weighted summing the local aggregation features using the learned attention weights:

[0175]

[0176] For the second branch, the formula for the global context features is:

[0177] f iG = Max(MLP(f i k ))

[0178] More specifically, to further illustrate the technical effects of the present solution, the present embodiment provides a more detailed description of the solution.

[0179] In this embodiment, on the plant point cloud dataset collected and reconstructed by this method, a comparative experiment is carried out with a classical semantic segmentation network to test the performance of this method.

[0180] The overall accuracy formula used to measure the performance of the semantic segmentation network is:

[0181]

[0182] The mean intersection over union formula used to measure the performance of the semantic segmentation network is:

[0183]

[0184] where c represents the number of semantic label classes. In addition, the precision and recall are also used to evaluate the results of the semantic segmentation, and the formulas are respectively:

[0185]

[0186]

[0187] The semantic segmentation objective of the 3D plant point cloud is to divide the points into four categories: leaf, stem, red, and cluster. The semantic segmentation results are shown in Table 1, and the visualization of the segmentation results is as Figure 8 shown:

[0188]

[0189]

[0190] Table 1

[0191] As can be seen from Table 1, for the semantic segmentation network of noisy plant point clouds proposed in the present invention, namely the SAFF-Net network, the OA and mIoU are 93.7% and 83.4% respectively. The overall accuracy of this model is 2.1% higher than that of PointNet, 2.2% higher than that of PointNet++, 1.5% higher than that of DGCNN, and 3.5% higher than that of PointWeb. In terms of the average intersection over union, it is 2.9% higher than that of PointNet, 4.0% higher than that of PointNet++, 3.0% higher than that of DGCNN, and 6.5% higher than that of PointWeb. The model of the present invention has achieved the highest IoU accuracy for the three semantic labels of miscellaneous points, leaves, and stems. The SAFF-Net network has achieved good precision and recall for all semantic classes. Especially for the stem class, its precision is 38% higher than that of PointNet, 4.1% higher than that of PointNet++, 15.3% higher than that of DGCNN, and 12.3% higher than that of PointWeb. Since the network has high precision and recall for both the miscellaneous points and leaf classes, it shows that the model has good segmentation ability for closely connected miscellaneous points and leaves. Through the above analysis, it can be seen that the introduction of spatial attention and local feature fusion can effectively learn context features and also learn the shape structure information of the point cloud, thereby improving the segmentation accuracy.

[0192] Example 4

[0193] A semantic segmentation system for noisy plant point clouds based on self-attention feature fusion includes an image acquisition module, a three-dimensional reconstruction module, a preprocessing module, a semantic segmentation network module, and a semantic segmentation module:

[0194] The image acquisition module is used to obtain two-dimensional sequence images;

[0195] The three-dimensional reconstruction module is used to perform three-dimensional reconstruction based on the two-dimensional sequence images to obtain the three-dimensional point cloud of the plant;

[0196] The preprocessing module is used to preprocess and manually annotate the three-dimensional point cloud of the plant to obtain a plant point cloud data set;

[0197] The described semantic segmentation network module is used to construct a noisy plant point cloud semantic segmentation network based on self-attention feature fusion, and train the noisy plant point cloud semantic segmentation network through a plant point cloud data set; the semantic segmentation module is used to perform semantic segmentation on the plant point cloud through the trained noisy plant point cloud semantic segmentation network to obtain the predicted semantic label of each point.

[0198] In this embodiment, the image acquisition module acquires high-precision and multi-angle plant 2D sequence images through a camera;

[0199] In this embodiment, the 3D reconstruction module includes a feature point acquisition unit, a feature point matching unit, a sparse point cloud generation unit, and a dense point cloud generation unit; among them:

[0200] The feature point acquisition unit uses the Scale-Invariant Feature Transform (SIFT) operator to detect potential feature points from the plant 2D sequence images and selects the most robust points as the final feature points.

[0201] The feature point matching unit uses the kd-tree model in the K-dimensional space to calculate the Euclidean distance between the feature points of two images to determine the matching degree, determines the matching points according to the high and low of the matching degree, and calculates the corresponding relationship between the feature points and the camera pose of the camera in the three-dimensional space.

[0202] The sparse point cloud generation unit uses the principle of triangulation to solve the three-dimensional coordinates of the camera pose of the image pair and its feature point pair, thereby generating a three-dimensional sparse point cloud.

[0203] The dense point cloud generation unit uses the CMVS and PMVS algorithms to expand the pixels around the feature points of the sparse point cloud to form a dense point cloud, that is, obtains the 3D point cloud of the plant.

[0204] In this embodiment, the preprocessing module includes a point cloud downsampling unit, a color threshold segmentation unit, and a noise point removal unit; among them:

[0205] The point cloud downsampling unit uses the three-dimensional voxel grid method to create a three-dimensional voxel grid through the input point cloud data, and then uses the centroid in each voxel to approximately represent other points in the voxel, so as to achieve the purpose of downsampling.

[0206] The color threshold segmentation unit manually adjusts appropriate parameter thresholds according to the different RGB values of different point cloud colors, retains the green plants, and filters out the white background and black soil.

[0207] The noise point removal unit uses the Statistical outlier removal algorithm to delete points that are too far away from their adjacent points. The Statistical outlier removal in open3D is used to delete points that are too far away from their adjacent points compared to the average value. It requires two input parameters:

[0208] nb_neighbors, which specifies the number of adjacent elements to consider when calculating the average distance of a given point.

[0209] std_ratio, which allows setting the threshold level according to the standard deviation of the average distance in the point cloud. The lower this number, the more aggressive the filter and the more points will be deleted.

[0210] More specifically, manual annotation is implemented using the cloudcompare software. Specifically:

[0211] Input the point cloud that needs to be manually annotated into the cloudcompare software, and manually segment different semantic segmentation label categories through the segmentation tool and label each point. The entire point cloud can be divided into leaf class, stem class, red paper class, and miscellaneous point class.

[0212] More specifically, the semantic segmentation network constructed by the semantic segmentation network module specifically performs the following operations: predicting the class labels of all points in the input point cloud to complete semantic label prediction, where:

[0213] The 3D point cloud semantic segmentation network adopts an encoder-decoder structure, and it specifically performs the following steps:

[0214] Use four encoders to hierarchically encode features through the self-attention feature fusion module, gradually reducing the number of point clouds and increasing the dimension of each point cloud; each self-attention feature fusion module includes downsampling, local feature fusion module, and double-branch attention pooling module. The number of points gradually decreases from N to N / 256, and the change process of the number of point clouds is N→N / 4→N / 8→N / 64→N / 256. The feature dimension increases from d to 512, and the change process of the point cloud feature dimension is d→64→128→256→512;

[0215] Use four decoders to decode the features layer by layer. Each decoder upsamples the point features through the nearest neighbor interpolation algorithm and stitches them with the intermediate features of the corresponding dimension encoding layer using skip connections to gradually restore the number of point clouds to N;

[0216] Through two shared fully-connected layers, where the dimensionality changes as (N, 128) → (N, 32) → (N, C), a dropout layer with a randomly set dropout rate of 0.5 is applied after the first fully-connected layer. A Log-SoftMax layer is applied after the first fully-connected layer to obtain the predicted semantics for all points, with a size of N×C, where C is the number of semantic label classes.

[0217] In the specific implementation process, this system is used to implement a point cloud segmentation method with noise, which is simple to implement, convenient to operate, and easy to promote in practical applications.

[0218] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, rather than limitations on the implementation manners of the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the claims of the present invention.

Claims

1. A method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion, characterized in that: Including the following steps: S1: Obtain a two-dimensional sequence image; S2: Perform three-dimensional reconstruction based on the two-dimensional sequence image to obtain the three-dimensional point cloud of the plant; In the step S2, for performing three-dimensional reconstruction based on the two-dimensional sequence image to obtain the three-dimensional point cloud of the plant, the specific steps are as follows: S201. Use the Scale-Invariant Feature Transform (SIFT) algorithm to find extreme points in the image scale space of the two-dimensional sequence image, detect potential feature points, and select the most robust points as the final feature points; S202. Introduce a K-dimensional space binary tree kd-tree model. Calculate the Euclidean distance between the final feature points of any pair of images in the two-dimensional sequence image through the kd-tree model to obtain their matching degree. Determine the matching points according to the high or low matching degree, and calculate the corresponding relationship between the final feature points of the image pair and the camera pose in the three-dimensional space; S203. Solve the camera pose of the image pair and its final feature point pair through the triangulation principle to obtain three-dimensional coordinates, thereby generating a three-dimensional sparse point cloud; S204. Use the multi-view stereo clustering algorithm to cluster the two-dimensional sequence images, and then use the multi-view stitching algorithm to expand the sparse point cloud to obtain a dense point cloud as the final three-dimensional point cloud of the plant S3: Preprocess and manually annotate the three-dimensional point cloud of the plant to obtain a plant point cloud dataset; The noisy plant point cloud semantic segmentation network adopts an encoder-decoder structure, including 4 self-attention feature fusion modules, 2 shared fully connected layers, and 3 upsampling layers; The noisy plant point cloud semantic segmentation network obtains multi-level encoded features through 4 self-attention feature fusion modules; the obtained features are decoded layer by layer through 3 upsampling layers. Each upsampling layer upsamples the features in the input point cloud dataset points through the nearest neighbor interpolation algorithm, and uses skip connections to splice with the intermediate features of the corresponding dimension encoding layer; the 2 shared fully connected layers are a dropout layer and a Log-SoftMax layer respectively; the spliced point cloud passes through the dropout layer and the Log-SoftMax layer in sequence to obtain the predicted semantics of all points in the point cloud dataset; S4: Construct a noisy plant point cloud semantic segmentation network based on self-attention feature fusion, and train the noisy plant point cloud semantic segmentation network through the plant point cloud dataset; S5. Perform semantic segmentation on the plant point cloud through the trained noisy plant point cloud semantic segmentation network to obtain the predicted semantic label of each point.

2. The method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion according to claim 1, wherein: In the step S3, for preprocessing the three-dimensional point cloud of the plant, specifically: sequentially perform point cloud downsampling, color threshold segmentation, and noise point removal on the three-dimensional point cloud of the plant: When performing point cloud downsampling on the three-dimensional point cloud of the plant, use the three-dimensional voxel grid method to create a three-dimensional voxel grid through the input point cloud data; within each voxel of the three-dimensional voxel grid, approximate the other points in the voxel with the center of gravity of all points in the voxel, represent all points in the voxel with a center of gravity point, and after processing all voxel blocks, obtain the filtered point cloud to complete the downsampling; For the three-dimensional plant point cloud after downsampling the point cloud, according to the different RGB values of different point cloud colors, manually adjust the appropriate parameter thresholds to retain the plants in the three-dimensional plant point cloud and filter the background and soil in the three-dimensional plant point cloud; For the three-dimensional plant point cloud after color threshold segmentation, use the statistical outlier removal algorithm to delete the points with too large a distance from the adjacent points in the three-dimensional plant point cloud to obtain the three-dimensional plant point cloud after preprocessing.

3. The method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion according to claim 2, characterized in that: Use the statistical outlier removal algorithm to delete the points with too large a distance from the adjacent points in the three-dimensional plant point cloud. Specifically: Traverse all points in the three-dimensional point cloud of the plant and calculate the average distance between each point and its K nearest neighbors ; Calculate the mean of the distances of all these points and the standard deviation , to determine the distance threshold; Manually set the number of neighbors, specifying the number of adjacent elements to consider when calculating the average distance of a given point; manually set the standard deviation, specifying the threshold level for setting the standard deviation of the average distance in the point cloud; Define the points with a distance between two points exceeding the threshold as noise points, and remove the noise points through the noise point removal formula: 。 4. The method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion according to claim 2, wherein: In the step S3 described above, manual annotation is specifically: For the noise points existing in the three-dimensional plant point cloud after preprocessing and continuously distributed with the leaves and stems, manually segment different semantic segmentation label categories through the cloudcompare software, label each point, and obtain the plant point cloud data set.

5. The method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion according to claim 1, characterized in that: The self-attention feature fusion module includes a downsampling module, a local feature fusion module, and a dual-branch attention pooling module; the downsampling module, the local feature fusion module, and the dual-branch attention pooling module are connected in sequence; The downsampling module downsamples the point cloud in the input point cloud data set points, reduces the number of point clouds, and at the same time upgrades the features of each point. Specifically: use farthest point sampling, continuously iteratively select the farthest point from the existing sampling point set to obtain the sampled point cloud; for each point in it, use the KNN algorithm to find the K points closest to the center point, generate a local neighborhood, each local neighborhood passes through a multi-layer perceptron, and finally perform max pooling to obtain the output of downsampling; The local feature fusion module aggregates the shallow features including the correlation between all point pairs within the neighborhood, the position and distance information of the points, and the high-level features including shape and semantic information to obtain local context features; The first branch of the dual-branch attention pooling module adaptively filters the locally low-correlation features of the local context features using the channel attention mechanism, which is used to guide the network to focus on the shape structure information of the plant when extracting the semantic segmentation features of the plant point cloud; the second branch obtains the global context features by performing max pooling on the highest-level feature map of the local context features; finally, fuse the local and global context features to obtain multi-level encoded features.

6. The method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion according to claim 5, characterized in that: The input scale of the local feature fusion module is N×(3 + d), where N represents the number of points, 3 represents the position coordinates in three-dimensional space, and d represents the feature attributes; the KNN algorithm is used once for each point to obtain the K points with the closest Euclidean distance; for the i-th point in the input, assuming its coordinates are , the coordinates of the K neighbors are respectively Combine and the coordinate information of its neighbor points, and splice the central point coordinates, neighbor point coordinates, coordinate differences, and the distance between the central point and the neighbor points through local space encoding; adjust through a shared perceptron to obtain a new feature that aggregates the Euclidean distance space information : In the formula, ⊕ represents the concatenation operation, and ‖⋅‖ calculates the distance from the center point to all neighbor points.

7. The method for semantic segmentation of noisy plant point clouds based on self-attention feature fusion according to claim 5, characterized in that: The dual-branch attention pooling module has four inputs, namely geometric distance, neighboring point features, coordinate difference, and local aggregation features. Among them, the coordinate difference is the vector obtained by subtracting the neighboring point coordinates from the center point coordinates, and the local aggregation features are the output of the local feature fusion module. Suppose g(i) and g(k) are two input feature vectors of this module, and the feature distance is defined as: In the formula, |∙| is to find the absolute value, mean is the average value function; use the parameter λ to control the weight of the feature distance, and λ is set to 0.1; Wherein, represents the geometric distance, represents the coordinate difference between the center point and the neighbor point; for the first branch of the dual-branch attention pooling module, the feature influence factor and the local aggregation feature are merged by splicing: The attention weights are calculated through a linear layer and the Softmax function : The local context features can be obtained by weighted summing the local aggregated features using the learned attention weights: For the second branch, the formula for the global context features is: 。 8. A noisy plant point cloud semantic segmentation system based on self-attention feature fusion, characterized in that: Including an image acquisition module, a 3D reconstruction module, a preprocessing module, a semantic segmentation network module, and a semantic segmentation module: The described image acquisition module is used to obtain two-dimensional sequence images; The described 3D reconstruction module is used to perform 3D reconstruction based on the two-dimensional sequence images to obtain the 3D point cloud of the plant; The described preprocessing module is used to preprocess and manually annotate the 3D point cloud of the plant to obtain the plant point cloud dataset; The described semantic segmentation network module is used to construct a noisy plant point cloud semantic segmentation network based on self-attention feature fusion, and train the described noisy plant point cloud semantic segmentation network through the plant point cloud dataset; The described semantic segmentation module is used to perform semantic segmentation on the plant point cloud through the trained noisy plant point cloud semantic segmentation network to obtain the predicted semantic label of each point.

Images