A method and apparatus for semantic segmentation of 3D point clouds in farms for unmanned agricultural machinery path planning

Abstract

This invention provides a method and apparatus for semantic segmentation of 3D point clouds in farmland for unmanned agricultural machinery path planning. The method includes: acquiring image data of farmland using a drone and converting the image data into 3D point cloud data; inputting the 3D point cloud data into a farmland 3D point cloud semantic segmentation model for classification to obtain the land cover types corresponding to different point clouds; the farmland 3D point cloud semantic segmentation model is trained through the following steps: acquiring multiple subsets of point cloud data based on the 3D point cloud data, extracting local geometric features and global semantic information of each sampling point; determining the predicted category of the sampling point based on the local geometric features and global semantic information; and training a deep learning model based on the predicted category and the true category to obtain the farmland 3D point cloud semantic segmentation model. The farmland 3D point cloud semantic segmentation method and apparatus for unmanned agricultural machinery path planning provided by this invention improves the accuracy of land cover recognition in farmland scenes.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method and apparatus for semantic segmentation of 3D point clouds in farms for unmanned agricultural machinery path planning. Background Technology

[0002] With the development of intelligent agriculture, unmanned agricultural machinery is increasingly used in precision navigation, path planning, and task allocation. The realization of these functions requires high-precision three-dimensional semantic information based on farmland features. However, due to the unstructured, multi-scale, and complex environmental characteristics of farmland scenes, traditional methods for extracting feature information are insufficient to meet practical needs, especially when constructing high-precision semantic maps in large-scale scenarios, which presents numerous challenges.

[0003] Currently, the acquisition of farmland feature information mainly relies on geometric maps, topological maps, or grid maps. However, these methods still have significant limitations when facing the complexity, diversity, and large-scale feature extraction of farmland environments. They cannot achieve a balance between detail preservation and global information fusion, resulting in deficiencies in semantic consistency, feature capture, and feature classification accuracy. Consequently, the accuracy of feature information acquisition is difficult to guarantee, thus reducing the efficiency and reliability of automated operations in farmland scenarios (such as driving route planning, task allocation, and navigation). Summary of the Invention

[0004] This invention provides a method and apparatus for semantic segmentation of 3D point clouds in farms for unmanned agricultural machinery path planning, in order to solve the technical problem of low accuracy of ground feature segmentation in the prior art.

[0005] In a first aspect, the present invention provides a method for semantic segmentation of farm 3D point clouds for unmanned agricultural machinery path planning, comprising the following steps.

[0006] Image data of farmland is acquired by drones and then converted into three-dimensional point cloud data;

[0007] The 3D point cloud data is input into the farm's 3D point cloud semantic segmentation model for classification to obtain the land cover types corresponding to different point clouds; the farm's 3D point cloud semantic segmentation model is trained through the following steps:

[0008] Multiple subsets of point cloud data are obtained based on 3D point cloud data;

[0009] Extract the local geometric features and global semantic information of each sampling point in the point cloud data subset;

[0010] The predicted category of the sampling point is determined based on the local geometric features and the global semantic information;

[0011] A pre-defined deep learning model is trained based on the predicted category and the true category of the sampling point to obtain a semantic segmentation model of the farm's 3D point cloud.

[0012] In some embodiments, acquiring multiple subsets of point cloud data based on 3D point cloud data includes:

[0013] The three-dimensional point cloud data is processed based on the voxel mesh downsampling algorithm to obtain sample point cloud data;

[0014] The sample point cloud data is divided into multiple point cloud data subsets of the same size.

[0015] In some embodiments, extracting the local geometric features of each sampling point in the subset of point cloud data includes:

[0016] The K-nearest neighbor algorithm is used to obtain all neighboring points of each sampling point;

[0017] The local spatial features of the sampling point are obtained based on the sampling point and its neighboring points; the local spatial features include the distance decay index and the local geometric bounding box.

[0018] The local spatial features are encoded using a multilayer perceptron to obtain the local geometric features of the sampling points.

[0019] In some embodiments, global semantic information of each sampling point in the point cloud data subset is extracted, including:

[0020] Calculate the volume ratio of the local neighborhood space to the global space for each sampling point; the local neighborhood space is the space containing all neighboring points of the sampling point; the global space is the space corresponding to the subset of point cloud data.

[0021] The global semantic information of the sampling points is determined based on the volume ratio.

[0022] In some embodiments, determining the predicted category of the sampling point based on the local geometric features and the global semantic information includes:

[0023] The local geometric features are fused with the global semantic information to obtain the aggregated features of the sampling points;

[0024] Based on the aggregated features of the sampling points and the aggregated features of the neighborhood points of the sampling points, the predicted category of the sampling points is determined.

[0025] In some embodiments, training a preset deep learning model based on the predicted category and the true category of the sampling point includes:

[0026] The cross-entropy loss is determined based on the predicted category and the true category of the sampling point;

[0027] The model parameters of the deep learning model are adjusted based on the cross-entropy loss.

[0028] In some embodiments, the method further includes:

[0029] Farmland semantic maps are generated based on the land cover types corresponding to different point clouds; the farmland semantic maps are used to display the three-dimensional distribution information of different land cover types in farmland.

[0030] Secondly, the present invention provides a farm 3D point cloud semantic segmentation device for unmanned agricultural machinery path planning, comprising the following modules.

[0031] The acquisition module is used to acquire image data of farmland through a drone and convert the image data into three-dimensional point cloud data;

[0032] The semantic segmentation module is used to input the 3D point cloud data into the farm's 3D point cloud semantic segmentation model for classification, thereby obtaining the land cover types corresponding to different point clouds. The farm's 3D point cloud semantic segmentation model is trained through the following steps:

[0033] Multiple subsets of point cloud data are obtained based on 3D point cloud data;

[0034] Extract the local geometric features and global semantic information of each sampling point in the point cloud data subset;

[0035] The predicted category of the sampling point is determined based on the local geometric features and the global semantic information;

[0036] A pre-defined deep learning model is trained based on the predicted category and the true category of the sampling point to obtain a semantic segmentation model of the farm's 3D point cloud.

[0037] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning as described above.

[0038] Fourthly, a non-transitory computer-readable storage medium stores a computer program that, when executed by a processor, implements a farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning as described above.

[0039] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning as described above.

[0040] This invention provides a method and apparatus for semantic segmentation of 3D point clouds in farmland for unmanned agricultural machinery path planning. It acquires image data of farmland using a drone, converts the image data into 3D point cloud data, and inputs the 3D point cloud data into a farmland 3D point cloud semantic segmentation model for classification, obtaining the land cover types corresponding to different point clouds. The farmland 3D point cloud semantic segmentation model is trained through the following steps: acquiring multiple subsets of point cloud data based on the 3D point cloud data, extracting local geometric features and global semantic information of each sampling point, determining the predicted category of the sampling point based on the local geometric features and global semantic information, and training a deep learning model based on the predicted category and the true category to obtain the farmland 3D point cloud semantic segmentation model. This effectively solves the feature extraction problem of multi-scale targets in farmland scenes, improves the recognition ability of complex land cover, and enhances the accuracy of land cover recognition in farmland scenes. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating the farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning provided by the present invention.

[0043] Figure 2 This is a schematic diagram of the farm point cloud construction process based on UAV aerial surveying provided by the present invention.

[0044] Figure 3 This is a schematic diagram of the structure of the farm 3D point cloud semantic segmentation model provided by the present invention.

[0045] Figure 4 This is a schematic diagram of the structure of the farm 3D point cloud semantic segmentation device for unmanned agricultural machinery path planning provided by the present invention.

[0046] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0047] Currently, there are many shortcomings in the field of 3D point cloud semantic segmentation in farmland scenes, which makes it difficult to meet the demand for high-precision semantic information for the precise navigation and path planning of unmanned agricultural machinery.

[0048] Methods based on geometric maps, topological maps, and mesh maps have limitations: While geometric maps can capture the three-dimensional structural information of the environment, they rely on simple geometric features and struggle to reflect the detailed information in complex farmland scenes. For example, common irregular structures in farmland (such as irrigation facilities and paddy field boundaries) often cannot be accurately represented by traditional geometric descriptions, resulting in insufficient semantic understanding of farmland features. Furthermore, these methods typically rely on manually defined rules or predefined paths, leading to low operational efficiency and difficulty in adapting to unstructured farmland environments. Although topological maps and mesh maps have been applied in path planning and task allocation, they cannot effectively integrate environmental semantic information, making it difficult to provide refined navigation support, and their computational efficiency is low in large-scale scenarios.

[0049] Projection and voxel methods simplify the data structure by transforming 3D point cloud data into 2D images or voxel meshes. However, this transformation inevitably leads to a reduction in point cloud resolution and a loss of detail, particularly when dealing with small targets in farmland scenes (such as field roads and irrigation pipes). Furthermore, these methods have high computational complexity, making them inefficient for processing large-scale point cloud data.

[0050] Point-by-point processing methods, such as PointNet, operate directly on the original point cloud, preserving geometric and semantic information well and making them suitable for handling complex scenes. However, PointNet and subsequent improved methods (such as RandLA-Net) still face scalability issues when processing large-scale point clouds. For example, their ability to extract features from multi-scale targets in farmland scenes is limited, making it difficult to balance local details and global semantic consistency, leading to a decrease in classification accuracy and segmentation performance. Furthermore, these methods are weak in handling class imbalance, and small-class targets (such as hard surfaces or irrigation facilities) are often overlooked due to the sparseness of the point cloud.

[0051] In summary, current segmentation methods typically only address regular scenarios such as paved roads, and struggle to simultaneously meet the demands of large-scale, multi-scale, and high semantic consistency in 3D point cloud semantic segmentation of farmland. These shortcomings limit the precision operation and path planning capabilities of unmanned agricultural machinery in complex farmland environments, such as irregular farmland scenes like field ridges, paddy fields, and irrigation facilities. Therefore, there is an urgent need for a novel point cloud segmentation technology capable of efficiently extracting local and global features and adapting to multi-scale targets to drive the development of agricultural automation.

[0052] To address the aforementioned technical challenges, this invention proposes a 3D point cloud semantic segmentation method for farmland, geared towards unmanned agricultural machinery path planning. The method involves acquiring farmland image data via a drone and converting it into 3D point cloud data. This 3D point cloud data is then input into a farmland 3D point cloud semantic segmentation model for classification, yielding different feature types corresponding to different point clouds. The farmland 3D point cloud semantic segmentation model is trained through the following steps: acquiring multiple subsets of point cloud data based on the 3D point cloud data; extracting local geometric features and global semantic information for each sampling point; determining the predicted category of the sampling point based on the local geometric features and global semantic information; and training a deep learning model based on the predicted and true categories to obtain the farmland 3D point cloud semantic segmentation model. This method efficiently extracts local geometric features and global semantic information from farmland scenes, effectively addressing the shortcomings of existing technologies in handling unstructured farmland, multi-scale targets, and semantic consistency. It achieves high-precision semantic segmentation of features, providing technical support for precise navigation, path planning, and task allocation for unmanned agricultural machinery. This invention not only improves the accuracy and robustness of semantic segmentation but also significantly enhances classification consistency and small target recognition capabilities in complex farmland scenes, thereby meeting the needs of modern agricultural automation and intelligence.

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] Figure 1 This is a flowchart illustrating the farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning provided by the present invention, as shown below. Figure 1 As shown, this invention provides a method for semantic segmentation of 3D point clouds in farms for unmanned agricultural machinery path planning. The method includes:

[0055] Step 101: Acquire image data of farmland using a drone and convert the image data into three-dimensional point cloud data.

[0056] Specifically, current point cloud acquisition and segmentation for autonomous driving are based on vehicle-mounted equipment. However, this application uses drones to collect point clouds to guide autonomous driving of ground vehicles. Specifically, a drone equipped with a high-precision image sensor (e.g., DJI Phantom 4RTK) can be used to perform a full-coverage scan of the target farmland area, obtain high-definition image data of the farmland, identify farmland features, and thus obtain a farmland map to assist autonomous driving, etc.

[0057] In some embodiments, the drone flies at an altitude of 25-50 meters and a speed of 2 m / s, with a vertical and directional overlap of 70% to ensure complete coverage and data quality.

[0058] Then, the image data can be converted into 3D point cloud data using professional software (such as DJI Terra).

[0059] In some embodiments, after converting image data into 3D point cloud data, the point cloud data is also checked for compliance.

[0060] For example, Figure 2 This is a schematic diagram of the farm point cloud construction process based on UAV aerial surveying provided by the present invention, such as... Figure 2 As shown, firstly, inspection points are set up in the farmland. Then, drones are used to conduct photogrammetry on the farmland at each inspection point, and point clouds are generated through office processing. Finally, the point cloud data is checked for quality. If the check is qualified, the process ends. If the check is unqualified, the drone conducts photogrammetry on the farmland again and generates point clouds until qualified point cloud data is obtained.

[0061] Step 102: Input the three-dimensional point cloud data into the farm's three-dimensional point cloud semantic segmentation model for classification to obtain the land cover types corresponding to different point clouds.

[0062] Specifically, a pre-trained farm 3D point cloud semantic segmentation model is used to perform semantic segmentation on the 3D point cloud data, that is, to classify and identify the land cover types of the 3D point cloud data to obtain the land cover types corresponding to different point clouds.

[0063] The farm's 3D point cloud semantic segmentation model was trained through the following steps:

[0064] Multiple subsets of point cloud data are obtained based on 3D point cloud data;

[0065] Extract the local geometric features and global semantic information of each sampling point in the point cloud data subset;

[0066] The predicted category of the sampling point is determined based on the local geometric features and the global semantic information;

[0067] A pre-defined deep learning model is trained based on the predicted category and the true category of the sampling point to obtain a semantic segmentation model of the farm's 3D point cloud.

[0068] Specifically, the farm's 3D point cloud semantic segmentation model adopts a random sampling-encoding-decoding structure for multi-scale feature extraction. In the sampling module, multiple subsets of point cloud data are acquired; in the encoding module, local geometric features and global semantic information of each sampling point in the point cloud data subset are extracted; and in the decoding module, the predicted land cover category of the sampling point is identified based on the extracted features.

[0069] In this embodiment, custom datasets and public datasets (such as Semantic3D) can be used for joint training to improve the generalization ability of the model.

[0070] The semantic segmentation method for farm 3D point clouds for unmanned agricultural machinery path planning provided in this application can efficiently extract local geometric features and global semantic information in farmland scenes. It effectively addresses the shortcomings of existing technologies in handling unstructured farmland, multi-scale targets, and semantic consistency, achieving high-precision semantic segmentation of ground features. This provides technical support for the precise navigation, path planning, and task allocation of unmanned agricultural machinery. It not only improves the accuracy and robustness of semantic segmentation but also significantly enhances classification consistency and small target recognition capabilities in complex farmland scenes, thus meeting the needs of modern agricultural automation and intelligence. Furthermore, it offers high ease of operation and application flexibility, adapting to the needs of various farmland scenarios, and demonstrates excellent energy-saving and environmental protection performance, showcasing significant technical advantages and broad application prospects.

[0071] In some embodiments, acquiring multiple subsets of point cloud data based on 3D point cloud data includes:

[0072] The three-dimensional point cloud data is processed based on the voxel mesh downsampling algorithm to obtain sample point cloud data;

[0073] The sample point cloud data is divided into multiple point cloud data subsets of the same size.

[0074] Specifically, a voxel mesh downsampling algorithm is used to process the 3D point cloud data to reduce the number of points and obtain sample point cloud data. Then, a batch data generation algorithm is used to divide the sample point cloud data into multiple subsets of point cloud data of the same size to ensure that each batch of data can be uniformly input into the model for training.

[0075] The farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning provided in this application reduces redundant data by voxel downsampling of point cloud data, reduces data size while preserving original geometric information as much as possible, and divides the point cloud into multiple point cloud data subsets of the same size, generating a standardized dataset suitable for model input, ensuring that the data input to the model has consistency and uniformity; in addition, it also reduces the time for processing large-scale point cloud data.

[0076] In some embodiments, extracting the local geometric features of each sampling point in the subset of point cloud data includes:

[0077] The K-nearest neighbor algorithm is used to obtain all neighboring points of each sampling point;

[0078] The local spatial features of the sampling point are obtained based on the sampling point and its neighboring points; the local spatial features include the distance decay index and the local geometric bounding box.

[0079] The local spatial features are encoded using a multilayer perceptron to obtain the local geometric features of the sampling points.

[0080] Specifically, the K-Nearest Neighbors (KNN) algorithm is used to obtain all neighboring points of each sampling point. All neighboring points of each sampling point constitute the local spatial neighborhood of that sampling point. Based on the local spatial neighborhood of that sampling point, the local spatial features of that sampling point are extracted. The local spatial features include the distance decay index and the local geometric bounding box, etc.

[0081] Then, the local spatial features are encoded based on the multilayer perceptron (MLP), and the self-attention mechanism is combined to enhance the representation ability of local geometric features of point cloud, thereby generating local geometric features with higher semantic understanding ability.

[0082] The semantic segmentation method for farm 3D point cloud for unmanned agricultural machinery path planning provided in this application captures the microstructural information of complex features by extracting local geometric features of the point cloud, thereby improving the ability to identify complex features and enhancing the accuracy of feature recognition in farmland scenes.

[0083] In some embodiments, global semantic information of each sampling point in the point cloud data subset is extracted, including:

[0084] Calculate the volume ratio of the local neighborhood space to the global space for each sampling point; the local neighborhood space is the space containing all neighboring points of the sampling point; the global space is the space corresponding to the subset of point cloud data.

[0085] The global semantic information of the sampling points is determined based on the volume ratio.

[0086] Specifically, the volume ratio of the local neighborhood space to the global space of each sampling point is calculated, and the global semantic information of the sampling point is determined based on this volume ratio.

[0087] The local neighborhood space is the space comprised of all neighboring points of the sampled point. The global space is the space corresponding to the subset of point cloud data containing the sampled point.

[0088] The farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning provided in this application captures global semantic information by using the volume feature of the volume ratio between the local neighborhood space and the global space, which improves the ability to identify complex features and enhances the accuracy of feature identification in farmland scenes.

[0089] In some embodiments, determining the predicted category of the sampling point based on the local geometric features and the global semantic information includes:

[0090] The local geometric features are fused with the global semantic information to obtain the aggregated features of the sampling points;

[0091] Based on the aggregated features of the sampling points and the aggregated features of the neighborhood points of the sampling points, the predicted category of the sampling points is determined.

[0092] Specifically, the local geometric features of the sampling points are fused or aggregated with the global semantic information to obtain the aggregated features of the sampling points.

[0093] Then, in the point cloud classification stage, the aggregation features of the sampling point and its neighboring points are aggregated in combination with the neighborhood consistency constraint. Based on the aggregation features and the aggregation features of the sampling point's neighborhood points, the predicted category of the sampling point is determined.

[0094] The semantic segmentation method for farm 3D point clouds for unmanned agricultural machinery path planning provided in this application enhances the model's segmentation performance for multi-scale targets, its ability to identify large-scale targets, and especially its accurate identification of complex structures by aggregating local and global features. Classification is performed by aggregating the aggregated features of sampling points and their neighboring points, expanding the receptive field and improving classification accuracy and consistency. Combined with residual connections, feature loss is prevented and the model's classification consistency for large target regions is enhanced, effectively reducing classification errors in large target regions and significantly improving the identification ability of small-category targets such as irrigation facilities and hard surfaces.

[0095] In some embodiments, training a preset deep learning model based on the predicted category and the true category of the sampling point includes:

[0096] The cross-entropy loss is determined based on the predicted category and the true category of the sampling point;

[0097] The model parameters of the deep learning model are adjusted based on the cross-entropy loss.

[0098] Specifically, the cross-entropy loss is determined based on the predicted and true categories of the sampling points, and the model parameters of the deep learning model are adjusted based on the cross-entropy loss to obtain a trained 3D point cloud semantic segmentation model for the farm.

[0099] In this embodiment, the model can be optimized using the Adam optimizer with an initial learning rate of 0.01, trained for 100 epochs to ensure stable convergence. Hyperparameter tuning is performed by comparing the performance contributions of different modules, including setting the number of neighborhood points K (optimal value is 16), thereby achieving a balance between local feature capture and noise suppression.

[0100] The farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning provided in this application embodiment can use a cross-entropy weight adjustment strategy to solve the class imbalance problem and improve the recognition ability of small class targets (such as irrigation facilities and hard ground).

[0101] Figure 3 This is a schematic diagram of the structure of the farm 3D point cloud semantic segmentation model provided by the present invention, as shown below. Figure 3 As shown, in the encoding module (Encoder) of the farm's 3D point cloud semantic segmentation model, a random sampling module is used to perform voxel downsampling and block segmentation of the point cloud data to obtain a subset of point cloud data of consistent size. A feature fusion module is used to aggregate the local geometric features and global features of the sampling points to obtain aggregated features. In the decoding module (Decoder) of the farm's 3D point cloud semantic segmentation model, a nearest neighbor search algorithm such as K-nearest neighbor is used to obtain the local neighborhood set of the point cloud, construct a local space and extract local spatial features. A multilayer perceptron (MLP) is used to encode the local spatial features to obtain local geometric features. A classifier is used to aggregate the aggregated features of the sampling points and their neighboring points, and the land cover category of the sampling points is identified based on these features. The navigation accuracy requirement for agricultural machinery automatic driving is ±2.5cm, therefore, the accuracy requirements for point cloud data and land cover segmentation results are extremely high. The model structure design proposed in this application improves the model accuracy and robustness, enabling more accurate automatic driving navigation, etc.

[0102] In some embodiments, the method further includes:

[0103] Farmland semantic maps are generated based on the land cover types corresponding to different point clouds; the farmland semantic maps are used to display the three-dimensional distribution information of different land cover types in farmland.

[0104] Specifically, the semantically segmented point cloud data is visualized, and a farmland semantic map is generated based on the classification results (i.e., the land cover types corresponding to different point clouds). This farmland semantic map is used to display the three-dimensional distribution information of different land cover types in farmland, including the three-dimensional distribution information of farmland targets such as paddy fields, entrances and exits, roads, buildings, and irrigation facilities.

[0105] The farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning provided in this application displays the semantic segmentation results and transforms them into a farmland semantic map. This map can be applied to scenarios such as path planning, navigation, and task allocation for unmanned agricultural machinery, providing reliable ground information support for unmanned agricultural machinery, meeting the accuracy requirements of path planning and task allocation, significantly improving the efficiency and reliability of agricultural automation, and reducing the need for manual intervention.

[0106] Figure 4 This is a schematic diagram of the structure of the farm 3D point cloud semantic segmentation device for unmanned agricultural machinery path planning provided by the present invention, as shown below. Figure 4 As shown, the present invention provides a farm 3D point cloud semantic segmentation device for unmanned agricultural machinery path planning, including an acquisition module 401 and a semantic segmentation module 402.

[0107] The acquisition module 401 is used to acquire image data of farmland through a drone and convert the image data into three-dimensional point cloud data.

[0108] The semantic segmentation module 402 is used to input the 3D point cloud data into the farm's 3D point cloud semantic segmentation model for classification, thereby obtaining the land cover types corresponding to different point clouds; the farm's 3D point cloud semantic segmentation model is trained through the following steps:

[0109] Multiple subsets of point cloud data are obtained based on 3D point cloud data;

[0110] Extract the local geometric features and global semantic information of each sampling point in the point cloud data subset;

[0111] The predicted category of the sampling point is determined based on the local geometric features and the global semantic information;

[0112] A pre-defined deep learning model is trained based on the predicted category and the true category of the sampling point to obtain a semantic segmentation model of the farm's 3D point cloud.

[0113] In some embodiments, acquiring multiple subsets of point cloud data based on 3D point cloud data includes:

[0114] The three-dimensional point cloud data is processed based on the voxel mesh downsampling algorithm to obtain sample point cloud data;

[0115] The sample point cloud data is divided into multiple point cloud data subsets of the same size.

[0116] In some embodiments, extracting the local geometric features of each sampling point in the subset of point cloud data includes:

[0117] The K-nearest neighbor algorithm is used to obtain all neighboring points of each sampling point;

[0118] The local spatial features of the sampling point are obtained based on the sampling point and its neighboring points; the local spatial features include the distance decay index and the local geometric bounding box.

[0119] The local spatial features are encoded using a multilayer perceptron to obtain the local geometric features of the sampling points.

[0120] In some embodiments, global semantic information of each sampling point in the point cloud data subset is extracted, including:

[0121] Calculate the volume ratio of the local neighborhood space to the global space for each sampling point; the local neighborhood space is the space containing all neighboring points of the sampling point; the global space is the space corresponding to the subset of point cloud data.

[0122] The global semantic information of the sampling points is determined based on the volume ratio.

[0123] In some embodiments, determining the predicted category of the sampling point based on the local geometric features and the global semantic information includes:

[0124] The local geometric features are fused with the global semantic information to obtain the aggregated features of the sampling points;

[0125] Based on the aggregated features of the sampling points and the aggregated features of the neighborhood points of the sampling points, the predicted category of the sampling points is determined.

[0126] In some embodiments, training a preset deep learning model based on the predicted category and the true category of the sampling point includes:

[0127] The cross-entropy loss is determined based on the predicted category and the true category of the sampling point;

[0128] The model parameters of the deep learning model are adjusted based on the cross-entropy loss.

[0129] In some embodiments, it also includes:

[0130] The generation module is used to generate farmland semantic maps based on the land cover types corresponding to different point clouds; the farmland semantic maps are used to display the three-dimensional distribution information of different land cover types in farmland.

[0131] Specifically, the farm 3D point cloud semantic segmentation device for unmanned agricultural machinery path planning provided by the present invention can realize all the method steps implemented in the above-mentioned farm 3D point cloud semantic segmentation method embodiment for unmanned agricultural machinery path planning, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0132] It should be noted that the division of units / modules in the above embodiments of the present invention is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0133] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device may include: a processor 501, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504. The processor 501 can call logical instructions in the memory 503 to execute the farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning provided in the above-described method embodiments. This method includes:

[0134] Image data of farmland is acquired by drones and then converted into three-dimensional point cloud data;

[0135] The 3D point cloud data is input into the farm's 3D point cloud semantic segmentation model for classification to obtain the land cover types corresponding to different point clouds; the farm's 3D point cloud semantic segmentation model is trained through the following steps:

[0136] Multiple subsets of point cloud data are obtained based on 3D point cloud data;

[0137] Extract the local geometric features and global semantic information of each sampling point in the point cloud data subset;

[0138] The predicted category of the sampling point is determined based on the local geometric features and the global semantic information;

[0139] A pre-defined deep learning model is trained based on the predicted category and the true category of the sampling point to obtain a semantic segmentation model of the farm's 3D point cloud.

[0140] In some embodiments, acquiring multiple subsets of point cloud data based on 3D point cloud data includes:

[0141] The three-dimensional point cloud data is processed based on the voxel mesh downsampling algorithm to obtain sample point cloud data;

[0142] The sample point cloud data is divided into multiple point cloud data subsets of the same size.

[0143] In some embodiments, extracting the local geometric features of each sampling point in the subset of point cloud data includes:

[0144] The K-nearest neighbor algorithm is used to obtain all neighboring points of each sampling point;

[0145] The local spatial features of the sampling point are obtained based on the sampling point and its neighboring points; the local spatial features include the distance decay index and the local geometric bounding box.

[0146] The local spatial features are encoded using a multilayer perceptron to obtain the local geometric features of the sampling points.

[0147] In some embodiments, global semantic information of each sampling point in the point cloud data subset is extracted, including:

[0148] Calculate the volume ratio of the local neighborhood space to the global space for each sampling point; the local neighborhood space is the space containing all neighboring points of the sampling point; the global space is the space corresponding to the subset of point cloud data.

[0149] The global semantic information of the sampling points is determined based on the volume ratio.

[0150] In some embodiments, determining the predicted category of the sampling point based on the local geometric features and the global semantic information includes:

[0151] The local geometric features are fused with the global semantic information to obtain the aggregated features of the sampling points;

[0152] Based on the aggregated features of the sampling points and the aggregated features of the neighborhood points of the sampling points, the predicted category of the sampling points is determined.

[0153] In some embodiments, training a preset deep learning model based on the predicted category and the true category of the sampling point includes:

[0154] The cross-entropy loss is determined based on the predicted category and the true category of the sampling point;

[0155] The model parameters of the deep learning model are adjusted based on the cross-entropy loss.

[0156] In some embodiments, the method further includes:

[0157] Farmland semantic maps are generated based on the land cover types corresponding to different point clouds; the farmland semantic maps are used to display the three-dimensional distribution information of different land cover types in farmland.

[0158] Specifically, the processor 501 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD). The processor can also adopt a multi-core architecture.

[0159] When the logical instructions in memory 503 can be implemented as software functional units and sold or used as independent products, they can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] In some embodiments, a computer program product is also provided, the computer program product including a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning provided in the above method embodiments.

[0161] Specifically, the computer program product provided in this application embodiment can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0162] In some embodiments, a computer-readable storage medium is also provided, the computer-readable storage medium storing a computer program for causing a computer to execute the farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning provided in the above method embodiments.

[0163] Specifically, the computer-readable storage medium provided by the present invention can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0164] It should be noted that the computer-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical storage (e.g., CD, DVD, BD, HVD), and semiconductor storage (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0165] In this invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0166] In this invention, "determining B based on A" means that factor A must be considered when determining B. It is not limited to "B can be determined based solely on A," but should also include: "determining B based on A and C," "determining B based on A, C, and E," "determining C based on A, and further determining B based on C," etc. Additionally, it can include using A as a condition for determining B, for example, "when A satisfies the first condition, B is determined using the first method"; or "when A satisfies the second condition, B is determined," etc.; or "when A satisfies the third condition, B is determined based on the first parameter," etc. Of course, it can also be a condition where A is a factor in determining B, for example, "when A satisfies the first condition, C is determined using the first method, and B is further determined based on C," etc.

[0167] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0168] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-executable instructions. These computer-executable instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0169] These processor-executable instructions may also be stored in a processor-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the processor-readable memory produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0170] These processors can execute instructions that can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0171] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A semantic segmentation method for 3D point clouds in farms for unmanned agricultural machinery path planning, characterized in that, include: Image data of farmland is acquired by drones and then converted into three-dimensional point cloud data; The 3D point cloud data is input into the farm's 3D point cloud semantic segmentation model for classification to obtain the land cover types corresponding to different point clouds; the farm's 3D point cloud semantic segmentation model is trained through the following steps: Multiple subsets of point cloud data are obtained based on 3D point cloud data; Extract the local geometric features and global semantic information of each sampling point in the point cloud data subset; The predicted category of the sampling point is determined based on the local geometric features and the global semantic information; A preset deep learning model is trained based on the predicted category and the true category of the sampling point to obtain a semantic segmentation model of the farm's 3D point cloud. The extraction of local geometric features for each sampling point in the subset of point cloud data includes: The K-nearest neighbor algorithm is used to obtain all neighboring points of each sampling point; The local spatial features of the sampling point are obtained based on the sampling point and its neighboring points; the local spatial features include the distance decay index and the local geometric bounding box. The local spatial features are encoded using a multilayer perceptron to obtain the local geometric features of the sampling points; Extracting global semantic information from each sampling point in the point cloud data subset, including: Calculate the volume ratio of the local neighborhood space to the global space for each sampling point; the local neighborhood space is the space containing all neighboring points of the sampling point; the global space is the space corresponding to the subset of point cloud data. The global semantic information of the sampling points is determined based on the volume ratio.

2. The farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning according to claim 1, characterized in that, The acquisition of multiple subsets of point cloud data based on 3D point cloud data includes: The three-dimensional point cloud data is processed based on the voxel mesh downsampling algorithm to obtain sample point cloud data; The sample point cloud data is divided into multiple point cloud data subsets of the same size.

3. The farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning according to claim 1, characterized in that, Determining the predicted category of the sampling point based on the local geometric features and the global semantic information includes: The local geometric features are fused with the global semantic information to obtain the aggregated features of the sampling points; Based on the aggregated features of the sampling points and the aggregated features of the neighboring points of the sampling points, the predicted category of the sampling points is determined.

4. The farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning according to claim 1, characterized in that, The process of training a pre-defined deep learning model based on the predicted category and the true category of the sampling points includes: The cross-entropy loss is determined based on the predicted category and the true category of the sampling point; The model parameters of the deep learning model are adjusted based on the cross-entropy loss.

5. The farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning according to claim 1, characterized in that, The method further includes: Farmland semantic maps are generated based on the land cover types corresponding to different point clouds; the farmland semantic maps are used to display the three-dimensional distribution information of different land cover types in farmland.

6. A semantic segmentation device for farm 3D point clouds for unmanned agricultural machinery path planning, characterized in that, include: The acquisition module is used to acquire image data of farmland through a drone and convert the image data into three-dimensional point cloud data; The semantic segmentation module is used to input the 3D point cloud data into the farm's 3D point cloud semantic segmentation model for classification, thereby obtaining the land cover types corresponding to different point clouds. The farm's 3D point cloud semantic segmentation model is trained through the following steps: Multiple subsets of point cloud data are obtained based on 3D point cloud data; Extract the local geometric features and global semantic information of each sampling point in the point cloud data subset; The predicted category of the sampling point is determined based on the local geometric features and the global semantic information; A preset deep learning model is trained based on the predicted category and the true category of the sampling point to obtain a semantic segmentation model of the farm's 3D point cloud. The extraction of local geometric features for each sampling point in the subset of point cloud data includes: The K-nearest neighbor algorithm is used to obtain all neighboring points of each sampling point; The local spatial features of the sampling point are obtained based on the sampling point and its neighboring points; the local spatial features include the distance decay index and the local geometric bounding box. The local spatial features are encoded using a multilayer perceptron to obtain the local geometric features of the sampling points; Extracting global semantic information from each sampling point in the point cloud data subset, including: Calculate the volume ratio of the local neighborhood space to the global space for each sampling point; the local neighborhood space is the space containing all neighboring points of the sampling point; the global space is the space corresponding to the subset of point cloud data. The global semantic information of the sampling points is determined based on the volume ratio.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores a computer program that, when executed by a processor, implements the farm 3D point cloud semantic segmentation method for unmanned agricultural machinery path planning as described in any one of claims 1 to 5.

Images