Cotton point cloud segmentation picking method

Abstract

The invention discloses a cotton point cloud segmentation picking method, and belongs to the technical field of fruit and vegetable picking, and the method comprises the following steps: firstly, collecting an RGB image and a depth image of a cotton region through a depth camera, and generating three-dimensional point cloud data; and then a Point TransformerV3 model is used to carry out feature extraction and space segmentation on the point cloud data, and cotton fruits and backgrounds are accurately identified. And through a post-processing stage, optimizing the segmentation result, and finally outputting the spatial position and morphological characteristics of the cotton for controlling an automatic mechanical arm to perform efficient and accurate picking. According to the method, the problems of poor adaptability and low segmentation precision in a complex environment in a traditional point cloud segmentation method are solved, the automation level and precision of cotton picking are improved, and accurate cotton fruit identification and picking are realized.

Description

Technical Field

[0001] The invention relates to the technical field of fruit and vegetable picking, in particular to a cotton point cloud segmentation and picking method. Background Art

[0002] With the continuous development of agricultural automation and intelligence, fruit and vegetable picking robots have become an important tool for improving agricultural production efficiency. Traditional fruit and vegetable picking methods rely on manual operations, which are inefficient, labor-intensive, and difficult to ensure the quality and integrity of fruit picking. Existing automated picking technologies mostly rely on visual recognition and sensor data to locate and grasp target objects, but in complex agricultural environments, existing methods face problems such as lighting changes, complex backgrounds, and fruit and vegetable occlusion, resulting in picking accuracy and efficiency that are difficult to meet demand.

[0003] In recent years, point cloud data processing technology has been widely used in the field of robotics, especially in 3D object recognition and environmental perception. However, since point cloud data usually has problems such as noise and sparsity, how to efficiently and accurately process and analyze point cloud data remains a challenge. Traditional deep learning methods have difficulty in effectively capturing spatial relationships when processing irregular point cloud data, which affects the segmentation accuracy. Summary of the invention

[0004] The purpose of the present invention is to provide a cotton point cloud segmentation and picking method, which segments cotton point cloud data according to a depth camera and a PointTransformerV3 network model to achieve accurate cotton fruit recognition and picking. The present invention is achieved through the following technical solutions.

[0005] The present invention provides a cotton point cloud segmentation and picking method, comprising:

[0006] Obtain cotton point cloud data and perform preprocessing;

[0007] The preprocessed cotton point cloud data is subjected to feature extraction through the Point TransformerV3 network model to obtain the semantic features of the cotton point cloud data; the semantic features of the cotton point cloud data are segmented to obtain the cotton point cloud data including labels;

[0008] The cotton point cloud data including labels is optimized by using a statistical outlier removal algorithm to obtain optimized cotton point cloud data;

[0009] Based on the optimized cotton point cloud data, the robotic arm is controlled to perform the cotton picking operation.

[0010] In practical applications, the traditional deep learning method is difficult to effectively capture the spatial relationship when processing irregular point cloud data, which affects the segmentation accuracy and thus affects the cotton picking accuracy. The present invention extracts the semantic features in the cotton point cloud data through the Point TransformerV3 network model and segments it, obtaining a more accurate cotton point cloud data segmentation result containing labels, and accurately identifies the cotton fruit and background through the calculation of the probability formula. Secondly, the segmented cotton point cloud data is further optimized to increase the accuracy of cotton picking. The geometric coordinates of each point are obtained based on the three-dimensional coordinates of the optimized cotton point cloud data, and then the grasping posture of the cotton is estimated through the simulation of the geometric model. The geometric coordinates and grasping posture provide a basis for accurately grasping the target cotton when the robotic arm is actually used to pick cotton.

[0011] Optionally, the obtaining of cotton point cloud data includes: collecting an RGB image and a depth image of a cotton area to obtain complete point cloud data of the cotton area; the RGB image provides visual features of the cotton; the depth image provides a depth value of each pixel, indicating spatial position information of the cotton;

[0012] Use the camera intrinsic parameters to convert the depth value of each pixel into a three-dimensional coordinate;

[0013] Generate initial point cloud data of cotton area based on visual features and 3D coordinates of cotton .

[0014] Optionally, the preprocessing includes generating the initial point cloud data of the cotton area Perform noise removal, downsampling and normalization respectively;

[0015] The noise removal includes using a statistical outlier removal algorithm to clean the initial point cloud data of the cotton area The noise in the cotton point cloud is removed by calculating the distance between each point and its neighboring points. If the distance between the two exceeds the set threshold, the point is removed. If the distance between the two does not exceed the set threshold, the point is retained and the distance between other points and their neighboring points is calculated to obtain the denoised cotton point cloud data. ; Among them, the points that exceed the set threshold are noise points; the whole process will traverse all the points in the point cloud data and analyze the neighborhood of each point to ensure that the real cotton area points are retained while removing the noise points.

[0016] The downsampling includes using a VoxelGrid filter to remove the noise from the cotton point cloud data. The space is divided into multiple voxels, and the centroid of each voxel is retained to reduce the amount of cotton point cloud data, and the downsampled cotton point cloud data is obtained. ; By reducing the amount of cotton point cloud data, the amount of calculation of cotton point cloud data can be reduced;

[0017] The standardization process includes: Perform standardization processing to obtain standardized cotton point cloud data , ensuring that the standardized cotton point cloud data is adapted to the input of the Point TransformerV3 network model; the standardization formula is:

[0018] ,

[0019] In the formula, This is the cotton point cloud data after downsampling Mean, This is the cotton point cloud data after downsampling Standard deviation. Ensure that point cloud data is within a uniform scale range to improve processing stability.

[0020] Optionally, the Point TransformerV3 network model includes an input layer, an embedding layer, a multi-head self-attention layer, a Transformer encoder layer, a local and global feature aggregation layer, and an output layer connected in sequence; the Point TransformerV3 network model performs the following operations:

[0021] The input layer inputs the preprocessed cotton point cloud data into the embedding layer;

[0022] The embedding layer maps the cotton point cloud data from the input layer to a high-dimensional space, generates multiple high-dimensional embedding vectors and inputs them into the multi-head self-attention layer;

[0023] The multi-head self-attention layer compares multiple high-dimensional embedding vectors and calculates the attention weights between points, so that the cotton point cloud data can obtain a more refined feature representation and input it into the Transformer encoder layer;

[0024] The Transformer encoder layer uses the self-attention mechanism to perform deep learning and modeling on the cotton point cloud data output by the multi-head self-attention layer, so that the cotton point cloud data can extract complex semantic information and input it into the local and global feature aggregation layer;

[0025] The local and global feature aggregation layer uses a local feature extraction method to capture the local information of the cotton point cloud data output from the Transformer encoder layer, and combines the global features to extract the global information of the point cloud, and inputs the cotton point cloud data with the local and global information of the point cloud into the output layer; the local feature extraction method used is, for example, the kk-nearest neighbor aggregation method;

[0026] The output layer classifies the cotton point cloud data processed by the local and global feature aggregation layers according to the task type, and outputs the category or label of each point.

[0027] Optionally, the multi-head self-attention layer also captures the relationship between local and global features by calculating the similarity between points, including:

[0028] Standardized cotton point cloud data Query the characteristics of each point in ,key Sum Calculation:

[0029] ,

[0030] In the formula, Yes The characteristic vector of , and are all learned weight matrices;

[0031] Using Query and key The similarity between points is calculated by the following formula :

[0032] ,

[0033] In the formula, Yes Key Dimensions, based on similarity The attention weight between each point and all points is calculated by the following formula:

[0034] ,

[0035] In the formula, the softmax operation converts the similarity Transformed into probability distribution, representing the dependency between points, is the attention weight matrix between all points, which is used to measure the degree of attention of each point to other points and represents the similarity and dependency between points;

[0036] Weighted by attention weights , the updated features are obtained by the following formula :

[0037] .

[0038] The updated feature vector of each point It will be updated based on global information. Through the multi-head mechanism, the model can learn different types of dependencies in different subspaces, thereby more comprehensively understanding the structure of the point cloud.

[0039] Optionally, the Point TransformerV3 network model gradually extracts deeper cotton point cloud features by using multiple Transformer encoder layers. Each layer of Transformer encoder includes a multi-head self-attention layer and a feed-forward neural network FFN. The output of each layer of Transformer encoder is residually connected with the input, and then processed by layer normalization to ensure the stability of the network and the effective transmission of information. The final output feature of the Transformer encoder layer is calculated by the following formula:

[0040] ,

[0041] In the formula, is the output obtained through the self-attention mechanism, is a feed-forward network, It is the final output feature of the Transformer encoder layer, which contains complex semantic information. is the layer normalization operation.

[0042] By stacking multiple Transformer encoder layers, PointTransformer is able to gradually extract deeper point cloud features, thereby better understanding the geometry and spatial relationships of the point cloud.

[0043] Optionally, the output layer classifies the cotton point cloud data processed by the local and global feature aggregation layers according to the task type, including calculating the probability of each point belonging to different categories through the softmax activation function, performing two category calculations on each point, and selecting the category result corresponding to the maximum probability as the prediction result; each point The probability is calculated by the following formula:

[0044] ,

[0045] In the formula Yes The probability of Indicate point The category labels include cotton and background. Represents each point Specific category index, the value is 0 or 1, 0 is background, 1 is cotton, Yes The category score of Is the category index Score The index of Represents the sum of the indices for all category scores.

[0046] The category probability calculated by the softmax function can be used to classify each point, thereby achieving point cloud segmentation. The point cloud segmentation result is the classification label of each point. In the cotton picking task, the goal is to separate the cotton fruit from the background. Each point is ultimately assigned to the cotton or background class based on its category probability. Finally, the Point TransformerV3 network model is based on each point The class probability Output the segmentation result to distinguish cotton from the background.

[0047] Optionally, the optimization of the cotton point cloud data including labels by using a statistical outlier removal algorithm includes: by calculating the distance between each point and a neighboring point, if the distance exceeds a set threshold, the point is removed; if the distance does not exceed the set threshold, the point is retained, and the distances between other points and their neighboring points are continuously calculated, thereby removing noise and isolated points in the point cloud data to obtain optimized cotton point cloud data. , , Represents the three-dimensional coordinates of each point in the optimized cotton point cloud data. Indicates the serial number of the point in the standardized cotton point cloud data. The value range is , is the total number of points, , and They represent the horizontal coordinate, vertical coordinate and height of the midpoint of the cotton area point cloud data. Through geometric analysis and region merging, the problem of inaccurate segmentation results caused by overlapping areas between multiple fruits is solved.

[0048] Optionally, the step of controlling the robotic arm to perform the cotton picking operation based on the optimized cotton point cloud data includes:

[0049] From the optimized cotton point cloud data Extracting the geometric center coordinates of each cotton, and then estimating the grasping posture of the cotton through geometric model fitting based on the geometric center coordinates of each cotton, wherein the grasping posture includes a grasping angle and a direction;

[0050] According to the geometric center coordinates of each cotton and the corresponding grasping posture, the grasping path is planned to determine the grasping path of the robot arm; in the process of grasping path planning, the path planning cost is used Plan with the minimum as the goal.

[0051] Optionally, the path planning cost Calculated by the following formula:

[0052] ,

[0053] In the formula, is the path planning cost, which represents the overall cost of the robot from the starting position to the target position. The larger the distance between the points on the path, The larger the value, the greater the overall consumption of the robot arm, so it should be minimized in practical applications. .

[0054] Beneficial Effects

[0055] (1) By combining the powerful feature extraction capability of the Point TransformerV3 network model with the point cloud data collected by the depth camera, the proposed cotton point cloud segmentation and picking robot arm method not only achieves a significant improvement in cotton segmentation accuracy, but also ensures the efficient execution of the picking task through post-processing optimization.

[0056] (2) The present invention solves the problems of inaccurate target positioning and poor environmental adaptability in traditional picking methods by means of precise control and real-time feedback mechanism of the robotic arm, and provides technical support for the automation and intelligence of cotton picking. Through the implementation of the present invention, the efficiency and accuracy of cotton picking can be greatly improved, manual intervention can be reduced, and the picking cost can be reduced, thus contributing to the development of agricultural automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Shown is a flow chart of the cotton point cloud segmentation and picking method of the present invention;

[0059] Figure 2 The figure shows a schematic diagram of the cotton positioning by the robot arm of the present invention without using the Point TransformerV3 network model;

[0060] Figure 3 The figure shows a schematic diagram of the robot arm of the present invention using the Point TransformerV3 network model to locate cotton;

[0061] Figure 4 The figure shows a schematic diagram of the robot arm of the present invention picking cotton without using the Point TransformerV3 network model;

[0062] Figure 5 The figure shows a schematic diagram of the robot arm of the present invention picking cotton using the Point TransformerV3 network model. DETAILED DESCRIPTION

[0063] The following is a further description in conjunction with the accompanying drawings and specific embodiments. In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features.

[0064] Example 1

[0065] This embodiment provides a cotton point cloud segmentation and picking method, including:

[0066] Obtain cotton point cloud data and perform preprocessing;

[0067] The preprocessed cotton point cloud data is subjected to feature extraction through the Point TransformerV3 network model to obtain the semantic features of the cotton point cloud data; the semantic features of the cotton point cloud data are segmented to obtain the cotton point cloud data including labels;

[0068] The cotton point cloud data including labels is optimized by using a statistical outlier removal algorithm to obtain optimized cotton point cloud data;

[0069] Based on the optimized cotton point cloud data, the robotic arm is controlled to perform the cotton picking operation.

[0070] In practical applications, the traditional deep learning method is difficult to effectively capture the spatial relationship when processing irregular point cloud data, which affects the segmentation accuracy and thus affects the cotton picking accuracy. The present invention extracts the semantic features in the cotton point cloud data through the Point TransformerV3 network model and segments it, obtaining a more accurate cotton point cloud data segmentation result containing labels, and accurately identifies the cotton fruit and background through the calculation of the probability formula. Secondly, the segmented cotton point cloud data is further optimized to increase the accuracy of cotton picking. The geometric coordinates of each point are obtained based on the three-dimensional coordinates of the optimized cotton point cloud data, and then the grasping posture of the cotton is estimated through the simulation of the geometric model. The geometric coordinates and grasping posture provide a basis for accurately grasping the target cotton when the robotic arm is actually used to pick cotton.

[0071] Example 2

[0072] Based on Example 1, this example introduces a cotton point cloud segmentation and picking method process, such as Figure 1 As shown, specifically including the following:

[0073] The acquisition of cotton point cloud data includes: collecting RGB images and depth images of the cotton area to obtain complete cotton area point cloud data; the RGB image provides visual features of the cotton; the depth image provides a depth value of each pixel, indicating the spatial position information of the cotton;

[0074] Use the camera intrinsic parameters to convert the depth value of each pixel into a three-dimensional coordinate;

[0075] Generate initial point cloud data of cotton area based on visual features and 3D coordinates of cotton .

[0076] The preprocessing includes generating the initial point cloud data of the cotton area Perform noise removal, downsampling and normalization respectively;

[0077] The noise removal includes using a statistical outlier removal algorithm to clean the initial point cloud data of the cotton area The noise in the cotton point cloud is removed by calculating the distance between each point and its neighboring points. If the distance between the two exceeds the set threshold, the point is removed. If the distance between the two does not exceed the set threshold, the point is retained and the distance between other points and their neighboring points is calculated to obtain the denoised cotton point cloud data. ; Among them, the points that exceed the set threshold are noise points; the whole process will traverse all the points in the point cloud data and analyze the neighborhood of each point to ensure that the real cotton area points are retained while removing the noise points.

[0078] The downsampling includes using a VoxelGrid filter to remove the noise from the cotton point cloud data. The space is divided into multiple voxels, and the centroid of each voxel is retained to reduce the amount of cotton point cloud data, and the downsampled cotton point cloud data is obtained. ; By reducing the amount of cotton point cloud data, the amount of calculation of cotton point cloud data can be reduced;

[0079] The standardization process includes: Perform standardization processing to obtain standardized cotton point cloud data , ensuring that the standardized cotton point cloud data is adapted to the input of the Point TransformerV3 network model; the standardization formula is:

[0080] ,

[0081] In the formula, This is the cotton point cloud data after downsampling Mean, This is the cotton point cloud data after downsampling Standard deviation. Ensure that point cloud data is within a uniform scale range to improve processing stability.

[0082] The Point TransformerV3 network model includes an input layer, an embedding layer, a multi-head self-attention layer, a Transformer encoder layer, a local and global feature aggregation layer, and an output layer connected in sequence; the Point TransformerV3 network model performs the following operations:

[0083] The input layer inputs the preprocessed cotton point cloud data into the embedding layer;

[0084] The embedding layer maps the cotton point cloud data from the input layer to a high-dimensional space, generates multiple high-dimensional embedding vectors and inputs them into the multi-head self-attention layer;

[0085] The multi-head self-attention layer compares multiple high-dimensional embedding vectors and calculates the attention weights between points, so that the cotton point cloud data can obtain a more refined feature representation and input it into the Transformer encoder layer;

[0086] The Transformer encoder layer uses the self-attention mechanism to perform deep learning and modeling on the cotton point cloud data output by the multi-head self-attention layer, so that the cotton point cloud data can extract complex semantic information and input it into the local and global feature aggregation layer;

[0087] The local and global feature aggregation layer uses a local feature extraction method to capture the local information of the cotton point cloud data output from the Transformer encoder layer, and combines the global features to extract the global information of the point cloud, and inputs the cotton point cloud data with the local and global information of the point cloud into the output layer; the local feature extraction method used is, for example, the kk-nearest neighbor aggregation method;

[0088] The output layer classifies the cotton point cloud data processed by the local and global feature aggregation layers according to the task type, and outputs the category or label of each point.

[0089] The multi-head self-attention layer also captures the relationship between local and global features by calculating the similarity between points, including:

[0090] Standardized cotton point cloud data Query the characteristics of each point in ,key Sum Calculation:

[0091] ,

[0092] In the formula, Yes The characteristic vector of , and are all learned weight matrices;

[0093] Using Query and key The similarity between points is calculated by the following formula :

[0094] ,

[0095] In the formula, Yes Key Dimensions, based on similarity The attention weight between each point and all points is calculated by the following formula:

[0096] ,

[0097] In the formula, the softmax operation converts the similarity Transformed into probability distribution, representing the dependency between points, is the attention weight matrix between all points, which is used to measure the degree of attention of each point to other points and represents the similarity and dependency between points;

[0098] Weighted by attention weights , the updated features are obtained by the following formula :

[0099] .

[0100] The updated feature vector of each point It will be updated based on global information. Through the multi-head mechanism, the model can learn different types of dependencies in different subspaces, thereby more comprehensively understanding the structure of the point cloud.

[0101] The Point TransformerV3 network model gradually extracts deeper cotton point cloud features by using multiple Transformer encoder layers. Each layer of Transformer encoder includes a multi-head self-attention layer and a feed-forward neural network FFN. The output of each layer of Transformer encoder is residually connected with the input, and then processed by layer normalization to ensure the stability of the network and the effective transmission of information. The final output feature of the Transformer encoder layer is calculated by the following formula:

[0102] ,

[0103] In the formula, is the output obtained through the self-attention mechanism, is a feed-forward network, It is the final output feature of the Transformer encoder layer, which contains complex semantic information. is the layer normalization operation.

[0104] By stacking multiple Transformer encoder layers, PointTransformer is able to gradually extract deeper point cloud features, thereby better understanding the geometry and spatial relationships of the point cloud.

[0105] The output layer classifies the cotton point cloud data processed by the local and global feature aggregation layers according to the task type, including calculating the probability of each point belonging to different categories through the softmax activation function, performing two category calculations on each point, and selecting the category result corresponding to the maximum probability as the prediction result; each point The probability is calculated by the following formula:

[0106] ,

[0107] In the formula Yes The probability of Indicate point The category labels include cotton and background. Represents each point Specific category index, the value is 0 or 1, 0 is background, 1 is cotton, Yes The category score of Is the category index Score The index of Represents the sum of the indices for all category scores.

[0108] The category probability calculated by the softmax function can be used to classify each point, thereby achieving point cloud segmentation. The point cloud segmentation result is the classification label of each point. In the cotton picking task, the goal is to separate the cotton fruit from the background. Each point is ultimately assigned to the cotton or background class based on its category probability. Finally, the Point TransformerV3 network model is based on each point The class probability Output the segmentation result to distinguish cotton from the background.

[0109] The optimization of the cotton point cloud data including labels by the statistical outlier removal algorithm includes: by calculating the distance between each point and the neighboring points, if it exceeds the set threshold, the point is removed, if it does not exceed the set threshold, the point is retained, and the distances of other points and their neighboring points are continuously calculated, thereby removing noise and isolated points in the point cloud data, and obtaining the optimized cotton point cloud data , , Represents the three-dimensional coordinates of each point in the optimized cotton point cloud data. Indicates the serial number of the point in the standardized cotton point cloud data. The value range is , is the total number of points, , and They represent the horizontal coordinate, vertical coordinate and height of the midpoint of the cotton area point cloud data. Through geometric analysis and region merging, the problem of inaccurate segmentation results caused by overlapping areas between multiple fruits is solved.

[0110] The operation of controlling the robotic arm to perform cotton picking based on the optimized cotton point cloud data includes:

[0111] From the optimized cotton point cloud data Extracting the geometric center coordinates of each cotton, and then estimating the grasping posture of the cotton through geometric model fitting based on the geometric center coordinates of each cotton, wherein the grasping posture includes a grasping angle and a direction;

[0112] According to the geometric center coordinates of each cotton and the corresponding grasping posture, the grasping path is planned to determine the grasping path of the robot arm; in the process of grasping path planning, the path planning cost is used Plan with the minimum as the goal.

[0113] The path planning cost Calculated by the following formula:

[0114] ,

[0115] In the formula, is the path planning cost, which represents the overall cost of the robot from the starting position to the target position. The larger the distance between the points on the path, The larger the value, the greater the overall consumption of the robot arm, so it should be minimized in practical applications. .

[0116] When picking cotton, the robot control system accurately plans the picking path based on the segmentation results through the inverse kinematics algorithm. By integrating the environmental perception feedback mechanism, the robot can adjust the motion trajectory in real time, avoid obstacles, and ensure the efficiency and safety of the picking process. The robot generates grasping points and calculates the appropriate posture based on the point cloud segmentation results, so as to perform high-precision picking in a dynamic environment. During the entire picking process, the system can adaptively handle different lighting, postures and environmental complexities, improving the automation level and production efficiency of cotton picking.

[0117] Combination Figure 2 and Figure 3 , respectively, are the comparisons of cotton positioning without using the Point TransformerV3 network model and with the Point TransformerV3 network model. It can be seen that when the Point TransformerV3 network model is used for cotton positioning, the recognition result of cotton is more accurate, which not only significantly reduces the misjudgment of background noise, but also can more accurately locate the center point of cotton, improving the accuracy of cotton target segmentation and positioning. This shows that the Point TransformerV3 network effectively captures the local and global correlation information in the three-dimensional point cloud data through its efficient feature extraction and attention mechanism, thereby optimizing the positioning and recognition effect of cotton.

[0118] Combination Figure 4 and Figure 5 , respectively, are the comparisons of cotton picking without using the Point TransformerV3 network model and with using the Point TransformerV3 network model. It can be seen that when the Point TransformerV3 network model is not used, the positioning accuracy of the cotton grasping point is low, and it is easy to offset or misjudgment, resulting in grasping failure or affecting the picking efficiency. After using the Point TransformerV3 network model, through its efficient feature extraction capability and self-attention mechanism, the network can more accurately identify the geometric structure and spatial distribution characteristics of cotton, thereby accurately locating the center point of the cotton and optimizing the grasping action path of the robot arm.

[0119] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A cotton point cloud segmentation and picking method, characterized in that: include: Obtain cotton point cloud data and perform preprocessing; The preprocessed cotton point cloud data is subjected to feature extraction through the Point TransformerV3 network model to obtain the semantic features of the cotton point cloud data; the semantic features of the cotton point cloud data are segmented to obtain the cotton point cloud data including labels; The cotton point cloud data including labels is optimized by using a statistical outlier removal algorithm to obtain optimized cotton point cloud data; Based on the optimized cotton point cloud data, the robotic arm is controlled to perform the cotton picking operation.

2. The cotton point cloud segmentation and picking method according to claim 1 is characterized in that: The acquisition of cotton point cloud data includes: collecting RGB images and depth images of the cotton area to obtain complete cotton area point cloud data; the RGB image provides visual features of the cotton; the depth image provides a depth value of each pixel, indicating the spatial position information of the cotton; Use the camera intrinsic parameters to convert the depth value of each pixel into a three-dimensional coordinate; Generate initial point cloud data of cotton area based on visual features and 3D coordinates of cotton .

3. The cotton point cloud segmentation and picking method according to claim 2 is characterized in that: The preprocessing includes generating the initial point cloud data of the cotton area Perform noise removal, downsampling and normalization respectively; The noise removal includes using a statistical outlier removal algorithm to clean the initial point cloud data of the cotton area The noise in the cotton point cloud is removed by calculating the distance between each point and its neighboring points. If the distance between the two exceeds the set threshold, the point is removed. If the distance between the two does not exceed the set threshold, the point is retained and the distance between other points and their neighboring points is calculated to obtain the denoised cotton point cloud data. ; Among them, the points exceeding the set threshold are noise points; the downsampling includes using a VoxelGrid filter to remove the noise from the cotton point cloud data The space is divided into multiple voxels, and the centroid of each voxel is retained to reduce the amount of cotton point cloud data, and the downsampled cotton point cloud data is obtained. ; The standardization process includes: Perform standardization processing to obtain standardized cotton point cloud data , ensuring that the standardized cotton point cloud data is adapted to the input of the Point TransformerV3 network model; the standardization formula is: ， In the formula, This is the cotton point cloud data after downsampling Mean, This is the cotton point cloud data after downsampling Standard deviation.

4. The cotton point cloud segmentation and picking method according to claim 1 is characterized in that: The PointTransformerV3 network model includes an input layer, an embedding layer, a multi-head self-attention layer, a Transformer encoder layer, a local and global feature aggregation layer, and an output layer connected in sequence; the Point TransformerV3 network model performs the following operations: The input layer inputs the preprocessed cotton point cloud data into the embedding layer; The embedding layer maps the cotton point cloud data from the input layer to a high-dimensional space, generates multiple high-dimensional embedding vectors and inputs them into the multi-head self-attention layer; The multi-head self-attention layer compares multiple high-dimensional embedding vectors and calculates the attention weights between points, so that the cotton point cloud data can obtain a more refined feature representation and input it into the Transformer encoder layer; The Transformer encoder layer uses the self-attention mechanism to perform deep learning and modeling on the cotton point cloud data output by the multi-head self-attention layer, so that the cotton point cloud data can extract complex semantic information and input it into the local and global feature aggregation layer; The local and global feature aggregation layer uses the local feature extraction method to capture the local information of the cotton point cloud data output from the Transformer encoder layer, and combines the global features to extract the global information of the point cloud. The cotton point cloud data with the captured local and global information of the point cloud is input into the output layer. The output layer classifies the cotton point cloud data processed by the local and global feature aggregation layers according to the task type, and outputs the category or label of each point.

5. The cotton point cloud segmentation and picking method according to claim 4 is characterized in that: The multi-head self-attention layer also captures the relationship between local and global features by calculating the similarity between points, including: Standardized cotton point cloud data Query the characteristics of each point in ,key Sum Calculation: , In the formula, Yes The characteristic vector of , and are all learned weight matrices; Using Query and key The similarity between points is calculated by the following formula : ， In the formula, Yes Key Dimensions, based on similarity The attention weight between each point and all points is calculated by the following formula: ， In the formula, the softmax operation converts the similarity Transformed into probability distribution, representing the dependency between points, is the attention weight matrix between all points, which is used to measure the degree of attention of each point to other points and represents the similarity and dependency between points; Weighted by attention weights , the updated features are obtained by the following formula : 。 6. The cotton point cloud segmentation and picking method according to claim 5 is characterized in that: The PointTransformerV3 network model gradually extracts deeper cotton point cloud features by using multiple Transformer encoder layers. Each layer of Transformer encoder includes a multi-head self-attention layer and a feedforward neural network FFN. The output of each layer of Transformer encoder is residually connected with the input, and then processed by layer normalization. The final output feature of the Transformer encoder layer is calculated by the following formula: ， In the formula, is the output obtained through the self-attention mechanism, is a feed-forward network, It is the final output feature of the Transformer encoder layer, which contains complex semantic information. is the layer normalization operation.

7. The cotton point cloud segmentation and picking method according to claim 6 is characterized in that: The output layer classifies the cotton point cloud data processed by the local and global feature aggregation layers according to the task type, including calculating the probability of each point belonging to different categories through the softmax activation function, performing two category calculations on each point, and selecting the category result corresponding to the maximum probability as the prediction result; each point The probability is calculated by the following formula: ， In the formula Yes The probability of Indicate point The category labels include cotton and background. Represents each point Specific category index, the value is 0 or 1, 0 is background, 1 is cotton, Yes The category score of Is the category index Score The index of Represents the sum of the indices for all category scores.

8. The cotton point cloud segmentation and picking method according to claim 7 is characterized in that: The optimization of the cotton point cloud data including labels by the statistical outlier removal algorithm includes: by calculating the distance between each point and the neighboring points, if it exceeds the set threshold, the point is removed, if it does not exceed the set threshold, the point is retained, and the distances of other points and their neighboring points are continuously calculated, thereby removing noise and isolated points in the point cloud data, and obtaining the optimized cotton point cloud data , , Represents the three-dimensional coordinates of each point in the optimized cotton point cloud data. Indicates the serial number of the point in the standardized cotton point cloud data. The value range is , is the total number of points, , and They respectively represent the horizontal coordinate, vertical coordinate and height of the midpoint of the cotton area point cloud data.

9. The cotton point cloud segmentation and picking method according to claim 8, characterized in that: The operation of controlling the robotic arm to perform cotton picking based on the optimized cotton point cloud data includes: From the optimized cotton point cloud data Extracting the geometric center coordinates of each cotton, and then estimating the grasping posture of the cotton through geometric model fitting based on the geometric center coordinates of each cotton, wherein the grasping posture includes a grasping angle and a direction; According to the geometric center coordinates of each cotton and the corresponding grasping posture, the grasping path is planned to determine the grasping path of the robot arm; in the process of grasping path planning, the path planning cost is used Plan with the minimum as the goal.

10. The cotton point cloud segmentation and picking method according to claim 9, characterized in that: The path planning cost Calculated by the following formula: ， In the formula, is the path planning cost, which represents the overall cost of the robot from the starting position to the target position.

Images