A method for detecting young tea shoots based on feature fusion graph neural network

Abstract

The present invention provides a method for detecting young tea leaves based on a feature fusion graph neural network. The method relates to the field of intelligent agriculture, and comprises the following steps: step (1): acquiring data, using the light trap described in claim 1 to obtain a point cloud and an RGB image of the target to be detected. step (2): using CNN to extract features from the two-dimensional image. step (3): applying a key point detection algorithm to the point cloud to detect key points. step (4): fusing image features with point cloud features. step (5): establishing a graph structure, using the key points detected in step (3) as vertices and building a graph. step (6): using a graph neural network to predict the position of young tea leaves. step (7): generating a prediction frame. step (8): using a soft-NMS algorithm to generate the best prediction frame. The present invention overcomes the occlusion problem in the detection of young tea leaves by using a light trap point cloud acquisition system and uses an improved graph neural network to improve detection accuracy, thereby achieving accurate identification of young tea leaves.

Description

Technical Field

[0001] The present invention belongs to the field of agricultural intelligence, and specifically relates to a tea shoot detection system and method based on 360-degree three-dimensional imaging of a single TOF camera. Background Art

[0002] Intelligent tea picking is crucial for reducing the workload. Current research on tea bud detection primarily focuses on two-dimensional (2D) images, combining traditional algorithms with 2D images and deep learning algorithms. Traditional algorithms extract tea bud features, then segment the bud image using a threshold based on super-green features. Finally, the entire tea shoot is identified. Deep learning algorithms use general object detection algorithms to classify and locate tea shoots.

[0003] However, the above technologies still have the following drawbacks: ① Due to the growth characteristics of tea leaves, the 2D image-based tea shoot detection method is less effective when detecting occluded objects, and cannot detect obscured buds and leaves. ② Traditional algorithms and deep learning algorithms perform poorly when processing unordered point cloud data, and cannot achieve high detection accuracy. Summary of the Invention

[0004] The present invention improves the problems that the existing technology has poor reconstruction effect of occluded objects during three-dimensional reconstruction and the accuracy of point cloud target detection using only convolutional neural networks has a certain upper limit. A tea shoot detection system and method based on 360-degree three-dimensional imaging of a single TOF camera are proposed.

[0005] The present invention proposes a method for detecting young tea leaves based on a feature fusion graph neural network. The technical solution includes the following steps:

[0006] Step (1): Data acquisition, using a light trap to obtain the point cloud and RGB image of the target to be detected;

[0007] Step (2): Use convolutional neural network (CNN) to extract features from the two-dimensional image;

[0008] Step (3): Apply key point detection algorithm to the point cloud to detect key points;

[0009] Step (4): Fusion of image features and point cloud features;

[0010] Step (5): Build a graph structure, using the key points detected in step (3) as vertices and build a graph;

[0011] Step (6): Use graph neural network to predict the location of tea leaves;

[0012] Step (7): Generate prediction box;

[0013] Step (8): The soft-NMS algorithm generates the best prediction box.

[0014] The purpose of this invention is to use a ToF camera in combination with a light trap to perform three-dimensional reconstruction of occluded objects, and combine it with an improved feature fusion graph neural network to change the weakness of the traditional tea shoot detection method in terms of poor detection ability for occluded targets, thereby improving the detection accuracy.

[0015] The specific steps are as follows:

[0016] Step (1): Acquisition of point cloud

[0017] We designed an inverted pyramid-shaped light trap consisting of four plane mirrors. The bottom of the inverted pyramid is open to allow enough light to enter the light trap. Another plane mirror is placed above the light trap at a 45-degree angle. A time-of-flight sensor is used to emit a light beam to the plane mirror, placing the observed object at the bottom of the inverted pyramid. After multiple reflections, the light beam can cover more than 99% of the object's surface area and obtain the three-dimensional coordinates of all points.

[0018]

[0019] d total Indicates the total distance traveled by the light from the ToF sensor position to the surface of the target object. k It represents the distance traveled by the light between two reflections, and d represents the distance from the last time the light beam touches the mirror to the surface of the object being measured.

[0020]

[0021] d means Is the total distance measured by the ToF sensor. Due to the reflection theorem of light, the light beam will return to the sensor along the original path, so the actual distance is half of the measured distance. k It can also be calculated as the position of the beam and the next mirror reflection minus the position of the previous mirror reflection, that is, d k =p k+1 -p k , where p k is the reflection point of the light at the mirror surface for the kth time, and p k+1 can be Calculated, is the unit direction vector along the direction of the reflected light beam, which we can calculate from Snell's law

[0022]

[0023] n kis the normal direction of the mirror. When we know the initial position p0 of ToF and the direction vector r0 of the initial incident light, we can recursively calculate p for all k∈[1...K] k and In this way, the three-dimensional coordinates of the object under test can be obtained, and further the three-dimensional point cloud of the object under test can be obtained. The reflection trajectory of the light in the light trap is as follows: Figure 2 shown.

[0024] Step (2): Use CNN to extract features from 2D images

[0025] Input a 2D image into a pretrained ResNet50 network to extract image features. ResNet50 includes an identity block and a conv block. It incorporates residual learning into traditional networks, addressing the issues of gradient dispersion and accuracy loss in deep networks. This allows networks to become deeper, ensuring both accuracy and speed. This example uses ResNet50 to extract features from an image, ultimately generating a feature vector with a depth of 2048.

[0026] Step (3): Apply key point detection algorithm to the point cloud to detect key points

[0027] This method uses a keypoint extraction algorithm based on point cloud saliency. The algorithm consists of a saliency calculation module and a keypoint generation module. We define saliency as the combination of feature activation signals at a specific layer of a pre-trained descriptor and the gradient of the same layer with respect to the input point cloud. We leverage the pre-trained descriptor network to generate robust keypoints directly from 3D data. We then project the saliency into the spherical coordinate system of the input point cloud and extract the most informative regions. We combine these regions with features from the original descriptor to extract robust and repeatable keypoints.

[0028] The key point detection algorithm obtains the final key point prediction by fusing the feature vector obtained by the point cloud saliency method, the original features of each point, and the point cloud context features, and inputting them into two fully connected layers.

[0029] Step (4): Fusion of image features and point cloud features

[0030] The two-dimensional image features obtained by using resnet50 in step (2) and the point cloud features in step (3) are fused together through the concatenate function to obtain a new feature vector. The fused new feature vector is used as the initial feature of the subsequent graph structure.

[0031] Step (5): Establish graph structure

[0032] A graph structure consists of two elements: edges and vertices, that is, G = (V, E), where V represents the vertex of the graph structure V = {v1, v2, ..., v n}, take the key point detected in step (2) as vertex v i E is the edge connecting the vertices, which can be obtained based on certain conditions:

[0033] E={(v i ,v j )|||x i -x j ||2<r}

[0034] Where x i with x j are the horizontal coordinates of different vertices, and r is the distance between two vertices. The features of each vertex not only include the initial features obtained by the previous fusion but also the features of the edges associated with it, that is, the features of the vertex are represented by V i =V0 is updated to

[0035] V i =[V0,V1,V2,...,V 128 ]

[0036] V0,V1,V2,...,V 128 For the remaining vertices associated with the vertex, V i is the updated vertex. By continuously updating the features of the vertices, we can determine the category of the object in the local area based on the vertices.

[0037] Step (6): Use graph neural network to predict the location of tea leaves and picking points

[0038] (a) First, manually calibrate the target frame of the tea leaves in the 3D point cloud collected in step (1) so that the target frame can just surround the object we need.

[0039] (b) The graph structure obtained in step (5) is used as the input of the network. The graph neural network updates the vertex features by aggregating the edge features. After t+1 iterations, the vertex and edge features are updated to

[0040]

[0041]

[0042] in and are the features of vertices and edges after t iterations. t The (.) function calculates the characteristics of the edge between two vertices, ρ(.) aggregates the characteristics of the edge to the vertex, gt (.) Calculate the vertex features after aggregating the edge features. Then associate a feature extraction function MLP for each vertex, and you can get:

[0043]

[0044]

[0045] After t iterations, we can use the obtained vertex features to predict the category of the target and obtain the regression box of the target.

[0046] Step (7): Generate prediction box

[0047] We get the prediction box by predicting a four-degree-of-freedom vector b = (x, y, z, l, h, w, θ). The (x, y, z) coordinates are the center coordinates of the regression box. The (h, w, θ) are the length, width, and angle of the predicted bounding box. In the initial classification, the vertices are divided into foreground points and background points. The foreground points contain the information we need. If the vertex falls in the foreground point, it will be sent to the MLP responsible for position prediction. loc (Multi-layer Perceptron for position prediction) and Huber Loss are applied to predict the regression box. Since there may be multiple vertices on the same object, multiple prediction boxes will be generated for the same object. The next step is to use an algorithm to obtain the best prediction box.

[0048] Step (8): Soft-NMS algorithm generates the best prediction box

[0049] The soft-NMS algorithm is used to remove the redundant prediction boxes generated in step (5). The NMS algorithm iteratively selects the most appropriate prediction box by calculating the IoU between the prediction box Bounding Box and the ground truth.

[0050]

[0051] Soft-NMS introduces a penalty coefficient λ to filter the most appropriate prediction box by reducing the confidence level.

[0052]

[0053]

[0054] S i =λS i

[0055] Where S i is the detection box B iThe confidence score of , Threshold and δ are two hyper parameters. After calculation, we can get the final prediction result.

[0056] The beneficial effects of the present invention are as follows: The present invention uses light traps and ToF sensors to obtain tea point clouds, overcoming the drawback of traditional point cloud acquisition methods that are unable to obtain point clouds for occluded targets. Combined with graph neural networks, the present invention improves the detection accuracy of targets, which has important engineering practical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is the technical route of the present invention;

[0058] Figure 2 The light trap point cloud acquisition device involved in the present invention;

[0059] Figure 3 A schematic diagram of the reflection of the light beam emitted by the ToF sensor involved in the present invention in the light trap;

[0060] Figure 4 The present invention relates to a point cloud key point detection algorithm;

[0061] Figure 5 This is a diagram of the graph neural network structure involved in the present invention. DETAILED DESCRIPTION

[0062] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0063] Step (1): Acquisition of point cloud

[0064] We designed an inverted pyramid-shaped light trap consisting of four equilateral triangular plane mirrors with sides of 1.5 meters. The bottom of the inverted pyramid is open to allow sufficient light to enter the light trap. Another 50cm*80cm rectangular plane mirror is placed above the light trap at a 45-degree angle to reflect the light beam emitted by the sensor. A time-of-flight sensor is used to transmit a light beam to the plane mirror, and the observed object is placed at the bottom of the inverted pyramid. After multiple reflections, the light beam can cover more than 99% of the object's surface area, and the three-dimensional coordinates of all points are obtained.

[0065]

[0066] d total Indicates the total distance traveled by the light from the ToF sensor position to the surface of the target object. kIt represents the distance traveled by the light between two reflections, and d represents the distance from the last time the light beam touches the mirror to the surface of the object being measured.

[0067]

[0068] d means Is the total distance measured by the ToF sensor. Due to the reflection theorem of light, the light beam will return to the sensor along the original path, so the actual distance is half of the measured distance. k It can also be calculated as the position of the beam and the next mirror reflection minus the position of the previous mirror reflection, that is, d k =p k+1 -p k , and p k+1 can be Calculated, is the unit direction vector along the direction of the reflected light beam, which we can calculate from Snell's law

[0069]

[0070] n k is the normal direction of the mirror. When we know the initial position p0 of ToF and the direction vector r0 of the initial incident light, we can recursively calculate p for all k∈[1...K] k and In this way, the three-dimensional coordinates of the object under test can be obtained, and further the three-dimensional point cloud of the object under test can be obtained. The reflection trajectory of the light in the light trap is as follows: Figure 4 shown.

[0071] Step (2): Use CNN to extract features from 2D images

[0072] Input a 2D image into a pretrained ResNet50 network to extract image features. ResNet50 includes an identity block and a conv block. It incorporates residual learning into traditional networks, addressing the issues of gradient dispersion and accuracy loss in deep networks. This allows networks to become deeper, ensuring both accuracy and speed. This example uses ResNet50 to extract features from an image, ultimately generating a feature vector with a depth of 2048.

[0073] Step (3): Apply key point detection algorithm to the point cloud to detect key points

[0074] This method uses a keypoint extraction algorithm based on point cloud saliency. The algorithm consists of a saliency calculation module and a keypoint generation module. We define saliency as a combination of the feature activation signal on a specific layer of the pre-trained descriptor and the gradient of the same layer with respect to the input point cloud. We use the pre-trained descriptor network to generate robust keypoints directly from 3D data. The saliency is then projected into the spherical coordinate system of the input point cloud and the regions with the most information are extracted. We combine these regions with the features of the original descriptor to extract robust and repeatable keypoints. In the point cloud saliency algorithm, for a given point cloud with N points We extract the i ∈P} N The gradient of the pre-trained network ▽F is defined as Then the initial significance is defined as S(P), and S(P) = F l P ·▽F l P , and calculate the second significance score s by the initial significance score i , this score is calculated by the distance from the center of the point cloud in spherical coordinates and is defined as:

[0075]

[0076] where j∈{1,2} is defined as the i ∈P is the x,y,z of each Cartesian coordinate system. is defined as the distance from point i to the center of the point cloud. In other words, we weight the initial saliency score by the offset from the point cloud median. Therefore, points farther away receive higher scores, while points closer to the center receive lower scores. Finally, the mean and variance of the input saliency are normalized.

[0077] The keypoint generation module takes saliency, raw descriptors, and point cloud context as input to produce the final keypoint predictions. The raw descriptors are obtained using a pretrained 3DFeatureNet and their dimensionality is reduced using PCA to produce a smoother feature space. In this example, the point cloud context features are generated using four pretrained X-Conv layers and two fully connected layers. Finally, the saliency, raw descriptors, and point cloud are combined into a feature vector using the concatenate function and fed into two fully connected networks to produce the final keypoint predictions.

[0078] Step (4): Fusion of image features and point cloud features

[0079] The two-dimensional image features obtained by using resnet50 in step (2) and the point cloud features in step (3) are fused together through the concatenate function to obtain a new feature vector of n×3136 dimensions. The fused new feature vector is used as the initial feature of the subsequent graph structure.

[0080] Step (5): Establish graph structure

[0081] A graph structure consists of two elements: edges and vertices, that is, G = (V, E), where V represents the vertex of the graph structure V = {v1, v2, ..., v n}, take each point in the point cloud as a vertex v i E is the edge connecting the vertices, which can be obtained based on certain conditions:

[0082] E={(v i ,v j )|||x i -x j ||2<r}

[0083] Among them, r is selected as 0.4m, and the upper limit of a vertex edge is set to 128. If it does not reach 128, it will be left empty. The features of each vertex include not only its own original features but also the features of the edges associated with it. That is, the features of the vertex are composed of V i =V0 is updated to

[0084] V i =[V0,V1,V2,...,V 128 ]

[0085] By continuously updating the features of vertices, we can determine the category of objects in the local area based on the vertices.

[0086] Step (6): Use graph neural network to predict the location of tea leaves and picking points

[0087] (a) First, manually calibrate the target frame of the tea leaves in the 3D point cloud collected in step (1) so that the target frame can just surround the object we need.

[0088] (b) The graph structure obtained in step (3) is used as the input of the network. The graph neural network updates the vertex features by aggregating the edge features. After t+1 iterations, the vertex and edge features are updated to

[0089]

[0090]

[0091] where v t and et are the features of vertices and edges after t iterations. t The (.) function calculates the characteristics of the edge between two vertices, ρ(.) aggregates the characteristics of the edge to the vertex, g t (.) Calculate the vertex features after aggregating the edge features. Then associate a feature extraction function MLP for each vertex, and you can get:

[0092]

[0093]

[0094] After t iterations, we can use the obtained vertex features to predict the category of the target and obtain the regression box of the target.

[0095] Step (7): Generate prediction box

[0096] We get the prediction box by predicting a four-degree-of-freedom vector b = (x, y, z, l, h, w, θ). The (x, y, z) coordinates are the center coordinates of the regression box. In the initial classification, the vertices are divided into foreground points and background points. The foreground points contain the information we need. If the vertex falls in the foreground point, it will be sent to the MLP responsible for position prediction. loc , Huber Loss is applied to predict the regression box. Since there may be multiple vertices on the same object, multiple prediction boxes will be generated for the same object. The next step is to use an algorithm to obtain the best prediction box.

[0097] Step (8): Soft-NMS algorithm generates the best prediction box

[0098] The soft-NMS algorithm is used to remove the redundant prediction boxes generated in step (5). The NMS algorithm iteratively selects the most appropriate prediction box by calculating the IoU between the prediction box Bounding Box and the ground truth.

[0099]

[0100] Soft-NMS introduces a penalty coefficient λ to filter the most appropriate prediction box by reducing the confidence level.

[0101]

[0102] Threshold is a hyperparameter. In this example, it is set to 0.6. After calculation, we can get the final prediction result.

[0103] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative uses of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0104] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A method for detecting young tea leaves based on feature fusion graph neural network, characterized in that: The steps include: Step (1): Data acquisition, using a light trap to obtain the point cloud and RGB image of the target to be detected; Step (2): Use convolutional neural network (CNN) to extract features from the two-dimensional image; Step (3): Apply key point detection algorithm to the point cloud to detect key points; Step (4): Fusion of image features and point cloud features; Step (5): Build a graph structure, using the key points detected in step (3) as vertices and build a graph; Step (6): Use graph neural network to predict the location of tea leaves; Step (7): Generate prediction box; Step (8): The soft-NMS algorithm generates the best prediction box; Step (1) specifically includes: A light trap in the shape of an inverted pyramid is designed, consisting of four plane mirrors. The bottom of the pyramid is open to allow sufficient light to enter the light trap. Another plane mirror is placed above the light trap at a 45-degree angle. A TOF sensor is used to emit a light beam to the plane mirror, placing the object to be observed at the bottom of the inverted pyramid. After multiple reflections, the light beam can cover more than 99% of the object's surface area, and the three-dimensional coordinates of all points are obtained: d total It represents the total distance traveled by the light from the ToF sensor position to the surface of the target object, d k It represents the distance traveled by the light between two reflections, and d represents the distance from the last time the light beam touches the mirror to the surface of the object being measured; d means Is the total distance measured by the ToF sensor. Due to the reflection theorem of light, the light beam will return to the sensor along the original path, so the actual distance is half of the measured distance, where d k It can also be calculated as the position of the beam and the next mirror reflection minus the position of the previous mirror reflection, that is, d k =p k+1 -p k , where p k is the reflection point of the light at the mirror surface for the kth time, and p k+1 can be Calculated, is the unit direction vector along the direction of reflection of the light beam, calculated by Snell's law n k is the normal direction of the mirror. When the initial position p0 of ToF and the direction vector r0 of the initial incident light are known, the p of all k∈[1...K] can be recursively calculated. k and In this way, the three-dimensional coordinates of the object being measured can be obtained, and further the three-dimensional point cloud of the object being measured can be obtained.

2. The method for detecting young tea leaves based on a feature fusion graph neural network according to claim 1, wherein: The specific process of step (2) is: The two-dimensional image is input into the pre-trained resnet50 network for image feature extraction. Resnet50 includes an identity_block and a conv_block, and finally a feature vector with a depth of 2048 is obtained.

3. The method for detecting young tea leaves based on a feature fusion graph neural network according to claim 1, wherein: Step (3) specifically includes: A key point extraction algorithm based on point cloud saliency is used, including a saliency calculation module and a key point generation module. Saliency is defined as the combination of the feature activation signal on a specific layer of the pre-trained descriptor and the gradient of the same layer relative to the input point cloud. The pre-trained descriptor network is used to generate robust key points directly from 3D data. The saliency is then projected into the spherical coordinate system of the input point cloud, and the areas with the most information are extracted. These areas are combined with the features of the original descriptor to extract robust and repeatable key points. The key point detection algorithm obtains the final key point prediction by fusing the feature vectors obtained by the point cloud saliency method, the original features of each point, and the point cloud context features, and then inputting them into two fully connected layers.

4. The method for detecting young tea leaves based on a feature fusion graph neural network according to claim 1, wherein: The specific process of step (4) is: The two-dimensional image features obtained by resnet50 are fused with the point cloud features through the concatenate function to obtain a new feature vector, and the fused new feature vector is used as the initial feature of the subsequent graph structure.

5. The method for detecting young tea leaves based on a feature fusion graph neural network according to claim 1, wherein: The specific process of step (5) is: A graph structure consists of two elements: edges and vertices, that is, G = (V, E), where V represents the vertex of the graph structure V = {v1, v2, ..., v n }, take the key point detected in step (2) as vertex v i ; E is the edge connected by vertices, which can be obtained according to certain conditions: E={(v i ,v j )|||x i -x j ||2<r} Where x i with x j are the horizontal coordinates of different vertices, r is the distance between two vertices, and the features of each vertex include not only the initial features obtained by the previous fusion but also the features of the edges associated with it, that is, the features of the vertex are represented by V i =V0 is updated to V i =[V0,V1,V2,…,V 128 ]V0,V1,V2,...,V 128 For the remaining vertices associated with the vertex, V i It is the updated vertex. By continuously updating the features of the vertex, the category of the object in the local area can be determined based on the vertex.

6. The method for detecting young tea leaves based on a feature fusion graph neural network according to claim 1, wherein: The specific process of step (6) is: a) First, manually calibrate the target frame of the tea leaves in the collected 3D point cloud so that the target frame can just surround the object we need; b) The obtained graph structure is used as the input of the network. The graph neural network updates the vertex features by aggregating the edge features. After t+1 iterations, the vertex and edge features are updated to in and are the features of vertices and edges after t iterations, f t The (.) function calculates the characteristics of the edge between two vertices, ρ(.) aggregates the characteristics of the edge to the vertex, g t (.) Calculate the vertex features after aggregating the edge features, and then associate a feature extraction function MLP for each vertex to obtain: After t iterations, the obtained vertex features can be used to predict the category of the target and obtain the regression box of the target.

7. The method for detecting young tea leaves based on a feature fusion graph neural network according to claim 1, wherein: Generate prediction box: The prediction box is obtained by predicting a four-degree-of-freedom vector b = (x, y, z, l, h, w, θ). The (x, y, z) coordinates are the center coordinates of the regression box, and (h, w, θ) are the length, width and angle of the predicted bounding box. In the preliminary classification, the vertices are divided into foreground points and background points. The foreground points contain the information we need. If the vertex falls in the foreground point, it will be sent to the MLP responsible for position prediction. loc , Huber Loss is applied to predict the regression box. Since there may be multiple vertices on the same object, multiple prediction boxes will be generated for the same object. The next step will be to use the algorithm to obtain the best prediction box.

8. The method for detecting young tea leaves based on a feature fusion graph neural network according to claim 1, wherein: The specific process of step (8) is: The soft-NMS algorithm is used to remove the redundant prediction boxes. The NMS algorithm iteratively selects the most appropriate prediction box by calculating the IoU between the prediction box BoundingBox and the true value Ground Truth: Soft-NMS introduces a penalty coefficient λ to filter the most appropriate prediction box by reducing the confidence level: Where B i , B j They are different detection boxes respectively. Threshold is a hyperparameter. After calculation, the final prediction result can be obtained.

Images