A fruit recognition and positioning method based on 3D-2D feature fusion

Abstract

The present invention discloses a fruit recognition and positioning method based on 3D-2D feature fusion, which obtains 3D point cloud data of fruits through a depth camera; converts the preprocessed point cloud data into a tensor; uses PointNet++ to obtain local, global features and result features of the point cloud; preprocesses the acquired 2D data, and uses a convolutional neural network to extract features from the 2D data; performs multi-scale feature extraction on the global features of the point cloud and the global features of the image, and after conversion through a deconvolution layer and a point convolution layer, splices along the channel dimension to obtain overall features at different scales; then fuses the result features of the image and the result features of the point cloud with the above-mentioned overall features; uses a training set of fused features as input to perform deep learning training. The present invention complements the advantages of point cloud and image data, performs all-round and multi-dimensional feature extraction and analysis on fruits, and can effectively improve the accuracy and reliability of fruit recognition.

Description

Technical Field

[0001] The invention belongs to the field of fruit recognition, and in particular relates to a fruit recognition and positioning method based on 3D-2D feature fusion. Background Art

[0002] In the field of fruit recognition, traditional fruit recognition technology mainly uses image processing methods to identify the type and maturity of fruits, but it has the disadvantages of being sensitive to lighting conditions, color differences, etc., resulting in low recognition accuracy. In addition, the information obtained from plane images is always limited. With the development of 3D technologies such as lidar and depth cameras, fruit recognition technology based on 3D technology has been proposed, but the three-dimensional information obtained by 3D technology requires more complex algorithms to obtain the required part, so the use of a single technology has become limited, and the multimodal fusion method has made new improvements to fruit recognition technology. There are various fusion technologies in multimodal fusion, such as multi-sensor fusion technology, decision fusion technology, deep learning fusion technology, etc. These multimodal fusion methods are all for more comprehensive and accurate recognition and positioning of targets, and provide support for the realization of intelligent and automated applications. Therefore, multimodal fusion technology has become a more cutting-edge technical method. In fruit recognition and positioning, in order to improve the accuracy of recognition and positioning, more comprehensive feature information is needed. Therefore, a multimodal fusion method is needed to overcome these problems. Summary of the invention

[0003] Purpose of the invention: In order to overcome the shortcomings of the above-mentioned prior art, the present invention proposes a fruit recognition and positioning method based on 3D-2D feature fusion, which utilizes feature fusion and deep learning algorithms to meet the requirements of all-round and multi-dimensional feature extraction and analysis of fruit recognition, and can effectively improve the accuracy and reliability of fruit recognition.

[0004] Technical solution: The invention discloses a fruit identification and positioning method based on 3D-2D feature fusion, comprising the following steps:

[0005] (1) Obtain the point cloud data of the fruit through a depth camera and preprocess the point cloud data; use the nearest neighbor algorithm K-NN to construct the local neighborhood of each point; convert the preprocessed point cloud data into a tensor format and normalize the coordinates. Each point includes three-dimensional coordinate information (x, y, z) and color information;

[0006] (2) Obtaining local, global and result features of point cloud data through PointNet++; taking the points in the local neighborhood obtained in step (1) as input, and obtaining the local features of each point through the feature extraction module; aggregating the local features of each point through the local region aggregation module to obtain more extensive local context information; passing the aggregated features through the regional context encoder to obtain encoded feature information; passing the encoded feature information through the global feature aggregation module to obtain the global features that represent the entire point cloud data; after the feature extraction and aggregation process, the PointNet++ network finally outputs the result features of the point cloud;

[0007] (3) Obtain a two-dimensional image of the fruit through a camera and perform data preprocessing; add corresponding label data to each fruit image and divide the data set into a training set, a validation set, and a test set; use the VGG model to train the training set, use the validation set to evaluate the trained VGG model, and apply the trained VGG model to the test set to check the performance; in the trained VGG model, use the convolution layer and pooling layer to extract the global features of the fruit image, and convert the extracted high-level abstract feature representation into a tensor form; use the fully connected layer and softmax to obtain the result features of the image;

[0008] (4) Perform multi-scale feature extraction on the global features of the point cloud data and the global features of the image, respectively pass through the deconvolution layer and the point convolution transformation, and then splice along the channel dimension to obtain the overall features at different scales; then perform feature fusion on the result features of the image and the result features of the point cloud with the above-mentioned overall features to obtain a fused feature set;

[0009] (5) Divide the fused feature set into a training set, a validation set, and a test set, where the fused feature training set is used as input and the YOLOv5 model is used for deep learning training; set a comprehensive loss function to calculate the error between the true label and the predicted label in the target detection task;

[0010] (6) Use the validation set and test set of the fused features to verify and evaluate the trained model respectively; the model will output information such as the location, category, and confidence of the detected target object; perform non-maximum suppression on the target detection results output by the model to eliminate redundant detection results; visualize the processed detection results into the original image so that the performance of the model can be evaluated through visualization.

[0011] Furthermore, the preprocessing of the point cloud data in step (1) includes removing noise points, filtering, and removing outliers.

[0012] Furthermore, the feature extraction module in step (2) extracts features from each point in the point cloud through a multi-layer perceptron to obtain a point-level feature representation:

[0013] f i+1 =MLP(u i+1 )

[0014] Among them, f i+1 represents the feature representation of the i+1th layer, u i+1 represents the input of the i+1th layer, and MLR represents a multi-layer perceptron, which is used to perform nonlinear transformation on the input.

[0015] Furthermore, the local area aggregation module in step (2) further fuses the extracted point-level features with local information through a maximum pooling aggregation strategy to capture the relationship and information between local features:

[0016]

[0017] Among them, T i represents the feature representation of the center point, N(i) represents the set of neighborhood points around the center point, It represents the feature representation of the center point after local area aggregation, and MAXPOOL represents the maximum pooling operation on the feature.

[0018] Furthermore, in step (2), the regional context encoder captures the context information between different regions in the point cloud through a multi-layer perceptron to understand the connection and structure between the various parts in the point cloud; the global feature aggregation module integrates and summarizes the global information of the entire point cloud through the maximum pooling aggregation strategy and splicing operation to obtain a global feature representation:

[0019]

[0020] Among them, g i represents the global feature representation, represents the updated feature representation of each point, f 1∶N represents the feature representation of all points in the point cloud data, MLR represents the multi-layer perceptron used to perform nonlinear mapping on the features of all points, MAXPOOL represents the maximum pooling operation on the features, and f i It represents the feature representation of the i-th point in the point cloud, and CONCAT represents the concatenation operation of the local features and the global features of the point cloud.

[0021] Furthermore, the data preprocessing in step (3) includes denoising, resizing, and grayscale processing.

[0022] Furthermore, the overall features at different scales obtained by splicing along the channel dimension in step (4) are realized by the following formula:

[0023]

[0024] Among them, S X Represents the input of small-scale features, S Z Represents the input of mesoscale features, S D represents the input of large-scale features, represents the output of small-scale features, represents the output of mesoscale features, Represents the output of large-scale features, S represents the overall features of different scales obtained by splicing, PCONV represents the point convolution operation, DECONV represents the deconvolution operation, and CCT represents the splicing operation along the channel dimension.

[0025] Furthermore, the feature fusion of the result features of the image and the result features of the point cloud with the overall features in step (4) is achieved by the following formula:

[0026]

[0027] Among them, K represents the part of the feature extraction network used to extract shallow features from the input data, J represents the part of the feature extraction network used to extract deeper features from shallow features, L represents the part of the feature extraction network used to obtain the model output results from deep features, and J P , K P and L P represents the network part corresponding to processing point cloud data, J I , K I and L I It represents the network part corresponding to processing image data, F represents fusion features, RF represents result features, GF represents overall features, and B represents the network part that processes overall features.

[0028] Furthermore, in step (5), the ratio of the training set, validation set and test set is 8:1:1.

[0029] Furthermore, the comprehensive loss function in step (5) is:

[0030]

[0031] Among them, L represents the comprehensive loss, OL represents the target existence loss, CL represents the classification loss, BL represents the bounding box regression loss, α represents the weight parameter of the target existence loss, β represents the weight parameter of the bounding box regression loss, and U P and U T They represent the intersection-over-union ratio of the predicted box and the true box, C represents the number of categories, and y i Represents the one-hot encoding form of the true label, p i Represents the probability value of the model output, x, y are the center coordinates of the prediction box, w, h are the width and height of the prediction box, xh ,y h 、w h 、h h They are the center coordinates, width and height of the real box respectively.

[0032] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are: the present invention can automatically identify the type of fruit and coordinate positioning, while complementing the advantages of point cloud and image data to provide more comprehensive and rich information, and adopts multi-scale feature extraction to perform all-round and multi-dimensional feature extraction and analysis of fruits, which can effectively improve the detection rate and accuracy of fruit recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the present invention;

[0034] Figure 2 Get the point cloud data map of Apple for the depth camera;

[0035] Figure 3 Schematic diagram of the fruit recognition and positioning model structure based on 3D-2D feature fusion

[0036] Figure 4 It is a flowchart of the global feature extraction process;

[0037] Figure 5 This is a visualization diagram of the detection results using the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0039] like Figure 1 As shown, the present invention proposes a fruit recognition and positioning method based on 3D-2D feature fusion, which specifically includes the following steps:

[0040] Step 1: 3D data acquisition and conversion.

[0041] The point cloud data of the apple is obtained through the depth camera, and the point cloud data will contain the three-dimensional coordinate information of the fruit; Figure 2As shown in the figure, the quality of the acquired point cloud data is checked, including preprocessing operations such as removing noise points, filtering, and removing outliers to ensure the quality of the obtained data; the preprocessed point cloud data is converted into a tensor format and the coordinates are normalized. Each point includes three-dimensional coordinate information (x, y, z) and color information; the nearest neighbor algorithm K-NN is used to construct a local neighborhood for each point. These local neighborhood points will be used in the local feature extraction in step (2); the processed data format is converted into the input format of PointNet++: (B, N, 3+D), where B is the batch size, N is the number of points, 3 is the three-dimensional coordinate information, and D is the dimension of the color attribute.

[0042] Step 2: Extraction of point cloud features.

[0043] PointNet++ obtains local, global and result features of point cloud data through feature extraction module, local region aggregation module, regional context encoder and global feature aggregation module. The feature extraction module extracts feature representation from the input point cloud data, extracts features from each point in the point cloud through a multi-layer perceptron, and obtains point-level feature representation. The feature extraction module can be expressed using the following formula:

[0044] f i+1 =MLP(u i+1 )

[0045] Among them, f i+1 represents the feature representation of the i+1th layer, u i+1 represents the input of the i+1th layer, and MLR represents a multi-layer perceptron, which is used to perform nonlinear transformation on the input.

[0046] The local area aggregation module further fuses the local information by using the maximum pooling aggregation strategy to extract the point-level features, thereby capturing the relationship and information between local features. The local area aggregation module can be expressed using the following formula:

[0047]

[0048] Among them, T i represents the feature representation of the center point, N(i) represents the set of neighborhood points around the center point, It represents the feature representation of the center point after local area aggregation, and MAXPOOL represents the maximum pooling operation on the feature.

[0049] The regional context encoder uses a multi-layer perceptron to capture the contextual information between different regions in the point cloud, understand the connection and structure between the various parts of the point cloud, and thus improve the accuracy and robustness of the feature representation; the global feature aggregation module uses the maximum pooling aggregation strategy and splicing operation to integrate and summarize the global information of the entire point cloud to obtain a global feature representation; after the feature extraction and aggregation process, the PointNet++ network finally outputs the result features of the point cloud. The regional context encoder and the global feature aggregation module can be expressed using the following formula:

[0050]

[0051] Among them, f 1∶N represents the feature representation of all points in the point cloud data, MLR represents the multi-layer perceptron used to perform nonlinear mapping on the features of all points, MAXPOOL represents the maximum pooling operation on the features, and g i represents the global feature representation, f i It represents the feature representation of the i-th point in the point cloud, CONCAT represents the concatenation operation of the local features and global features of the point cloud, Denotes the updated feature representation of each point.

[0052] The points in the local neighborhood in step 1 are taken as input, and the multi-layer perceptron in the feature extraction module is used to extract features for each point in the point cloud to obtain point-level feature representation; the point-level features of each point are aggregated by the local area aggregation module through maximum pooling to obtain the local context information of the apple point cloud data; the multi-layer perceptron in the regional context encoder is used to capture the context information between different areas in the apple point cloud; the obtained feature information is used to integrate and summarize the global information of the entire point cloud using the aggregation strategy and splicing operation of the global feature aggregation module to obtain the global features that characterize the entire apple point cloud data.

[0053] Step 3: 2D data collection and feature extraction.

[0054] The two-dimensional image of the apple is obtained through the camera; the quality of the obtained two-dimensional image is checked, including image denoising, resizing, grayscale processing and other preprocessing operations to reduce interference factors in the image and enhance the image quality; corresponding label data is added to each apple image, and the data set is divided into training set, validation set and test set in the ratio of 0.7:0.1:0.2.

[0055] Use the VGG model to train the training set, use the validation set to evaluate the trained VGG model, and apply the trained VGG model to the test set to check the performance; in the trained VGG model, use the convolution layer and pooling layer to extract the feature representation of the apple image, and convert the extracted high-level abstract feature representation into tensor form to facilitate the feature fusion in step 4; use the fully connected layer and softmax to obtain the result features of the image.

[0056] Step 4: Feature fusion: Perform multi-scale feature extraction on the global features of the point cloud and the global features of the image, transform them through the deconvolution layer and point convolution respectively, and then splice them along the channel dimension to obtain the overall features.

[0057] The formula for multi-scale feature extraction is as follows:

[0058]

[0059] Among them, S X Represents the input of small-scale features, S Z Represents the input of mesoscale features, S D represents the input of large-scale features, represents the output of small-scale features, represents the output of mesoscale features, Represents the output of large-scale features, S represents the overall features of different scales obtained by splicing, PCONV represents the point convolution operation, DECONV represents the deconvolution operation, and CCT represents the splicing operation along the channel dimension.

[0060] like Figure 4 As shown in the figure, multi-scale features with dimensions of (64, 64, 1024), (128, 128, 512), and (256, 256, 256) are used respectively, input into the multi-scale feature extraction network, and transformed through the deconvolution layer and point convolution respectively, and then spliced ​​along the channel dimension to obtain the overall feature of dimension (256, 256, 3).

[0061] Then the resulting features of the image and the point cloud are combined with the above overall features as follows Figure 3 Feature fusion is performed as shown:

[0062]

[0063] Among them, K represents the part of the feature extraction network used to extract shallow features from the input data, J represents the part of the feature extraction network used to extract deeper features from shallow features, L represents the part of the feature extraction network used to obtain the model output results from deep features, and J P , K P and L P represents the network part corresponding to processing point cloud data, JI , K I and L I Indicates the network part corresponding to processing image data, F indicates fusion feature, RF indicates result feature, GF indicates overall feature, and B indicates the network part that processes overall feature. The fusion feature set is divided into training set, validation set and test set in a ratio of 0.8∶0.1∶0.1.

[0064] Step 5: Model training: Take the training set of fused features as input and use the YOLOv5 model for deep learning training; set a comprehensive loss function to calculate the error between the true label and the predicted label in the target detection task. The comprehensive loss function is:

[0065]

[0066] Among them, L represents the comprehensive loss, OL represents the target existence loss, CL represents the classification loss, BL represents the bounding box regression loss, α represents the weight parameter of the target existence loss, β represents the weight parameter of the bounding box regression loss, and U P and U T They represent the intersection-over-union ratio of the predicted box and the true box, C represents the number of categories, and y i Represents the one-hot encoding form of the true label, p i Represents the probability value of the model output, x, y are the center coordinates of the prediction box, w, h are the width and height of the prediction box, x h ,y h 、w h 、h h They are the center coordinates, width and height of the real box respectively.

[0067] After each training cycle, the validation set is used for verification to obtain the model parameters. After the full training cycle is completed, the model with the highest accuracy is selected as the best model.

[0068] Step 6: Model testing: Use the test set to verify and evaluate the trained model. The model will output information such as the location and confidence of the detected apples. Perform non-maximum suppression (NMS) on the apple detection results output by the model to eliminate redundant detection results; Figure 5 As shown, the processed detection results are visualized into the original image to obtain a visualized recognition result.

[0069] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A fruit identification and positioning method based on 3D-2D feature fusion, characterized in that: The following steps are involved: (1) Obtaining point cloud data of the fruit through a depth camera and preprocessing the point cloud data; Use the nearest neighbor algorithm K-NN to construct the local neighborhood of each point; Convert the preprocessed point cloud data into tensor format and normalize the coordinates. Each point includes three-dimensional coordinate information (x, y, z) and color information. (2) Obtain local, global and result features of point cloud data through PointNet++; The points in the local neighborhood obtained in step (1) are used as input, and the local features of each point are obtained through the feature extraction module; the local features of each point are aggregated through the local area aggregation module to obtain more extensive local context information; the aggregated features are passed through the regional context encoder to obtain encoded feature information; the encoded feature information is passed through the global feature aggregation module to obtain the global features that represent the entire point cloud data; After feature extraction and aggregation, the PointNet++ network finally outputs the result features of the point cloud; (3) obtaining a two-dimensional image of the fruit through a camera and performing data preprocessing; Add corresponding label data to each fruit image and divide the data set into training set, validation set and test set; use the VGG model to train the training set, use the validation set to evaluate the trained VGG model, and apply the trained VGG model to the test set to check the performance; in the trained VGG model, use the convolution layer and pooling layer to extract the global features of the fruit image, and convert the extracted high-level abstract feature representation into tensor form; use the fully connected layer and softmax to obtain the result features of the image; (4) Perform multi-scale feature extraction on the global features of the point cloud data and the global features of the image, respectively pass through the deconvolution layer and the point convolution transformation, and then splice along the channel dimension to obtain the overall features at different scales; then perform feature fusion on the result features of the image and the result features of the point cloud with the above-mentioned overall features to obtain a fused feature set; (5) Divide the fused feature set into a training set, a validation set, and a test set, where the fused feature training set is used as input and the YOLOv5 model is used for deep learning training; set a comprehensive loss function to calculate the error between the true label and the predicted label in the target detection task; (6) Use the validation set and test set of fused features to verify and evaluate the trained model respectively; The model will output information such as the location, category, and confidence of the detected target object; Perform non-maximum suppression on the target detection results output by the model to eliminate redundant detection results; Visualize the processed detection results into the original image so that the performance of the model can be evaluated through visualization; The feature fusion of the result features of the image and the result features of the point cloud with the overall features in step (4) is achieved by the following formula: Among them, K represents the part of the feature extraction network used to extract shallow features from the input data, J represents the part of the feature extraction network used to extract deeper features from shallow features, L represents the part of the feature extraction network used to obtain the model output results from deep features, and J P , K P and L P represents the network part corresponding to processing point cloud data, J I , K I and L I It represents the network part corresponding to processing image data, F represents fusion features, RF represents result features, and GF represents overall features.

2. A fruit identification and positioning method based on 3D-2D feature fusion according to claim 1, characterized in that: The preprocessing of the point cloud data in step (1) includes removing noise points, filtering, and removing outliers.

3. A fruit identification and positioning method based on 3D-2D feature fusion according to claim 1, characterized in that: The feature extraction module in step (2) extracts features from each point in the point cloud through a multi-layer perceptron to obtain point-level feature representation: f i+1 =MLP(u i+1 ) Among them, f i+1 represents the feature representation of the i+1th layer, u i+1 represents the input of the i+1th layer, and MLR represents a multi-layer perceptron, which is used to perform nonlinear transformation on the input.

4. A fruit identification and positioning method based on 3D-2D feature fusion according to claim 1, characterized in that: The local area aggregation module in step (2) further fuses the extracted point-level features with local information through the maximum pooling aggregation strategy to capture the relationship and information between local features: Among them, T N(i) represents the feature representation of the center point, N(i) represents the set of neighborhood points around the center point, It represents the feature representation of the center point after local area aggregation, and MAXPOOL represents the maximum pooling operation on the feature.

5. The fruit identification and positioning method based on 3D-2D feature fusion according to claim 1 is characterized in that: In step (2), the regional context encoder captures the context information between different regions in the point cloud through a multi-layer perceptron to understand the connection and structure between the various parts of the point cloud; the global feature aggregation module integrates and summarizes the global information of the entire point cloud through the maximum pooling aggregation strategy and splicing operation to obtain a global feature representation: Among them, g i represents the global feature representation, represents the updated feature representation of each point, f 1:N represents the feature representation of all points in the point cloud data, MLR represents the multi-layer perceptron used to perform nonlinear mapping on the features of all points, MAXPOOL represents the maximum pooling operation on the features, and f i It represents the feature representation of the i-th point in the point cloud, and CONCAT represents the concatenation operation of the local features and the global features of the point cloud.

6. A fruit identification and positioning method based on 3D-2D feature fusion according to claim 1, characterized in that: The data preprocessing in step (3) includes denoising, resizing, and grayscale processing.

7. The fruit identification and positioning method based on 3D-2D feature fusion according to claim 1, characterized in that: The overall features at different scales obtained by splicing along the channel dimension in step (4) are realized by the following formula: Among them, S X Represents the input of small-scale features, S Z Represents the input of mesoscale features, S D represents the input of large-scale features, represents the output of small-scale features, represents the output of mesoscale features, Represents the output of large-scale features, S represents the overall features of different scales obtained by splicing, PCONV represents the point convolution operation, DECONV represents the deconvolution operation, and CCT represents the splicing operation along the channel dimension.

8. The fruit identification and positioning method based on 3D-2D feature fusion according to claim 1, characterized in that: The ratio of the training set, validation set and test set in step (5) is 8:1:

1.

9. The fruit identification and positioning method based on 3D-2D feature fusion according to claim 1, characterized in that: The comprehensive loss function in step (5) is: Among them, L represents the comprehensive loss, OL represents the target existence loss, CL represents the classification loss, BL represents the bounding box regression loss, α represents the weight parameter of the target existence loss, β represents the weight parameter of the bounding box regression loss, and U P and U T They represent the intersection-over-union ratio of the predicted box and the true box, C represents the number of categories, and y i Represents the one-hot encoding form of the true label, p i Represents the probability value of the model output, x, y are the center coordinates of the prediction box, w, h are the width and height of the prediction box, x h ,y h 、w h 、h h They are the center coordinates, width and height of the real box respectively.

Images