Seed detection method based on three-dimensional point cloud and hyperspectral feature fusion

Abstract

The invention provides a seed detection method based on three-dimensional point cloud and hyperspectral feature fusion, and the method combines computer vision, spectral analysis and deep learning technologies to realize accurate classification and quality detection of wheat seeds. The method comprises the following steps: firstly, acquiring spatial form data, including size, volume and surface structure characteristics, of a seed through a three-dimensional reconstruction technology; secondly, spectral information of the seeds is obtained through a hyperspectral imaging technology, and spectral reflection characteristics of a specific wave band are extracted; and then, constructing a multi-dimensional feature set based on the three-dimensional reconstruction data and the hyperspectral data, performing feature extraction and classification identification by using a convolutional neural network algorithm, and judging the variety and the quality grade of the seed. According to the method, data features are efficiently and accurately extracted through multi-modal fusion of spatial features and spectral features in combination with the CNN algorithm, the accuracy and reliability of wheat seed classification are effectively improved, and the method can be widely applied to the fields of seed screening, variety identification, agricultural production quality control and the like.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and spectral analysis, and particularly to a multi-purpose seed detection method based on the fusion of three-dimensional point cloud and hyperspectral features, especially achieving high-precision classification by combining a convolutional neural network (CNN), belonging to the fields of agricultural engineering and intelligent detection technology. Background Art

[0002] With the development of agricultural technology, the classification and quality detection of wheat seeds are of great significance for improving agricultural production efficiency and seed quality. Traditional wheat seed classification methods mainly rely on manual observation or single-image recognition technology, which have problems of low efficiency and insufficient accuracy. In addition, it is difficult to accurately distinguish wheat seeds of different varieties or quality grades with single-dimensional information (such as color and morphology).

[0003] In recent years, three-dimensional reconstruction technology and hyperspectral imaging technology have gradually been applied to the agricultural field. Three-dimensional reconstruction technology can collect three-dimensional morphological information of seeds (such as size, volume, and surface structure), while hyperspectral imaging technology can obtain the spectral reflection characteristics of seeds, reflecting their material composition. However, the research on combining these two technologies and classifying them through a deep learning model (such as a convolutional neural network) is not yet mature. Therefore, there is an urgent need for a wheat seed classification method that fuses three-dimensional reconstruction data and hyperspectral data to improve the accuracy and reliability of classification. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-purpose seed detection method that combines three-dimensional reconstruction and hyperspectral detection and is implemented through a convolutional neural network (CNN). The present invention significantly improves the accuracy and efficiency of wheat seed classification through multi-modal data fusion.

[0005] To solve the above technical problems, the technical solution provided by the present invention is: a seed detection method based on the fusion of three-dimensional point cloud and hyperspectral features, the method comprising the following steps: S1. Point cloud data acquisition and preprocessing: ① Point cloud data acquisition: Use a three-dimensional scanning device to perform multi-view scanning on wheat seeds to obtain point cloud data of the seeds including size, surface morphology, and spatial structure information; when collecting, place the wheat seeds on a turntable and perform multi-angle scanning by rotating the turntable; ② Point cloud data preprocessing: Perform preprocessing on the collected point cloud data, including denoising, coordinate normalization, and meshing processing, to generate regular three-dimensional grid data; project the processed point cloud data into multi-view two-dimensional depth maps, or directly use it for feature extraction after voxelization; S2. 3D Data Feature Extraction and Classification Based on CNN: ① Data Input: Input the preprocessed point cloud into different CNN networks: Input the depth map into a two-dimensional convolutional neural network (2D CNN) to extract surface structure features; Input the voxel data into a three-dimensional convolutional neural network (3D CNN) to extract spatial morphological features; Input the point cloud data into the PointNet neural network to extract global three-dimensional space features; ② Feature Extraction: Perform the following operations in the convolutional neural network: Convolution operation: Extract the spatial morphological features of wheat seeds through the convolutional layer; Pooling operation: Reduce the dimension through max pooling or average pooling to reduce feature redundancy; Feature fusion: Fuse the multi-level features extracted through the fully connected layer to form three-dimensional features; ③ Classification: The extracted feature vectors are fused through the fully connected layer, and the Softmax classifier is used to complete the preliminary classification; S3. Hyperspectral Data Acquisition and Preprocessing: ① Hyperspectral Data Acquisition: Use hyperspectral imaging equipment to acquire hyperspectral data of wheat seeds in the wavelength range of 400 - 1000 nm; ② Data Preprocessing: Perform spectral data preprocessing, including denoising, band selection, and normalization; Extract the key bands of the hyperspectral data to reduce data redundancy; S4. Hyperspectral Data Feature Extraction and Classification Based on CNN: Input the hyperspectral data into the CNN network: Input the spectral vector into a one-dimensional convolutional neural network (1D CNN) to extract spectral dimension features; Input the spectral-spatial cube into a three-dimensional convolutional neural network (3D CNN) to extract joint spatial and spectral features; The extracted hyperspectral features are mapped to the classification space through the fully connected layer, and the Softmax classifier is used to complete the classification; S5. Decision-Level Fusion of Classification Results: ① Weight Assignment: Calculate the weight coefficients of each result according to the confidence and accuracy of the classification results of the three-dimensional reconstruction point cloud data and the hyperspectral classification results; ② Decision Fusion: Determine each final classification result based on the confidence, accuracy, and weight coefficients of the classification results; ③ Result Output: Output the final classification results of wheat seeds, including variety, quality grade, or health status.

[0006] Compared with the prior art, the present invention has the following beneficial effects: Multimodal data fusion to improve classification accuracy Different from the existing single-vision or spectral classification methods, the present invention combines three-dimensional reconstruction data with hyperspectral data to comprehensively analyze the morphological structure characteristics and spectral reflection characteristics of wheat seeds. Through multimodal data fusion, the information on the surface morphology and internal material composition of the seeds is fully exploited, significantly improving the accuracy and robustness of classification. Experimental results show that the classification accuracy of the present invention exceeds 98%, significantly superior to traditional methods. The present invention uses a convolutional neural network (CNN) to extract features and classify the three-dimensional reconstruction data and hyperspectral data, realizing the automatic learning and representation of features. Through network structures such as 2D CNN, 3D CNN, PointNet, and 1D CNN, efficient feature extraction is performed for different types of data, avoiding the subjectivity and complexity in traditional manual feature extraction and enhancing the generalization ability of the model. In the three-dimensional reconstruction part, the present invention generates a two-dimensional depth map of the seeds through multi-view projection technology and combines 3D CNN and PointNet to extract three-dimensional spatial features, effectively capturing the details and spatial structure of the seed surface. In the hyperspectral detection part, 1D CNN and 3D CNN are used to extract spectral and spatial joint features, realizing multi-scale and all-round feature extraction, thereby improving the classification effect. To further improve the stability of classification, the present invention comprehensively combines the three-dimensional reconstruction classification result and the hyperspectral classification result through a decision-level fusion method, and uses a weighted voting method or a weighted average method for result fusion. This method can dynamically assign weights according to the classification confidence, further enhancing the accuracy and reliability of the final classification result. The present invention adopts a non-contact detection method based on images and spectra, which will not cause any damage to wheat seeds throughout the process, is applicable to various scenarios such as variety identification, quality grade assessment, and health status detection, and has strong applicability and promotion value. Compared with the traditional methods that rely on manual observation or chemical detection, the present invention, through the automated processes of three-dimensional reconstruction and hyperspectral detection, as well as the application of the CNN deep learning model, significantly improves the detection efficiency and automation level, reduces manual intervention, and lowers the detection cost. Through high-precision three-dimensional reconstruction and hyperspectral detection technologies, combined with the powerful feature extraction ability of the deep learning network, the classification method of the present invention has strong robustness and anti-interference ability, and can adapt to changes in different lighting, angles, and seed morphologies, with stronger adaptability.

[0007] Furthermore, in the step S1, the acquisition of the point cloud data of wheat seeds specifically includes: placing the wheat seeds to be detected on a rotatable scanning platform, and using a three-dimensional scanning device to perform multi-angle scanning. The rotation angle is 0°-360°, and a group of high-precision point cloud data is acquired every 60°. The acquired point cloud data includes single or multiple wheat seeds, ensuring that the surface morphological characteristics and spatial information of the seeds are covered, and the point cloud scanning accuracy is better than 0.1 mm. The three-dimensional scanning device includes: a structured light scanner, which is suitable for high-precision three-dimensional point cloud acquisition, and the scanning resolution is not lower than 0.1 mm; a laser scanner, which is suitable for the acquisition of high-density point cloud data of seeds with complex shapes; a depth camera, which can realize multi-view data acquisition and is used to generate a three-dimensional depth map.

[0008] Further, in the step S1, the preprocessing of the point cloud data specifically includes: using statistical filtering and radius filtering to remove noise points and low-density points; performing coordinate normalization to standardize the point cloud coordinates to the range [0,1]; using the octree segmentation algorithm to voxelize the point cloud data, and the size of the voxel unit is set to 1 mm³; converting the point cloud data into multiple two-dimensional depth maps through multi-view projection, and the number of views is 6.

[0009] Further, in the step S2, the training process of the convolutional neural network (CNN) used for three-dimensional data feature extraction is as follows: inputting the multi-view depth map of the point cloud into a two-dimensional convolutional neural network (2D CNN); inputting the voxelized point cloud data into a three-dimensional convolutional neural network (3D CNN); inputting the original point cloud data into the PointNet neural network; setting the training parameters: BatchSize = 32, epoch = 100, and the initial learning rate is 0.0001; using the trained network to test the validation set and the test set, and selecting the network model with the highest classification accuracy as the final three-dimensional feature extraction model.

[0010] Further, in the step S3, the process of acquiring hyperspectral data is as follows: using a hyperspectral imaging device to perform line-scan imaging on wheat seeds in the wavelength range of 400 nm to 1000 nm, and setting the spectral resolution to 1 nm; using a constant light source and a diffuser plate to ensure uniform illumination and avoid noise interference in the spectral data; finally obtaining hyperspectral cube data containing spatial and spectral information.

[0011] Further, the preprocessing process of the hyperspectral data includes: performing Savitzky-Golay filtering on the hyperspectral data to remove noise; extracting key bands through principal component analysis (PCA) to reduce data redundancy; performing min-max normalization to ensure the standardized input of the spectral data; using linear discriminant analysis to reduce the spectral dimension while retaining key features.

[0012] Further, in step S4, the training process of the model used for hyperspectral data feature extraction is as follows: Input the hyperspectral vector into a one-dimensional convolutional neural network (1D CNN) for spectral feature extraction; Input the hyperspectral-spatial data cube into a three-dimensional convolutional neural network (3D CNN) to extract joint spatial and spectral features; Set the training parameters: BatchSize = 16, epoch = 150, and the initial learning rate is 0.0001; Test the models with different training rounds, evaluate the classification performance on the test set, and select the hyperspectral data feature extraction model with the highest accuracy.

[0013] ⑤ Further, the decision-level fusion of the classification results in step S5 includes the following steps: Calculate the confidence and accuracy of the classification results of the three-dimensional data and the hyperspectral data; Use the weighted voting method: Perform weighted voting on the two classification results according to the confidence to obtain the final classification result; Or use the weighted average method: Perform weighted summation on the two types of classification results according to the weights to obtain the final classification result; Output the variety, quality grade, or health status of the wheat seeds.

[0014] Further, the verification and optimization of the results in step S5 include the following: Use the test set data to evaluate the final classification result, and calculate the accuracy, recall rate, and F1 score; If the classification accuracy is lower than 95%, perform data augmentation on the data set, including operations such as random rotation, translation, and scaling of the images; Adjust the parameters of the convolutional neural network (such as the learning rate, network depth) and retrain until the classification result reaches the expected accuracy. Description of the Drawings

[0015] Figure 1 It is a schematic flowchart of the seed detection method based on the fusion of three-dimensional point cloud and hyperspectral features proposed by the present invention.

[0016] Figure 2 It is a schematic diagram of the point cloud after point cloud data acquisition and preprocessing.

[0017] Figure 3 It is a schematic diagram of hyperspectral data acquisition and preprocessing.

[0018] Figure 4 It is a schematic diagram of the CNN network structure for three-dimensional reconstruction and hyperspectral data feature extraction. Detailed Embodiments

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope of protection of the present application. Without conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here, and the execution order between the steps can be adjusted according to requirements. The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0020] Combined with Figures 1 to 4 , an embodiment of the present invention proposes a multi-purpose seed detection method based on the fusion of three-dimensional point cloud and hyperspectral features. As Figure 1 shown, the method includes the following steps: 1. Three-dimensional point cloud data acquisition and preprocessing: Data acquisition: Place wheat seeds on a rotating platform and use a structured light scanner, a laser scanner, or a depth camera to perform multi-view scanning to obtain the point cloud data of the seeds, including size, surface morphology, and spatial structure information. The platform automatically rotates 6 angles, and one frame of high-precision point cloud data is collected at each angle, with a resolution of not less than 0.1 mm to ensure the accuracy and integrity of the point cloud data.

[0021] Point cloud preprocessing: Remove low-density points, isolated noise points, abnormal points, and scanning noise points through statistical filtering or radius filtering algorithms. Normalize the spatial coordinates of the point cloud data to the range [0,1]. Use the octree segmentation method to discretize the point cloud data, and set the voxel size to 1 mm³. Project the point cloud data from multiple perspectives into two-dimensional depth maps to provide input data for subsequent feature extraction. As Figure 2 shown.

[0022] 2. Hyperspectral data acquisition and preprocessing: Data acquisition: Use a hyperspectral imaging device to set the band range to 400-1000 nm, and the spectral resolution is better than 1 nm. During acquisition, place the wheat seeds on the spectral acquisition platform, set the light source to a constant intensity, and use a diffuser plate for uniform illumination to eliminate the influence of uneven illumination.

[0023] Data preprocessing: Use Savitzky-Golay filtering or wavelet transform to remove spectral noise. Select the spectral bands that contribute most to classification as key bands through principal component analysis (PCA) correlation analysis. Use linear discriminant analysis (LDA) to reduce the spectral dimension while retaining key features. Perform min-max normalization on the spectral data reflectance to standardize the input data. As Figure 3 shown.

[0024] 3. Feature extraction and classification of 3D point cloud data based on CNN: Input multi-view depth Figure 2 dimensional input into a 2D convolutional neural network (2D CNN), voxel data into a 3D convolutional neural network (3D CNN), and point cloud data into the PointNet model. 2D CNN: Extract the edge and surface texture features of the depth map through convolutional layers and pooling layers. 3D CNN: Perform three-dimensional convolution on the voxelized data to extract spatial structure features. As Figure 4 shown. PointNet: Directly process the point cloud data to extract global morphological features. Fuse various features through fully connected layers and input them into the Softmax classifier to output the classification results of 3D point cloud data.

[0025] 4. Feature extraction and classification of hyperspectral data based on CNN: Input single-pixel spectral vectors into a 1D CNN to extract spectral dimension features. Three-dimensional convolutional neural network (3D CNN): Input spectral-spatial cube data into a 3D CNN to extract joint spectral and spatial features. As Figure 4 shown. The extracted feature vectors are mapped through fully connected layers and input into the Softmax classifier to output the hyperspectral classification results.

[0026] 5. Decision-level fusion of classification results: Calculate the weight coefficients of each result according to the confidence and accuracy of the classification results of 3D reconstructed point cloud data and hyperspectral classification results, and then use the weighted voting method or weighted average method to confirm the final classification result. Finally, output the final classification results of wheat seeds, including variety identification, quality grade, or health status. Specifically, the weighted voting method: Allocate weights according to the confidence of 3D reconstruction classification results and hyperspectral classification results for voting. The weighted average method: Calculate the final result based on the confidence weights of the classification models.

[0027] 6. Performance evaluation and optimization: Use indicators such as accuracy, recall rate, and F1 score to evaluate the classification effect. If the classification accuracy is lower than 95%, optimize the dataset through data augmentation (such as image translation, rotation, scaling); readjust network parameters such as learning rate and batch size, and retrain the model to ensure that the performance meets the application requirements.

[0028] The above describes the present invention and its implementation manners. Such description is not restrictive. What is shown in the accompanying drawings is only one of the implementation manners of the present invention, and the actual structure is not limited thereto. Generally speaking, if those of ordinary skill in the art are inspired by it and design, without creative efforts, structural manners and embodiments similar to the technical solution without departing from the gist of the present invention, they shall fall within the protection scope of the present invention.

Claims

1. A seed detection method based on fusion of three-dimensional point cloud and hyperspectral features, characterized in that: The method comprises the following steps: S1. Point cloud data acquisition and preprocessing: ① Point cloud data acquisition: Use 3D scanning equipment to perform multi-view 3D scanning of wheat seeds to obtain point cloud data of seeds, including size, surface morphology and spatial structure information; Place wheat seeds on a turntable during collection, and realize multi-angle scanning by rotating the turntable; ② Point cloud data preprocessing: De-noising: Remove low-density points, isolated noise points, scanning noise and abnormal points in the point cloud; Coordinate normalization: Normalize the spatial coordinates of point cloud data; Voxelization: Segment the voxels of Ji'an point cloud data into 3D grids; Multi-view projection: Project the preprocessed point cloud data to multiple viewpoints to generate a 2D depth map for subsequent convolutional neural network input; S2. 3D data feature extraction and classification based on convolutional neural network: ① Data input: input the preprocessed point cloud data into the convolutional neural network in different forms: Depth map: input the 2D depth map of multi-view projection into the 2D convolutional neural network; Voxel data: input the voxelized data into the 3D convolutional neural network; Point cloud data: input the original point cloud data into the PointNet or PointNet++ neural network; ② Feature extraction: perform the following operations in the convolutional neural network: Convolution operation: extract the spatial morphological features of wheat seeds through the convolutional layer; Pooling operation: reduce the dimension through maximum pooling or average pooling to reduce feature redundancy; Feature fusion: form 3D features by fusing the extracted multi-level features through the fully connected layer; ③ Classification: input the extracted 3D features into the Softmax classifier, perform preliminary classification of wheat seeds, and output the classification results; S3. Hyperspectral data acquisition and preprocessing: ① Hyperspectral data acquisition: use hyperspectral imaging equipment, set the band range to 400nm to 1000nm, and the spectral resolution is better than 1nm; wheat seeds are placed on the spectral acquisition platform and the light source irradiation conditions are kept constant; ② Data preprocessing: De-noising: use filtering algorithm to remove noise signals; Spectral normalization: perform minimum-maximum normalization on spectral data; Band selection: select key bands with classification contribution; S4. Hyperspectral data feature extraction and classification based on convolutional neural network: ① Hyperspectral data input: The hyperspectral data is input into the convolutional neural network in the following format: The spectral vector of each pixel is input into the one-dimensional convolutional neural network; The spectral-spatial data cube is input into the three-dimensional convolutional neural network; ② Feature extraction: Spectral feature extraction: In the one-dimensional convolutional neural network, the features of the spectral dimension are extracted by one-dimensional convolution; Spatial-spectral joint feature extraction: In the three-dimensional convolutional neural network, the joint features of the spectral dimension and the spatial dimension are extracted simultaneously by three-dimensional convolution; Feature fusion: The extracted spectral features and spatial-spectral joint features are fused through the fully connected layer; ③ Classification: The extracted and fused hyperspectral features are input into the Softmax classifier, and the wheat seeds are classified based on the spectral information, and the classification results are output; S5. Decision-level fusion of classification results: ① Weight allocation: Calculate the weight coefficient of each result based on the classification results of the 3D reconstructed point cloud data and the confidence and accuracy of the hyperspectral classification results; ② Decision fusion: Determine the final classification results based on the confidence and accuracy of the classification results and the weight coefficient; ③ Result output: Output the final classification results of wheat seeds, including variety, quality grade or health status.

2. The seed detection method based on fusion of three-dimensional point cloud and hyperspectral features according to claim 1 is characterized in that: The three-dimensional scanning equipment includes: a structured light scanner, which is suitable for high-precision three-dimensional point cloud acquisition, with a scanning resolution of not less than 0.1mm; a laser scanner, which is suitable for high-density point cloud data acquisition of complex-shaped seeds; and a depth camera, which can realize multi-view data acquisition and is used to generate a three-dimensional depth map.

3. The seed detection method based on fusion of three-dimensional point cloud and hyperspectral features according to claim 1 is characterized in that: The preprocessing of the point cloud data specifically includes: denoising: using statistical filtering or radius filtering algorithm to remove low-density points, isolated noise points, scanning noise and abnormal points; coordinate normalization: normalizing the spatial coordinates of the point cloud data to the range of [0,1] to facilitate convolution operations; voxelization: using the octree segmentation algorithm to perform three-dimensional grid processing on the point cloud data, and the size of each voxel unit is not greater than 1 mm³; multi-view projection: projecting the point cloud data from different angles to generate a two-dimensional depth map, and each view covers different surface features of the seed.

4. The seed detection method based on fusion of three-dimensional point cloud and hyperspectral features according to claim 1 is characterized in that: The three-dimensional data feature extraction and classification based on convolutional neural network includes: inputting the depth map of multi-view projection into the two-dimensional convolutional neural network to extract surface structure features; inputting voxelized data into the three-dimensional convolutional neural network to extract spatial morphological features; inputting the original point cloud data into the PointNet or PointNet++ network to extract global spatial features; performing feature-level fusion on the above features, uniformly representing the feature vectors through the fully connected layer, and inputting them into the Softmax classifier to complete classification.

5. The seed detection method based on fusion of three-dimensional point cloud and hyperspectral features according to claim 1 is characterized in that: The collection of the hyperspectral data includes: setting the band range of the hyperspectral imaging device to 400nm to 1000nm, with a spectral resolution better than 1nm; maintaining constant light source illumination during the collection process, and using a uniform diffuser to reduce uneven lighting; using the line scanning mode of the hyperspectral camera to achieve line-by-line spectral collection of the seed surface, ensuring that the spatial resolution is not less than 100µm.

6. The seed detection method based on fusion of three-dimensional point cloud and hyperspectral features according to claim 1 is characterized in that: The preprocessing of the hyperspectral data includes: spectral denoising: using Savitzky-Golay smoothing filtering or wavelet transform to remove spectral noise; band selection: selecting the spectral band that contributes most to classification as the key band through principal component analysis or correlation analysis; spectral normalization: using the minimum-maximum normalization method to standardize the spectral data; data dimensionality reduction: using linear discriminant analysis to reduce the spectral dimension while retaining key features.

7. The seed detection method based on fusion of three-dimensional point cloud and hyperspectral features according to claim 1 is characterized in that: The hyperspectral data feature extraction and classification based on convolutional neural network includes: using a one-dimensional convolutional neural network to extract features from the spectral vector of a single pixel and extracting spectral dimension features; using a three-dimensional convolutional neural network to perform convolution operations on spectral-spatial cube data and extract joint features of spectral and spatial dimensions; mapping the spectral features to the classification space through a fully connected layer and using a Softmax classifier to complete classification.

8. The seed detection method based on fusion of three-dimensional point cloud and hyperspectral features according to claim 1 is characterized in that: The decision fusion includes: weighted voting method: weights are assigned according to the classification accuracy of the classification model, and the classification result with the highest confidence is voted; weighted average method: weighted fusion of the results is performed according to the confidence of the classification results to generate the final classification result; through the dynamic adjustment mechanism of fusion weights, the fusion strategy is updated in real time according to the changes in the characteristics of the input data to optimize the classification performance.

9. The seed detection method based on fusion of three-dimensional point cloud and hyperspectral features according to claim 1 is characterized in that: The method also includes: verifying the final classification result, evaluating the classification performance by using accuracy, recall rate and F1 score; and analyzing the classification error by using a confusion matrix, optimizing the model used for three-dimensional data feature extraction and classification and the model used for hyperspectral data feature extraction pre-classification.

10. The seed detection method based on fusion of three-dimensional point cloud and hyperspectral features according to claim 1, characterized in that: This method is applicable to the classification of various wheat seed varieties, including variety identification, quality grade assessment and seed health status detection.

Images