Corn ear phenotypic parameter measurement method, system, photography method and device

Abstract

A corn ear phenotypic parameter measurement method, system, photography method, and device relate to the interdisciplinary field of agricultural science and computer vision, addressing the existing lack of efficient, accurate, and automated methods for comprehensive corn ear phenotypic parameter measurement. A corn ear photography device is constructed to obtain multi-angle two-dimensional images of the corn ear; the multi-angle two-dimensional images of the corn ear are processed to obtain two-dimensional phenotypic parameters of the corn ear; the corn ear is three-dimensionally reconstructed to obtain a three-dimensional reconstructed image of the corn ear; corn ear defect detection is performed on the three-dimensional reconstructed image of the corn ear based on an improved PointNet++ model to obtain a corn ear defect classification result; the corn ear phenotypic parameters are measured and a corn ear phenotypic parameter data report is generated. The present invention is used to achieve efficient, accurate, and automated comprehensive corn ear phenotypic parameter measurement.

Description

Technical Field

[0001] The present invention relates to the field of intersectional technology of agricultural science and technology and computer vision, and in particular to a method, system, shooting method and device for measuring phenotypic parameters of corn ears. Background Art

[0002] Corn is an important global food crop, and its yield and quality directly impact food security. Corn ears, a direct indicator of yield and quality, are crucial indicators in corn breeding and cultivation research, with phenotypic parameters such as ear length, ear width, number of ear rows, and number of kernels per row serving as key indicators. However, traditional methods for measuring corn ear phenotyping, such as manual measurement and vernier scale measurement, suffer from large measurement errors, low efficiency, and an inability to fully capture morphological characteristics.

[0003] Traditional manual measurement methods, such as manually measuring ear length and diameter, are not only inefficient but also susceptible to human error. This method is difficult to implement in large-scale maize phenotyping and cannot provide accurate spatial geometric data.

[0004] Many existing 3D reconstruction technologies exist, including structured light 3D reconstruction, LiDAR (laser scanning 3D reconstruction), stereo vision, photometric stereo, and deep learning-based image-based 3D reconstruction. However, MVS (multi-view stereo) 3D reconstruction generates a high-density point cloud by combining multi-view images with depth calculations, offering higher accuracy and robustness. Structured light 3D reconstruction projects a known light pattern and uses a camera to capture the reflected information for 3D reconstruction, requiring high environmental requirements. LiDAR uses laser beams to scan and acquire surface data, but the equipment is expensive and its performance is poor when reconstructing fine details and small objects. Stereo vision uses two cameras to capture images from different angles, but has high requirements for viewing angle and image quality. Photometric stereo uses images under different lighting conditions to infer the surface morphology of an object, but its performance is poor for objects with complex textures. Deep learning-based image-based 3D reconstruction uses deep learning models to extract features from images for 3D reconstruction, but requires a large amount of data and training.

[0005] At the same time, the existing measurement of corn ear phenotypic parameters also has the following defects:

[0006] 1. Insufficient automation: Existing phenotypic parameter extraction methods mostly rely on manual or semi-automated tools, requiring manual calibration and adjustment. These methods are cumbersome and cannot be fully automated, limiting their application in large-scale crop monitoring.

[0007] 2. Incomplete parameter extraction: Existing methods usually only extract some basic phenotypic parameters (such as length and diameter), but are unable to comprehensively and accurately extract the multi-dimensional phenotypic characteristics of the fruit cluster.

[0008] 3. Inconsistent data processing: The data formats, algorithm differences, and parameter processing methods between different technical platforms and devices are not uniform, resulting in a lack of consistency in the final phenotypic data, affecting cross-platform analysis and application of data.

[0009] Corn ears are an important basis for crop yield and quality assessment. The accurate identification of diseased areas is of great significance in intelligent breeding and quality screening. Most existing 3D point cloud segmentation technologies use PointNet (point cloud network) or PointNet++ as the basic model. Although they have certain semantic recognition capabilities, they are limited in extracting complex surface textures and structural details, making it difficult to accurately capture fine-grained semantic features, which affects recognition accuracy. The PointNet++ model achieves multi-scale feature extraction and segmentation of dense point clouds through a hierarchical SA (set abstraction) and FP (feature propagation) architecture. However, in the practical application of corn ear point clouds, the following difficulties still exist:

[0010] 1. Sparse features in low-density areas are difficult to aggregate: The number of point clouds in areas with missing grains, rotten grains, and bald tips is sparse, and the SA layer does not fully extract features from these areas, which easily leads to the formation of "blind spots" after segmentation.

[0011] 2. Weak relationship modeling for irregular morphological regions: The morphology of diseased areas or special parts is usually very irregular, such as bald points and decayed areas. The structural information in the point cloud is sparse, and traditional aggregation methods have difficulty in adaptively capturing such complex geometric shapes.

[0012] 3. Insufficient local-global context fusion: The SA layer can only aggregate features within a preset neighborhood scale, making it difficult to dynamically interact with local and global information, resulting in limited performance in identifying similar semantic clusters. The PointNet++ model effectively extracts local features through multi-scale aggregation operations, but its ability to integrate global context information across different scales is limited, leading to the misclassification of similar semantic regions, such as the healthy and rotten parts of a corn ear.

[0013] Image-based 3D reconstruction technology has achieved some success in phenotyping various plant species, but accurate 3D reconstruction of corn ears and the measurement of their phenotypic parameters remain at a research and exploration stage. Existing technologies mostly rely on point cloud data reconstruction or manual calibration, lacking a comprehensive, efficient, accurate, and automated approach.

[0014] In the prior art, Chinese patent document CN118196282A discloses a "device and method for three-dimensional reconstruction of corn ears based on neural radiation fields." The device includes a host control terminal, a control panel, a servo, a pan-tilt head, an angle sensor, a camera, a display module, a fixed chassis, a background plate, and a tripod. The method uses the host control terminal and camera to capture a multi-frame image sequence of corn ears from 0 to 360 degrees, which serves as training sample data for subsequent models. A corn ear dataset is generated and preprocessed to restore the camera pose. Three-dimensional reconstruction is performed using a neural radiation field network to obtain corn ear point cloud information. Finally, phenotypic parameters of the corn ears are calculated. However, this technical solution relies on point cloud data reconstruction, which has high data dependence and limited generalization capabilities. Furthermore, the captured photos are easily affected by the environment, resulting in inaccurate data.

[0015] In summary, the existing technology lacks an efficient, accurate and automated method for measuring comprehensive corn ear phenotypic parameters. Summary of the Invention

[0016] The present invention solves the problem that the prior art lacks an efficient, accurate and automated method for measuring comprehensive corn ear phenotypic parameters.

[0017] The method for measuring corn ear phenotypic parameters of the present invention comprises the following steps:

[0018] Step 1: Build a corn ear shooting device;

[0019] Step 2, obtaining a multi-angle two-dimensional image of the corn ear based on a corn ear photographing device;

[0020] Step 3, processing the multi-angle two-dimensional image of the corn ear to obtain the two-dimensional phenotypic parameters of the corn ear;

[0021] Step 4, performing three-dimensional reconstruction of the corn ear based on the multi-angle two-dimensional image of the corn ear to obtain a three-dimensional reconstructed image of the corn ear;

[0022] Step 5: Perform corn ear defect detection on the 3D reconstructed image of the corn ear based on the improved PointNet++ model to obtain a corn ear defect classification result;

[0023] Step 6: Based on the two-dimensional phenotypic parameters of the corn ear, the three-dimensional reconstructed image of the corn ear, and the classification result of the corn ear defects, the measurement of the phenotypic parameters of the corn ear is completed, and a corn ear phenotypic parameter data report is generated.

[0024] Furthermore, in the embodiment of the present invention, the processing of the multi-angle two-dimensional image of the corn ear in step 3 is specifically as follows:

[0025] The multi-angle two-dimensional image of the corn ear is segmented to obtain the corn ear region image, the corn ear region image is denoised to obtain the contour information of the corn ear region image, and the contour information of the corn ear is feature extracted to obtain the corn ear length and the corn ear width respectively, thereby completing the processing of the multi-angle two-dimensional image of the corn ear.

[0026] Furthermore, in an embodiment of the present invention, the three-dimensional reconstruction of the corn ear based on the multi-angle two-dimensional image of the corn ear in step 4 is specifically performed as follows:

[0027] Based on SFM, feature extraction and feature matching are performed on multi-angle two-dimensional images of corn ears to generate sparse point clouds. The sparse point clouds are expanded through MVS to obtain dense point clouds. A network model is generated based on the dense point clouds, and the network model is mesh smoothed. Texture mapping is performed on the mesh-smoothed network model to complete the three-dimensional reconstruction of the corn ear.

[0028] Furthermore, in an embodiment of the present invention, the step 5 of performing corn ear defect detection on the three-dimensional reconstructed image of the corn ear based on the improved PointNet++ model to obtain a corn ear defect classification result includes the following steps:

[0029] Step 51: Build a PointNet++ model and improve the PointNet++ model;

[0030] Step 52: Perform corn ear defect detection on the 3D reconstructed image of the corn ear based on the improved PointNet++ model to obtain a corn ear defect classification result.

[0031] Furthermore, in the embodiment of the present invention, the PointNet++ model is improved in step 51, specifically:

[0032] The relative position encoding module, local grouping rearrangement module and LRSA module are embedded in sequence after the SA second grouping layer of the PointNet++ model. The relative position encoding module is used to provide the relative position information of the output features of the SA second grouping layer. The local grouping rearrangement module rearranges the relative position information of the output features of the SA second grouping layer. The LRSA module is used to enhance the adaptive features of abnormal areas in the features after the relative position information is rearranged.

[0033] Further, in an embodiment of the present invention, the corn ear phenotypic parameter data report in step 6 includes the detection results of the number of rows and kernels in each row of corn ears, the measurement results of the length and width of corn ears, the corn kernel counting results, the detection results of bald tips of corn ears, the detection results of ear disease of corn ears, the detection results of missing kernels in corn ears and the detection results of abnormal color of corn ears.

[0034] The corn ear photographing device of the present invention is constructed according to the corn ear phenotypic parameter measurement method of the present invention, and the device includes a host control terminal 1, a rotating turntable 2, an RGB-D camera 4, an adjustment arm 5, a double-layer bracket 6, a fill light 7, a storage rack 8, a light shield 10, a motor 11 and a remote control 3;

[0035] The rotating turntable 2 is arranged on the lower layer of the double-layer bracket 6, the adjusting arm 5 is installed above the rotating turntable 2, the RGB-D camera 4 is installed on the adjusting arm 5, a plurality of fill lights 7 are arranged on the inner side of the upper layer of the double-layer bracket 6, the storage rack 8 is arranged in the middle of the rotating turntable 2, and a fixing nail is provided in the center of the storage rack 8, and the corn cob is inserted into the fixing nail. The host control end 1 is connected to the RGB-D camera 4, and the remote control 3 is used to control the motor 11 to adjust the start and stop and speed of the rotating turntable 2. The light shield 10 is arranged on the outside of the double-layer bracket 6.

[0036] The corn ear photographing method of the present invention is implemented according to the corn ear photographing device of the present invention, specifically as follows:

[0037] The motor 11 is controlled by the remote control 3 to rotate the rotating turntable 2 360°. The rotating turntable 2 drives the RGB-D camera 4 to aim at the corn ear inserted on the fixed nail of the rack 8. The host control terminal 1 is used to control the RGB-D camera 4 to shoot a 360° video of the corn ear inserted on the fixed nail of the rack 8. Each second of the 360° video of the corn ear is divided into a photo to complete the shooting of the corn ear.

[0038] The corn ear phenotypic parameter measurement system of the present invention is implemented according to the above-mentioned corn ear phenotypic parameter measurement method, and includes the following modules:

[0039] Build the module and construct the corn ear shooting device;

[0040] A shooting module, based on a corn ear shooting device, obtains a multi-angle two-dimensional image of the corn ear;

[0041] A processing module processes the multi-angle two-dimensional images of the corn ear to obtain the two-dimensional phenotypic parameters of the corn ear;

[0042] A reconstruction module, which performs three-dimensional reconstruction of the corn ear based on the multi-angle two-dimensional image of the corn ear to obtain a three-dimensional reconstructed image of the corn ear;

[0043] The detection module performs corn ear defect detection on the 3D reconstructed image of the corn ear based on the improved PointNet++ model to obtain the corn ear defect classification results;

[0044] The generation module completes the measurement of the phenotypic parameters of the corn ear based on the two-dimensional phenotypic parameters of the corn ear, the three-dimensional reconstructed image of the corn ear and the classification results of the corn ear defects, and generates a data report on the phenotypic parameters of the corn ear.

[0045] The present invention solves the problem that the existing technology lacks an efficient, accurate, and automated method for measuring comprehensive corn ear phenotypic parameters. Specific beneficial effects include:

[0046] 1. The corn cob phenotypic parameter measurement method of the present invention utilizes computer vision and 3D reconstruction technology to build a corn cob shooting device to collect multi-angle 2D images of corn cobs, performs 3D reconstruction of the corn cobs based on the multi-angle 2D images of the corn cobs, obtains a 3D reconstructed image of the corn cobs, utilizes an improved PointNet++ model to perform point cloud segmentation and defect classification, performs corn cob defect detection on the 3D reconstructed image of the corn cobs, obtains a corn cob defect classification result, and efficiently, accurately, and automatically completes the measurement of corn cob phenotypic parameters based on the 2D phenotypic parameters of the corn cobs, the 3D reconstructed image of the corn cobs, and the corn cob defect classification result, and generates a corn cob phenotypic parameter data report;

[0047] 2. The improved PointNet++ model of the present invention uses a relative position encoding module, a local grouping rearrangement module, and an LRSA (local region self-attention) module to sequentially embed the relative position encoding module, the local grouping rearrangement module, and the LRSA (local region self-attention) module after the second grouping layer of the SA. The relative position encoding module integrates the geometric relationship between points into the feature expression, enhancing the spatial structure perception capability. The local grouping rearrangement module rearranges the disordered point features into a regular grid, providing a standardized input format for subsequent self-attention calculations. These two preprocessing steps effectively "format" the spatial and topological information before being processed by the LRSA module. This enables the attention layer to more accurately capture the semantic dependencies of local regions, establish stronger local context modeling capabilities, enhance the point cloud feature's ability to express spatial structural differences, and improve the accuracy of defect classification.

[0048] The corn ear phenotypic parameter measurement method described in the present invention can efficiently and accurately reconstruct the three-dimensional model of the corn ear and automatically extract its phenotypic parameters for use in agricultural research, breeding improvement and precision agriculture. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0050] Figure 1 This is a flow chart of the method for measuring corn ear phenotypic parameters according to embodiment one;

[0051] Figure 2 This is a diagram of the corn ear photographing device described in Embodiment 2;

[0052] Figure 3 The original image of corn ears and the selected image in Lab color space described in the first embodiment;

[0053] Figure 4 This is the maximum inter-class variance algorithm threshold segmentation result diagram described in Implementation Method 1;

[0054] Figure 5 The three-dimensional reconstructed image of the corn ear described in the first embodiment;

[0055] Figure 6 It is the improved PointNet++ model described in Implementation 1. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe various embodiments of the present invention in conjunction with the accompanying drawings. The embodiments described with reference to the accompanying drawings are exemplary and intended to be used to explain the present invention, but should not be understood as limiting the present invention.

[0057] Embodiment 1. The method for measuring corn ear phenotypic parameters according to this embodiment comprises the following steps:

[0058] Step 1: Build a corn ear shooting device;

[0059] Step 2, obtaining a multi-angle two-dimensional image of the corn ear based on a corn ear photographing device;

[0060] Step 3, processing the multi-angle two-dimensional image of the corn ear to obtain the two-dimensional phenotypic parameters of the corn ear;

[0061] Step 4, performing three-dimensional reconstruction of the corn ear based on the multi-angle two-dimensional image of the corn ear to obtain a three-dimensional reconstructed image of the corn ear;

[0062] Step 5: Perform corn ear defect detection on the 3D reconstructed image of the corn ear based on the improved PointNet++ model to obtain a corn ear defect classification result;

[0063] Step 6: Based on the two-dimensional phenotypic parameters of the corn ear, the three-dimensional reconstructed image of the corn ear, and the classification result of the corn ear defects, the measurement of the phenotypic parameters of the corn ear is completed, and a corn ear phenotypic parameter data report is generated.

[0064] In this embodiment, the processing of the two-dimensional image of the corn ear in step 3 is specifically as follows:

[0065] The multi-angle two-dimensional image of the corn ear is segmented to obtain the corn ear region image, the corn ear region image is denoised to obtain the contour information of the corn ear region image, and the contour information of the corn ear is feature extracted to obtain the corn ear length and the corn ear width respectively, thereby completing the processing of the multi-angle two-dimensional image of the corn ear.

[0066] In this embodiment, the three-dimensional reconstruction of the corn ear based on the multi-angle two-dimensional image of the corn ear in step 4 is specifically as follows:

[0067] Based on SFM, feature extraction and feature matching are performed on multi-angle two-dimensional images of corn ears to generate sparse point clouds. The sparse point clouds are expanded through MVS to obtain dense point clouds. A network model is generated based on the dense point clouds, and the network model is mesh smoothed. Texture mapping is performed on the mesh-smoothed network model to complete the three-dimensional reconstruction of the corn ear.

[0068] In this embodiment, the step 5 of performing corn ear defect detection on the three-dimensional reconstructed image of the corn ear based on the improved PointNet++ model to obtain the corn ear defect classification result includes the following steps:

[0069] Step 51: Build a PointNet++ model and improve the PointNet++ model;

[0070] Step 52: Perform corn ear defect detection on the 3D reconstructed image of the corn ear based on the improved PointNet++ model to obtain a corn ear defect classification result.

[0071] In this embodiment, the PointNet++ model is improved in step 51, specifically:

[0072] The relative position encoding module, local grouping rearrangement module and LRSA module are embedded in sequence after the SA second grouping layer of the PointNet++ model. The relative position encoding module is used to provide the relative position information of the output features of the SA second grouping layer. The local grouping rearrangement module rearranges the relative position information of the output features of the SA second grouping layer. The LRSA module is used to enhance the adaptive features of abnormal areas in the features after the relative position information is rearranged.

[0073] In this embodiment, the corn ear phenotypic parameter data report in step 6 includes the detection results of the number of rows and kernels in the corn ears, the measurement results of the ear length and width of the corn ears, the corn kernel counting results, the detection results of the bald tips of the corn ears, the detection results of the ear disease of the corn ears, the detection results of the missing kernels in the corn ears and the detection results of the abnormal color of the corn ears.

[0074] In the existing technology, there is a lack of efficient, accurate, and automated comprehensive corn ear phenotypic parameter measurement methods. To solve the above technical problems, this embodiment proposes a corn ear phenotypic parameter measurement method, which uses efficient and accurate three-dimensional reconstruction technology to obtain the geometric morphology of corn ears and perform automated measurement of phenotypic parameters, such as Figure 1 As shown, the specific steps include:

[0075] Step 1: Build a corn ear shooting device for shooting corn ears from multiple angles;

[0076] Build a corn ear photography setup based on the experimental requirements, ensuring the stability of the rotating turntable 2, RGB-D camera 4, and fill light 7. Calibrate the RGB-D camera 4 to ensure consistent parameters for each shot.

[0077] Step 2, using a corn ear shooting device and an RGB-D camera 4 to shoot a 360° image of the corn ear to obtain a multi-angle two-dimensional image of the corn ear, specifically comprising the following steps:

[0078] Step 21: Capture multi-angle two-dimensional images of corn ears. The remote controller 3 controls the motor 11 to rotate the rotating turntable 2, which drives the RGB-D camera 4 to focus on a certain angle of the corn ear. The RGB-D camera 4 can cover all angles of the corn ear.

[0079] Step 22, using the host control terminal 1 to control the RGB-D camera 4 to shoot a 360° video of the corn ear, dividing each second in the video into a photo, thereby obtaining color images of the corn ear at different angles, that is, multi-angle two-dimensional images of the corn ear.

[0080] Step 3, performing two-dimensional image processing on the collected multi-angle two-dimensional images of the corn ear to obtain two-dimensional phenotypic parameters of the corn ear, specifically comprising the following steps:

[0081] Step 31, using the multi-angle two-dimensional image of the corn ear captured by the RGB-D camera 4 to perform corn ear region segmentation using the Lab (brightness, red-green, yellow-blue) color model and the maximum inter-class variance method;

[0082] Unlike the RGB (red, green, and blue) space, the Lab color space is considered a uniform color space. The changes in different color areas within this space are similar to the perception of human vision. Therefore, it is conducive to computers extracting the feature information contained in multi-angle two-dimensional images of real corn ears.

[0083] The Lab color space is a uniform color space that divides color information into three components:

[0084] The L (luminance) channel reflects the brightness information of the image, that is, the change from black to white, with a value range of 0 to 100.

[0085] The a (red-green) channel reflects the color change from green to red, and its value range is -128 to 127.

[0086] The b (yellow-blue) channel reflects the color change from blue to yellow, with a value range of -128 to 127.

[0087] A key advantage of the Lab color space is its perceptual consistency. Due to its properties in terms of color and brightness separation, the Lab color space is more suitable for detailed analysis of color than the traditional RGB space.

[0088] The conversion formula between RGB space and Lab color space is as follows:

[0089] ; (1)

[0090] ; (2)

[0091] ; (3)

[0092] ; (4)

[0093] ; (5)

[0094] Where, They are the three components in the XYZ color space, and R, G, and B are the three components in the RGB space. is brightness (brightness), and is chroma (hue), The reference white point value, used for nonlinear correction.

[0095] The maximum inter-class variance method, also known as the Otsu (maximum inter-class variance) algorithm, can separate a multi-angle 2D image of a corn ear into two distinct regions. Based on the background and target, the maximum variance is selected as the segmentation threshold. The larger the variance, the greater the difference between the two regions. By combining a uniform color space with automated threshold selection, the corn ear region image can be accurately and efficiently extracted.

[0096] Convert the multi-angle 2D image of corn ears from RGB space to Lab color space. In Lab color space, the L channel provides brightness information. Using the a and b channels can further separate the color features of the corn ear area image and distinguish it from the color of the background area. Figure 3As shown in the figure, the left picture is the original image taken by the camera, and the right picture is the Lab color space selection picture. When performing corn ear segmentation, the L channel can be used for brightness threshold segmentation to further extract the corn ear area image. By thresholding the L channel using the maximum inter-class variance method, the corn ear area image and the background area image can be effectively separated from the multi-angle two-dimensional image of the corn ear. Especially when the brightness difference between the corn ear area and the background area is obvious, the maximum inter-class variance method can automatically determine the optimal threshold for segmentation, such as Figure 4 As shown in the figure, the color difference information in the a and b channels can also be used to further optimize the color, identify and eliminate possible background noise, and ensure a more accurate image of the corn ear region. Combining this with subsequent median filtering and morphological processing can further remove noise and correct the outline of the corn ear.

[0097] Step 32, removing noise from the corn ear region image by median filtering and morphological processing, obtaining contour information of the corn ear by using an internal gradient algorithm, and obtaining the number of corn ear rows and the number of kernels per row;

[0098] In corn ear region images, median filtering can effectively remove noise caused by environmental conditions and camera angle variations while also preventing loss of corn ear outline information. After extracting the corn ear region image, further extracting the corn ear outline information can help us better analyze and measure features such as the shape and size of the corn ear. The internal gradient algorithm is used to extract the corn ear outline information. Based on the gradient information of the corn ear region image, the internal gradient algorithm detects areas with large pixel value changes within the corn ear region image, which typically correspond to object edges.

[0099] Use contour detection to count the kernel boundaries on the surface of the corn ear. Use edge detection, Hough transform, or morphological processing to identify the circular or elliptical structure of the corn kernels. Detect circular or elliptical features in multi-angle 2D images of the corn ear to identify the number of rows arranged on the surface of the corn ear.

[0100] K-means clustering is used to classify the detected corn kernels into different rows. The center positions of different rows are calculated to ensure that the rows of corn kernels are arranged correctly. The number of ear rows is calculated by detecting the horizontal distribution density of corn kernels. :

[0101] ; (6)

[0102] The number of corn kernels in each row can be counted using a feature-based segmentation method to obtain the row kernel count.

[0103] Step 33, obtaining the minimum circumscribed rectangle of the corn cob and the corn kernels, drawing horizontal and vertical lines in the minimum circumscribed rectangle, obtaining two-dimensional pixel coordinates of two pairs of intersection points, and thus obtaining the length and width of the corn cob.

[0104] Calculate the minimum enclosing rectangle of a corn ear. Draw horizontal and vertical lines within the minimum enclosing rectangle to obtain two-dimensional pixel coordinates. Using a reference object of known length and the principle of pinhole imaging, convert pixels to actual distances to calculate the length and width of the corn ear.

[0105] Using a reference object of known length, establish the proportional relationship between pixels and actual physical size:

[0106] ; (7)

[0107] ; (8)

[0108] ; (9)

[0109] Where, The length of the corn ear; It is the width of the corn ear.

[0110] Step 4, performing three-dimensional reconstruction of the corn ear based on the multi-angle two-dimensional image of the corn ear to obtain a three-dimensional reconstructed image of the corn ear, specifically comprising the following steps:

[0111] Step 41: perform camera calibration to obtain internal and external parameters of the RGB-D camera, and extract and match feature points in the multi-angle two-dimensional image of the corn ear using an image processing algorithm;

[0112] Step 42, using SFM to generate a sparse point cloud, calculates the camera pose and preliminary coordinates of the 3D feature points, specifically including the following steps:

[0113] Step 421: Before performing 3D reconstruction, the camera pose must be restored and a sparse point cloud must be generated using SFM. SFM estimates the internal and external parameters of the RGB-D camera by pairing and matching multiple multi-angle 2D images of corn ears and generates a sparse point cloud. This process mainly includes two key steps: feature extraction and feature matching. Recognizable feature points are extracted from multiple multi-angle 2D images of corn ears, such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded Robust Features), or ORB (Object Request Broker). Corresponding feature points are found between the multi-angle 2D images of corn ears, and a correspondence between the multi-angle 2D images of corn ears is established. The RGB-D camera position and orientation, i.e., the camera pose, of each multi-angle 2D image of corn ears is calculated using the matched feature points. Based on the matching of camera poses and feature points, a sparse point cloud is generated to represent the basic structure of the scene. COLMAP (3D Reconstruction Tool) and OpenSFM (Open Structure from Motion) can be used for SFM reconstruction to produce an output containing camera pose and sparse point cloud data.

[0114] Step 422: Data format conversion. OpenMVS (Multi-view Stereo Reconstruction System) does not directly support SFM data generated by COLMAP or OpenSFM. Therefore, the SFM output must be converted to a format that OpenMVS can use. OpenMVS provides a tool, InterfaceCOLMAP (3D Reconstruction Tool Interface), which converts COLMAP's cameras.txt (camera parameter file), images.txt (image list file), and points3D.txt (3D point list) files into .mvs (Multi-view Stereo) files, the input format for OpenMVS. The converted .mvs files contain RGB-D camera parameters, sparse point clouds, and other necessary metadata.

[0115] Step 43, generate a depth map through MVS and synthesize it into a dense point cloud, further optimize the point cloud quality, and finally perform surface reconstruction to obtain a 3D reconstructed image of the corn ear, as shown in the figure. Figure 5 As shown, the specific steps include:

[0116] Step 431, dense point cloud reconstruction is to expand the sparse point cloud through MVS to generate a dense point cloud; through disparity estimation (or depth estimation), a depth map is calculated for each pair of matching corn ear multi-angle two-dimensional images. The depth value of each pixel reflects the distance of the object surface corresponding to the pixel from the camera. The consistency of the depth map is ensured by multi-view matching to remove erroneous estimates and noise. OpenMVS will consider the depth information of the same object from multiple perspectives and perform optimization. Using the calculated depth map and camera pose, the dense point cloud is gradually restored. The three-dimensional coordinates of each point are inversely calculated from the depth map to form the final dense point cloud. In OpenMVS, this step is implemented by the DensifyPointCloud tool. It will generate a .mvs file containing a large number of three-dimensional points and prepare for subsequent surface reconstruction;

[0117] Step 432, convert the dense point cloud into a simpler and more structured mesh model through triangulation to perform surface reconstruction; surface reconstruction is to triangulate the dense points of the dense point cloud to form a closed geometric surface. Connect the points in the dense point cloud by constructing a surface. Common surface reconstruction methods include Poisson reconstruction or Delaunay triangulation. Connect adjacent points in the dense point cloud through triangles to generate a mesh model. This process converts the dense point cloud into a triangular mesh (a geometric structure composed of faces, edges and vertices). In OpenMVS, the ReconstructMesh tool can complete this process. It will generate a triangular mesh based on the dense point cloud and output a .mvs file containing the generated mesh model information;

[0118] Step 433, optimize the mesh model. After the surface reconstruction, there may be surface noise or uneven areas. At this time, the mesh model needs to be optimized to make it smoother and finer. By removing irregular triangles and noise points, the mesh is smoothed to make the surface of the mesh model smoother. Refine the details of the mesh model to make the surface more realistic and the boundaries clearer. OpenMVS provides the RefineMesh tool to further optimize the mesh model, remove messy details and improve the quality of the mesh model, generating a smoother and finer mesh model.

[0119] Step 434, texture mapping is performed to map the color information obtained from the input multi-angle two-dimensional image of the corn ear to each triangular face of the mesh model. Based on the camera pose and the mesh model, OpenMVS will map the color information in the multi-angle two-dimensional image of the corn ear to the surface of the mesh model, so that the surface of the mesh model presents the content details of the multi-angle two-dimensional image of the corn ear. The texture coordinates are calculated and a texture map (usually one or more image texture layers) is generated. Through the TextureMesh tool, OpenMVS can generate a three-dimensional reconstruction image of the corn ear with texture. The output result is usually a .mvs file containing texture data;

[0120] In step 435, the generated 3D reconstructed image of the corn ear can be exported to common 3D file formats, such as .ply (point cloud file) and .obj (object file). This format can then be imported into various 3D visualization software for viewing and further processing. Software such as MeshLab (a 3D mesh processing and reconstruction tool) and CloudCompare (point cloud processing software) can be used to view and process mesh models and dense point clouds.

[0121] MVS 3D reconstruction can handle complex morphologies and is superior to methods such as laser scanning and stereo vision in terms of low cost and high efficiency. At the same time, MVS 3D reconstruction has relatively low requirements for lighting changes and texture, and is suitable for more complex and detailed crop phenotypic measurements. Therefore, it has obvious advantages in the application of corn ear 3D reconstruction.

[0122] Step 5: Perform corn ear defect detection on the 3D reconstructed image of the corn ear based on the improved PointNet++ model to obtain a corn ear defect classification result, which specifically includes the following steps:

[0123] Step 51: Create a data set and use CloudCompare to annotate the dense point cloud of the corn ear, marking normal corn kernels, missing kernels, bald tips, and diseased parts.

[0124] Step 52: Build a PointNet++ model and improve the PointNet++ model;

[0125] Since the PointNet++ model has the ability to directly process unstructured point cloud data and the characteristics of hierarchical multi-scale feature extraction, the PointNet++ model is used to detect corn ear defects, and can realize the preliminary segmentation of point clouds for features of different spatial scales. However, if the PointNet++ model is directly used for point cloud segmentation, it will lead to insufficient segmentation accuracy and robustness due to low density, irregular shape and noise point interference. In order to solve the above technical problems, this embodiment introduces the LRSA module after the SA second grouping layer of the PointNet++ model, and uses local gridding and self-attention mechanism to realize adaptive feature enhancement and noise self-suppression of abnormal areas, which can realize the classification of normal kernels, missing kernels, bald tips and diseased parts on corn ears, thereby realizing defect detection of corn ears.

[0126] The LRSA module is a lightweight attention mechanism for local regions. Originally developed from the need to model two-dimensional structural data such as images, it aims to address the limitations of convolution operations, which have limited receptive fields in local regions and cannot dynamically model spatial relationships. By introducing a self-attention mechanism within a region, the LRSA module enables dynamic information interaction and feature enhancement between pixels or points while maintaining local computational efficiency. Compared to traditional convolution, the LRSA module is not only position-sensitive but also adaptively mines salient features and structural relationships within a neighborhood, demonstrating superior performance in tasks such as image semantic segmentation and object detection. It is particularly adept at capturing contextual associations between edge regions and small objects.

[0127] However, point cloud data is inherently sparse and disordered. Simply combining the PointNet++ model with the LRSA module results in complex and irregular local structures. Traditional LRSA modules face challenges in processing this type of disordered data because they rely on the regularity and positional relationships of the input data, which point cloud data lacks. To address these technical issues, RPE (Relative Position Encoding) is employed to provide clear relative position information for each point. This ensures that the LRSA module can accurately calculate the relationships between points in the disordered point cloud, thereby enhancing its performance in point cloud segmentation tasks.

[0128] At the same time, the local structures in point cloud data are often irregular. If only the PointNet++ model, LRSA module and RPE are combined, it will be difficult for the LRSA module to effectively model the spatial dependencies between points. Due to the uneven distribution of neighborhood points in the point cloud, LRSA tends to ignore important local information when directly processing these irregular local structures. To solve this problem, LGRM (Local Rearrangement Module) is introduced after RPE. LGRM converts the local structure of the point cloud into a regular format through rearrangement based on relative position information, enabling the LRSA module to better calculate local dependencies on structured input data, thereby improving the model's adaptability and accuracy to local structures.

[0129] By combining RPE and LGRM, the PointNet++ model effectively addresses the adaptation issues of the LRSA module for point cloud data. RPE enhances the LRSA module's ability to capture the spatial relationships between points in the point cloud, while LGRM ensures that the LRSA module can fully leverage the advantages of the self-attention mechanism on regularized local structures. This combined improvement makes the LRSA module more efficient when processing sparse, disordered, and irregular point cloud data, significantly improving the performance and robustness of 3D point cloud segmentation tasks.

[0130] The main parts of the improved PointNet++ model are: SA layer, RPE, LGRM, LRSA module, FP layer and classification head, such as Figure 6 As shown in the figure, these parts constitute the core structure of the improved PointNet++ model. After the second grouping layer of the SA model, the relative position encoding module, the local grouping rearrangement module, and the LRSA module are embedded. The relative position encoding module provides the relative position information of the output features of the second grouping layer of the SA. The local grouping rearrangement module rearranges the relative position information of the features. The LRSA module is used to enhance the adaptive features of abnormal areas in the features after the relative position information is rearranged. Specifically, the following steps are included:

[0131] In step 521, the SA layer is responsible for extracting local features from the dense point cloud and performing downsampling. Four SA layers are used in the improved PointNet++ model, as follows:

[0132] SA1 (SA first grouping layer): The input point cloud dimension is 9 (including coordinates and features), the output point cloud dimension is 64, the radius is 0.1, the number of points in the neighborhood is 32, and three convolutional layers are used. Perform feature extraction.

[0133] SA2 (SA second grouping layer): The input point cloud dimension is 64, the output point cloud dimension is 128, the radius is 0.2, the number of points in the neighborhood is 64, and three convolutional layers are used. Perform feature extraction.

[0134] SA3 (SA third grouping layer): The input point cloud dimension is 128, the output point cloud dimension is 256, the radius is 0.4, the number of points in the neighborhood is 128, and three convolutional layers are used. Perform feature extraction.

[0135] SA4 (SA fourth grouping layer): The input point cloud dimension is 256, the output point cloud dimension is 512, the radius is 0.8, the number of points in the neighborhood is 256, and three convolutional layers are used. Perform feature extraction.

[0136] The role of these SA layers is to extract local features from the dense point cloud layer by layer and gradually reduce the number of point clouds for subsequent processing.

[0137] In step 522, RPE is used to calculate the relative coordinates and Euclidean distance of each point cloud in the output features of the second grouping layer of SA relative to the centroid of the layer, and 128-dimensional position encoding features are obtained through 1×1 convolution + BatchNorm (batch normalization) + ReLU (rectified linear unit).

[0138] Calculate the centroid of the point cloud after the second layer of aggregation:

[0139] ; (10)

[0140] Where, is the normalization factor, For the The weight vector of the position, Is a set of weight vectors, calculated for each point , and distance , spliced ​​together to get , For the relative position encoding vector.

[0141] Through 1*1 convolution mapping:

[0142] ; (11)

[0143] Where, is the weight matrix of the 1×1 convolution layer. 1×1 convolution is equivalent to point-by-point full connection. is the bias vector of the same layer, For the The relative position encoding value of each position is added point by point to the RPE feature and the original point feature:

[0144] ; (12)

[0145] Where, is the initial position code, Encode the updated position.

[0146] Step 523, LGRM rearranges the remaining point features (B×128×256) of the second grouping layer of SA into a 16×16 grid, and performs mirror filling on the points that are less than 256 to generate a structured grid tensor (B×128×16×16);

[0147] Calculate total network capacity:

[0148] ; (13)

[0149] Where, is the height of the grid, The width of the grid.

[0150] Padding: If the total size of the grid Greater than the number of points , then it needs to be filled, the amount of filling is , these padded features will be concatenated to the original feature tensor.

[0151] Rearrange: Rearrange the feature tensor into a shape of This grid structure enables local operations (such as convolution or attention mechanism) to be performed in the neighborhood. is the batch size, is the number of channels.

[0152] Step 524: Use the LRSA module to enhance the adaptive features of the abnormal area in the features after the relative position information is rearranged:

[0153] 1. Input the annotated dense point cloud into the PointNet++ model, which contains XYZ (spatial position information) and RGB (color information), and use the SA first and SA second grouping layers in the PointNet++ model to extract low-level spatial geometric features;

[0154] 2. After the second SA grouping layer, after REP and LGRM, and before the third SA grouping layer, the LRSA module is inserted. Its lightweight local attention mechanism is used to perform context modeling on the middle-level features output by the second SA grouping layer, thereby enhancing the semantic dependency between different parts.

[0155] 3. Use the LRSA module output as the input of the SA third packet layer to further abstract it;

[0156] 4. Feature propagation and upsampling are performed layer by layer through the FP layer, and finally the semantic label of each point is output to identify rotten, missing grains, bald tips and normal areas.

[0157] In this embodiment, the LRSA module is chosen to be embedded between the SA second grouping layer and the SA third grouping layer of the PointNet++ model, mainly based on the semantic feature positioning and expression characteristics in the network hierarchy at this stage. The features output by the SA second grouping layer are in the transition stage from local geometry to mid-level semantic abstraction. At this time, the point cloud features still retain relatively rich spatial structure and fine-grained differences, which are suitable for introducing the attention mechanism for contextual relationship modeling to enhance the semantic dependency and regional consistency between points. At the same time, the introduction of the LRSA module before further abstraction of the SA third grouping layer helps to improve the network's ability to recognize local complex morphologies, such as ear rot or grain-missing areas, to avoid information loss caused by deep feature compression. Compared with inserting after the SA first grouping layer or after the SA third grouping layer, fusing the attention module between the SA second grouping layer and the SA third grouping layer can maximize the performance advantages of the local attention mechanism on the basis of taking into account both spatial structure perception and semantic abstraction depth, thereby effectively improving the model's expression and generalization capabilities in point cloud segmentation tasks.

[0158] Relative position encoding incorporates the geometric relationships between points (relative coordinates and distances) into feature representation, enhancing spatial structure perception. LGRM reorders disordered point features into a regular grid, providing a standardized input format for subsequent self-attention calculations. These two preprocessing steps effectively "format" the spatial and topological information before being processed by the LRSA module, enabling the attention layer to more accurately capture the semantic dependencies of local regions.

[0159] The core design of the LRSA module is as follows:

[0160] 1. Divide the input features into several overlapping local patches (region blocks).

[0161] 2. Apply a multi-head self-attention mechanism to each local patch to model the internal feature relationship of the local area.

[0162] In order to perform feature interactions within each local region, each patch is first flattened into a sequence:

[0163] ; (14)

[0164] Where, is the number of patches, is the patch size, is the area of ​​each local region, The number of channels per pixel.

[0165] Within each patch, a multi-head self-attention mechanism is used to capture the long-range dependencies between different positions within the local patch.

[0166] The generation process of query (Q), key (K), and value (V) is as follows:

[0167] ; (15)

[0168] ; (16)

[0169] ; (17)

[0170] Where, is an element in the input sequence, is the query weight matrix, is the key weight matrix, is the value weight matrix, , is the dimension of the key vector.

[0171] The attention weight calculation formula is:

[0172] ; (18)

[0173] Where, This is the transpose of the 𝐾 matrix, swapping the rows and columns of 𝐾.

[0174] The result is linearly projected:

[0175] ; (19)

[0176] Where, is the weight matrix of the output linear layer, is the output feature after attention mechanism and linear transformation.

[0177] Finally, combine the residual connection:

[0178] ; (20)

[0179] Where, is the original feature vector of the input, is the final output feature.

[0180] 3. Restore all patches to their original feature map shape and fuse the overlapping areas.

[0181] 4. Apply ConvFFN (local convolutional feedforward network) to further extract local features.

[0182] After the local attention module, ConvFFN is introduced to enhance feature expression.

[0183] The first step is linear transformation and activation:

[0184] ;(twenty one)

[0185] Where, is the weight matrix of the first linear transformation, is the feature vector after linear transformation and activation.

[0186] In the second step, depth-wise separable convolution is applied to further extract local structural information:

[0187] ;(twenty two)

[0188] Where, is the feature vector after depth-wise separable convolution operation.

[0189] The third step is linear projection to the original channel dimension:

[0190] ;(twenty three)

[0191] Where, is the weight matrix of the second linear transformation, is the eigenvector after linear projection.

[0192] 5. Strengthen feature expression through residual connection and output the final features.

[0193] Applying residual connections yields the final output:

[0194] ;(twenty four)

[0195] Where, is the final output feature.

[0196] In step 525, the FP layer is mainly used to gradually restore the resolution of the point cloud and propagate features from high layers to low layers. The network contains four FP layers, and their functions are as follows:

[0197] FP4: Fuse the output of the fourth SA grouping layer with the output of the third SA grouping layer for feature propagation.

[0198] FP3: Fuse the output of the third SA grouping layer with the output of the second SA grouping layer to continue propagating features.

[0199] FP2: Fuse the output of the second grouping layer of SA2 with the output of the first grouping layer of SA to continue propagating features.

[0200] FP1: The output of the first grouping layer of SA is fused with the original input features to finally restore the original point cloud resolution.

[0201] The FP layer ensures that the spatial structure information of the point cloud can be preserved by gradually restoring the resolution and propagating features to each point.

[0202] In step 526, the final part of the improved PointNet++ model is the classification head, which is responsible for predicting the category of each point based on its features. Specifically:

[0203] conv1 (a 1x1 convolutional layer): outputs 128-dimensional features.

[0204] bn1 (batch normalization layer): used to stabilize the training process.

[0205] drop1 (Dropout layer): used to prevent overfitting.

[0206] conv2 (a 1x1 convolutional layer): outputs num_classes (number of categories) channels, representing the classification probability of each point.

[0207] Softmax (normalized exponential function): Normalize the category probability of each point through log_softmax (logarithmic Softmax function) and output the category label of each point.

[0208] In step 527, the loss function uses NLL Loss (negative log-likelihood loss) to calculate the gap between the prediction and the target label, specifically:

[0209] The forward function in the Pointnet2SemSegLoss class calculates the negative log-likelihood loss between the predicted point cloud classification result pred (prediction value) and the target label target (target value).

[0210] The shape of the prediction result pred is , where is the batch size, is the number of points, is the number of categories.

[0211] The shape of the target label target is , represents the true category of each point.

[0212] This implementation adds RPE, LGRM, and LRSA modules to the PointNet++ model, which not only retains the original multi-scale feature extraction and segmentation capabilities, but also significantly enhances the adaptive processing capabilities for low-density, irregularly shaped, and noisy points by introducing relative position information and regularized local structures, thereby greatly improving the accuracy and robustness of corn ear three-dimensional point cloud segmentation.

[0213] Step 53: Train the improved PointNet++ model for point cloud segmentation and defect classification. After training, use the improved PointNet++ model to segment the point cloud, extract the corresponding defect areas, evaluate and visualize the segmentation results, and optimize the improved PointNet++ model to improve accuracy and apply it to actual detection tasks. Using the improved PointNet++ model for deep learning on the dataset enables defect classification of corn ears into normal kernels, missing kernels, bald tips, and diseased areas, thereby enabling corn ear defect detection.

[0214] At the same time, the improved PointNet++ model and the PointNet++ model are compared in terms of classification accuracy, precision, and average intersection-over-union (IoU) parameters.

[0215] Table 1

[0216]

[0217] Step 531, detecting the number of rows of corn ears and the number of kernels per row: detecting the number of rows of corn ears and the number of corn kernels per row through image feature extraction and deep learning methods.

[0218] Step 532, ear length and ear width measurement: extract the outline of the corn ear and calculate the longest and maximum width to measure the ear length and ear width.

[0219] Step 533, corn kernel counting: accurately counting each corn kernel on the corn ear using target detection or image segmentation technology.

[0220] Step 534, bald tip detection: determine whether there is a bald tip defect based on the sparseness of corn kernels in the top area of ​​the corn ear.

[0221] Step 535, ear disease detection: Use color space analysis and deep learning models to detect the diseased area on the surface of the corn ear.

[0222] Step 536, missing kernel detection: identifying missing kernels by comparing the distribution of corn kernels on the surface of the corn ear.

[0223] Step 537, color anomaly detection: identifying the abnormal color area on the surface of the corn ear based on the Lab color space.

[0224] Step 6: Conduct experimental tests on the acquired two-dimensional phenotypic parameters of the corn ear, the three-dimensional reconstructed image of the corn ear, and the classification results of the corn ear defects, and perform accuracy and robustness analyses. Based on the two-dimensional phenotypic parameters of the corn ear, the three-dimensional reconstructed image of the corn ear, and the classification results of the corn ear defects, complete the measurement of the corn ear phenotypic parameters, generate a corn ear phenotypic parameter data report, and construct a visual operation platform, including a data acquisition interface and a result display interface.

[0225] Step 61, perform accuracy analysis and robustness analysis on the two-dimensional phenotypic parameters of the corn ear, the three-dimensional reconstructed image of the corn ear, and the corn ear defect classification results to see whether the preset accuracy is achieved. If the preset accuracy is not achieved, it is necessary to improve the corn ear shooting device or the corn ear phenotypic parameter measurement method. If the preset accuracy can be achieved, small-scale parameter measurement can be achieved, and the real data of the manual measurement of the corn ear is compared with the experimental data to calculate the R value of the corn ear length. 2 (coefficient of determination) is 0.97, RMSE (root mean square error) is 2.01 mm, MAE (mean absolute error) is 0.11 mm, and R 2 The average precision is 0.95, the RMSE is 1.02 mm, and the MAE is 0.70 mm.

[0226] Deep learning classification evaluation: Accuracy is used to evaluate overall performance. Prec measures the accuracy of the model when predicting a certain category. mIoU (mean intersection over union) is used to evaluate the quality of the segmentation task.

[0227] Step 62 , generating a structured data report of corn ears, using software such as PyQt5 (Python GUI graphical user interface) to provide visualization of measurement results and statistical analysis;

[0228] Extract structured information from corn ears, including detecting ear row and kernel count, ear length and width, kernel count, and detection of bald tips, missing kernels, and color anomalies. Data visualization is achieved using a PyQt5 user interface, including both the data acquisition and result display interfaces. A structured data report for corn ears is generated, providing intuitive measurement results and statistical analysis.

[0229] This implementation proposes for the first time to insert the LRSA module between the SA second grouping layer and the SA third grouping layer of the PointNet++ model to establish stronger local context modeling capabilities and enhance the ability of point cloud features to express spatial structural differences. The LRSA module improves the recognition accuracy of fine-grained areas, such as rotten or missing corn cob areas, and significantly outperforms existing solutions in multiple segmentation indicators, such as mIoU and Accuracy. While maintaining the efficiency of network training and deployment, this implementation enhances the robustness of the PointNet++ model to structural changes and different corn cob morphologies. An innovative local area modeling scheme combining spatial window division and attention mechanism is proposed to improve the PointNet++ model's ability to perceive detailed structures.

[0230] Embodiment 2. The corn ear shooting device described in this embodiment is constructed according to the corn ear phenotypic parameter measurement method described in embodiment 1, including a host control terminal 1, a rotating turntable 2, an RGB-D camera 4, an adjustment arm 5, a double-layer bracket 6, a fill light 7, a shelf 8, a light shield 10, a motor 11, and a remote control 3;

[0231] The rotating turntable 2 is arranged on the lower layer of the double-layer bracket 6, the adjusting arm 5 is installed above the rotating turntable 2, the RGB-D camera 4 is installed on the adjusting arm 5, a plurality of fill lights 7 are arranged on the inner side of the upper layer of the double-layer bracket 6, the storage rack 8 is arranged in the middle of the rotating turntable 2, and a fixing nail is provided in the center of the storage rack 8, and the corn cob is inserted into the fixing nail. The host control end 1 is connected to the RGB-D camera 4, and the remote control 3 is used to control the motor 11 to adjust the start and stop and speed of the rotating turntable 2. The light shield 10 is arranged on the outside of the double-layer bracket 6.

[0232] like Figure 2 As shown, the functions of the components in the corn ear photographing device are as follows:

[0233] 1. Host Control Terminal 1: Host Control Terminal 1 connects to RGB-D camera 4, controls RGB-D camera 4 to capture data, and performs data collection tasks. Automated control is achieved through computer software, and RGB-D camera 4 capture tasks can be executed through program control or remote commands.

[0234] 2. Rotating turntable 2: Rotating turntable 2 is a rotatable circular turntable that can rotate 360° and provide different shooting angles. It has a diameter of 70 cm. An adjustment arm 5 is installed on rotating turntable 2. Rotating turntable 2 can control RGB-D camera 4 to shoot corn ears from multiple angles.

[0235] 3. Remote control 3: The remote control 3 provides a wireless control function. The remote control 3 can remotely adjust the start, stop and speed of the rotating turntable 2.

[0236] 4. RGB-D camera 4: In this embodiment, the shooting device is selected from Intel Realsense D435i, which has the functions of recording video, shooting color images, shooting depth images and collecting 3D point cloud data. The RGB-D camera 4 is installed on the adjustment arm 5 and can perform multi-angle shooting to improve the accuracy of three-dimensional reconstruction.

[0237] 5. Adjustment arm 5: An adjustment arm 5 is mounted on the rotating turntable 2. The RGB-D camera 4 is placed on the adjustment arm 5. The adjustment arm 5 can adjust the shooting angle and shooting height of the RGB-D camera 4. The RGB-D camera 4 can be controlled to shoot corn ears from multiple angles. The adjustment arm 5 provides flexible positioning of the RGB-D camera 4 to accommodate different types of corn ears, ensuring a variety of shooting angles and improving the integrity of the 3D reconstruction.

[0238] 6. Double-layer bracket 6: Double-layer bracket 6 is used to support the entire device and ensure the stability of rotating turntable 2. The upper bracket is 78 cm from the ground, and the lower bracket is 8 cm from the ground. Double-layer bracket 6 is a rectangular structure with a length of 70 cm, a width of 65 cm, and a height of 70 cm.

[0239] 7. Fill Lights 7: Four fill lights 7 are installed on the upper layer of the double-layer bracket 6 to ensure sufficient and uniform lighting. They use LED (light-emitting diode) light sources with a high CRI (color rendering index) to reduce shadow effects and improve image quality.

[0240] 8. Storage rack 8: It is provided with corn cob fixing nails to facilitate the stable fixation of corn cobs. The corn cobs are inserted into the fixing nails. At the same time, the storage rack 8 is rotatable to meet various shooting needs and reduce image blur caused by movement.

[0241] 9. Pulley 9: The lower layer of the double-layer bracket 6 is provided with four pulleys 9 to facilitate the movement of the corn ear shooting device.

[0242] 10. Lens hood 10: Lens hood 10 is black and can reduce the impact of the surrounding environment and improve shooting consistency.

[0243] 11. Motor 11: can rotate the rotation speed of the turntable 2 to ensure a uniform and stable shooting process.

[0244] Implementation method three. The corn ear shooting method of this embodiment, the method is implemented according to the corn ear shooting device of embodiment two, specifically:

[0245] The motor 11 is controlled by the remote control 3 to rotate the rotating turntable 2 360°. The rotating turntable 2 drives the RGB-D camera 4 to aim at the corn ear inserted on the fixed nail of the rack 8. The host control terminal 1 is used to control the RGB-D camera 4 to shoot a 360° video of the corn ear inserted on the fixed nail of the rack 8. Each second of the 360° video of the corn ear is divided into a photo to complete the shooting of the corn ear.

[0246] Embodiment 4. The corn ear phenotypic parameter measurement system of this embodiment is implemented according to the corn ear phenotypic parameter measurement method of embodiment 1, and includes the following modules:

[0247] Build the module and construct the corn ear shooting device;

[0248] A shooting module, based on a corn ear shooting device, obtains a multi-angle two-dimensional image of the corn ear;

[0249] A processing module processes the multi-angle two-dimensional images of the corn ear to obtain the two-dimensional phenotypic parameters of the corn ear;

[0250] A reconstruction module, which performs three-dimensional reconstruction of the corn ear based on the multi-angle two-dimensional image of the corn ear to obtain a three-dimensional reconstructed image of the corn ear;

[0251] The detection module performs corn ear defect detection on the 3D reconstructed image of the corn ear based on the improved PointNet++ model to obtain the corn ear defect classification results;

[0252] The generation module completes the measurement of the phenotypic parameters of the corn ear based on the two-dimensional phenotypic parameters of the corn ear, the three-dimensional reconstructed image of the corn ear and the classification results of the corn ear defects, and generates a data report on the phenotypic parameters of the corn ear.

[0253] The above is a detailed introduction to the corn ear phenotypic parameter measurement method, system, shooting method and device proposed in the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for measuring corn ear phenotypic parameters, characterized in that: The following steps are involved: Step 1: Build a corn ear shooting device; Step 2, obtaining a multi-angle two-dimensional image of the corn ear based on a corn ear photographing device; Step 3, processing the multi-angle two-dimensional image of the corn ear to obtain the two-dimensional phenotypic parameters of the corn ear; Step 4, performing three-dimensional reconstruction of the corn ear based on the multi-angle two-dimensional image of the corn ear to obtain a three-dimensional reconstructed image of the corn ear; Step 5, performing corn ear defect detection on the 3D reconstructed image of the corn ear based on the improved PointNet++ model to obtain a corn ear defect classification result, including the following steps: Step 51: Build a PointNet++ model and improve the PointNet++ model; Step 52: performing corn ear defect detection on the 3D reconstructed image of the corn ear based on the improved PointNet++ model to obtain a corn ear defect classification result; The improvements to the PointNet++ model are as follows: After the second SA grouping layer of the PointNet++ model, a relative position encoding module, a local grouping rearrangement module, and an LRSA module are sequentially embedded. The relative position encoding module provides the relative position information of the output features of the second SA grouping layer. The local grouping rearrangement module rearranges the relative position information of the output features of the second SA grouping layer. The LRSA module enhances the adaptive features of abnormal areas in the features after the relative position information rearrangement. The local grouping rearrangement module rearranges the remaining point features of the SA second grouping layer into a grid, and performs mirror filling on the points that are less than the total grid capacity to generate a structured grid tensor, specifically: Get the total grid capacity of a structured grid : ； in, is the height of the grid, is the width of the grid; If the total size of the grid Greater than the number of points , then it needs to be filled, the amount of filling is , splice the padded features onto the original feature tensor; Rearrange the padded feature tensor into a shape of The grid is obtained to obtain the features after the relative position information is rearranged. is the batch size, is the number of channels; Step 6: Based on the two-dimensional phenotypic parameters of the corn ear, the three-dimensional reconstructed image of the corn ear, and the classification result of the corn ear defects, the measurement of the phenotypic parameters of the corn ear is completed, and a corn ear phenotypic parameter data report is generated.

2. The method for measuring corn ear phenotypic parameters according to claim 1, wherein The processing of the multi-angle two-dimensional image of the corn ear in step 3 is specifically as follows: The multi-angle two-dimensional image of the corn ear is segmented to obtain the corn ear region image, the corn ear region image is denoised to obtain the contour information of the corn ear region image, and the contour information of the corn ear is feature extracted to obtain the corn ear length and the corn ear width respectively, thereby completing the processing of the multi-angle two-dimensional image of the corn ear.

3. The method for measuring corn ear phenotypic parameters according to claim 1, wherein The three-dimensional reconstruction of the corn ear based on the multi-angle two-dimensional image of the corn ear in step 4 is specifically as follows: Based on SFM, feature extraction and feature matching are performed on multi-angle two-dimensional images of corn ears to generate sparse point clouds. The sparse point clouds are expanded through MVS to obtain dense point clouds. A mesh model is generated based on the dense point cloud, and the mesh model is smoothed. Texture mapping is performed on the smoothed mesh model to complete the three-dimensional reconstruction of the corn ear.

4. The method for measuring corn ear phenotypic parameters according to claim 1, wherein The corn ear phenotypic parameter data report in step 6 includes the detection results of the number of rows and kernels in each row of corn ears, the measurement results of the length and width of corn ears, the corn kernel counting results, the detection results of bald tips of corn ears, the detection results of ear disease of corn ears, the detection results of missing kernels in corn ears and the detection results of abnormal color of corn ears.

5. A system for measuring corn ear phenotypic parameters, the system being implemented according to the method for measuring corn ear phenotypic parameters according to claim 1, characterized in that: Includes the following modules: Build the module and construct the corn ear shooting device; A shooting module, based on a corn ear shooting device, obtains a multi-angle two-dimensional image of the corn ear; A processing module processes the multi-angle two-dimensional images of the corn ear to obtain the two-dimensional phenotypic parameters of the corn ear; A reconstruction module, which performs three-dimensional reconstruction of the corn ear based on the multi-angle two-dimensional image of the corn ear to obtain a three-dimensional reconstructed image of the corn ear; The detection module performs corn ear defect detection on 3D reconstructed images of corn ears based on the improved PointNet++ model to obtain corn ear defect classification results. It includes the following modules: Module 1: Build and improve the PointNet++ model; Module 2: Detect corn ear defects on the 3D reconstructed image of corn ears based on the improved PointNet++ model to obtain corn ear defect classification results; The improvements to the PointNet++ model are as follows: After the second SA grouping layer of the PointNet++ model, a relative position encoding module, a local grouping rearrangement module, and an LRSA module are sequentially embedded. The relative position encoding module provides the relative position information of the output features of the second SA grouping layer. The local grouping rearrangement module rearranges the relative position information of the output features of the second SA grouping layer. The LRSA module enhances the adaptive features of abnormal areas in the features after the relative position information rearrangement. The local grouping rearrangement module rearranges the remaining point features of the SA second grouping layer into a grid, and performs mirror filling on the points that are less than the total grid capacity to generate a structured grid tensor, specifically: Get the total grid capacity of a structured grid : ； in, is the height of the grid, is the width of the grid; If the total size of the grid Greater than the number of points , then it needs to be filled, the amount of filling is , splice the padded features onto the original feature tensor; Rearrange the padded feature tensor into a shape of The grid is obtained to obtain the features after the relative position information is rearranged. is the batch size, is the number of channels; The generation module completes the measurement of the phenotypic parameters of the corn ear based on the two-dimensional phenotypic parameters of the corn ear, the three-dimensional reconstructed image of the corn ear and the classification results of the corn ear defects, and generates a data report on the phenotypic parameters of the corn ear.

Images