Corn ear phenotypic parameter measuring method and system, and shooting method and device

Abstract

The invention discloses a corn ear phenotypic parameter measuring method and system and a shooting method and device, relates to the technical field of crossing of agricultural science and computer vision, and solves the problem that an efficient, accurate and automatic comprehensive corn ear phenotypic parameter measuring method is lacked in the prior art. Building a corn ear shooting device to obtain a corn ear multi-angle two-dimensional image; processing the multi-angle two-dimensional image of the corn ear to obtain two-dimensional phenotypic parameters of the corn ear; performing three-dimensional reconstruction on the corn ear to obtain a corn ear three-dimensional reconstruction image; performing corn ear defect detection on the corn ear three-dimensional reconstruction image based on an improved PointNet + + model to obtain a corn ear defect classification result; completing the measurement of the corn ear phenotypic parameters, and generating a corn ear phenotypic parameter data report. The method is used for realizing efficient, accurate and automatic comprehensive corn ear phenotype parameter measurement.

Description

Technical Field

[0001] The present invention relates to the cross - technical field of agricultural science and technology and computer vision, and particularly relates to a method and system for measuring phenotypic parameters of maize ears, a shooting method and a device. Background Art

[0002] As an important food crop globally, the yield and quality of maize directly affect food security. The maize ear, as a direct manifestation of the yield and quality of maize, its phenotypic parameters are important indicators for maize breeding and cultivation research, such as ear length, ear width, number of rows of kernels, number of kernels per row, etc. However, traditional methods for measuring the phenotypic parameters of maize ears have problems such as large measurement errors, low efficiency, and inability to comprehensively obtain morphological characteristics, such as manual measurement, vernier caliper measurement, etc.

[0003] Traditional manual measurement methods, such as manually measuring the length, diameter, etc. of the ear, are not only inefficient but also easily affected by human errors. This method is difficult to implement in large - scale maize phenotypic analysis and cannot provide accurate spatial geometric data.

[0004] There are many existing 3D reconstruction technologies, including: structured light 3D reconstruction, LiDAR (laser scanning 3D reconstruction), Stereo Vision (stereo vision 3D reconstruction), Photometric Stereo (photometric measurement method), deep - learning - based image 3D reconstruction, etc. However, MVS (multi - view stereo vision) 3D reconstruction generates high - density point clouds by combining multi - perspective images with depth calculation, and has higher accuracy and robustness. Structured light 3D reconstruction projects a known light pattern and uses a camera to obtain reflection information for 3D reconstruction, which has high requirements for the environment; LiDAR uses a laser beam to scan and obtain object surface data, but the equipment cost is high, and the effect is poor when reconstructing details and small objects; Stereo Vision obtains images from different angles through two cameras, but has high requirements for the viewing angle and image quality; Photometric Stereo infers the object surface morphology using images under different lighting conditions, but has a poor effect on objects with complex textures; Deep - learning - based image 3D reconstruction: extracts features from images through a deep - learning model for 3D reconstruction, but requires a large amount of data and training.

[0005] Meanwhile, the existing measurement of maize ear phenotypic parameters also has the following defects: 1. Insufficient automation: Most of the existing methods for extracting phenotypic parameters rely on manual or semi - automated tools, which require manual calibration and adjustment, are cumbersome to operate, and cannot be fully automated. This limits its application in large - scale crop monitoring.

[0006] 2. Incomplete parameter extraction: Existing methods usually only extract some basic phenotypic parameters (such as length, diameter), but cannot comprehensively and accurately extract the multi-dimensional phenotypic characteristics of corn ears.

[0007] 3. Inconsistent data processing: The data formats, algorithm differences, and parameter processing methods among different technical platforms and devices are not unified, resulting in the lack of consistency in the final phenotypic data and affecting the cross-platform analysis and application of the data.

[0008] As an important basis for crop yield and quality evaluation, the accurate identification of the diseased area of corn ears is of great significance in intelligent breeding and quality screening. Most of the existing 3D point cloud segmentation technologies use PointNet (Point Cloud Network) or PointNet++ as the basic model. Although they have certain semantic recognition capabilities, they have limitations in the extraction of complex surface textures and structural details, and it is difficult to accurately capture fine-grained semantic features, affecting the recognition accuracy. The PointNet++ model realizes multi-scale feature extraction and segmentation of dense point clouds through a hierarchical SA (Set Abstraction) and FP (Feature Propagation) architecture. However, in the actual application of corn ear point clouds, the following problems still exist: 1. Difficult aggregation of sparse features in low-density areas: The number of point clouds in the areas with missing grains, spoilage, and bald tips is scarce, and the SA layer does not extract sufficient features from these areas, and it is easy to form "blind spots" after segmentation.

[0009] 2. Weak relationship modeling in morphologically irregular areas: The morphology of diseased areas or special parts is usually very irregular, such as bald tips and spoilage parts, and the structural information in the point cloud is sparse. Traditional aggregation methods are difficult to adaptively capture such complex geometric morphologies.

[0010] 3. Insufficient local-global context fusion: The SA layer can only aggregate features within a preset neighborhood scale, and it is difficult to achieve dynamic interaction between local and global information, resulting in limited performance in identifying similar semantic clusters. The PointNet++ model effectively extracts local features through multi-scale aggregation operations, but has limited ability to fuse global context information between different scales, resulting in similar semantic regions being easily misclassified, such as the healthy part and the spoilage part of corn ears.

[0011] At present, image-based 3D reconstruction technology has achieved certain results in the phenotypic analysis of various plants, but the precise 3D reconstruction of corn ears and its phenotypic parameter measurement methods are still in the research and exploration stage. Most of the existing technologies rely on point cloud data reconstruction or manual calibration, lacking an efficient, accurate, and automated comprehensive method.

[0012] In the prior art, Chinese patent document CN118196282A discloses a "Three-dimensional reconstruction device and method for maize ear based on neural radiance fields". The device includes a host control terminal, a control board, a servo motor, a rotating cloud platform, an angle sensor, a camera, a display module, a fixed chassis, a background board, and a tripod. The method uses the host control terminal and the camera to capture a multi-frame image sequence of a maize ear surrounding 0 to 360°, which is used as the training sample data for the subsequent model; a maize ear data set is made and data preprocessing is carried out to restore the camera pose; a neural radiance field network is used for three-dimensional reconstruction to obtain the maize ear point cloud information; and the maize ear phenotypic parameters are calculated. However, this technical solution relies on point cloud data reconstruction, has a high data dependence, limited generalization ability, and the captured photos are easily affected by the environment, resulting in inaccurate captured data.

[0013] In summary, the prior art lacks an efficient, accurate, and automated comprehensive method for measuring maize ear phenotypic parameters. Summary of the Invention

[0014] The present invention solves the problem that the prior art lacks an efficient, accurate, and automated comprehensive method for measuring maize ear phenotypic parameters.

[0015] The method for measuring maize ear phenotypic parameters of the present invention includes the following steps: Step 1, set up a maize ear photographing device; Step 2, based on the maize ear photographing device, obtain multi-angle two-dimensional images of the maize ear; Step 3, process the multi-angle two-dimensional images of the maize ear to obtain the two-dimensional phenotypic parameters of the maize ear; Step 4, perform three-dimensional reconstruction on the maize ear based on the multi-angle two-dimensional images of the maize ear to obtain the three-dimensional reconstruction image of the maize ear; Step 5, perform maize ear defect detection on the three-dimensional reconstruction image of the maize ear based on the improved PointNet++ model to obtain the maize ear defect classification result; Step 6, based on the two-dimensional phenotypic parameters of the maize ear, the three-dimensional reconstruction image of the maize ear, and the maize ear defect classification result, complete the measurement of the maize ear phenotypic parameters and generate a maize ear phenotypic parameter data report.

[0016] Further, in the embodiment of the present invention, the processing of the multi-angle two-dimensional images of the maize ear in step 3 is specifically as follows: Segment the multi-angle two-dimensional images of the maize ear to obtain the maize ear region image, denoise the maize ear region image to obtain the contour information of the maize ear region image, extract features from the contour information of the maize ear, and respectively obtain the maize ear length and the maize ear width, then the processing of the multi-angle two-dimensional images of the maize ear is completed.

[0017] Further, in the embodiment of the present invention, the three-dimensional reconstruction of the maize ear based on the multi-angle two-dimensional images of the maize ear in step 4 is specifically as follows: Based on SFM, feature extraction and feature matching are respectively performed on the multi-angle two-dimensional images of the maize ear to generate a sparse point cloud. The sparse point cloud is expanded by MVS to obtain a dense point cloud. A network model is generated according to the dense point cloud, the network model is subjected to mesh smoothing, and texture mapping is performed on the network model after mesh smoothing to complete the three-dimensional reconstruction of the maize ear.

[0018] Further, in the embodiment of the present invention, the defect detection of the maize ear on the three-dimensional reconstruction image of the maize ear in step 5 based on the improved PointNet++ model to obtain the maize ear defect classification result includes the following steps: Step 51, construct a PointNet++ model and improve the PointNet++ model; Step 52, based on the improved PointNet++ model, perform defect detection on the three-dimensional reconstruction image of the maize ear to obtain the maize ear defect classification result.

[0019] Further, in the embodiment of the present invention, the improvement of the PointNet++ model in step 51 is specifically as follows: A relative position encoding module, a local grouping rearrangement module, and an LRSA module are sequentially embedded after the SA second grouping layer of the PointNet++ model. The relative position encoding module provides the relative position information of the output features of the SA second grouping layer. The relative position information of the output features of the SA second grouping layer is rearranged by the local grouping rearrangement module, and the adaptive features of the abnormal regions in the features after the rearrangement of the relative position information are enhanced by the LRSA module.

[0020] Further, in the embodiment of the present invention, the maize ear phenotype parameter data report in step 6 includes the detection results of the number of rows and grains per row of the maize ear, the measurement results of the ear length and ear width of the maize ear, the maize grain counting result, the maize ear bald tip detection result, the maize ear ear disease detection result, the maize ear grain deficiency detection result, and the maize ear color abnormality detection result.

[0021] The maize ear photographing device of the present invention is constructed according to the maize ear phenotype parameter measurement method of the present invention. The device includes a host control terminal 1, a rotating turntable 2, an RGB-D camera 4, an adjusting arm 5, a double-layer bracket 6, a supplementary light 7, a placing rack 8, a light-shielding cover 10, a motor 11, and a remote controller 3; The described rotary turntable 2 is arranged on the lower layer of the double-layer bracket 6. The adjusting arm 5 is installed above the rotary turntable 2. The RGB-D camera 4 is installed on the adjusting arm 5. A plurality of supplementary light lamps 7 are arranged on the inner side of the upper layer of the double-layer bracket 6. The storage rack 8 is arranged in the middle of the rotary turntable 2. A fixing nail is provided at the center of the storage rack 8, and the corn ear is inserted over the fixing nail. The host control terminal 1 is connected to the RGB-D camera 4. The remote controller 3 is used to control the motor 11 to adjust the start, stop and rotation speed of the rotary turntable 2. The light shield 10 is arranged outside the double-layer bracket 6.

[0022] The method for photographing corn ears according to the present invention is realized according to the corn ear photographing device described in the present invention, specifically: The motor 11 is controlled by the remote controller 3 to rotate the rotary turntable 2 by 360°. The rotary turntable 2 drives the RGB-D camera 4 to aim at the corn ear inserted over the fixing nail of the storage rack 8. The host control terminal 1 is used to control the RGB-D camera 4 to photograph the 360° video of the corn ear inserted over the fixing nail of the storage rack 8. Each second of the 360° video of the corn ear is sliced into a photo to complete the photographing of the corn ear.

[0023] The corn ear phenotypic parameter measurement system according to the present invention is realized according to the above-mentioned corn ear phenotypic parameter measurement method, and includes the following modules: A building module for building a corn ear photographing device; A photographing module for obtaining multi-angle two-dimensional images of corn ears based on the corn ear photographing device; A processing module for processing the multi-angle two-dimensional images of corn ears to obtain two-dimensional phenotypic parameters of corn ears; A reconstruction module for three-dimensionally reconstructing the corn ear based on the multi-angle two-dimensional images of the corn ear to obtain a three-dimensional reconstruction image of the corn ear; An inspection module for inspecting corn ear defects on the three-dimensional reconstruction image of the corn ear based on the improved PointNet++ model to obtain a corn ear defect classification result; A generation module for completing the measurement of the phenotypic parameters of the corn ear and generating a corn ear phenotypic parameter data report based on the two-dimensional phenotypic parameters of the corn ear, the three-dimensional reconstruction image of the corn ear and the corn ear defect classification result.

[0024] The present invention solves the problem that the prior art lacks an efficient, accurate and automated comprehensive method for measuring corn ear phenotypic parameters. The specific beneficial effects include: 1. The corn ear phenotypic parameter measurement method described in the present invention uses computer vision and 3D reconstruction technology to build a corn ear photographing device to collect multi-angle 2D images of the corn ear, performs 3D reconstruction on the corn ear based on the multi-angle 2D images of the corn ear to obtain a 3D reconstruction image of the corn ear, uses an improved PointNet++ model to perform point cloud segmentation and defect classification, detects defects of the corn ear on the 3D reconstruction image of the corn ear to obtain a corn ear defect classification result, and efficiently, accurately and automatically completes the measurement of the corn ear phenotypic parameters based on the 2D phenotypic parameters of the corn ear, the 3D reconstruction image of the corn ear and the corn ear defect classification result, and generates a corn ear phenotypic parameter data report; 2. In the corn ear phenotypic parameter measurement method described in the present invention, the improved PointNet++ model sequentially embeds a relative position encoding module, a local grouping rearrangement module and an LRSA (local region self-attention) module after the second grouping layer of SA. Through the relative position encoding module, the geometric relationship between points is integrated into the feature expression, enhancing the spatial structure perception ability; through the local grouping rearrangement module, the disordered point features are rearranged into a regular grid, providing a standardized input format for subsequent self-attention calculation. After these two preprocessings effectively "format" the spatial and topological information, and then hand it over to the LRSA module for processing, it can enable the attention layer to more accurately capture the semantic dependencies of local regions, establish a stronger local context modeling ability, enhance the expression ability of point cloud features to spatial structure differences, and improve the accuracy of defect classification; The corn ear phenotypic parameter measurement method described in the present invention can efficiently and accurately reconstruct the 3D model of the corn ear and automatically extract its phenotypic parameters for agricultural research, breeding improvement and precision agriculture. Description of the Drawings

[0025] The above and / or additional aspects and advantages of the present invention will become obvious and easy to understand from the following description of the embodiments in conjunction with the drawings, where: Figure 1 is the flow chart of the corn ear phenotypic parameter measurement method described in Embodiment 1; Figure 2 is the corn ear photographing device diagram described in Embodiment 2; Figure 3 is the original corn ear diagram and Lab color space selection diagram described in Embodiment 1; Figure 4 is the threshold segmentation result diagram of the maximum inter-class variance algorithm described in Embodiment 1; Figure 5 is the 3D reconstruction image of the corn ear described in Embodiment 1; Figure 6 is the improved PointNet++ model described in Embodiment 1. Specific Embodiments

[0026] The following will clearly and completely describe various embodiments of the present invention with reference to the accompanying drawings. The embodiments described by referring to the accompanying drawings are exemplary and are intended to explain the present invention, and should not be construed as a limitation to the present invention.

[0027] Embodiment 1. The method for measuring maize ear phenotypic parameters described in this embodiment includes the following steps: Step 1, set up a maize ear photographing device; Step 2, based on the maize ear photographing device, obtain multi-angle two-dimensional images of the maize ear; Step 3, process the multi-angle two-dimensional images of the maize ear to obtain two-dimensional phenotypic parameters of the maize ear; Step 4, perform three-dimensional reconstruction on the maize ear based on the multi-angle two-dimensional images of the maize ear to obtain a three-dimensional reconstruction image of the maize ear; Step 5, perform maize ear defect detection on the three-dimensional reconstruction image of the maize ear based on the improved PointNet++ model to obtain a maize ear defect classification result; Step 6, based on the two-dimensional phenotypic parameters of the maize ear, the three-dimensional reconstruction image of the maize ear, and the maize ear defect classification result, complete the measurement of the maize ear phenotypic parameters and generate a maize ear phenotypic parameter data report.

[0028] In this embodiment, the processing of the two-dimensional image of the maize ear in Step 3 is specifically as follows: Segment the multi-angle two-dimensional images of the maize ear to obtain a maize ear region image, denoise the maize ear region image to obtain the contour information of the maize ear region image, extract features from the contour information of the maize ear, and respectively obtain the ear length and ear width of the maize ear, then complete the processing of the multi-angle two-dimensional images of the maize ear.

[0029] In this embodiment, the three-dimensional reconstruction of the maize ear based on the multi-angle two-dimensional images of the maize ear in Step 4 is specifically as follows: Based on SFM, respectively extract features and perform feature matching on the multi-angle two-dimensional images of the maize ear to generate a sparse point cloud, expand the sparse point cloud through MVS to obtain a dense point cloud, generate a network model according to the dense point cloud, perform mesh smoothing on the network model, and perform texture mapping on the network model after mesh smoothing to complete the three-dimensional reconstruction of the maize ear.

[0030] In this embodiment, the maize ear defect detection on the three-dimensional reconstruction image of the maize ear based on the improved PointNet++ model in Step 5 to obtain a maize ear defect classification result includes the following steps: Step 51, constructing a PointNet++ model and improving the PointNet++ model; Step 52: Perform corn ear defect detection on the three-dimensional reconstructed image of the corn ear based on the improved PointNet++ model to obtain a corn ear defect classification result.

[0031] In this implementation, the PointNet++ model in step 51 is improved as follows: The relative position encoding module, local grouping rearrangement module and LRSA module are embedded in sequence after the SA second grouping layer of the PointNet++ model. The relative position encoding module is used to provide the relative position information of the output features of the SA second grouping layer. The relative position information of the output features of the SA second grouping layer is rearranged by the local grouping rearrangement module. The LRSA module is used to enhance the adaptive features of the abnormal areas in the features after the relative position information is rearranged.

[0032] In this embodiment, the corn ear phenotypic parameter data report in step 6 includes the detection results of the number of corn ear rows and the number of kernels in each row, the measurement results of the corn ear length and ear width, the corn grain counting results, the corn ear bald tip detection results, the corn ear ear disease detection results, the corn ear missing kernel detection results and the corn ear color abnormality detection results.

[0033] In the prior art, there is a lack of efficient, accurate, and automated comprehensive corn ear phenotypic parameter measurement methods. To solve the above technical problems, this embodiment proposes a corn ear phenotypic parameter measurement method, which uses efficient and accurate three-dimensional reconstruction technology to obtain the geometric morphology of corn ears and perform automated measurement of phenotypic parameters, such as Figure 1 As shown, the specific steps include: Step 1, building a corn ear shooting device for shooting corn ears at multiple angles; A corn ear shooting device was built according to the experimental requirements to ensure the stability of the rotating turntable 2, the RGB-D camera 4 and the fill light 7. The RGB-D camera 4 was calibrated to ensure that the parameters of each shooting were consistent.

[0034] Step 2, using a corn ear shooting device and an RGB-D camera 4 to shoot the corn ear at 360 degrees to obtain a multi-angle two-dimensional image of the corn ear, specifically comprising the following steps: Step 21, multi-angle two-dimensional image acquisition of corn ears, the remote controller 3 controls the motor 11 to drive the rotating turntable 2 to rotate, and the rotating turntable drives the RGB-D camera 4 to aim at a certain angle of the corn ear, and the RGB-D camera 4 can cover all angles of the corn ear; Step 22: Use the host control terminal 1 to control the RGB-D camera 4 to capture a 360° video of the corn ear, and divide each second in the video into a photo, so as to obtain color images of the corn ear at different angles, that is, multi-angle two-dimensional images of the corn ear.

[0035] Step 3: Perform two-dimensional image processing on the collected multi-angle two-dimensional images of the corn ear to obtain two-dimensional phenotypic parameters of the corn ear, specifically including the following steps: Step 31: Segment the multi-angle two-dimensional images of the corn ear captured by the RGB-D camera 4 through the Lab (luminance, red-green, yellow-blue) color model and the maximum inter-class variance method; Different from the RGB (red-green-blue) space, the Lab color space is considered a uniform color space. Inside this space, the changes in different color regions are similar to the perception effect of the human eye vision. Therefore, it is beneficial for the computer to extract the feature information contained in the real multi-angle two-dimensional images of the corn ear.

[0036] The Lab color space is a uniform color space, which divides color information into three components: The L (luminance) channel reflects the luminance information of the image, that is, the change from black to white, and its value range is from 0 to 100.

[0037] The a (red-green) channel reflects the color change from green to red, and its value range is from -128 to 127.

[0038] The b (yellow-blue) channel reflects the color change from blue to yellow, and its value range is from -128 to 127.

[0039] A key advantage of the Lab color space is its perceptual consistency. Due to its characteristics in color and luminance separation, the Lab color space is more suitable for the detailed analysis of colors than the traditional RGB space.

[0040] The conversion formulas between the RGB space and the Lab color space are as follows: ; (1) ; (2) ; (3) ; (4) ; (5) In the formula, are respectively the three components in the XYZ color space, R, G, and B are respectively the three components in the RGB space, is the luminance (brightness), and are the chromaticities (hues), values of the reference white point respectively, which are used for non - linear correction. The Otsu method, also known as the Otsu (between - class variance) algorithm, can divide the multi - angle two - dimensional images of corn ears into two different regions. Then, according to the background and the target, a maximum variance is selected as the segmentation threshold. The larger the variance, the greater the difference between the two parts. By combining a uniform color space and automated threshold selection, the image of the corn ear region can be accurately and efficiently extracted.

[0041] Convert the multi - angle two - dimensional images of corn ears from the RGB color space to the Lab color space. In the Lab color space, the L channel provides brightness information. Using the a and b channels can further separate the color features of the corn ear region image from the color of the background region. As

[0042] shown, the left figure is the original image taken by the camera, and the right figure is the selected image in the Lab color space. When performing corn ear segmentation, the L channel can be used for brightness threshold segmentation to further extract the corn ear region image. By thresholding the L channel using the Otsu method, the corn ear region image and the background region image can be effectively separated from the multi - angle two - dimensional images of corn ears. Especially when the brightness difference between the corn ear region and the background region is obvious, the Otsu method can automatically determine the optimal threshold for segmentation, as Figure 3 shown. In addition, using the color difference information of the a and b channels can further optimize the color, identify and exclude some possible background noises, and ensure that the extracted corn ear region image is more accurate. Combining subsequent median filtering and morphological processing can further remove noises and correct the contour of the corn ear. Figure 4 Step 32: Remove the noises in the corn ear region image through median filtering and morphological processing, and use the internal gradient algorithm to obtain the contour information of the corn ear, and get the number of rows and grains per row of the corn ear;

[0043] In the corn ear region image, median filtering can effectively remove the noise caused by environmental conditions and changes in shooting angles, while avoiding the loss of the corn ear contour information. After extracting the corn ear region image, further extracting the contour information of the corn ear can help us better analyze and measure the characteristics such as the shape and size of the corn ear. The internal gradient algorithm is used to extract the contour information of the corn ear. The internal gradient algorithm is based on the gradient information of the corn ear region image to detect the regions where the pixel values change greatly in the corn ear region image. Usually, these regions correspond to the edges of the object.

[0044] ​Using the contour detection method, count the boundaries of corn kernels on the surface of the corn ear. Identify the circular or elliptical structures of the corn kernels by edge detection, Hough transform or morphological processing. Detect the number of kernel rows arranged on the surface of the corn ear by detecting the circular or elliptical features in the multi-angle two-dimensional images of the corn ear.

[0045] Divide the detected corn kernels into different rows by K-means clustering. Calculate the central positions of different rows to ensure the correct arrangement of the corn kernel rows. Calculate the number of kernel rows by detecting the horizontal distribution density of the corn kernels. : ; (6) The number of corn kernels in each row can be counted by a feature-based segmentation method to obtain the number of kernels per row.

[0046] Step 33, obtain the minimum bounding rectangles of the corn ear and the corn kernels, draw horizontal and vertical lines in the minimum bounding rectangles to obtain the two-dimensional pixel coordinates of the two pairs of intersection points, so as to obtain the ear length and ear width of the corn ear.

[0047] Calculate the minimum bounding rectangle of the corn ear. Draw horizontal and vertical lines in the minimum bounding rectangle to obtain two-dimensional pixel coordinates. Use a reference object with a known length and utilize the principle of pinhole imaging to convert pixels and actual distances, and the ear length and ear width of the corn ear can be calculated.

[0048] Through a reference object with a known length, establish a proportional relationship between pixels and actual physical dimensions: ; (7) ; (8) ; (9) In the formula, is the ear length of the corn ear; is the ear width of the corn ear.

[0049] Step 4, perform 3D reconstruction on the corn ear based on the multi-angle two-dimensional images of the corn ear to obtain the 3D reconstruction image of the corn ear, which specifically includes the following steps: Step 41, perform camera calibration to obtain the internal and external parameters of the RGB-D camera, and extract and match the feature points in the multi-angle two-dimensional images of the corn ear through image processing algorithms; Step 42, use SFM to generate a sparse point cloud, and calculate the initial coordinates of the camera pose and 3D feature points, which specifically includes the following steps: Step 421, Before performing 3D reconstruction, it is necessary to first recover the camera pose and generate a sparse point cloud through SFM. SFM estimates the internal and external parameters of the RGB-D camera by pairwise matching of multiple 2D images of the corn ear from multiple angles and generates a sparse point cloud. This process mainly includes two key steps: feature extraction and feature matching. Distinguishable feature points are extracted from multiple 2D images of the corn ear from multiple angles, such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), or ORB (Object Request Broker), etc. Corresponding feature points are found between the 2D images of the corn ear from multiple angles to establish the correspondence between the 2D images of the corn ear from multiple angles. The position and orientation of the RGB-D camera for each 2D image of the corn ear from multiple angles, that is, the camera pose, are calculated through the matched feature points. Based on the camera pose and the matching of feature points, a sparse point cloud is generated, representing the basic structure of the scene. Two tools, COLMAP (3D reconstruction tool) and OpenSFM (Open Structure from Motion), can be used to perform SFM reconstruction, generating an output containing camera pose and sparse point cloud data; Step 422, Data format conversion, OpenMVS (Multi-View Stereo Reconstruction System) does not directly support the SFM data generated by COLMAP or OpenSFM. Therefore, it is necessary to convert the SFM output result into a format that OpenMVS can use. OpenMVS provides a tool, InterfaceCOLMAP (3D reconstruction tool interface), for converting the cameras.txt (camera parameter file), images.txt (image list file), and points3D.txt (3D point list) files of COLMAP into.mvs (Multi-View Stereo) files,.mvs is the input format of OpenMVS. The converted.mvs file contains RGB-D camera parameters, sparse point cloud, and other necessary metadata.

[0050] Step 43, Perform 3D reconstruction through MVS to generate a depth map and synthesize it into a dense point cloud, further optimize the quality of the point cloud, and finally perform surface reconstruction to obtain the 3D reconstruction image of the corn ear, as Figure 5 shown, specifically including the following steps: Step 431: Dense point cloud reconstruction expands the sparse point cloud through MVS to generate a dense point cloud; through disparity estimation (or depth estimation), a depth map is calculated for each pair of matched multi-angle two-dimensional images of corn ears. The depth value of each pixel reflects the distance between the object surface corresponding to the pixel and the camera. The consistency of the depth map is ensured through multi-view matching to remove incorrect estimations and noise. OpenMVS considers the depth information of the same object from multiple perspectives for optimization. Using the calculated depth map and camera poses, the dense point cloud is gradually restored. The three-dimensional coordinates of each point are back-calculated from the depth map to form the final dense point cloud. In OpenMVS, this step is implemented through the DensifyPointCloud (dense point cloud) tool. It generates an.mvs file containing a large number of three-dimensional points and prepares for subsequent surface reconstruction; Step 432: Convert the dense point cloud into a more concise and structured mesh model through triangular meshing for surface reconstruction; surface reconstruction triangulates between the dense points of the dense point cloud to form a closed geometric surface. Connect the various points in the dense point cloud by constructing the surface. Common surface reconstruction methods include Poisson reconstruction or Delaunay triangulation. Connect adjacent points in the dense point cloud with triangles to generate a mesh model. This process converts the dense point cloud into a triangular mesh (a geometric structure composed of faces, edges, and vertices). In OpenMVS, the ReconstructMesh (reconstruct mesh) tool can complete this process. It generates a triangular mesh based on the dense point cloud and outputs an.mvs file containing the information of the generated mesh model; Step 433: Optimize the mesh model. After surface reconstruction, there may be areas with surface noise or unevenness. At this time, it is necessary to optimize the mesh model to make it smoother and more refined. Remove irregular triangles and noise points and perform mesh smoothing to make the surface of the mesh model smoother. Refine the details of the mesh model to make the surface more realistic and the boundaries clearer. OpenMVS provides the RefineMesh (mesh refinement) tool to further optimize the mesh model, remove cluttered details, and improve the quality of the mesh model, generating a smoother and more refined mesh model; Step 434: Perform texture mapping to map the color information obtained from the multi-angle two-dimensional images of the input corn ear to each triangular face of the mesh model. Based on the camera pose and the mesh model, OpenMVS maps the color information in the multi-angle two-dimensional images of the corn ear to the surface of the mesh model, enabling the surface of the mesh model to present the content details of the multi-angle two-dimensional images of the corn ear. Calculate the texture coordinates to generate texture maps (usually one or more image texture layers). Through the TextureMesh tool, OpenMVS can generate a three-dimensional reconstruction image of the corn ear with texture. The output result is usually an.mvs file and contains texture data; Step 435: The generated three-dimensional reconstruction image of the corn ear can be exported to common three-dimensional file formats such as.ply (point cloud file),.obj (object file), etc., and can be imported into various three-dimensional visualization software for viewing and further processing. Software such as MeshLab (a three-dimensional mesh processing and reconstruction tool) and CloudCompare (point cloud processing software) can be used to view and process the mesh model and the dense point cloud.

[0051] MVS three-dimensional reconstruction can handle complex shapes and is superior to methods such as laser scanning and stereo vision in terms of low cost and high efficiency. At the same time, MVS three-dimensional reconstruction has relatively low requirements for light changes and texture, and is suitable for relatively complex and fine crop phenotype measurements. Therefore, it has obvious advantages in the application of three-dimensional reconstruction of corn ears.

[0052] Step 5: Based on the improved PointNet++ model, perform defect detection on the three-dimensional reconstruction image of the corn ear to obtain the corn ear defect classification result, which specifically includes the following steps: Step 51: Make a dataset. Use CloudCompare to perform data annotation on the dense point cloud of the corn ear, and label normal corn kernels, missing grains, bare tips, and diseased parts separately; Step 52: Construct a PointNet++ model and improve the PointNet++ model; Since the PointNet++ model has the ability to directly process unstructured point cloud data and the characteristics of hierarchical multi-scale feature extraction, the PointNet++ model is used to detect corn ear defects, and can realize the preliminary segmentation of point clouds for features of different spatial scales. However, if the PointNet++ model is directly used for point cloud segmentation, it will lead to insufficient segmentation accuracy and robustness due to low density, irregular shape and noise point interference. In order to solve the above technical problems, this implementation method introduces the LRSA module after the SA second grouping layer of the PointNet++ model, and uses local gridding and self-attention mechanism to realize adaptive feature enhancement and noise self-suppression of abnormal areas, which can realize the classification of normal kernels, missing kernels, bald tips, and diseased parts on corn ears, thereby realizing defect detection of corn ears.

[0053] The LRSA module is a lightweight attention mechanism for local regions. It was originally derived from the modeling needs of two-dimensional structural data such as images. It aims to make up for the shortcomings of convolution operations in local regions, which have limited receptive fields and cannot dynamically model spatial relationships. By introducing a self-attention mechanism within the region, the LRSA module can achieve dynamic information interaction and feature enhancement between pixels or points while maintaining local computational efficiency. Compared with traditional convolution, the LRSA module is not only position-sensitive, but also can adaptively mine significant features and structural relationships within the neighborhood, thereby demonstrating superior performance in tasks such as image semantic segmentation and target detection, and is particularly good at capturing contextual associations between edge regions and small targets.

[0054] However, for point clouds, point cloud data itself is sparse and disordered. If the PointNet++ model and the LRSA module are simply combined, its local structure will become complex and irregular. The traditional LRSA module faces challenges in processing such disordered data because the LRSA module relies on the regularity and positional relationship of the input data, while point cloud data lacks such structure. In order to solve the above technical problems, RPE (Relative Position Encoding) is used to provide clear relative position information for each point, ensuring that the LRSA module can accurately calculate the relationship between points in the disordered point cloud, thereby enhancing its performance in point cloud segmentation tasks.

[0055] Meanwhile, the local structures in point cloud data are often irregular. If only the PointNet++ model, LRSA module, and RPE are combined, it will be difficult for the LRSA module to effectively model the spatial dependency relationships between points. Due to the uneven distribution of neighboring points in the point cloud, when the LRSA directly processes these irregular local structures, it is easy to ignore important local information. To solve this problem, a LGRM (Local Rearrangement Module) is introduced after the RPE. The LGRM converts the local structure of the point cloud into a regular format through rearrangement based on relative position information, enabling the LRSA module to better calculate local dependencies on structured input data, thereby enhancing the model's adaptability and accuracy to local structures.

[0056] By combining the RPE and LGRM, the PointNet++ model can effectively solve the adaptation problem of the LRSA module in point cloud data. The RPE enhances the LRSA module's ability to capture the spatial relationships between points in the point cloud, while the LGRM ensures that the LRSA module can fully utilize the advantages of the self-attention mechanism on regularized local structures. Such a combined improvement makes the LRSA module more efficient in processing sparse, disordered, and irregular point cloud data, significantly enhancing the performance and robustness in the 3D point cloud segmentation task.

[0057] The main parts of the improved PointNet++ model are: SA layer, RPE, LGRM, LRSA module, FP layer, and classification head, as Figure 6 shown. These parts successively constitute the core structure of the improved PointNet++ model. After the second SA grouping layer of the PointNet++ model, a relative position encoding module, a local grouping rearrangement module, and an LRSA module are embedded. The relative position encoding module provides the relative position information of the output features of the second SA grouping layer. The local grouping rearrangement module rearranges the relative position information of the features, and the LRSA module enhances the adaptive features of the abnormal regions in the features after the rearrangement of the relative position information. The specific steps are as follows: Step 521, the SA layer is responsible for extracting local features from the dense point cloud and performing downsampling. Four SA layers are used in the improved PointNet++ model, specifically as follows: SA1 (the first SA grouping layer): The input point cloud has a dimension of 9 (including coordinates and features), and the output point cloud has a dimension of 64. It uses a radius of 0.1 and the number of points in the neighborhood is 32. Feature extraction is performed through three convolutional layers for feature extraction.

[0058] SA2 (the second SA grouping layer): The input point cloud has a dimension of 64, and the output point cloud has a dimension of 128. It uses a radius of 0.2 and the number of points in the neighborhood is 64. Feature extraction is performed through three convolutional layers Feature extraction is performed.

[0059] SA3 (SA third grouping layer): The input point cloud dimension is 128, the output point cloud dimension is 256, using a radius of 0.4 and 128 points in the neighborhood, through three convolutional layers Feature extraction is performed.

[0060] SA4 (SA fourth grouping layer): The input point cloud dimension is 256, the output point cloud dimension is 512, using a radius of 0.8 and 256 points in the neighborhood, through three convolutional layers Feature extraction is performed.

[0061] The role of these SA layers is to extract local features layer by layer from the dense point cloud and gradually reduce the number of point clouds for subsequent processing.

[0062] Step 522: Use RPE to calculate the relative coordinates and Euclidean distances of each point cloud in the output features of the SA second grouping layer relative to the centroid of this layer, and obtain 128-dimensional position encoding features through 1×1 convolution + BatchNorm (batch normalization) + ReLU (rectified linear unit).

[0063] Calculate the centroid of the point cloud after the second layer aggregation: ; (10) In the formula, is the normalization factor, is the weight vector at the th position, is the set of weight vectors. Calculate for each point, and the distance , and splice to get , is the th relative position encoding vector.

[0064] Map through 1*1 convolution: ; (11) In the formula, is the weight matrix of the 1×1 convolutional layer. The 1×1 convolution is equivalent to a pointwise fully connected layer, is the bias vector of the same layer, is the value of the relative position encoding at the th position. Add the RPE feature and the original point feature point by point: ; (12) In the formula, is the initial position encoding, is the updated position encoding.

[0065] Step 523, the LGRM rearranges the remaining point features of the SA second grouping layer (B×128×256) into a 16×16 grid, and mirrors and fills the number of points less than 256 to generate a structured grid tensor (B×128×16×16); Calculate the total network capacity: ; (13) In the formula, is the height of the grid, is the width of the grid.

[0066] Filling: If the total size of the grid is greater than the number of points , filling is required, and the number of fillings is , and these filled features will be concatenated to the original feature tensor.

[0067] Rearrangement: Rearrange the feature tensor into a grid with a shape of , such a grid structure enables local operations (such as convolution or attention mechanism) to be performed within the neighborhood, is the batch size, is the number of channels.

[0068] Step 524, use the LRSA module to enhance the adaptive features in the abnormal regions of the features after rearranging the relative position information: 1. Input the dense point cloud after data annotation into the PointNet++ model, which contains XYZ (spatial position information) and RGB (color information), and use the SA first grouping layer and SA second grouping layer in the PointNet++ model to extract low-level spatial geometric features; 2. After the SA second grouping layer, after REP and LGRM, insert the LRSA module before the SA third grouping layer, and use its lightweight local attention mechanism to perform context modeling on the middle-level features output by the SA second grouping layer to enhance the semantic dependence between different parts; 3. Use the output of the LRSA module as the input of the SA third grouping layer to continue with deep abstraction; 4. Perform feature propagation and upsampling layer by layer through the FP layer, and finally output the semantic label of each point to identify the corrupted, missing grain, bald tip, and normal regions.

[0069] In this embodiment, it is selected to embed the LRSA module between the second grouping layer and the third grouping layer of the PointNet++ model, mainly based on the semantic feature localization and expression characteristics at this stage in the network hierarchy. The features output by the second grouping layer of SA are in the transition stage from local geometry to middle-level semantic abstraction. At this time, the point cloud features still retain relatively rich spatial structures and fine-grained differences, which are suitable for introducing an attention mechanism for context relationship modeling to enhance the semantic dependence and regional consistency between points. At the same time, introducing the LRSA module before further abstraction in the third grouping layer of SA helps to improve the network's recognition ability for local complex shapes, such as ear rot or missing grain regions, and avoid information loss caused by deep feature compression. Compared with inserting after the first grouping layer of SA or after the third grouping layer of SA, fusing the attention module between the second grouping layer and the third grouping layer of SA can maximize the performance advantages of the local attention mechanism on the basis of taking into account spatial structure perception and semantic abstraction depth, thereby effectively improving the expression ability and generalization ability of the model in the point cloud segmentation task.

[0070] Through relative position encoding, the geometric relationships (relative coordinates and distances) between points are incorporated into the feature expression, enhancing the spatial structure perception ability; through LGRM, the unordered point features are rearranged into a regular grid, providing a standardized input format for subsequent self-attention calculation. These two preprocessing steps effectively "format" the spatial and topological information and then hand it over to the LRSA module for processing, enabling the attention layer to more accurately capture the semantic dependence of local regions.

[0071] The core design of the LRSA module is as follows: 1. Divide the input features into several overlapping local patches (regional blocks).

[0072] 2. Apply the multi-head self-attention mechanism within each local patch to model the internal feature relationships of the local region.

[0073] To perform feature interaction within each local region, each patch is first flattened into a sequence: ; (14) In the formula, is the number of patches, is the patch size, is the area of each local region, is the number of channels per pixel.

[0074] Within each patch, the multi-head self-attention mechanism is adopted to capture the long-range dependence relationships between different positions within the local patch.

[0075] The generation process of the query (Q), key (K), and value (V) is as follows: ; (15) ; (16) ; (17) In the formula, is an element in the input sequence, is the query weight matrix, is the key weight matrix, is the value weight matrix, , is the dimension of the key vector.

[0076] The calculation formula of the attention weight is:[[]] ; (18) In the formula, is the transpose of the 𝐾 matrix, which interchanges the rows and columns of 𝐾.

[0077] The result is linearly projected:[[]] ; (19) In the formula, is the weight matrix of the output linear layer, is the output feature after the attention mechanism and linear transformation.

[0078] Finally, combine with the residual connection:[[]] ; (20) In the formula, is the original feature vector of the input, is the final output feature.

[0079] 3. Restore all patches to the shape of the original feature map and perform fusion of the overlapping regions.

[0080] 4. Apply ConvFFN (local convolutional feed-forward network) to further extract local features.

[0081] After the local attention module, introduce ConvFFN to enhance the feature representation.

[0082] The first step is linear transformation and activation:[[]] ; (21) In the formula, is the weight matrix of the first linear transformation, is the feature vector after linear transformation and activation.

[0083] The second step is to apply depthwise separable convolution to further extract local structural information:[[]] ; (22) In the formula, is the feature vector after the depthwise separable convolution operation.

[0084] Step 3, linearly project to the original channel dimension: ; (23) In the formula, is the weight matrix of the second linear transformation, is the feature vector after the linear projection.

[0085] 5. Strengthen the feature representation through residual connection and output the final features.

[0086] Apply the residual connection to obtain the final output: ; (24) In the formula, is the final output feature.

[0087] Step 525, the FP layer is mainly used to gradually restore the resolution of the point cloud and propagate the features from the high layer to the low layer. There are four FP layers in the network, and their functions are as follows: FP4: Fuse the output of the fourth grouping layer of SA with the output of the third grouping layer of SA for feature propagation.

[0088] FP3: Fuse the output of the third grouping layer of SA with the output of the second grouping layer of SA to continue propagating the features.

[0089] FP2: Fuse the output of the second grouping layer of SA2 with the output of the first grouping layer of SA to continue propagating the features.

[0090] FP1: Fuse the output of the first grouping layer of SA with the original input features and finally restore to the original point cloud resolution.

[0091] The FP layer ensures that the spatial structure information of the point cloud can be retained by gradually restoring the resolution and propagating the features to each point.

[0092] Step 526, the last part of the improved PointNet++ model is the classification head, which is responsible for predicting the category of each point according to its features, specifically: conv1 (a 1x1 convolutional layer): Outputs 128-dimensional features.

[0093] bn1 (batch normalization layer): Used to stabilize the training process.

[0094] drop1 (Dropout layer): Used to prevent overfitting.

[0095] conv2 (a 1x1 convolutional layer): Outputs channels of num_classes (number of classes), representing the classification probability of each point.

[0096] softmax (normalized exponential function): Normalizes the class probabilities of each point through log_softmax (logarithmic Softmax function) and outputs the class label of each point.

[0097] Step 527, the loss function uses NLL Loss (negative log-likelihood loss) to calculate the gap between the prediction and the target label, specifically: The forward function in the Pointnet2SemSegLoss (point cloud network semantic segmentation loss function) class calculates the negative log-likelihood loss between the predicted point cloud classification result pred (predicted value) and the target label target (target value).

[0098] The shape of the prediction result pred is , where is the batch size, is the number of points, is the number of classes.

[0099] The shape of the target label target is , representing the true class of each point.

[0100] In this embodiment, the RPE, LGRM, and LRSA modules are newly added to the PointNet++ model, which not only retains the original multi-scale feature extraction and segmentation capabilities, but also significantly enhances the adaptive processing capabilities for low-density, irregularly shaped, and noisy points by introducing relative position information and regularized local structures, thereby greatly improving the accuracy and robustness of the 3D point cloud segmentation of corn ears.

[0101] Step 53, train the improved PointNet++ model for point cloud segmentation and defect classification; after training, use the improved PointNet++ model for point cloud segmentation, extract the corresponding defect regions, evaluate and visualize the segmentation results, optimize the improved PointNet++ model to improve the accuracy and apply it to actual detection tasks. Using the improved PointNet++ model for deep learning on the dataset can achieve defect classification of normal corn kernels, missing grains, bald tips, and diseased parts on corn ears, thereby realizing corn ear defect detection.

[0102] At the same time, compare the classification accuracy, precision, mean intersection over union and other parameter indicators of the improved PointNet++ model and the PointNet++ model; Table 1

[0103] Step 531, Detection of ear row number and kernels per row: Detect the row number of the corn ear and the number of corn kernels in each row through image feature extraction and deep learning methods.

[0104] Step 532, Measurement of ear length and ear width: Extract the contour of the corn ear and calculate the longest and maximum widths to measure the ear length and ear width.

[0105] Step 533, Counting of corn kernels: Use object detection or image segmentation technology to accurately count each corn kernel on the corn ear.

[0106] Step 534, Detection of barren tip: Judge whether there is a barren tip defect by the sparsity of corn kernels in the top area of the corn ear.

[0107] Step 535, Detection of ear diseases: Use color space analysis and deep learning models to detect the diseased areas on the surface of the corn ear.

[0108] Step 536, Detection of missing kernels: Identify the missing kernel phenomenon by comparing the distribution of corn kernels on the surface of the corn ear.

[0109] Step 537, Detection of abnormal color: Identify the areas with abnormal color on the surface of the corn ear based on the Lab color space.

[0110] Step 6: Conduct experimental tests on the obtained two-dimensional phenotypic parameters of the corn ear, three-dimensional reconstructed images of the corn ear, and corn ear defect classification results, perform accuracy analysis and robustness analysis. Based on the two-dimensional phenotypic parameters of the corn ear, three-dimensional reconstructed images of the corn ear, and corn ear defect classification results, complete the measurement of the phenotypic parameters of the corn ear, generate a data report on the phenotypic parameters of the corn ear, and construct a visualization operation platform, including a data acquisition interface and a result display interface at the same time; Step 61, Conduct accuracy analysis and robustness analysis on the two-dimensional phenotypic parameters of the corn ear, three-dimensional reconstructed images of the corn ear, and corn ear defect classification results to see if the preset accuracy is achieved. If the preset accuracy is not achieved, the corn ear shooting device or the method for measuring the phenotypic parameters of the corn ear needs to be improved. If the preset accuracy can be achieved, small-size parameter measurement can be realized. Compare the real data measured manually for the corn ear with the experimental data, and calculate the R 2 (coefficient of determination) is 0.97, RMSE (root mean square error) is 2.01 mm, MAE (mean absolute error) is 0.11 mm, and the R 2 for the ear width of the corn ear is 0.95, RMSE is 1.02 mm, and MAE is 0.70 mm.

[0111] Deep learning classification evaluation: Use Accuracy to evaluate the overall performance, Prec to measure the accuracy of the model when predicting a certain category, and mIoU to evaluate the quality of the segmentation task.

[0112] Step 62, generate a structured data report for corn ears, and use software such as PyQt5 (Python GUI graphical user interface) to provide visual measurement results and statistical analysis; Extraction of structured information of corn ears, including detection of the number of rows and kernels per row, measurement of ear length and width, corn kernel counting, bald tip, missing kernels, and color anomaly detection. Visualize the data, and use PyQt5 to build a user interface, including a data acquisition interface and a result display interface. Generate a structured data report for corn ears, providing intuitive measurement results and statistical analysis.

[0113] In this embodiment, for the first time, an LRSA module is inserted between the SA second grouping layer and the SA third grouping layer of the PointNet++ model to establish a stronger local context modeling ability and enhance the expression ability of point cloud features for spatial structure differences; through the LRSA module, the recognition accuracy of fine-grained regions is improved, such as the spoiled or missing kernel regions of corn ears, which is significantly better than existing solutions in multiple segmentation metrics, such as mIoU and Accuracy; while maintaining the network training and deployment efficiency, this embodiment enhances the robustness of the PointNet++ model to structural changes and different corn ear morphologies; an innovative local region modeling scheme combining spatial window division and attention mechanism is proposed, which improves the perception ability of the PointNet++ model for detailed structures.

[0114] Embodiment 2. The corn ear shooting device described in this embodiment is constructed according to the corn ear phenotypic parameter measurement method described in Embodiment 1, and includes a host control terminal 1, a rotating turntable 2, an RGB-D camera 4, an adjustment arm 5, a double-layer bracket 6, a supplementary light 7, a storage rack 8, a light shield 10, a motor 11, and a remote control 3; The rotating turntable 2 is arranged on the lower layer of the double-layer bracket 6, the adjustment arm 5 is installed above the rotating turntable 2, the RGB-D camera 4 is installed on the adjustment arm 5, multiple supplementary lights 7 are arranged on the inner side of the upper layer of the double-layer bracket 6, the storage rack 8 is arranged in the middle of the rotating turntable 2, a fixing nail is provided at the center of the storage rack 8, the corn ear is inserted into the fixing nail, the host control terminal 1 is connected to the RGB-D camera 4, and the remote control 3 is used to control the start, stop, and rotation speed of the rotating turntable 2 by adjusting the motor 11, and the light shield 10 is arranged outside the double-layer bracket 6.

[0115] As Figure 2 shown, the functions of each component in the corn ear shooting device are as follows: 1. Host control terminal 1: The host control terminal 1 is connected to the RGB-D camera 4 to control the RGB-D camera 4 to perform data shooting and execute the data acquisition task. Through computer software, automatic control is realized. The shooting task of the RGB-D camera 4 can be executed through program control or remote instructions.

[0116] 2. Rotating turntable 2: The rotating turntable 2 is a rotatable circular turntable that can rotate 360°, can provide different shooting angles, has a diameter of 70 cm, and an adjustment arm 5 is installed on the rotating turntable 2. The rotating turntable 2 can control the RGB-D camera 4 to shoot the corn ear from multiple angles.

[0117] 3. Remote controller 3: The remote controller 3 provides a wireless control function. The remote controller 3 can remotely adjust the start / stop and rotation speed of the rotating turntable 2.

[0118] 4. RGB-D camera 4: In this embodiment, the shooting device selects the Intel Realsense D435i, which has the functions of video recording, color image shooting, depth image shooting, and 3D point cloud data acquisition. The RGB-D camera 4 is installed on the adjustment arm 5 and can perform multi-angle shooting to improve the accuracy of 3D reconstruction.

[0119] 5. Adjustment arm 5: An adjustment arm 5 is installed on the rotating turntable 2, and the RGB-D camera 4 is placed on the adjustment arm 5. The adjustment arm 5 can adjust the shooting angle and shooting height of the RGB-D camera 4, can control the RGB-D camera 4 to shoot the corn ear from multiple angles, and the adjustment arm 5 provides a flexible positioning method for the RGB-D camera 4 to adapt to different types of corn ears, ensuring the diversity of shooting angles and improving the integrity of 3D reconstruction.

[0120] 6. Double-layer bracket 6: The double-layer bracket 6 is used to support the overall device and can ensure the stability of the rotating turntable 2. The distance from the upper bracket to the ground is 78 cm, and the distance from the lower bracket to the ground is 8 cm. The double-layer bracket 6 is a rectangular structure with a length of 70 cm, a width of 65 cm, and a height of 70 cm.

[0121] 7. Fill light 7: Four fill lights 7 are set on the upper layer of the double-layer bracket 6, which can ensure sufficient light source and uniform illumination. An LED (light-emitting diode) light source with a CRI (high color rendering index) is used to reduce the influence of shadows and improve the image quality.

[0122] 8. Storage rack 8: Corn ear fixing nails are provided, which are convenient for the stable fixation of corn ears. The corn ears are inserted into the fixing nails. At the same time, the storage rack 8 can rotate to meet various shooting needs and reduce image blurring caused by movement.

[0123] 9. Pulley 9: Four pulleys 9 are set on the lower layer of the double-layer bracket 6, which is convenient for moving the corn ear shooting device.

[0124] 10. Light shield 10: The light shield 10 is black, which can reduce the influence of the surrounding environment and improve the shooting consistency.

[0125] 11. Motor 11: It can rotate the rotation speed of the turntable 2 to ensure a uniform and stable shooting process.

[0126] Embodiment 3. The maize ear shooting method described in this embodiment is implemented according to the maize ear shooting device described in Embodiment 2, specifically as follows: Control the motor 11 through the remote control 3 to make the turntable 2 rotate 360°. The turntable 2 drives the RGB-D camera 4 to aim at the maize ear inserted on the fixing pin of the storage rack 8. Use the host control terminal 1 to control the RGB-D camera 4 to shoot a 360° video of the maize ear inserted on the fixing pin of the storage rack 8, and cut each second of the 360° video of the maize ear into a photo to complete the shooting of the maize ear.

[0127] Embodiment 4. The maize ear phenotypic parameter measurement system described in this embodiment is implemented according to the maize ear phenotypic parameter measurement method described in Embodiment 1, and includes the following modules: Building module, building a maize ear shooting device; Shooting module, obtaining multi-angle two-dimensional images of maize ears based on the maize ear shooting device; Processing module, processing the multi-angle two-dimensional images of maize ears to obtain two-dimensional phenotypic parameters of maize ears; Reconstruction module, performing three-dimensional reconstruction on the maize ear based on the multi-angle two-dimensional images of the maize ear to obtain a three-dimensional reconstruction image of the maize ear; Detection module, performing maize ear defect detection on the three-dimensional reconstruction image of the maize ear based on the improved PointNet++ model to obtain a maize ear defect classification result; Generation module, based on the two-dimensional phenotypic parameters of maize ears, the three-dimensional reconstruction image of maize ears and the maize ear defect classification result, complete the measurement of the phenotypic parameters of maize ears and generate a maize ear phenotypic parameter data report.

[0128] The above has introduced in detail the maize ear phenotypic parameter measurement method, system, shooting method and device proposed by the present invention. Specific examples are used in this article to elaborate on the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. Method for measuring phenotypic parameters of maize ear, characterized in that, It includes the following steps: Step 1, set up a corn ear shooting device; Step 2, based on the corn ear shooting device, obtain multi-angle two-dimensional images of the corn ear; Step 3, process the multi-angle two-dimensional images of the corn ear to obtain two-dimensional phenotypic parameters of the corn ear; Step 4, perform three-dimensional reconstruction on the corn ear based on the multi-angle two-dimensional images of the corn ear to obtain a three-dimensional reconstruction image of the corn ear; Step 5, perform corn ear defect detection on the three-dimensional reconstruction image of the corn ear based on the improved PointNet++ model to obtain a corn ear defect classification result; Step 6, based on the two-dimensional phenotypic parameters of the corn ear, the three-dimensional reconstruction image of the corn ear, and the corn ear defect classification result, complete the measurement of the phenotypic parameters of the corn ear and generate a corn ear phenotypic parameter data report.

2. The method for measuring phenotypic parameters of corn ears according to claim 1, characterized in that, The processing of the multi-angle two-dimensional images of the corn ear in Step 3 is specifically as follows: Segment the multi-angle two-dimensional images of the corn ear to obtain a corn ear region image, denoise the corn ear region image to obtain the contour information of the corn ear region image, extract features from the contour information of the corn ear, and respectively obtain the ear length and ear width of the corn ear, then complete the processing of the multi-angle two-dimensional images of the corn ear.

3. The method for measuring phenotypic parameters of maize ears according to claim 1, characterized in that, The three-dimensional reconstruction of the corn ear based on the multi-angle two-dimensional images of the corn ear in Step 4 is specifically as follows: Based on SFM, extract features and perform feature matching on the multi-angle two-dimensional images of the corn ear respectively to generate a sparse point cloud, expand the sparse point cloud through MVS to obtain a dense point cloud, generate a network model according to the dense point cloud, perform mesh smoothing on the network model, and perform texture mapping on the network model after mesh smoothing to complete the three-dimensional reconstruction of the corn ear.

4. The method for measuring phenotypic parameters of maize ear according to claim 1, characterized in that, The corn ear defect detection on the three-dimensional reconstruction image of the corn ear based on the improved PointNet++ model in Step 5 to obtain a corn ear defect classification result includes the following steps: Step 51, construct a PointNet++ model and improve the PointNet++ model; Step 52, perform corn ear defect detection on the three-dimensional reconstruction image of the corn ear based on the improved PointNet++ model to obtain a corn ear defect classification result.

5. The method for measuring the phenotypic parameters of maize ears according to claim 4, wherein The improvement of the PointNet++ model in Step 51 is specifically as follows: Embed a relative position encoding module, a local grouping rearrangement module, and an LRSA module in sequence after the SA second grouping layer of the PointNet++ model. Use the relative position encoding module to provide the relative position information of the output features of the SA second grouping layer, rearrange the relative position information of the output features of the SA second grouping layer through the local grouping rearrangement module, and use the LRSA module to enhance the adaptive features of the abnormal regions in the features after the rearrangement of the relative position information.

6. The maize ear phenotype parameter measurement method according to claim 1, characterized in that, The phenotypic parameter data report of the corn ear in step 6 includes the detection results of the number of rows and grains per row of the corn ear, the measurement results of the ear length and ear width of the corn ear, the corn grain counting results, the detection results of the barren tip of the corn ear, the detection results of the ear diseases of the corn ear, the detection results of the missing grains of the corn ear, and the detection results of the abnormal ear color of the corn ear.

7. Corn ear photographing device, the device is constructed according to the corn ear phenotypic parameter measurement method described in claim 1, characterized in that, The device includes a host control terminal (1), a rotating turntable (2), an RGB-D camera (4), an adjusting arm (5), a double-layer bracket (6), a supplementary light (7), a storage rack (8), a light shield (10), a motor (11), and a remote control (3); The rotating turntable (2) is arranged on the lower layer of the double-layer bracket (6). The adjusting arm (5) is installed above the rotating turntable (2). The RGB-D camera (4) is installed on the adjusting arm (5). A plurality of supplementary lights (7) are arranged on the inner side of the upper layer of the double-layer bracket (6). The storage rack (8) is arranged in the middle of the rotating turntable (2). A fixing nail is provided at the center of the storage rack (8). The corn ear is inserted into the fixing nail. The host control terminal (1) is connected to the RGB-D camera (4). The remote control (3) is used to control the motor (11) to adjust the start, stop, and rotation speed of the rotating turntable (2). The light shield (10) is arranged outside the double-layer bracket (6).

8. Method for photographing maize ear, the method being implemented according to the maize ear photographing device described in claim 7, characterized in that, Specifically: The motor (11) is controlled by the remote control (3) to rotate the rotating turntable (2) by 360°. The rotating turntable (2) drives the RGB-D camera (4) to aim at the corn ear inserted into the fixing nail of the storage rack (8). The host control terminal (1) is used to control the RGB-D camera (4) to take a 360° video of the corn ear inserted into the fixing nail of the storage rack (8). Each second of the 360° video of the corn ear is cut into a photo to complete the shooting of the corn ear.

9. Maize ear phenotypic parameter measurement system, the system is implemented according to the maize ear phenotypic parameter measurement method described in claim 1, characterized in that, It includes the following modules: A building module for building a corn ear shooting device; A shooting module for obtaining multi-angle two-dimensional images of the corn ear based on the corn ear shooting device; A processing module for processing the multi-angle two-dimensional images of the corn ear to obtain two-dimensional phenotypic parameters of the corn ear; A reconstruction module for three-dimensionally reconstructing the corn ear based on the multi-angle two-dimensional images of the corn ear to obtain a three-dimensional reconstruction image of the corn ear; A detection module for detecting corn ear defects on the three-dimensional reconstruction image of the corn ear based on an improved PointNet++ model to obtain a corn ear defect classification result; A generation module for completing the measurement of the phenotypic parameters of the corn ear based on the two-dimensional phenotypic parameters of the corn ear, the three-dimensional reconstruction image of the corn ear, and the corn ear defect classification result, and generating a phenotypic parameter data report of the corn ear.

Images