Unpowered rolling boll phenotype measurement method, system, and apparatus

Abstract

This application provides a method, system, and apparatus for measuring the phenotype of unpowered rolling ears of grain, belonging to the field of ear phenotype measurement. The method includes: acquiring a rolling video of the ear and extracting multiple video frames; performing region extraction on each video frame to obtain the OBB region, ROI region, and ear center point of the ear in each video frame; extracting the effective grain region in each video frame based on the ROI region, and fusing the effective grain regions from each video frame to obtain an effective grain image of the ear; calculating the boundary trajectory of the ear based on each OBB region and each ear center point, and constructing an effective surface region image of the ear based on the boundary trajectory; constructing multiple sets of point pairs based on the effective grain image and the effective surface region image, and measuring the ear phenotype based on the multiple sets of point pairs. The unpowered rolling ear phenotype measurement method, system, and apparatus provided in this application can improve the efficiency and accuracy of ear phenotype measurement.

Description

Technical Field

[0001] This application belongs to the field of ear phenotyping technology, and more specifically, relates to a method, system and device for measuring ear phenotyping without power. Background Technology

[0002] Maize is one of the world's most important food crops. Phenotypic traits of the maize ear are core indicators for evaluating the yield potential of varieties and selecting superior parents. These traits can include ear length, ear diameter, number of rows per ear, number of kernels per row, total number of kernels, and weight per 100 kernels. Traditional manual seed evaluation methods rely on visual inspection and manual counting, which are inefficient and prone to error, severely hindering the advancement of large-scale breeding programs.

[0003] With the development of computer vision and artificial intelligence technologies, some methods have emerged in recent years for identifying fruit ears using simultaneous multi-camera acquisition. This involves deploying multiple cameras around the ear to simultaneously capture images of the entire ear's surface. However, simultaneous multi-camera acquisition requires multiple cameras to simultaneously capture different sides of the ear, resulting in complex control logic and significant control challenges. Summary of the Invention

[0004] The purpose of this application is to provide a method, system, and apparatus for measuring the phenotypic characteristics of unpowered rolling ears of fruit, which can improve the reliability and accuracy of ear phenotypic measurement by acquiring rolling video of ears of fruit, analyzing and fusing the rolling video, and then measuring the phenotypic characteristics of ears of fruit.

[0005] A first aspect of this application provides a method for measuring the phenotypic characteristics of unpowered rolling ears of fruit, comprising: Obtain a rolling video of the ear of fruit and extract multiple video frames from the rolling video, including at least one video frame of the ear of fruit rolling around once. Region extraction is performed on each video frame to obtain the OBB region, ROI region, and center point of the ear of fruit in each video frame; Based on the ROI region in each video frame, the effective grain region in each video frame is extracted, and the effective grain regions of each video frame are image fused to obtain the effective grain image of the ear. The boundary trajectory of the ear is calculated based on each OBB region and the center point of each ear, and an effective surface area image of the ear is constructed based on the boundary trajectory; the boundary trajectory includes the center rolling trajectory of the ear, the tip rolling trajectory of the ear, and the base rolling trajectory of the ear. Multiple point pairs were constructed based on effective grain images and effective surface area images, and the phenotype of the ear was measured based on the multiple point pairs. Each point pair included a rolling start point and a rolling end point of the ear.

[0006] A second aspect of this application provides a non-powered rolling ear phenotyping device, comprising: The data acquisition module is used to acquire the rolling video of the ear of fruit and extract multiple video frames from the rolling video. The multiple video frames include at least one video frame of the ear of fruit rolling around once. The first processing module is used to extract regions from each video frame to obtain the OBB region, ROI region and center point of the ear of fruit in each video frame. The second processing module is used to extract the effective grain region in each video frame based on the ROI region in each video frame, and to perform image fusion of the effective grain regions in each video frame to obtain the effective grain image of the ear of grain. The third processing module is used to calculate the boundary trajectory of the ear based on each OBB region and the center point of each ear, and to construct an effective surface area image of the ear based on the boundary trajectory; the boundary trajectory includes the center rolling trajectory of the ear, the tip rolling trajectory of the ear, and the base rolling trajectory of the ear. The fourth processing module is used to construct multiple sets of point pairs based on the effective grain image and the effective surface area image, and to measure the phenotype of the ear based on the multiple sets of point pairs. Each set of point pairs includes a rolling start point and a rolling end point of the ear.

[0007] A third aspect of this application provides a non-powered rolling ear phenotypic measurement system, characterized in that it includes: A support platform is used to support and rotate the ear of fruit; the support platform is continuously adjustable within a preset angle range. Image acquisition equipment, used to capture video of the ears of fruit rolling on the support platform; And, as mentioned in the second aspect above, a non-powered rolling ear phenotyping device.

[0008] A fourth aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for measuring the phenotypic characteristics of unpowered rolling ears of fruit.

[0009] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for measuring the phenotypic characteristics of unpowered rolling ears of fruit.

[0010] The beneficial effects of the non-powered rolling ear phenotyping measurement method, system, and apparatus provided in this application are as follows: First, the rolling video full-cycle sampling in this embodiment ensures 360° data acquisition of the ear without blind spots, avoiding local perspective deviations, and eliminating the need for multiple cameras to work simultaneously and complex control logic. Second, precise segmentation of the OBB and ROI regions effectively eliminates interfering information such as pedicels and leaves, significantly improving the accuracy of grain identification. Finally, automated point-to-point matching is much more efficient than traditional manual measurement, while eliminating human error and improving the reliability and accuracy of ear phenotypic measurement. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the structure of a non-powered rolling ear phenotyping system provided in an embodiment of this application; Figure 2 A flowchart illustrating a method for measuring the phenotypic characteristics of a non-powered rolling ear of fruit provided in an embodiment of this application; Figure 3 This is a structural block diagram of a non-powered rolling ear phenotyping device provided in an embodiment of this application; Figure 4 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0014] With the development of computer vision and artificial intelligence technologies, the measurement of ear phenotypes is becoming increasingly intelligent. Generally, it involves two stages: ear image acquisition and ear image processing. By measuring and analyzing the ear phenotype from the ear images, relevant indicators of the ear phenotype are obtained.

[0015] Optionally, in the ear image acquisition stage, some methods employ a roller drive method, using circulating rolling rollers to drive the ear to roll in the opposite direction, while a fixed camera captures images of the rolling ear at different positions to obtain ear images. Other methods use a mechanical rotary table method, using a stepper motor to drive the ear to rotate, while a camera captures images from different angles to obtain ear images. Still others employ a multi-camera synchronous acquisition method, deploying multiple cameras around the ear to simultaneously capture images.

[0016] However, the idler roller drive method may have the following problems: (1) It requires the installation of an idler roller chain device and the use of a stepper motor to drive the idler roller movement. (2) The equipment has a complex structure and is heavy, making it difficult to use in the field. (3) It is costly: the motor, controller and precision mechanical structure require power supply, resulting in high equipment cost.

[0017] The mechanical converter table method may have the following problems: (1) mechanical damage, the motor clamp causes mechanical damage to the ear of fruit. (2) high cost, the motor, controller and precision mechanical structure lead to high equipment cost.

[0018] The multi-camera synchronous acquisition method may have the following problems: (1) The system is complex: the joint calibration of multiple cameras is complicated and the equipment cost is high. (2) A single camera can only shoot a single side of the ear of fruit, and multiple cameras are needed to shoot different sides of the ear of fruit simultaneously, which is difficult to control.

[0019] Optionally, in the ear image processing stage, some methods employ 3D reconstruction, capturing fixed ear images from multiple perspectives and reconstructing a 3D model of the corn ear using Structure from Motion (SfM) or 3D Gaussian Splatting (3DGS). Other methods use single-frame image deep learning, acquiring single-frame images of the ear and analyzing them based on object detection algorithms such as YOLO.

[0020] However, 3D reconstruction may have the following problems: (1) Large computational load: algorithms such as SFM / 3DGS have huge computational loads and long processing time per ear, making it difficult to meet high-throughput requirements. (2) Unstable reconstruction quality: it is sensitive to ear surface texture and lighting conditions, and the texture of the reconstructed 3D model is deformed, making it unsuitable for direct use in image detection and analysis. (3) Based on the reconstructed ear 3D model, a secondary algorithm is developed to calculate the kernel phenotype on the ear surface, which is difficult to implement.

[0021] The single-frame image deep learning method may have the following problems: incomplete information, a single frame image can only cover about 40%-50% of the surface of the ear, there is a serious distortion problem at the edge of the ear, and it is impossible to accurately estimate global parameters such as the number of rows in the ear.

[0022] In summary, the existing technologies have the following problems: (1) Damage and cost contradiction: existing methods either pose a risk of mechanical damage to breeding materials or have complex and costly mechanical and imaging systems. (2) Algorithm limitations: there is a lack of an algorithm that can efficiently and robustly fuse multi-view, locally visible information into full-surface information of the ear to solve the problem of measuring the full-surface phenotypic characteristics of the ear. (3) Poor environmental adaptability: existing equipment is mostly fixed, high-precision laboratory equipment, which is difficult to adapt to the portable and rapid seed testing needs in complex and unstructured environments such as field fields.

[0023] To address the above-mentioned technical problems, this application provides a method and system for measuring the phenotypic characteristics of unpowered rolling ears of fruit, thereby overcoming the problems existing in the prior art.

[0024] The unpowered rolling ear phenotyping system provided in this application embodiment may include a support platform, an image acquisition device, and an unpowered rolling ear phenotyping apparatus. The support platform is used to support and roll the ear; the support platform is continuously adjustable within a preset angle range. The image acquisition device is used to acquire video of the ear rolling on the support platform. The unpowered rolling ear phenotyping apparatus can perform the measurement method provided in this application embodiment.

[0025] Specifically, Figure 1 This is a schematic diagram of the structure of a non-powered rolling ear phenotyping system provided in an embodiment of this application, as shown below. Figure 1 As shown in the embodiments of this application, the support platform can be a multi-functional tilting support platform, i.e., a tilting plate. The image acquisition device can be an industrial camera. The measuring device can be a computer.

[0026] In addition, the measurement system may also include: a collection plate for collecting fallen ears of fruit; an electronic balance for weighing the ears; a limiting guide rail for limiting the rolling direction of the ears and preventing them from deviating and rolling off the support platform; an illumination device for providing lighting; a synchronous tilting linkage mechanism for adjusting the platform's tilt angle so that the camera and light source tilt synchronously, maintaining the optical axis vertical and the illumination angle constant; and a portable integrated interface for supporting quick assembly and disassembly, facilitating portability and deployment in field environments.

[0027] In the embodiments of this application, the tilt angle of the support plate is continuously adjustable from 0° to 45°, with a locking accuracy of ±0.5°. It mainly provides basic support for the rolling of the ear of fruit, and the working mode can be switched by adjusting the angle.

[0028] The guide rail spacing of the limiting guide rail is adjustable (determined according to the maximum diameter of the ear of fruit, such as 30cm), with an adjustment accuracy of ≥1mm.

[0029] The inclined plate has a matte collection surface, which is a low-reflectivity collection surface. The diffuse reflectivity of this collection surface is ≤5%, and the surface roughness Ra≥0.4μm, which can eliminate specular reflection interference and improve the grain splitting accuracy of the ear.

[0030] Industrial cameras with a frame rate of ≥60fps (100fps recommended) and a resolution of ≥1920×1080 can capture high frame rate videos of rolling ears of fruit.

[0031] The illumination device is a dual-sided LED strip light source, which forms a 45° angle with the normal of the collection surface. The illuminance on one side is ≥2000 Lux, which is used to provide uniform, shadowless illumination and enhance the contrast of the grain outline.

[0032] The electronic balance has a weighing range of 0~5kg, an accuracy of 0.1g, and a sampling frequency of ≥10Hz. It is used to weigh the ears of fruit in real time, and the data is automatically associated with the sample ID. The synchronous tilting linkage mechanism uses gear / linkage transmission with a transmission ratio of 1:1 and a synchronization accuracy of ±1°. When adjusting the platform tilt angle, the camera and light source tilt synchronously to keep the optical axis vertical and the illumination angle constant.

[0033] The portable integrated interface is a standardized mechanical and electrical interface that supports quick assembly and disassembly, making it easy to carry and deploy in field environments.

[0034] The electronic balance, illumination device, and synchronous tilting linkage mechanism are optional.

[0035] For example, this application provides a non-powered rolling ear phenotyping system, comprising: The multi-functional tilting support platform has an adjustable and lockable tilt angle within the range of 0° to 45°. It is used to support the ears of fruit and allow them to move naturally under the influence of gravity.

[0036] Limiting guide rails are installed on both sides of the inclined support platform to prevent the ears of fruit from rolling beyond the boundary.

[0037] A matte surface is applied to the inclined support platform to eliminate specular reflections.

[0038] The image acquisition device includes an industrial camera with a frame rate of not less than 60fps, used to capture video of the rolling ears of fruit.

[0039] The electronic balance weighing module is used to weigh the ears of fruit on the platform in real time.

[0040] The illumination device includes an array of light-emitting diodes (LEDs) at a 45° angle to the normal of the acquisition surface, for providing uniform illumination.

[0041] The synchronous tilt linkage mechanism is used to mount the image acquisition device and the lighting device. When the platform tilt angle is adjusted, the image acquisition device and the lighting device tilt synchronously at the same angle to keep the camera optical axis perpendicular to the acquisition platform and the angle between the light source and the normal of the platform unchanged.

[0042] Computers are used to measure ear phenotypic characteristics based on collected data.

[0043] In the embodiments of this application, the unpowered rolling ear phenotyping system may include two operating modes: First working mode: Harvesting by rolling ears. The platform is tilted at 12°~18°, and the ears roll naturally under the influence of gravity. The camera continuously captures images at a frame rate of ≥60fps, covering the entire 360° surface. This is the optimal working mode.

[0044] Second working mode: zero-degree ear seed testing, with the platform tilted at 0° and placed horizontally, used for static image acquisition and weighing of the ears.

[0045] The system provided in this application can eliminate mechanical damage to breeding materials caused by motor clamping and reduce system costs. It utilizes gravity to drive the ears of fruit to roll naturally on an inclined platform, eliminating the need for any motors or mechanical clamps, thus achieving zero damage and low cost.

[0046] The measurement system of this application adopts a lightweight and modular design. It only needs to integrate a tilting platform and a camera to complete the video acquisition of the ear of fruit rolling. It can also be equipped with a light source and a weighing module to expand the system's functions and performance. The device has a simple structure, is easy to carry and deploy quickly.

[0047] In addition, this application also provides a method for measuring the phenotypic characteristics of unpowered rolling ears of fruit. By processing the acquired ear images, accurate ear phenotypic measurement can be achieved without the need for simultaneous acquisition by multiple cameras. Simultaneously, it can also solve the problems of incomplete information and large edge distortion in single-frame images, enabling full-surface data acquisition and phenotypic extraction of the ear.

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0049] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a method for measuring the phenotypic characteristics of a non-powered rolling ear of fruit according to an embodiment of this application. The method may include steps S101 to S105.

[0050] S101, acquire the rolling video of the ear of fruit, and extract multiple video frames from the rolling video, the multiple video frames including at least one video frame of the ear of fruit rolling around once.

[0051] This application embodiment can acquire video of the ear of fruit rolling on a support platform using an industrial camera, and extract the rolling video frame by frame to obtain multiple video frames of the ear of fruit during the rolling process. Each video frame can represent a different rolling pattern of the ear of fruit.

[0052] Specifically, the ear of fruit to be tested can be placed on an inclined support platform with an inclination angle of 12° to 18°, and an industrial camera can be started to collect high frame rate video of the ear of fruit rolling naturally for at least one cycle under the action of gravity, thus obtaining the rolling video.

[0053] In the embodiments of this application, the frame extraction rate represents the number of frames extracted from the rolling video of the ear of fruit. Each frame of image requires a series of subsequent image analyses, thus determining the overall computing performance.

[0054] The more frames extracted, the greater the overlap between adjacent frames, resulting in lower computational efficiency but higher accuracy; conversely, the fewer frames extracted, the lower the overlap between adjacent frames, leading to higher computational efficiency but lower accuracy. At least four frames need to be extracted to ensure basic coverage of the entire ear surface information (grain distortion still exists). Therefore, in this embodiment, 10-20 frames can be extracted to ensure the accuracy of ear surface information calculation. These 10-20 video frames include the state of the ear naturally rolling at least one revolution under gravity.

[0055] S102, perform region extraction on each video frame to obtain the OBB region, ROI region and center point of the ear in each video frame.

[0056] After obtaining multiple video frames, this embodiment of the application performs ear instance segmentation, directional bounding box detection, and ear-standardized region of interest extraction on each video frame.

[0057] Specifically, for each frame of the video, a deep learning-based instance segmentation model (such as the YOLO series) can be called to extract the ear mask and calculate the minimum circumscribed bounding box (OBB) region.

[0058] Furthermore, in this embodiment, the ear of fruit in each video frame can be rotated so that its axis is parallel to the Y-axis of the image and centered through affine transformation. Then, the square Region of Interest (ROI) of the ear is extracted based on its length.

[0059] In one embodiment of this application, the instance segmentation model is obtained by annotating at least 200 ear instance segmentation datasets and then training them using the YOLO-SEG model.

[0060] Harvest-normalized ROI images can be used to build grain detection models: these images are continuously collected as training data during use to improve model inference accuracy and adapt to different imaging backgrounds and harvester types. Based on harvester normalization (harvest rotation until the ear axis is parallel to the image's Y-axis and centered), grain bounding boxes are labeled to generate a grain detection dataset, which helps improve grain segmentation accuracy and calculate grain length and width.

[0061] S103 extracts the effective grain region in each video frame based on the ROI region in each video frame, and performs image fusion on the effective grain regions in each video frame to obtain the effective grain image of the ear.

[0062] After extracting the Region of Interest (ROI) for each video frame, a grain detection model can be used to identify the ROI and obtain the grain region for each video frame. Then, radial distortion correction can be performed on the grains in the grain region to obtain the effective grain region for that video frame. Finally, the effective grain regions from each video frame can be fused to obtain the effective grain image of the ear of grain.

[0063] S104: Calculate the boundary trajectory of the ear based on each OBB region and the center point of each ear, and construct an effective surface area image of the ear based on the boundary trajectory. The boundary trajectory includes the center rolling trajectory of the ear, the tip rolling trajectory of the ear, and the base rolling trajectory of the ear.

[0064] This embodiment of the application, after obtaining multiple video frames, can calculate the ear diameter and circumference. Simultaneously, it can locate the ear center point in multiple video frames, thereby determining the ear's central axis. Subsequently, this embodiment of the application can construct the ear center point's central rolling trajectory based on the ear center point in each video frame. Then, based on the intersection of the central axis and the OBB region, it determines the ear tip rolling trajectory and the ear base rolling trajectory. The central rolling trajectory, the ear tip rolling trajectory, and the ear base rolling trajectory together constitute the ear's boundary trajectory.

[0065] After obtaining the boundary trajectory, an effective label area image of the ear can be constructed based on the boundary trajectory, that is, a grain imprint map of the entire surface of the ear is formed.

[0066] S105, construct multiple sets of point pairs based on effective grain images and effective surface area images, and measure the phenotype of the ear based on multiple sets of point pairs, wherein each set of point pairs includes a rolling start point and a rolling end point of the ear.

[0067] This application embodiment can construct multiple sets of point pairs based on effective grain images and effective surface area images. Subsequently, these can be combined with the grain point set generated by the ear rolling around once to construct a topological graph that can indicate the grain connection relationship. Finally, by solving the topological graph, the number of rows and grains in the ear can be obtained, thereby enabling the measurement of the ear phenotype.

[0068] The point pairs for calculating the number of rows in the ear are located on both sides of the ear's rolling direction; the point pairs for calculating the number of seeds per row are located on both sides of the tip and base of the ear.

[0069] In embodiments of this application, the phenotype of the ear of fruit is measured based on multiple sets of point pairs, including: Construct a Delaunay triangulation with multiple pairs of points.

[0070] Calculate the shortest path for each pair of points in the Delaunay triangulation to obtain multiple shortest paths.

[0071] The number of rows and the number of kernels in each row of the ear are determined by using multiple shortest paths, thus obtaining the phenotypic data of the ear.

[0072] Specifically, in this embodiment, for each pair of points (one start point and one end point), the shortest path through the triangular grid from the start point to the end point can be calculated using Dijkstra's shortest path algorithm. The number of nodes on each path is counted as the number of rows in the ear. The average number of rows in the ear is obtained by calculating the results of multiple paths. Then, an even number constraint is applied to this number of rows in the ear so that the count of rows in the ear meets the breeding agronomic requirements, and finally, the number of rows in the ear is obtained. The embodiments of this application utilize the same approach to determine the number of kernels per ear row.

[0073] In this embodiment of the application, during the frame-by-frame calculation of the ear video frames, phenotypic parameters such as ear length, ear diameter, total number of seeds, and number of rows in each frame can be calculated. These parameters are updated and saved. Furthermore, after completing the video calculation and analysis, an overall analysis is performed, and through outlier cleanup and other processing, more accurate phenotypic analysis results are obtained. The output is in CSV, JSON, and XLS formats, ensuring traceability throughout the entire process.

[0074] In this embodiment, data acquisition utilizes a tilting platform combined with an industrial camera to capture rolling ear videos, extracting 10-20 frames to balance computational efficiency and accuracy while ensuring comprehensive coverage of ear surface information. During image processing, a deep learning model is used to extract the ear's OBB, ROI regions, and center point. Standardized ROI images can be used for model training, improving kernel segmentation accuracy and adapting to different imaging backgrounds and ear types. In image fusion and trajectory construction, multiple frames of effective kernel regions are fused, and boundary trajectories are combined to form a full-surface kernel imprint map, providing comprehensive data for phenotypic measurement. During phenotypic measurement, point pairs and a topology map are constructed, and the number of rows and kernels per row are calculated using a shortest path algorithm. Multiple phenotypic parameters can be calculated for each frame, and after outlier cleanup, multi-format files are output, ensuring accurate results and full traceability, providing precise and efficient technical support for maize breeding and other related work.

[0075] In some embodiments of this application, extracting effective seed regions from each video frame based on the Region of Interest (ROI) in each video frame includes: extracting seeds from each ROI based on a pre-determined target detection model to obtain seed regions in each video frame; and performing distortion correction on each seed region to obtain effective seed regions in each video frame.

[0076] In the embodiments of this application, distortion correction is performed on each grain region to obtain the effective grain region in each video frame, including: selecting the region in each grain region where the distance to the central axis of the ear of grain is less than a preset distance as the effective grain region, thereby obtaining the effective grain region in each video frame.

[0077] Specifically, the application embodiment can use target detection models such as YOLO-DET to detect kernels in the ROI region of the ear of fruit, and calculate the length and width of the kernels on the side of the ear of fruit by statistically analyzing the length and width of the detection box.

[0078] Since the distance from the central axis of the ear determines the degree of radial distortion of the kernels (the farther the kernels are from the central axis, the more severe the distortion, which is reflected in the smaller the kernel width; for example, the width of kernels at the edge of the ear is already severely distorted), the radial distortion of the detected kernels is corrected based on the direction of the ear axis (central axis). (Assuming the ear is a cylinder with a variable cross-section, the position and width of the kernels on its surface are adjusted according to the distance between the kernels and the central axis.)

[0079] The effective grain area is determined by setting the radial bandwidth parameter of the ear axis (band_scale, which is a multiple of the average grain width). That is, the range from the center axis of the ear that is less than band_scale * average grain width is considered the effective grain area.

[0080] The direct result is that in each frame of the image, only the kernels that are close to the central axis of the ear (with less distortion) are retained on the surface of the ear.

[0081] The `band_scale` parameter, which relates to the number of frames used for the ear to roll 360 degrees, determines the overlap between the effective kernels retained in adjacent frames. Grid search optimization ensures sufficient overlap of effective kernels in adjacent frames to prevent kernel omission.

[0082] The band_scale parameter has a negative correlation with the number of frames used for the ear to roll 360 degrees: the more frames there are, the smaller the change in ear rolling angle between adjacent frames, and the larger the overlapping area of ​​the image. Therefore, the band_scale parameter can be set smaller to ensure the accuracy of the effective grain area. Conversely, the fewer frames there are, the larger the change in rolling angle between adjacent frames, and the smaller the overlapping area. In order to ensure that the entire surface of the ear is completely covered and to prevent grains from being missed, the band_scale parameter needs to be set larger to retain a wider effective area.

[0083] Example explanation: Scenario 1 (High frame rate, e.g., 20 frames per second): Twenty video frames were captured after the ear of grain rolled one revolution. The rotation angle between adjacent frames was 18 degrees (360 / 20), resulting in a very high degree of overlap. In this case, the `band_scale` parameter can be set to a small value, such as 1.0. This means that only grains whose distance from the central axis is less than one times the average grain width are retained. Due to the large number of frames and the high overlap, even if only a small number of grains at the very center are retained in each frame, subsequent image fusion can still stitch together an image of all grains on the entire ear surface without omission. This processing can minimize the inclusion of grains with severe edge distortion in the analysis, improving data accuracy.

[0084] Scenario 2 (low frame rate, e.g., 6 frames / week): Only 6 video frames were captured after the ear of grain rolled one revolution. The rotation angle between adjacent frames was 60 degrees (360 / 6), resulting in low overlap. Therefore, to ensure sufficient coverage of the ear's surface in each frame and avoid uncaptured "blank areas," the `band_scale` parameter needs to be set larger, for example, to 2.5. This means retaining grains whose distance from the central axis is less than 2.5 times the average grain width. While this introduces some edge grains with slightly larger distortion, it ensures sufficient overlap between frames, allowing subsequent fusion steps to successfully stitch all frames into a complete and continuous image of the ear's surface, thus preventing the omission of grain information.

[0085] This application embodiment can use a seed detection model to obtain seeds in each frame, perform radial distortion correction on the seeds, and obtain an effective seed region. The coordinates of the seeds within the effective region detected in each frame are then transformed to the same image coordinate system.

[0086] Then, in this embodiment, the effective seed information (center point / four corner points of the detection box) extracted from the ROI region can be mapped back to the image space captured by the camera frame by frame, thus imprinting the effective seeds into the image space. In this image space, the effective seeds in subsequent new frames need to be fused and deduplicated with the historical seeds that have been "imprinted" into the image space of previous frames (determined based on the cross-union ratio between seeds and the confidence level of the seeds).

[0087] This application employs a deduplication strategy based on Intersection over Union (IoU) and confidence level. The specific judgment logic is as follows: Coordinate space alignment: First, the valid seed information (center point or detection box corner point) detected in the current new frame is mapped back to the global "imprint" image coordinate system.

[0088] Candidate Matching Filtering: For each valid seed in the new frame, in the set of historical seeds that have been “imprinted”, find all historical seeds whose intersection-over-union (IoU) with their detection boxes is greater than a preset threshold T_iou (e.g., T_iou = 0.5) and use them as candidate matching objects for the new seed.

[0089] Deduplication and update rules: Case A: A match exists (IoU > T_iou) Compare the confidence level (conf_new) of the new frame seed with the confidence level (conf_hist) of the candidate historical seed.

[0090] If conf_new > conf_hist: If the new frame is considered to provide better observation results, then the historical seed information is replaced, that is, the historical seed is updated with the detection box coordinates, confidence score and other data of the new seed, and the historical seed is marked as "refreshed".

[0091] If conf_new <= conf_hist: if the information of historical grains is considered more reliable, then the historical grain information is retained and the current grain data of the new frame is directly discarded.

[0092] Special case (many-to-one matching): If the IoU of a new frame seed is greater than the threshold with multiple historical seeds, then only the historical seed with the highest IoU will be matched with the above logic, and the rest will be considered as no match.

[0093] Case B: No match (IoU <= T_iou) The seed in the current new frame is considered to represent a new, unrecorded seed position. Therefore, it is added as a new seed node to the historical seed set in the global "imprint" image space.

[0094] Through the above logic, the system can dynamically and intelligently fuse local observation results from multiple frames, realize the deduplication and updating of grains, and finally construct a complete, continuous and non-redundant full-surface effective grain image of the ear, laying a solid foundation for subsequent accurate phenotypic measurements.

[0095] This application's embodiments accurately extract the kernel region of the ear using a target detection model and introduce a radial distortion correction mechanism, effectively solving the morphological distortion problem of kernels at the ear's edge caused by the shooting angle. By setting the radial bandwidth parameter of the ear axis to filter the effective kernel region, it not only retains the core kernel data with less distortion but also avoids missing kernel information by optimizing the overlap between adjacent frames. After coordinate transformation and multi-frame fusion deduplication, a complete and high-precision three-dimensional dataset of ear kernels can be constructed, providing key technical support for the automation of maize seed evaluation. It can significantly improve the accuracy and efficiency of measuring traits such as kernel count and kernel weight, and reduce labor costs.

[0096] In some embodiments of this application, the boundary trajectory of the ear is calculated based on each OBB region and each ear center point, including: The central rolling trajectory of each ear is calculated based on the center point of each ear.

[0097] Determine the central axis of the ear in each OBB region, extract the two intersection points of the central axis of the ear and each OBB region, and construct the ear tip point set and ear base point set based on the set of the two intersection points.

[0098] The ear tip rolling trajectory is determined based on the ear tip point set, and the ear base rolling trajectory is determined based on the ear base point set.

[0099] In the embodiments of this application, constructing an effective surface area image of the ear based on the boundary trajectory includes: constructing an effective surface area image of the ear based on the center rolling trajectory, the ear tip rolling trajectory, and the ear base rolling trajectory.

[0100] The embodiments of this application can dynamically calculate the ear diameter and circumference by calculating the ear outline and OBB region frame by frame, and determine the central axis of the ear.

[0101] Meanwhile, the embodiments of this application can calculate the rolling trajectory of the ear center based on the set of center points of the ear.

[0102] In the embodiments of this application, the two intersections of the OBB region and the central axis of the ear constitute the point sets of the ear tip and the base, respectively. The two point sets represent the rolling trajectories of the ear tip and the base, respectively, and are represented as the ear tip rolling trajectory and the ear base rolling trajectory.

[0103] In this embodiment, the grain coverage range of the ear rolling 360 degrees can be determined based on the rolling distance of the ear's center point. The start and end points of the ear rolling one revolution are dynamically calculated on the rolling trajectory of the ear's center point. The range of the ear rolling 360 degrees is represented by the rolling trajectory of the ear's center point between the start and end points.

[0104] Subsequently, curve fitting can be performed on the rolling trajectory points of the ear center point, and two normal lines of the curve can be generated at the coordinates of the start and end points on the curve.

[0105] Ultimately, the two generated normal lines, the rolling trajectory line at the tip of the ear, and the rolling trajectory line at the base of the ear together constitute the effective surface area image of the ear rolling around once, which is to form a complete grain imprint map of the entire surface of the ear.

[0106] This application embodiment constructs a three-dimensional rolling trajectory of the ear's center, tip, and base, accurately reconstructing the complete morphology of the ear surface and providing accurate data dimensions for maize phenotypic analysis. Compared to the limitations of traditional two-dimensional images that can only observe local features of the ear, this application embodiment achieves complete acquisition of kernel information across the entire ear surface through dynamic tracking of the intersection of the OBB bounding box and the central axis, providing reliable computational data for the automated measurement of key traits such as ear row count and kernel weight. Secondly, the effective surface area image constructed based on the trajectory line can be directly used for intelligent identification and counting of ear kernels. Combined with multi-view image acquisition in the field, it can significantly improve the efficiency and accuracy of maize seed evaluation. Finally, this method does not require complex three-dimensional reconstruction equipment; high-precision three-dimensional morphological reconstruction can be achieved using only a sequence of images from a regular camera, lowering the hardware threshold for phenotypic analysis.

[0107] This application addresses the issues of incomplete information and significant edge distortion in single-frame images, enabling full-surface data acquisition and phenotypic extraction of the ear. Furthermore, it achieves topological analysis of the grains on the ear surface, accurately calculating parameters such as the number of ear rows.

[0108] This application embodiment only requires an inclined plane and an industrial camera to build a basic video acquisition device for the rolling of ears of fruit, but the optical center of the industrial camera is required to be perpendicular to the inclined plane of the rolling ears of fruit; an electronic balance can be selected to obtain the weight data of the ears of fruit; an LED light source and a light shield can be selected to improve the imaging conditions.

[0109] In this embodiment of the application, during video acquisition, the detection areas for the inlet and outlet of the rolling ear in the real-time video can be adjusted. The size and position of the inlet and outlet captured by the camera can be interactively adjusted in the software to adapt to the rolling of the ear in any direction without changing the position and orientation of the camera. Using image processing methods, a corresponding event is triggered once a significant change in the image information (brightness / color, etc.) of two small areas is detected. During real-time camera shooting, a video recording event is triggered only when the ear enters the image detection inlet, and a stop video recording event is triggered when it rolls past the outlet, and the video is saved. One video is generated for each ear rolling once, and the video length is approximately 1 second. Each acquired video contains complete information about one complete rotation of the ear. The video contains virtually no invalid or redundant data, saving storage space. The throughput of ear video acquisition can reach one ear per second.

[0110] This application's embodiments support parallel computing, improving video analysis efficiency. After each ear of grain video is acquired, it can be sent to the video analysis pipeline, where the ear phenotype is calculated using the aforementioned ear tracking and grain imprinting algorithms.

[0111] Compared with the prior art, the embodiments of this application have the following beneficial effects: (1) Zero-damage collection. The incidence of mechanical damage is low and does not affect the subsequent use of breeding materials.

[0112] (2) High throughput. In standard mode, the throughput can reach 3,000 ears / hour; which is far superior to all other test equipment. (3) High precision. The accuracy rate of ear row number identification is ≥95%; the mean absolute error (MAE) of total grain number estimation is ≤15 grains / ear.

[0113] (4) Low cost. The cost of the whole machine is about 1 / 5 of that of a traditional mechanical rotary table system and 1 / 10 of that of a roller drive system.

[0114] (5) Portable and easy to deploy. It does not require an external power supply, has a modular design, can be quickly assembled and disassembled, and is suitable for complex environments such as fields.

[0115] (6) Full-process traceability. The ear video contains complete surface information of the ear, and the video compression greatly saves storage space; the automated analysis process, standardized data output and multi-channel diagnostic video provide a complete chain of evidence.

[0116] Corresponding to the non-powered rolling ear phenotypic measurement method in the above embodiment, Figure 3 This is a structural block diagram of a non-powered rolling ear phenotyping device provided in one embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. References Figure 3The non-powered rolling ear phenotyping device 20 includes: a data acquisition module 201, a first processing module 202, a second processing module 203, a third processing module 204, and a fourth processing module 205.

[0117] The data acquisition module 201 is used to acquire the rolling video of the ear of fruit and extract multiple video frames from the rolling video, wherein the multiple video frames include at least one video frame of the ear of fruit rolling around once.

[0118] The first processing module 202 is used to extract regions from each video frame to obtain the OBB region, ROI region and center point of the ear of fruit in each video frame.

[0119] The second processing module 203 is used to extract the effective grain region in each video frame based on the ROI region in each video frame, and to perform image fusion of the effective grain regions in each video frame to obtain the effective grain image of the ear of grain.

[0120] The third processing module 204 is used to calculate the boundary trajectory of the ear based on each OBB region and the center point of each ear, and to construct an effective surface area image of the ear based on the boundary trajectory. The boundary trajectory includes the center rolling trajectory of the ear, the tip rolling trajectory of the ear, and the base rolling trajectory of the ear.

[0121] The fourth processing module 205 is used to construct multiple sets of point pairs based on the effective grain image and the effective surface area image, and to measure the phenotype of the ear based on the multiple sets of point pairs, wherein each set of point pairs includes a rolling start point and a rolling end point of the ear.

[0122] In one embodiment of this application, the second processing module 203 is used to extract seeds from each ROI region based on a predetermined target detection model to obtain seed regions in each video frame; and to perform distortion correction on each seed region to obtain effective seed regions in each video frame.

[0123] In one embodiment of this application, the second processing module 203 is used to filter the regions in each grain region where the distance to the central axis of the ear of grain is less than a preset distance as effective grain regions, thereby obtaining the effective grain regions in each video frame.

[0124] In one embodiment of this application, the second processing module 203 is used to perform image fusion of the grains in each effective grain region according to the time sequence of video frames, so as to fuse and remove duplicates of each grain to obtain an effective grain image of the ear.

[0125] In one embodiment of this application, the third processing module 204 is used to calculate the center rolling trajectory of the ear based on the center point of each ear; determine the central axis of the ear in each OBB region; extract the two intersection points of the central axis of the ear and each OBB region in each OBB region; and construct a set of ear tip points and a set of ear base points based on the set of the two intersection points; determine the ear tip rolling trajectory based on the ear tip point set; and determine the ear base rolling trajectory based on the ear base point set.

[0126] In one embodiment of this application, the third processing module 204 is used to construct an effective surface area image of the ear based on the central rolling trajectory, the ear tip rolling trajectory, and the ear base rolling trajectory.

[0127] In one embodiment of this application, the fourth processing module 205 is used to construct a Delaunay triangulation of multiple pairs of points; calculate the shortest path of each pair of points in the Delaunay triangulation to obtain multiple shortest paths; and determine the number of rows of the ear and the number of grains in each row based on the multiple shortest paths to obtain the phenotypic data of the ear.

[0128] See Figure 4 , Figure 4 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 4 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of the modules in the aforementioned device embodiments, for example... Figure 3 The functions of the data acquisition module 201, the first processing module 202, the second processing module 203, the third processing module 204, and the fourth processing module 205 are shown.

[0129] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0130] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0131] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory.

[0132] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation method described in the non-powered rolling ear phenotypic measurement method provided in the embodiments of this application, or they can execute the implementation method of the electronic device described in the embodiments of this application, which will not be repeated here.

[0133] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0134] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0135] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0136] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0138] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0139] This application provides a non-powered rolling ear phenotyping system, including a support platform for supporting and rolling the ear; wherein the support platform is continuously adjustable within a preset angle range; an image acquisition device for acquiring video of the ear rolling on the support platform; and a non-powered rolling ear phenotyping device 20 as described above.

[0140] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of unpowered rolling boll phenotype measurement, characterized in that, include: Acquire a rolling video of the ear of fruit, and extract multiple video frames from the rolling video, wherein the multiple video frames include at least one video frame of the ear of fruit rolling one full circle; Region extraction is performed on each video frame to obtain the OBB region, ROI region, and center point of the ear of fruit in each video frame; Based on the ROI region in each video frame, the effective grain region in each video frame is extracted, and the effective grain regions of each video frame are image fused to obtain the effective grain image of the ear of grain. The boundary trajectory of the ear is calculated based on each OBB region and each ear center point, and an effective surface area image of the ear is constructed based on the boundary trajectory; the boundary trajectory includes the center rolling trajectory of the ear, the tip rolling trajectory of the ear, and the base rolling trajectory of the ear. Multiple point pairs are constructed based on the effective grain image and the effective surface area image, and the phenotype of the ear is measured based on the multiple point pairs, wherein each point pair includes a rolling start point and a rolling end point of the ear.

2. The unpowered rolling pod phenotyping method of claim 1, wherein, The step of extracting the effective seed region from each video frame based on the ROI region in each video frame includes: Based on a pre-determined target detection model, seed regions are extracted from each ROI region to obtain the seed regions in each video frame. Distortion correction is performed on each of the seed regions to obtain the effective seed regions in each of the video frames.

3. The unpowered rolling pod phenotyping method of claim 2, wherein, The step of performing distortion correction on each of the seed regions to obtain the effective seed region in each of the video frames includes: The effective grain regions are selected by filtering the regions in each grain region where the distance from the central axis of the ear of grain is less than a preset distance, thus obtaining the effective grain regions in each video frame.

4. The unpowered rolling pod phenotyping method of claim 1, wherein, The step of fusing the effective grain regions in each of the video frames to obtain the effective grain image of the ear of grain includes: The grains in each of the effective grain regions are image-fused according to the time sequence of the video frames to remove duplicates and obtain the effective grain image of the ear of grain.

5. The unpowered rolling pod phenotyping method of claim 1, wherein, The calculation of the boundary trajectory of the ear based on each of the OBB regions and each ear center point includes: Calculate the center rolling trajectory of the ear based on the center point of each ear; Determine the central axis of the ear in each OBB region, extract the two intersection points of the central axis of the ear in each OBB region and each OBB region, and construct the ear tip point set and ear base point set based on the set of the two intersection points; The ear tip rolling trajectory is determined based on the ear tip point set, and the ear base rolling trajectory is determined based on the ear base point set.

6. The unpowered rolling pod phenotyping method of claim 5, wherein, The construction of the effective surface area image of the ear of fruit based on the boundary trajectory includes: An effective surface area image of the ear is constructed based on the central rolling trajectory, the ear tip rolling trajectory, and the ear base rolling trajectory.

7. The unpowered rolling pod phenotyping method of claim 1, wherein, The measurement of the phenotype of the ear of fruit based on the multiple sets of point pairs includes: Construct the Delaunay triangulation of the multiple pairs of points; Calculate the shortest path for each pair of points in the Delaunay triangulation to obtain multiple shortest paths; Based on the multiple shortest paths, the number of rows in the ear and the number of kernels in each row are determined to obtain the phenotypic data of the ear.

8. A non-powered rolling ear phenotyping device, characterized in that, include: The data acquisition module is used to acquire a rolling video of the ear of fruit and extract multiple video frames from the rolling video, wherein the multiple video frames include at least one video frame of the ear of fruit rolling around once. The first processing module is used to extract regions from each video frame to obtain the OBB region, ROI region and center point of the ear of fruit in each video frame; The second processing module is used to extract the effective grain region in each video frame based on the ROI region in each video frame, and to perform image fusion of the effective grain regions in each video frame to obtain the effective grain image of the ear of grain. The third processing module is used to calculate the boundary trajectory of the ear based on each OBB region and each ear center point, and to construct an effective surface area image of the ear based on the boundary trajectory; the boundary trajectory includes the center rolling trajectory of the ear, the tip rolling trajectory of the ear, and the base rolling trajectory of the ear. The fourth processing module is used to construct multiple sets of point pairs based on the effective grain image and the effective surface area image, and to measure the phenotype of the ear of grain based on the multiple sets of point pairs, wherein each set of point pairs includes a rolling start point and a rolling end point of the ear of grain.

9. A non-powered rolling ear phenotypic measurement system, characterized in that, include: A support platform is used to support and rotate the ear of fruit; wherein the support platform is continuously adjustable within a preset angle range; An image acquisition device is used to acquire rolling video of the ear of fruit on the support platform; And, as described in claim 8, the non-powered rolling ear phenotyping device.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images