A method for in-situ three-dimensional phenotypic information acquisition of crops

Abstract

This invention discloses a method for in-situ three-dimensional phenotypic information acquisition of crops, relating to the field of agricultural information acquisition technology. The method includes: acquiring raw images captured by a camera in real time during the autonomous walking of a mobile chassis along an optimal path, and determining the presence of an AprilTag based on the raw images; if an AprilTag exists, determining the camera's pose relative to the AprilTag, adjusting the robot's pose until the crop is located in the center of the raw image, at which point the mobile chassis stops moving; acquiring raw video footage captured by the camera during the rotation of the motor arm; and determining the corresponding three-dimensional phenotypic information of each crop based on the raw video footage. This invention reduces the cost of acquiring in-situ three-dimensional phenotypic information of crops.

Description

Technical Field

[0001] This invention relates to the field of agricultural information acquisition technology, and in particular to a method for in-situ three-dimensional phenotypic information acquisition of crops. Background Technology

[0002] Plant breeders examine crop phenotypic information to screen for high-yielding, disease-resistant, and high-quality varieties for testing and promotion. Accurately acquiring plant information scientifically forms the data foundation for subsequent scientific research. Plant phenotypic information is typically categorized into one-dimensional, two-dimensional, and three-dimensional information based on data dimensions. As the data dimensions increase, the amount of information also increases, and the acquisition of three-dimensional phenotypic information is gradually becoming a trend in high-throughput phenotypic analysis and attracting increasing attention from researchers. However, limited by existing equipment and processing algorithms, current crop three-dimensional phenotypic information acquisition and analysis are almost entirely conducted in laboratory environments. This removes the crops from their original environment, introducing certain errors into phenotypic evaluation. In fact, breeding experts prefer in-situ, non-destructive, high-precision three-dimensional crop information detection.

[0003] For in-situ 3D reconstruction of crops in the field, there are two main approaches: 3D information acquisition sensors and multi-view image reconstruction based on 2D images. Commonly used 3D information acquisition sensors include laser scanners, structured light cameras, binocular vision, LiDAR, and time-of-flight cameras, which can directly output 3D point cloud information. Currently, most 3D information acquisition sensors are handheld, which is time-consuming, labor-intensive, costly, and has low accuracy. Alternatively, robotic arms can be used to carry 3D sensors for multi-view imaging, but this is costly and structurally complex. Therefore, how to acquire 3D information of crops in a lighter, smarter, more accurate, and lower-cost manner is a key technical issue that needs attention. Summary of the Invention

[0004] The purpose of this invention is to provide a method for in-situ three-dimensional phenotypic information acquisition of crops, thereby reducing the cost of acquiring such information.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] A method for in-situ three-dimensional phenotypic information acquisition of crops is disclosed, the method being implemented based on a crop in-situ three-dimensional phenotypic information acquisition robot; the robot includes: a mobile chassis, a lower-level computer, and a three-dimensional phenotypic information acquisition device, the three-dimensional phenotypic information acquisition device including: a motor arm and a camera; the lower-level computer is connected to both the motor arm and the upper-level computer, the motor arm is mounted on the mobile chassis, and the camera is mounted on the motor arm; the upper-level computer is connected to a data processing backend.

[0007] The process of implementing the method based on the robot includes:

[0008] During the autonomous movement of the mobile chassis along the optimal path, the host computer acquires the original images captured by the camera in real time and determines whether an AprilTag exists based on the original images. The AprilTag is affixed to the side of the crop, and the information in the AprilTag includes: variety, plant number, and planting date.

[0009] When the AprilTag appears, the pose of the camera relative to the AprilTag is determined, and based on the pose of the robot relative to the AprilTag, the pose of the robot is adjusted by the lower computer until the crop is in the center of the original image, after which the moving chassis stops moving.

[0010] The host computer controls the motor arm to rotate one revolution through the slave computer, and acquires the original video captured by the camera during the rotation of the motor arm, and sends the original video to the data processing backend;

[0011] The host computer controls the mobile chassis to continue its autonomous walking process and returns "The host computer acquires the original images captured by the camera in real time" until the original video corresponding to all crops in the target area is obtained;

[0012] The data processing backend determines the three-dimensional phenotypic information of each crop based on the original video of each crop. The three-dimensional phenotypic information includes: stem diameter, projected leaf area, actual leaf area, leaf area density, leaf tilt angle, and plant type phenotypic information.

[0013] Optionally, the process of determining the optimal path includes:

[0014] Construct the velocity sampling space of the robot;

[0015] From the starting point to the ending point, at each position in the velocity sampling space, multiple sampling velocity groups are determined at each position; the sampling velocity groups include: linear velocity and angular velocity;

[0016] Based on the sampling rate groups at each location, multiple paths from the starting point to the ending point are determined;

[0017] Calculate the score for each path, and determine the path with the highest score as the optimal path.

[0018] Optionally, the velocity sampling space is the union of velocity boundary constraints, acceleration constraints, and environmental obstacle constraints.

[0019] Optionally, the velocity boundary limit V m for:

[0020] V m ={(v,ω)|v∈[v min vmax ],ω∈[ω min ω max ]};

[0021] Where v is the linear velocity; v min The minimum linear velocity; v max ω is the maximum linear velocity; ω is the angular velocity; ω min ω is the minimum angular velocity; max This is the maximum angular velocity;

[0022] The acceleration limit V a for:

[0023]

[0024] Among them, v c a is the linear velocity at the current moment; vmax ω represents the maximum linear acceleration; Δt represents the time interval; ω represents the maximum linear acceleration. c a is the angular velocity at the current moment; ωmax This is the maximum angular acceleration;

[0025] The environmental obstacles limit V d for:

[0026]

[0027] Where dist(v,ω) is the shortest distance between the trajectory and the obstacle under linear velocity v and angular velocity ω;

[0028] Velocity sampling space V s for:

[0029] V s =V m ∩V a ∩V d ;

[0030] Here, ∩ represents the union operation.

[0031] Optionally, the process of determining whether an AprilTag exists based on any original image includes:

[0032] The original image is converted into a grayscale image, the grayscale image is segmented into multiple blocks, and the maximum and minimum grayscale values ​​of each block are determined.

[0033] The maximum and minimum gray values ​​of each block are subjected to 3-neighborhood maximum and minimum filtering to obtain the filtered maximum and minimum gray values ​​of the corresponding blocks.

[0034] The average of the maximum and minimum filtered gray values ​​of each block is determined as the threshold of the corresponding block.

[0035] By utilizing continuous boundary segmentation and fast decoding, and based on the threshold of each block, it is determined whether the AprilTag exists in the original image.

[0036] Optionally, determining the pose of the camera relative to the AprilTag includes:

[0037] Obtain the image coordinates of the four corner points of AprilTag, as well as the camera's intrinsic parameter matrix and distortion parameters;

[0038] Based on the intrinsic parameter matrix and the distortion parameters, the image coordinates are converted into camera coordinates;

[0039] Using the PnP algorithm, the camera coordinates are determined based on the image coordinates of each corner point and the corresponding camera coordinates.

[0040] Using the homography transformation matrix, the camera coordinates of each corner point are converted into the world coordinates of the corresponding corner point, and the camera coordinates of the camera are converted into the world coordinates of the camera.

[0041] The pose of the camera relative to the AprilTag is determined based on the world coordinates of each corner point and the world coordinates of the camera.

[0042] Optionally, based on the original videos corresponding to each crop, the three-dimensional phenotypic information of the corresponding crop is determined, including:

[0043] Select any crop as the current crop;

[0044] Collect multiple frames of original images from different perspectives from the original video corresponding to the current crop;

[0045] Extract feature points from each original image of the current crop and perform feature matching to obtain the corresponding matched image;

[0046] The matching images are reconstructed using the Neus network to obtain the corresponding 3D model;

[0047] Based on the physical dimensions of AprilTag, assign absolute values ​​to the relative dimensions of the 3D model;

[0048] The three-dimensional phenotypic information of the current crop is determined based on the relative size of the three-dimensional model.

[0049] Optionally, the robot further includes: an aluminum alloy support frame and carbon fiber guide rails;

[0050] The aluminum alloy bracket is mounted on the mobile chassis, the carbon fiber guide rail is mounted on the aluminum alloy bracket, and the three-dimensional phenotypic information acquisition device is connected to the carbon fiber guide rail.

[0051] Optionally, the robot further includes: a human-computer interaction display;

[0052] The human-computer interaction display is mounted on the aluminum alloy bracket.

[0053] Optionally, the three-dimensional phenotypic information acquisition device further includes: a motor, a moving slider, and a gimbal;

[0054] The motor is connected to the carbon fiber guide rail and the motor arm respectively. The carbon fiber guide rail and the motor arm are slidably connected through the movable slider. The camera is mounted on the motor arm through the gimbal.

[0055] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0056] This invention discloses a method for in-situ three-dimensional phenotypic information acquisition of crops. It cleverly utilizes a simplified, low-cost mechanical structure to acquire in-situ three-dimensional phenotypic information of crops. Three-dimensional reconstruction can be achieved based solely on RGB images, avoiding the use of expensive three-dimensional sensors and thus reducing costs. The three-dimensional reconstruction algorithm based on deep learning (i.e., the Neus network) can perform in-situ three-dimensional reconstruction of crops with higher accuracy compared to traditional reconstruction algorithms, which is beneficial for subsequent extraction of phenotypic information. The detection process is optimized by cleverly introducing Apriltag, which can simultaneously achieve multiple functions such as crop localization, robot-assisted navigation, variety information storage, and pose correction, enabling more efficient automatic acquisition of three-dimensional phenotypic information and reducing the cost of collecting in-situ three-dimensional phenotypic information of crops. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a schematic diagram of the crop in-situ three-dimensional phenotypic information acquisition method provided in an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of the automated acquisition and analysis process for in-situ three-dimensional phenotypic information of crops;

[0060] Figure 3 A schematic diagram of a robot for collecting in-situ three-dimensional phenotypic information of crops;

[0061] Figure 4 A schematic diagram of a three-dimensional phenotypic information acquisition device and a carbon fiber guide rail structure;

[0062] Figure 5 This is a schematic diagram of the 3D reconstruction results when the crop is strawberry.

[0063] Symbol explanation:

[0064] Mobile chassis—1, 3D phenotypic information acquisition device—2, motor arm—3, camera—4, aluminum alloy bracket—5, carbon fiber guide rail—6, human-computer interaction display—7, motor—8, moving slider—9, gimbal—10. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] The purpose of this invention is to provide a method for collecting in-situ three-dimensional phenotypic information of crops, aiming to reduce the cost of collecting in-situ three-dimensional phenotypic information of crops.

[0067] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0068] like Figures 1-2 As shown, the crop in-situ three-dimensional phenotypic information acquisition method in this embodiment is based on the following... Figures 3-4 The robot shown is used for in-situ 3D phenotypic information acquisition of crops. The robot includes a mobile chassis 1, a lower-level computer, and a 3D phenotypic information acquisition device 2. The 3D phenotypic information acquisition device 2 includes a motor arm 3 and a camera 4. The lower-level computer is connected to the motor arm 3 and the upper-level computer respectively. The motor arm 3 is mounted on the mobile chassis 1, and the camera 4 is mounted on the motor arm 3. The upper-level computer is connected to the data processing backend.

[0069] The process of implementing the robot-based method includes:

[0070] Step 1: During the autonomous walking process of the mobile chassis along the optimal path, the host computer acquires the original images captured by the camera in real time and determines whether the AprilTag exists based on the original images.

[0071] The AprilTag is affixed to the side of the crop, and the information in the AprilTag includes: variety, plant number, and planting date.

[0072] Specifically, after the robot starts operating, it is first manually remotely controlled to move in the target area. The 3D LiDAR is used to acquire the surrounding point cloud information to map the scene. The robot is then positioned based on its own odometry. After manually setting the starting point and the ending point, the optimal path is planned using a dynamic window algorithm.

[0073] As an optional implementation method, the process of determining the optimal path includes:

[0074] Step 101: Construct the robot's velocity sampling space.

[0075] As an alternative implementation, the velocity sampling space is the union of velocity boundary constraints, acceleration constraints, and environmental obstacle constraints.

[0076] As an optional implementation, the velocity boundary limit V m for:

[0077] V m ={(v,ω)|v∈[v min v max ],ω∈[ω min ω max ]}.

[0078] Where v is the linear velocity; v min The minimum linear velocity; v max ω is the maximum linear velocity; ω is the angular velocity; ω min ω is the minimum angular velocity; max This represents the maximum angular velocity.

[0079] Acceleration Limit V a for:

[0080]

[0081] Among them, v c a is the linear velocity at the current moment; vmax ω represents the maximum linear acceleration; Δt represents the time interval; ω represents the maximum linear acceleration. c a is the angular velocity at the current moment; ωmax This is the maximum angular acceleration.

[0082] Environmental Obstacles Limitation V d for:

[0083]

[0084] Where dist(v,ω) is the shortest distance between the trajectory and the obstacle under linear velocity v and angular velocity ω.

[0085] Velocity sampling space V s for:

[0086] V s =V m ∩V a ∩V d .

[0087] Here, ∩ represents the union operation.

[0088] Step 102: At each position from the starting point to the ending point, determine multiple sampling velocity groups at each position in the velocity sampling space; the sampling velocity groups include: linear velocity and angular velocity.

[0089] Specifically, the process of determining the number of sampling rate groups includes:

[0090] Determine the velocity sampling space V s Subsequently, the dynamic window algorithm uniformly samples the velocity sampling space at a certain sampling interval (resolution). Within the velocity sampling space, sampling resolutions are set for linear velocity and angular velocity, respectively, using E... v E ω Therefore, the number of sampling rate groups can be determined, as shown in the following formula:

[0091] n = [(v high -v low ) / E v ]·[(ω high -ω low ) / E ω ].

[0092] Among them, v high For velocity sampling space V s The upper limit of the midline velocity space; v low For velocity sampling space V s Lower limit of the midline velocity space; ω high For velocity sampling space V s The upper limit of the angular velocity space; ω low For velocity sampling space V s The lower bound of the angular velocity space. After sampling a set of (v,ω), the position is updated using the kinematic model of the mobile robot.

[0093] Step 103: Based on the sampling rate groups at each location, determine multiple paths from the starting point to the ending point.

[0094] Step 104: Calculate the score for each path and determine the path with the highest score as the optimal path.

[0095] Specifically, after determining the robot's various sampling velocity groups, the simulated trajectories for some of these groups are feasible, but some are substandard. Therefore, it's necessary to evaluate and select the best among the multiple sampled trajectories. The optimal trajectory is selected by comparing scores using a standard evaluation trajectory, and the speed corresponding to this optimal trajectory is chosen as the driving speed.

[0096] G (v,ω) =σ(α·heading(v,ω))+σ(β·dist(v,ω))+σ(γ·

[0097] velocity(v,ω)).

[0098] Among them, G (v,ω) The evaluation function scores the path at online speed v and angular velocity ω; σ(·) represents normalization; α, β, and γ are coefficients of the evaluation function; heading(v,ω) is the azimuth evaluation function of the path at online speed v and angular velocity ω, heading(v,ω)=π-Δθ, where Δθ is the error of the angle between the direction of the endpoint position of the path at online speed v and angular velocity ω and the line connecting the actual endpoint; dist(v,ω) is the distance evaluation function of the path at online speed v and angular velocity ω, representing the shortest distance between the path at online speed v and angular velocity ω and the obstacle; velocity(v,ω) is the velocity evaluation function of the path at online speed v and angular velocity ω, representing the magnitude of the velocity, which can be directly represented by the magnitude of the linear velocity v. The larger the velocity evaluation function, the faster the speed on the planned trajectory, and the higher the evaluation score.

[0099] As an optional implementation, the process of determining whether an AprilTag exists based on any original image includes:

[0100] Step 111: Convert the original image into a grayscale image, segment the grayscale image to obtain multiple blocks, and determine the maximum and minimum grayscale values ​​of each block.

[0101] Step 112: Perform 3-neighborhood maximum and minimum grayscale filtering on the maximum and minimum grayscale values ​​of each block to obtain the filtered maximum and minimum grayscale values ​​of the corresponding block.

[0102] Step 113: Determine the average of the maximum and minimum filtered gray values ​​of each block as the threshold for the corresponding block.

[0103] Step 114: Using continuous boundary segmentation and fast decoding, determine whether the AprilTag exists in the original image based on the threshold of each block.

[0104] Specifically, the robot moves according to the navigation algorithm described above, while the camera captures a video stream at 2K resolution and 15 frames per second, continuously detecting whether the AprilTag appears in the frame. The specific detection method is as follows:

[0105] (1) Adaptive Thresholding Segmentation: First, the original RGB image is converted to a grayscale image. Second, the grayscale image is divided into 4×4 grid blocks, and the maximum and minimum grayscale values ​​of each block are calculated. Third, the maximum and minimum grayscale values ​​of all blocks are subjected to 3-neighborhood max-min filtering. Finally, the mean of the filtered maximum and minimum grayscale values ​​((max+min) / 2) is used as the threshold for block segmentation, assigning white (255, 255, 255) or black (0, 0, 0) to each pixel. After adaptive thresholding, a binarized image is obtained. The advantage of this step is increased robustness and improved computational efficiency.

[0106] (2) Continuous boundary segmentation: The purpose is to find the boundary points of possible Apriltag boundaries in the binarized image. The part where black pixels and white pixels meet is segmented using union-find, resulting in multiple sets of boundary points, which are unordered. Since there are multiple black and white blocks in a single binarized image, the algorithm will give each black and white block a unique ID number as an identifier after segmentation.

[0107] (3) Fast Decoding: First, fit the disordered cluster of boundary points into a quadrilateral and sort them according to the angle of the centroid. Second, for the sorted boundary points, select points within a certain range from the center point in sequence to fit straight lines, iterate the index continuously, and calculate the total error of each straight line. Third, perform low-pass filtering on the total error and select the corner points corresponding to the four straight lines with the largest total error as the corner points of the quadrilateral. Then, fit straight lines between the corner points of the quadrilateral and obtain the corner points of the four straight lines as the vertices of AprilTag. Finally, compare the content of the quadrilateral with the known code words, filter out the incorrect candidate quadrilaterals, output the detection result of AprilTag, and record the ID.

[0108] Step 2: When the AprilTag appears, determine the camera's pose relative to the AprilTag, and based on the robot's pose relative to the AprilTag, adjust the robot's pose via the lower-level machine until the crop is in the exact center of the original image, then move the chassis to stop moving.

[0109] As an optional implementation, determining the camera pose relative to the AprilTag includes:

[0110] Step 21: Obtain the image coordinates of the four corner points of AprilTag, as well as the camera's intrinsic parameter matrix and distortion parameters.

[0111] Step 22: Based on the intrinsic parameter matrix and distortion parameters, convert each image coordinate into camera coordinates.

[0112] Step 23: Using the PnP algorithm, determine the camera coordinates based on the image coordinates of each corner point and the corresponding camera coordinates.

[0113] Step 24: Using the homography transformation matrix, convert the camera coordinates of each corner point into the world coordinates of the corresponding corner point, and convert the camera coordinates of the camera into the world coordinates of the camera.

[0114] Step 25: Determine the pose of the camera relative to the AprilTag based on the world coordinates of each corner point and the world coordinates of the camera.

[0115] Specifically, the AprilTag pose recognition process includes:

[0116] 1) Camera intrinsic parameters and distortion parameters: The camera's intrinsic parameter matrix and distortion parameters are obtained through camera calibration. The intrinsic parameter matrix includes: camera focal length and principal point.

[0117] 2) Image coordinates of the four corner points of AprilTag: AprilTag is usually square, so we can assume that the four corner points of AprilTag are cube-shaped in image space. The image coordinates of the four corner points of AprilTag can be calculated using the physical dimensions of AprilTag.

[0118] 3) Image coordinate to camera coordinate conversion: Using the camera's intrinsic parameter matrix and distortion parameters, image coordinates are converted to camera coordinates. The formula for converting camera coordinates to image coordinates is:

[0119]

[0120] Where (i,j) are the image coordinates; K is the intrinsic parameter matrix; (X c ,Y c Z c () represents the camera coordinates.

[0121] 4) Homography Transformation Matrix Estimation: Descriptors corresponding to each feature point are extracted through feature point matching. Matching these descriptors identifies matching feature point pairs. The RANSAC algorithm is used to eliminate incorrect matches, and the system of equations is solved to calculate the Homography homography transformation matrix. The homography transformation matrix maps points on the image plane to the world plane.

[0122]

[0123] Among them, (X) w ,Y w Z w ) represents world coordinates; H represents the homography transformation matrix.

[0124] 5) Camera pose estimation: The Perspective-n-Point (PnP) algorithm is used to estimate the camera pose based on the camera coordinates and the image coordinates of the four corner points of AprilTag. The camera pose includes the rotation matrix R and the translation matrix T.

[0125] 6) Coordinate system transformation: Convert the pose in the camera coordinate system to the pose in the world coordinate system in order to understand the position and orientation of AprilTag in the actual scene.

[0126] Through the above six steps, the robot's pose relative to the Apriltag can be determined. The pose is then sent to the lower-level machine via the robot's operating system, controlling the robot to adjust its pose. During this process, the Apriltag's pose is continuously acquired, forming a negative feedback control system with the robot's motion system until the robot adjusts to a suitable position for taking a picture.

[0127] Step 3: The host computer controls the motor arm to rotate one revolution through the slave computer, and acquires the original video captured by the camera during the rotation of the motor arm, and sends the original video to the data processing backend.

[0128] Step 4: The host computer controls the mobile chassis to continue the autonomous walking process and returns to Step 1 until the original video of all crops in the target area is obtained.

[0129] Specifically, the areas where the robots work, i.e. the target areas, include: greenhouses and between fields.

[0130] Specifically, after the posture adjustment is complete, the 3D phenotypic information acquisition device begins to move. The motor drives the motor arm to rotate 360 ​​degrees at a speed of 5 r / min. The video captured by the camera is named with the variety information and stored in the host computer. After storage, the host computer sends a signal to continue automatic movement until the camera scans the AprilTag again, and repeats the above process until the phenotypic acquisition of all varieties is completed.

[0131] Step 5: The data processing backend determines the three-dimensional phenotypic information of each crop based on the original video. The three-dimensional phenotypic information includes: stem diameter, projected leaf area, actual leaf area, leaf area density, leaf tilt angle, and plant type phenotypic information.

[0132] As an optional implementation, based on the original video of each crop, the three-dimensional phenotypic information of the corresponding crop is determined, including:

[0133] Step 51: Select any crop as the current crop.

[0134] Step 52: Collect multiple frames of original images from different perspectives from the original video corresponding to the current crop.

[0135] Step 53: Extract feature points from each original image of the current crop and perform feature matching to obtain the corresponding matched image.

[0136] Step 54: Use the Neus network to perform 3D reconstruction on the matched image to obtain the corresponding 3D model.

[0137] Specifically, the 3D model is a mesh model.

[0138] Specifically, the NeuS network had undergone 300,000 iterations of training before being used for 3D reconstruction. The loss function for training the NeuS network was:

[0139]

[0140] in, The loss value is represented by λ; both λ and δ are weighting coefficients. The difference between the reconstructed 3D model and a photograph of the actual crop; This is the SDF function used in constraint calculations; is an optional cross-entropy loss function.

[0141] When the crop is strawberry, the 3D model is as follows: Figure 5 As shown.

[0142] Step 55: Assign absolute values ​​to the relative dimensions of the 3D model based on the physical dimensions of AprilTag.

[0143] Specifically, before step 55, the process also includes: denoising the mesh model and segmenting the target plant.

[0144] Step 56: Determine the three-dimensional phenotypic information of the current crop based on the relative size of the three-dimensional model.

[0145] Specifically, the three-dimensional phenotypic information of the crop is calculated based on the relative relationship between the physical dimensions of the reference object and the three-dimensional model (i.e., the relative dimensions of the three-dimensional model).

[0146] As an optional implementation, the robot also includes: an aluminum alloy support 5 and a carbon fiber guide rail 6.

[0147] An aluminum alloy bracket 5 is mounted on a mobile chassis 1, a carbon fiber guide rail 6 is mounted on the aluminum alloy bracket 5, and a three-dimensional phenotypic information acquisition device 2 is connected to the carbon fiber guide rail 6.

[0148] As an alternative implementation, the robot also includes: a human-computer interaction display 7.

[0149] The human-computer interaction display 7 is mounted on the aluminum alloy bracket 5.

[0150] As an optional implementation, the three-dimensional phenotypic information acquisition device 2 also includes: a motor 8, a moving slider 9, and a gimbal 10.

[0151] Motor 8 is connected to carbon fiber guide rail 6 and motor arm 3 respectively. Carbon fiber guide rail 6 and motor arm 3 are slidably connected by movable slider 9. Camera 4 is mounted on motor arm 3 by gimbal 10.

[0152] Specifically, the robot's mobile platform is a tracked chassis measuring 680mm × 580mm × 300mm, powered by a lithium battery and driven by two independent DC motors with a rated power of 250W. The track wheels are 110mm wide, and each track structure consists of a track, one drive wheel, one driven wheel, and three shock-absorbing wheels. An aluminum alloy frame, measuring 450mm × 350mm × 900mm, is mounted on the tracked chassis. This frame is divided into three layers and can house the hardware circuitry, the host computer control system, the automatic navigation system, and the phenotypic information acquisition system. A carbon fiber guide rail 6, 1200mm long, is installed on the top layer of the frame. A slider 9 is mounted on the guide rail and can slide freely along it. A lightweight, simplified 3D phenotypic information acquisition device 2 is mounted on the slider 9. This device consists of a motor 8, a motor arm 3, a gimbal 10, and a camera 4. Motor 8 is a low-speed, high-torque motor, and the gimbal 10 is a miniature gimbal. Motor 8 is fixed to the movable slider 9, with its output shaft pointing vertically downwards. A carbon fiber motor arm 3 is fixed to the output shaft of motor 8. Motor arm 3 is 420mm long and has a sliding groove at its end for mounting a gimbal 10. A camera 4 or a mobile phone is mounted on the gimbal 10, allowing for free adjustment of the shooting angle. The sliding groove on motor arm 3 is 150mm long, allowing the camera 4 to be freely adjusted within a range of 250mm-400mm from the subject. The robot's phenotypic information acquisition device is entirely made of carbon fiber, effectively reducing weight while maintaining strength. Tests have shown that this structure effectively avoids shaking during movement. Motor 8 has an adjustable speed, with a rated speed of 5r / min, so it typically takes 12 seconds to capture a 360-degree shot around an object. For motion control, the robot uses an STM32 lower-level computer, a wheel odometer, and a driver to form a closed-loop control system. The robot is equipped with a Jetson Xavier as the upper-level controller, communicating with the lower-level computer via CAN communication to control the robot's automatic movement. The host computer controller can also process laser mapping data and issue specific walking commands, and receive information collected by the 3D phenotypic information acquisition device 2. The host computer's built-in WiFi module can transmit data to the backend. The data processed in the backend is automatically fed back to the user terminal.

[0153] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for in-situ three-dimensional phenotypic information acquisition of crops, characterized in that, The method is based on a crop in-situ three-dimensional phenotypic information acquisition robot; the robot includes: a mobile chassis, a lower-level computer, and a three-dimensional phenotypic information acquisition device, the three-dimensional phenotypic information acquisition device includes: a motor arm and a camera; the lower-level computer is connected to the motor arm and the upper-level computer respectively, the motor arm is mounted on the mobile chassis, and the camera is mounted on the motor arm; the upper-level computer is connected to the data processing backend; The process of implementing the method based on the robot includes: During the autonomous movement of the mobile chassis along the optimal path, the host computer acquires the original images captured by the camera in real time and determines whether an AprilTag exists based on the original images. The AprilTag is affixed to the side of the crop, and the information in the AprilTag includes: variety, plant number, and planting date. When the AprilTag appears, the pose of the camera relative to the AprilTag is determined, and based on the pose of the robot relative to the AprilTag, the pose of the robot is adjusted by the lower computer until the crop is in the center of the original image, after which the moving chassis stops moving. The host computer controls the motor arm to rotate one revolution through the slave computer, and acquires the original video captured by the camera during the rotation of the motor arm, and sends the original video to the data processing backend; The host computer controls the mobile chassis to continue its autonomous walking process and returns "The host computer acquires the original images captured by the camera in real time" until the original video corresponding to all crops in the target area is obtained; The data processing backend determines the three-dimensional phenotypic information of each crop based on the original video of each crop; the three-dimensional phenotypic information includes: stem diameter, projected leaf area, actual leaf area, leaf area density, leaf tilt angle and plant type phenotypic information; Determining the pose of the camera relative to the AprilTag includes: Obtain the image coordinates of the four corner points of AprilTag, as well as the camera's intrinsic parameter matrix and distortion parameters; Based on the intrinsic parameter matrix and the distortion parameters, the image coordinates are converted into camera coordinates; Using the PnP algorithm, the camera coordinates are determined based on the image coordinates of each corner point and the corresponding camera coordinates. Using the homography transformation matrix, the camera coordinates of each corner point are converted into the world coordinates of the corresponding corner point, and the camera coordinates of the camera are converted into the world coordinates of the camera. Based on the world coordinates of each corner point and the world coordinates of the camera, the pose of the camera relative to the AprilTag is determined; Based on the original videos corresponding to each crop, the three-dimensional phenotypic information of the corresponding crop is determined, including: Select any crop as the current crop; Collect multiple frames of original images from different perspectives from the original video corresponding to the current crop; Extract feature points from each original image of the current crop and perform feature matching to obtain the corresponding matched image; The matching images are reconstructed using the Neus network to obtain the corresponding 3D model; Based on the physical dimensions of AprilTag, assign absolute values ​​to the relative dimensions of the 3D model; The three-dimensional phenotypic information of the current crop is determined based on the relative size of the three-dimensional model.

2. The method for in-situ three-dimensional phenotypic information acquisition of crops according to claim 1, characterized in that, The process of determining the optimal path includes: Construct the velocity sampling space of the robot; From the starting point to the ending point, at each position in the velocity sampling space, multiple sampling velocity groups are determined at each position; the sampling velocity groups include: linear velocity and angular velocity; Based on the sampling rate groups at each location, multiple paths from the starting point to the ending point are determined; Calculate the score for each path, and determine the path with the highest score as the optimal path.

3. The method for in-situ three-dimensional phenotypic information acquisition of crops according to claim 2, characterized in that, The velocity sampling space is the union of velocity boundary constraints, acceleration constraints, and environmental obstacle constraints.

4. The method for in-situ three-dimensional phenotypic information acquisition of crops according to claim 3, characterized in that, The velocity boundary limit V m for: V m ={(v,ω)|v∈[v min ,v max ],ω∈[ω min ,ω max ]}; Where v is the linear velocity; v min The minimum linear velocity; v max ω is the maximum linear velocity; ω is the angular velocity; ω min ω is the minimum angular velocity; max This is the maximum angular velocity; The acceleration limit V a for: Among them, v c a is the linear velocity at the current moment; vmax ω represents the maximum linear acceleration; Δt represents the time interval; ω represents the maximum linear acceleration. c a is the angular velocity at the current moment; ωmax This is the maximum angular acceleration; The environmental obstacles limit V d for: Where dist(v,ω) is the shortest distance between the trajectory and the obstacle under linear velocity v and angular velocity ω; Velocity sampling space V s for: V s =V m ∩V a ∩V d ; Here, ∩ represents the union operation.

5. The method for in-situ three-dimensional phenotypic information acquisition of crops according to claim 1, characterized in that, The process of determining whether an AprilTag exists based on any original image includes: The original image is converted into a grayscale image, the grayscale image is segmented into multiple blocks, and the maximum and minimum grayscale values ​​of each block are determined. The maximum and minimum gray values ​​of each block are subjected to 3-neighborhood maximum and minimum filtering to obtain the filtered maximum and minimum gray values ​​of the corresponding blocks. The average of the maximum and minimum filtered gray values ​​of each block is determined as the threshold of the corresponding block. By utilizing continuous boundary segmentation and fast decoding, and based on the threshold of each block, it is determined whether the AprilTag exists in the original image.

6. The method for in-situ three-dimensional phenotypic information acquisition of crops according to claim 1, characterized in that, The robot also includes: an aluminum alloy frame and carbon fiber guide rails; The aluminum alloy bracket is mounted on the mobile chassis, the carbon fiber guide rail is mounted on the aluminum alloy bracket, and the three-dimensional phenotypic information acquisition device is connected to the carbon fiber guide rail.

7. The method for in-situ three-dimensional phenotypic information acquisition of crops according to claim 6, characterized in that, The robot also includes: a human-computer interaction display; The human-computer interaction display is mounted on the aluminum alloy bracket.

8. The method for in-situ three-dimensional phenotypic information acquisition of crops according to claim 6, characterized in that, The three-dimensional phenotypic information acquisition device also includes: a motor, a moving slider, and a gimbal; The motor is connected to the carbon fiber guide rail and the motor arm respectively. The carbon fiber guide rail and the motor arm are slidably connected through the movable slider. The camera is mounted on the motor arm through the gimbal.

Images