Crop canopy three-dimensional reconstruction method based on track generation type space information acquisition

Abstract

The invention discloses a crop canopy three-dimensional reconstruction method based on track generation type space information acquisition, relates to the technical field of plant three-dimensional reconstruction, can adapt to different user requirements and plant characteristics by setting a plurality of multi-view shooting track modes, breaks through the limitation of traditional horizontal surround shooting, and improves the reconstruction efficiency. Complete images of overlook, look-up and side middle and lower parts can be obtained, information loss of the top and the bottom of a three-dimensional model is made up, and the reconstruction effect of crops with complex structures, such as vine plants, is greatly improved. Meanwhile, coordinate discretization and inverse kinematics solution are carried out on a continuous track, the track is converted into an executable instruction of a mechanical arm, and an acquisition device is driven to move and shoot, so that plant shaking caused by rotation of a rotary table is avoided, non-rigid deformation in an image sequence is reduced, and the defects of ghosting, trailing, local data missing and the like in three-dimensional point cloud are effectively eliminated. And a more reliable three-dimensional data support is provided for crop phenotype analysis, pest and disease detection and the like.

Description

Technical Field

[0001] This application relates to the field of plant three-dimensional reconstruction technology, specifically to a method for three-dimensional reconstruction of crop canopy based on trajectory-generating spatial information acquisition. Background Technology

[0002] The three-dimensional morphological structure of plants is a true reflection of their growth and development. By combining image processing technology with 3D modeling to reconstruct 3D plant models, plant morphological characteristics can be more realistically represented. Plant phenotypic features play an important role in weed identification, pest and disease detection, and yield prediction, making image-based 3D plant modeling one of the most promising directions in plant 3D modeling. With the continuous development of machine vision technology, methods based on multi-view image reconstruction have achieved high accuracy in obtaining plant phenotypic data and have therefore been widely applied.

[0003] Current domestic research mainly focuses on using cameras fixed on a plant rotation platform for multi-view photography. For example, Chinese patent CN209247059U discloses a plant three-dimensional phenotypic information acquisition device. By using a rotation drive unit to drive the plant sample to rotate intermittently, a 3D camera and a visible light image camera can be used to acquire three-dimensional point cloud images and visible light color images of the plant sample during the rotation intervals, thereby obtaining the three-dimensional phenotypic information of the plant sample. Chinese patent CN109738441B discloses a plant phenotypic three-dimensional reconstruction information acquisition device and its control method. The plant rotation platform rotates 360°, thereby acquiring multiple 360-degree images of the plant cross-section and canopy for subsequent three-dimensional reconstruction of the plant phenotypic profile, without the need for complex processing of each image.

[0004] While the aforementioned patent documents achieve automation in constructing 3D models of plants, the perspective is limited to a horizontal, circling view, failing to capture top-down, bottom-up, or lower-middle-side images. This results in missing top and bottom information in the reconstructed 3D model, leading to poor reconstruction results for structurally complex plants (such as vines). Furthermore, plants (especially those with lush foliage and loose structures) are prone to swaying during turntable rotation, causing non-rigid deformation in the continuously captured image sequence. This ultimately produces defects such as ghosting, trailing, or localized data loss in the 3D point cloud. Summary of the Invention

[0005] In order to solve the above-mentioned technical problems, this application proposes the following technical solution: In a first aspect, embodiments of this application provide a method for three-dimensional reconstruction of crop canopy based on trajectory-generating spatial information acquisition, including: Initialize the crop canopy 3D reconstruction device system and calibrate the plant positions; Multiple multi-view shooting trajectory modes were determined, and different multi-view shooting trajectory modes are suitable for different user needs and plant characteristics; After selecting the multi-view shooting trajectory mode, the continuous trajectory determined by the multi-view shooting trajectory mode will be discretized and inverse kinematics will be solved to convert the continuous trajectory into joint action commands that can be executed by the robotic arm. The actuator control, status monitoring and data transmission of the crop canopy 3D reconstruction device are integrated to automatically acquire multi-view 2D images of crops containing metadata; A three-dimensional point cloud model of the crop is generated by combining the multi-view two-dimensional images of the crop with the corresponding shooting pose information. The three-dimensional point cloud model is used for three-dimensional reconstruction of the crop canopy.

[0006] In one possible implementation, the crop canopy three-dimensional reconstruction device includes: a support frame, a lifting mechanism, a rotating mechanism, a robotic arm, a data acquisition trajectory control unit, and a three-dimensional information acquisition and reconstruction unit. The bracket includes a base and a support column vertically fixed on the base; The lifting mechanism includes an electric push rod and a tray. The electric push rod is vertically mounted on the base. The drive end of the electric push rod is fixedly connected to the base, and the movable end is fixedly connected to the tray. The rotating mechanism is a gear assembly, and the robotic arm is a robotic arm connected to the gear assembly via a robotic arm connector. The acquisition trajectory control unit includes limit switches, laser rangefinders, encoders, control boxes, motor drive modules, relay modules, and power supply modules; The three-dimensional information acquisition and reconstruction unit includes a computing unit, a soft light plate, and an acquisition camera mounted on the robotic arm. The soft light plate is connected to the gear assembly via a soft light plate connector.

[0007] In one possible implementation, the gear assembly includes an external gear bearing, a drive gear meshing with the external gear bearing, the drive gear being fixedly connected to the rotational output end of a stepper motor, and a gear encoder meshing with the drive gear. The external gear bearing is disposed at the top of the support column adjacent to the tray. A robotic arm connector and a diffuser connector are symmetrically arranged on the external gear bearing. The external gear bearing supports the robotic arm and the diffuser respectively through the robotic arm connector and the diffuser connector. The rotation of the external gear bearing drives the camera to rotate 360° around the observed plant to capture images, ensuring that every part of the observed plant can be observed.

[0008] In one possible implementation, the initialization of the crop canopy three-dimensional reconstruction device system and the calibration of plant positions include: After the crop canopy three-dimensional reconstruction device is started, it drives the joints of the robotic arm to move to the preset safe initial position, controls the stepper motor to rotate slowly until the external gear bearing limit switch is triggered, records this position as the mechanical zero point of the rotation angle, completes the absolute position calibration of the chassis rotation, and reads the initial value of the laser rangefinder sensor and records it as the reference value of the tray height. The camera's optical center is aligned with the center of the plant. The angles of each joint servo and the chassis rotation angle are read at this moment. Combined with the forward kinematics model, the three-dimensional coordinates of the plant's center in the robot arm's base coordinate system are calculated and stored as the target observation point.

[0009] In one possible implementation, the multi-view shooting trajectory mode includes: a preset trajectory mode, a manually edited trajectory mode, and an occlusion-minimized trajectory generation mode; The preset trajectory mode includes a variety of standard trajectories, and automatically generates a complete sequence of trajectory points based on the plant center coordinates and set parameters; The manual trajectory editing mode allows users to directly draw or set key points in a virtual 3D space through a graphical interface, generating the robotic arm's motion trajectory based on the user-defined path. The occlusion minimization trajectory generation mode first controls the robotic arm to drive the camera to perform a rapid pre-scan of the plant, acquiring a set of preliminary images covering the entire plant. The complexity of the plant surface is assessed through image analysis to determine that for areas with dense foliage and severe occlusion, more shooting points are planned; for areas with sparse foliage and less occlusion, fewer shooting points are planned.

[0010] In one possible implementation, after selecting the multi-view shooting trajectory mode, the continuous trajectory determined by the multi-view shooting trajectory mode is discretized in coordinates and inverse kinematics is solved to convert the continuous trajectory into joint motion commands executable by the robotic arm, including: The generated continuous trajectory is discretized into a series of discrete spatial coordinate points. ; Subsequently, for each discrete trajectory point By using inverse kinematics calculations, the target rotation angles required for the servo motors of each joint of the robotic arm can be determined.

[0011] In one possible implementation, the step of discretizing the generated continuous trajectory into a series of discrete spatial coordinate points includes: For calculating the arc length of a continuous trajectory curve: in: , , The first derivative of the component function; From parameters arrive Arc length formula: Let the number of discrete points be N, then the arc length interval is: Where L is the total arc length; The target arc length point is: For each Solve the equation Get parameters The parameter sequence is obtained by iterative method. : The coordinates of the discrete trajectory point in the plant coordinate system are: .

[0012] In one possible implementation, the statement for each discrete trajectory point By using inverse kinematics calculations, the target rotation angles required for the servo motors of each joint of the robotic arm are determined, including: Transform the points in the plant coordinate system to the base coordinate system, with the origin at the base of the robotic arm and the X-axis pointing towards the center of the plant: in: The rotation angle of the external gear. For trajectory points x-coordinate For trajectory points The ordinate; The position of the robotic arm base in the plant coordinate system is: in: The radial distance from the base to the center of the plant; The coordinates of the trajectory point in the base coordinate system are: Based on the target point in the base coordinate system Solve for joint angles : Position constraint equations: in: The length of the robotic arm's upper arm. The length of the robotic arm's forearm. This is the length of the end effector link of the robotic arm; set up The visual pitch angle pointing towards the center of the plant is: The constraint equation for the camera pointing towards the center of the plant: in: Adjust the camera mounting angle; Verify whether the joint angle meets the limit. And whether the target point is within the reachable space of the robotic arm: Substitute the joint angles obtained from the solution into the formula to calculate the theoretical coordinates of the lens, and compare them with the trajectory planning coordinates to verify the accuracy of the solution.

[0013] In one possible implementation, the actuator control, status monitoring, and data transmission of the integrated crop canopy 3D reconstruction device, automatically acquiring multi-view 2D images of the crop containing metadata, includes: The electric linear actuator is controlled by a dual-channel relay module. The stepper motor is controlled by calculating the angle deviation based on the target angle and the current angle fed back by the rotary encoder. The stepper motor's step pulse frequency and direction are adjusted through a PID control algorithm to ensure that the actual angle reaches the target angle quickly and accurately. The rotary encoder monitors the position of the stepper motor in real time and feeds the position information back to the main controller. During the shooting process, the main controller monitors the height change of the tray in real time by reading the laser rangefinder. Following the planned trajectory, the robotic arm is controlled to move to each shooting point in sequence. At each point, after the robotic arm stabilizes, the main controller triggers the camera to take a picture and controls the fill light to light up synchronously. After the shooting is completed, multi-view two-dimensional images and corresponding robotic arm pose and tray height data are obtained.

[0014] In one possible implementation, the step of generating a 3D point cloud model of the crop by combining the multi-view 2D images of the crop with the corresponding shooting pose information, wherein the 3D point cloud model is used for 3D reconstruction of the crop canopy, including: After receiving multi-view 2D images of the crop and their corresponding shooting pose information, a depth map is generated for each image based on the pose information through multi-view stereo matching, with depth values... Obtained by minimizing the photometric consistency error: in: For depth values Find the minimum value. For robust loss function, These are the weighting coefficients. Represents pixels in the reference image Photometric value, For the first Image luminance values ​​for each view This is a camera projection function that projects points in 3D space onto a 2D image plane to obtain the corresponding pixel coordinates. For the camera intrinsic parameter matrix, For the first The extrinsic parameter matrix of each camera, For the first The translation vector of each camera. For pixels homogeneous coordinates; Multiple depth maps are merged into a unified voxel representation: in: for In the Weights under each view This indicates a lower truncation operation that limits the lower bound of the normalized symbol distance to -1. The truncation operation restricts the upper bound of the normalized symbol distance to 1. for In the Projected coordinates under each view For the first Depth value under each view For the camera optical center, for To the The Euclidean distance of the optical center of each camera The truncation distance threshold is used to limit the maximum effective range of symbolic distances. Distances exceeding this range will be truncated to avoid extreme values ​​interfering with the fusion results. A 3D point cloud model is obtained by extracting isosurfaces from the TSDF field: in: It is a three-dimensional point grid space. for The absolute value of the distance to the nearest surface. This is the distance threshold, used to extract the cutoff threshold of the surface, retaining only points that are sufficiently close to the surface; The three-dimensional point cloud model is filtered and smoothed to remove noise and optimize surface quality to obtain the final three-dimensional point cloud model of the crop.

[0015] In this embodiment, by setting multiple multi-view shooting trajectory modes, it can adapt to different user needs and plant characteristics, breaking the limitations of traditional horizontal surround shooting. It can acquire complete images from top, bottom, and lower side views, compensating for the lack of information at the top and bottom of the 3D model, and significantly improving the reconstruction effect of structurally complex crops such as vines. Simultaneously, by discretizing the continuous trajectory and solving its inverse kinematics, the trajectory is converted into executable commands for the robotic arm, driving the acquisition device to move and capture images. This avoids plant swaying caused by turntable rotation, reduces non-rigid deformation in the image sequence, and effectively eliminates defects such as ghosting, trailing, and local data loss in the 3D point cloud. Furthermore, the system integrates control, monitoring, and transmission functions to achieve automated image acquisition, improving acquisition efficiency and data accuracy, and providing more reliable 3D data support for crop phenotypic analysis and pest and disease detection. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a method for three-dimensional reconstruction of crop canopy based on trajectory-generating spatial information acquisition, provided in an embodiment of this application; Figure 2 This is a schematic diagram of a three-dimensional reconstruction device for crop canopy using trajectory generation-based spatial information acquisition, provided in an embodiment of this application. Figure 3 A schematic diagram of the robotic arm connector and the diffuser connector provided in the embodiments of this application; Figure 4 This is a schematic diagram of the encoder installation position provided in an embodiment of this application; Figure 5 This is a schematic diagram of the installation position of the limit switch provided in an embodiment of this application; Figure 6 A schematic diagram of a spiral trajectory in a preset trajectory mode of a multi-view shooting trajectory mode provided in an embodiment of this application; Figure 7 A schematic diagram of a user-defined trajectory in the manually edited trajectory mode of the multi-view shooting trajectory mode provided in the embodiments of this application; Figure 8 This is a schematic diagram of trajectory generation with minimal occlusion in the multi-view shooting trajectory mode provided in the embodiments of this application; Figure 1-8 In Chinese, the symbol is represented as: 1-Base, 2-Support column, 3-Electric push rod, 4-Pattern, 5-External gear bearing, 6-Drive gear, 7-Stepper motor, 8-Gear encoder, 9-Robotic arm connector, 10-Soft light plate connector, 11-Robotic arm, 12-Soft light plate, 13-Camera, 14-Large arm, 15-Small arm, 16-End effector, 17-Third servo motor, 18-Second servo motor, 19-First servo motor, 20-Limit switch, 21-Laser rangefinder sensor, 22-Control box, 23-Stepper motor driver, 24-Computer. Detailed Implementation

[0017] The present solution will now be described in conjunction with the accompanying drawings and specific embodiments.

[0018] See Figure 1 The crop canopy three-dimensional reconstruction method based on trajectory generation spatial information acquisition provided in this embodiment includes: S101, Initialize the crop canopy three-dimensional reconstruction device system and calibrate the plant positions.

[0019] See Figure 2 The crop canopy three-dimensional reconstruction device includes: a support frame, a lifting mechanism, a rotating mechanism, a robotic arm 11, a data acquisition trajectory control unit, and a three-dimensional information acquisition and reconstruction unit.

[0020] The bracket includes a base 1 and a support column 2 vertically fixed on the base 1. The lifting mechanism includes an electric push rod 3 and a tray 4. The base 1 is a circular rigid structure used to fix the support column 2 and the electric push rod 3. The support column 2 is used to connect the circular base 1 to the rotating mechanism and to support the rotating mechanism.

[0021] The electric push rod 3 is vertically fixed to the circular base 1 and is used to connect and support the tray 4. The height of the plant on the tray 4 can be adjusted by controlling the electric push rod 3. The circular tray 4 is fixed to the movable end of the top of the electric push rod 3. The tray 4 is used to carry and place the plant and can move vertically up and down relative to a set horizontal reference plane under the drive of the electric push rod 3.

[0022] The rotating mechanism is a gear assembly, and the robotic arm 11 is a multi-axis robotic arm 11. The robotic arm 11 is connected to the gear assembly via a robotic arm connector 9.

[0023] See Figure 3 and Figure 4The gear assembly includes an external gear bearing 5, a drive gear 6 meshing with the external gear bearing 5, the drive gear 6 being fixedly connected to the rotation output end of a stepper motor 7, and a gear encoder 8 meshing with the drive gear 6. The external gear bearing 5 is positioned at the top of the support column 2, adjacent to the tray 4. A robotic arm connector 9 and a diffuser connector 10 are symmetrically arranged on the external gear bearing 5. The external gear bearing 5 supports the robotic arm 11 and the diffuser 12 respectively through the robotic arm connector 9 and the diffuser connector 10. The rotation of the external gear bearing 5 drives the camera 13 to rotate 360° around the observed plant, ensuring that every part of the observed plant can be observed.

[0024] The robotic arm 11 includes a large arm 14, a small arm 15, and an end effector 16. One end of the large arm 14 is connected to a servo motor spool and is driven by a third servo motor 17 for pitch movement. The other end of the large arm 14 is connected to a servo motor 18 for pitch movement. One end of the small arm 15 is bolted to the body of the second servo motor 18 and is driven by the second servo motor 18 for pitch movement. The end effector 16 serves as a mounting base for the camera 13, hinged to the end of the small arm 15, and is driven by a first servo motor 19 for pitch rotation, used to adjust the shooting angle of the camera 13. The robotic arm 11 is fixed to the robotic arm connector 9 via the third servo motor 17. Then, an embedded controller is connected to the camera 13 and the robotic arm 11 via signal lines, thereby achieving integrated control of the pitch movement of the robotic arm 11 and the recognition and positioning of the camera 13.

[0025] The acquisition trajectory control unit includes a limit switch 20, a laser rangefinder 21, a gear encoder 8, a control box 22, a motor drive module, a relay module, and a power supply module.

[0026] join Figure 5 The limit switch 20 is a push-button limit switch, which is fixedly installed on the circular plate below the external gear bearing 5 with bolts. A bolt is installed in a threaded hole of the external gear bearing 5. The length of the bolt is 5 mm longer than the thickness of the gear, so that the mechanical limit switch 20 can be pressed when rotating. This is used to detect the specific position of the external gear bearing 5 and to determine the rotation angle of the robotic arm 11.

[0027] The laser rangefinder 21 is fixedly mounted on the circular base 1. Its laser beam points vertically upward toward the bottom of the tray 4, measures the height of the tray 4 relative to the circular base 1, and feeds the height data back to the main controller.

[0028] The encoder is mounted on the same plane as the drive gear 6 and meshes with it, providing real-time, high-precision feedback on the motor's angular displacement and speed. The control box 22, placed on the circular base 1, is responsible for receiving all signals from the trajectory control unit and for the overall system logic control. The motor drive module includes a servo driver and a stepper motor driver 23, which drive the joint servos of the robotic arm 11 and the rotary motor of the external gear bearing 5, respectively. The relay module is connected to the main controller, controlling the push rod by controlling the on / off combinations of the relays. The power supply module includes a main power supply and a DC-DC step-down module. The main power supply provides the total input power to the system, while the DC-DC step-down module steps down the main power supply to the operating voltage required by each component of the system.

[0029] The 3D information acquisition and reconstruction system includes a computing unit, a camera 13, and a diffuser 12. The computing unit, a high-performance computer 24, receives image sequences with precise pose information and runs algorithms to reconstruct a 3D model from a 2D image. The camera 13 is fixedly mounted on the end effector 16, with its optical axis perpendicular to the rotation axis of the end effector 16, and is used to acquire 2D images of the plant. The diffuser 12 is mounted on a diffuser connector 10 and is used to provide uniform illumination, eliminate shadows, and improve image quality to facilitate 3D reconstruction.

[0030] In this embodiment, after the main controller is powered on, it performs a power self-test to confirm that the voltage output of each part is normal. Then, it drives each joint of the robotic arm 11 to move to the preset safe initial position. It controls the stepper motor 7 to rotate slowly until the push-button limit switch 20 is triggered, and records this position as the mechanical zero point of the rotation angle, completing the absolute position calibration of the chassis rotation. At the same time, it reads the initial value of the laser rangefinder sensor 21 and records it as the reference value of the height of the tray 4.

[0031] In manual control mode, the optical center of camera 13 is aligned with the center of the plant. The main controller reads the current angles of each joint servo motor and the chassis rotation angle, and, combined with the forward kinematics model, calculates the three-dimensional coordinates (Xp, Yp, Zp) of the plant center in the base coordinate system of robotic arm 11, and stores it as the target observation point.

[0032] S102 determines multiple multi-view shooting trajectory modes, and different multi-view shooting trajectory modes are suitable for different user needs and plant characteristics.

[0033] In this embodiment, three multi-view shooting trajectory modes are provided, which users can choose according to their needs: Preset trajectory modes: The system includes a variety of standard trajectories (such as horizontal circles, multi-height layer scanning, cylindrical spirals, etc.) for users to choose from. After the user selects a trajectory, the system automatically generates a complete sequence of trajectory points based on the plant center coordinates and the set parameters (radius, height, number of sampling points, etc.).

[0034] Manual trajectory editing mode: Users can draw or set key points directly in virtual three-dimensional space through the graphical interface on the computer 24, and the system will generate the motion trajectory of the robotic arm 11 according to the user-defined path.

[0035] Occlusion minimization trajectory generation mode: The system first controls the robotic arm 11 to drive the camera 13 to perform a rapid pre-scan of the plant, acquiring a set of preliminary images covering the entire plant. The computer 24 evaluates the complexity of the plant surface through image analysis algorithms (such as depth estimation, occlusion detection, and leaf density analysis): for areas with dense branches and leaves and severe occlusion, the algorithm automatically plans denser shooting points; for areas with sparse branches and leaves and less occlusion, the number of shooting points is appropriately reduced.

[0036] S103, after selecting the multi-view shooting trajectory mode, the continuous trajectory determined by the multi-view shooting trajectory mode will be discretized in coordinates and solved inverse kinematics, and the continuous trajectory will be converted into joint action commands that can be executed by the robotic arm.

[0037] A schematic diagram of the 3D trajectory view of the plant 3D reconstruction acquisition trajectory in the preset trajectory mode is shown below. Figure 6 A schematic diagram of the 3D trajectory view of the plant 3D reconstruction acquisition trajectory in the manually edited trajectory mode is shown below. Figure 7 A schematic diagram of the 3D trajectory view of the plant 3D reconstruction acquisition trajectory using the occlusion minimization generation trajectory mode is shown below. Figure 8 The three diagrams above all use the base of the plant as the display coordinate system, with the coordinate axes labeled... In three dimensions, and with the associated sampling point indices showing the overall trajectory change, among which... Figure 6 , Figure 7 The focus is on presenting the changing patterns of camera height along the trajectory. Figure 8 In addition to displaying the changes in camera height, it also presents the dynamic changes in the distance from the camera to the object, which can intuitively reflect the density distribution characteristics of the shooting points planned by the algorithm for different areas of the plant.

[0038] In this embodiment, the system performs trajectory coordinate discretization processing on the generated continuous trajectory, converting it into a series of discrete spatial coordinate point sequences. Subsequently, for each discrete trajectory point... The system calculates the target rotation angle required for each joint servo motor of the robotic arm 11 through inverse kinematics calculations.

[0039] The system uses a discrete method with equal arc length spatial parameters to process the planned trajectory. The core of this method is that the arc lengths of adjacent trajectory points are equal after discretization, thus achieving uniform speed characteristics when driving the robotic arm 11.

[0040] For the arc length unit of the spatial parameter curve calculation trajectory: in , , The first derivative of the component function. These are the three coordinate components in the plant coordinate system.

[0041] From parameters to the formula for arc length u: Let the number of discrete points be N, then the arc length interval is: Where: L is the total arc length.

[0042] The target arc length point is: in: For discrete point indexes.

[0043] For each Solve the equation Get parameters The parameter sequence is obtained using Newton's iteration method. : The coordinates of the discrete trajectory point in the plant coordinate system are: Subsequently, the system performs inverse kinematics calculations for each discrete trajectory point to determine the target angles of each joint required for the robotic arm 11 to reach that pose. The solution steps are as follows: Coordinate system transformation: Transform the points in the plant coordinate system to the base 1 coordinate system (origin at base 1 of robotic arm 11, X-axis pointing to the center of the plant). Let the radial distance from base 1 to the center of the plant be... The external gear rotates at an angle of 100°. .

[0044] in: For trajectory points x-coordinate For trajectory points The ordinate.

[0045] The position of the base 1 of the robotic arm 11 in the plant coordinate system is: The coordinates of the trajectory point in the coordinate system of base 1 are: Inverse kinematics solution: Given the target point in the coordinate system of base 1 Solve for joint angles .

[0046] Position constraint equations: in: The upper arm is 14 cm long. Forearm length 15, This is the length of the end link.

[0047] set up The visual pitch angle pointing towards the center of the plant is: The constraint equation for camera 13 pointing to the center of the plant: in: Install the offset angle for camera 13.

[0048] Constraint check: Verify whether the joint angle meets the limit. And whether the target point is within the reachable space of the robotic arm 11: Forward kinematics verification: The main controller substitutes the joint angles obtained from the solution into the formula, calculates the theoretical coordinates of the lens, and compares them with the trajectory planning coordinates to verify the accuracy of the solution.

[0049] S104, integrates the actuator control, status monitoring and data transmission of the crop canopy three-dimensional reconstruction device to automatically acquire multi-view two-dimensional images of crops containing metadata.

[0050] The electric linear actuator 3 is controlled via a dual-channel relay module. When the actuator needs to extend, the ESP32 controls the first channel of the relay module to close and the second channel to open, so that the positive terminal of the electric linear actuator 3 motor is connected to the positive terminal of the power supply and the negative terminal is connected to the negative terminal of the power supply, thus extending the actuator. When the actuator needs to retract, the ESP32 controls the first channel of the relay module to open and the second channel to close, so that the positive terminal of the electric linear actuator 3 is connected to the negative terminal of the power supply and the negative terminal is connected to the positive terminal of the power supply, thus retracting the actuator.

[0051] The controller calculates the angle deviation based on the target angle and the current angle fed back by the rotary encoder. It then adjusts the stepping pulse frequency and direction of the stepper motor 7 using a PID control algorithm, ensuring the actual angle reaches the target angle quickly and accurately. The rotary encoder monitors the position of the stepper motor 7 in real time and feeds the position information back to the main controller.

[0052] During the shooting process, the main controller monitors the height changes of the tray 4 in real time by reading the laser rangefinder 21. Following a pre-planned trajectory, the controller sequentially moves the robotic arm 11 to each shooting point. At each point, after the robotic arm 11 stabilizes, the main controller triggers the camera 13 to take a picture and simultaneously controls the fill light to illuminate. After shooting is completed, the image file and corresponding metadata such as the pose of the robotic arm 11 and the height of the tray 4 are transmitted to the computer 24 via a wireless network.

[0053] S105, after combining the crop multi-view two-dimensional images and the corresponding shooting pose information, a three-dimensional point cloud model of the crop is generated, and the three-dimensional point cloud model is used for three-dimensional reconstruction of the crop canopy.

[0054] After receiving the multi-view two-dimensional image set, the corresponding precise camera pose information, and the height of the tray 4, the computer 24 uses a multi-view stereo vision (MVS) algorithm based on depth map fusion with known camera pose to reconstruct the point cloud.

[0055] The specific process is as follows: Based on the known camera pose, a depth map is generated for each image using multi-view stereo matching. Depth values. Obtained by minimizing the photometric consistency error: in: For depth values Find the minimum value. For robust loss function, These are the weighting coefficients. Represents pixels in the reference image Photometric value, For the first Image luminance values ​​for each view This is a camera projection function that projects points in 3D space onto a 2D image plane to obtain the corresponding pixel coordinates. For the camera intrinsic parameter matrix, For the first The extrinsic parameter matrix of each camera, For the first The translation vector of each camera. For pixels Homogeneous coordinates.

[0056] Multiple depth maps are merged into a unified voxel representation: in: for In the Weights under each view This indicates a lower truncation operation that limits the lower bound of the normalized symbol distance to -1. The truncation operation restricts the upper bound of the normalized symbol distance to 1. for In the Projected coordinates under each view For the first Depth value under each view For the camera optical center, for To the The Euclidean distance of the optical center of each camera The truncation distance threshold is used to limit the maximum effective range of symbolic distances. Distances exceeding this range will be truncated to avoid extreme values ​​interfering with the fusion results.

[0057] Extracting isosurfaces from a TSDF field ( ) Obtain the 3D point cloud: in: It is a three-dimensional point grid space. for The absolute value of the distance to the nearest surface. This is the distance threshold, used to extract the cutoff threshold of the surface, retaining only points that are sufficiently close to the surface.

[0058] The point cloud is post-processed with filtering and smoothing to remove noise and optimize surface quality. Poisson reconstruction is then performed using the optimized 3D point cloud. Specifically, an octree is constructed using an adaptive spatial mesh generation method. A node function is set for each node of the octree. The function space is composed of Open, basis functions An n-dimensional convolution with box filtering was used. The vector field was then calculated. Vector field It can be accurately and efficiently represented as a node function. The linear sum of the terms is obtained. The Poisson equation is solved iteratively using the Laplace matrix. A specific threshold is calculated to obtain an isosurface, and the MarchingCubes algorithm is used to obtain a triangular mesh.

[0059] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0060] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A crop canopy three-dimensional reconstruction method based on trajectory generative spatial information acquisition, characterized in that, include: Initialize the crop canopy 3D reconstruction device system and calibrate the plant positions; Multiple multi-view shooting trajectory modes were determined, and different multi-view shooting trajectory modes are suitable for different user needs and plant characteristics; After selecting the multi-view shooting trajectory mode, the continuous trajectory determined by the multi-view shooting trajectory mode will be discretized and inverse kinematics will be solved to convert the continuous trajectory into joint action commands that can be executed by the robotic arm. The actuator control, status monitoring and data transmission of the crop canopy 3D reconstruction device are integrated to automatically acquire multi-view 2D images of crops containing metadata; A three-dimensional point cloud model of the crop is generated by combining the multi-view two-dimensional images of the crop with the corresponding shooting pose information. The three-dimensional point cloud model is used for three-dimensional reconstruction of the crop canopy. 2.The crop canopy three-dimensional reconstruction method based on trajectory generative spatial information collection according to claim 1, wherein, The crop canopy three-dimensional reconstruction device includes: a support frame, a lifting mechanism, a rotating mechanism, a robotic arm, a data acquisition trajectory control unit, and a three-dimensional information acquisition and reconstruction unit; The bracket includes a base and a support column vertically fixed on the base; The lifting mechanism includes an electric push rod and a tray. The electric push rod is vertically mounted on the base. The drive end of the electric push rod is fixedly connected to the base, and the movable end is fixedly connected to the tray. The rotating mechanism is a gear assembly, and the robotic arm is a robotic arm connected to the gear assembly via a robotic arm connector. The acquisition trajectory control unit includes limit switches, laser rangefinders, encoders, control boxes, motor drive modules, relay modules, and power supply modules; The three-dimensional information acquisition and reconstruction unit includes a computing unit, a soft light plate, and an acquisition camera mounted on the robotic arm. The soft light plate is connected to the gear assembly via a soft light plate connector. 3.The crop canopy three-dimensional reconstruction method based on trajectory generative spatial information collection of claim 2, wherein, The gear assembly includes an external gear bearing, a drive gear meshing with the external gear bearing, the drive gear being fixedly connected to the rotation output end of a stepper motor, and a gear encoder meshing with the drive gear. The external gear bearing is disposed at the top of the support column adjacent to the tray. A robotic arm connector and a diffuser connector are symmetrically arranged on the external gear bearing. The external gear bearing supports the robotic arm and the diffuser respectively through the robotic arm connector and the diffuser connector. The rotation of the external gear bearing drives the camera to rotate 360° around the observed plant to capture images, ensuring that every part of the observed plant can be observed.

4. The crop canopy three-dimensional reconstruction method based on trajectory generative spatial information acquisition according to claim 3, characterized in that, The initialization of the crop canopy three-dimensional reconstruction device system and the calibration of plant positions include: After the crop canopy three-dimensional reconstruction device is started, it drives the joints of the robotic arm to move to the preset safe initial position, controls the stepper motor to rotate slowly until the external gear bearing limit switch is triggered, records this position as the mechanical zero point of the rotation angle, completes the absolute position calibration of the chassis rotation, and reads the initial value of the laser rangefinder sensor and records it as the reference value of the tray height. The camera's optical center is aligned with the center of the plant. The angles of each joint servo and the chassis rotation angle are read at this moment. Combined with the forward kinematics model, the three-dimensional coordinates of the plant's center in the robot arm's base coordinate system are calculated and stored as the target observation point.

5. The crop canopy three-dimensional reconstruction method based on trajectory generative spatial information acquisition according to claim 4, characterized in that, The multi-view shooting trajectory modes include: preset trajectory mode, manually edited trajectory mode, and trajectory generation mode with minimal occlusion. The preset trajectory mode includes a variety of standard trajectories, and automatically generates a complete sequence of trajectory points based on the plant center coordinates and set parameters; The manual trajectory editing mode allows users to directly draw or set key points in a virtual 3D space through a graphical interface, generating the robotic arm's motion trajectory based on the user-defined path. The occlusion minimization trajectory generation mode first controls the robotic arm to drive the camera to perform a rapid pre-scan of the plant, acquiring a set of preliminary images covering the entire plant. The complexity of the plant surface is assessed through image analysis to determine that for areas with dense foliage and severe occlusion, more shooting points are planned; for areas with sparse foliage and less occlusion, fewer shooting points are planned.

6. The method for three-dimensional reconstruction of crop canopy based on trajectory-generating spatial information acquisition according to claim 5, characterized in that, After selecting the multi-view shooting trajectory mode, the continuous trajectory determined by the multi-view shooting trajectory mode will be discretized and inverse kinematics solved to convert the continuous trajectory into joint motion commands that the robotic arm can execute, including: The generated continuous trajectory is discretized into a series of discrete spatial coordinate points. ; Subsequently, for each discrete trajectory point By using inverse kinematics calculations, the target rotation angles required for the servo motors of each joint of the robotic arm can be determined.

7. The method for three-dimensional reconstruction of crop canopy based on trajectory-generating spatial information acquisition according to claim 6, characterized in that, The step of discretizing the generated continuous trajectory into a series of discrete spatial coordinate points includes: For calculating the arc length of a continuous trajectory curve: in: , , The first derivative of the component function; From parameters arrive Arc length formula: Let the number of discrete points be N, then the arc length interval is: Where L is the total arc length; The target arc length point is: For each Solve the equation Get parameters The parameter sequence is obtained by iterative method. : The coordinates of the discrete trajectory point in the plant coordinate system are: 。 8. The method for three-dimensional reconstruction of crop canopy based on trajectory generation spatial information acquisition according to claim 7, characterized in that, For each discrete trajectory point By using inverse kinematics calculations, the target rotation angles required for the servo motors of each joint of the robotic arm are determined, including: Transform the points in the plant coordinate system to the base coordinate system, with the origin at the base of the robotic arm and the X-axis pointing towards the center of the plant: in: The rotation angle of the external gear. For trajectory points x-coordinate For trajectory points The ordinate; The position of the robotic arm base in the plant coordinate system is: in: The radial distance from the base to the center of the plant; The coordinates of the trajectory point in the base coordinate system are: Based on the target point in the base coordinate system Solve for joint angles : Position constraint equations: in: The length of the robotic arm's upper arm. The length of the robotic arm's forearm. This refers to the length of the end effector link of the robotic arm; set up The visual pitch angle pointing towards the center of the plant is: The constraint equation for the camera pointing towards the center of the plant: in: Adjust the camera mounting angle; Verify whether the joint angle meets the limit. And whether the target point is within the reachable space of the robotic arm: Substitute the joint angles obtained from the solution into the formula to calculate the theoretical coordinates of the lens, and compare them with the trajectory planning coordinates to verify the accuracy of the solution.

9. The method for three-dimensional reconstruction of crop canopy based on trajectory generation spatial information acquisition according to claim 3, characterized in that, The integrated crop canopy 3D reconstruction device controls the actuators, monitors the status, and transmits data to automatically acquire multi-view 2D images of the crop containing metadata, including: The electric linear actuator is controlled by a dual-channel relay module. The stepper motor is controlled by calculating the angle deviation based on the target angle and the current angle fed back by the rotary encoder. The stepper motor's step pulse frequency and direction are adjusted through a PID control algorithm to ensure that the actual angle reaches the target angle quickly and accurately. The rotary encoder monitors the position of the stepper motor in real time and feeds the position information back to the main controller. During the shooting process, the main controller monitors the height change of the tray in real time by reading the laser rangefinder. Following the planned trajectory, the robotic arm is controlled to move to each shooting point in sequence. At each point, after the robotic arm stabilizes, the main controller triggers the camera to take a picture and controls the fill light to light up synchronously. After the shooting is completed, multi-view two-dimensional images and corresponding robotic arm pose and tray height data are obtained.

10. The method for three-dimensional reconstruction of crop canopy based on trajectory-generating spatial information acquisition according to claim 9, characterized in that, The process of generating a 3D point cloud model of the crop by combining the multi-view 2D images of the crop with the corresponding shooting pose information, wherein the 3D point cloud model is used for 3D reconstruction of the crop canopy, including: After receiving multi-view 2D images of the crop and their corresponding shooting pose information, a depth map is generated for each image based on the pose information through multi-view stereo matching, with depth values... Obtained by minimizing the photometric consistency error: in: For depth values Find the minimum value. For robust loss function, These are the weighting coefficients. Represents pixels in the reference image Photometric value, For the first Image luminance values ​​for each view This is a camera projection function that projects points in 3D space onto a 2D image plane to obtain the corresponding pixel coordinates. For the camera intrinsic parameter matrix, For the first The extrinsic parameter matrix of each camera, For the first The translation vector of each camera. For pixels homogeneous coordinates; Multiple depth maps are merged into a unified voxel representation: in: for In the Weights under each view This indicates a lower truncation operation that limits the lower bound of the normalized symbol distance to -1. The truncation operation restricts the upper bound of the normalized symbol distance to 1. for In the Projected coordinates under each view For the first Depth value under each view For the camera optical center, for To the The Euclidean distance of the optical center of each camera The truncation distance threshold is used to limit the maximum effective range of symbolic distances. Distances exceeding this range will be truncated to avoid extreme values ​​interfering with the fusion results. A 3D point cloud model is obtained by extracting isosurfaces from the TSDF field: in: It is a three-dimensional point grid space. for The absolute value of the distance to the nearest surface. This is the distance threshold, used to extract the cutoff threshold of the surface, retaining only points that are sufficiently close to the surface; The three-dimensional point cloud model is filtered and smoothed to remove noise and optimize surface quality to obtain the final three-dimensional point cloud model of the crop.

Images