A multi-scenario self-propelled high-throughput non-destructive crop phenotype acquisition equipment

Abstract

The present invention discloses a multi-scenario self-propelled crop phenotype high-throughput non-destructive acquisition equipment, which belongs to the field of crop phenotype detection. The equipment includes a mobile robot platform, a satellite positioning module, a three-dimensional visual sensor, an edge computing device and a multi-sensor phenotypic data acquisition unit. The mobile robot platform adopts a four-wheel independent drive and independent steering mode, and the height is adjustable; the satellite positioning module controls the mobile robot platform to move to the target plot, and the edge computing device dynamically adjusts the steering angle of the mobile robot platform and the upper platform height according to the plant canopy depth image and three-dimensional point cloud information collected by the three-dimensional visual sensor; the edge computing device analyzes each modal phenotypic sensor data collected by the multi-sensor phenotypic data acquisition unit to obtain a variety of phenotypic data; it can also realize the automatic mapping of multi-modal phenotypic sensor data and target plot number, as well as line break operations. The present invention realizes multi-scenario self-propelled crop full growth period phenotypic high-throughput non-destructive measurement.

Description

Technical Field

[0001] The present invention relates to the field of crop phenotype detection, and in particular to a multi-scenario self-propelled crop phenotype high-throughput non-destructive acquisition device. Background Art

[0002] Realizing high-throughput and accurate acquisition of crop phenotypic information is a key step in improving the scale and efficiency of crop breeding, and is also an important research content in the current crop breeding direction. When acquiring crop information, due to the differences in indoor and field crop growth environments, the growth forms and parameters to be tested of different crops are different, and crops in different growth stages all have testing needs. Research on high-throughput, accurate, rapid, and non-destructive crop phenotypic detection devices and methods that can adapt to different crops, different testing environments (indoor, field), and different growth stages has become a hot topic and difficulty.

[0003] In this context, high-throughput phenotypic detection technology devices for different crops have emerged at home and abroad, which can be divided into two categories. One is the indoor phenotypic detection device, which is relatively mature in research at home and abroad. Typical examples include the indoor phenotypic detection platform developed by LemnaTec in Germany and the fully automatic non-destructive measurement system for phenotypic parameters of potted rice developed by Huazhong University of Science and Technology and Huazhong Agricultural University in China, which can achieve high-throughput phenotypic measurement of different crops. However, these platforms are fixed indoors, expensive, and the objects they process are potted crops. The other type is a field phenotypic detection device, such as the Chinese invention patent application “Device and method for active field high-throughput measurement of crop phenotypes” with publication number CN105547152A, which places the imaging part on the experimental field through a truss, and uses a manipulator to push the crops around the detection object to both sides before measurement. The truss is inconvenient to move in the field and has a complex structure; the Chinese invention patent “Field crop phenotypic monitoring robot” with publication number CN109466655A proposes a three-wheeled field mobile platform, but does not describe the phenotypic detection sensors and measurement methods that it can carry; the phenotyping robot proposed in the Chinese invention patent “A field-based high-throughput crop phenotyping robot and its control system” with publication number CN110006424A cannot walk across seedlings, and a special driving area needs to be reserved during the detection process, and the influence of the natural environment on the detection results cannot be eliminated. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-scenario self-propelled crop phenotype high-throughput non-destructive acquisition equipment, which can realize multi-scenario self-propelled crop phenotype high-throughput non-destructive measurement of the whole growth period.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A multi-scenario self-propelled crop phenotype high-throughput non-destructive acquisition equipment, the equipment comprising: a mobile robot platform, a satellite positioning module, a three-dimensional vision sensor, an edge computing device and a multi-sensor phenotype data acquisition unit.

[0007] The mobile robot platform adopts four-wheel independent drive and independent steering, and its height is adjustable; the three-dimensional vision sensor and the multi-sensor phenotypic data acquisition unit are both arranged on the upper platform of the mobile robot platform.

[0008] The satellite positioning module is used to obtain the starting position of the mobile robot platform; the edge computing device is used to plan a path according to the starting position and the position of the target plot, and control the mobile robot platform to move to the target plot according to the planned path.

[0009] When the four wheels of the mobile robot platform move between the crop rows of the target plot, the three-dimensional vision sensor is used to collect the depth image and three-dimensional point cloud information of the plant canopy in the visual area in real time from above the plants; the multi-sensor phenotypic data acquisition unit is used to collect the multimodal phenotypic sensor data of the plant canopy of the currently detected crop row in the visual area in real time from above the plants.

[0010] The edge computing device is used to extract crop row information within the visual area from the plant canopy depth image, and dynamically adjust the steering angle of the mobile robot platform according to the crop row information, so that the plant canopy of the currently detected crop row in the visual area is always at the center of the mobile robot platform.

[0011] The edge computing device is also used to calculate in real time the average distance from crops in the visual area to the three-dimensional vision sensor based on the depth information in the plant canopy depth image, and dynamically adjust the height of the upper platform of the mobile robot platform based on the average distance to keep the distance from the multi-sensor phenotyping data acquisition unit to the plant canopy unchanged.

[0012] Edge computing devices are also used to analyze each modality of phenotypic sensing data to obtain multiple phenotypic data of the canopy of the current crop row plant being inspected.

[0013] The edge computing device is used to dynamically identify the target plot number in the material label inserted on the target plot from the RGB image of the plant canopy depth image, and automatically map the multimodal phenotypic sensor data with the target plot number.

[0014] The edge computing device is also used to calculate the row spacing based on the crop row information extracted in the visual area. When the three-dimensional point cloud information collected by the three-dimensional vision sensor gradually decreases to disappear, it is determined that the end of the row has been reached. The average value of all row spacings calculated for the current detected crop row is used as the row spacing value for the line break. The mobile robot platform is controlled to turn vertically and then move laterally, and the lateral movement distance is made equal to the row spacing value for the line break, so as to align with the head of the next row of crops and complete the line break.

[0015] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0016] The mobile robot platform of the embodiment of the present invention adopts a four-wheel independent drive and independent steering mode, which can be simultaneously applicable to the measurement needs of different crop phenotypic detection parameters in field and indoor environments, and has adjustable height, suitable for a variety of crops with different growth heights in different growth periods; a satellite and three-dimensional vision fusion navigation method is adopted to realize unmanned autonomous walking of the mobile robot platform throughout the growth period; a multi-sensor phenotypic data acquisition unit collects multi-modal phenotypic sensor data from above the plant in real time, and can automatically map the multi-modal phenotypic sensor data with the target plot number, which improves the detection throughput and efficiency compared with the traditional method; the height of the mobile robot platform is adjustable, so that the multi-sensor phenotypic data acquisition unit has a certain distance from the canopy of the detection object, ensuring the stability and consistency of the collected phenotypic traits. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is a schematic diagram of the structure of a multi-scenario self-propelled high-throughput non-destructive acquisition equipment for crop phenotypes provided in Example 1 of the present invention.

[0019] Explanation of symbols:

[0020] Mobile robot platform-1, satellite positioning module-2, electric telescopic rod-3, multiple phenotypic sensors-4, multi-sensor phenotypic data acquisition unit-5, three-dimensional visual sensor-6, multi-modal phenotypic data analysis unit-7, edge computing device-8. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] In order to solve the defects of the prior art, the present invention integrates mobile robot technology, multi-sensor acquisition and imaging technology, and multi-modal data analysis technology to construct a multi-scenario self-propelled crop full-growing period phenotype high-throughput non-destructive acquisition and precise analysis equipment, which realizes unmanned efficient collection and intelligent analysis of phenotypic traits of crops in the field and facility environment throughout the growth period. On the one hand, it solves the limitation that the fixed phenotyping platform cannot be moved and can only be detected on fixed plots, and on the other hand, it greatly improves the detection efficiency of phenotypic traits. It is also suitable for field and indoor crop full-growing period phenotypic detection, realizing accurate, efficient, high-throughput, non-destructive measurement of crop phenotypic information. It provides high-throughput phenotyping solutions for breeding research teams, related enterprises, crop cultivation research teams, etc.

[0023] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] like Figure 1 As shown, a multi-scenario self-propelled crop phenotype high-throughput non-destructive acquisition equipment in this embodiment includes: a mobile robot platform 1, a satellite positioning module 2, a three-dimensional vision sensor 6, an edge computing device 8 and a multi-sensor phenotyping data acquisition unit 5.

[0025] The mobile robot platform 1 adopts a four-wheel independent drive and independent steering mode, and is height-adjustable; the three-dimensional visual sensor 6 and the multi-sensor phenotype data acquisition unit 5 are both arranged on the upper platform of the mobile robot platform 1.

[0026] The satellite positioning module 2 is used to obtain the starting position of the mobile robot platform 1; the edge computing device 8 is used to plan a path according to the starting position and the position of the target plot, and control the mobile robot platform 1 to move to the target plot according to the planned path.

[0027] When the four wheels of the mobile robot platform 1 move between the crop rows of the target plot, the three-dimensional vision sensor 6 is used to collect the plant canopy depth image and three-dimensional point cloud information in the visual area from above the plants in real time; the multi-sensor phenotypic data acquisition unit 5 is used to collect the multi-modal phenotypic sensor data of the plant canopy of the currently detected crop row in the visual area in real time from above the plants.

[0028] The edge computing device 8 is used to extract crop row information within the visual area from the plant canopy depth image, and dynamically adjust the steering angle of the mobile robot platform 1 according to the crop row information, so that the plant canopy of the currently detected crop row in the visual area is always at the center position of the mobile robot platform 1.

[0029] The edge computing device 8 is also used to calculate in real time the average distance from the crops in the visual area to the three-dimensional vision sensor 6 based on the depth information in the plant canopy depth image, and dynamically adjust the upper platform height of the mobile robot platform 1 based on the average distance so that the distance from the multi-sensor phenotyping data acquisition unit 5 to the plant canopy remains unchanged.

[0030] The edge computing device 8 is also used to analyze each modality of phenotypic sensing data to obtain a variety of phenotypic data of the canopy of the current detected crop row plant.

[0031] The edge computing device 8 is used to dynamically identify the target plot number in the material label inserted on the target plot from the RGB image of the plant canopy depth image, and automatically map the multimodal phenotypic sensor data with the target plot number.

[0032] The edge computing device 8 is also used to calculate the row spacing based on the crop row information extracted in the visual area. When the three-dimensional point cloud information collected by the three-dimensional vision sensor 6 gradually decreases to disappear, it is determined that the end of the row has been reached, and the average value of all row spacings calculated for the currently detected crop row is used as the row spacing value for the line break. The mobile robot platform 1 is controlled to turn vertically and then move laterally, and the lateral movement distance is made equal to the row spacing value for the line break, so as to align with the head of the next row of crops and complete the line break.

[0033] The equipment of the present invention is suitable for phenotypic detection in both field and indoor environments, and is suitable for detection of crops at different growth stages and different growth heights. The detection process moves across the seedlings, collects information from above the crops, and can measure the height of the crops in real time during the detection process.

[0034] During the data collection process, the three-dimensional visual sensor 6 will dynamically identify the material label inserted in each planting plot according to the RGB image, upload the text on the label, and the edge computing device 8 will correspond it to the phenotypic data collected at this time. In order to prevent missed recognition caused by occlusion, the satellite positioning module 2 will also dynamically detect the real-time position information of the equipment, and the collected phenotypic data will be assigned a position information mark, and the absolute position of each target plot (planting plot) is known, so the collected phenotypic data can be automatically mapped with the corresponding planting plot position in the end, so as to realize unmanned crop phenotypic trait collection. That is, the edge computing device 8 is also used to assign the real-time position mark of the mobile robot platform 1 dynamically detected by the satellite positioning module 2 to the multimodal phenotypic sensor data, so as to realize the automatic mapping of the multimodal phenotypic sensor data with the target plot position. The satellite positioning module 2 and the three-dimensional visual sensor 6 are used to dynamically detect the position information of the mobile robot platform 1 during the whole growth period, and the planting plot number is identified, and the collected phenotypic data is automatically mapped with the corresponding planting plot, so as to realize unmanned crop phenotypic trait collection.

[0035] Reference Figure 1The mobile robot platform 1 includes: an upper platform, a four-wheel mobile chassis and four electric telescopic rods 3. The four-wheel mobile chassis includes two two-wheel mobile chassis, and the two wheels in each two-wheel mobile chassis are arranged longitudinally front and back. The moving ends of the four electric telescopic rods 3 are respectively connected to the four corners of the bottom surface of the upper platform, and the fixed ends of the two electric telescopic rods 3 along the longitudinal movement direction of the wheels are arranged on a two-wheel mobile chassis; the three-dimensional visual sensor 6 and the multi-sensor phenotypic data acquisition unit 5 are both arranged on the top surface of the upper platform. The control end of each wheel in each two-wheel mobile chassis and the control end of the four electric telescopic rods 3 are connected to the signal output end of the edge computing device 8.

[0036] The mobile robot platform 1 adopts a lifting design, and the height of the mobile robot platform 1 is intelligently adjusted throughout the growth period through the electric lifting and retracting rod 3. The height adjustment range of the electric lifting and retracting rod 3 is 0-1.5 meters. The equipment measures crops with a height of less than 3 meters, and is suitable for phenotypic detection of different crops throughout the growth period.

[0037] The electric telescopic rod is arranged on the chassis of the mobile robot platform 1 (mobile robot platform for the whole growth period), and the platform height is changed by lifting the upper platform. The four corners of the top of the platform are provided with bases, and the multi-sensor phenotypic data acquisition unit 5 is mounted on the base and fixed by screws. The multi-modal phenotypic data analysis unit 7 is arranged at the edge of the multi-sensor phenotypic data acquisition unit 5. The edge computing device 8 included in the multi-modal phenotypic data analysis unit 7 receives the information of the satellite positioning module 2, the information of the three-dimensional visual sensor 6, and the position information of the electric telescopic rod 3 in real time, and calculates and outputs the control command to the mobile robot platform 1 internally, so as to realize the autonomous movement of the above-mentioned robot and the change of the platform height. At the same time, the edge computing device 8 receives the data of multiple phenotypic sensors 4 and performs phenotypic analysis internally. During the detection process, the self-propelled crop phenotypic high-throughput non-destructive acquisition and precise analysis intelligent equipment in the field and indoor environment moves across the seedlings, collects information from above the crops, and can measure the crop height in real time during the detection process.

[0038] In this embodiment, during use, the unmanned autonomous walking of this equipment is realized by the satellite and three-dimensional vision fusion navigation method. Specifically, navigation control is performed by carrying a satellite positioning module 2 and a three-dimensional vision sensor 6. Among them, the satellite positioning module 2 receives the signal propagation time of multiple satellites and the position information of the satellites. The self-position (longitude, latitude, altitude, heading angle, etc.) of the mobile robot platform 1 can be obtained through the triangulation positioning principle. Combined with the map data and the determined target position, a suitable path can be planned to control the multi-scene self-propelled crop full growth period phenotype high-throughput lossless acquisition and precise analysis equipment to move autonomously from the hangar to the target detection plot; the three-dimensional vision sensor 6 collects two-dimensional images and three-dimensional point cloud information, dynamically collects two-dimensional images and analyzes and extracts crop row information during the movement of this equipment, and performs real-time dynamic path detection and planning, and performs posture control of the mobile robot platform 1 to ensure that the multi-scene self-propelled crop full growth period phenotype high-throughput lossless acquisition and precise analysis equipment drives autonomously in the crop row, without human control throughout the process.

[0039] In the implementation process, the three-dimensional vision sensor 6 dynamically collects the RGB image and depth information of the plant canopy area while walking in the crop row. The semantic segmentation model is used on the edge computing device 8 to segment the plant area. Then, the image processing method is used to extract the median line of the single-row plant area. This line is mapped to the real world coordinate system to obtain the navigation line. The mapping method is: for a point (u, v) on the median line in the image coordinate system, the camera intrinsic parameter matrix K and the extrinsic parameter matrix [R|T] and the depth value Z obtained by the three-dimensional vision sensor 6 can be used to calculate its coordinates (X in the world coordinate system) by formula (1): w ,Y w ,Z w ).

[0040]

[0041] Where R is the rotation matrix of the camera, T is the translation vector of the camera, (X c ,Y c ,Z c ) is the coordinate of the pixel coordinate system point (u,v) in the camera coordinate system, (X w ,Y w ,Z w ) is the coordinate of the point in the world coordinate system, Z c It can be directly acquired by the three-dimensional vision sensor 6.

[0042] The angular deviation and lateral deviation of the mobile robot platform 1 are calculated in the real world coordinate system, and finally a dual-input single-output fuzzy controller is used to perform precise deviation correction and posture adjustment, with the angular deviation and lateral deviation as input and the steering angle as output. This navigation method can ensure that the crop canopy is always in the center of the equipment and improve the quality of phenotypic data collection. During driving, the above method can obtain three median lines of the crop areas in the left and right rows and the middle row, and they are all mapped to the real world coordinate plane. The row spacing is calculated by the equations of the three straight lines in the plane, and it is calculated every 10 frames of images. When the end of the row is reached, the three-dimensional visual sensor 6 dynamically scans the surrounding objects. According to the collected three-dimensional point cloud, the relative position of the mobile robot platform 1 and the crops below can be determined. When the point cloud of the crops below becomes smaller and smaller and finally disappears, it is determined that the end of the row has been reached. At this time, the previously collected row spacing is also averaged to obtain the final row spacing value D for the line change. Since the mobile platform is a four-wheel drive and four-wheel steering platform, it can directly change rows in a 90° lateral movement mode. When it moves horizontally by a distance D, the equipment is aligned with the head of the next row of crops, and the autonomous row change operation is completed. The three-dimensional point cloud information of the plant collected by the three-dimensional visual sensor 6 is used to identify the end of the operation row and calculate the row spacing, thereby completing the autonomous row change operation.

[0043] It can be concluded that the edge computing device 8 is used to extract the crop row information in the visual area from the plant canopy depth image, and dynamically adjust the steering angle of the mobile robot platform 1 according to the crop row information, so that the plant canopy of the currently detected crop row in the visual area is always at the center of the mobile robot platform 1, specifically including:

[0044] A semantic segmentation model is used to segment a plant canopy depth image into a single-row plant area; an image processing method is used to extract the segmented median line of the single-row plant area of ​​the current detected crop row; the median line of the single-row plant area of ​​the current detected crop row is mapped to a real-world coordinate system to obtain a navigation line; based on the navigation line, an angular deviation and a lateral deviation of a mobile robot platform 1 are calculated in the real-world coordinate system; the angular deviation and the lateral deviation are input into a fuzzy controller to output a steering angle; and the mobile robot platform 1 is dynamically adjusted according to the steering angle so that the plant canopy of the current detected crop row in the visual area is always at the center of the mobile robot platform 1.

[0045] The edge computing device 8 is also used to calculate the row spacing based on the crop row information extracted in the visual area, specifically including: using an image processing method to extract the midline of the single-row plant area of ​​the left and right rows of the segmented current detected crop row; mapping the midline of the single-row plant area of ​​the left and right rows to the real-world coordinate system to obtain two straight lines; and calculating the row spacing based on the two straight lines and the navigation line.

[0046] In the implementation process, the equipment intelligently adjusts the height through the electric lifting and retracting rod 3. Specifically, the target area in the field of view of the three-dimensional vision sensor 6 (i.e., the area to be scanned by other phenotype sensors) is determined according to the relative position of the other phenotype sensors and the three-dimensional vision sensor 6. The edge computing device 8 calculates the average distance from the object in the target area to the three-dimensional vision sensor 6 in real time based on the depth information obtained by the three-dimensional vision sensor 6. Assuming the origin of the world coordinate system is set at the camera, the Z coordinate of a point in the target area can still be calculated by equation (1): w The value is the distance from this point to the camera, so the average distance from the object in the area to the camera It can be calculated as:

[0047]

[0048] The distance d between the sensor and the crop is set in advance, and the edge computing device 8 finally calculates the distance between the sensor and the crop according to d and The control command is sent to the mobile robot platform 1 according to the distance difference, and the platform height of the electric lifting and retracting rod 3 is dynamically adjusted to ensure that the distance between the multi-sensor phenotyping data acquisition unit 5 and the crop canopy is stable, thereby improving the quality of phenotyping data acquisition.

[0049] The multi-sensor phenotypic data acquisition unit 5 includes: a visible light sensor, a thermal infrared sensor and a hyperspectral sensor. The visible light sensor, the thermal infrared sensor and the hyperspectral sensor are all connected to the edge computing device 8.

[0050] The edge computing device 8 is used to obtain the driving speed of the mobile robot platform 1 when the four wheels of the mobile robot platform 1 move between the crop rows of the target plot, and based on the matching function of the driving speed and the acquisition parameters of the visible light sensor, thermal infrared sensor, hyperspectral sensor and three-dimensional vision sensor 6, obtain the acquisition parameters of the visible light sensor, thermal infrared sensor, hyperspectral sensor and three-dimensional vision sensor 6 that match the driving speed, and control the acquisition of the visible light sensor, thermal infrared sensor, hyperspectral sensor and three-dimensional vision sensor 6 according to the matched acquisition parameters; the acquisition parameters include acquisition frame rate and exposure time.

[0051] After acquiring a variety of phenotypic sensor data, the edge computing device 8 independently analyzes the multimodal data such as images, point clouds, and spectra collected by different sensors. The edge computing device 8 is also used to obtain the green leaf area ratio, projected perimeter, convex hull area ratio, and texture features based on the visible light image collected by the visible light sensor; obtain the temperature value of each position of the plant based on the infrared sensing information collected by the thermal infrared sensor; and obtain the total spectral reflectance, spectral reflectance under 224 bands, average spectral reflectance, and derived first-order and second-order derivatives based on the hyperspectral image collected by the hyperspectral sensor. The edge computing device 8 is also used to obtain plant height, leaf area, and projected area based on the plant canopy depth image collected by the three-dimensional vision sensor 6.

[0052] According to the visible light image collected by the visible light sensor, the green leaf area ratio, projected perimeter, convex hull area ratio and texture features are obtained, specifically including: splicing the collected visible light images of each frame of the current detected crop row into a long image of a complete row of plants; using a trained image recognition model to cut out the minimum rectangular area of ​​a single crop from the long image of a row of plants; using a trained semantic segmentation model to segment a single crop from the minimum rectangular area of ​​a single crop to obtain a single crop image; using the Super G algorithm to segment the single crop image to obtain a single crop green part foreground image; determining the crop green area corresponding to the single crop green part foreground image, and taking the ratio of the crop green area to the total crop area as the green leaf area ratio; performing edge detection and geometric calculation on the single crop image to obtain the projected perimeter and convex hull area ratio; and extracting texture feature parameters from the single crop image using the grayscale-gradient co-occurrence matrix method.

[0053] According to the infrared sensing information collected by the thermal infrared sensor, the temperature value of each position of the plant is obtained, specifically including: amplifying the collected infrared sensing information and applying a filter to eliminate noise; performing analog-to-digital conversion, calibration and correction on the infrared sensing information after noise elimination; using the Canny operator and Harris corner point detection method to detect the plant area, and extracting the temperature value of each position of the plant in the plant area.

[0054] According to the hyperspectral image collected by the hyperspectral sensor, the total spectral reflectance, the spectral reflectance in 224 bands, the average spectral reflectance and the derived first-order derivative and second-order derivative are obtained, which specifically includes: correcting the collected hyperspectral image by a whiteboard to obtain a 224-band corrected image; dividing the 224-band corrected image to obtain a binary image; using a filter to remove background noise points in the binary image; using a trained instance segmentation model to segment the single crop in the binary image with the background noise points removed; performing a mask operation on the binary image of the single crop and the 224-band corrected image to obtain a hyperspectral grayscale image of the target foreground; taking the pixel weighted value of the spectral grayscale image of the target foreground as the total spectral reflectance; taking the ratio of the spectral reflectance to the number of pixels in a single band as the average spectral reflectance in a single band; according to the average spectral reflectance in a single band, the spectral reflectance in 224 bands, the average spectral reflectance and the derived first-order derivative and second-order derivative are calculated.

[0055] According to the plant canopy depth image collected by the three-dimensional vision sensor 6, the plant height, leaf area and projection area are obtained, specifically including: using the scale-invariant feature transform (SIFT) operator to detect the plant canopy depth images of adjacent date frames obtained by the RGB-D camera, and calculating the point cloud alignment coarse matching matrix and the displacement distance of adjacent images; according to the displacement distance of adjacent images, selecting the data frame for point cloud matching; using the color ICP (iterative closest point) algorithm to determine the fine matching matrix, and combining the data frame for point cloud matching to generate the point cloud of the crop row; using the clustering method to segment the point cloud of each plant from the point cloud of the crop row, and measuring the plant height, leaf area and projection area.

[0056] Figure 1 The multimodal phenotypic data analysis unit 7 shown in the figure can independently analyze multimodal data such as images, point clouds, spectra, etc. collected by different sensors, and perform data preprocessing such as image denoising, spectral data calibration, and point cloud reconstruction through the edge computing device 8 included in the multimodal phenotypic data analysis unit 7. Through a variety of data processing methods, efficient and accurate extraction of multi-type phenotypic data such as crop color texture, morphological structure, and physiological components can be achieved.

[0057] During the implementation process, the multi-sensor phenotypic data acquisition unit 5 will continuously collect crop canopy phenotypic data. The unit is equipped with multiple sensors such as visible light, thermal infrared, and hyperspectral. Due to environmental changes, such as occlusion, light changes, crop growth morphology and height differences, a single fixed light source cannot meet the imaging requirements. Therefore, the equipment of the present invention also includes: four linear light sources. The control ends of the four linear light sources are connected to the signal output end of the edge computing device 8. The edge computing device 8 is used to generate a crop 3D point cloud based on the plant canopy depth image and three-dimensional point cloud information collected by the three-dimensional visual sensor 6, and obtain the crop's growth angle relative to the ground and the crop height based on the crop 3D point cloud, so as to control the posture, position and brightness of the four linear light sources according to the crop's growth angle relative to the ground and the crop height, so that the leaves in the crop canopy area always receive uniform light and have sufficient light intensity.

[0058] The equipment also includes: four light source adjustment mechanisms. The four linear light sources are respectively arranged on the four light source adjustment mechanisms in a one-to-one correspondence. The control ends of the four light source adjustment mechanisms are connected to the signal output end of the edge computing device 8. Each light source adjustment mechanism includes: a slide rail, a first servo motor and a second servo motor. The linear light source is arranged on the slide rail. The edge computing device 8 is used to control the vertical movement of the slide rail through the first servo motor to adjust the height of the linear light source. The edge computing device 8 is used to control the pitch angle of the linear light source through the second servo motor. The edge computing device 8 is used to adjust the brightness of the linear light source by controlling the power-on voltage of the linear light source.

[0059] The present invention also provides another control scheme: a controller is designed in the edge computing device 8 to control the posture, position and brightness of the four linear light sources, so that the leaves in the crop canopy area always receive uniform illumination and have sufficient light intensity. The controller receives information from the front-end three-dimensional visual sensor 6, generates a crop 3D point cloud, obtains the growth angle of the crop relative to the ground and the height of the crop, and the light source is arranged on a slide rail. The servo motor at the end of the slide rail controls the movement of the slide rail, that is, controls the height of the light source. Another servo motor is arranged at the light source installation location to control the pitch angle of the light source. Finally, the brightness of the light source is adjusted by controlling the power-on voltage of the light source. At the same time, during the acquisition process, the driving speed of the mobile robot platform 1 will also be fed back to the controller. The driving speed has a corresponding matching function with the acquisition frame rate and exposure time of each sensor. The controller calculates the corresponding sensor acquisition frame rate and exposure time according to the speed, and sends it to the multi-sensor phenotypic data acquisition unit 5. Through the sensor acquisition software internal parameter setting, the acquisition frame rate, exposure time, etc. of the matching sensor are adjusted. The above method can ensure the data acquisition quality of multiple types of sensors and realize the synchronous acquisition of multiple sensor phenotypic traits.

[0060] The multi-sensor phenotypic data acquisition unit 5 adopts a modular design, which integrates multiple sensors such as visible light, thermal infrared, hyperspectral, and fluorescence imaging. Through a self-made controller, it performs adaptive control of different sensor acquisition light sources and acquisition methods to achieve synchronous acquisition of multiple sensor phenotypic traits. The acquisition frequency is adaptively aligned according to the driving speed of the mobile robot platform 1 to ensure the quality of data acquisition from multiple types of sensors.

[0061] The equipment of the present invention mainly includes: a mobile robot platform 1, a multi-sensor phenotypic data acquisition unit 5 and a multi-modal phenotypic data analysis unit 7. The mobile robot platform for the entire growth period adopts a four-wheel independent drive and independent steering mode, with adjustable height, and can travel across rows above crops of different growth periods and different heights. It adopts satellite positioning and three-dimensional visual navigation to achieve automatic unmanned driving, and real-time detection of target crop area marks, corresponding to the data collected by the multi-sensor phenotypic data acquisition unit 5, to achieve autonomous walking of the phenotypic detection platform. The multi-sensor phenotypic data acquisition unit 5 mainly includes sensors such as visible light, hyperspectral, three-dimensional, thermal infrared, and fluorescent imaging. According to the differences in different sensor imaging modes and light source requirements, it automatically controls the automatic collection of various sensor data. The multi-modal phenotypic data analysis unit 7 performs intelligent analysis on multiple types of data such as images, point clouds, and spectra collected by various sensors to achieve accurate measurement of phenotypic traits such as color texture, morphological structure, physiological information, and growth potential of crops in facilities and field environments.

[0062] In summary, the advantages of the equipment of the present invention are as follows:

[0063] (1) Strong versatility and wide adaptability.

[0064] The present invention adopts a high-ground-clearance mobile platform with four-wheel drive and four-wheel steering, equipped with a variety of different functional collection sensors, which can be simultaneously applicable to the measurement needs of different crop phenotypic detection parameters in field and indoor environments, and the operating height is adjustable, suitable for a variety of crops with different growth heights in different growth periods.

[0065] (2) Fully automatic, high efficiency and high throughput.

[0066] The satellite and 3D vision fusion navigation method is used to realize the unmanned autonomous walking of the mobile robot platform throughout the growth period. It can simultaneously collect multiple types of data information such as visible light, thermal infrared, and hyperspectral, and can automatically associate the detected data with the target cell (object). Compared with traditional methods, the detection throughput and efficiency are high, and no human intervention is required.

[0067] (3) High precision, strong detection stability and good consistency.

[0068] The detection robot is height-adjustable, and the detection sensor is height-adjustable automatically, ensuring a certain distance between the sensor and the canopy of the detection object, ensuring the accuracy of the collected data, and ensuring good stability and consistency of the collected phenotypic traits.

[0069] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0070] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A multi-scenario self-propelled high-throughput non-destructive crop phenotype acquisition equipment, characterized in that: The equipment includes: a mobile robot platform, a satellite positioning module, a three-dimensional vision sensor, an edge computing device, and a multi-sensor phenotypic data acquisition unit; The mobile robot platform adopts a four-wheel independent drive and independent steering mode, and its height is adjustable; the three-dimensional visual sensor and the multi-sensor phenotypic data acquisition unit are both arranged on the upper platform of the mobile robot platform; the mobile robot platform includes: an upper platform, a four-wheel mobile chassis and four electric telescopic rods; the four-wheel mobile chassis includes two two-wheel mobile chassis, and the two wheels in each two-wheel mobile chassis are arranged longitudinally front and back; the moving ends of the four electric telescopic rods are respectively connected to the four corners of the bottom surface of the upper platform, and the fixed ends of the two electric telescopic rods along the longitudinal movement direction of the wheels are arranged on a two-wheel mobile chassis; the three-dimensional visual sensor and the multi-sensor phenotypic data acquisition unit are both arranged on the top surface of the upper platform; the control end of each wheel in each two-wheel mobile chassis and the control end of the four electric telescopic rods are connected to the signal output end of the edge computing device; The satellite positioning module is used to obtain the starting position of the mobile robot platform; the edge computing device is used to plan a path according to the starting position and the position of the target plot, and control the mobile robot platform to move to the target plot according to the planned path; When the four wheels of the mobile robot platform move between the crop rows of the target plot, the three-dimensional vision sensor is used to collect the depth image and three-dimensional point cloud information of the plant canopy in the visual area from above the plant in real time; the multi-sensor phenotypic data acquisition unit is used to collect multi-modal phenotypic sensor data of the plant canopy of the currently detected crop row in the visual area from above the plant in real time; The edge computing device is used to extract crop row information within the visual area from the plant canopy depth image, and dynamically adjust the steering angle of the mobile robot platform according to the crop row information, so that the plant canopy of the currently detected crop row within the visual area is always at the center of the mobile robot platform; The edge computing device is also used to calculate in real time the average distance from the crops in the visual area to the three-dimensional vision sensor according to the depth information in the plant canopy depth image, and dynamically adjust the height of the upper platform of the mobile robot platform according to the average distance so that the distance from the multi-sensor phenotyping data acquisition unit to the plant canopy remains unchanged; The edge computing device is also used to analyze the phenotypic sensing data of each modality to obtain multiple phenotypic data of the canopy of the crop row plant currently being detected; The edge computing device is also used to dynamically identify the target plot number in the material label inserted on the target plot from the RGB image of the plant canopy depth image, and automatically map the multimodal phenotypic sensor data with the target plot number; The edge computing device is also used to calculate the row spacing based on the crop row information extracted in the visual area. When the three-dimensional point cloud information collected by the three-dimensional vision sensor gradually decreases to disappear, it is determined that the end of the row has been reached. The average value of all row spacings calculated for the current detected crop row is used as the row spacing value for the line break. The mobile robot platform is controlled to turn vertically and then move laterally, and the lateral movement distance is made equal to the row spacing value for the line break, so as to align with the head of the next row of crops and complete the line break.

2. The multi-scenario self-propelled crop phenotype high-throughput non-destructive acquisition equipment according to claim 1, characterized in that: The edge computing device is also used to assign the real-time location tag of the mobile robot platform dynamically detected by the satellite positioning module to the multimodal phenotypic sensor data, thereby realizing automatic mapping of the multimodal phenotypic sensor data with the target plot location.

3. The multi-scenario self-propelled crop phenotype high-throughput non-destructive acquisition equipment according to claim 1, characterized in that: The height adjustment range of the electric telescopic rod is 0-1.5 meters; The equipment measures crops within 3 meters in height.

4. The multi-scenario self-propelled crop phenotype high-throughput non-destructive acquisition equipment according to claim 1, characterized in that: The multi-sensor phenotypic data acquisition unit includes: a visible light sensor, a thermal infrared sensor and a hyperspectral sensor; Visible light sensors, thermal infrared sensors, and hyperspectral sensors are all connected to edge computing devices; The edge computing device is used to obtain the driving speed of the mobile robot platform when the four wheels of the mobile robot platform move between the crop rows of the target plot, obtain the acquisition parameters of the visible light sensor, thermal infrared sensor, hyperspectral sensor and three-dimensional vision sensor that match the driving speed based on the matching function of the driving speed and the acquisition parameters of the visible light sensor, thermal infrared sensor, hyperspectral sensor and three-dimensional vision sensor, and control the acquisition of the visible light sensor, thermal infrared sensor, hyperspectral sensor and three-dimensional vision sensor according to the matched acquisition parameters; the acquisition parameters include acquisition frame rate and exposure time; The edge computing device is also used to obtain the green leaf area ratio, projected perimeter, convex hull area ratio and texture characteristics based on the visible light images collected by the visible light sensor; obtain the temperature value of each position of the plant based on the infrared sensing information collected by the thermal infrared sensor; and obtain the total spectral reflectance, spectral reflectance in 224 bands, average spectral reflectance and derived first-order and second-order derivatives based on the hyperspectral images collected by the hyperspectral sensor. Edge computing devices are also used to obtain plant height, leaf area, and projected area based on plant canopy depth images collected by three-dimensional vision sensors.

5. The multi-scenario self-propelled high-throughput non-destructive acquisition equipment for crop phenotypes according to claim 4, characterized in that: According to the visible light image collected by the visible light sensor, the green leaf area ratio, projection perimeter, convex hull area ratio and texture features are obtained, including: The collected visible light images of the current crop row are stitched together into a long image of a complete row of plants; the smallest rectangular area of ​​a single crop is cut out from the long image of a row of plants using a trained image recognition model; a single crop is segmented from the smallest rectangular area of ​​a single crop using a trained semantic segmentation model to obtain a single crop image; the single crop image is segmented using the Super G algorithm to obtain a single crop green part foreground image; the green area of ​​the crop corresponding to the single crop green part foreground image is determined, and the ratio of the crop green area to the total crop area is used as the green leaf area ratio; edge detection and geometric calculation are performed on the single crop image to obtain the projection perimeter and convex hull area ratio; the texture feature parameters are extracted from the single crop image using the grayscale-gradient co-occurrence matrix method; According to the infrared sensing information collected by the thermal infrared sensor, the temperature values ​​of various positions of the plant are obtained, including: The collected infrared sensor information is amplified and a filter is applied to eliminate noise; the infrared sensor information after noise elimination is converted into analog and digital, calibrated and corrected; the plant area is detected using the Canny operator and Harris corner point detection method, and the temperature value of each position of the plant in the plant area is extracted; According to the hyperspectral images collected by the hyperspectral sensor, the total spectral reflectance, the spectral reflectance in 224 bands, the average spectral reflectance and the derived first-order and second-order derivatives are obtained, including: The collected hyperspectral image is corrected by a whiteboard to obtain a 224-band corrected image; the 224-band corrected image is divided to obtain a binary image; a filter is used to remove background noise points in the binary image; a trained instance segmentation model is used to segment individual crops from the binary image with the background noise points removed; the binary image of the individual crop is masked with the 224-band corrected image to obtain a hyperspectral grayscale image of the target foreground; the pixel weighted value of the spectral grayscale image of the target foreground is taken as the total spectral reflectance; the ratio of the spectral reflectance to the number of pixels in a single band is taken as the average spectral reflectance in the single band; the spectral reflectance, average spectral reflectance, and the derived first-order derivative and second-order derivative in 224 bands are calculated based on the average spectral reflectance in the single band; According to the plant canopy depth image collected by the 3D vision sensor, the plant height, leaf area and projection area are obtained, including: The scale-invariant feature transformation operator was used to detect the plant canopy depth images of adjacent date frames and calculate the displacement distance of adjacent images. According to the displacement distance of adjacent images, the data frame for point cloud matching was selected. The color ICP algorithm was used to determine the fine matching matrix, and the point cloud of the crop row was generated by combining the data frame for point cloud matching. The clustering method was used to segment the point cloud of each plant from the point cloud of the crop row, and the plant height, leaf area and projection area were measured.

6. The multi-scenario self-propelled crop phenotype high-throughput non-destructive acquisition equipment according to claim 1, characterized in that: The equipment also includes: four linear light sources; The control ends of the four linear light sources are all connected to the signal output ends of the edge computing device; The edge computing device is used to generate a crop 3D point cloud based on the plant canopy depth image and three-dimensional point cloud information collected by the three-dimensional vision sensor, and obtain the growth angle of the crop relative to the ground and the crop height based on the crop 3D point cloud, so as to control the posture, position and brightness of the four linear light sources according to the growth angle of the crop relative to the ground and the crop height, so that the leaves in the crop canopy area always receive uniform light and light intensity.

7. The multi-scenario self-propelled crop phenotype high-throughput non-destructive acquisition equipment according to claim 6, characterized in that: The equipment also includes: four light source adjustment mechanisms; The four linear light sources are respectively arranged on the four light source adjustment mechanisms in a one-to-one correspondence; The control ends of the four light source adjustment mechanisms are all connected to the signal output ends of the edge computing device; Each light source adjustment mechanism includes: a slide rail, a first servo motor and a second servo motor; The linear light source is arranged on the slide rail; The edge computing device is used to control the vertical movement of the slide rail through the first servo motor to adjust the height of the linear light source; The edge computing device is used to control the pitch angle of the linear light source through a second servo motor; The edge computing device is used to adjust the brightness of the linear light source by controlling the power-on voltage of the linear light source.

8. The multi-scenario self-propelled crop phenotype high-throughput non-destructive acquisition equipment according to claim 1, characterized in that: The edge computing device is used to extract crop row information within the visual area from the plant canopy depth image, and dynamically adjust the steering angle of the mobile robot platform according to the crop row information, so that the plant canopy of the currently detected crop row within the visual area is always at the center of the mobile robot platform, specifically including: Use semantic segmentation model to segment single-row plant regions in plant canopy depth images; An image processing method is used to extract the median line of the segmented single-row plant area of ​​the currently detected crop row; Mapping the median line of the single-row plant area of ​​the current detected crop row to the real-world coordinate system to obtain a navigation line; Calculating the angular deviation and lateral deviation of the mobile robot platform in a real world coordinate system based on the navigation line; The angular deviation and lateral deviation are input into the fuzzy controller, and the steering angle is output; The mobile robot platform is dynamically adjusted according to the steering angle so that the plant canopy of the crop row currently being detected in the visual area is always located at the center of the mobile robot platform.

9. The multi-scenario self-propelled crop phenotype high-throughput non-destructive acquisition equipment according to claim 8, characterized in that: The edge computing device is also used to calculate the row spacing based on the crop row information extracted within the visual area, including: An image processing method is used to extract the median line of the single-row plant area of ​​the two rows on the left and right of the currently detected crop row; Map the midlines of the single-row plant areas on the left and right rows to the real-world coordinate system to obtain two straight lines; The line distance is calculated based on the two straight lines and the guide line.

Images