A phenotyping device, method, and system for measuring phenotypic traits of one or more plants in a target canopy.

Abstract

A phenotyping device, method, and system for measuring phenotypic traits of one or more plants in a target canopy, the phenotyping device comprising a canopy deployer for deploying the target canopy, a camera unit, means for controlling the angle and distance of the camera unit relative to the canopy deployer, and an electronic control unit for controlling the camera unit. The phenotyping device, method, and system enable the recording of raw data from the target canopy during a data collection session, the data collection session including information about plants not visible to the camera unit without deploying the canopy.

Description

Technical Field

[0001] This invention generally relates to digital phenotyping systems and methods, and more specifically, to measuring plant traits. Furthermore, this disclosure relates to apparatus and methods for digital image processing. Background Technology

[0002] Numerical phenotypes refer to the estimation of plant traits from image data. In plant breeding, numerical phenotypes are used to generate data in a cost-effective manner. In crop production, image-based methods for assessing crop health and yield are used to optimize agronomic management, such as selecting field areas for herbicide treatment and choosing the amount of fertilizer to use.

[0003] Existing digital phenotyping devices and methods analyze image data acquired from above the canopy, while many agronomically relevant traits lie below the top of the canopy. The problem is that the cameras in known devices can only see the upper part of the canopy, so the collected images cannot provide relevant and accurate information about phenotypic traits below the top of the canopy.

[0004] The estimation of plant phenotypic traits needs to serve as the basis for decision-making in crop production and crop production research and development. Plant phenotypic trait estimations are used, for example, for decision-making and optimization. Information about plants measured in the form of plant phenotypic trait estimates is the cornerstone for assessing the relative performance of different plant genotypes (varieties or candidate varieties in breeding programs), agronomic practices, and the effectiveness and risks of agronomic inputs such as fertilizers, herbicides, and pesticides.

[0005] Therefore, based on the foregoing discussion, it is necessary to overcome the drawbacks associated with traditional devices and methods used for digital phenotypes. Summary of the Invention

[0006] This disclosure aims to improve the relevance and accuracy of image data generated from phenotypic traits not visible from above. This objective is achieved through a phenotyping device and method capable of mechanically unfolding the canopy and acquiring image data from portions below the top, thereby improving the relevance and accuracy of image data generated from phenotypic traits not visible from above.

[0007] On one hand, embodiments of this disclosure provide a phenotyping device for measuring phenotypic traits of one or more plants in a target canopy. The phenotyping device includes: one or more canopy spreaders for exposing the one or more plants to image within the target canopy; one or more camera units; means for controlling the angle and distance of the one or more camera units relative to the one or more canopy spreaders; and an electronic control unit for controlling the camera units.

[0008] On the other hand, embodiments of this disclosure provide a phenotyping method for measuring phenotypic traits of one or more plants in a target canopy. The method includes the following steps: defining an input data specification for raw data collected from a current set of raw data collection targets by one or more camera units of a phenotyping device; defining a usage orientation-height specification; defining a device-specific device position angle specification; defining a usage movement specification; adjusting at least one parameter of a device for controlling the angle and distance of the one or more camera units relative to a canopy spreader according to the device-specific device position angle specification; initiating a raw data collection session; moving the phenotyping device, in contact with a portion of one or more plants in the target canopy, according to the defined usage orientation-height specification and the usage movement specification, to expose the one or more plants for imaging while measuring their phenotypic traits; recording the raw data of the one or more plants in the target canopy while moving the device in contact with the portion of the target canopy according to the usage orientation-height specification and the defined usage movement specification; and processing the recorded raw data to estimate the phenotypic traits of one or more plants in the target canopy.

[0009] On the other hand, embodiments of the present invention provide a phenotyping system for measuring phenotypic traits of one or more plants in a target canopy, the system comprising at least one phenotyping device according to the present disclosure; means for moving the phenotyping device; and means for processing data generated by the phenotyping method, wherein the means includes a plurality of computing units.

[0010] According to this disclosure, the phenotyping devices, methods, and systems are capable of unfolding the target canopy of plants to reveal portions of the plants belonging to the original data collection target within the target canopy that are not visible without unfolding the canopy. This improves the quality and quantity of phenotypic trait estimates obtained by processing video from the canopy by attaching a mechanical structure to a moving camera, such that the mechanical structure (i.e., the canopy unfolder) reveals new portions of the canopy to the camera as the camera moves. Attached Figure Description

[0011] Embodiments of this disclosure will now be described by way of example only with reference to the following figures.

[0012] Figure 1 This is a schematic top view of a phenotypic device according to an embodiment, showing a phenotypic device including two height sensors and one tension sensor.

[0013] Figure 2 This is a schematic side view of the phenotypic device according to an embodiment, showing the position of the device relative to a particular plant in the canopy at different points in time.

[0014] Figure 3 This is a schematic top view of the phenotypic device according to an embodiment, showing an example of the device's orientation in the canopy.

[0015] Figure 4 This is a schematic diagram illustrating an example of an image from the target canopy obtained by the camera unit of the device according to an embodiment.

[0016] Figure 5 The storage and process of raw data in the embodiment are illustrated.

[0017] Figure 6 An exemplary embodiment of a phenotypic device according to the present disclosure is shown. Detailed Implementation

[0018] The following detailed description illustrates embodiments of the present disclosure and ways in which these embodiments can be implemented. The present disclosure provides phenotyping devices, methods, and systems for measuring phenotypic traits of one or more plants in a target canopy.

[0019] In one aspect, embodiments of this disclosure provide a phenotyping device for measuring phenotypic traits of one or more plants in a target canopy, the device comprising: one or more canopy spreaders for exposing the one or more plants to image in the target canopy; one or more camera units; means for controlling the angle and distance of the one or more camera units relative to the one or more canopy spreaders; and an electronic control unit for controlling the camera units.

[0020] In an alternative embodiment, a phenotyping device for measuring phenotypic traits of one or more plants in a target canopy includes at least one of the following: a storage component, a power source, one or more tension sensors, one or more height sensors, other sensors, one or more control units, one or more cooling units, a solar panel power unit, one or more mechanical support structures, one or more computing units, a means for moving the phenotyping device (e.g., a robotic system or vehicle), and a mechanical unit. Raw data recorded by one or more camera units is stored on the storage component. The power source provides power to electronic components requiring power. The tension sensors measure mechanical stresses associated with unfolding the canopy using the one or more canopy unfolders, and the one or more tension sensors are connected to the one or more canopy unfolders. The one or more height sensors are connected to the storage component and mechanically connected to the phenotyping device. The other sensors are connected to the storage component and mechanically connected to the phenotyping device, wherein measurements recorded by the other sensors are included in the raw data stored on the storage component. The one or more control units activate and deactivate the data collection session and otherwise control the one or more camera units. The one or more cooling units are connected to electronic components requiring cooling. The solar panel power unit is connected to the power source and charges the power source. The one or more mechanical support structures are mechanically connected to the phenotypic device.

[0021] In one aspect, embodiments of this disclosure provide a phenotyping device for measuring phenotypic traits of one or more plants in a target canopy, comprising: a frame; a canopy deployer attached to the frame and used to deploy one or more plants in the target canopy; a camera position holding unit connected to the frame; a camera unit connected to the camera position holding unit; and an electronic control unit. While the device moves and partially contacts the plants in the target canopy, the frame maintains a fixed angle and distance between the canopy deployer and the camera position holding unit.

[0022] Throughout this disclosure, the term "phenotypic trait" refers to measurements, estimates, and counts relating to the structure and growth of plants in the canopy. Phenotypic traits include the structure, quantity, species, variety, density, and location of plants in the canopy. Furthermore, phenotypic traits include the size and quantity of plant sub-parts (e.g., yield components of plants in the canopy and leaves), as well as the mechanical and spectral characteristics of the canopy and plants. In addition, phenotypic traits include statistics of other phenotypic traits, such as mean, variance and quantiles, estimated distributions, and spatial distributions. Furthermore, phenotypic traits include time series of estimates, counts, and statistics related to plants in the canopy. Phenotypic traits also include agronomic outcomes, such as yield and disease resistance or susceptibility.

[0023] The term "target canopy" refers to a piece of land covered with plants of interest in their physical structure. In one example, a target canopy is an experimental field used for plant breeding. In another example, a target canopy could be a canopy grown in a growth chamber or greenhouse. In yet another example, a target canopy is, for example, a sub-section of a field used for agronomic production. The plant species grown in a target canopy can range from grasses to cereals, legumes, crops, or any other type of plant.

[0024] Throughout this disclosure, the term “expanding the canopy” refers to pushing or pulling the canopy of plants belonging to set A such that portions of plants in another set B, as well as optional portions of plants in set a, are exposed for imaging.

[0025] The term "device that moves in partial contact with the target canopy" refers to a canopy deployer manipulating the position of one or more plants in the target canopy that are not part of the current raw data collection target group to improve the visibility of plants in the current raw data collection target group to one or more camera units. In other words, the device makes partial contact with the target canopy.

[0026] The canopy expander of this phenotyping device, or optionally more than one canopy expander in alternative embodiments, creates an opening in the canopy by expanding it, making the portion of the plant below the top of the canopy of the current set of raw data collection targets visible to one or more camera units and optionally to other sensor groups. The canopy expander may include, but is not limited to, one of the following: a rod or shaft that pushes or pulls away other plants obstructing visibility from the camera unit to the currently imaged target plant, wherein the width of the canopy expander depends on the width of the target canopy. Optionally, only a small portion of the target canopy is of interest, for example, 30% of the width of the breeding field in the middle of the breeding field, and the width of the canopy expander is only sufficient to cover the small portion of interest.

[0027] In one example, the phenotyping device includes two canopy spreaders, each a 140 cm long axis, attached parallel to each other and spaced 10-40 cm apart. In another example, the phenotyping device includes a single canopy spreader, a 150 cm long axis. In a third example, the canopy spreader is an axis containing plow-like structures that separate plants in different planting rows. Optionally, the width of the canopy spreader is adjustable, for example, between 95 cm and 180 cm. Thus, the width of one or more canopy spreaders can increase from 95 cm, 105 cm, 115 cm, 125 cm, 135 cm, 145 cm, 155 cm, 165 cm, or 175 cm to 100 cm, 110 cm, 120 cm, 130 cm, 140 cm, 150 cm, 160 cm, 170 cm, or 178 cm. Having several narrow canopy spreaders or one wider canopy spreader has a remarkable effect, significantly improving visibility of the current group of raw data collection targets. If the device has only one narrow spreader, plants pushed aside by the canopy spreader during operation will not remain behind but will rise up, thus obstructing the camera's visibility of the current group of raw data collection targets.

[0028] In one example, the device for controlling the angle and distance of the camera unit includes an adjustable frame that includes a camera position holding unit, and a canopy deployer is attached to the adjustable frame via an adjustable connector capable of adjusting the angle at which the canopy deployer is attached. The angle may need to be modified to obtain better visibility from the camera unit to the current group's raw data collection target.

[0029] A camera unit may include one or more cameras. Alternatively, in alternative embodiments, a camera unit may also include: a power element, a flash or light source, a spectral sensor, a laser scanner, a lidar sensor, or other distance measurement sensors or attachments. The camera unit continuously captures still images or records video during the measurement process to collect raw data from the current set of raw data collection targets. The camera unit may record bandwidths visible or invisible to the human eye. The camera unit may also include other spectral sensors. The camera unit may include laser scanners and lidar sensors, as well as other distance measurement sensors. A camera unit may include a plurality of cameras or other sensors. One advantage of having a plurality of cameras is that it enables stereo imaging. The camera unit is electrically connected to a power source to provide power to the camera unit. The camera unit is connected to a storage unit for data transmission, and still images and video are stored on the storage unit. Optionally, a camera unit may include a plurality of camera units. Therefore, in embodiments of this disclosure, a camera unit in one or more camera units may include at least one of the following: one or more cameras, one or more spectral sensors, a laser scanner, or a lidar sensor.

[0030] The means for controlling the angle and distance between the camera unit and the canopy expander can control the distance and angle between the canopy expander and the camera unit according to device-specific device position angle specifications. As a result, the raw data collected by the camera unit from the current set of raw data collection targets meets the input data specifications. When the canopy expands the target canopy according to the usage orientation height specification and optionally moves the canopy expander at a speed specified using the movement specification, the means for controlling the angle and distance between the camera unit and the canopy expander maintains the position and orientation of one or more camera units fixed relative to the canopy expander so that the camera unit has visibility of the current set of raw data collection targets.

[0031] Optionally, the size and angle of the means for controlling the angle and distance of the camera unit relative to the canopy expander are adjustable, allowing the angle and distance of the camera unit relative to the canopy expander to be adjusted according to different equipment-specific equipment position angle specifications. Adjusting the angle and distance parameters of the camera unit connected to the canopy expander has the advantage of improving the quality of the recorded raw data.

[0032] In one example, the means for controlling the angle and distance of the camera unit relative to the canopy expander includes an adjustable frame connected to the canopy expander, wherein the adjustable frame has a camera position holding unit, which is, for example, an adjustable hinge connector connecting the camera unit and the adjustable frame, in which the angle of the hinge can be adjusted according to a device-specific device position angle specification.

[0033] In one example, the adjustable frame includes two telescopic adjustable structures. These adjustable structures can be, for example, adjustable shafts, with the shorter telescopic shaft ranging in length from 30 cm to 55 cm and the longer telescopic shaft ranging in length from 50 cm to 150 cm. The two telescopic shafts are connected via a 90-degree fixed-angle connector. The adjustable frame also includes a camera position holding unit that connects the end of the longer telescopic shaft opposite the 90-degree fixed-angle connector to the camera unit. In this example, the camera position holding unit is attached to the camera unit and can control the angle between the camera unit and the longer telescopic shaft within a plane defined by the directions of the two telescopic shafts, between 0 and 340 degrees. The angle between the camera unit and the adjustable frame is defined as the angle between the direction of the longer telescopic shaft and line A, where line A is the line connecting the center of the camera unit's lens to the center of the image captured by the camera unit. Adjusting the size of the telescopic shafts and the angle between the camera unit and the longer telescopic shaft allows for obtaining high-quality images and videos in different usage environments, such as those where canopy height and breeding field width vary. In one example, higher quality is achieved by optimizing the camera orientation so that at each time point in the image or video captured by the camera unit, the projection of the relevant part of the plant in the current group's raw data collection target is captured with as many pixels as possible.

[0034] In one example, at the first position A, the canopy plants are not easily bent, and only a small opening can be formed using a canopy spreader. Therefore, the camera unit must record raw data from a high position, resulting in low resolution. At the second position B, the canopy plants are flexible, and a wider opening can be made in the canopy using a canopy spreader, allowing the camera unit to record raw data from a lower position with higher raw data resolution.

[0035] In another example, the camera position holding unit is, for example, a threaded hole in an adjustable frame that allows the camera unit to be attached to the adjustable frame with screws. In yet another example, the camera position holding unit is, for example, a hole in an adjustable frame that serves as a bracket, into which the camera unit can be pressed.

[0036] To meet the input data specifications of the raw data collected by one or more camera units, it may be necessary to adjust the angle at which the camera units are attached to the adjustable frame. Therefore, if the phenotyping device is used in several different environments where different types of crops are grown and different experimental practices are conducted, the ability to adjust the angle and distance between the camera units and the canopy spreader is useful.

[0037] In one embodiment, the distance between the camera unit and the canopy expander is fixed and cannot be adjusted; that is, in this embodiment, when the distance is fixed, there is no need to adjust the angle and distance between the camera units. An advantage of this embodiment is that the angle and distance of the camera units are the same for all the raw data collected by the phenotyping device. In this embodiment, the resolution obtained varies in different environments, but the resolution obtained using very good camera units is sufficient for many applications.

[0038] In another example, the means for controlling the angle and distance of the camera unit relative to the canopy deployer is a drone, to which the camera unit, along with some sensors from an optional other sensor set, is attached. The drone is programmed to fly and maintain a fixed position and orientation relative to the canopy deployer, as specified by device-specific device position angle specifications.

[0039] In another example, the means for controlling the angle and distance of the camera unit relative to the canopy expander includes: means for moving a camera operated by another user according to a protocol, and for maintaining the position and orientation of the camera unit, optionally some of the other sensors in the sensor group, relative to the canopy expander according to the protocol.

[0040] In another example, the means for controlling the angle and distance of the camera unit relative to the canopy expander is capable of adjusting the distance between the camera and the canopy expander in orthogonal x, y, and z directions, and is also capable of controlling the angle of the camera unit when recording raw data from the current group of raw data collection targets.

[0041] In embodiments of this disclosure, the one or more canopy expanders may be in a fixed position relative to the location of one or more camera units. Attaching one or more canopy expanders to a fixed position relative to the location of one or more camera units when moving the device within the target canopy has the technical effect of significantly improving the visibility of the target canopy vegetation to the one or more camera units. Unlike one or more camera units acquiring images or videos from the top of the canopy, in this disclosure, the entire height of the canopy is displayed when traversing only a small portion of the target canopy at a time, corresponding to the original data collection target of the current group. The canopy expander translates the movement of the device into increased visibility, thereby enhancing the value of the original data collected by the one or more camera units.

[0042] In an alternative embodiment, the phenotyping device includes additional sensors, such as one or more tension sensors or one or more height sensors. The phenotyping device may also include at least one of the following: a microphone for recording speech, an accelerometer, an orientation sensor, a GPS receiver, or any other sensor that can be used to collect data from the target canopy. A voice control device is used to control the phenotyping device, and the collected data is stored as raw data in a storage component.

[0043] In embodiments, the one or more tension sensors are used to measure the mechanical resistance associated with deploying the canopy using the one or more canopy deployers. In one example, the canopy deployer is attached to an adjustable frame via a mechanical hinge, and the tension of the mechanical hinge generated when the canopy deployer deploys the canopy as the phenotyping device is moved within the canopy is measured by the tension sensor and recorded on a storage component. In another example, the tension sensor is a mechanically serial sensor that measures the tension on the hinge connecting the canopy deployer to the adjustable frame, and the output of the serial sensor is recorded as video input to a camera unit. Optionally, the tension sensor is used to connect the canopy deployer to the means for controlling the angle and distance of the camera unit relative to the canopy deployer. In another embodiment, the one or more height sensors are attached to the canopy deployer to measure the distance from the canopy deployer to the ground. In one example, two height sensors are attached to, for example, the end of a 140 cm long axial canopy deployer, and when the phenotyping device is oriented according to a usage orientation height specification, these two height sensors point to a line parallel to the normal to the approximate tangential plane of the ground. In this example, the phenotyping device is used to record raw data from rectangular breeding test plots, each approximately 120 cm wide and 300 cm long, with a distance of 30 cm between plots parallel to the width. When the phenotyping device is moved at a height of 20 cm below an approximate tangential plane to the ground, parallel to this plane, the axial canopy spreader is aligned parallel to the width of the test plot, with the center of the axis located in the middle of the plot in the width direction. The ends of the canopy spreader move within the areas separating the test plots, where plants are not tested and the vegetation is low. In this example, a height sensor is located at the end of the canopy spreader axis, pointing towards the ground so that it can directly see the ground and estimate the distance from the canopy spreader to the ground with a resolution of + / - 5 cm.

[0044] The storage component stores raw data that is recorded by the one or more camera units, by one or more tension sensors, received from the other set of sensors, or received from one or more height sensors during a data collection session.

[0045] In one example, the storage component includes a network connection for transmitting raw data to the server storage component via the Internet. In another example, the storage component includes temporary storage where raw data is temporarily stored during a data collection session, and whereby the temporarily stored data is transferred from the temporary storage to the server storage component via the network connection or the Internet after the data collection session. Optionally, the power supply, camera unit, and storage component are interconnected via one or more electronic connector units that transmit electricity or data, or both. Optionally, in one embodiment, for example, a smartphone is used to provide the camera unit, power supply, storage unit, and electronic control unit.

[0046] In one embodiment, the phenotypic device is moved by a robotic system that controls the camera unit via an electronic control unit, the electronic control unit including a wired or wireless connection to the main robotic system.

[0047] In another example, the electronic control unit is directly connected to the camera unit via a wired or wireless connection. In yet another example, the electronic control unit is connected to an electronic connector unit. In one embodiment, the electronic control unit starts and stops video recording by the camera unit, while continuously recording the outputs of different sensors in addition to recording the time of sensor inputs. In this embodiment, based on the time of video recording or the time of recorded sensor inputs, the recorded outputs of different sensors can be matched with subsequently recorded video.

[0048] The power source includes one or more rechargeable or disposable batteries, which provide energy to a subset of electronic components that require power from the device. Optionally, a solar power unit is used to supply power to the power source. In one embodiment, the power source includes two batteries, with the solar power unit charging one battery and the other battery providing power to other electronic components of the device.

[0049] Optionally, one or more cooling units are used to reduce the temperature of electrical components, specifically by transferring heat away from the electrical components and reflecting radiation such as sunlight. In one example, the cooling unit uses pre-cooled ice blocks attached to the camera unit and power supply. In another example, the cooling unit uses thermoelectric cooling technology to reduce the temperature of one or more electronic components of the device.

[0050] The one or more height sensors, the one or more tension sensors, and the group of other sensors are connected to the target measurement location in the phenotypic device.

[0051] Optionally, the phenotypic device includes one or more computing units, wherein computations are performed for a first machine learning system, optionally a second machine learning system, and optionally a third machine learning system to process raw data stored on a storage component. In one embodiment, the computing unit is a smartphone carrying the first machine learning system. In another embodiment, the phenotypic device includes a computing unit that further includes a central processing unit (CPU), a graphics processing unit (GPU), and other components required to perform computations necessary to carry the first machine learning system and process the raw data stored on the storage unit.

[0052] Optionally, the phenotyping device includes one or more mechanical support structures that enable movement by a human user, robot, or vehicle by, for example, transferring the weight of the camera unit to the mechanical support structure supported by the user. In one embodiment, the mechanical support structure is attached to an adjustable frame to which the canopy deployer and camera unit are attached. In another embodiment, the mechanical support structure is attached to the canopy deployer. Other components can move along with the phenotyping device as the user carries it via an axis. The mechanical support structure may include a handle to improve ergonomics.

[0053] In embodiments where the mechanical support structure includes a handle, the mechanical support structure is, for example, two bending axes. In this embodiment, the phenotypic device includes: a frame having one or more canopy deployers and a camera unit attached to the frame. The mechanical support structure, including the two bending axes and the handle, is connected to the frame via a first dual-axis connection. The mechanical support structure is also connected to the person carrying the device via a second dual-axis connection. The dual-axis connection ensures the device remains perpendicular to the ground as long as the person walks upright. The dual-axis connection can, for example, be connected to a body plate secured to the body using straps. The frame includes: a canopy deployer retaining axis; a camera unit retaining axis connected perpendicular to the canopy deployer retaining axis; a plurality of connecting rods for connecting the axes of the mechanical support structure, the canopy deployer retaining axis, and the camera unit retaining axis; and weights for balancing the momentum generated by the mass of the one or more canopy deployers, the camera unit, and the frame to improve the ergonomics of the person carrying the device. The one or more canopy deployers are attached to the canopy deployer retaining axis, and the one or more canopy deployers are parallel to and spaced apart from each other.

[0054] Optionally, the device is connected to an ergonomic support unit that reduces user load. In one embodiment, the ergonomic support unit is attached to an adjustable frame of the device. In one example, the ergonomic support unit includes a backpack unit that also includes backpack-style shoulder straps connected to a rigid backpack frame. Then, when the user carries the oriented device, the rigid backpack frame is connected to a first pole pointing towards the sky, which is parallel to the normal direction of an approximate tangential plane to the ground. A second pole is hinged to the first pole at a distance of 30 cm from end A of the second pole, and this hinge is connected to the sky-pointing end of the first pole. A strong elastic band is then connected to the rigid backpack frame and end A of the second pole. A rope is connected to end B of the second pole and to the backpack frame. By adjusting the length of the rope connecting the device to the second pole, the length of the second pole, and the length of the elastic band, the elastic band pulls end A of the second pole toward the backpack, inducing momentum through the hinge connector, which causes end B of the second pole to rotate toward the sky. In this way, because the weight of the watch device is transferred via the rods to the rigid backpack frame, the shoulder straps, and the shoulders of the person carrying the ergonomic support unit, the user requires less force to maintain the height of the watch device.

[0055] In another embodiment, the phenotyping device further includes a mechanical unit for controlling the height (i.e., distance to the ground) of the canopy deployer when the phenotyping device is moved within the target canopy. The mechanical unit may be connected to the canopy deployer, or to the means for controlling the angle and distance of the camera unit relative to the canopy deployer, or to the mechanical support structure. The mechanical unit includes a motion height controller and an optional auxiliary camera unit. The mechanical unit uses the output of the auxiliary camera unit and one or both of the raw data as input to estimate the canopy height. Then, based on the height specification and the estimated canopy height, the mechanical unit uses the motion height controller to control the height of the canopy deployer. The mechanical unit may optionally include other sensors that provide input for estimating the canopy height.

[0056] In one example, the height specification is used to maintain the contact point between the canopy spreader and the target canopy plant at a specified height, for example, 20 cm below the top of the tallest plant in the target canopy.

[0057] In another example, the height specification is used to maintain contact between the canopy spreader and the target canopy plant at a height of 20 cm below 95% of the plant height in the target canopy. In a third example, the height specification is used to maintain the canopy spreader at a height of 60 cm above the ground. In yet another example, the height specification is used to maintain contact between the canopy spreader and the plant that the canopy spreader contacts at each moment at a height of 70% of the plant height that the canopy spreader contacts.

[0058] Optionally, the mechanical unit includes an automatic calculation unit comprising a microcontroller or other device capable of performing calculations based on one or both of the inputs from the auxiliary camera unit and the raw data, and outputting a control sequence for a motorized height controller to adjust the height of the canopy expander to the height specification. The motorized height controller is driven by a stepper motor, servo motor, pneumatic system, or hydraulic system. Optionally, the mechanical unit uses only a subset of the raw data measurements.

[0059] The subset of electronic components requiring power includes one or more of the following: one or more camera units, storage components, optional one or more cooling units, computing units, electronic control units, one or more computing units, a set of other sensors, one or more height sensors, and one or more tension sensors. The subset of components requiring cooling includes electronic components whose performance may be affected by thermal conditions and require cooling to ensure reliable operation under hot summer conditions.

[0060] On the other hand, embodiments of this disclosure provide a phenotyping method for measuring phenotypic traits of one or more plants in a target canopy. The method includes the following steps: defining an input data specification for raw data collected from a current set of raw data collection targets by one or more camera units of a phenotyping device; defining a usage orientation-height specification; defining a device-specific device position angle specification; defining a usage movement specification; adjusting at least one parameter of a device for controlling the angle and distance of the one or more camera units relative to a canopy spreader according to the device-specific device position angle specification; initiating a raw data collection session; moving the phenotyping device, in partial contact with plants in the target canopy, according to the defined usage orientation-height specification and the usage movement specification, to expose the one or more plants for imaging while measuring their phenotypic traits; recording raw data of the one or more plants in the target canopy while moving the device in contact with the target canopy according to the usage orientation-height specification and the defined usage movement specification; and processing the recorded raw data to estimate the phenotypic traits of the one or more plants in the target canopy.

[0061] By defining the orientation and height of the phenotyping device relative to the canopy, the use of the phenotyping device at that orientation and height, and the direction of movement of the phenotyping device relative to the canopy, the orientation and height of the phenotyping device are defined. In one embodiment, the height parameter of the orientation and height specification is defined in a direction parallel to the planting rows of the wheat field, relative to an approximate tangential plane of the ground or growing medium under the canopy, and the direction in which the phenotyping device moves is such that the canopy spreader is perpendicular to the planting rows.

[0062] In another embodiment, the height parameter is defined in a direction parallel to the planting rows of the barley breeding plot, relative to an approximate tangent plane of the canopy's approximate top, and the orientation of the phenotyping device during movement such that the canopy spreader is at, for example, a 30-degree angle relative to the direction of the planting rows. The use of directional height specifications is tailored to the crop variety and planting practices.

[0063] In one example, the application orientation height specification for barley differs from that for soybeans. In one example, the canopy spreader is an axis. In this example, the application orientation height specification stipulates that the canopy spreader maintains an angle of 180 ± 10 degrees relative to the tangential plane of the approximate top of the canopy, moves in the direction of the tangential plane of the approximate top of the canopy, and is held at a height of 20 ± 5 cm below the approximate top of the target canopy, such that the direction of movement of the phenotyping device is parallel to the planting row, and the canopy spreader remains perpendicular to the planting row.

[0064] According to the device-specific device position angle specification, each of the one or more camera units is oriented and positioned relative to the one or more canopy expanders. The device position angle specification defines the distance and angle of each camera unit relative to the canopy expander when the phenotyping device is moved according to the use movement specification and the use orientation height specification, and that the canopy expander will be positioned relative to the current set of raw data collection targets at an approximately constant distance and orientation in each frame and each image of the raw data collected by the one or more camera units. That is, raw data is collected according to the input data specification. Thus, the current set of raw data collection targets is positioned relative to the camera unit according to the input data specification associated with each of the one or more camera units, wherein the input data specification is used to specify the device position angle specification. In other words, when the phenotyping device is positioned and oriented according to the use orientation height specification, and optionally moved in the canopy at a speed defined by the use movement specification along a direction specified by the use orientation height specification, the raw data recorded by the phenotyping device is recorded according to the input data specification, with each position, angle, and speed parameter within the tolerances specified in the respective specification. Equipment position and angle specifications vary depending on crop type, sowing or sowing practices (such as field width, canopy height, and other mechanical properties of the canopy). The equipment position and angle specifications for the camera unit define the orientation and distance of the camera unit relative to the canopy deployer; however, when the equipment is used according to other specifications, its relationship to the canopy deployer also indirectly defines its distance and angle relative to the current group's raw data collection target.

[0065] Furthermore, the orientation-height specification includes tolerances for deviations in orientation and height within which the raw data recorded by one or more camera units of the phenotyping device meets the input data specification. In one example, the orientation-height specification stipulates that the phenotyping device be moved at a height of, for example, 70 cm + / - 5 cm above the soil surface, where + / - 5 cm is the tolerance. In another example, the orientation-height specification stipulates that the phenotyping device must be moved at a height of 70% + / - 10% of the average height of the target canopy plants, where the tolerance is + / - 10% of the average plant height.

[0066] Meeting the input data specification ensures that the first machine learning system can process the collected raw data into estimates of phenotypic traits. Given the optical parameters of one or more camera units, the input data specification then indirectly specifies how far the one or more camera units should be from the current set of raw data collection targets in order to provide the machine learning system with a sufficient number of necessary pixels. In other words, the input data specification specifies how the current set of raw data collection targets must be recorded in the raw data. The input data specification can be defined based on the size of the object, the angle of the object, and the position of the object in the image / video frames of the raw data. Alternatively, when using phenotypic devices, the input data specification can be defined based on the properties of one or more camera units and their distance from the current set of raw data collection targets. For example, a camera unit with a 120-degree field of view must be positioned such that, during a data collection session, the distance from the camera unit to the current set of raw data collection targets is, for example, 30-150 cm, 100-140 cm, or 50-120 cm. The necessary number of pixels refers to, for example, the number of pixels needed to better identify an ear of grain belonging to one of the original data collection targets of the current group, such that an ear of grain with 400 pixels is better identified than one with 100 pixels. A sufficient resolution for calculating, for example, the number of seeds in a barley ear is <0.8 mm / pixel.

[0067] Input data specification can improve quality; that is, input data specification maximizes the use of pixels to collect raw data from the current group's raw data collection target, and thus maximizes the quality of information accumulation and trait estimation.

[0068] In one example, the plant part of interest is the spike-like inflorescence of a barley plant. In this example, in order to estimate the size of the spike from the raw data, the image portion containing the spike has a sufficient resolution, for example, 50 by 50 pixels in the first machine learning system, and the camera unit, as one of one or more camera units, records the raw data that will be used to estimate the size of the spike.

[0069] In this example, the length of the spikelet of the plant in the target canopy is greater than 30 mm, and the resolution of the camera unit is 3800 x 2000 pixels. In this example, the distance from the camera unit to the spikelet of the plant belonging to the current group of raw data collection targets is set such that the size of each pixel measured from the current group of raw data collection targets within the field of view boundary of the camera unit is less than 0.6 mm x 0.6 mm, so that for objects with a length greater than or equal to 30 mm, a resolution greater than 50 x 50 pixels is obtained, as this number of pixels is sufficient to accurately estimate the spikelet size. The actual distance parameter between the camera unit and the canopy spreader in the device-specific device position angle specification depends on the characteristics of the camera and lens, such as the camera's field of view.

[0070] In one embodiment, defining device-specific device position angle specifications includes: specifying the distance and angle of each camera unit relative to one or more canopy expanders and adjusting the distance and angle to parameter values ​​when the phenotypic device is moved according to the usage direction height specification and the usage movement specification, and using the parameter values ​​to collect raw data from the current group of raw data collection targets to meet the input data specification.

[0071] When the phenotyping device is moved, it is used to unfold the target canopy according to the usage orientation height specification to expose the raw data collection target and record the raw data. During the data collection session, the phenotyping device is moved within the target canopy according to the usage movement specification so that, as the one or more canopy unfolders move, the current set of raw data collection targets changes with the movement according to the manufacturing space-visible-covered-dynamics, and information is obtained from a wider portion of the target canopy. The phenotyping device is moved within the target canopy to record the raw data by a human user, robotic system, or vehicle, wherein the phenotyping device is mechanically connected to the robotic system or vehicle via a mechanical unit.

[0072] In one example, at a given time t0 during a data collection session, the current group of raw data collection targets includes: a group of plants in the target canopy at time t0 during the data collection session, to which one or more camera units are visible the rest of the plants in the current group of raw data collection targets due to the canopy spreading. In other words, the current group of raw data collection targets at time t0 includes: the plants exposed to one or more camera units at a given time t0 due to the canopy spreading.

[0073] In one embodiment, the current group of raw data collection targets upright plants that have been previously released from under the canopy spreader. In another embodiment, the current group of raw data collection targets both upright plants and plants that are currently upright after being released from under the canopy spreader.

[0074] It must be emphasized that when moving phenotyping devices within the canopy, raw data for the current group of raw data collection targets may be acquired at several time points before the current group's raw data collection targets are completely or partially covered by other plants.

[0075] Raw data includes video, still images, tension measurements, height measurements, spectral imaging data, acceleration measurements, and any other types of measurements obtained using one or more camera units, one or more tension sensors, height sensors, and a set of other sensors.

[0076] A data collection session is a period of time during which raw data is collected from the target canopy. In one example, a data collection session lasts 10–30 seconds. Optionally, the user uses one of one or more control units to start and end the data collection session.

[0077] When the phenotyping device is moved within the canopy or is being moved, the current raw data collection target will change with the movement according to the manufacturing space-seen-covered dynamic, and raw data will be obtained within a higher coverage area of ​​the target canopy. The manufacturing space-seen-covered dynamic includes the following iterative steps.

[0078] First, at time t1, the canopy deployer pushes aside the plant individuals in group B to expose the current group's raw data collection target at t1 to one or more camera units. This target corresponds to a plant individual belonging to group A. In other words, the canopy deployer is creating space for the line of sight from one or more camera units to the current group's raw data collection target.

[0079] Then, at time t2, the phenotyping device and its attached canopy spreader move and push aside the plant individuals belonging to group C. The plant individuals in group B that were earlier pushed aside by the canopy spreader have risen and become visible to one or more camera units, and have become the original data collection targets at time t2 because the parts of them that were invisible without their canopies unfolding are now visible to one or more camera units. Now, at t2, the plants in group B completely or partially cover the plant individuals in group A, and the plants in group A are no longer part of the original data collection targets for the current group.

[0080] In the next step, at time t3, the canopy expander pushes the plant individuals in group D aside to create space for observation of the individuals in group C, which have risen to be visible. The visible individuals in group C become the original data collection targets for the current group, while the plant individuals in group B are completely or partially covered by the individuals in group C.

[0081] Optionally, the phenotypic device is moved within the target canopy according to a movement specification. The movement specification includes a movement speed in a direction specified by a direction-of-use height specification. Furthermore, the movement specification includes speed tolerances such that when the speed of the phenotypic device is within the tolerance and the movement direction is within the directional tolerance specified by the direction-of-use height specification, the recorded raw data is within the tolerances of the input data specification, and the raw data is processed by one or more machine learning systems into one or more sets of outcome estimates, for example, using a first machine learning system and optionally a second machine learning system and optionally a third machine learning system. The machine learning system is used to process raw data from one or more data collection sessions into one or more sets of outcome estimates. Each of the one or more sets of outcome estimates corresponds to a set of estimates for one of the phenotypic traits.

[0082] In one example, when the speed is higher than the speed specified using the movement specification, plants pushed away from the target area covering the current group of raw data collection will not have time to rise back to an upright position during the time they are in the field of view of the camera unit, and therefore, these plants are not recorded in the raw data.

[0083] In one example, a movement specification is used to move the phenotyping device in the direction specified by the orientation height specification, such that the speed of the phenotyping device is 0.25 m / s–0.5 m / s. In this example, at this speed, the plant individual has time to rise after being pushed aside by the canopy spreader before becoming the target of the current group of raw data collection according to the manufacturing space-being seen-being covered-dynamics. The plant individual in the current group of raw data collection targets is positioned in the raw data according to the input data specification corresponding to the distance from the plant in the current group of raw data collection targets to one or more camera units, for example, at a distance of 120–145 cm from one or more camera units. Furthermore, the orientation height specification stipulates that the canopy spreader should be maintained at a constant height of 17–23 cm below the top of the canopy, and the canopy height is estimated by averaging the distance 30 cm before the position of the current group of raw data collection targets.

[0084] In another example, the canopy deployer is moved at a constant height of 25 cm below the top of the canopy, as specified using an orientation height specification. The canopy deployer moves forward an average of 4.5 cm between time points t1 and t2, the time between t1 and t2 being specified by a speed defined using a movement specification. At time point t1, the plant group in group B, creating space for the plants seen in group A, is released, and at time point t2, the plants in group B will completely or partially cover the plants in group A as seen by one or more camera units at t1. The movement speed is adjusted to allow sufficient time for the plants to rise. Experimental results show that a suitable speed is 0.6 m / s to 0.15 m / s.

[0085] Optionally, during a data collection session, raw data from several target canopies are recorded, and a third machine learning system is used to split the data collection session into several data collection sessions, each containing raw data from only one target canopy.

[0086] The first machine learning system is used to process raw data from one or more data collection sessions into one or more sets of outcome estimates. Each of the set of outcome estimates corresponds to a set of estimates for one of the phenotypic traits.

[0087] In one example, raw data from each data collection session is processed by a first machine learning system into an estimate of the number of ears in the target canopy. A set of outcome estimates includes an estimate of a phenotypic trait or other agronomic traits of interest associated with each target canopy, such as yield. Examples of phenotypic traits to be estimated for the target canopy include the number of ears, the number of seeds per ear, biomass of different plants, area estimates of disease and pest symptoms, estimates of plant part size, plant morphology, DUS traits, and gene pool characteristics. Phenotypic traits to be estimated by the first machine learning system and to be compiled into a set of outcome estimates include: various symptoms related to water and drought stress, symptoms related to nutrient deficiencies, and symptoms related to the harmful side effects of herbicides and pesticides. Phenotypic traits to be estimated also include the quantity, size, and species of other plants growing in the target canopy, such as weeds and predetermined species plants belonging to other varieties or genotypes.

[0088] Machine learning systems use deep neural networks or other machine learning models to transform inputs, including raw data from each data collection session, into outputs, such as estimates of phenotypic traits. In one embodiment, the machine learning system first processes estimates of individual still images extracted from videos in the raw data, and then aggregates the estimates for the entire data collection session. In one example, a data collection session contains raw data from a breeding test plot. In one embodiment, the machine learning system estimates the spatial distribution of yield components based on the raw data collected during the data collection session. In one embodiment, a set of outcome estimates includes the outputs of the machine learning system for each data collection session, i.e., estimates of agronomic traits and other traits of interest generated by the machine learning system based on the raw data. Optionally, the set of outcome estimates also includes the raw data associated with the data collection session.

[0089] Optionally, the method and system further include a second machine learning system. The second machine learning system operates by one or more computing units and takes as input one or more sets of outcome estimates related to one or more data collection sessions to generate one or more sets of outcome estimates based on the output of the first machine learning system, and outputs a second set of outcome estimates. The second machine learning system takes as input one or more sets of outcome estimates generated by the first machine learning system and related to the data collection sessions, and outputs a second set of outcome estimates. The second set of outcome estimates includes: a spatial error correction map and other scores, which can be calculated by aggregating one or more sets of outcome estimates from several data collection sessions.

[0090] A phenotyping system for measuring phenotypic traits of one or more plants in a target canopy using imaging-based estimation of phenotypic traits includes one or more computing units, in which computations are performed for a first machine learning system, an optional second machine learning system, and an optional third machine learning system to process raw data stored on a storage component. In one embodiment, the computing unit is a cloud server that can host the first machine learning system, the optional second machine learning system, and the optional third machine learning system. In another embodiment, the computing unit is attached to a canopy phenotyping device to be able to obtain multiple sets of result estimates in real time.

[0091] Optionally, the first machine learning system uses an orientation-height specification as input. Optionally, the first machine learning system uses one or more of the following as input: a device-specific device position angle specification and the distance between any components of the phenotyping device when each component has been positioned and oriented according to the device position angle specification. In one example, the first machine learning system uses the distance between one or more camera units and one or more canopy expanders as input. In another example, the first machine learning system uses the position and orientation of one or more height sensors and the distance from the height sensors to one or more camera units as input, or uses the dimensions of the components of the phenotyping device as input, wherein the distance between one or more camera units and the canopy expander can be calculated based on the component dimensions and the device-specific device position angle specification.

[0092] In one embodiment, the first machine learning system is configured to, when processing raw data into one or more sets of result estimates, assume in the computation that the raw data recorded by each of one or more camera units has been recorded according to camera unit-specific input data specifications. In one example, the input data specifications specify that one or more canopy expanders are seen at the bottom (minimum 35 pixels) of the image, and that the canopy expanders are aligned horizontally with the image; and that the length of the observed spike must be greater than 40 pixels. In another example, the input data specifications include angular and distance parameters of one or more camera units relative to the current set of raw data collection targets, their tolerances, and camera parameters of one or more camera units, both of which together define the size of the object in the current set of raw data collection targets in the raw data collected by one or more camera units. In another embodiment, by using the known physical dimensions of the object seen in the image (e.g., a canopy expander) as input, and by back-estimating the distance from the lens based on the observed pixel-level dimensions, the known dimensions, and the camera parameters, the first machine learning system estimates the angles and distances from one or more camera units to the canopy expander based on the raw data.

[0093] When the raw data has been recorded according to the input data specification, the plant parts in the image or video of the current group of raw data collection targets are recorded at a sufficient resolution. These parts are of interest for estimating the phenotypic traits associated with the plant parts, and all plants belonging to the current group of raw data collection targets are within the field of view of a specific camera unit responsible for acquiring images or videos of the plant parts of interest so that the first machine learning system can estimate the target phenotypic traits required for the current group of raw data collection targets.

[0094] In one example, the target canopy is a wheat breeding test plot with a width of 120 cm, the canopy deployer is a straight axis with a length of 140 cm, and the phenotyping device includes a camera unit. When the phenotyping device is moved, the canopy deployer moves in a direction approximately parallel to the ground surface and orthogonal to the planting rows, and the canopy deployer is oriented partially towards the ground and towards the centerline of the test plot, which includes points equidistant from the boundaries of the plot in the width direction. Within this centerline, the angle of the camera unit is measured based on a line from the center of the camera unit's lens to the center of the image.

[0095] In one embodiment, the breeding plot is 120 cm wide, but the current set of raw data collection targets only includes plants spaced 30 cm apart, with the midpoint of the interval located in the middle of the width of the breeding plot. In this embodiment, if the current set of raw data collection targets includes plants from the full 120 cm width, the camera unit can be positioned closer to the canopy spreader.

[0096] In one example, the input data specifications for the raw data collected by camera unit A (which is one or more camera units) are: the angle between the straight line from the center of the lens of camera unit A to the center of the field of view of camera unit A and the normal to the approximate soil surface, which is 26-45 degrees; and, in terms of the height dimension of the image, the spike of the plant must be contained in an approximate horizontal plane at the center of the image. Each of the one or more camera units has device-specific input data specifications. To further clarify, the image of the plant portion corresponding to the trait estimated from the video and image data must be visible and large enough in terms of pixels, which defines the maximum distance from the camera unit used to the plant portion corresponding to the trait of the current set of raw data collection targets. The mechanical properties of the plant in the target canopy determine the depth to which the canopy spreader can spread the plant into the target canopy. The spreading depth defines an angle from which one or more camera units can obtain an image or video of the plant portion of the current set of raw data collection targets. The projection angle of the recorded images and videos is the sum of the effects of the orientation height specification, the device position angle specification, and any fluctuations, orientation obstacles, and jitter that occur when using the phenotyping device. The projection angle must be kept within the tolerances specified in the input data specification so that the first machine learning system can properly process the raw data. In this context, "properly" means that the plant portion in the video or image recorded by the camera unit is located within the image area assumed by the first machine learning system to be the plant portion. In other words, when the phenotyping device is used to ensure that the resulting angles and distances are within the tolerances of the input data specification, the trait estimation is more accurate and a certain level of precision is guaranteed, for example, by measuring the angle and distance with a Pearson correlation greater than 0.85 between the estimated value and the manually measured ground truth. When the angles and distances are outside the tolerances, the first machine learning system may be unable to process the raw data at all or produce poor results. In another example, the machine learning system processes the raw data into an estimate of ear length, but the device position angle specification is poorly adjusted, and the ear is not visible in the collected raw data. The first machine learning system then cannot process the raw data into an estimate of ear length. Although the input data specifications for raw data are defined relative to the current group of raw data collection targets, when the phenotypic device is moved according to the usage orientation height specification and the usage movement specification, the angle of the camera unit specified in the device position angle specification is defined and adjusted relative to the canopy expander, and the distance and angle of the current group of raw data collection targets relative to the canopy expander will be fixed.

[0097] Optionally, device-specific device position and angle specifications are sought experimentally. In one example, a phenotypic device is used to record raw data from a breeding plot 120 cm wide, with a canopy spreader 140 cm on the axis, and the canopy spreader is moved along the approximate canopy surface in a direction perpendicular to the plot width, in a direction parallel to the plot width dimension, and at a height approximately 20 cm below the top of the canopy, according to the usage height specification. The user first adjusts the distance of a camera unit to the center of the canopy spreader so that the plants become visible when the canopy is spread across the entire width of the plot; these plants are the targets of the current group of raw data collection. The user then tests various movement speeds ranging from 0.1 m / s to 0.7 m / s and adjusts the position and angle of the camera unit relative to the canopy spreader for each speed so that the camera unit records raw data when the device is held according to the usage direction height specification and moved according to the usage movement specification, so that the portion estimated by the first machine learning system that is related to the phenotypic traits of the plants targeted by the current group of raw data collection is included in the images or videos recorded by the camera unit. In one embodiment, when adjusting the angle of the device position angle specification, the camera is oriented such that the canopy expander is located at the lower boundary of the camera image to maximize the area of ​​the current group of raw data collection targets in the image or video obtained by the camera unit.

[0098] Optionally, the user can experimentally search for height parameter values ​​using the orientation height specification. The optimal height parameter value is one that maximizes the canopy's spread without damaging the plants, opening the line of sight from one or more camera units to the plants at the current group's raw data collection target, thereby ensuring that the projection from the camera to the current group's raw data collection target is as orthogonal as possible. It is important to understand that plants typically grow along a direction approximating the ground normal, and the angle between the lines from each lens to the image center should be as perpendicular as possible to the growth direction corresponding to the ground normal.

[0099] It is important to emphasize that the height parameters of the orientation-height specification and the device-specific device position angle specification can be sought experimentally. The aim of the experiment is to find parameter values ​​that maximize the average number of pixels in the plant portion associated with the estimated phenotypic traits of the plant in the current set of raw data collection targets, where the plant portion in the current set of raw data collection targets is located in the portion of the image in the field of view of each camera unit, as assumed by the first machine learning system. Once the user finds parameter values ​​that provide sufficient resolution for the plant portion he / she wants to study, these parameter values ​​are used to specify the use of the orientation-height specification and the device-specific device position angle specification.

[0100] The input data specification is a function of the first machine learning system's requirements for input data, the size of the target canopy, and other attributes. The movement specification is a function of the target canopy characteristics, where the target canopy effectively sets a limit on the movement speed when collecting raw data; the movement specification includes a range of feasible speeds. The device is moved at a height and in a direction specified by the usage height direction. The device position angle specification defines the position and angle of the camera unit relative to the canopy expander to take into account the usage height direction and the usage movement direction, so that the raw data collected by the camera unit will be collected to satisfy the input data specification. The angle and distance between the camera unit and the canopy expander are controlled by adjusting one or more parameters of a device that controls the angle and position relative to the canopy expander.

[0101] In one embodiment, the device-specific device position angle specification is not changed during the data collection session. In other embodiments, the device-specific device position angle specification is modified during the data collection session, and the change in the device-specific device position angle specification is stored in a storage unit.

[0102] In another example, the user wants to estimate disease traits and symptoms located very close to the ground and is not interested in the top of the canopy. In this case, the camera is positioned closer to the target of the current group's raw data collection; although the top of the plant portion may be partially missed in the raw data, this provides a higher pixel resolution for disease symptoms.

[0103] When moving phenotyping equipment within the canopy, shutter speeds for one or more high-resolution camera units should be less than 1 / 3000 s, typically between 1 / 4000 s and 1 / 8000 s. ISO sensitivity and aperture are set to provide high sensitivity. If the part of the plant of interest is located close to the ground, less light is available, requiring higher sensitivity (or longer shutter speeds) to obtain high-resolution data.

[0104] In another example, the image quality is optimal for the plant portion near the top of the canopy when the shutter speed is 1 / 8000s. In this example, the optimal shutter speed for recording data from the bottom of the canopy at the same camera unit position is 1 / 4000s.

[0105] In one embodiment, several camera units are used simultaneously, and device-specific device position angle parameters and imaging parameters (shutter speed, ISO sensitivity) are selected, so that different camera units obtain images and videos of different parts of the plant from the current group of raw data collection targets.

[0106] In one embodiment, the phenotyping device is used to collect data for a plant breeding program. An experimental station has 500 experimental plots (each 10m x 1.2m in area), each corresponding to a target canopy. The phenotyping device is moved within the target canopy. The user initiates a data collection session at one end of the experimental plot using an electronic control unit, moves to the other end of the experimental plot while collecting raw data, and terminates the data collection session at the other end of the experimental plot using the electronic control unit. During the data collection session, the canopy expander makes a new set of raw data collection targets visible to the camera unit by expanding the canopy, so that all aboveground parts of the plants in the current set of raw data collection targets can be fully seen, except for occasional occlusion by other plants. As the user moves, the raw data collection targets continuously change, and raw data of all or almost all plant individuals in the target canopy are acquired. The raw data is stored in a storage component so that during the data collection session, the raw data is written to temporary storage, which in this embodiment is an external hard disk drive. The user initiates a separate data collection session for each experimental plot. When the phenotyping device is taken to a location with network connectivity (wireless or cable), it connects to the server storage component via a communication tool (e.g., the Internet) and transfers raw data from temporary storage to the server storage component. Once the raw data is stored in the server storage component, the first machine learning system takes the raw data from different data collection sessions as input and outputs estimates for the total count of spikes, the seed count in each spike, the total area of ​​disease symptoms, total biomass, and several DUS traits for each data collection session (corresponding to an experimental plot). In addition to obtaining trait estimates by averaging or summing across the data collection sessions, the first machine learning system also outputs a spatial map of the yield components for each experimental plot recorded during the data collection sessions. Furthermore, the first machine learning system calculates and outputs a yield measurement correction factor for each data collection session corresponding to the experimental plot by taking the spatial map of the yield components as input, thereby eliminating the influence of heterogeneity in the experimental plot.

[0107] The output of the first machine learning system serves as the input to a second machine learning system, which outputs a second set of result estimates. This second set of estimates includes a further spatial error correction factor. By considering less spatial variation within the fields, this spatial error correction factor makes yield measurements from different experimental fields corresponding to the data collection session more comparable. This less spatial variation cannot be observed by observing a single field and requires simultaneous aggregation of information from different experimental fields. The first and second set of result estimates are then provided to the user.

[0108] In one embodiment, the means for controlling the angle and distance of the camera unit relative to the canopy expander includes: a frame including a camera position holding unit, by means of which the distance and angle between the camera unit and the canopy expander can be fixed as parameters according to device-specific device position angle specifications.

[0109] In another aspect, one embodiment of the present invention provides a phenotyping system for measuring phenotypic traits of one or more plants in a target canopy, including at least one phenotyping device according to the present disclosure; means for moving the phenotyping device; and means for processing data generated by the phenotyping method, the means including one or more computing units.

[0110] In one embodiment, a phenotyping system for measuring phenotypic traits of one or more plants in a target canopy may further include one or more computing units that carry one or more machine learning systems to operate a first machine learning system configured to process data collected from a current set of raw data collected by one or more camera units of one or more phenotyping devices in a future free storage unit into one or more sets of result estimates.

[0111] The following is a detailed description of the attached diagram.

[0112] Reference Figure 1 The diagram shows a top view of a phenotyping device according to an embodiment of the present disclosure, wherein the phenotyping device includes: a canopy deployer 110 including two height sensors 120; a frame 130; a tension sensor 140 connecting the canopy deployer 110 to the frame 130 to measure mechanical resistance generated due to canopy deployment; a camera unit 150; and a camera position holding unit 160 connecting the camera unit 150 to the frame 130.

[0113] Reference Figure 2 This illustration shows a schematic diagram of a phenotyping device according to an embodiment of the present disclosure, illustrating the position of the phenotyping device relative to a specific plant within the canopy when viewed from the side at different time points. The phenotyping device includes a camera unit 250, a canopy deployer 210, and a frame. The frame has a first telescopic structure 230 for vertical distance adjustment and a second telescopic structure 205 for horizontal distance adjustment. The frame is moved in a direction 201 approximately relative to the tangent to the ground below the target canopy 222, according to a height direction specification, wherein the movement direction 201 of the phenotyping device, the approximate direction of the tangential plane of the ground below the target canopy 222, and the direction of the tangential plane of the approximate top of the canopy 272 are parallel 202.

[0114] The first telescopic structure 230 can adjust the distance between the camera unit 250 and the current group of raw data collection targets. A second telescopic structure 205 in the frame, used for horizontal distance adjustment, can be used to modify the horizontal distance between the camera unit 250 and the canopy spreader 210. Adjusting the vertical and horizontal distances between the camera unit and the canopy spreader 210 also adjusts the vertical and horizontal distances between the camera unit and the current group of raw data collection targets; therefore, the angle, distance, and orientation between the camera unit and the frame specified in the input data specification can be achieved. In this example, the entire area captured in the field of view 262 of the camera unit 250 shows the distance and angle of the camera unit 250 relative to the phenotyping device on which the selected plants can be recorded in the raw data.

[0115] When the phenotyping device is moved in direction 201 according to the height specification, the canopy is pushed down, and as the canopy plants are released from under the canopy spreader 210, the top of the canopy plants moves along trajectory 209, wherein the distance from the approximate top of the canopy to the height at which the phenotyping device pushes the canopy down is distance 207. When the phenotyping device is moved in direction 201 according to the height specification, the canopy spreader will move at a constant height in a direction parallel to plane 208, which is oriented to be parallel to an approximate tangential surface of the ground below the target canopy 222.

[0116] During the time interval from time point t0 to t8, when the phenotyping device is moved along the direction 201 specified by the height specification according to the speed specified in the movement specification, the positions of the selected plants in the target canopy relative to the phenotyping device at time points t0 to t10 are shown as positions 251, 252, 253, 254, 255, 256, 257, 258, and 259. At position 251, the selected plant has not yet been affected by the canopy spreader or by plants that have been displaced due to the influence of the canopy spreader on other plants. At positions 252, 253, and 254, the selected plant is tilted toward the ground when other plants affected by the canopy spreader push the selected plant toward the ground.

[0117] At position 255, as the device moves approximately 1 cm in the direction specified in the movement guidelines, the selected plant is released from under the canopy spreader. At position 256, the selected plant has been released from under the canopy spreader and rises along approximate trajectory 209 back to its normal upright position, but is not yet fully upright. At positions 257 and 258, the selected plant has been fully restored to its normal upright position and remains part of the original data collection target for the current group.

[0118] At position 259, the selected plant has fully risen back to its normal upright position, but is no longer part of the current group's raw data collection target because the selected plant's line of sight to camera unit 250 is blocked by other plants in the current group's raw data collection target.

[0119] The vertical distance of the selected plant relative to the phenotyping device is shown as distance 261, and the selected plant is part of the current group's original data target set.

[0120] The angle 224 between the canopy expander 210 and the camera unit 250 is defined as the angle between the canopy expander 210 and the line 204 formed from the lens center of the camera unit to the image center, and this angle 224 is adjusted according to the device position angle specification.

[0121] In one embodiment, line 204 can be used in the input data specification to define how the camera unit should be positioned and oriented relative to the current group of raw data collection targets. Normal 203 is the normal to the approximate surface of the ground beneath the target canopy. Optionally, angle 224 between normal 203 and line 204 can be used in the input data specification. Optionally, the device position angle specification specifies that the canopy expander is included in the field of view 262 of the camera unit 250 at the lowest 50 pixels of the image.

[0122] Figure 3 A top view of the orientation of the phenotyping device in the canopy is shown in one embodiment for measuring phenotypic traits of plants in a target canopy with region 376, wherein the plants in the target canopy in region 374 are the current group of raw data collection targets at the time of illustration when the device is moved in direction 301 parallel to the direction of the planting rows 322. The phenotyping device is moved toward direction 301 according to the orientation height specification and along a direction perpendicular to the direction of the planting rows. The two plants 324 in the rightmost row are not part of the current group of raw data collection targets, while the two plants 373 in the rightmost row are part of the current group of raw data collection targets. Figure 3 At the moment shown, the plants in subregion 374 of target canopy 376 are the raw data collection targets for the current group. The two plants 378 in the leftmost planting row have not yet been affected by the canopy expander 310, but as the canopy expander moves forward, the plants 378 will be affected by the canopy expander, and after moving further along the direction 301 specified by the height direction specification, the plants 378 will become part of the raw data collection targets for the current group.

[0123] The phenotyping device includes a frame 330 that connects a canopy spreader 310 to a camera position holding unit, which is further connected to a camera unit 350. From a top view, the camera unit 350 is oriented in the direction of the planting row and perpendicular to the canopy spreader 310. The acquired image includes region 374, which encompasses the current group's raw data collection target.

[0124] Figure 4 An example illustration of an image obtained by a camera unit from a target canopy is shown, with a center 480. In the target canopy, planting rows have been sown along direction 442, where phenotyping devices are moved along direction 401 (i.e., parallel to the planting rows) according to a height-direction specification. Three plants 482 in the rightmost row are in the field of view of camera unit 450, but are not part of the current group of raw data collection targets 484. Plant 486 in the rightmost row is part of the current group of raw data collection targets 484. Plant 487 in the leftmost row of the target canopy is located below the canopy spreader at the moment the image is captured, and the canopy spreader is pushing plant 487 aside to make room for the line of sight from the camera unit to the current group of raw data collection targets. Plants behind the schematic line 488 perpendicular to the planting rows are not part of the current group of raw data collection targets.

[0125] Figure 5 The illustration shows the storage and flow of raw data in one embodiment, wherein raw data is recorded by camera unit 550 and optionally other sensors. The raw data is stored in storage unit 590 and processed by a first machine learning system in computing unit 596. The machine learning system operating on computing unit 596 outputs one or more sets of result estimates 598. The storage unit includes temporary storage 592 and server storage component 594. When the phenotypic device operates under field conditions, the raw data is stored on an external hard drive. When the user returns to a more urbanized environment with a fast internet connection, the data is transferred from temporary storage to the server storage component via the internet.

[0126] Figure 6An exemplary embodiment of a phenotyping device according to the present disclosure is shown. In an embodiment, the phenotyping device includes a frame 630 having a first canopy deployer 610a, a second canopy deployer 610b, and a third canopy deployer 610c attached to the frame 630. The phenotyping device also includes a camera unit 650. A mechanical support structure includes two bending shafts 602a, 602b and a handle 603, and is connected to the frame via a first dual-axis connection 601a. ​​The mechanical support structure is also connected to a person carrying the device via a second dual-axis connection 601b. The frame 630 in this example includes: a canopy deployer holding shaft 630a; a camera unit holding shaft 630b perpendicularly connected to the canopy deployer holding shaft; a plurality of links 630c for connecting the shafts 602a, 602b of the mechanical support structure, the canopy deployer holding shaft 630a, and the camera unit holding shaft 630b; and a canopy deployer weight 630d. The canopy expanders 610a, 610b, and 610c are attached to the canopy expander retaining shaft 630a, and the canopy expanders 610a, 610b, and 610c are parallel to each other and spaced 20 cm apart.

Claims

1. A phenotyping device for measuring phenotypic traits of one or more plants in a target canopy (376), comprising: - One or more axes (110, 210, 310, 410, 610a, 610b, 610c) are configured to move parallel to the ground surface and contact the target canopy portion as the phenotyping device moves, to expose the one or more plants for imaging in the target canopy (376); - One or more camera units (150, 250, 350, 550, 650); - A means for controlling the angle and distance of the one or more camera units (150, 250, 350, 550, 650) relative to the one or more axes (110, 210, 310, 410, 610a, 610b, 610c), configured to keep the position and orientation of the one or more camera units relative to the axes fixed; and - An electronic control unit for controlling the camera unit.

2. The phenotypic device according to claim 1, wherein, The means for controlling the angle and distance of the one or more camera units includes an adjustable frame (130, 330, 630), the adjustable frame (130, 330, 630) includes a camera position holding unit, and the one or more axes are attached to the adjustable frame via an adjustable connector.

3. The phenotyping device according to claim 2, wherein, The adjustable frame also includes two telescopic adjustable structures (230, 205).

4. The phenotypic device according to any one of the preceding claims, wherein, The camera unit in the one or more camera units includes at least one of the following: one or more cameras, one or more spectral sensors, laser scanners, or lidar sensors.

5. The phenotypic device according to any one of claims 1 to 3, wherein, The device also includes at least one of the following: one or more tension sensors (140) or one or more height sensors (120).

6. The phenotypic device according to claim 2 or 3, wherein, The device also includes one or more mechanical support structures attached to the adjustable frame or to one or more shafts.

7. The phenotyping device according to claim 6, wherein, The device also includes a mechanical unit connected to the one or more axes, or to a device for controlling the angle and distance of the one or more camera units, or to the one or more mechanical support structures.

8. The phenotyping device according to claim 7, wherein, The mechanical unit includes a motorized altitude controller and one or more sensors for estimating the height of the canopy.

9. A phenotypic method for measuring phenotypic traits of one or more plants in a target canopy (376), the method comprising the steps of: - Define the input data specification for the raw data collected by one or more camera units (150, 250, 350, 550, 650) from the current group of raw data collection targets (256, 257, 258, 373) by the modeling device; - Define the direction and height specifications; - Define specific equipment position and angle specifications; - Define the use of the movement specification; - According to the specific device position angle specifications of the device, adjust at least one parameter of the means for controlling the angle and distance of the one or more camera units (150, 250, 350, 550, 650) relative to one or more axes (110, 210, 310, 410, 610a, 610b, 610c) to keep the position and orientation of the one or more camera units relative to the axes fixed; -Initiate the raw data collection session; -Move one or more axes of the phenotyping device that are in contact with one or more plant portions in the target canopy, according to the defined usage orientation height specification and the usage movement specification, for exposing the one or more plants for imaging when measuring the phenotypic traits of the one or more plants; - While moving the device in contact with the target canopy (376) according to the usage direction height specification and the defined usage movement specification, record the raw data of the one or more plants in the target canopy (376); and - Process the recorded raw data to estimate the phenotypic traits of the one or more plants in the target canopy (376).

10. The phenotyping method according to claim 9, wherein, Defining the input data specification includes: defining the angle parameters and distance parameters of the one or more camera units, and the tolerances of the angle parameters and distance parameters of the one or more camera units relative to the current group of raw data collection targets.

11. The phenotyping method according to claim 9 or 10, wherein, The defined orientation and height specification includes defining tolerances for deviations in orientation and height, within which the raw data recorded by the one or more camera units of the phenotypic device satisfies the input data specification.

12. The phenotyping method according to claim 9, wherein, Defining the device-specific position angle specifications includes: when the device is moved according to the usage direction height specifications and the usage movement specifications, specifying the distance and angle of each camera unit relative to the one or more axes according to the camera input data specifications and adjusting the distance and angle to parameter values, wherein the current group of raw data collection targets is positioned relative to the one or more camera units.

13. The phenotyping method according to claim 9, wherein, The usage movement specification includes: a movement speed in a direction specified by the usage direction height specification and a tolerance for the speed, such that when the speed of the phenotypic device is within the tolerance and the movement direction is within the directional tolerance specified by the usage direction height specification, the recorded raw data is within the tolerance of the input data specification, and the raw data is processed by one or more machine learning systems into one or more sets of result estimates.

14. The phenotyping method according to any one of claims 9, 10, 12, and 13, wherein, The method further includes: defining an azimuth and height relative to the target canopy, and a direction of movement relative to the target canopy, wherein the phenotyping device is used at the azimuth and height.

15. A phenotyping system for measuring phenotypic traits of one or more plants in a target canopy, comprising: - At least one phenotypic device according to any one of claims 1 to 8; - A means for moving the phenotypic device; as well as - A means for processing data generated by a phenotypic method, wherein the means for processing data generated by a phenotypic method includes one or more computing units (596).

16. The phenotyping system according to claim 15, wherein, The one or more computing units (596) carry one or more machine learning systems configured to process data from the current group of raw data collection targets into one or more sets of result estimates, wherein the data from the current group of raw data collection targets is collected by one or more camera units (150, 250, 350, 550, 650) in the at least one phenotypic device.

Images