Method and system for processing image data containing position data

Abstract

Systems and methods for processing image data of a crop are provided. One example computer-implemented method includes accessing image data specific to a corn plant, wherein the image data includes an image of the corn plant and depth data indicative of a range between the corn plant and a camera capturing the image; and identifying, by a computing device, features of the corn plant using the trained model, the features including ears of the corn plant and / or nodes from which the ears grow. The method further includes transforming, by the computing device, the coordinates specific to the feature to a height of the feature of the corn plant, and storing, by the computing device, the height of the feature of the corn plant in a memory.

Description

Cross-references to related applications

[0001] This application claims the benefit and priority of U.S. Provisional Application No. 63 / 529,327, filed July 27, 2023. The entire disclosure of the above application is incorporated herein by reference. Technical Field

[0002] This disclosure generally relates to methods and systems for processing image data of crops, wherein the image data includes relative position data, thereby enabling the determination of the location of specific features associated with the crop. Background Technology

[0003] This section provides background information relating to this disclosure, which is not necessarily prior art.

[0004] Crops are planted, grown, and harvested in different growing areas. During the breeding or testing of certain crops, different characteristics can be measured. For example, yield (in bushels per acre) can be determined at harvest as a performance indicator of the crop. Similarly, crop height, root strength, or other suitable phenotypic traits can be determined or measured before, during, or after the crop has grown in its growing area. Breeding can then be employed based on the measured data to develop crops with desired performance and / or phenotypic traits. Summary of the Invention

[0005] This section provides a general overview of this disclosure and is not a full disclosure of its entire scope or all its features.

[0006] The exemplary embodiments of this disclosure generally relate to a computer-implemented method for processing image data (including relative position data) of a crop, thereby determining the location of specific features associated with the crop. In one exemplary embodiment, the method typically includes: accessing image data specific to a corn plant, the image data including an image of the corn plant and depth data indicating the extent between the corn plant and a camera capturing the image; accessing image data specific to a corn plant, the image data including an image of the corn plant and depth data indicating the extent between the corn plant and the camera capturing the image; identifying features of the corn plant, including ears of corn and / or nodes from which the ears grow, using a trained model via a computing device; transforming the coordinates of the feature-specific features into the height (or size) of the corn plant feature via a computing device; and storing the height (or size) of the corn plant feature in memory via a computing device.

[0007] In another example implementation, a method for processing image data of a crop includes: accessing plant-specific image data, the image data including an image of the plant and depth data indicating the extent between the plant and a camera that captured the image; identifying features of the plant using a trained model via a computing device; transforming the coordinates of the features into dimensions of the plant features via the computing device; and storing the dimensions of the plant features in a memory via the computing device.

[0008] Example embodiments of this disclosure also relate to a non-transitory computer-readable storage medium including executable instructions for processing image data. In one example embodiment, such a non-transitory computer-readable storage medium includes executable instructions that, when executed by at least one processor, cause the at least one processor to perform the operations described herein.

[0009] Example embodiments of this disclosure also relate to systems for processing image data of crops, including relative position data, thereby determining the location of specific features associated with the crop. In one example embodiment, such a system typically includes at least one computing device configured to perform the operations described herein.

[0010] Further applicability will become apparent from the description provided herein. The descriptions and specific examples within the scope of this invention are intended for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0011] The accompanying drawings described herein are for illustrative purposes only, and not for all possible implementations, and are not intended to limit the scope of this disclosure.

[0012] Figure 1 An example system of this disclosure is shown, configured to process image data of crops, wherein the image data includes relative position data, thereby enabling the determination of the location of specific features associated with the crop.

[0013] Figure 2 Example image of a plot of land including corn crops, where several features of the corn crops were identified.

[0014] Figure 3 The example input image and the output data from the trained model appended (or superimposed) to the input image are used to identify corn nodes and / or ears with detection confidence, which can be determined by... Figure 1 The system provides; Figure 4 As shown Figure 1 The system provides indicators for variables used to transform coordinate data; Figure 5 To be able to Figure 1 A block diagram of an example computing device used in the system; and Figure 6 A flowchart of an example method is shown, which can be used (or implemented in) Figure 1 In the system, image data of crops is processed, including relative position data, thereby determining the location of specific features associated with the crops.

[0015] Throughout the various views of the accompanying drawings, corresponding reference numerals indicate the corresponding components. Detailed Implementation

[0016] Example embodiments will now be described more fully with reference to the accompanying drawings. The descriptions and specific examples included herein are for illustrative purposes only and are not intended to limit the scope of this disclosure.

[0017] For specific crop assessments, such as ear height (or ear size) of maize, the assessment is performed manually, with one person entering the planting area and measuring the ear height of each maize plant, recording the information in a log. Besides the labor intensity and lack of scalability of the manual measurement process, the accuracy and objectivity of the assessment can be reduced due to the subjective nature of the person performing the assessment, resulting in distorted data and / or potentially introducing errors into subsequent decisions or processes that rely on that data. Prior to this disclosure, providing accuracy and objectivity while also offering scalability and reducing and / or eliminating manual processing went beyond conventional techniques.

[0018] The unique aspect is that the system and method in this paper process crop image data, which includes relative position data, enabling the determination of the location of specific features associated with the crop. For example In some implementations, it is automated; in some implementations, it is independent of and / or contains no and / or requires no human measurement and / or evaluation; etc.

[0019] Figure 1 An example system 100 is shown, in which one or more aspects of this disclosure may be implemented. While system 100 is presented in one arrangement, other embodiments may include components (or additional components) of system 100 arranged in other ways depending on, for example, the source and / or type of image data, field arrangement, type of capturing device for capturing images of field crops, type of field crops, etc.

[0020] exist Figure 1 In an example implementation, system 100 typically includes a computing device 102 and a database 104, the database being coupled to (and / or otherwise communicating with) the computing device 102, as indicated by the arrow line between them. Figure 1 In this embodiment, computing device 102 and database 104 are shown separately, but it should be understood that in other system implementations, database 104 may be included in whole or in part in computing device 102.

[0021] System 100 also includes fields 106, which include rows of crops ( For example , Figure 1 (The four rows in the image are arranged in a circle, with each corn plant indicated by a circle.) Figure 1 There are limited illustrations, but field 106 typically covers several acres ( For example 1 acre, 10 acres, 50 acres, 100 acres or more or less, etc., and crop rows are spread throughout the field 106 (making field 106 typically include multiple rows). For example (More than four rows, etc.) (but this is not required in all implementations, and thus in some implementations, field 106 may include fewer than four rows, etc.)). It should also be understood that field 106 can be any planting area, from small experimental plots to greenhouses to large commercial areas covering several acres, etc.

[0022] In addition to the above, it should be further understood that although field 106 is described as including multiple rows of corn, other plants and / or other varieties or types of plants may also be included in field 106 or other fields in system 100.

[0023] In other words, Figure 2 An example corn plant 202 is shown, comprising a stalk 204, supporting roots 206, and a corn ear 208. As further shown, the ear 208 is illustrated relative to nodes A, B, and C. Node A represents a node of the corn ear 208, while node B represents the base of the corn ear 208, and node C represents the top of the corn ear 208. As indicated, each node has a certain height relative to the supporting roots 206 or the ground from which the corn plant grows. For example, the ear height may include the distance between the node from which the ear grows and the ground (…). Right now Figure 2 The distance from the ground to node A). Alternatively, ear height may include the size of the maize ear, such as the length from the node to the tip. It should be understood that Figure 1 Each corn plant shown has the same nodes in each ear of corn.

[0024] Refer again Figure 1 The system 100 further includes a surveyor's computing device 108. Figure 1In this example embodiment, the surveyor computing device 108 is shown separately from the computing device 102 and the database 104. That is, it should be understood that in one or more other embodiments, the surveyor computing device 108 may include all or part of the computing device 102 and / or the database 104.

[0025] As shown, in this embodiment, the surveyor calculation device 108 is configured to move between rows of corn in field 106. Therefore, the surveyor calculation device 108 defines dimensions, and particularly widths compatible with field 106 and other fields, as well as the plants and / or rows in field 106. The surveyor calculation device 108 includes a base 110, which in turn includes tracks, wheels, etc. (broadly speaking, a transport mechanism or device) to enable the surveyor calculation device 108 to move between and / or along two (or more) rows of corn in field 106. For example (e.g., mobile device 108). In other example embodiments, the surveyor's computing device 108 may be fixed in the field 106. For example (where the field includes greenhouses, etc.), and the corn plants can be moved by a fixed measuring and calculating device 108 (e.g., where the corn plants are located on a conveyor belt in a greenhouse, etc.).

[0026] In this example embodiment, the surveyor computing device 108 also includes a plurality of cameras 112 mounted on a bracket 114 extending upward from the base 110 of the surveyor computing device 108. Furthermore, in this example, each camera 112 is mounted on the bracket 114 at a specific height from the ground. For example Figure 1 medium height h (etc.), wherein the height is known to the surveyor's calculation device 108. For example... Figure 1 As shown in the detailed diagram, in this example embodiment, the surveyor's computing device 108 includes three cameras 112, each positioned at a different orientation relative to the bracket 114, thereby defining different angles relative to the bracket 114. In this manner, one camera 112 is generally pointing downwards, one camera 112 is generally pointing horizontally, and one camera 112 is generally pointing upwards. For example, the cameras 112 can be positioned at sufficient angles to provide sufficient overlap between the captured images, such as, for example, about fifteen degrees, about thirty degrees, about forty-five degrees, etc. In a particular instance, the cameras 112 can be positioned at different angles relative to each other, specifically including a central camera 112 arranged at an angle of about 90 degrees to the long axis defined by the bracket 114, and two other cameras 112, in this embodiment, defining angles offset from the central camera 112 by about ±35 degrees. For example(e.g., one tilted upwards at approximately 35 degrees and the other tilted downwards at approximately 35 degrees). That is to say, it should be understood that in other instances, cameras 112 may be arranged at other angles, for example, relative to each other, and / or relative to the support 114, the ground, crops, etc. Therefore, cameras 112 are configured to capture images of corn crops in field 106, and typically capture almost all of the corn crops ( For example (From the supporting roots to the tip of the ear / tassel, etc.).

[0027] In addition, in this embodiment, each of the cameras 112 is configured to capture the color spectral data and depth of the object. Right now Camera 112 is a depth camera or a stereo vision camera, etc. An example depth camera used herein is the Intel RealSense depth camera. Each camera 112 may include, for example, an Intel RealSense™ D435i or ZED 2i depth camera, which provides RGB plus depth information and inertial measurement unit (IMU) data as output. The depth information and IMU information can be used to detect approximately 3-10 degrees of freedom, or more specifically, approximately 6 degrees of freedom (or 6DoF) of movement and rotation. Furthermore, in this example embodiment, as explained above, each of the cameras 112 is oriented to provide approximately 69 degrees of horizontal field of view and approximately 112 degrees of vertical field of view (approximately 7 degrees of overlap between cameras 112). It should be understood that in other embodiments, the horizontal and vertical fields of view may differ.

[0028] In addition, such as Figure 1 As shown, camera 112 is a certain distance away from an example plant 116 in field 106. For example , Figure 1 Distance in d (etc.), thus the camera 112 is configured to capture distance d As an object ( Right now The depth of the plant 116 and an image of the plant 116. In other embodiments, the surveyor computing device 108 may include additional sensors to determine the distance between the camera 112 and the plant 116.

[0029] In conjunction with the above, the surveyor computing device 108 is also configured to capture inertial measurement unit (IMU) data via one or more sensors. The IMU data includes angular velocities and specific force / accelerations collected by the surveyor computing device 108 as it traverses crop rows in the field 106. In this example embodiment, the one or more sensors include a camera 112 ( For exampleThe internal IMU (such as that in the INTEL RealSense™ D435i camera, etc.) includes a BOSCH BMI055 sensor with a 62.5 Hz accelerometer and a 200 Hz gyroscope data rate.

[0030] In the illustrated embodiment, although the surveyor computing device 108 includes three cameras 112 facing the same direction to capture images of a row of crops in field 106, in other embodiments, the surveyor computing device 108 may include a different number of cameras, for example, including additional cameras facing in the opposite (or different) direction from the cameras 112 to capture image data of adjacent rows of crops, thereby enabling the simultaneous capture of images of two rows of crops in field 106. In still other embodiments, the surveyor computing device may be configured to be stationary or fixed in position, or mounted on a static structure. For example An arched structure to facilitate the desired positioning of the camera 112 as described herein, thereby identifying the plant by scanning a barcode, QR code, or other identifier. For example The plant is placed on a tray, conveyor belt, trolley, track, etc., to identify the plant (compared to the aforementioned location), and then the plant is moved relative to the measuring device, thereby capturing an image of the plant. In this example, a fixed measuring device may be more suitable for a greenhouse, while a mobile measuring device may be more suitable for fields, etc. That is to say, it should be understood that the description herein applies to a fixed measuring device used to determine the size of plants therein, although the relative movement is reversed.

[0031] Continue to refer to Figure 1 In this example embodiment, the surveyor computing device 108 includes a tracking sensor 118, which is configured to... For example Location is detected and / or determined under GPS denial conditions, via WIFI signals or other technologies, because the surveyor's computing device 108 moves and / or in instances of captured images. The detected location data can be represented, for example, as GPS time coordinates. For example (e.g., latitude, longitude, time / date) or other forms. The surveyor's computing device 108 can be further configured to associate or link location data with images captured by camera 112 (and other cameras) and / or associated depth data. Thus far, and as will be described in detail below, the tracking sensor 118 can optionally aid in image deduplication performance. When performed, deduplication allows the computing device 102 to limit the image to aid in single counting of corn ears, for example, based on feature tracking and identity assignment via the tracking sensor 118.

[0032] It should be understood that the surveyor computing device 108 may include other sensors, one or more of which may be configured to detect and / or determine various aspects of the corn plant 116 or field 106, the environment, and / or the surveyor computing device 108, etc.

[0033] As part of the foregoing, it should also be understood that the surveyor's computing device 108 is communicatively coupled to the computing device 102 via one or more networks (as indicated by the dotted lines). These one or more networks may include, but are not limited to, a local area network (LAN), a wide area network (WAN), etc. For example The network may include the Internet, mobile networks, virtual networks, and / or another suitable public and / or private network, or a combination thereof, capable of supporting communication between the surveyor computing device 108 and the computing device 102. In a particular instance, one or more networks may include a mobile network associated with a mobile network hotspot, which provides communication between the surveyor computing device 108 and the mobile device (…). For example Real-time communication between the user and their associated tablet or other suitable computing device.

[0034] In light of the foregoing, during crop cultivation in field 106, a surveyor computing device 108 is configured to periodically move through field 106 and capture images of corn plants 116 via camera 112. These images, combined with depth data (depth data of camera 112 relative to plants 116), IMU data, and location data that may be used by the surveyor computing device 108, are collectively referred to herein as image data. In one or more embodiments, the surveyor computing device 108 is further configured to compile image data from camera 112 and various sensors and transmit the image data to computing device 102, which is in turn configured to store the image data in database 104. The image data may be compiled and transmitted at one or more regular or irregular intervals, or at the end of a line, at the end of field 106, or otherwise. The image data may be stored specifically for field 106 as needed or required, or relative to specific plants therein, or otherwise.

[0035] In one or more embodiments, the surveyor computing device 108 is further configured to compile image data from the camera 112 and various sensors and store the compiled images therein. In such embodiments, the images are at least initially retained in the surveyor computing device 108, wherein the computing device 102 (and database 104) are at least partially included in the surveyor computing device 108, thereby adapting the configuration of the computing device 102 to the surveyor computing device 108. Furthermore, in this example embodiment (where the computing device 102 (and database 104) are at least partially included in the surveyor computing device 108), download and upload times associated with transmitting images are avoided or at least delayed to accelerate the processing described herein. For example (This involves real-time or near-real-time processing, etc.)

[0036] In addition to image data, it should be understood that database 104 may also include data related to field 106 and plants 116 within field 106. This data may include specific plant types, varieties, and genetic characteristics. For example (such as superposition of traits), management practices ( For example ,deal with( For example (fertilizers, fungicides, etc.), irrigation, cultivation, etc.), unique identifiers for each plant, and location data of specific plants, etc.

[0037] In this example embodiment, the computing device 102 (either as a standalone computing device or in the form of being included in the surveyor computing device 108) is then configured to access image data in the database 104 to identify ears of corn in images of corn plants 116 from the field 106 and to determine the height associated with the ears of corn of the identified corn plants.

[0038] In light of the above, the computing device 102 is configured to preprocess the image data from the surveyor's computing device 108 (for each accessed image), specifically, for example, to correct the orientation of a given image. For example(Image flipping, rotating, etc.). In particular, for example, computing device 102 can be configured to use a real-time video stream from camera 112 to determine the correct orientation of the data capture. This can be further utilized with IMU data to perform angular momentum and linear momentum calculations from the origin. That is, preprocessing of image data can include other operations, such as, for example, blurring the image based on depth data associated with the image or otherwise altering the image. For example, given that the corn plants are known to be 24 inches away from the surveyor computing device 108, and particularly from camera 112, computing device 102 can be configured to blur images beyond a depth of 30 inches from camera 112 or some other threshold, to limit the background in one or more models mentioned herein. Preprocessing can also include stitching and / or aggregating images from camera 112, and / or stitching and / or aggregating different images based on location or capture time, so as to provide a single image of a row of plants 116 as the surveyor computing device 108 traverses the field 106. Alternatively, the original image data can be aggregated or referenced together based on the timestamps associated with the data. In a specific instance of such aggregation, the original image data could have The .bag format allows the use of ROS (Robot Operating System) to capture data from multiple sensors together and package the data with corresponding timestamps (in words). (.bag format). Then, the packaged data can be converted based on the type of sensor used to collect the data. .jpeg .ply、 .txt and other formats.

[0039] Images used for training ( For example As part of preprocessing (such as training images), the computing device 102 is configured to prompt the user to annotate the image, for example, identifying the corn ear and the nodes growing from it. The corn ear and nodes are defined by bounding boxes applied by the user to the image. For example Recognition is achieved through mouse input, touchscreen input, etc. Specifically, the computing device 102 is configured to display an image to the user and allow the user to draw corresponding bounding boxes on the ear / node, and to label the boxes. For example (e.g., "ear of grain", "joint"). The computing device 102 can be configured to prompt a user or multiple users to label one hundred, three hundred or more or fewer images, thereby defining an image training set. It should be understood that a second user can verify the bounding boxes and / or labels before or after adding images to the image training set.

[0040] In this example implementation, the computing device 102 is then configured to train a suitable model to perform ear and / or node detection based on the image.

[0041] Relatedly, computing device 102 can be configured to train a convolutional neural network (CNN) model using a portion of the training set. Training involves exposing the input image, along with the output bounding box and label, to the model while tuning the CNN's parameters so that it can produce output from the input. After training is complete, computing device 102 can then be configured to feed the image to a set of validation images (…). Right now The training set (which does not include a reference bounding box / label) is used to attach bounding boxes and / or labels, and the model is validated by matching the attached bounding boxes / labels with the previously labeled boxes / labels of the image (and / or by manual inspection of the image, bounding boxes, and labels, etc.). Once the model is trained and validated... For example When providing accuracy threshold levels, etc., the computing device 102 can be configured to store the trained model in the database 104 for use as described below.

[0042] In this example implementation, specifically, the CNN model includes the YOLOv5 (You Only Look Once Version 5) model. For example YOLOv5 (YOLOv5x, YOLOv5x6, etc.) (or other versions of YOLO). The YOLOv5 model uses a convolutional neural network (CNN) for object detection, which is configured to detect multiple objects, predict categories, and identify the location of objects. In one instance, the configuration includes 53 CNN layers (Darknet53), which are then stacked 53 more layers to generate 106 layers. Detection can then be performed at layers 82, 94, and 106. The model further uses convolutional layers instead of pooling layers, which may suppress the loss of low-level features.

[0043] More generally, a YOLOv5 network can comprise three parts: a backbone network, which comprises CNN layers of varying scales aggregating image features; a neck network, which comprises a set of layers for combining image features and passing them to prediction; and a head network, which extracts features from the neck and performs localization and classification. As part of the foregoing, in one instance, the input image can have any size and / or aspect ratio; bounding box attributes can include the input (416, 416, 3), one or more deep CNNs of the YOLOv5 model, downsampling of the input image with a stride of 32, and a (13, 13, 255) feature map at a scale of 1. Furthermore, training the YOLOv5 model can include: one or more ground truth bounding boxes for one (or more) objects, center cells specified for predicting one or more objects, and cell target confidence of 1. Additionally, detection can be performed at layers 82, 94, and 106 by applying 1x1 convolutional kernels to the downsampled image, where the feature maps are (13x13), (26x26), and (52x52).

[0044] It should be understood that the model may be retrained or updated at one or more intervals based on additional training data such as the same or different types or varieties of crops.

[0045] In this example implementation, after the model has been trained, validated, and stored, computing device 102 is configured to access image data from database 104. For example (e.g., captured by camera 112 and preprocessed as described above), and the image is provided to a trained model, whereby the computing device 102 is configured by the trained model to attach bounding boxes to images of corn ears included in the images. For each image, one or more bounding boxes may include corn ears and / or may include nodes of corn ears. For example, Figure 3 An example input image is shown with bounding boxes appended by a trained model. The bounding boxes include the bounding box for each visible ear of corn in the image, and the bounding box for each node of each visible ear of corn. Additionally, as shown, the computing device 102 is configured by the trained model to append bounding boxes as needed. Right now In this example, labels and / or classifications are applied to the corn ears and nodes respectively.

[0046] It should be understood that in other implementations, the model may apply different bounding boxes for ears and / or nodes or apply different labels.

[0047] In one or more embodiments, detection is based on the overlap between the ear and the node. That is, a maize plant may potentially include many nodes, but not all nodes are locations from which the ear of corn grows. Therefore, detecting the ear and / or node includes detecting ears and nodes that overlap at least some extent to confirm the detection of the ear of corn. In this context, nodes are identified to determine the height or other dimensions of the ear of corn relative to the plant 116 and / or the ground.

[0048] By identifying the corn ear and nodes in the image, the computing device 102 is then configured to determine a specific height of the ear and / or node based on the image data. In this example embodiment, the computing device 102 is configured to identify the location of the corn node, such as the center pixel or center coordinates of the node's bounding box (which overlaps with the ear's bounding box). It should be understood that in other embodiments, another aspect of the bounding box of the corn node or ear can be identified as the "location" of a feature (e.g., ， Topmost pixel / coordinate, bottommost pixel / coordinate, average pixel / coordinate, etc.). The computing device 102 is then configured to access the depth data of the image provided to the training model and identify points with bounding boxes such as ears of grain or nodes. For example, Depth data associated with the top, bottom, etc. The computing device 102 is also configured to access specific data from the surveyor's computing device 108, including, for example, inherent data from the camera 112, the position of the camera 112, etc. For example (Including the height of the camera above the ground, the distance between the camera and the planted area, etc.) and other suitable data.

[0049] The computing device 102 is then configured to transform the coordinates of the points using depth data. In this example, x can represent the two-dimensional pixel coordinates in the image, and X can represent the three-dimensional coordinates of the ear or node (relative to the surveyor computing device 108), while K is the camera intrinsic matrix that indicates various aspects of the camera 112. This is given in the expression below.

[0050] refer to Figure 4 The focal length is represented by Fx and Fy, and the camera center is Cx and Cy, with an axis tilt of S. Based on these variables, the computing device is configured to transform the coordinates of the points in the bounding box into 3D camera coordinates.

[0051] Next, in this example embodiment, computing device 102 is configured to further transform the three-dimensional camera coordinates from the camera origin to the world origin based on the following expression, wherein the R matrix includes rotations as needed for the image, and the T matrix includes translations describing the movement of camera 112 during image capture.

[0052] Finally, the computing device 102 is configured to determine the ear height based on the transformed coordinates.

[0053] The calculation device 102 (either as a separate calculation device or included in the surveyor calculation device 108) is configured to output ear height and / or node height. For example, refer to Figure 3 The computing device 102 can be configured to attach ear height and node height to the labels of the bounding box. As shown, in this example, ear height can be measured in inches (or other units) and can include confidence values. It should be understood that height data can be output in different forms and / or otherwise presented to the user as needed or required.

[0054] It should be understood that when multiple heights share a relatively consistent location ( For example When based on GPS coordinates, etc., the computing device 102 can be configured to discard heights (or other dimensions) closer to the ground, indicating that the ears belong to the same plant, thereby retaining only the height of the highest ear.

[0055] Furthermore, considering the above, it should be understood that maize plants can be captured in multiple images. Therefore, the computing device 102 can be configured to potentially perform deduplication based on images or ear height, such as plant location. In this embodiment, deduplication allows the computing device 102 to further count maize ears based on feature tracking and identity assignment. Counting utilizes images to accurately count ears to ensure reliable phenotypic measurements. The method can be extended to different traits and crops (…). Right now (This is not limited to ear height or count), which may further ensure or improve the accuracy of classification. Deduplication methods may include, for example, simple online real-time tracking (SORT), BDP-Tile using GPS measurements, etc.

[0056] In one implementation, the computing device 102 may be configured to average the ear heights of the same plant, such as those determined from different images, thereby retaining only one value for the ear height of the plant. It should be understood that in other system implementations, other techniques may be employed to deduplicate the ear height data (or images).

[0057] While the above description is provided with reference to a corn ear or a node extending from it, it should be understood that other objects associated with the corn plant can be identified and their relative positions determined. For example, during pollination, bags can be placed over the corn ear. The calculation device 102 (alone or as part of or in combination with the measuring calculation device 108) can be configured to perform the above operations to determine the height or other dimensions of the bags applied to the corn plant (particularly the corn ear). In addition to dimensions, the count of bags per plant, or the count of bags per seedbed in a greenhouse, or the number of bags / ears at each location, can also be determined.

[0058] Additionally, while the above description is provided with reference to a surveyor's computing device 108 configured to move through field 106 to capture images of plants 116 in field 106, it should be understood that the features described herein can also be implemented in other devices. For example, in one example embodiment, the combination may include a camera 112 and a tracking sensor 118 mounted thereon. For example (e.g., coupled to or mounted on the front portion of the assembly), its configuration and / or arrangement are similar to those described above. The assembly is then configured to move through the field 106 in the manner generally described above with respect to the surveyor's calculation device 108 (therefore, in some respects, the assembly can be considered (or can represent)). Figure 1 The measuring device 108 shown. In this way, when the group passes through the field 106 to harvest the plants 116, etc., data can be collected from the field 106 (via the camera 112 and the sensor 118, etc.).

[0059] Figure 5 It shows that it can be used Figure 1 The system 100 includes an example computing device 500. The computing device 500 may include, for example, one or more servers, workstations, personal computers, laptop computers, tablet computers, smartphones, virtual devices, etc. Alternatively, the computing device 500 may include a single computing device, or it may include multiple computing devices located adjacent to or distributed across a geographical area, provided that the computing devices are specifically configured to operate as described herein.

[0060] exist Figure 1 In an example implementation, computing device 102 and / or measuring computing device 108 ( For example The system 100 (including camera 112, sensors, etc.) includes one or more computing devices consistent with and / or implemented therein, similar to computing device 500. Database 104 can also be understood to include one or more computing devices at least partially consistent with computing device 500 and / or implemented therein. However, as described below, system 100 should not be limited to computing device 500, as different computing devices and / or arrangements of computing devices can be used. Furthermore, different components and / or arrangements of components can also be used in other computing devices.

[0061] like Figure 5 As shown, the example computing device 500 includes a processor 502 and a memory 504 coupled to (and in communication with) the processor 502. The processor 502 may include one or more processing units ( For example(In multi-core configurations, etc.). For example, processor 502 may include, but is not limited to, a central processing unit (CPU), a microcontroller, a reduced instruction set computer (RISC) processor, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a gate array, and / or any other circuitry or processor capable of implementing the functions described herein.

[0062] As described herein, memory 504 is one or more means that permits the storage and retrieval of data, instructions, etc. Relatedly, memory 504 may include one or more computer-readable storage media, such as, but not limited to, dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), solid-state devices, flash drives, CD-ROMs, thumb drives, floppy disks, magnetic tapes, hard disks, and / or any other type of volatile or non-volatile physical or tangible computer-readable media, to store such data, instructions, etc. Specifically, memory 504 is configured to store data, including but not limited to image data (…). For example Images, depth data, etc.), models, parameters, phenotypic data, and / or other types of data (and / or data structures) suitable for the purposes described herein.

[0063] Furthermore, in various embodiments, computer-executable instructions may be stored in memory 504 for execution by processor 502 to cause processor 502 to perform one or more operations described herein that are connected to various components of system 100. For example (such as one or more operations of method 600, etc.), making memory 504 a physical, tangible, and non-transitory computer-readable storage medium. Such instructions generally improve the efficiency and / or performance of processor 502 performing one or more of the various operations described herein, thereby enabling computing device 500 to be transformed into a dedicated computing device. It should be understood that memory 504 may include various different memories, each implemented in conjunction with one or more functions or processes described herein.

[0064] In an example implementation, computing device 500 also includes components coupled to processor 502. For example The output device 506 represents and communicates with the computing device 500 (such as a unit, etc.). The output device 506 can output information (visually or otherwise) to the user of the computing device 500 (such as a researcher, grower, etc.). For example (e.g., bounding box, ear height, node height). It should be further understood that various interfaces can be displayed or otherwise output on the computing device 500 and the output device 506. For exampleAn output device 506 is an interface defined by web-based applications, websites, etc., to display or present certain information to a user. The output device 506 may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic LED (OLED) display, an "electronic ink" display, a speaker, a printer, etc. In some embodiments, the output device 506 may include multiple devices. Alternatively or additionally, the output device 506 may include printing capabilities, enabling the computing device 500 to print text, images, etc., on paper and / or other similar media.

[0065] Additionally, the computing device 500 includes an input device 508 that receives input from a user. Right now User input (e.g., image separation metadata, data collection annotations, crop selection, images, identification of spikes / nodes, etc.) can be provided. Input device 508 may include a single input device or multiple input devices. Input device 508 is coupled to (and communicates with) processor 502 and may include one or more of, for example, a camera, keyboard, pointing device, touch-sensitive panel, or other suitable user input device. It should be understood that in at least one embodiment, input device 508 may be integrated with and / or included with output device 506. For example Touchscreen displays, etc.

[0066] Furthermore, the computing device 500 shown also includes a network interface 510 coupled to (and communicating with) the processor 502 and the memory 504. The network interface 510 may include, but is not limited to, a wired network adapter, a wireless network adapter, a mobile network adapter, or an adapter capable of communicating with one or more different networks. For example Local Area Network (LAN), Wide Area Network (WAN) For example Other devices that communicate with computing device 500 via one or more of the following: the Internet, mobile networks, virtual networks, and / or another suitable public and / or private network, wherein the one or more different networks include those capable of supporting communication between computing device 500 and other computing devices. For example One or more networks or other suitable networks (between computing device 102, database 104, etc.) including wired and / or wireless communication with other computing devices used as described herein.

[0067] Figure 6An example method 600 for processing image data of crops, including location data, is shown to determine the location of specific points associated with the crops. This document describes the example method 600 as being connected to system 100 and can be implemented wholly or partially in computing device 102 of system 100, which can be implemented as a stand-alone computing device or as included in a surveyor computing device 108 connected to method 600. Furthermore, for illustrative purposes, reference is also made to… Figure 5 The example method 600 is described using computing device 500. However, it should be understood that method 600 or other methods described herein are not limited to system 100 or computing device 500. And conversely, the systems, data structures, and computing devices described herein are not limited to example method 600.

[0068] At point 602, the surveyor's computing device 108 captures image data (such as) associated with the plants 116 in the field 106. Figure 1 (As shown). The image data includes one or more images of the corn plant 116, the position of the surveyor computing device 108 when capturing the one or more images, and depth data also captured by the surveyor computing device 108. It should be understood that the image data may include other data, such as, for example, sensor data, environmental data, etc.

[0069] In this regard, image data is transferred from the surveyor's computing device 108 to the computing device 102 (and / or the database 104) (for implementations where the two are separate).

[0070] Next, the image data is preprocessed at 604. Specifically, in this example, one or more images are rotated or flipped as needed and potentially stitched together to provide a single image of plant 116. In one or more instances, the preprocessing includes deduplication of the images to limit the number of times plant 116 appears in the image (e.g., only once, etc.), as explained in more detail above.

[0071] In one or more instances, background details in the image may be blurred, removed, or otherwise reduced to avoid or limit the exposure of any background vegetation to the following steps. Specifically, for example, computing device 102 may use depth information from camera 112 to target any object in an adjacent row ( For example Thresholding is performed on objects that are closer to and / or farther from camera 112 to help avoid deletion in rows that are not points of interest, which in turn can reduce or eliminate errors in height calculation processing.

[0072] The preprocessed image data is then stored in memory (e.g., database 104, etc.).

[0073] Combined with the training phase, such as Figure 6 As instructed, on computing device 102 or other suitable computing device, the preprocessed image is displayed to a user such as a researcher or grower, and the user annotates specific points of the plant in the image at step 606. In this example, the image includes a corn ear and / or nodes growing from the corn ear. Annotations may include, for example, placing bounding boxes on or around the entire corn ear (or most of it) and around the nodes (or most of it). Multiple bounding boxes may be annotated in each image. In addition to one or more bounding boxes, the user also categorizes labels (such as, for example, ear or node) to specify that the bounding box indicates an ear or node. It should be understood that step 606 may involve annotating dozens, hundreds, or even thousands of images as part of a training phase to define an image training set. The annotations are then saved along with the images.

[0074] In this example, at 608, the computing device 102 then trains a CNN model based on the labeled images to identify spikes and / or nodes of the plant. The training may involve 50%, 60%, 70%, or 80% of the labeled images, or more or less, etc.

[0075] Subsequently, at point 610, the computing device 102 validates the trained model based on the remaining labeled images. Specifically, the computing device 102 provides images (without labels) to the trained model and then... For example The identified spikes / nodes (via bounding boxes, etc.) are compared with the annotations in the input image. When the performance of the trained model is sufficient, or one or more thresholds are met, the trained model is validated and stored. For example (in memory 504, etc.) for use in the highly defined stage.

[0076] Although the CNN model was trained in Method 600, it should be understood that, as mentioned above, other suitable models can be used in other method implementations.

[0077] In the height determination stage, as described above, at steps 602 and 604, images are captured and preprocessed as needed. At 612, the computing device 102 identifies spikes / nodes of the plant in one or more images based on a trained model, or more broadly, identifies features. That is, the computing device 102 provides the trained model with images of the plant (…). Right now (Preprocessed images), and the model is trained to identify each feature of the plant through bounding boxes, where the features are spikes and / or nodes.

[0078] The model's output includes bounding boxes with different features, such as... Figure 3 The image is shown as four bounding boxes consisting of two nodes and two ears of corn (but without height labels).

[0079] At 614, the computing device 102 initially identifies the overlap between the node boundary frame and the ear boundary frame, indicating the germination characteristics of the plant 116. Furthermore, at 614, the computing device identifies the center coordinates (or other coordinates) of the node boundary frame. In other embodiments, the coordinates can be coordinates of the ear boundary frame or other coordinates of the node boundary frame. For example, the coordinates may include uppermost coordinates, lowermost coordinates, average coordinates, etc., or other coordinates of one or both of the node boundary frame and / or ear boundary frame. It should be understood that specific coordinates may vary within a particular boundary frame, but for multiple plants, the coordinates are generally kept consistent to provide an indication of one or more relative dimensions between plants.

[0080] Using coordinate and depth data of specific features, as well as other data included in the image data (including bounding boxes), the computing device then transforms the coordinates at 616. Specifically, specific points in the bounding box (such as, for example, top or bottom points) are associated with the image and the image's depth data. Points in the image are represented by coordinates, and these coordinates are then transformed using the depth data, generally as described in system 100 above.

[0081] At point 618, the computing device 102 determines the height of points included in the image, or, in this specific scenario, the height of the ear and / or node of the corn plant, and then outputs the height to the user. This determination is based on transformed coordinates, the camera's height above the ground, etc. Relatedly, the computing device 102 further annotates the height of features on the image, particularly the height of the ear and node, for example, as... Figure 3 As shown. Although the height is defined in this article, one or more other dimensions of the plant can be determined based on the description herein.

[0082] It should be understood that when the computing device 102 is fully or partially integrated with the surveyor's computing device 108, the height determination is in real time. For example Within a minute of capture (or near real-time) For example The process can be completed in a manner that takes approximately one minute, two minutes, or at most five minutes. Relatedly, the computing device 102 can limit the resolution of the captured image. For example (e.g., reducing the resolution of RGB images by 50% or more, or skipping images) For example Other methods include skipping every other image or skipping two out of three images, applying multithreading, reducing the frequency of image capture, or reducing the amount of data to be processed to help achieve the desired throughput from image capture to a highly defined level.

[0083] Therefore, it should be understood that in some embodiments, the functionality described herein can be described using computer-executable instructions stored on a computer-readable medium and executable by one or more processors. A computer-readable medium is a non-transitory computer-readable medium. For example, and not as a limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that is accessible by a computer. Combinations of the above should also be included within the scope of computer-readable media.

[0084] It should also be understood that one or more aspects of this disclosure, when configured to perform one or more of the functions, methods and / or processes described herein, can transform a general-purpose computing device into a special-purpose computing device.

[0085] Based on the foregoing description, it should be understood that the above-described embodiments of this disclosure can be implemented using computer programming or engineering techniques, including computer software, firmware, hardware, or any combination or subset thereof, wherein the technical effects can be achieved by at least one of the following operations: (a) accessing corn plant-specific image data, the image data including an image of the corn plant and depth data indicating the extent between the corn plant and a camera capturing the image; (b) identifying features of the corn plant using a trained model, the features including ears of corn and / or nodes from which the ears grow; (c) transforming feature-specific coordinates into the height of the corn plant feature; (d) storing the height of the corn plant feature in memory; (e) preprocessing the image before identifying the corn plant feature; and (f) attaching the height to the feature on the image.

[0086] Based on the foregoing description, it should also be understood that the above-described embodiments of this disclosure can be implemented using computer programming or engineering techniques, including computer software, firmware, hardware, or any combination or subset thereof, wherein the technical effects can be further ( For example In addition to the above list, or as an alternative to the above list, at least one of the following operations may be performed to achieve: (a) accessing plant-specific image data, the image data including an image of the plant and depth data indicating the range between the plant and the camera that captured the image; (b) identifying plant features using a trained model via a computing device; (c) transforming feature-specific coordinates into the dimensions of plant features via a computing device; and (d) storing the dimensions of plant features in memory via a computing device.

[0087] Examples and implementations are provided so that this disclosure will be complete and will fully communicate the scope to those skilled in the art. Numerous specific details, such as examples of particular components, apparatus, and methods, are set forth to provide a complete understanding of embodiments of this disclosure. It will be apparent to those skilled in the art that specific details are not required, exemplary embodiments may be embodied in many different forms, and neither should be construed as limiting the scope of this disclosure. In some example embodiments, well-known processes, well-known apparatus structures, and well-known techniques are not described in detail. Furthermore, advantages and improvements achievable using one or more example embodiments disclosed herein may be provided, or not provided, in whole or in part, and still fall within the scope of this disclosure.

[0088] The specific values ​​disclosed herein are, in essence, instances and do not limit the scope of this disclosure. The disclosure herein of specific values ​​and ranges of values ​​for a given parameter does not preclude other values ​​and ranges of values ​​that may be used in one or more instances disclosed herein. Furthermore, it is contemplated that any two specific values ​​of a particular parameter described herein may define endpoints that are also applicable to the range of values ​​of the given parameter. Right now The disclosure of a first and second value of a given parameter can be interpreted as disclosing that any value between the first and second values ​​can also be used for the given parameter. For example, if parameter X is exemplified herein as having a value A and also exemplified as having a value Z, then parameter X is contemplated to have a range of values ​​from about A to about Z. Similarly, the disclosure of two or more ranges of values ​​of a parameter (whether such ranges are nested, overlapping, or distinct) covers all possible combinations of ranges of values ​​that may be claimed using the endpoints of the disclosed ranges. For example, if parameter X is exemplified herein as having a value in the range of 1-10, or 2-9, or 3-8, then it is also contemplated that parameter X may have other ranges of values, including 1-9, 1-8, 1-3, 1-2, 2-10, 2-8, 2-3, 3-10, and 3-9.

[0089] The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context explicitly indicates otherwise. The terms “comprises,” “comprising,” “including,” and “having” are inclusive and therefore indicate the presence of the stated feature, integer, step, operation, element, and / or component, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein should not be construed as necessarily requiring them to be performed in the particular order discussed or shown, unless specifically indicated otherwise. It should also be understood that additional or alternative steps may be employed.

[0090] When a feature is described as being “on,” “joined to,” “connected to,” “coupled to,” “associated with,” “communicating with,” or “included” in another element or layer, it may be directly located on, joined to, connected to, or coupled to the other feature, or associated with, communicate with, or be included in the other feature, or there may be intervening features present. As used herein, the terms “and / or” and the phrase “at least one of…” include any and all combinations of one or more of the associated listed items.

[0091] Although the terms first, second, third, etc., may be used herein to describe various features, these features should not be limited by these terms. These terms may only be used to distinguish one feature from another. Unless the context clearly indicates otherwise, terms such as “first,” “second,” and other numerical terms, as used herein, do not imply order or sequence. Therefore, a first feature described herein may be referred to as a second feature without departing from the teachings of the exemplary embodiments described.

[0092] The above description of embodiments has been provided for illustrative and descriptive purposes. This description is not intended to be exhaustive or limiting of this disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but are interchangeable where appropriate and can be used in the chosen embodiment, even if not specifically shown or described. The same can be varied in many ways. Such variations should not be considered as departing from this disclosure, and all such modifications are intended to be included within the scope of this disclosure.

Claims

1. A computer-implemented method for processing image data of crops, the method comprising: Access corn plant-specific image data, which includes an image of the corn plant and depth data indicating the range between the corn plant and the camera that captured the image; Using a trained model and a computing device, the characteristics of the corn plant are identified, including the ears of the corn plant and / or the nodes growing from the ears. The computing device transforms the coordinates specific to the feature into the height of the feature of the corn plant. as well as The height of the corn plant, a characteristic described in the computing device, is stored in a memory.

2. The computer-implemented method of claim 1, wherein the trained model comprises a convolutional neural network (CNN) model; and The characteristics of the corn plant include the ear of the corn plant and the nodes growing from the ear.

3. The computer-implemented method of claim 1, further comprising preprocessing the image before identifying the features of the corn plant; and / or The features used to identify the corn plant include: Identify the first bounding box of the ear of the maize plant and the second bounding box of the node from which the ear grows; Confirm the overlap between the first bounding box and the second bounding box; as well as The coordinates of the center of the second bounding box of the section are identified as the coordinates specific to the feature.

4. The computer-implemented method of claim 3, wherein transforming the coordinates specific to the feature comprises: The three-dimensional coordinates of the first and second bounding boxes are transformed into a two-dimensional uniform image via an intrinsic matrix.

5. The computer-implemented method of claim 4, wherein transforming the coordinates specific to the feature comprises: Based on the transformation between the camera origin and the world origin, the three-dimensional coordinates of the first bounding box and the second bounding box are translated / rotated.

6. The computer-implemented method of claim 1, wherein transforming the coordinates specific to the feature to the height of the feature further comprises determining the height relative to the ground based on the coordinates.

7. The computer-implemented method of claim 1, the method further comprising attaching the height to the feature on the image, thereby indicating the height using the feature on the image.

8. A non-transitory computer-readable storage medium comprising executable instructions for processing image data, the instructions causing the at least one processor, when executed by at least one processor, to: Access plant-specific image data, which includes an image of the plant and depth data indicating the range between the plant and the camera that captured the image; The plant features are identified using a trained model, which includes a convolutional neural network (CNN) model. Transform the coordinates specific to the feature into the height dimension of the feature of the plant; as well as The dimensions of the plant's characteristics are stored in the memory.

9. A system for processing image data of crops, the system comprising: Memory; as well as A computing device communicatively coupled to the memory, the computing device being configured to: Access corn plant-specific image data, which includes an image of the corn plant and depth data indicating the range between the corn plant and the camera that captured the image; The trained model is used to identify features of the corn plant, including the ears of the corn plant and / or the nodes growing from the ears; The coordinates specific to the feature are transformed to the height of the feature of the corn plant; as well as The height of the corn plant is stored in the memory.

10. The system of claim 9, wherein the computing device includes a surveyor computing device, the surveyor computing device including one or more cameras; and The measuring computing device is configured to capture image data specific to the corn plant via the one or more cameras.

11. The system of claim 9, wherein the trained model comprises a convolutional neural network (CNN) model, and the CNN model comprises a YOLOv5 (You Only Look Once Version 5) model.

12. The system of claim 9, wherein the trained model includes a YOLOv5 (You Only LookOnce Version 5) model.

13. The system of claim 9, wherein the characteristic of the maize plant includes the node from which the ear of the maize plant grows.

14. The system of claim 9, wherein the computing device is further configured to preprocess the image before recognizing the features of the corn plant via the trained model; and The computing device is configured to: Identify the first bounding box of the ear of the maize plant and the second bounding box of the node from which the ear grows; Confirm the overlap between the first bounding box and the second bounding box; and The coordinates of the center of the second bounding box of the section are identified as the coordinates specific to the feature.

15. The system of claim 14, wherein the computing device is configured to: transform the coordinates specific to the feature when: The three-dimensional coordinates of the first and second bounding boxes are transformed into a two-dimensional uniform image via an intrinsic matrix.

16. The system of claim 15, wherein the computing device is configured to: transform the coordinates specific to the feature when: Based on the transformation between the camera origin and the world origin, the three-dimensional coordinates of the first bounding box and the second bounding box are translated / rotated.

17. The system of claim 9, wherein the computing device is configured to determine the height relative to the ground based on the coordinates when transforming the coordinates specific to the feature.

18. The system of claim 9, wherein the computing device is further confirmed to attach the height to the feature on the image, thereby indicating the height using the feature on the image.

Images