Systems and methods for vision-based plant detection and reconnaissance application techniques

Abstract

In one embodiment, a method includes obtaining image data from one or more sensors of a device in response to an input to initiate a continuous process for reconnaissance; analyzing one or more input images from the image data, generating a grid of two-dimensional (2D) reference points projected onto the ground plane to create a set of matched three-dimensional (3D) anchor points and their positions in the 3D space of the farmland using augmented reality (AR); providing the one or more input images, the tracking mesh, and the location in the 3D space of the farmland to a machine learning (ML) model having a convolutional neural network (CNN); and generating inference results including the detected array of objects with the ML model and selecting a most likely inference plant location and classification for the detected array of objects.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 363,981, filed on May 2, 2022, entitled SYSTEMS AND METHODS FOR VISION-BASED PLANT DETECTION AND SCOUTING APPLICATION TECHNOLOGY, the entire contents of which are hereby incorporated by reference. Technical Field

[0003] Embodiments of the present disclosure generally relate to systems and methods for vision-based plant detection and scouting application techniques. Background Art

[0004] Crop advisors, farmers, and seed dealers use field scouting applications to determine crop conditions. The scouting application allows users to record areas or points of interest manually, on foot, or in a vehicle. Users can take pictures with a mobile device and receive results for weed infestation, nitrogen status, leaf damage, disease identification, pest monitoring, and emergence analysis. Summary of the invention

[0005] Disclosed herein are systems and methods for vision-based plant detection and scouting application technologies. In one aspect of the present disclosure, a computer-implemented method for scouting plants in a farmland is provided, the computer-implemented method comprising: obtaining image data from one or more sensors of a device in response to an input for initiating a continuous process of scouting plants; analyzing one or more input images from the image data; generating a grid of two-dimensional (2D) reference points projected onto a ground plane to create a set of matching three-dimensional (3D) anchor points and their positions in the 3D space of the farmland using an augmented reality (AR) framework. The computer-implemented method also includes: providing the one or more input images, the grid, and the positions in the 3D space of the farmland to a machine learning (ML) model having a convolutional neural network (CNN); and generating an inference result including an array of detected objects with the ML model and selecting the most likely inferred plant position and classification for the array of detected objects.

[0006] The method includes machine learning models, augmented reality, and optionally computer vision biodetection to detect and map plants in three-dimensional space for an enhanced crop scouting tool.

[0007] Yet another aspect of the present disclosure includes using object tracking anchors to generate a corrected inference position to compensate for movement of the device during image capture.

[0008] Yet another aspect of the present disclosure includes projecting the corrected inference result position into the 3D space using the augmented reality (AR) framework.

[0009] Yet another aspect of the present disclosure includes creating plant tags in the 3D space based on the positions of the corrected inference positions, and classifying the plant tags into target plants to be identified and other non-target plants.

[0010] Another aspect of the present disclosure includes: inputting the position of the target plant into a row detection and tracking algorithm to detect linear rows of the target plant in the farmland and grouping the position of the target plant into the linear rows; determining a predicted row direction vector and a central position equidistant between adjacent rows based on a moving window of row tracking results; using the predicted row direction vector to project the estimated central position between the adjacent rows; converting the position of the target plant to be relative to the predicted central position and projecting it onto an axis orthogonal to the predicted row direction vector within a certain angle range; and using kernel density estimation to analyze the projection at each angle to determine the direction and spacing of each adjacent row.

[0011] Yet another aspect of the present disclosure includes processing all plants using a fuzzy logic rule set to determine whether to add a target plant or row to an existing row or to discard inference results based on the fuzzy logic rule set and filtering, wherein if a detected plant is positioned more than a set distance from the device, the detected plant is discarded.

[0012] Yet another aspect of the present disclosure includes adding labels and markers to the 3D space of the AR framework upon determining a new detection of a plant position for an inference result of a plant, wherein the labels and markers are visible to a user on a display of the device.

[0013] Yet another aspect of the present disclosure includes the label indicating a growth stage of the plant, and also adding a spacing between the plant and neighboring plants to the 3D space of the AR frame to indicate an assessment of the planting.

[0014] Yet another aspect of the present disclosure includes upon detecting a new row or plant, connecting the plants in the row via the shortest possible path, and displaying the connection and separation distance between adjacent plants to the display of the device.

[0015] Yet another aspect of the present disclosure includes the device being an edge device to determine crop emergence levels and spacing between adjacent plants in a row.

[0016] In one aspect of the present disclosure, a computing device is provided, the computing device comprising: a display device for displaying a user interface to display plants present in a farmland, the plants being enhanced with computer-generated sensory information for scouting the plants; and a processor coupled to the display device. The processor is configured to: obtain image data from one or more sensors; analyze one or more input images from the image data; generate three-dimensional (3D) anchor points and their positions in the 3D space of the farmland using an augmented reality (AR) framework; provide the one or more input images and the positions in the 3D space of the farmland to a machine learning (ML) model having a convolutional neural network (CNN); and generate an inference result including an array of detected objects with the ML model and select the most likely inferred plant position and classification for the array of detected objects.

[0017] Yet another aspect of the present disclosure includes the processor being configured to generate a corrected inference result position using object tracking anchors to compensate for movement of the device during image capture.

[0018] Yet another aspect of the present disclosure includes the processor being configured to project the corrected inference result position into the 3D space using the augmented reality (AR) framework.

[0019] Yet another aspect of the present disclosure includes the processor being configured to create plant tags in the 3D space based on the positions of the corrected inference result positions, and to classify the plant tags into target plants to be identified and other non-target plants.

[0020] Another aspect of the present disclosure includes the processor being configured to: provide the input position of the target plant to a row detection and tracking algorithm to detect linear rows of the target plant in the farmland and group the positions of the target plant into the linear rows; determine a predicted row direction vector and a central position equidistant between adjacent rows based on a moving window of row tracking results; use the predicted row direction vector to project the estimated central position between the adjacent rows; convert the position of the target plant to be relative to the predicted central position and project it onto an axis orthogonal to the predicted row direction vector within a certain angle range; and use kernel density estimation to analyze the projection at each angle to determine the direction and spacing of each adjacent row.

[0021] Yet another aspect of the present disclosure includes the processor being configured to process all plants using a fuzzy logic rule set to determine whether to add a target plant or row to an existing row or to discard inference results based on the fuzzy logic rule set and filtering.

[0022] Yet another aspect of the present disclosure includes the processor being configured to add tags and markers to the 3D space of the AR framework upon determining a new detection of a plant position for an inference result of a plant, wherein the tags and markers are visible to a user on a display of the device.

[0023] Yet another aspect of the present disclosure includes the label indicating a growth stage of the plant, and also adding a spacing between the plant and neighboring plants to the 3D space of the AR frame to indicate an assessment of the planting.

[0024] Yet another aspect of the disclosure includes the tag characterizing the interval as one of a good interval, a misplaced seed interval, and a skipped seed interval.

[0025] Yet another aspect of the present disclosure includes the processor being configured to connect the plants in the row via the shortest possible path upon detection of a new row or plant. The display device being configured to display the connection and the separation distance between adjacent plants.

[0026] Within the scope of the present application, it should be understood that the various aspects, embodiments, examples and alternatives and their respective features set forth herein can be adopted independently or in any possible and compatible combination. Where features are described with reference to a single aspect or embodiment, it should be understood that such features apply to all aspects and embodiments unless otherwise specified or such features are incompatible. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present disclosure is illustrated by way of example and not limitation in the figures of the accompanying drawings and in which:

[0028] Figure 1 An example of a system for collecting and analyzing farmland data according to one embodiment is shown.

[0029] Figure 2 The architecture of an implement 200 for delivering an application (eg, a fluid application, a fluid mixture application) to an agricultural field is shown according to one embodiment.

[0030] Figure 3A , Figure 3B , Figure 3C and Figure 3D A flow chart illustrating one embodiment of a computer-implemented method 300 and 350 of a continuous scouting process that uses machine learning models, augmented reality, and computer vision bio-detection to detect and map plants in three-dimensional space for an enhanced crop scouting tool.

[0031] Figure 3E , Figure 3F and Figure 3GA flow chart illustrating one embodiment of a computer-implemented method 390 and 392 of a continuous scouting process using a machine learning model and augmented reality to detect and map plants in three-dimensional space for an enhanced crop scouting tool.

[0032] Figure 4 A flow chart of one embodiment of a computer-implemented method for implementing one or more rule sets for plant location and identification is shown.

[0033] Figure 5 A monitor or display of a device having a user interface 501 for displaying annotated target plants in 3D space using augmented reality is shown according to one embodiment.

[0034] Figure 6 A monitor or display of an apparatus 600 is shown having a user interface 601 for displaying annotated target plants in 3D space using augmented reality according to one embodiment.

[0035] Figure 7 An example of a system 700 is shown that includes a machine 702 (eg, a tractor, a combine harvester, etc.) and an implement 740 (eg, a planter, a cultivator, a plow, a sprayer, a spreader, an irrigation implement, etc.) according to one embodiment. DETAILED DESCRIPTION

[0036] Disclosed herein are systems and methods for vision-based plant detection and scouting application technologies. In one embodiment, at least one of an implement, a machine, an agricultural vehicle, an aerial device, a drone, a self-propelled device (e.g., a robot, an off-road vehicle, an ATV, a UTV), or an electronic edge device (e.g., a mobile device, a tablet device) with a sensor (e.g., an image capture device) collects agricultural data of emerging plants. The computer-implemented method includes using a machine learning model, augmented reality, and optionally computer vision biodetection to detect and map plants in three-dimensional space for an enhanced crop scouting tool. The computer-implemented method for scouting plants in a farmland includes: obtaining image data from one or more sensors of a device; analyzing one or more input images from the image data; and generating a grid of two-dimensional (2D) reference points projected onto a ground plane to create a set of matching three-dimensional (3D) anchor points and their positions in the 3D space of the farmland using augmented reality (AR). The method also includes: applying computer vision to one or more input images to determine biomass regions (e.g., colors of pixels of the biomass regions) of the one or more images; determining linear plant rows for a row tracking process based on the tracking grid and the locations of the biomass regions; and providing the one or more input images, the tracking grid, and the locations of the biomass regions to a machine learning (ML) model having a convolutional neural network (CNN); and generating, with the ML model, inference results including a detected object array and selecting the most likely inferred plant location and classification for the detected object array.

[0037] In the following description, many details are set forth. However, it will be apparent to those skilled in the art that embodiments of the present disclosure may be practiced without these specific details. In some cases, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the present disclosure.

[0038] Referring to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views, Figure 1 An example of a system for collecting and analyzing agricultural data from a farmland to display customized agricultural data according to one embodiment is shown. The machines and implements of the system 100 perform agricultural operations on the farmland (e.g., planting seeds in the field, applying fluid applications to plants, capturing image data to determine stand counts, plant heights, growth stages, and spacing of emerging plants to assess the quality of the planting operation).

[0039] For example, system 100 may be implemented as a cloud-based system having a server, a data processing device, a computer, etc. Aspects, features, and functions of system 100 may be implemented in a server, a wireless node, a planter, a planter monitor, a sprayer, a sidedress bar, a combiner, a laptop, a tablet, a computer terminal, a client device, a user device, a handheld computer, a personal digital assistant, a cellular phone, a camera, a smart phone, a mobile phone, a computing device, or a combination of any of these or other data processing devices.

[0040] System 100 may include a network computer or an embedded processing device within another device (e.g., a display device), an implement or machine (e.g., a tractor cab, a farm vehicle), or a processor having a processor that is more than 100,000 bytes. Figure 1Other types of data processing systems with fewer components or possibly more components are shown. System 100 (e.g., a cloud-based system) and agricultural operations can use implements or machines to control and monitor planting and fluid application. System 100 includes machines 140, 142, 144, 146 and implements 141, 143, 145 connected to the corresponding machines 140, 142 and 144, and devices 148 (e.g., user devices, edge devices). Device 148 can be moved by a user through a field to provide vision-based plant detection and scouting application technology according to the methods disclosed herein (e.g., methods 300 and 400). Implements (e.g., planters, cultivators, plows, sprayers, spreaders, irrigation implements) can include flow devices for controlling and monitoring the application (e.g., sowing, spraying, fertilizing) of crops and soil within an associated field (e.g., fields 103, 107, 109). The system 100 includes an agricultural analysis system 102, which may include a weather storage area 150 with current and historical weather data, a weather prediction module 152 with weather predictions for different regions, a field database 134 with farmland data, and at least one processing system 132 for executing instructions for controlling and monitoring different operations (e.g., fluid application). The storage medium 136 may store instructions, software, software programs, etc. for execution by the processing system and for performing operations of the agricultural analysis system 102. In one example, the storage medium 136 may contain a fluid application prescription (e.g., a fluid application prescription that associates a geographically referenced location in a field with an application rate). The implement 141 (or any of the implements) may include an implement having a pump, a flow sensor, and / or a flow controller, which may specifically be an element that communicates with the network 180 for sending control signals or receiving application data. When the system 102, wireless node, machine (e.g., 140, 142, 144, 146), or implement (e.g., 141, 143, 145) is connected to the network 180, the network 180 (e.g., any wireless network, any cellular network (e.g., 4G, 5G), the Internet, a wide area network, WiMax, satellite, IP network, etc.) allows Figure 1The system 102, wireless nodes, machines, devices and implements communicate with each other. Examples of agricultural monitors are described in PCT Publication Nos. WO2008 / 086318, WO2012 / 129442, WO2013 / 049198, WO2014 / 026183 and WO2014 / 018717. An example of an agricultural monitor is the 20|20® monitor from Precision Planting, LLC. In one example, the monitor preferably includes a graphical user interface ("GUI"), a memory, a central processing unit ("CPU") and a bus node. The bus node preferably includes a controller area network ("CAN") node, which includes a CAN transceiver, a controller and a processor. The monitor is preferably in electrical communication with a speed sensor (e.g., a radar speed sensor mounted to a tractor) and a global positioning receiver ("GPS") receiver mounted to the tractor (or in some embodiments, to the tool bar of the implement).

[0041] As the agricultural implement passes through the field, the monitor A of the first machine (e.g., 140, 142, 144, 146) collects application data or scouting data at various points in the field. The first machine can be connected to the agricultural implement and pass the agricultural implement through the field. The application data can be sowing information, such as singulation percentage, skip, multiple, down force, applied liquid, depth measurement, agronomic measurement, and liquid application data. The scouting data of the plant can be collected by an image sensor of the machine or implement.

[0042] Figure 2 The architecture of an implement 200 for delivering an application (e.g., a fluid application, a fluid mixture application) to an agricultural field is shown according to one embodiment. The implement 200 includes at least one storage tank 250, flow lines 260 and 261, a flow controller 252 (e.g., a valve), and at least one variable rate pump 254 (e.g., an electric pump, a centrifugal pump, a piston pump, etc.) for pumping fluid (e.g., a fluid application, a semi-fluid mixture) from the at least one storage tank to different application units 210 to 217 of the implement and controlling the application rate thereof. At least one flow sensor 270 may be utilized on the implement 200 row by row or upstream of the location where the fluid branches off to the application unit, such as Figure 2 The flow controller 252 can be line by line, rather than Figure 2 Shown is based on implement width.

[0043] The application units are mechanically coupled to the frames 220 to 227, which are mechanically coupled to the pole 10. Each application unit 210 to 217 may include an image sensor, a flow sensor, and a component with a placement mechanism (e.g., a planting contact member, a feeler, a guide member) for obtaining the correct orientation and / or positioning of the fluid outlet relative to the plants in the field. The frames 220 to 227 and the pole 10 may also include image sensors for various applications, including scouting applications for methods 300 and 400.

[0044] Figure 3A , Figure 3B , Figure 3C and Figure 3D A flow chart of one embodiment of a computer-implemented method 300 and 350 of a continuous scouting process using machine learning models, augmented reality, and computer vision bio-detection to detect and map plants in three-dimensional space for an enhanced crop scouting tool is shown. Methods 300 (e.g., row tracking process) and 350 (e.g., plant tracking process) are performed by processing logic, which may include hardware (circuitry, dedicated logic, etc.), software (such as running on a general-purpose computer system or a dedicated machine or device), or a combination of the two. In one embodiment, methods 300 and 350 are performed by processing logic of a processing system of a system 102, a machine, an equipment, an implement, an agricultural vehicle, an aerial device, a monitor, a display device, a user edge device, a self-guided device, or a self-propelled device (e.g., a robot, a drone, an ATV, a UTV, etc.). The processing system executes instructions of a software application or program with the processing logic. The software application or program can be launched by the processing system. In one example, the monitor or display device receives user input and provides a customized display for the operation of methods 300 and 350.

[0045] exist Figure 3A At operation 302 of method 300, one or more sensors (e.g., image sensors, cameras) of a device (e.g., edge device, self-propelled device) are positioned to observe plants in a field, and a software application is launched on a processing system of the device and displayed on a monitor or display device as a user interface of the device. The processing system can be integrated or coupled with a machine that performs an application process (e.g., planting, tilling, fertilizing, spraying, etc.). Alternatively, the processing system can be integrated with such a device (e.g., drone, edge device with image capture device) that is associated with the machine that captures images before, during, or after the application process. In one example, the user interface displays a real-time view of the plants in the field.

[0046] At operation 304, image data is obtained from one or more sensors of the device in response to a user input for initiating a continuous process of scouting plants using the device. The continuous scouting process may initiate a row tracking process to detect linear rows of biomass, and after completing several iterations of the row tracking process, initiate a plant tracking process and receive input from the row tracking process. The device may be located on an implement, a machine that pulls the implement during a current application flow, an aerial device, a user edge device, a self-guided device, a self-propelled device, etc. The sensors may be in-situ sensors located on each row unit of the implement, spaced apart every few row units, or located on the machine.

[0047] The software application can provide instructions for angular guidance of the device while capturing images of plants. Sensory data from the device's accelerometer, gyroscope, or other motion-based sensor can be used to suggest positioning and angular guidance of the device on a user interface to improve the performance of machine learning, augmented reality, and computer vision. The software application can provide instructions to the user, including providing a location of the device between plant rows, providing a suggested orientation of the device, and providing a suggested tilt of the device while capturing image data.

[0048] At operation 306, the computer-implemented method analyzes one or more input images from the image data, generates a tracking grid of 2D reference points projected onto a ground plane to create a set of matching 3D anchor points and their locations in 3D space using augmented reality (e.g., an augmented reality framework), and the one or more input images are saved on the device. Augmented reality (AR) is an interactive experience of a real-world environment in which objects present in the real world are enhanced by computer-generated sensory information, sometimes across multiple sensory modalities, including vision, hearing, touch, body sensation, and smell. In one example, emerging plants present in a field are augmented with computer-generated sensory information to improve the user's scouting experience.

[0049] At operation 311, computer vision is applied to one or more images to determine biomass regions in the one or more images. The computer vision can determine the color of the pixels of the biomass. At operation 314, the biomass regions are grouped by location using the nearest 2D reference point generated from the one or more images (e.g., still images), and the offset from the nearest 2D reference point to the inferred location of the plant is recorded.

[0050] At operation 316, the 3D reference points are then projected back onto the current 2D camera frame based on the current viewing frustum, and the 2D translation of each reference point is recorded.

[0051] At operation 318, a 2D translation is then applied to the corresponding biomass region to generate a corrected biomass region to compensate for movement of the device during image processing, and the resulting corrected biomass region position is projected into 3D space using an augmented reality (AR) framework. This results in the 3D point being temporarily stored as a pin marking the plant in 3D space.

[0052] exist Figure 3B At operation 322, as part of the row tracking process, the location of the biomass area is input into a custom line detection algorithm to detect linear rows. At operation 323, the method adds the detected linear rows to the continuous row direction and spatial tracking of the plant rows in 3D space.

[0053] At operation 324, the method is based on a row tracking process (e.g., Figure 3A The method determines whether the plant tracking process is ready to be initiated by determining whether a threshold number of iterations (e.g., several iterations, 3 to 5 iterations, etc.) has been completed. If so, the plant tracking process is initiated at operation 325. If not, the method returns to operation 304 for additional iterations of the row tracking process.

[0054] At operation 326, the method determines whether the plant tracking process is over (e.g., the user ends or terminates the continuous scouting process and this ends the plant tracking process). After the user ends the plant tracking process, at operation 334, the rows and plants that have been detected and displayed to the user are aggregated to display various statistics, such as spacing between plants in a row, row spacing, emergence levels, and financial impact. Additionally, the software application is designed so that the user can selectively draw plant tags in 3D space and select / change classifications, as well as remove false positive auto-tags.

[0055] If the vegetation tracking process continues, the method returns to operation 304 for another iteration.

[0056] exist Figure 3CAt operation 358 of the plant tracking process 350, one or more images obtained at operation 304 are then preprocessed, resized, and formatted according to the specifications of a machine learning (ML) model with a convolutional neural network (CNN), and provided as input to the ML model. The inputs may include images, tracking grids, biomass areas, and locations from the row tracking process. At operation 360, the ML model (e.g., an object detection model) is then executed using the ML engine of the processing system of the device for object detection. The architecture and weights of the object detection model used by the software application are developed and trained using a dataset representing plants in the farmland and annotations added to these inference outputs. Iterations of the object detection model improve the classification and positioning of plants in 3D space during use, and are fully packaged on an edge device that includes an ML engine (e.g., a neural engine), an AR framework with a software library, an efficient camera and image pipeline, GPS, and pixel-level local image post-processing for computer vision and biomass detection. Different crops can be evaluated and characterized with a single general neural network model. The updated model is trained and improved without the need for app store updates. The software application creates an intuitive user experience (similar to a gaming experience) based on the use of augmented reality, object detection modeling, and computer vision techniques on edge devices to determine crop emergence levels and spacing between neighboring plants in a row while offline or online. Augmented reality, ML modeling, and computer vision are performed on edge devices so that the edge device can be offline without internet access and still be able to determine crop emergence levels and spacing between neighboring plants in a row. The implementation of computer vision and biomass detection provides improved bias for the ML model.

[0057] At operation 362, once the ML model execution is complete, the inference output of the ML model (e.g., an array of detected objects with confidence levels) is filtered and adjusted using non-maximum suppression to provide a confidence level based at least in part on the information from the Figure 3A The most likely plant location and classification are selected based on the color determined in operation 311. At operation 364, the inference results are grouped by location using the nearest 2D reference point generated by one or more images (e.g., a still image), and the offset from the nearest 2D reference point to the inference position of the plant is recorded. At operation 366, the 3D reference points are then projected back onto the current 2D camera frame based on the current view cone, and the 2D translation of each reference point is recorded. At operation 368, the 2D translation is then applied to the corresponding inference result to generate a corrected inference result to compensate for the movement of the device during image capture, and the resulting corrected inference result position is projected into 3D space using an augmented reality (AR) framework. This causes the 3D point to be temporarily stored as a pin marking the plant in 3D space.

[0058] In one example, at operation 370, once plant tags have been created in 3D space, the plant tags are divided into another category of desired target plants (e.g., any crops, corn, soybeans, weeds, etc.) to be identified and scouted and other non-target plants. At operation 372, all plants are then processed using a fuzzy logic rule set. According to the fuzzy logic rule set and filtering, these rules determine whether to add the target plant or row to the existing row or discard the result. When the ML model finds that the plant is a repeated detection, they are grouped together with the already existing plants and rows. The computer-implemented method is a novel algorithm designed to adjust the location and emergence classification of existing plants with new matching data, thereby slowly increasing the accuracy of plant detection over time. When it is determined that the plant is a new detection at operation 374, the tags and marks visible to the user are placed in the 3D space of the AR frame. At operation 378, when a new row is detected or added, the plants in the row are connected via the shortest possible path and their connection and distance are displayed to the user on the device. Based on the lateral drop off in the detected linear row line, the likely point where the detected plant stem touches the ground is located in 3D space. This creates an accurate point for assessing spacing measurements between plants.

[0059] At operation 380, existing rows and individual plants that are determined to have a low likelihood of being correct at each cycle of new data are removed.

[0060] In one example, computational limitations of the edge device create a significant time delay between the capture of a camera frame from the edge device's camera and the completion of image data processing by the object detection model at operations 308 and 310. To compensate for this delay, the algorithm creates a 2D point grid with corresponding 3D point anchors on the captured camera frame at operation 306. Projecting these 3D point anchors back onto the camera frame plane (e.g., operation 314) creates a new grid that can be used to compensate for any changes in the camera view between the original frame and the current frame.

[0061] In addition, grouping objects in the original camera frame with points on the grid and applying the same translation allows efficient motion compensation for a large number of ground-based object detections. This motion is caused when the edge device moves during image processing. Object detection CNN models can often produce false positive detections during conventional use. Class elimination and region of interest (ROI) framing can be used to reduce false positives. To compensate for these false positives, a rule set (e.g., operation 372) is developed that uses the determined 3D position of each plant to logically eliminate or combine results that appear with low probability at potential plant locations.

[0062] Figure 3E , Figure 3F and Figure 3GA flow chart illustrating one embodiment of a computer-implemented method 390 and 392 of a continuous scouting process that uses a machine learning model and augmented reality to detect and map plants in three-dimensional space for an enhanced crop scouting tool. Figure 3E and Figure 3F 390 (e.g., a row tracking process) and 390 (e.g., a plant tracking process) are performed by processing logic, which may include hardware (circuitry, dedicated logic, etc.), software (such as running on a general-purpose computer system or a dedicated machine or device), or a combination of the two. In one embodiment, methods 390 and 392 are performed by processing logic of a processing system of system 102, a machine, an equipment, an implement, an agricultural vehicle, an aerial device, a monitor, a display device, a user edge device, a self-guided device, or a self-propelled device (e.g., a robot, a drone, an ATV, a UTV, etc.). The processing system executes instructions of a software application or program with processing logic. The software application or program can be started by the processing system. In one example, the monitor or display device receives user input and provides a customized display for the operation of methods 390 and 392. The software application can be run on a handheld device with a camera, a touch screen, a sensor, and processing resources with sufficient power to run an augmented reality (AR) engine. The device uses an object detection CNN to infer the location and emergence classification of selected plants in a camera image frame, a machine learning engine capable of running a model, and an AR framework. The trained model accepts images and supports different classifications, such as corn, soybean, and weed classification.After initialization, the software application activates the AR framework and uses the device's cameras and sensors to map the surroundings in three dimensions.

[0063] exist Figure 3E At operation 302 of method 390, one or more sensors (e.g., image sensors, cameras) of a device (e.g., edge device, self-propelled device) are positioned to observe plants in a field, and a software application is launched on a processing system of the device and displayed on a monitor or display device as a user interface of the device. The processing system can be integrated or coupled with a machine that performs an application process (e.g., planting, tilling, fertilizing, spraying, etc.). Alternatively, the processing system can be integrated with such a device (e.g., drone, edge device with image capture device) that is associated with the machine that captures images before, during, or after the application process. In one example, the user interface displays a real-time view of the plants in the field.

[0064] At operation 304, image data is obtained from one or more sensors of the device in response to a user input for initiating a continuous process of scouting plants using the device. The continuous scouting process may initiate a row tracking process to detect linear rows of biomass, and after completing several iterations of the row tracking process, initiate a plant tracking process and receive input from the row tracking process. The device may be located on an implement, a machine that pulls the implement during a current application flow, an aerial device, a user edge device, a self-guided device, a self-propelled device, etc. The sensors may be in-situ sensors located on each row unit of the implement, spaced apart every few row units, or located on the machine.

[0065] The software application activates an augmented reality framework that requires the device's camera and sensors to map the surrounding environment in three dimensions. The software application can provide instructions for angular guidance of the device while capturing images of the plants. Sensory data from the device's accelerometer, gyroscope, or other motion-based sensors can be used to suggest positioning and angular guidance of the device on a user interface to improve the performance of machine learning and augmented reality. The software application can provide instructions to the user, including providing the location of the device between plant rows, providing a suggested orientation of the device, and providing a suggested tilt of the device when capturing image data.

[0066] At operation 306, the computer-implemented method analyzes one or more input images from the image data (e.g., analyzing still images from a video stream from a camera), generates a tracking grid of 2D reference points projected onto a ground plane to create a set of matching 3D object tracking anchors and their locations in 3D space using augmented reality (e.g., an augmented reality framework), and the one or more input images are saved on the device.

[0067] At operation 358, one or more images obtained at operation 304 are then preprocessed, resized, and formatted according to the specifications of a machine learning (ML) model with a convolutional neural network (CNN), and provided as input to the ML model. The inputs may include images, tracking grids, and positions from a row tracking process. At operation 360, the ML model (e.g., an object detection model) is then executed using the ML engine of the processing system of the device for object detection. The architecture and weights of the object detection model used by the software application are developed and trained using a dataset representing plants in a farm field and annotations added to these inference outputs. Iterations of the object detection model improve the classification and positioning of plants in 3D space during use, and are fully packaged on an edge device that includes an ML engine (e.g., a neural engine), an AR framework with a software library, an efficient camera and image pipeline, and a GPS. Different crops can be evaluated and characterized with a single general neural network model. The updated model is trained and improved without the need for an app store update. The software application creates an intuitive user experience (similar to a gaming experience) based on the use of augmented reality and object detection modeling on the edge device to determine crop emergence levels and spacing between neighboring plants in a row while offline or online. Augmented reality and ML modeling are performed on the edge device so that the edge device can be offline without internet access and still be able to determine crop emergence levels and spacing between neighboring plants in a row.

[0068] At operation 362 , once the ML model execution is complete, the inference output of the ML model (eg, an array of detected objects with confidence levels) is filtered and adjusted using non-maximum suppression to select the most likely plant location and classification.

[0069] exist Figure 3F At operation 368 of the method, the method uses the object tracking anchor point to generate a corrected reasoning result position to compensate for the movement of the device during image capture, and projects the resulting corrected reasoning result position for the corrected result position into 3D space using an augmented reality (AR) framework. This results in the 3D point being temporarily stored as a pin marking the plant in 3D space.

[0070] In one example, at operation 370, once the plant markers have been created in 3D space, the plant markers are classified into another category of desired target plants (e.g., any crop, corn, soybeans, etc.) to be identified and scouted and other non-target plants. At operation 371, the locations of the target plant markers are sent to a row detection and tracking algorithm (method 392).

[0071] At operation 372, all plants are then processed using a fuzzy logic rule set. Based on the fuzzy logic rule set and filtering, the rules determine whether to add the target plant or row to the existing rows, or discard the results. In order to reduce results inferred from objects that are too far from the device's image sensor, when the user completes the run, if the detected plant is positioned more than a set distance from the device, the plant can be discarded. When the ML model finds that a plant is a duplicate detection, the plant is grouped with already existing plants and rows. The computer-implemented method is a novel algorithm designed to adjust the position and emergence classification of existing plants with new matching data, thereby slowly increasing the accuracy of plant detection over time. When it is determined at operation 380 that the plant is a new detection, labels and markers visible to the user are placed in the 3D space of the AR frame.

[0072] At operation 326, the method determines whether the plant tracking process is ended (e.g., the user ends or terminates the continuous scouting process and this ends the plant tracking process). After the user ends the plant tracking process, at operation 334, the rows and plants that have been detected and presented to the user are aggregated to display various statistics, such as spacing between plants in a row, row spacing, emergence levels, and financial impact.

[0073] To further leverage the collected data to provide accurate reconnaissance, a user interface is provided on the device so that the user can provide more information about the detected skip. For example, the detected skip can be characterized as Wrong seeds or none Germ This helps the user detect patterns and determine if the problem is due to the seed, machinery or environmental / ditch conditions.

[0074] Additionally, the software application is designed so that the user can selectively draw plant tags in 3D space and select / change classifications, as well as remove false positive automatic tags. If the plant tracking process continues, the method returns to Figure 3E Operation 304 is performed for another iteration. Figure 3G A row tracking method 392 is shown according to one embodiment. The method 392 may be implemented with a row detection and tracking algorithm that detects and tracks row positions and orientations. At operation 393, a corrected position (e.g., a corrected 3D point) of a target plant is received from an ML model ( Figure 3FAt operation 394, the algorithm determines a predicted row direction vector and a center position equidistant between adjacent rows based on a moving window of past row tracking results. The row tracking results in the window may be averaged to determine the predicted row direction vector and center position. If there are no past row tracking results, the orientation of the image sensor and the position of the device in the field may initially be used to predict the row direction vector and center position. The predicted row direction vector may be used to project an estimated center position between adjacent rows.

[0075] At operation 395, the corrected 3D point is then converted to a central position relative to the prediction and projected onto an axis orthogonal to the predicted direction vector rotated within a certain angle range. The projection of each angle is analyzed using kernel density estimation to determine the direction and spacing of each adjacent row. At operation 396, the direction and spacing values ​​are then used to calculate new row center, direction, and position values, which are appended to the window array of the current row tracking results. When new row tracking results arrive at the window array, the previous row tracking results can be moved out of the window array. These new row tracking results are then used to calculate row geometry when other features of method 390 require.

[0076] In statistics, kernel density estimation (KDE) is a nonparametric method that applies kernel smoothing to probability density estimation, that is, to estimate the probability density function of a random variable based on a kernel as a weight. KDE solves the basic data smoothing problem, where inference is made about a population based on a finite sample of data. One of the applications of kernel density estimation is to estimate the class-conditional marginal density of data when using a naive Bayes classifier, which can improve its predictive accuracy.

[0077] Figure 4 A flow chart of one embodiment of a computer-implemented method for implementing one or more rule sets for plant location and identification is shown. The method 400 is performed by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine or device), or a combination of both.

[0078] This one or more rule sets utilize row detection based on a custom straight line detection algorithm.At operation 401, method 400 applies fuzzy logic rule set to each newly detected plant of method 300 or 390, to adjust their category and probability based on the region where the plant is located, optionally their alignment with the biomass region and their distance from the user and other factors.At operation 402, method 400 filters out the plant whose probability or position is not within an acceptable threshold.At operation 403, method 400 adjusts the newly detected plant position to align with the nearest existing row.At operation 404, these newly detected plants are compared with existing plants and rows.At operation 405, the method determines whether to find a match based on proximity and angle.In one example, the newly detected plant between adjacent plant rows or within half of the row interval outside adjacent plant rows will be considered as matching with one of the adjacent rows.Otherwise, the newly detected plant is not considered as matching with one of the adjacent rows.Also the angle of the straight line of the newly detected plant relative to the nearby plant row can be used to determine whether the newly detected plant is considered as matching with the nearby plant row.

[0079] At operation 406, if a match is found, the new plants and rows are combined with the existing plants and rows. If a match cannot be found, at operation 414, the method determines that the new plants are new detections and these plants will be added to the nearest existing row.

[0080] When detecting crops in closely spaced rows, the crops may overlap, making it difficult for the object detection model to identify the presence of crops that are further away from the POV (viewing angle) of the image being analyzed. This results in lower confidence for some detections, causing them to fall below a minimum confidence threshold and be excluded from the final result. To compensate for this, algorithms were created that exploit the 3D geometry of the detected crop rows.

[0081] Once the row is established, the average spacing between plants in the row is determined. This average spacing is then applied along the row to find points between existing plants where plants may be present but have not yet been detected. Biodetection is used to determine the likely locations of the plants. These points (e.g., 3D points) are then projected into 2D on the camera frame. The location of the 2D points is then recorded and set as the focus for object detection model post-processing. When the object detection model produces results for the captured camera frame, post-processing checks for any results that fall on one of these focal points and reduces the minimum confidence threshold required to allow the results to pass the filtering step. This allows the software application to dynamically increase and decrease the degree of filtering of the object detection model results based on the likelihood of the region within the input image (e.g., operation 362).

[0082] In some embodiments, the operations of the methods disclosed herein may be changed, modified, combined or deleted. The methods in the embodiments of the present disclosure may be performed with a device, equipment or data processing system as described herein. The device, equipment or data processing system may be a conventional general-purpose computer system, or a special-purpose computer designed or programmed to perform only one function may be used.

[0083] Figure 5 A monitor or display of a device with a user interface 501 for displaying annotated target plants in 3D space using augmented reality is shown according to one embodiment. Processing logic executes instructions of a software application (e.g., a field application) launched by the processing system to generate the user interface 501 displayed by the monitor or display of the device (e.g., an edge device).

[0084] The software application can provide different display areas that the user can select on the user interface 501. In one example, the user enters data into different input fields (e.g., farm name, crop type, left row number, target population, field name, row spacing, right row number, and user ID) before displaying plant emergence. The user moves the device to establish initial orientation according to the instructions of the software application, and then the user walks between adjacent rows of emerging plants. The display area can be marked with selectable growth stages (e.g., VE stage when corn seedlings emerge from the soil and do not form leaf collars, V1 stage, V2, V3, V4, etc. when plants have a visible leaf collar). In this example, augmented reality has added annotations (e.g., 5.51, 7.58 ", 8.28 ", 7.24 ", 8.05 ", etc.) to the growth stage and the intervals between adjacent plants, and the shortest possible path connecting the plants in the same row connects 510 and 520. Optionally, computer vision and augmented reality are used to accurately score emergence and intervals in an intuitive and easy-to-use manner. The spacing values ​​may optionally be displayed with different background colors to characterize the spacing as good spacing (e.g., a green background indicating spacing values ​​of 6 or less between plants in the same row), misplaced seed spacing (e.g., a yellow background indicating spacing values ​​greater than 6" and less than 8"), and skipped seed spacing (e.g., a red background indicating spacing values ​​greater than 8"). The spacing values ​​may be adjusted for good spacing, misplaced seed spacing, and skipped seed spacing based on the crop type being scouted.

[0085] The user interface 501 may include a selectable option 550 for completing the scouting and a selectable option 552 for resuming plant scouting by continuing to capture images of the plants, which results in additional iterations of the object detection model. The software application may provide additional plant information, including stand counts, the number of seeds planted, and the number of plants grown per unit area. The expected results based on the number of seeds planted are compared to the number of plants grown to calculate a percentage. The stand counts are used to assess seed quality (germination rate) and whether replanting is necessary.

[0086] Figure 6 A monitor or display of a device 600 is shown with a user interface 601 for displaying annotated target plants in 3D space using augmented reality according to one embodiment. Processing logic executes instructions of a software application (e.g., a field application) launched by the processing system to generate the user interface 601 displayed by the monitor or display of the device (e.g., an edge device).

[0087] The software application can provide different display areas that the user can select on the user interface 601. In one example, a display area marked with a growth stage (e.g., VE stage when a corn seedling emerges from the soil and has not formed a leaf collar, V1 stage when the plant has a visible leaf collar, V2, V3, V4, etc.) is selectable. In this example, the user is selecting the growth stage of the target plant. Optionally, the straight line 610 moves from top to bottom to indicate that a new CNN cycle or iteration is being executed. The user interface 601 can display a growth stage indicator, a run option indicator, and a keep walking indicator to indicate the walking distance for reconnaissance. The user interface can also display setup options, resume run options, restart run options, and cancel run options.

[0088] Figure 7An example of a system 700 according to one embodiment is shown, the system including a machine 702 (e.g., a tractor, a combine harvester, etc.) and an implement 740 (e.g., a planter, a cultivator, a plow, a sprayer, a spreader, an irrigation implement, etc.). The machine 702 includes a processing system 720, a memory 705, a machine network 710 (e.g., a controller area network (CAN), a serial bus protocol network, an ISOBUS network, etc.), and a network interface 715 for communicating with other systems or devices including the implement 740. According to an embodiment of the present disclosure, the machine network 710 includes a sensor 712 (e.g., a speed sensor), a controller 711 (e.g., a GPS receiver, a radar unit) for controlling and monitoring machine operations, and an optional image capture device 714 for capturing images of crop and soil conditions of a field. The network interface 715 may include at least one of a GPS transceiver, a WLAN transceiver (e.g., WiFi), an infrared transceiver, a Bluetooth transceiver, Ethernet, or other interfaces for communicating with other devices and systems including the implement 740. The network interface 715 can be integrated with the machine network 710, or can be separate from the machine network 710, such as Figure 7 An I / O port 729 (eg, a diagnostic / on-board diagnostic (OBD) port) can communicate with another data processing system or device (eg, a display device, a sensor, etc.).

[0089] In one example, the machine performs the operation of a combine (combine harvester) for harvesting grain crops. The machine combines the harvesting, threshing and winnowing operations in a single harvesting operation. An optional header 780 (e.g., a grain platform, a flexible platform) includes a cutting mechanism to cause the cutting of the crop to be positioned in the auger. The header 780 includes an orienting device 782 or mechanism for orienting the crop (e.g., corn, soybeans) to improve image capture by the image capture device 784.

[0090] The processing system 720 may include one or more microprocessors, processors, systems on a chip (integrated circuits), or one or more microcontrollers. The processing system includes processing logic 726 for executing software instructions of one or more programs and a communication unit 728 (e.g., a transmitter, a transceiver) for sending and receiving communications to and from a machine via the machine network 710 or the network interface 715 or to and from an implement via the implement network 750 or the network interface 760. The communication unit 728 may be integrated with the processing system or separate from the processing system. In one embodiment, the communication unit 728 communicates data with the machine network 710 and the implement network 750 via a diagnostic / OBD port of an I / O port 729.

[0091] Processing logic 726, including one or more processors, may process communications including agricultural data received from communication unit 728. System 700 includes memory 705 for storing data and programs (software 706) for execution by the processing system. Memory 705 may store, for example, software components such as image capture software, software for performing scouting techniques of the disclosed operations or methods, or any other software applications or modules, images 708 (e.g., captured crop images), alerts, maps, etc. Memory 705 may be any known form of machine-readable non-transitory storage medium, such as semiconductor memory (e.g., flash memory; SRAM; DRAM, etc.) or non-volatile memory, such as a hard disk or solid-state drive. The system may also include an audio input / output subsystem (not shown), which may include a microphone and speaker for, for example, receiving and sending voice commands or for user authentication or authorization (e.g., biometrics).

[0092] Processing system 720 is in bidirectional communication with memory 705, machine network 710, network interface 715, header 780, display device 730, display device 725, and I / O port 729 via communication links 731 to 737, respectively.

[0093] Display devices 725 and 730 may provide a visual user interface for a user or operator. The display device may include a display controller. In one embodiment, the display device 725 (or computing device 725) is a portable tablet device or computing device with a touch screen that displays images (e.g., captured images, local view map layers, high-definition field maps of planted or harvested data or other agricultural variables or parameters, yield maps, alerts, etc.), or displays reconnaissance tools generated using machine learning, augmented reality, and computer vision as disclosed in the operations or methods described herein. The processing system 720 and the memory 705 may be integrated with the computing device 725 or separate from the computing device. The operations may include configuring the machine or implement, reporting data, controlling the machine or implement including sensors and controllers, and storing the generated data. The display device 730 can be a display (e.g., a display provided by an original equipment manufacturer (OEM)) that displays images and data for scouting applications, local view map layers, planting or harvesting data, yield data, thereby controlling a machine (e.g., a planter, tractor, combine, sprayer, etc.), maneuvering a machine, and monitoring a machine or an implement connected to a machine (e.g., a planter, combine, sprayer, etc.) using sensors and controllers located on the machine or implement.

[0094] The cab control module 770 may include additional control modules for enabling or disabling certain components or devices of the machine or implement. For example, if a user or operator is unable to control the machine or implement using one or more of the display devices, the cab control module may include switches to shut down or turn off components or devices of the machine or implement.

[0095] The implement 740 (e.g., a planter, a cultivator, a plow, a sprayer, a spreader, an irrigation implement, etc.) includes an implement network 750, a processing system 762 having processing logic 764, a network interface 760, and optional input / output ports 766 for communicating with other systems or devices including the machine 702. The implement network 750 (e.g., a controller area network (CAN), a serial bus protocol network, an ISOBUS network, etc.) includes an image capture device 756 for capturing images of crop development and soil conditions, sensors 752 (e.g., speed sensors, seed sensors for detecting seed passage, downforce sensors, actuator valves, OEM sensors, etc.), a controller 754 (e.g., a GPS receiver), and a processing system 762 for controlling and monitoring the operation of the machine. The OEM sensor may be a moisture sensor or flow sensor for a combiner, a speed sensor for a machine, a seed force sensor for a planter, a liquid application sensor for a sprayer, or a vacuum, lift, or lower sensor for an implement. For example, the controller may include a processor that communicates with a plurality of seed sensors. The processor is configured to process the images or seed sensor data captured by the image capture device 756 and transmit the processed data to the processing system 762 or 720. The controller and sensors can be used to monitor the motors and drives on the planter, including variable rate drive systems for changing plant populations. The controller and sensors can also provide harvesting width control to close individual rows or sections of the planter. The sensors and controllers can sense changes in the electric motors that control each row of the planter individually. These sensors and controllers can sense the seed delivery rate in the seed tubes for each row of the planter.

[0096] The network interface 760 may be a GPS transceiver, a WLAN transceiver (e.g., WiFi), an infrared transceiver, a Bluetooth transceiver, Ethernet, or other interface for communicating with other devices and systems including the machine 702. The network interface 760 may be integrated with the machine network 750 or separate from the machine network 750, such as Figure 7 shown.

[0097] Processing system 762 is in bidirectional communication with implement network 750 , network interface 760 , and I / O port 766 via communication links 741 - 743 , respectively.

[0098] The implements communicate with the machine via wired and possibly wireless two-way communications 704. The implement network 750 may communicate with the machine network 710 directly or via network interfaces 715 and 760. The implements may also be physically coupled to the machine to perform agricultural operations (e.g., planting, harvesting, spraying, etc.).

[0099] The memory 705 may be a machine-accessible, non-transitory medium having stored thereon one or more sets of instructions (e.g., software 706) embodying any one or more of the methods or functions described herein. The software 706 may also reside completely or at least partially within the memory 705 and / or the processing system 720 during its execution by the system 700, the memory and the processing system also constituting machine-accessible storage media. The software 706 may also be sent or received over a network via the network interface 715.

[0100] In one embodiment, a machine-accessible non-transitory medium (e.g., memory 705) contains executable computer program instructions that, when executed by a processing system, cause the system to perform operations or methods of the present disclosure, including operations to capture image data for use in a scouting application that can determine stand counts, plant heights, growth stages, and intervals of emerging plants to assess the quality of different agricultural operations. Although the computer-accessible non-transitory medium (e.g., memory 705) is shown as a single medium in the exemplary embodiment, the term "machine-accessible non-transitory medium" should be considered to include a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) that store one or more sets of instructions. The term "computer-accessible non-transitory medium" should also be considered to include any medium capable of storing, encoding, or carrying a set of instructions for execution by a machine and causing the machine to perform any one or more methods of the present disclosure. Therefore, the term "machine-accessible non-transitory medium" should be considered to include, but is not limited to, solid-state memory, optical and magnetic media, and carrier signals.

[0101] Example

[0102] The following are non-limiting examples.

[0103] Example 1 - A computer-implemented method for scouting plants in a farmland, comprising: obtaining image data from one or more sensors of a device in response to user input; analyzing one or more input images from the image data, generating a grid of two-dimensional (2D) reference points projected onto a ground plane to create a set of matching three-dimensional (3D) anchor points and their locations in the 3D space of the farmland using an augmented reality (AR) framework; providing the one or more input images, the tracking grid, and the locations in the 3D space of the farmland to a machine learning (ML) model having a convolutional neural network (CNN); and generating, with the ML model, an inference result including an array of detected objects and selecting the most likely inferred plant location and classification for the array of detected objects.

[0104] Example 2 - The computer-implemented method of Example 1, further comprising: using object tracking anchors to generate a corrected inference result position to compensate for movement of the device during image capture.

[0105] Example 3 - The computer-implemented method of Example 2, further comprising: projecting the corrected inference result position into the 3D space using the augmented reality (AR) framework.

[0106] Example 4 - The computer-implemented method of Example 3, further comprising: creating plant tags in the 3D space based on the positions of the corrected inference result positions; and classifying the plant tags into target plants to be identified and other non-target plants.

[0107] Example 5 - The computer-implemented method according to Example 4 further includes: inputting the position of the target plant into a row detection and tracking algorithm to detect linear rows of the target plant in the farmland and grouping the position of the target plant into the linear rows; determining a predicted row direction vector and a central position equidistant between adjacent rows based on a moving window of row tracking results; using the predicted row direction vector to project the estimated central position between the adjacent rows; converting the position of the target plant to be relative to the predicted central position and projecting it onto an axis orthogonal to the predicted row direction vector within a certain angle range; and using kernel density estimation to analyze the projection at each angle to determine the direction and spacing of each adjacent row.

[0108] Example 6 - The computer-implemented method of Example 5 further includes: using a fuzzy logic rule set to process all plants to determine whether to add a target plant or row to an existing row, whether to add the target plant as a separate marker, or to discard the inference result, wherein if the detected plant is positioned more than a set distance from the device, the detected plant is discarded.

[0109] Example 7 - The computer-implemented method of Example 6, further comprising: adding labels and markers to the 3D space of the AR framework upon determining a new detection at a plant position of the inference result for the plant, wherein the labels and markers are visible to a user on a display of the device.

[0110] Example 8 - The computer-implemented method of Example 7, wherein the label indicates a growth stage of the plant and further adds a spacing between the plant and neighboring plants to the 3D space of the AR frame to indicate an assessment of the planting.

[0111] Example 9 - The computer-implemented method of Example 8, further comprising: upon detecting a new row or plant, connecting the plants in the row via the shortest possible path; and displaying the connections and separation distances between adjacent plants to the display of the device.

[0112] Example 10 - The computer-implemented method of Example 1, wherein the device includes an edge device to determine crop emergence levels and spacing between adjacent plants in a row.

[0113] Example 11 - A computer device comprising: a display device for displaying a user interface to show plants present in a farmland, the plants being augmented with computer-generated sensory information for scouting the plants; and a processor coupled to the display device, the processor being configured to: obtain image data from one or more sensors; analyze one or more input images from the image data; generate three-dimensional (3D) anchor points and their positions in the 3D space of the farmland using an augmented reality (AR) framework; provide the one or more input images and the positions in the 3D space of the farmland to a machine learning (ML) model having a convolutional neural network (CNN); and generate, with the ML model, an inference result including an array of detected objects and select the most likely inferred plant position and classification for the array of detected objects.

[0114] Example 12 - The computing device of Example 11, wherein the processor is configured to generate a corrected inference result position using object tracking anchors to compensate for movement of the computing device during image capture.

[0115] Example 13 - The computing device of Example 12, wherein the processor is configured to project the corrected inference result position into the 3D space using the augmented reality (AR) framework.

[0116] Example 14 - A computing device according to Example 13, wherein the processor is configured to create plant tags in the 3D space based on the position of the corrected inference position, and to classify the plant tags into target plants to be identified and other non-target plants.

[0117] Example 15 - A computing device according to Example 14, wherein the processor is configured to: provide the input position of the target plant to a row detection and tracking algorithm to detect linear rows of the target plants in the farmland and group the positions of the target plants into the linear rows; determine a predicted row direction vector and a central position equidistant between adjacent rows based on a moving window of row tracking results; use the predicted row direction vector to project the estimated central position between the adjacent rows; convert the position of the target plant to be relative to the predicted central position and project it onto an axis orthogonal to the predicted row direction vector within a certain angle range; and use kernel density estimation to analyze the projection at each angle to determine the direction and spacing of each adjacent row.

[0118] Example 16 - A computing device according to Example 15, wherein the processor is configured to process all plants using a fuzzy logic rule set to determine whether to add a target plant or row to an existing row, whether to add the target plant as a separate tag, or to discard the inference result if the inference result exceeds a maximum threshold interval or falls below a minimum threshold interval, the threshold interval being a possible interval from the location of the inference result to one or more locations of existing plants and rows.

[0119] Example 17 - A computing device according to Example 16, wherein the processor is configured to add a label and a marker to the 3D space of the AR framework upon determining a new detection of a plant position for an inference result of a plant, wherein the label and the marker are visible to a user on a display of the device.

[0120] Example 18 - A computing device according to Example 17, wherein the label indicates a growth stage of the plant and further adds a spacing between the plant and neighboring plants to the 3D space of the AR frame to indicate an assessment of the planting.

[0121] Example 19 - The computing device of Example 18, wherein the label characterizes the interval as one of a good interval, a misplaced seed interval, and a skipped seed interval.

[0122] Example 20 - A computing device according to Example 18, wherein the processor is configured to connect the plants in the row via the shortest possible path upon detecting a new row or plant, wherein the display device is configured to display the connection and the separation distance between adjacent plants.

[0123] It should be understood that the above description is intended to be illustrative and not restrictive. Those skilled in the art will immediately understand many other embodiments after reading and understanding the above description. Therefore, the scope of the present disclosure should be determined with reference to the full scope of the equivalents to which the attached claims and such claims are entitled.

Claims

1. A computer-implemented method for scouting plants in a field, comprising: obtaining image data from one or more sensors of the device in response to an input for initiating a continuous process of scouting for vegetation; analyzing one or more input images from the image data, generating a tracking grid of two-dimensional (2D) reference points projected onto a ground plane to create a set of matching three-dimensional (3D) anchor points and their locations in 3D space of the farmland using an augmented reality (AR) framework; providing the one or more input images, the tracking grid, and the position in the 3D space of the field to a machine learning (ML) model having a convolutional neural network (CNN); and An inference result is generated using the ML model including an array of detected objects and a most likely inferred plant location and classification is selected for the array of detected objects.

2. The computer-implemented method of claim 1 , further comprising: Object tracking anchors are used to generate a corrected inference position to compensate for movement of the device during image capture.

3. The computer-implemented method of claim 2, further comprising: The corrected inference result position is projected into the 3D space using the augmented reality (AR) framework.

4. The computer-implemented method of claim 3, further comprising: creating a plant marker in the 3D space based on the position of the corrected inference result position; as well as The plant markers are divided into target plants to be identified and other non-target plants.

5. The computer-implemented method of claim 4, further comprising: inputting the locations of the target plants into a row detection and tracking algorithm to detect linear rows of the target plants in the agricultural field and grouping the locations of the target plants into the linear rows; Determining a predicted row direction vector and a center position equidistant between adjacent rows based on a moving window of row tracking results; projecting an estimated center position between the adjacent rows using the predicted row direction vector; Converting the position of the target plant to be relative to the predicted central position and projecting it onto an axis orthogonal to the predicted row direction vector within a certain angle range; as well as The projection at each angle is analyzed using kernel density estimation to determine the direction and spacing of each adjacent row.

6. The computer-implemented method of claim 5, further comprising: All plants are processed using a fuzzy logic rule set to determine whether to add a target plant or row to an existing row or to discard inference results based on the fuzzy logic rule set and filtering, wherein a detected plant is discarded if it is located more than a set distance from the device.

7. The computer-implemented method of claim 6, further comprising: Upon determining a new detection of a plant position for an inference result of a plant, adding a label and a marker to the 3D space of the AR framework, wherein the label and the marker are visible to a user on a display of the device.

8. The computer-implemented method of claim 7, wherein the label indicates a growth stage of the plant, and a spacing between the plant and neighboring plants is also added to the 3D space of the AR frame to indicate an assessment of the planting.

9. The computer-implemented method of claim 8, further comprising: Upon detecting a new row or plant, connecting the plants in the row via the shortest possible path; as well as The connections and separation distances between adjacent plants are displayed to the display of the device.

10. The computer implemented method of claim 1, wherein the device comprises an edge device to determine crop emergence levels and spacing between adjacent plants in a row.

11. A computer device comprising: a display device for displaying a user interface to show plants present in a farm field, the plants being enhanced with computer-generated sensory information for scouting the plants; as well as A processor coupled to the display device, the processor being configured to: obtain image data from one or more sensors; analyze one or more input images from the image data; generate three-dimensional (3D) anchor points and their positions in the 3D space of the farmland using an augmented reality (AR) framework; provide the one or more input images and the positions in the 3D space of the farmland to a machine learning (ML) model having a convolutional neural network (CNN); and generate an inference result including an array of detected objects with the ML model and select the most likely inferred plant position and classification for the array of detected objects.

12. The computing device of claim 11, wherein the processor is configured to generate a corrected inference result position using object tracking anchors to compensate for movement of the computing device during image capture. 13 . The computing device of claim 12 , wherein the processor is configured to project the corrected inference result position into the 3D space using the augmented reality (AR) framework.

14. The computing device of claim 13, wherein the processor is configured to create plant tags in the 3D space based on the positions of the corrected inference result positions, and to classify the plant tags into target plants to be identified and other non-target plants.

15. The computing device of claim 14, wherein the processor is configured to: inputting the locations of the target plants into a row detection and tracking algorithm to detect linear rows of the target plants in the agricultural field and grouping the locations of the target plants into the linear rows; Determine a predicted row direction vector and a center position equidistant between adjacent rows based on a moving window of row tracking results; projecting an estimated center position between the adjacent rows using the predicted row direction vector; Converting the position of the target plant to be relative to the predicted central position and projecting it onto an axis orthogonal to the predicted row direction vector within a certain angle range; as well as The projection at each angle is analyzed using kernel density estimation to determine the direction and spacing of each adjacent row.

16. The computing device of claim 15, wherein the processor is configured to process all plants using a fuzzy logic rule set to determine whether to add a target plant or row to an existing row or to discard inference results based on the fuzzy logic rule set and filtering.

17. A computing device according to claim 16, wherein the processor is configured to add labels and markers to the 3D space of the AR framework after determining a new detection of a plant position for an inference result of a plant, wherein the labels and markers are visible to a user on a display of the device. 18 . The computing device of claim 17 , wherein the label indicates a growth stage of the plant, and further adds a spacing between the plant and neighboring plants to the 3D space of the AR frame to indicate an assessment of planting.

19. The computing device of claim 18, wherein the label characterizes the interval as one of a good interval, a misplaced seed interval, and a skipped seed interval.

20. The computing device of claim 18, wherein the processor is configured to connect plants in the row via the shortest possible path upon detecting a new row or plant, wherein the display device is configured to display the connection and the separation distance between adjacent plants.

Images