Systems and methods for object tracking and location prediction

Abstract

Disclosed herein are methods, devices, modules, and systems that can be used to accurately track and subsequently target objects of interest relative to a moving body, such as a vehicle. The object tracking methods can be implemented by a detection system in which a sensor is coupled to the moving body. The targeting system can be used to target objects tracked by the detection system, such as for automating crop cultivation or maintenance. The devices disclosed herein can be configured to locate, identify, and autonomously target weeds with a beam, such as a laser beam, that can burn or irradiate the weeds. These methods, devices, modules, and systems can be used for agricultural crop management or home weed control.

Description

[Background technology]

[0001] cross reference This application claims the benefit of U.S. Provisional Application No. 63 / 300,999, filed January 19, 2022, which is incorporated herein by reference.

[0002] As technology advances, tasks previously performed by humans are becoming increasingly automated. Tasks performed in highly controlled environments, such as a factory assembly line, can be automated by instructing a machine to perform the task the same way every time, while tasks performed in unpredictable environments, such as driving around town or vacuuming a cluttered room, rely on dynamic feedback and adaptation to perform the task. Autonomous systems often struggle to identify and locate objects in unpredictable environments. Improvements in object tracking methods advance automation technology, improving the ability of autonomous systems to react and adapt to unpredictable environments. Summary of the Invention [Means for solving the problem]

[0003] In various aspects, the present disclosure provides a method for predicting a position of a tracked object relative to a moving vehicle, the method including: determining, within a first image at a first time, a first set of real positions of a first object set relative to the moving vehicle; determining, within a second image at a second time, a second set of real positions of a second object set relative to the moving vehicle, the second object set including one or more of the same objects as the first object set; applying a plurality of test shifts to the first set of real positions of the first object set to obtain a set of test positions of the first object set at a second time, each test shift of the plurality of test shifts being determined based on a distance between a first real position of the first object at the first time and a second real position of the second object at the second time; determining a plurality of offsets between each of the test positions of the set of test positions of the first object set at the second time and each of the second real position set of the second object set at the second time, each offset of the plurality of offsets corresponding to one test shift of the plurality of test shifts; selecting a refinement shift from the plurality of test shifts based on the plurality of offsets; and identifying a tracked object included in both the first object set and the second object set.

[0004] In some aspects, the method further includes applying a refinement shift to the first set of actual positions of the first set of objects at the first time to establish a set of shifted positions of the first set of objects at a second time. In some aspects, the method further includes determining a vector displacement of the tracked objects between the first time and the second time from the actual positions of the tracked objects in the first image at the first time and the actual positions of the tracked objects in the second image at the second time. In some aspects, the refinement shift is a test shift from a plurality of test shifts having a minimum corresponding offset. In some aspects, the method further includes determining a vector velocity of the moving vehicle.

[0005] In various aspects, the present disclosure provides a method for predicting positions of tracked objects relative to a moving vehicle, the method including: determining a first set of actual positions of a first set of objects relative to the moving vehicle in a first image at a first time; determining a second set of actual positions of a second set of objects relative to the moving vehicle in a second image at a second time, the second set of objects including one or more of the same objects as the first set of objects; applying a predicted shift to the first set of actual positions of the first set of objects to generate an intermediate set of positions, the predicted shift being based on a vector velocity of the moving vehicle and a time difference between the first and second times; and applying the second shift to the intermediate set of positions to generate intermediate sets of positions of the first set of objects at the second time. the first object in the first image at the second time, wherein the first object is located at a first intermediate position in the intermediate position set and the second object is located at a second time; acquiring a set of shift positions of the tracked objects included in the first object set, the second shift being determined based on a distance between a first intermediate position of the first object in the intermediate position set and a second actual position of the second object at the second time; identifying tracked objects included in the first object set; specifying a search area; selecting from the second actual position set a real position of the tracked object from the second object set in the second image at the second time, wherein the real position of the tracked object in the second image at the second time is within the search area; and determining a vector displacement of the tracked object between the first time and the second time from the real position of the tracked object in the first image at the first time and the real position of the tracked object in the second image at the second time.

[0006] In some aspects, an intermediate position of the tracked object from the intermediate position set is within the search area. In some aspects, the method includes specifying the search area around the intermediate position of the tracked object. In some aspects, the method includes specifying the search area around the actual position of the tracked object.

[0007] In various aspects, the present disclosure provides a method for predicting positions of tracked objects relative to a moving vehicle, the method including: determining, in a first image at a first time, a first set of actual positions of a first set of objects relative to the moving vehicle; determining, in a second image at a second time, a second set of actual positions of a second set of objects relative to the moving vehicle, the second set of objects including one or more of the same objects as the first set of objects; identifying tracked objects included in the first set of objects; applying a predicted shift to the actual positions of the tracked objects in the first image at the first time to generate shifted positions of the tracked objects at the second time, the predicted shift being based on a vector velocity of the moving vehicle and a time difference between the first and second times; specifying a search area; selecting from the second set of actual positions an actual position of the tracked object from the second set of objects in the second image at the second time, the actual position of the tracked object in the second image at the second time being within the search area; and determining a vector displacement of the tracked object between the first and second times from the actual position of the tracked object in the first image at the first time and the actual position of the tracked object in the second image at the second time.

[0008] In some aspects, the method includes specifying a search area around a shifted position of the tracked object. In some aspects, the method includes specifying a search area around an actual position of the tracked object. In some aspects, the shifted position of the tracked object is within the search area. In some aspects, the method includes applying the predicted shift to each position of the first set of actual positions to generate a set of shifted positions. In some aspects, the method further includes applying a second shift to the shifted positions to generate revised shifted positions of the tracked object, the second shift being based on a distance between a shifted position of the set of shifted positions and an actual position of the second set of actual positions. In some aspects, the method includes specifying a search area around the revised shifted position of the tracked object. In some aspects, the revised shifted position of the tracked object is within the search area.

[0009] In some aspects, the method further includes determining a trajectory of the tracked object over time based on the vector displacement of the tracked object between the first and second times and the vector velocity of the moving vehicle. In some aspects, the method further includes determining a predicted position of the tracked object at a third time based on the vector displacement of the tracked object between the first and second times and the elapsed time between the second and third times.

[0010] In various aspects, the present disclosure provides a method for predicting a position of a tracked object relative to a moving vehicle, the method including determining a first set of actual positions of a first set of objects relative to the moving vehicle within a first image at a first time, identifying tracked objects included in the first set of objects, determining a vector velocity of the moving vehicle, and determining a predicted position of the tracked object at a second time based on the vector velocity, a time difference between the first time and the second time, and the actual position of the tracked object in the first image at the first time.

[0011] In some aspects, the vector velocity is determined using optical flow, a rotary encoder, a global positioning system, or a combination thereof. In some aspects, the optical flow is performed using consecutive image frames. In some aspects, the optical flow is determined using image frames that overlap by at least 10%. In some aspects, the optical flow is determined using image frames that are displaced by about 90% or less.

[0012] In some aspects, the first real set of positions, the second real set of positions, the shifted set of positions, the predicted positions, or a combination thereof, includes one dimension and two dimensions. In some aspects, the one dimension is parallel to a direction of motion of the moving vehicle. In some aspects, the one dimension and the two dimensions are parallel to an image plane that includes the first image or the second image. In some aspects, the first real set of positions, the second real set of positions, the shifted set of positions, the predicted positions, or a combination thereof, further includes a third dimension. In some aspects, the third dimension is perpendicular to an image plane that includes the first image or the second image. In some aspects, the third dimension is determined based on a vector velocity of the moving vehicle and a vector displacement of the tracked object. In some aspects, the test shift, the predicted shift, or the refinement shift is applied along one dimension. In some aspects, the test shift, the predicted shift, or the refinement shift is applied in one dimension and two dimensions. In some aspects, the test shift, the predicted shift, or the refinement shift is applied in one dimension, two dimensions, and three dimensions.

[0013] In some aspects, the method further includes determining a parameter of the tracked object. In some aspects, the parameter includes a size, a shape, a plant category, an orientation, or a plant type. In some aspects, the plant category is a weed or a crop. In some aspects, the method further includes determining a confidence value for the parameter. In some aspects, the method further includes determining a second predicted position based on the predicted and actual position of the tracked object in the first image at the first time, the actual position of the tracked object in the second image at the second time, or both.

[0014] In some aspects, the method further includes targeting the tracked object at the predicted location. In some aspects, targeting the tracked object includes correcting for parallax effects, first image distortion aberrations, second image distortion aberrations, targeting distortion aberrations, or combinations thereof. In some aspects, targeting the tracked object includes aiming a targeting sensor, an instrument, or both, toward the predicted location. In some aspects, targeting the tracked object further includes dynamically tracking the object with the targeting sensor, an instrument, or both. In some aspects, dynamically tracking the object includes moving the targeting sensor, an instrument, or both to match a vector velocity of the moving vehicle such that the targeting sensor, an instrument, or both are aimed toward the predicted location while the moving vehicle is moving. In some aspects, the instrument includes a laser, a sprayer, or a grabber. In some aspects, targeting the tracked object further includes manipulating the tracked object with the instrument. In some aspects, manipulating the tracked object includes irradiating the tracked object with electromagnetic radiation, moving the tracked object, spraying the tracked object, or combinations thereof. In some embodiments, the electromagnetic radiation is infrared light.

[0015] In some aspects, the tracked objects are positioned on a surface. In some aspects, the surface is a ground surface, a dirt surface, or a farm field. In some aspects, the first set of objects, the second set of objects, the tracked objects, or a combination thereof include plants. In some aspects, the first set of objects, the second set of objects, the tracked objects, or a combination thereof include weeds. In some aspects, the first set of objects, the second set of objects, the tracked objects, or a combination thereof include crops. In some aspects, the tracked objects are plants. In some aspects, the tracked objects are weeds. In some aspects, the tracked objects are crops. In some aspects, the moving vehicle is a trailer or an autonomous vehicle. In some aspects, the method further includes collecting the first image, the second image, or both by a sensor. In some aspects, the sensor is coupled to the moving vehicle.

[0016] In various aspects, the present disclosure provides an object tracking system, the object tracking system including a sensor configured to collect a first image at a first time and a second image at a second time, the sensor coupled to a vehicle, and a targeting system having an instrument; a prediction system configured to determine within the first image at the first time a first set of real positions of a first set of objects relative to the vehicle, and to determine within the second image at the second time a second set of objects relative to the vehicle, the second set of objects including one or more of the same objects as the first set of objects; and configured to apply a plurality of test shifts to the first set of real positions of the first set of objects to obtain a set of test positions for the first set of objects at the second time. a refinement shift is configured to select a refinement shift from the plurality of test shifts based on the plurality of offsets; each test shift of the plurality of test shifts is determined based on a distance between a first real position of the first object at a first time and a second real position of the second object at a second time; determining a plurality of offsets between each of the test positions of the set of test positions of the first object set at the second time and each of the second real positions of the second object set at the second time, each offset of the plurality of offsets corresponding to a test shift of the plurality of test shifts; and selecting a refinement shift from the plurality of test shifts based on the plurality of offsets; and identifying tracked objects included in both the first object set and the second object set; and a targeting system configured to target the tracked objects using the instrument.

[0017] In some aspects, the vehicle is configured to move relative to the surface. In some aspects, the first set of objects, the second set of objects, the tracked objects, or a combination thereof are positioned on the surface. In some aspects, the first set of objects, the second set of objects, the tracked objects, or a combination thereof include vegetation. In some aspects, the first set of objects, the second set of objects, the tracked objects, or a combination thereof include weeds. In some aspects, the first set of objects, the second set of objects, the tracked objects, or a combination thereof include crops.

[0018] In some aspects, the vehicle includes an autonomous vehicle or trailer. In some aspects, the sensor is a camera. In some aspects, the sensor is fixed relative to the vehicle. In some aspects, the instrument is configured to direct toward the tracked object and manipulate the tracked object. In some aspects, the targeting system further includes a targeting sensor. In some aspects, the targeting sensor is configured to direct toward the tracked object and image the tracked object. In some aspects, the targeting system is configured to aim the instrument at the tracked object based on a position of the tracked object in a targeting image collected by the targeting sensor. In some aspects, the instrument is a laser. In some aspects, the object tracking system is configured to perform the methods described herein.

[0019] Incorporation by Reference All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference.

[0020] The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings, in which: [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 illustrates an isometric view of an autonomous laser weeding vehicle, according to one or more embodiments herein. [Diagram 2] FIG. 1 illustrates a top view of an autonomous laser weeding vehicle traveling through a crop field while implementing various techniques described herein. [Diagram 3] FIG. 1 illustrates a side view of a detection system positioned on an autonomous laser weeding vehicle in accordance with one or more embodiments herein. [Figure 4] FIG. 1 is a block diagram illustrating components of a prediction and targeting system for identifying, locating, targeting, and manipulating objects in accordance with one or more embodiments of the present disclosure. [Diagram 5] 1 is a flowchart illustrating a procedure for predicting weed location according to one or more embodiments herein. [Figure 6] 1 is a flowchart illustrating a procedure for de-duplicating weeds identified in a set of images and predicting weed locations in accordance with one or more embodiments herein. [Figure 7] 1 illustrates a procedure for de-duplicating identified weeds in a set of images and predicting weed locations, according to one or more embodiments herein. [Figure 8A] 1 illustrates a procedure for de-duplication of identified weeds in a set of images, according to one or more embodiments herein. [Figure 8B] 1 illustrates a procedure for de-duplication of identified weeds in a set of images, according to one or more embodiments herein. [Figure 8C]1 illustrates a procedure for de-duplication of identified weeds in a set of images, according to one or more embodiments herein. [Figure 9] FIG. 2 is a block diagram illustrating components of a detection terminal according to an embodiment of the present disclosure. [Figure 10] FIG. 1 is an example block diagram of a computing device architecture for a computing device capable of implementing various techniques described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Various exemplary embodiments of the present disclosure are described in detail below. Although specific embodiments are described, it should be understood that this description is for illustrative purposes only. Those skilled in the relevant art will recognize that other components and configurations can be used without departing from the spirit and scope of the present disclosure. Therefore, the following description and drawings are illustrative and should not be construed as limiting. Numerous specific details are described to provide a thorough understanding of the present disclosure. However, in certain instances, well-known or conventional details are not described to avoid obscuring the description. References to one or an embodiment in the present disclosure may be references to the same embodiment or any embodiment, and such references mean at least one of the exemplary embodiments.

[0023] Reference to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the present disclosure. The phrase "in one embodiment" appears in various places in this specification not necessarily all referring to the same embodiment, nor are separate or alternative exemplary embodiments mutually exclusive with other exemplary embodiments. Furthermore, various features are described that may be exhibited by some exemplary embodiments but not by other embodiments. Any feature of an example can be integrated or used with any other feature of any other example.

[0024] The terms used herein generally have their ordinary meaning in the art, within the context of this disclosure and within the specific context in which each term is used. Alternative language and synonyms may be used for any one or more of the terms discussed herein, and no special significance should be placed on whether a term is detailed or discussed herein. In some cases, synonyms for a particular term are provided. The description of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification, including examples of any term described herein, is for illustrative purposes only and is not intended to further limit the scope and meaning of the disclosure or any exemplary term. Similarly, the disclosure is not limited to the various exemplary embodiments provided herein.

[0025] Although not intended to limit the scope of the present disclosure, examples of instruments, devices, methods and their related results according to exemplary embodiments of the present disclosure are given below. Please note that titles or subtitles may be used in the examples for the convenience of the reader and are not intended to limit the scope of the present disclosure in any way. Unless otherwise defined, technical and scientific terms used herein have the meanings commonly understood by those skilled in the art to which the present disclosure pertains. In case of conflict, the present specification, including definitions, shall prevail.

[0026] Additional features and advantages of the present disclosure will be set forth in and, in part, will be apparent from the description which follows, or may be learned by the practice of the principles disclosed herein. The features and advantages of the present disclosure may be realized and obtained by means of the instructions and combinations particularly pointed out in the appended claims. These and other features of the present disclosure will become more fully apparent from the following description and appended claims, or may be learned by the practice of the principles as set forth herein.

[0027] For clarity of explanation, in some cases, the technology may be presented as including individual functional blocks that represent devices, device components, steps or routines in a method implemented in software, or a combination of hardware and software.

[0028] In the figures, certain structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order may not be required. Rather, in some embodiments, such features may be arranged in a different manner and / or order than that shown in the exemplary figures. Furthermore, the inclusion of a structural or method feature in a particular figure does not imply that such feature is required in all embodiments, and that in some embodiments it may not be included or may be combined with other features.

[0029] While the concepts of the present disclosure are susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and are herein described in detail. It should be understood, however, that there is no intention to limit the concepts of the present disclosure to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives consistent with the scope of the present disclosure and the appended claims.

[0030] Described herein are systems and methods for tracking objects relative to a moving body, such as a vehicle, and predicting the object's future location with high accuracy. Tracking objects, particularly from a moving body (e.g., a moving vehicle), can be difficult due to many factors, including the speed of the moving body, changes in the moving body's position, changes in the object's shape or position over time, the object's motion relative to the surface, full or partial obfuscation of the object, the density of the tracked object, its proximity to other objects, optical distortion aberrations of the detection system, and parallax effects due to variations in object distance and changes in viewing angle. In some embodiments, object tracking may consider the object's motion relative to the moving body, the object's motion relative to the surface, or both. The systems and methods of the present disclosure enable tracking of multiple objects from a moving vehicle over time to target objects in three dimensions with high accuracy at predicted future locations. The methods of the present disclosure may be used to determine the trajectory of an individual object relative to a moving system by precisely locating the same object in two or more images that are separate in time and space. Because objects such as weeds or other vegetation may be similar in appearance, existing software may be inadequate at distinguishing and tracking multiple objects with sufficiently high accuracy, especially as object density and displacement between image frames increases.

[0031] A system of the present disclosure, such as the laser weeding system shown in Figures 1, 2, and 3, may collect images of a surface as it moves along the surface. Objects identified in the collected images may be tracked to predict the object's position relative to the system at a later time. As described herein, an object identified in a first image may be paired with the same object identified in a second image, and a relative trajectory may be determined for each paired object. Pairing objects in the first and second images may be performed by applying a shift to the object in the first image and comparing the position of the object shifted from the first image to the position of the object in the second image. An object from the second image that is in a matching position near the shifted position of the object from the first image may be identified as the same object identified in the first image. Such objects that appear in both the first and second images may be de-duplicated by assigning the object's position in the first image and its position in the second image to the same object.

[0032] The tracked object set may be differentially detected by combining objects identified in two or more images, adding newly identified objects to the set, and removing objects that are no longer present. A trajectory for each combined object that appears in two or more images may be determined from its position in the two or more images, and the trajectory may be used to predict the object's position at a later time. The images may be consecutive or non-consecutive images. Alternatively, or in addition, the object's trajectory may be determined based on a single position and vector velocity of the moving body relative to the object or the surface on which the object is located. In some embodiments, this predicted position may be targeted to manipulate or interact with the object at a later time using a targeting system, for example including a laser. This tracking method allows for precise targeting of the object by a moving system, such as a laser weeding system.

[0033] As used herein, an "image" may refer to a representation of an area or object. For example, an image may be a visual representation of an area or object formed by electromagnetic radiation (e.g., light, x-ray, microwave, or radio waves) scattered from the area or object. In another example, an image may be a point cloud model formed by a Light Detection and Ranging (LIDAR) or Radio Detection and Ranging (RADAR) sensor. In another example, an image may be a sonogram generated by detecting sound, infrasonic, or ultrasonic waves reflected from an area or object. As used herein, "imaging" may be used to describe the process of collecting or generating a representation (e.g., an image) of an area or object.

[0034] As used herein, a position, such as a position of an object or a position of a sensor, may be expressed relative to a reference frame. Exemplary reference frames include a surface reference frame, a vehicle reference frame, a sensor reference frame, or an actuator reference frame. A position can be easily converted between reference frames, for example, by using a conversion factor or a calibration model. Although a position, a change in position, or an offset may be expressed in one reference frame, it should be understood that a position, a change in position, or an offset may be expressed in any reference frame or easily converted between reference frames.

[0035] As used herein, a "sensor" may refer to a device that can detect or measure an event, a change in an environment, or a physical property. For example, a sensor may detect light, such as visible light, ultraviolet light, or infrared light, and generate an image. Examples of sensors include a camera (e.g., a charge-coupled device (CCD) camera or a complementary metal-oxide semiconductor (CMOS) camera), a LIDAR detector, an infrared sensor, an ultraviolet sensor, or an x-ray detector.

[0036] As used herein, an "object" may refer to an item or distinct area that may be observed, tracked, manipulated, or targeted. For example, an object may be a plant, such as a crop or a weed. In another example, an object may be a piece of debris. In another example, an object may be a distinct area or point on a surface, such as a marking or surface irregularity.

[0037] As used herein, "targeting" or "aiming" may refer to pointing or directing a device or action at a particular location or object. For example, targeting an object may include pointing a sensor (e.g., a camera) or an instrument (e.g., a laser) at the object. Targeting or aiming may be dynamic, such that the device or action follows an object that moves relative to the targeting system. For example, a device positioned on a moving vehicle may dynamically target or aim at an object on the ground by following the object as the vehicle moves relative to the ground.

[0038] As used herein, a "weed" may refer to an unwanted plant, such as an unwanted type of plant, or a plant that grows in an undesirable location or at an undesirable time. For example, a weed may be a wild plant or an invasive plant. In another example, a weed may be a non-cultivated plant in a cultivated crop field. In another example, a weed may be a plant that grows outside or between the rows of a cultivated crop.

[0039] As used herein, "manipulating" an object may refer to performing an action on, interacting with, or changing the state of an object. For example, manipulation may include illuminating, lighting, heating, incinerating, killing, moving, lifting, grasping, spraying, or otherwise altering the object.

[0040] As used herein, "electromagnetic radiation" may refer to radiation from the entire electromagnetic spectrum, which may include, but is not limited to, visible light, infrared light, ultraviolet light, radio waves, gamma rays, or microwaves.

[0041] Autonomous weeding system Object tracking methods may be implemented by an autonomous weeding system to target and remove weeds. Tracking methods such as these may facilitate object tracking relative to a moving body, such as a moving vehicle. For example, an autonomous weeding system may be used to track a weed identified in an image or representation collected by a first sensor, such as a predictive sensor, relative to the autonomous weeding system over time while the system is moving relative to the target weed. The tracking information may be used to determine a predicted location of the weed relative to the system at a later time. The autonomous weeding system may then use the predicted location to locate the same weed in an image or representation collected by a second sensor, such as a targeting sensor. In some embodiments, the first sensor is a predictive camera and the second sensor is a targeting camera. One or both of the first and second sensors may be moving relative to the weed. For example, the predictive camera may be coupled to the autonomous weeding system and moved thereby.

[0042] Targeting the weeds may include precisely locating the weeds using a targeting sensor, targeting the weeds with a laser, and removing the weeds by burning them with a laser light, such as infrared light. The predictive sensor may be part of a predictive module configured to determine a predicted position of the target object, and the targeting sensor may be part of a targeting module configured to refine the predicted position of the target object to determine a target position and target the target object with a laser at the target position. The predictive module may be configured to communicate with the targeting module to adjust the camera handoff using point-to-point targeting as described herein. The targeting module may target the object at the predicted position. In some embodiments, the targeting module may dynamically target the object using the trajectory of the object while the system is moving to adjust the position of the targeting sensor, the laser, or both to maintain the target.

[0043] The autonomous weeding system may identify, target, and remove weeds without human input. Optionally, the autonomous weeding system may be positioned on a self-driving or steered vehicle, or may be a trailer towed by another vehicle, such as a tractor. As shown in FIG. 1, the autonomous weeding system may be part of or coupled to a vehicle 100, such as a tractor or self-driving vehicle. The vehicle 100 may travel through a crop field 200, as shown in FIG. 2. As the vehicle 100 travels through the field 200, it may identify, target, and remove weeds in a non-weeded section 210 of the field, leaving a weeded field 220 behind it. The object tracking methods described herein may be implemented by the autonomous weeding system to identify, target, and remove weeds while the vehicle 100 is moving. The high accuracy of tracking methods such as these allows the use of lasers or the like to precisely target weeds and remove them without damaging nearby crops.

[0044] Detection System In some embodiments, the object tracking methods described herein may be performed by a detection system. The detection system may include a prediction system and, optionally, a targeting system. In some embodiments, the detection system may be positioned on or coupled to a vehicle, such as an autonomous weeding vehicle or a tractor-towed laser weeding system trailer. The prediction system may include a prediction sensor configured to image the target area, and the targeting system may include a targeting sensor configured to image a portion of the target area. The imaging may include collecting a representation (e.g., an image) of the target area or a portion of the target area. In some embodiments, the prediction system may include multiple prediction sensors, allowing for coverage of a larger target area. In some embodiments, the targeting system may include multiple targeting sensors.

[0045] The region of interest may correspond to an overlap region between the field of view of the targeting sensor and the field of view of the predictive sensor. Such overlap may be simultaneous or separated in time. For example, the field of view of the predictive sensor encompasses the region of interest at a first time, and the field of view of the targeting sensor encompasses the region of interest at a second time, but not at the first time. Optionally, the detection system may move relative to the region of interest between the first and second times to facilitate a temporal separation of the overlap between the field of view of the predictive sensor and the field of view of the targeting sensor.

[0046] In some embodiments, the predictive sensor may have a wider field of view than the targeting sensor. The predictive system may further include an object identification module for identifying the object of interest in a predicted image or representation collected by the predictive sensor. The object identification module may distinguish the object of interest from other objects in the predicted image.

[0047] The prediction module may determine a predicted position of the target object and may transmit the predicted position to the targeting system. The predicted position of the object may be determined using the object tracking methods described herein.

[0048] The targeting system can direct a targeting sensor to a desired portion of the region of interest predicted to contain the object based on the predicted position received from the prediction system. In some embodiments, the targeting module can direct an instrument to the object. In some embodiments, the instrument may perform an action on or manipulate the object. In some embodiments, the targeting module can dynamically target the object while the system is moving, using the object's trajectory to adjust the position of the targeting sensor, the instrument, or both to maintain the target.

[0049] An example of a detection system 300 is provided in FIG. 3. The detection system may be part of or coupled to a vehicle 100, such as an autonomous weeding vehicle or a tractor-pulled laser weeding system trailer, moving along a surface, such as a crop field 200. The detection system 300 comprises a prediction module 310 including a prediction sensor having a predicted field of view 315, and a targeting module 320 including a targeting sensor having a targeting field of view 325. The targeting module may further include an instrument, such as a laser, having a target area that overlaps with the targeting field of view 325. In some embodiments, the prediction module 310 is positioned ahead of the targeting module 320 along the direction of travel of the vehicle 100 such that the targeting field of view 325 overlaps with the predicted field of view 315 by a time delay. For example, the predicted field of view 315 at a first time may overlap with the targeting field of view 325 at a second time. In some embodiments, the predicted field of view 315 at the first time may not overlap with the targeting field of view 325 at the first time.

[0050] The detection system of the present disclosure may be used to target objects on surfaces such as ground, dirt surfaces, floors, walls, fields (e.g., farms), lawns, roads, earthworks, piles, or depressions. In some embodiments, the surface may be an uneven surface such as uneven ground, uneven terrain, or a rough floor. For example, the surface may be uneven ground in a construction site, farm field, or mining tunnel, or the surface may be uneven terrain including a field, road, forest, hill, mountain, house, or building. The detection system described herein may locate objects on uneven surfaces more accurately, faster, or in a larger area than a single sensor system or a system lacking an object matching module.

[0051] Alternatively, or in addition, the detection system may be used to target objects that may be spaced apart from the surface on which the object is placed, such as the top of a tree away from the ground point of the tree, and / or may target objects that are locatable relative to the surface, for example, in the air or relative to the atmospheric surface. Further, the detection system may be used to target objects that may move relative to the surface, such as vehicles, animals, humans, or flying objects.

[0052] FIG. 4 illustrates a detection system including a prediction system 400 and a targeting system 450 for tracking an object O in targeting a moving body, such as the vehicle 100 shown in FIGS. 1-3. The prediction system 400, the targeting system 450, or both, may be positioned on or coupled to the moving body (e.g., a moving vehicle). The prediction system 400 may include a prediction sensor 410 configured to image an area, such as a surface area, that includes one or more objects having the object O. Optionally, the prediction system 400 may include a speed tracking module 415. The speed tracking module may estimate the speed of the moving body relative to the area (e.g., the surface). In some embodiments, the speed tracking module 415 may include a device for measuring the displacement of the moving body over time, such as a rotary encoder. Alternatively, or in addition, the speed tracking module may use images collected by the prediction sensor 400 and estimate the speed using optical flow.

[0053] The object identification module 420 may identify objects in images collected by the predictive sensor. For example, the object identification module 420 may identify weeds in an image and distinguish the weeds from other plants, such as crops, in the image. The object location module 425 may determine the location of objects identified by the object identification module 420 and compile a set of identified objects and their corresponding locations. Object identification and object location may be performed on a series of images collected over time by the predictive sensor 410. The set of identified objects and corresponding locations from two or more images from the object location module 425 may be sent to the de-duplication module 430.

[0054] The de-duplication module 430 may use object locations in a first image collected at a first time and object locations in a second image collected at a second time to identify objects, such as object O, that appear in both the first and second images. The set of identified objects and corresponding locations may be de-duplicated by the de-duplication module 430 by assigning object locations that appear in both the first and second images to the same object O. In some embodiments, the de-duplication module 430 may use velocity estimates from the velocity tracking module 415 to identify corresponding objects that appear in both images. The resulting de-duplicated set of identified objects may include unique objects, each having one or more corresponding locations determined at one or more time points. The differencing module 435 may receive the de-duplicated object set from the de-duplication module 430 and may differencing the de-duplicated set by removing objects. In some embodiments, an object may be removed if it is no longer tracked. For example, an object may be removed if it has not been identified in a predetermined number of images in the sequence of images. In another example, an object may be removed if it has not been identified within a predetermined period of time. In some embodiments, an object may continue to be tracked even if it no longer appears in the images collected by the predictive sensor 410. For example, an object may continue to be tracked if it is predicted to be present within the predicted field of view based on the predicted location of the object. In another example, an object may continue to be tracked if it is predicted to be present within range of the targeting system based on the predicted location of the object. The difference detection module 435 may provide the difference detected object set to the position prediction module 440.

[0055] The position prediction module 440 may determine a predicted position of the object O at a future time from the differentially detected object set. In some embodiments, the predicted position may be determined from two or more corresponding positions determined from images collected at two or more time points, or from a single position combined with speed information from the speed tracking module 415. The predicted position of the object O may be based on a vector velocity, including speed and direction, of the object O relative to the moving body between a position of the object O in a first image collected at a first time and a position of the object O in a second image collected at a second time. Optionally, the vector velocity may take into account the distance of the object O from the moving body along the imaging axis (e.g., the height or altitude of the object relative to the surface). Alternatively, or in addition, the predicted position of the object may be based on the position of the object O in the first image or the second image and the vector velocity of the vehicle determined by the speed tracking module 415.

[0056] The targeting system 450 may receive a predicted position of the object O at a future time from the prediction system 400 and may use the predicted position to accurately target the object with the instrument 475 at a future time. The targeting control module 460 of the targeting system 450 may receive the predicted position of the object O from the position prediction module 440 of the prediction system 435 and may direct the targeting sensor 465, the instrument 475, or both to orient to the predicted position of the object. Optionally, the targeting sensor 465 may collect images of the object O, and the position refinement module 470 may refine the predicted position of the object O based on the position of the object O determined from the images. In some embodiments, the position refinement module 470 may take into account optical distortion aberrations in the images collected by the prediction sensor 410 or the targeting sensor 465, or distortion aberrations in the angular motion of the instrument 475 or the targeting sensor 465 due to nonlinearities in the angular motion relative to the object O. The targeting control module 460 may command the instruments 475, and optionally the targeting sensor 465, to head toward the refined position of the object O. In some embodiments, the targeting control module 460 may adjust the position of the targeting sensor 465 or the instruments 475 to follow the object, taking into account the motion of the vehicle during targeting. An instrument 475, such as a laser, may then manipulate the object O. For example, the laser may direct infrared light toward the predicted or refined position of the object O. The object O may be a weed, and directing infrared light toward the weed's location may remove the weed.

[0057] In some embodiments, the prediction system 400 may further include a scheduling module 445. The scheduling module 445 may select objects identified by the prediction module and schedule which objects to target by the targeting system. The scheduling module 445 may schedule objects to target based on parameters such as object position, relative speed, instrument activation time, confidence score, or a combination thereof. For example, the scheduling module 445 may prioritize targeting objects predicted to move outside the field of view of the prediction or targeting sensor or outside the range of the instrument. Alternatively or in addition, the scheduling module 445 may prioritize identified or located targeting objects with high confidence. Alternatively or in addition, the scheduling module 445 may prioritize targeting objects with short activation times. In some embodiments, the scheduling module 445 may prioritize targeting objects based on a user's preferred parameters.

[0058] Prediction Module A prediction module of the present disclosure may be configured to track an object relative to a moving body using the tracking methods described herein. In some embodiments, the prediction module is configured to capture an image or representation of an area of ​​a surface using a predictive camera or sensor, identify an object of interest in the image, and determine a predicted position of the object.

[0059] The prediction module may include an object identification module configured to identify an object of interest and distinguish the object of interest from other objects in the predicted image. In some embodiments, the prediction module uses a machine learning model to identify and distinguish objects based on features extracted from a training dataset including labeled images of the objects. For example, a machine learning model of or associated with the object identification module may be trained to identify weeds and distinguish weeds from other plants, such as crops. In another example, a machine learning model of or associated with the object identification module may be trained to identify debris and distinguish debris from other objects. The object identification module may be configured to identify plants and distinguish between different plants, such as between crops and weeds. In some embodiments, the machine learning model may be a deep learning model, such as a deep learning neural network.

[0060] In some embodiments, the object identification module includes using a discriminative machine learning model, such as a convolutional neural network. The discriminative machine learning model can be trained with many images, such as high resolution images, of surfaces with and without target objects. For example, the machine learning model can be trained with images of fields with and without weeds. Once trained, the machine learning model can be configured to identify regions in the image that contain the target object. The regions may be defined by polygons, such as rectangles. In some embodiments, the regions are bounding boxes. In some embodiments, the regions are polygon masks that cover the identified regions. In some embodiments, the discriminative machine learning model can be trained to determine the location of the target object, such as a pixel location in a predicted image.

[0061] The prediction module may further include a speed tracking module for determining the speed of the vehicle to which the prediction module is coupled. In some embodiments, the positioning system and the detection system may be positioned on the vehicle. Alternatively or additionally, the positioning system may be positioned on the vehicle that is spatially coupled to the detection system. For example, the positioning system may be installed on the vehicle towing the detection system. The speed tracking module may include a positioning system, such as a wheel or rotary encoder, an inertial measurement unit (IMU), a global positioning system (GPS), a ranging sensor (e.g., laser, SONAR, or RADAR), or an internal navigation system (INS). For example, a wheel encoder may communicate with the wheels of the vehicle to estimate speed or distance traveled based on angular frequency, rotation frequency, rotation angle, or wheel revolutions. In some embodiments, the speed tracking module may utilize images from the prediction sensor to determine the speed of the vehicle using optical flow.

[0062] The prediction module may comprise a system controller, such as a system computer having storage, random access memory (RAM), a central processing unit (CPU), and a graphics processing unit (GPU). The system computer may comprise a tensor processing unit (TPU). The system computer must have sufficient RAM, storage space, CPU power, and GPU power to perform operations to detect and identify targets. The prediction sensor must provide images of sufficient resolution to perform operations to detect and identify objects. In some embodiments, the prediction sensor may be a camera, such as a charge-coupled device (CCD) camera or a complementary metal oxide semiconductor (CMOS) camera, a LIDAR detector, an infrared sensor, an ultraviolet sensor, an x-ray detector, or any other sensor capable of generating images.

[0063] Targeting Module The targeting module of the present disclosure may be configured to target an object tracked by the prediction module. In some embodiments, the targeting module may direct an instrument to an object to manipulate the object. For example, the targeting module may be configured to direct a laser beam toward a weed to incinerate the weed. In another example, the targeting module may be configured to instruct a gripping tool to grip the object. In another example, the targeting module may direct a spraying tool to spray a fluid onto the object. In some embodiments, the object may be a weed, a plant, an insect, a pest, a field, a piece of debris, an obstacle, a surface area, or any other object that can be manipulated. The targeting module may be configured to receive a predicted location of the target object from the prediction module and direct a targeting camera or a targeting sensor to the predicted location. In some embodiments, the targeting module may direct an instrument, such as a laser, to the predicted location. The location of the targeting sensor and the location of the instrument may be combined. In some embodiments, multiple targeting modules communicate with the prediction module.

[0064] The targeting module may include a targeting control module. In some embodiments, the targeting control module may control the targeting sensor, the instrument, or both. In some embodiments, the targeting control module may include an optical control system, which includes optical components configured to control an optical path (e.g., a laser beam path or a camera imaging path). The targeting control module may include software-driven electrical components that may control activation and deactivation of the instrument. Activation or deactivation may depend on the presence or absence of an object detected by the targeting camera. Activation or deactivation may depend on the position of the instrument relative to the position of the target object. In some embodiments, the targeting control module may activate an instrument, such as a laser emitter, once an object is identified and located by the predictive system. In some embodiments, the targeting control module may activate the instrument when the range or target area of ​​the instrument is positioned to overlap the target object position.

[0065] The targeting control module may deactivate the instrument when the object is grasped, sprayed, burned, or illuminated, i.e., when an area containing the object is targeted by the instrument, when the object is no longer identified by the target prediction module, when a specified period of time has elapsed, or any combination thereof. For example, the targeting control module may deactivate the emitter when an area on the surface containing weeds is scanned by the beam, when the weeds are illuminated or burned, or when the beam has been activated for a predetermined period of time.

[0066] The prediction module and the targeting module described herein may be used in combination to locate, identify, and target objects using an instrument. The targeting control module may comprise an optical control system as described herein. The prediction module and the targeting module may be in communication, such as, for example, electrical or digital communication. In some embodiments, the prediction module and the targeting module are directly or indirectly coupled. For example, the prediction module and the targeting module may be coupled to a support structure. In some embodiments, the prediction module and the targeting module are configured on or coupled to a vehicle, such as the vehicle shown in Figures 1 and 2. For example, the prediction module and the targeting module may be positioned on an autonomous vehicle. In another example, the prediction module and the targeting module may be positioned on a trailer towed by another vehicle, such as a tractor.

[0067] The targeting module may include a system controller, such as a system computer having storage, random access memory (RAM), a central processing unit (CPU), and a graphics processing unit (GPU). The system computer may include a tensor processing unit (TPU). The system computer must have sufficient RAM, storage space, CPU power, and GPU power to perform operations to detect and identify targets. The targeting sensor must provide images of sufficient resolution to perform operations to match objects with objects identified in the predicted image.

[0068] Object Tracking Methods Systems of the present disclosure, such as the detection systems, optical control systems, autonomous weeding systems, and computer systems described herein, implement object tracking methods for tracking one or more objects relative to a moving body (e.g., vehicle 100 shown in FIGS. 1-3). The object tracking methods may be used to identify and locate an object in images collected by a sensor (e.g., a predictive sensor), match the same object identified across two or more images, determine the trajectory of an object relative to a moving body (e.g., a moving vehicle), predict the future location of an object, or combinations thereof. In some embodiments, the object may be a weed. An exemplary application of the object tracking methods is described with respect to tracking weeds on a surface. However, these methods may be applied to any other object or group of objects, such as plants, debris, insects, pests, or other objects.

[0069] A method 500 for predicting the location of weeds relative to a moving body, such as the vehicle 100 shown in FIGS. 1-3, at a future time point is shown in FIG. 5. In step 510, a first image may be collected at a first time point, i.e., time (t). The first image may be collected by a sensor, such as a predictive sensor, that may be positioned on or coupled to the moving body (e.g., a moving vehicle). Weeds may be identified in the first image and their location at time (t) may be determined in step 520. A second image taken at time (t+dt) may be collected in step 530. In some embodiments, dt may be positive such that the second image is collected after the first image. In some embodiments, dt may be negative such that the second image is collected before the first image. The second image may be collected using the same sensor as the first image. In step 540, weeds may be identified in the second image and their location at time (t+dt) may be determined. Deduplication may be performed in step 550 to identify any weeds that appear in both the first and second images and associate the positions of such weeds at time (t) and time (t+dt) with the corresponding weeds. Using the positions of the weeds identified in both the first and second images at time (t) and time (t+dt) determined in deduplication step 550, a future position of the weed at a later time (t+dt+dt2) may be determined in step 560. Alternatively or additionally, a single position of the weed at time (t) or time (t+dt) may be used in combination with the velocity vector of the moving body to determine the future position of the weed. This future position may be used to target the weed at a later time and remove it, for example, using a laser. In some embodiments, the predicted position of the weed may include a trajectory of the weed relative to the moving body. This trajectory may enable dynamic tracking of the weed over time as it moves relative to the moving body.

[0070] Deduplication and difference detection As described herein, de-duplication may include pairing or matching the positions of identified objects (e.g., weeds) across two or more images such that the positions of the objects in each of the two or more images are associated with the same object. Differential detection may include removing tracked objects that are no longer present in the image. For example, differential detection may include removing tracked objects that are out of the field of view of the sensor. In some embodiments, a tracked object may be removed if it is not present across multiple image frames or if the predicted location of the object at the time of image collection is outside the field of view. In some embodiments, a tracked object may not be removed if it is not identified in the image but is predicted to be in the field of view. For example, an object may not be identified in an image if it is not visible at the time of image collection or if the software used to identify the object is unable to detect or recognize the object in the image.

[0071] The deduplication 550 of FIG. 5 is described in more detail with respect to FIG. 6, which describes a method 600 for determining a trajectory of a weed relative to a moving object, such as the vehicle 100 shown in FIGS. 1-3. A position of the weed at time (t), such as that determined in step 520 of FIG. 5, is provided in step 610. In step 620, a shift may be applied to the weed position at time (t) to determine, in step 630, a shifted weed position at time (t+dt). In some embodiments, the shift may be based on a measured or estimated displacement of the moving object (e.g., a moving vehicle) relative to the weed over a period (dt) from time (t) to time (t+dt), as described in more detail in FIG. 7. For example, the displacement of the moving object may be estimated using optical flow analysis of images collected by a sensor coupled to the moving object, or the displacement of the moving object may be measured using a rotary encoder or a GPS tracking device. Alternatively or additionally, the shift may be based on a displacement between a first weed identified in the first image and a second weed identified in the second image, as described in more detail in FIGS. 8A-8C. In some embodiments, the shift may be a sum of multiple shifts. For example, a first shift may be applied based on a measured or estimated displacement of the moving body relative to the weed over a period (dt) from time (t) to time (t+dt), generating an intermediate shift position. In some embodiments, dt may be positive such that time (t+dt) is after time (t). In some embodiments, dt may be negative such that time (t+dt) is before time (t). A second shift may be applied based on a displacement between the intermediate shift position and a weed position identified in the second image. A weed position at time (t+dt), such as that determined in step 540 of FIG. 5, is provided in step 640. The shifted weed position at time (t+dt), determined by applying the shift to the weed position at time (t), is compared in step 650 with the actual weed position at time (t+dt).

[0072] In some embodiments, the comparison performed in step 650 may include comparing the shifted position with each of the closest actual weed positions at time (t+dt) to determine whether the weed corresponding to the shifted position identified in the first image is the same as the weed corresponding to the actual weed position at time (t+dt) identified in the second image. Alternatively or additionally, the comparison may include identifying the position of any actual weed at time (t+dt) within a predetermined search radius of each shifted position to determine whether the weed corresponding to the shifted position identified in the first image is the same as the weed corresponding to the actual weed position identified in the second image and located within the search radius. Based on this comparison, in step 660, the weed positions in the first image are paired with weed positions in the second image, each corresponding to the position of the same weed at time (t) and (t+dt). This pairing process is used to de-duplicate the set of weeds identified in the first image and / or the second image. In step 670, for weeds identified in both the first and second images, a trajectory for the moving object may be determined, and thus may have a position corresponding to time (t) and time (t+dt). In some embodiments, the trajectory may include one, two, or three dimensional displacement over a period of time (dt) from time (t) to time (t+dt). This trajectory may be used to predict the future position of the weed at a later time (t+dt+dt2), as described in step 560 of FIG. 5.

[0073] Matching or pairing of objects between images for deduplication may be performed in a variety of ways. In some embodiments, objects may be identified across multiple images based on proximity to a location where the object is expected to reside. An example of such a method is described in more detail with respect to FIG. 7. In some embodiments, objects may be identified across multiple images by applying multiple displacements or shifts to a set of objects, identifying the displacement or shift that results in the lowest error (e.g., deviation from an observed object location), and matching the object to the shifted location based on proximity. An example of such a method is described in more detail with respect to FIGS. 8A-8C.

[0074] FIG. 7 illustrates an example of a method for de-duplicating images as described in method 600 of FIG. 6. Schematically illustrated in panel 1 of FIG. 7 is a first image of a surface collected at time (t) including weeds at locations x1, x2, x3, and x4 (shown as shaded stars, pentagons, triangles, and hexagons, respectively). The direction of movement of a moving object (e.g., a vehicle such as an autonomous car or a tow trailer) from which the image was collected relative to the surface is indicated by a left-pointing arrow. In panel 2, shifts s1, s2, s3, and s4 are applied to each of the weeds at locations x1, x2, x3, and x4, respectively, resulting in shift positions p1, p2, p3, and p4, shown as open stars, pentagons, triangles, and hexagons, respectively. The shifts s1, s2, s3, and s4 are determined based on the measured or estimated speed of the moving object relative to the surface over a period (dt) from time (t) to time (t+dt). In this example, each of the shifts s1, s2, s3, and s4 has the same magnitude and direction. In some embodiments, each of the shifts s1, s2, s3, and s4 can be the sum of two or more shifts. For example, a first shift may be determined based on a measured or estimated speed of the moving body, and a second shift may be determined based on a distance at time (t+dt) between an intermediate shift position and an actual weed position y1, y2, y3, or y4. The shift positions p1, p2, p3, and p4 represent estimates of the positions where the weeds at positions x1, x2, x3, and x4, respectively, are located at time (t+dt).

[0075] Panel 3 shows a schematic representation of a second image collected at time (t+dt), which includes weeds at positions y1, y2, y3, and y4, shown as shaded stars, pentagons, triangles, and circles, respectively, overlapping with the shift positions p1, p2, p3, and p4 determined in panel 2. Panel 3 also shows search regions r1, r2, r3, and r4 around the shift positions p1, p2, p3, and p4. Each search region encompasses the corresponding shift position. The positions of weeds at time (t+dt) that are located within the search regions are attributed to the same weed as the corresponding shift position. For example, the weed at position y1 (shaded star) is located within search region r1, which corresponds to shift position p1 (open star), so the weed at position y1 is identified as the same weed as the weed at position x1 present in the first image shown in panel 1. Similarly, the weed at position y2 is determined to be the same as the weed at position x2, and the weed at position y3 is determined to be the same as the weed at position x3. The weed at position y4 does not correspond to any of the shift positions, so the weed at position y4 is identified as a new weed that does not correspond to a weed identified in the first image. The shift position p4 does not correspond to a weed identified in the second image collected at time (t+dt), so the weed at position x4 is determined not to be identified in the second image. Alternatively or additionally, a search area may be applied around the weeds at positions y1, y2, y3, and y4, and the shift positions that fall within the search area may be attributed to the same weed as the corresponding weed position. In some embodiments, a weed identified in the first image may not be identified in the second image if the mobile object has moved such that the weed is no longer in the field of view of the sensor when the second image is collected. Alternatively, a weed identified in the first image may not be identified in the second image if the weed is obscured in the second image by another plant, etc. Image differencing may include removing weeds that are no longer being tracked. For example, an object may be removed if it has not been identified within a predetermined number of images in a sequence of images. In another example, an object may be removed if it has not been identified within a predetermined period of time. In some embodiments, an object may continue to be tracked for targeting even if it no longer appears in images collected by a predictive sensor.

[0076] Panel 4 shows the de-duplicated and difference-detected image including weeds at positions y1, y2, and y3 (corresponding to weeds at positions x1, x2, and x3, respectively) and a weed at position y4 (corresponding to a newly identified weed). Panel 5 shows vector trajectories v1, v2, and v3 between weed positions x1, x2, and x3 at time (t) and weed positions y1, y2, and y3 at time (t+dt). No vector trajectory is determined for the weed at position y4 since it does not have a corresponding position at time (t). Unlike the shifts s1, s2, and s3, the vector trajectories v1, v2, and v3 may not be the same due to various factors including differences in weed height (i.e., perpendicular to the image plane), rotation of the moving object relative to the surface from time (t) to time (t+dt), optical distortion aberrations, or changes in the shape of the weed from time (t) to time (t+dt). Furthermore, the vector trajectories v1, v2, and v3 may differ from the shifts s1, s2, and s3, respectively, due to imprecision in estimating or measuring the speed of the moving object, the slippage of the rotary encoder wheel, the height of the corresponding weed relative to the surface, the rotation of the moving object relative to the surface from time (t) to time (t+dt), optical distortion aberrations, the motion of the weed relative to the surface, or the change in the weed shape from time (t) to time (t+dt). The vector trajectories v1, v2, and v3 may be used to predict the position of the corresponding weed at a later time.

[0077] 8A-8C show a second example of a method for de-duplicating images described in method 600 of FIG. 6. Application of different test shifts to weeds identified in an image to determine the shift with the smallest offset is shown in FIGS. 8A, 8B, and 8C, respectively. FIG. 8A shows application of a first test shift. Schematically shown in panel 1 is a first image collected at time (t) containing weeds at locations x1, x2, x3, and x4, shown as shaded stars, pentagons, triangles, and hexagons, respectively, and a second image collected at time (t+dt) containing weeds at locations y1, y2, y3, and y4, shown as open stars, pentagons, triangles, and circles, respectively. In some embodiments, the weed locations x1, x2, x3, and x4 at time (t) may be predicted weed locations determined using the methods described herein, rather than weed locations determined from the images. In such cases, (dt) may be zero such that (t) is equal to (t+dt), or (dt) may be greater than or less than zero.

[0078] In panel 2, shifts s1, s2, s3, and s4 are applied to x1, x2, x3, and x4, and the corresponding shift positions p1, p2, p3, and p 4がThe resulting shifts are shown as dashed star, pentagon, triangle, and hexagon, respectively. The shift is based on the vector displacement (s1, shown as solid arrow) between position x1 (shaded star) and position y2 (open pentagon). The same shifts (s2, s3, and s4, shown as dashed arrows) are applied to the weeds at positions x2, x3, and x4. In panel 3, the shifted positions p1, p2, p3, and p4 are compared with the weed positions y1, y2, y3, and y4 at time (t+dt). The offsets are determined based on the distances d1, d2, d3, and d4 of the shifted positions p1, p2, p3, and p4, respectively, from the nearest weed at time (t+dt). For example, the shifted position p2 (dashed pentagon) is closest to the weed at position y2 (open pentagon), and the resulting distance is d2. Similarly, shift position p3 is closest to the weed at position y2, and the resulting distance is d3, and shift position p4 is closest to the weed at position y3, and the resulting distance is d4. Because the shift was based on the distance between position x1 and position y2, shift position p1 is aligned with position y2, and the resulting distance d1 is zero. The offset is determined based on the distances d1, d2, d3, and d4. For example, the offset may be the sum of d1, d2, d3, and d4, the average of d1, d2, d3, and d4, or a combination of d1, d2, d3, and d4.

[0079] FIG. 8B illustrates the application of a second test shift. Schematically shown in panel 1 is a first image collected at time (t) containing weeds at positions x1, x2, x3, and x4, shown as shaded stars, pentagons, triangles, and hexagons, respectively, and a second image collected at time (t+dt) containing weeds at positions y1, y2, y3, and y4, shown as open stars, pentagons, triangles, and circles, respectively. In some embodiments, the weed positions x1, x2, x3, and x4 at time (t) may be predicted weed positions determined using the methods described herein, rather than weed positions determined from the images. In such cases, (dt) may be zero such that (t) is equal to (t+dt), or (dt) may be greater than or less than zero.

[0080] In panel 2, shifts s1, s2, s3, and s4 are applied to x1, x2, x3, and x4, resulting in corresponding shifted positions p1, p2, p3, and p4, shown as dashed star, pentagon, triangle, and hexagon, respectively. The shifts are based on the vector displacement (s1, shown as solid arrow) between position x1 (shaded star) and position y3 (open triangle). The same shifts (s2, s3, and s4, shown as dashed arrow) are applied to the weeds at positions x2, x3, and x4. In panel 3, the shifted positions p1, p2, p3, and p4 are compared with the weed positions y1, y2, y3, and y4 at time (t+dt). The offsets are determined based on the distances d1, d2, d3, and d4 of the shifted positions p1, p2, p3, and p4, respectively, from the nearest weed at time (t+dt). For example, shift position p2 (dashed pentagon) is closest to the weed at position y2 (open pentagon) and the resulting distance is d2. Similarly, shift position p3 is closest to the weed at position y3 and the resulting distance is d3, and shift position p4 is closest to the weed at position y3 and the resulting distance is d4. Because the shift was based on the distance between position x1 and position y3, shift position p1 is aligned with position y3 and the resulting distance d1 is zero. The offset is determined based on the distances d1, d2, d3, and d4. For example, the offset may be the sum of d1, d2, d3, and d4, the average of d1, d2, d3, and d4, or a combination of d1, d2, d3, and d4.

[0081] FIG. 8C illustrates the application of a third test shift. Schematically shown in panel 1 is a first image collected at time (t) containing weeds at positions x1, x2, x3, and x4, shown as shaded stars, pentagons, triangles, and hexagons, respectively, and a second image collected at time (t+dt) containing weeds at positions y1, y2, y3, and y4, shown as open stars, pentagons, triangles, and circles, respectively. In some embodiments, the weed positions x1, x2, x3, and x4 at time (t) may be predicted weed positions determined using the methods described herein, rather than weed positions determined from the images. In such cases, (dt) may be zero such that (t) is equal to (t+dt), or (dt) may be greater than or less than zero.

[0082] In panel 2, shifts s1, s2, s3, and s4 are applied to x1, x2, x3, and x4, resulting in corresponding shifted positions p1, p2, p3, and p4, shown as dashed star, pentagon, triangle, and hexagon, respectively. The shifts are based on the vector displacement (s1, shown as solid arrows) between position x1 (shaded star) and position y1 (open star). The same shifts (s2, s3, and s4, shown as dashed arrows) are applied to the weeds at positions x2, x3, and x4. In panel 3, the shifted positions p1, p2, p3, and p4 are compared with the weed positions y1, y2, y3, and y4 at time (t+dt). The offsets are determined based on the distances d1, d2, d3, and d4 of the shifted positions p1, p2, p3, and p4, respectively, from the nearest weed at time (t+dt). For example, shift position p2 (dashed pentagon) is closest to the weed at position y2 (open pentagon), and the resulting distance is d2. Similarly, shift position p3 is closest to the weed at position y3, and the resulting distance is d3, and shift position p4 is closest to the weed at position y3, and the resulting distance is d4. Since the shift was based on the distance between position x1 and position y3, shift position p1 is aligned with position y3, and the resulting distance d1 is zero. Furthermore, shift positions p2 and p3 are closely aligned with positions y2 and y3, respectively, and therefore the distances d2 and d3 are small. The offset is determined based on the distances d1, d2, d3, and d4. For example, the offset may be the sum of d1, d2, d3, and d4, the average of d1, d2, d3, and d4, or a combination of d1, d2, d3, and d4.

[0083] The offsets resulting from the test shifts applied in Figures 8A, 8B, and 8C are compared to determine the shift that results in the smallest offset. In some embodiments, the offsets resulting from the test shifts corresponding to the displacement between the pairing of the weeds identified in the first image and the weeds identified in the second image are compared. In some embodiments, the shifts corresponding to all possible weed pairings are compared. In some embodiments, the pairing is selected based on the proximity of the expected position of the weeds identified in the first image to the actual position of the weeds identified in the second image. In this example, the test shift applied in Figure 8C produces the smallest offset. The refinement shift, in this example, corresponds to the shift shown in Figure 8C, is applied to the weed positions in the first image as described with respect to panels 2 and 3 of Figure 8C, and the resulting shift positions are compared to the weed positions in the second image.

[0084] The images are de-duplicated by identifying weed positions from the first image and weed positions from the second image that correspond to the same weed based on the distance between the weed position in the second image and the shift position. In this example, the weeds at positions x1, x2, and x3 are identified as the same weed at positions y1, y2, and y3, respectively. The weed at position y4 (open circle) is identified as a new weed that is present in the second image but not in the first image, and it is determined that the weed at position x4 is not identified in the second image. In some embodiments, a weed identified in the first image may not be identified in the second image if the mobile object has moved such that the weed is no longer in the field of view of the sensor when the second image is collected. Alternatively, a weed identified in the first image may not be identified in the second image if the weed is obscured in the second image by another plant, etc. Image differencing may include removing weeds that are no longer tracked. For example, an object may be removed if it has not been identified in a predetermined number of images in the sequence of images. In another example, an object may be removed if it has not been identified within a predetermined period of time. In some embodiments, an object may continue to be tracked even if it no longer appears in the image collected by the predictive sensor. For example, an object may continue to be tracked if it is predicted to be in the predicted field of view based on the predicted position of the object. In another example, an object may continue to be tracked if it is predicted to be within the range of the targeting system based on the predicted position of the object. Determine vector trajectories between the weed positions x1, x2, and x3 at time (t) and the weed positions y1, y2, and y3 at time (t+dt). Do not determine vector trajectories because the weed at position y4 does not have a corresponding position at time (t). The vector trajectories may be used to predict the corresponding weed positions at a later time.

[0085] In some embodiments, deduplication may include comparing parameters of tracked objects identified across two or more images to determine whether a first object in a first image is the same as a second object in a second image. Parameters such as size, shape, category, orientation, or type may be determined for objects identified in an image. For example, a plant may have parameters corresponding to the size of the plant, the shape of the leaf, the category of the plant (e.g., weed or crop), or the type of plant (e.g., onion, strawberry, corn, soybean, barley, oats, wheat, alfalfa, cotton, hay, tobacco, rice, sorghum, tomato, potato, grape, rice, lettuce, bean, pea, sugar beet, grass, broadleaf, foliage, or purslane). A first object identified in a first image may be the same as a second object identified in a second image if the parameters of the first object are the same or similar to the parameters of the second object. For example, a first object may be the same as a second object if the size of the first object is similar to the size of the second object. In another example, a first object may be the same as a second object if the type of the first object is the same as the type of the second object (e.g., if the first object and the second object are both weeds). In another example, a first object may be the same as a second object if the shape of the first object (e.g., the shape of the leaves) is similar to the shape of the second object. In some embodiments, a first object identified in a first image may be different from a second object identified in a second image if a parameter of the first object is different from a parameter of the second object. For example, a first object may be different from a second object if a size of the first object is different from a size of the second object. In another example, a first object may be different from a second object if a type of the first object is different from a type of the second object (e.g., if the first object is a weed and the second object is a crop).In another example, a first object may differ from a second object if the shape of the first object (e.g., the shape of the leaves) differs from the shape of the second object. In another example, a first object may differ from a second object if the orientation of the first object (e.g., the orientation of the leaves or the orientation of the stem) differs from the orientation of the second object.

[0086] The object parameters may be determined using a trained machine learning model. The machine learning model may be trained with a sample training dataset of images, such as high-resolution images of surfaces with and without plants, pests, or other objects. The training images may be labeled with one or more object parameters, which may be the location of the object, the size of the object (e.g., radius, diameter, surface area, or a combination thereof), the category of the object (e.g., weed, crop, equipment, pest, or surface irregularity), the type of plant (e.g., grass, broadleaf, purslane, onion, strawberry, corn, soybean, barley, oats, wheat, alfalfa, cotton, hay, tobacco, rice, sorghum, tomato, potato, grapes, rice, lettuce, beans, peas, sugar beet, etc.), or a combination thereof.

[0087] The deduplication may further include determining a confidence score for the object location or parameters and using the confidence score to evaluate whether a first object in the first image is the same as a second object in the second image. The confidence score may quantify the confidence of an object identification, classification, localization, or combination thereof that has been made. For example, the confidence score may quantify the confidence of classifying a plant as a weed. In another example, the confidence score may quantify the confidence that a plant is a particular plant type (e.g., grass, broadleaf, purslane, onion, strawberry, corn, soybean, barley, oats, wheat, alfalfa, cotton, hay, tobacco, rice, sorghum, tomato, potato, grape, rice, lettuce, bean, pea, sugar beet, etc.). In some embodiments, the confidence score may quantify the confidence that an object is not of a particular class or type. For example, the confidence score may quantify the confidence that an object is not a crop. A confidence score may be assigned to each of the identified objects in each collected image for each evaluation parameter (e.g., one or more of the object's location, size, shape, category, orientation, or type). The confidence score may be used to determine whether a first image is the same as a second object in a second image. Two objects in different images identified as having the same or similar parameters with high confidence may be identified as the same object. For example, a first object identified as a weed with high confidence may be the same as a second object identified as a weed with high confidence. Two objects in different images identified as having different parameters with high confidence may be identified as different objects. For example, a first object identified as a weed with high confidence may be different from a second object identified as a crop with high confidence. Two objects in different images identified as having different parameters with low confidence may not be different objects due to ambiguity in the determination of the parameters.For example, a first object identified with low confidence as a weed may not be different from a second object identified with low confidence as a crop.

[0088] In some embodiments, the confidence score may range from 0 to 1, with 0 corresponding to low confidence and 1 corresponding to high confidence. The value threshold at which confidence is considered high may be context dependent and may be adjusted based on the desired outcome. In some embodiments, high confidence values ​​may be considered 0.5 or more, 0.6 or more, 0.7 or more, 0.8 or more, or 0.9 or more. In some embodiments, low confidence values ​​may be considered less than 0.3, less than 0.4, less than 0.5, less than 0.6, less than 0.7, or less than 0.8. An object may be identified as a first object category if the first object category has a high confidence score and a second object category has a low confidence score. For example, an object may be identified as a weed if the weed category has a confidence score of 0.6 and the crop category has a confidence score of 0.1.

[0089] Location Prediction The object tracking methods described herein may include predicting a position of an object relative to a moving body (e.g., a moving vehicle) at a future time. In some embodiments, the predicted position may be utilized by the targeting system of the present disclosure to target an object at the predicted position at a future time. The predicted position may include one dimension, two dimensions, three dimensions, or a combination thereof. The one and two dimensions may be parallel to the plane of an image collected by the predictive sensor, parallel to the mean plane of the surface, or both. In some embodiments, the one dimension may be parallel to the direction of movement of the moving body. The three dimensions may be perpendicular to the plane of an image collected by the predictive sensor, perpendicular to the mean plane of the surface, or both. In some embodiments, the predicted position may include a trajectory. The trajectory may be used to predict a change in the object position over time. In some embodiments, the trajectory may be determined from the position of the object in the first image, the position of the object in the second image, the predicted position of the object, the vector velocity of the moving body, and combinations thereof.

[0090] The predicted position of the object may be determined from two or more of the position of the object in the first image, the position of the object in the second image, and the vector velocity (including magnitude and direction) of the moving body. The time elapsed from the collection of the first image and the collection of the second image, the time from the collection of the first image or the second image and a future time point, or both may be used. For example, the trajectory of the object may be determined from the position of the object in the first image, the position of the object in the second image. The vector velocity of the object relative to the moving body may be determined from the trajectory and the time elapsed from the collection of the first image and the collection of the second image. Using the vector velocity of the object, a predicted position of the object at a future time point may be determined based on the time between the collection of the second image and a future time point.

[0091] In another example, a vector velocity of an object may be determined from the position of the object in the first image and the vector velocity of the moving body. Using the vector velocity of the object, a predicted position of the object at a future time point may be determined based on the time between the collection of the second image and the future time point. In some embodiments, the predicted position may be adjusted based on a change in the vector velocity of the moving body between the position of the object in the first image or the second image and the future time point. In some embodiments, a predicted position determined from the position of the object in the first image and the position of the object in the second image may be more accurate than a predicted position determined from the position of the object in a single image.

[0092] Determining the predicted position may include estimating the height or altitude of the object relative to the moving body. Estimating the height or altitude of the object may improve the accuracy of the predicted position, improve the accuracy of the targeting, or both. The height or altitude of the object may correspond to the position of the object in three dimensions (e.g., perpendicular to the plane of the image collected by the predictive sensor, perpendicular to the average plane of the surface, or both). The height or altitude of the object may be determined based on the position of the object in the first image, the position of the object in the second image, and the displacement of the moving body during the time elapsed from the collection of the first image to the collection of the second image. In some embodiments, the displacement of the moving body may be determined based on a vector velocity of the moving body relative to the surface. In some embodiments, the displacement of the moving body may be determined based on an average displacement of the object located in the first image and the second image.

[0093] An object may be targeted at a predicted location by pointing a targeting sensor, an instrument, or both at the predicted location. The targeting sensor, the instrument, or both may be controlled by an actuator (e.g., a servo). In some embodiments, targeting an object at a predicted location may include applying a calibration factor to the predicted location. The calibration factor may transform the predicted location between a first reference frame (e.g., a surface reference frame, a vehicle reference frame, a predicted sensor image reference frame, a targeting sensor image reference frame, a targeting sensor actuator reference frame, or an instrument actuator reference frame) and a second reference frame (e.g., a surface reference frame, a vehicle reference frame, a predicted sensor image reference frame, a targeting sensor image reference frame, a targeting sensor actuator reference frame, or an instrument actuator reference frame). For example, the calibration factor may transform the predicted location from a surface reference frame to a targeting sensor reference frame. In another example, the calibration factor may transform the predicted location from a predicted sensor reference frame to a targeting sensor reference frame. In another example, the calibration factor may transform the predicted location from a surface reference frame to an instrument actuator reference frame. In another example, the calibration coefficients may transform the predicted positions from a predicted sensor reference frame to an instrument actuator reference frame. In some embodiments, the calibration coefficients may include a calibration spline or a geometric calibration.

[0094] Targeting an object may include tracking the object over time as it moves relative to the moving body. In some embodiments, the targeting sensor, instrument, or both may change position over time such that the target is maintained. For example, a laser instrument may be repositioned such that the laser continues to illuminate the object while the system is moving relative to the object. Repositioning of the targeting sensor, instrument, or both may be performed based on the trajectory of the object, the vector velocity of the moving body, or a combination thereof.

[0095] Determining or estimating speed The object tracking methods described herein may include determining or estimating a velocity of a moving body. The velocity may be relative to a surface or relative to one or more objects. The velocity may be a vector velocity that includes both a magnitude and a direction. The direction may include one dimension. In some embodiments, the direction may include two dimensions, three dimensions, or both. In some embodiments, a dimension may be determined relative to a direction of movement. For example, the direction of movement may be one dimensional.

[0096] Various methods may be used to determine or estimate the velocity of a moving body. In some embodiments, the velocity may be determined by measuring the displacement of the moving body over time, for example using devices such as wheel or rotary encoders, inertial measurement units (IMUs), global positioning systems (GPSs), ranging sensors (e.g., laser, SONAR, or RADAR), or internal navigation systems (INSs) positioned on or coupled to the moving body. Rotary or wheel encoders may convert the angular motion of the moving body's wheels into distance traveled.

[0097] Alternatively or additionally, the velocity of the moving object may be estimated using computer vision (CV). An example of computer vision that may be used to estimate the velocity of the moving object may be optical flow. Optical flow may include determining a shift between a first image and a second image collected by a sensor coupled to the moving object. The shift may be a pixel shift, a shift relative to a surface, a shift relative to one or more objects, or a combination thereof. In some embodiments, optical flow may detect the relative motion of the moving object and a surface, object, or multiple objects between the first image and the second image.

[0098] The determination or estimation of velocity may be performed more frequently than the identification or location of an object. In some embodiments, the velocity estimation may be performed using one or more image frames collected between the image frames used to identify or locate the object. For example, optical flow analysis may be less computationally intensive than object identification and location, so the estimation of velocity by optical flow may be performed more frequently than object identification and location. In some embodiments, two or more velocity measurements or estimates may be used to determine the vector velocity of a moving object. Alternatively or additionally, optical flow may be used to determine the vector velocity of an object relative to a surface. For example, optical flow may be used to determine the trajectory of a slowly moving object relative to a surface, such as an insect or pest. In some embodiments, the velocity of the moving object relative to the surface and the velocity of the object relative to the surface may be combined to determine the velocity of the moving object relative to the object.

[0099] In some embodiments, the first and second images used for optical flow may be displaced by about 90% or less. In some embodiments, the first and second images used for optical flow may be displaced by about 85% or less, about 80% or less, about 75% or less, about 70% or less, about 60% or less, or about 50% or less. In some embodiments, the first and second images used for optical flow may overlap by at least about 10%. In some embodiments, the first and second images used for optical flow may overlap by at least about 10%, at least about 20%, at least about 30%, at least about 40%, or at least about 50%. In some embodiments, the first and second images used for optical flow may overlap by about 10% to about 99%, about 10% to about 90%, about 20% to about 80%, or about 30% to about 70%.

[0100] Distortion correction The object tracking or targeting methods described herein may include correcting for various distortions introduced into the systems implementing those methods, which may include causes of lenses, reflective elements, and other optical elements, as well as differences in object height or elevation, surface irregularities, sensor pose changes, vehicle pose changes, or actuator motion.

[0101] The methods of the present disclosure may be implemented by an optical system that includes one or more optical elements (e.g., lenses, reflective elements, or beam splitters) that may introduce distortion aberrations in an image collected by an image sensor (e.g., a predictive sensor or a targeting sensor). Failure to account for image distortion aberrations may result in inaccuracies when converting to and from an image reference frame. Distortion aberrations may include radial distortion aberrations, chromatic distortion aberrations, or other distortion aberrations. For example, a lens may cause radial distortion aberrations such as barrel distortion aberrations, pincushion distortion aberrations, pincushion distortion aberrations, or combinations thereof. In another example, a reflective element (e.g., a mirror or dichroic element) may cause distortion aberrations due to non-planar surfaces, positioning off a 45° angle, or chromatic aberrations. Image distortion aberrations such as these may be corrected using calibrations such as geometric calibrations or calibration splines. In some embodiments, calibration to correct image distortion aberrations may include mapping points on a surface to pixel locations in an image collected from that surface. The image may be transformed to produce a uniform pixel density per real-world or surface unit.

[0102] The angular motion of the reflective element may introduce distortion aberrations into the targeting system of the present disclosure. In some embodiments, the motion of an actuator that aims at a targeting sensor or instrument (e.g., a laser) relative to the motion of the targeting sensor or instrument may be nonlinear over the range of motion due to the reflective element along the optical path. For example, the laser aiming may be controlled by rotation of a first mirror by a first actuator along a first axis (e.g., a tilt axis) and rotation of a second mirror by a second actuator along a second axis (e.g., a pan axis). The first mirror may be positioned in front of the second mirror along the optical path of the laser such that the motion of the laser along the first axis is nonlinear with respect to the motion of the actuator over the full range of motion. The nonlinearity may be more pronounced at extreme pan angles, resulting in pillow-shaped distortion aberrations. Such targeting distortion aberrations (e.g., pillow-shaped distortion aberrations of a targeting sensor or instrument) may be corrected using calibration coefficients such as geometric calibration or calibration splines. In some embodiments, the calibration coefficients may be determined by mapping the position of the targeting sensor or instrument on the surface to the position of the pan and tilt actuators. Correcting distortion aberrations in the targeting may improve the accuracy of the targeting.

[0103] Height or altitude differences between objects present in a first image may result in apparent distortion aberrations in the object positions in a second image due to parallax effects. Briefly, an object approaching an image sensor along an axis perpendicular to the image plane may appear to move away from the image sensor from a different angle in a first image to a second image collected by the same image sensor than an object moving away from the image sensor. As a result, objects of different heights or objects on uneven surfaces may have less accurate predicted positions. As described herein, the predicted position of an object may be refined based on the object's height or altitude (e.g., along an axis perpendicular to the plane of the image collected by the prediction sensor, an axis perpendicular to the average plane of the surface, or both axes).

[0104] Optical Control System The methods described herein can be implemented by an optical control system, such as a laser optics system, to target an object of interest. For example, the optics system can be used to target an object of interest identified in an image or representation collected by a first sensor, such as a predictive sensor, and locate the same object in an image or representation collected by a second sensor, such as a targeting sensor. In some embodiments, the first sensor is a predictive camera and the second sensor is a targeting camera. Targeting the object may include precisely locating the object using the targeting sensor and targeting the object with an instrument.

[0105] Described herein is an optical control system for directing a beam, e.g., a light beam, to a target location on a surface, such as the location of an object of interest. In some embodiments, the instrument is a laser. However, other instruments are within the scope of the present disclosure, including, but not limited to, a gripping instrument, a spraying instrument, a planting instrument, a harvesting instrument, a pollinating instrument, a marking instrument, a spraying instrument, or a depositing instrument.

[0106] In some embodiments, the emitter is configured to direct a beam along an optical path, e.g., a laser path. In some embodiments, the beam comprises electromagnetic radiation, e.g., light, radio waves, microwaves, or x-rays. In some embodiments, the light is visible light, infrared light, or ultraviolet light. The beam may be coherent. In one embodiment, the emitter is a laser, such as an infrared laser.

[0107] One or more optical elements may be positioned in the path of the beam. The optical elements may include a beam combiner, a lens, a reflective element, or any other optical element that may be configured to direct light, focus light, filter light, or otherwise control light. These elements may be arranged in the following order in the direction of the beam path: a beam combiner, then a first reflective element, then a second reflective element. In another example, one or both of the first reflective element or the second reflective element may be arranged in the direction of the beam path before the beam combiner. In another example, the optical elements may be arranged in the direction of the beam path after the beam combiner, then the first reflective element. In another example, one or both of the first reflective element or the second reflective element may be arranged in the direction of the beam path before the beam combiner. Any number of additional reflective elements may be positioned in the beam path.

[0108] The beam combiner may also be referred to as a beam combining element. In some embodiments, the beam combiner may be a zinc selenide (ZnSe), zinc sulfide (ZnS), or germanium (Ge) beam combiner. For example, the beam combiner may be configured to transmit infrared light and reflect visible light. In some embodiments, the beam combiner may be dichroic. In some embodiments, the beam combiner may be configured to pass electromagnetic radiation having a wavelength longer than a cutoff wavelength and reflect electromagnetic radiation having a wavelength shorter than a cutoff wavelength. In some embodiments, the beam combiner may be configured to pass electromagnetic radiation having a wavelength shorter than a cutoff wavelength and reflect electromagnetic radiation having a wavelength longer than a cutoff wavelength. In other embodiments, the beam combiner may be a polarizing beam splitter, a long pass filter, a short pass filter, or a band pass filter.

[0109] The optical control system of the present disclosure may further include a lens positioned in the optical path. In some embodiments, the lens may be a focusing lens, the focusing lens being positioned to focus the beam, the scattered light, or both. For example, the focusing lens may be positioned in the visible optical path to focus the scattered light onto the targeting camera. In some embodiments, the lens may be a defocusing lens, the defocusing lens being positioned to defocus the beam, the scattered light, or both. In some embodiments, the lens may be a collimating lens, the collimating lens being positioned to collimate the beam, the scattered light, or both. In some embodiments, more than one lens may be positioned in the optical path. For example, two lenses may be positioned in series in the optical path to expand or narrow the beam.

[0110] The position and orientation of one or both of the first and second reflective elements may be controlled by one or more actuators. In some embodiments, the actuators may be motors, solenoids, galvanometers, or servos. For example, the position of the first reflective element may be controlled by a first actuator, and the position and orientation of the second reflective element may be controlled by a second actuator. In some embodiments, a single reflective element may be controlled by multiple actuators. For example, the first reflective element may be controlled by a first actuator along a first axis and a second actuator along a second axis. Optionally, the mirror may be controlled by a first actuator, a second actuator, and a third actuator to provide multi-axis control of the mirror. In some embodiments, a single actuator may control the reflective element along more than one axis. In some embodiments, a single reflective element may be controlled by a single actuator.

[0111] The actuator can change the angle of incidence of the beam hitting the reflective element by changing the position of the reflective element by rotating the reflective element. Changing the angle of incidence can translate the location where the beam hits the surface. In some embodiments, the angle of incidence can be adjusted so that the location where the beam hits the surface is maintained while the optical system is moving relative to the surface. In some embodiments, a first actuator rotates a first reflective element about a first rotation axis to translate the location where the beam hits the surface along a first translation axis, and a second actuator rotates a second reflective element about a second rotation axis to translate the location where the beam hits the surface along a second translation axis. In some embodiments, the first actuator and the second actuator translate the location where the beam hits the surface of the first reflective element along the first and second translation axes by rotating the first reflective element about the first and second rotation axes. For example, a single reflective element can be controlled by a first actuator and a second actuator, and a single reflective element controlled by two actuators can be used to translate the location where the beam hits the surface along the first and second translation axes. In another example, a single reflective element may be controlled by one, two, or three actuators.

[0112] The first and second translational axes may be orthogonal. The coverage area on the surface may be defined by a maximum translational motion along the first and second translational axes. One or both of the first and second actuators may be servo-controlled, piezoelectrically actuated, piezo-inertial actuated, stepper motor controlled, galvanometer-driven, linear actuator controlled, or any combination thereof. One or both of the first and second reflective elements may be mirrors, e.g., dichroic or dielectric mirrors, prisms, beam splitters, or any combination thereof. In some embodiments, one or both of the first and second reflective elements may be any element capable of deflecting a beam.

[0113] The targeting camera may be positioned to capture light, e.g., visible light, traveling along a visible light path in a direction opposite to the beam path, e.g., the laser path. The light may be scattered by a surface, such as a surface having a target object, or an object, such as a target object, and travel along a visible light path toward the targeting camera. In some embodiments, the targeting camera is positioned to capture light reflected from the beam combiner. In other embodiments, the targeting camera is positioned to capture light transmitted through the beam combiner. Upon capturing such light, the targeting camera may be configured to image a target field of view on the surface. The targeting camera may be coupled to the beam combiner, or the targeting camera may be coupled to a support structure that supports the beam combiner. In one embodiment, the targeting camera does not move relative to the beam combiner, such that the targeting camera maintains a fixed position relative to the beam combiner.

[0114] The optical control system of the present disclosure may further include an exit window positioned in the beam path. In some embodiments, the exit window may be the last optical element the beam hits before exiting the optical control system. The exit window may include a material that is substantially transparent to visible light, infrared light, ultraviolet light, or any combination thereof. For example, the exit window may include glass, quartz, fused silica, zinc selenide, zinc sulfide, a transparent polymer, or any combination thereof. In some embodiments, the exit window may include a scratch-resistant coating, such as a diamond coating. The exit window may prevent dust, debris, water, or any combination thereof from reaching other optical elements of the optical control system. In some embodiments, the exit window may be part of a protective casing that surrounds the optical control system.

[0115] After exiting the optical control system, the beam can be directed along a beam path to a surface. In some embodiments, the surface includes a target object, e.g., a weed. Rotational motion of the reflective element can result in a laser sweep along a first translational axis and a laser sweep along a second translational axis. Rotational motion of the reflective element can control the location where the beam strikes the surface. For example, rotational motion of the reflective element can move the location where the beam strikes the surface to the location of the target object on the surface. In some embodiments, the beam is configured to damage the target object. For example, the beam can include electromagnetic radiation, and the beam can irradiate the object. In another example, the beam can include infrared light, and the beam can incinerate the object. In some embodiments, one or both of the reflective elements can be rotated such that the beam scans an area that surrounds and includes the object.

[0116] The predictive camera or predictive sensor may cooperate with an optical control system, such as an optical control system, to identify and locate a target object. The predictive camera may have a field of view that encompasses the coverage area of ​​the optical control system that is covered by the amiable laser sweep. The predictive camera may be configured to capture an image or representation of an area that includes the coverage area to identify and select a target object. The selected object may be assigned to the optical control system. In some embodiments, the field of view of the predictive camera and the coverage area of ​​the optical control system may be separated in time such that the field of view of the predictive camera encompasses the target at a first time and the coverage area of ​​the optical control system encompasses the target at a second time. Optionally, the predictive camera, the optical control system, or both may move relative to the target between the first and second times.

[0117] In some embodiments, multiple optical control systems can be combined to increase the coverage area on the surface. The multiple optical control systems can be configured such that the laser sweep along the translational axis of each optical control system overlaps with the laser sweep along the translational axis of an adjacent optical control system. The combined laser sweep defines a coverage area that at least one of the multiple beams from the multiple optical control systems can reach. The one or more prediction cameras may be positioned such that the field of view of the prediction camera covered by the one or more prediction cameras completely encompasses the coverage area. In some embodiments, the detection system may include two or more prediction cameras, each having a field of view. The fields of view of the prediction cameras can be combined to form a predicted field of view that completely encompasses the coverage area. In some embodiments, the predicted field of view may not completely encompass the coverage area at a single time point, but may encompass the coverage area over two or more time points (e.g., image frames). Optionally, the one or more prediction cameras may move relative to the coverage area over two or more time points to enable temporal coverage of the coverage area. A predictive camera or sensor may be configured to capture an image or representation of an area including the coverage area to identify and select an object to target. The selected object may be assigned to one of the multiple optical control systems based on the object's location and the area covered by the laser sweep of the individual optical control systems.

[0118] The optical control systems may be configured on a vehicle, such as the vehicle 100 shown in FIGS. 1-3. For example, the vehicle may be an autonomous vehicle. The autonomous vehicle may be a robot. In some embodiments, the vehicle may be controlled by a human. For example, the vehicle may be driven by a human driver. In some embodiments, the vehicle may be coupled to a second vehicle driven by a human driver, e.g., towed behind or pushed by the second vehicle. The vehicle may be remotely controlled by a human, e.g., by a remote control. In some embodiments, the vehicle may be remotely controlled via long wave signals, optical signals, satellite, or any other remote communication method. The optical control systems may be configured on the vehicle such that the coverage area overlaps with surfaces under, behind, in front of, or around the vehicle.

[0119] The vehicle may be configured to travel over a surface including multiple objects, having one or more target objects, e.g., a crop field including multiple plants and one or more weeds. The vehicle may include one or more of multiple wheels, a power source, a motor, a predictive camera, or any combination thereof. In some embodiments, the vehicle has sufficient clearance above the surface to move over the plants, e.g., crops, without damaging the plants. In some embodiments, the space between the inner edge of the left wheel and the inner edge of the right wheel is wide enough to pass over a row of plants without damaging the plants. In some embodiments, the distance between the outer edge of the left wheel and the outer edge of the right wheel is narrow enough to allow the vehicle to pass between two rows of plants, e.g., two rows of crops, without damaging the plants. In one embodiment, a vehicle with multiple wheels, multiple optical control systems, and a predictive camera can travel over a row of crops and emit one of the multiple beams toward a target, e.g., a weed, to incinerate or irradiate the weed.

[0120] Computer system and method The object identification and targeting method can be implemented using a computer system. In some embodiments, the detection system described herein includes a computer system. In some embodiments, the computer system can implement the object identification and targeting method autonomously without human input. In some embodiments, the computer system can implement the object identification and targeting method based on instructions provided by a human user via a detection terminal.

[0121] FIG. 9 illustrates components in a block diagram of a non-limiting, exemplary embodiment of a detection terminal 1400 according to various aspects of the disclosure. In some embodiments, the detection terminal 1400 is a device that displays a user interface to provide access to a detection system. As shown, the detection terminal 1400 includes a detection interface 1420. The detection interface 1420 enables the detection terminal 1400 to communicate with a detection system, such as the detection systems of FIG. 3 or FIG. 4. In some embodiments, the detection interface 1420 may include an antenna configured to communicate with the detection system, for example, by a remote control. In some embodiments, the detection terminal 1400 may also include a local communication interface, such as an Ethernet interface, a Wi-Fi interface, or other interface that allows other devices associated with the detection system to connect to the detection system via the detection terminal 1400. For example, the detection terminal may be a handheld device, such as a mobile phone, that runs a graphical interface to enable a user to remotely operate or monitor the detection system via Bluetooth, Wi-Fi, or a mobile network.

[0122] The detection terminal 1400 further includes a detection engine 1410. The detection engine may receive information regarding the status of a detection system, for example the detection system of Figure 3 or Figure 4. The detection engine may receive information regarding the number of identified objects, the identity of the identified objects, the location of the identified objects, the trajectory and predicted location of the identified objects, the number of targeted objects, the identity of the targeted objects, the location of the targeted objects, the location of the detection system, the elapsed time of a task performed by the detection system, the area covered by the detection system, the battery charge of the detection system, or a combination thereof.

[0123] Actual embodiments of the illustrated devices will include many more components therein known to those skilled in the art. For example, each illustrated device has a power source, one or more processors, computer-readable media for storing computer-executable instructions, etc. For clarity, these additional components are not illustrated herein.

[0124] In some examples, the procedures described herein (e.g., procedure 500 of FIG. 5, 600 of FIG. 6, or other procedures described herein) may be performed by a computing device or apparatus, such as a computing device having the computing device architecture 1600 shown in FIG. 10. In one example, the procedures described herein may be performed by a computing device having the computing device architecture 1600. The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device, a server (e.g., in a Software as a Service (SaaS) system or other server-based system), and / or any other computing device having resource capabilities to perform the processes described herein, including procedure 500 or 600. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other components configured to perform steps of the processes described herein. In some examples, a computing device may include a display (as an example of an output device or in addition to an output device), a network interface configured to communicate and / or receive data, any combination thereof, and / or other component(s). The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0125] The components of a computing device may be implemented in circuitry. For example, the components may include electronic circuitry or other electronic hardware that may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry) to perform various operations described herein, and / or may include and / or be implemented using computer software, firmware, or any combination thereof.

[0126] Procedures 500 and 600 are illustrated as logic flow diagrams, whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In terms of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed on one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to realize a process.

[0127] Additionally, the processes described herein may be executed under the control of one or more computer systems configured with executable instructions, implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors via hardware or a combination thereof. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0128] FIG. 10 illustrates an exemplary computing device architecture 1600 of an exemplary computing device capable of implementing various techniques described herein. For example, the computing device architecture 1600 may implement the procedures illustrated in FIGS. 5 and 6, control the detection system illustrated in FIGS. 3 or 4, or control the vehicle illustrated in FIGS. 1 and 2. The components of the computing device architecture 1600 are shown in electrical communication with each other using a connection 1605, such as a bus. The exemplary computing device architecture 1600 includes a processing unit (which may include a CPU and / or a GPU) 1610 and a computing device connection 1605 that couples various computing device components to the processor 1610, including a computing device memory 1615, such as a read only memory (ROM) 1620 and a random access memory (RAM) 1625. In some embodiments, the computing device may include a hardware accelerator.

[0129] The computing device architecture 1600 may include a cache of high speed memory directly connected to, closely connected to, or integrated as part of the processor 1610. The computing device architecture 1600 may copy data from the memory 1615 and / or storage device 1630 to the cache 1612 for quick access by the processor 1610. In this manner, the cache may provide a performance boost that avoids delays to the processor 1610 while waiting for data. These and other modules may control or be configured to control the processor 1610 to perform various actions. Other computing device memories 1615 may be available for use as well. The memory 1615 may include multiple different types of memories with different performance characteristics. The processor 1610 may include any general purpose processor as well as hardware or software services such as service 1 1632, service 2 1634, and service 3 1636 stored in the storage device 1630 configured to control the processor 1610, as well as special purpose processors where software instructions are built into the processor design. The processor 1610 may be a self-contained system that includes multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0130] To enable user interaction with the computing device architecture 1610, the input device 1645 can represent any number of input mechanisms, such as a microphone for audio, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The output device 1635 can also be one or more of several output mechanisms known to those skilled in the art, such as a display, projector, television, speaker device, etc. In some cases, a multimodal computing device can enable a user to provide multiple types of input to communicate with the computing device architecture 1600. The communication interface 1640 can generally manage and manage user input and computing device output. There is no limitation to operation with any particular hardware configuration, so the basic features herein can be easily substituted for improved hardware or firmware configurations as they are developed.

[0131] The storage device 1630 is a non-volatile memory and can be a hard disk or other type of computer readable medium capable of storing data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a random access memory (RAM) 1625, a read only memory (ROM) 1620, and hybrids thereof. The storage device 1630 can include services 1632, 1634, 1636 for controlling the processor 1610. Other hardware or software modules are also contemplated. The storage device 1630 can be connected to the computing device connection 1605. In one aspect, a hardware module that performs a particular function can include software components stored on a computer readable medium in association with hardware components, such as the processor 1610, the connection 1605, the output device 1635, etc., necessary to perform that function.

[0132] The term "computer-readable medium" includes, but is not limited to, portable or non-transportable storage devices, optical storage devices, and various other media that can store, contain, or carry instruction(s) and / or data. Computer-readable media can include non-transitory media that can store data and do not include carrier waves and / or transitory electronic signals propagating wirelessly or via wired connections. Examples of non-transitory media can include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory or memory devices. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0133] In some embodiments, computer readable storage devices, media, and memories may include cables or wireless signals carrying bit streams, etc. However, when referring to non-transitory computer readable storage media, media such as energy, carrier signals, electromagnetic waves, and the signals themselves are expressly excluded.

[0134] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For clarity of explanation, in some cases, the technology may be presented as including individual functional blocks, with functional blocks including devices, device components, steps or routines in a method implemented in software, or a combination of hardware and software. Additional components other than those shown in the drawings and / or described herein may also be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other examples, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0135] Particular embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. While a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Moreover, the order of operations may be rearranged. A process terminates when the operations are completed, but may include additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0136] The processes and methods according to the above examples can be implemented using computer-executable instructions stored on or otherwise available from a computer-readable medium. Such instructions can include, for example, instructions and data that cause a general purpose computer, special purpose computer, or processing device to perform or otherwise perform a particular function or group of functions. Some of the computer resources used can be accessible over a network. The computer-executable instructions can be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, network storage devices, etc.

[0137] A device implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may adopt any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor(s) may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functions described herein may also be embodied in peripheral devices or add-in cards. As a further example, such functions may also be implemented on a circuit board between different chips or different processes executing within a single device.

[0138] The instructions, media for carrying such instructions, computing resources for executing them, and other structures for supporting such computing resources are examples of means for providing the functionality described in this disclosure.

[0139] The various illustrative logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functions in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0140] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any functions described as modules or components may be implemented together in an integrated logic device, or separately as separate but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium including program code having instructions that, when executed, perform one or more of the above-described methods. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, or the like. Additionally or alternatively, the techniques may be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, that carries or communicates program code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer.

[0141] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine, or the like. A processor may also be implemented as a combination of computing devices, such as, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such components. Thus, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0142] While illustrative embodiments have been illustrated and described, it will be understood that various changes can be made therein without departing from the spirit and scope of the disclosure.

[0143] In the foregoing description, aspects of the present application have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, while exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concepts may be embodied and used in various other ways, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. Various features and aspects of the above-described applications may be used individually or in combination. Moreover, the embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative and not restrictive. For purposes of explanation, the methods have been described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in an order different from that described.

[0144] Those skilled in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein can be replaced with the less than ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.

[0145] Where a component is described as being "configured" to perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0146] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component and / or that is in direct or indirect communication with another component (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0147] Claim language or other language reciting "at least one" of a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting "at least one of A and B" may mean A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" may mean A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language reciting "at least one" of a set and / or "one or more" of a set does not limit the set to the items recited in the set. For example, claim language reciting "at least one of A and B" may mean A, B, or A and B, and may include additional items not recited in the set of A and B.

[0148] As used herein, the terms "about" and "approximately" in reference to numerical values ​​are used herein to include numerical values ​​that fall within 10%, 5%, or 1% of that numerical value in either direction (greater or smaller), unless otherwise stated or clear from the context (except where such numerical value exceeds 100% of its possible values).

[0149] Working Example The invention is further illustrated by the following non-limiting examples.

[0150] Example 1 Weeding in crop fields This example describes weeding in a crop field using the object tracking method of the present disclosure. A vehicle as shown in FIG. 1 and FIG. 3 equipped with a prediction system, a targeting system, and an infrared laser was positioned in the crop field shown in FIG. 2. The vehicle drove between the crop rows at a speed of about 2 miles per hour, and the prediction camera collected images of the field. The prediction system identified weeds in the images and tracked the weeds by pairing the same weed identified in two or more images and de-duplicating the images. The prediction system selected a weed to be weeded from the tracked weeds. The prediction system determined a predicted location of the weed based on the location of the weed in the first image, the location of the weed in the second image, and the time when the weed will be targeted. The prediction system sent the predicted location to the targeting system.

[0151] A targeting system was selected based on availability and proximity to the selected weed. The targeting system included a targeting camera and an infrared laser, the direction of which was adjusted by a mirror controlled by an actuator. The mirror reflected visible light from the surface to the targeting camera and infrared light from the laser back to the surface. The targeting system translated the predicted positions received from the prediction system into actuator positions. The targeting system adjusted the actuator to direct the targeting camera and the infrared laser beam to the predicted positions of the selected weeds. The targeting camera imaged the field at the predicted positions of the weeds and corrected its position to generate a target position. The targeting system adjusted the position of the targeting camera and the infrared laser beam based on the target positions of the weeds and activated the infrared beam toward the weeds' positions. The beam illuminated the weeds with infrared light for a time sufficient to damage or kill the weeds, while adjusting the position of the laser beam to account for the movement of the autonomous vehicle during illumination.

[0152] Example 2 Debris Tracking and Removal in the Built Environment In this example, a system and method for automated identification and removal of debris in a rough environment is described. A vehicle equipped with a predictive camera, a targeting camera, and a debris collection tool travels through a construction site. The construction site has a rough terrain surface. The predictive camera images a surface area of ​​the construction site and detects debris in the images. The predictive system identifies debris in images collected by the predictive camera and tracks the debris by pairing the same debris objects identified in two or more images and de-duplicating the images. The predictive system selects debris objects from the identified debris to be collected. The predictive system determines a predicted location of the debris based on the location of the debris in the first image, the location of the debris in the second image, and the vector velocity of the vehicle. The predictive system transmits the predicted location to a targeting system.

[0153] A targeting system is selected based on availability and proximity to the selected debris. The targeting system directs actuators controlling the targeting camera and debris collection tool to point the targeting camera and debris collection tool to the predicted location of the selected debris. The targeting camera images the construction site at the predicted location of the debris and modifies its location to generate a target location. The targeting system directs the actuators to point the targeting camera and debris collection tool to the target location of the debris, and the debris collection tool collects the debris.

[0154] While preferred embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the invention. It is understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of the invention. It is intended that the following claims define the scope of the invention, and that methods and structures within the scope of these claims, and their equivalents, are covered thereby.

Claims

1. 1. A method for predicting a position of a tracked object relative to a moving vehicle, comprising: determining a first set of actual positions of a first set of objects relative to the moving vehicle in a first image at a first time; determining a second set of actual positions of a second set of objects relative to the moving vehicle in a second image at a second time, the second set of objects including one or more of the same objects as the first set of objects; applying a plurality of test shifts to the first set of actual positions of the first set of objects to obtain a set of test positions of the first set of objects at the second time, wherein each test shift of the plurality of test shifts is determined based on a distance between a first actual position of a first object at the first time and a second actual position of a second object at the second time; determining a plurality of offsets between each of the test locations in the set of test locations for the first set of objects at the second time and each of the second set of actual locations for the second set of objects at the second time, each offset of the plurality of offsets corresponding to a test shift in the plurality of test shifts; selecting a refinement shift from the plurality of test shifts based on the plurality of offsets; identifying a tracked object included in both the first object set and the second object set; The method comprising:

2. 2. The method of claim 1, further comprising applying the refinement shift to the first set of actual positions of the first set of objects at the first time to establish a set of shifted positions for the first set of objects at the second time.

3. 2. The method of claim 1, further comprising determining a vector displacement of the tracked object between the first time and the second time from an actual position of the tracked object in the first image at the first time and an actual position of the tracked object in the second image at the second time.

4. The method of any one of claims 1 to 3, wherein the refinement shift is a test shift from the plurality of test shifts having a smallest corresponding offset.

5. The method of claim 1 further comprising determining a vector velocity of the moving vehicle.

6. 1. A method for predicting a position of a tracked object relative to a moving vehicle, comprising: determining a first set of actual positions of a first set of objects relative to the moving vehicle in a first image at a first time; determining a second set of actual positions of a second set of objects relative to the moving vehicle in a second image at a second time, the second set of objects including one or more of the same objects as the first set of objects; applying a predicted shift to the first set of actual positions of the first set of objects to generate a set of intermediate positions, the predicted shift being based on a vector velocity of the moving vehicle and a time difference between the first time and the second time; applying a second shift to the set of intermediate positions to obtain a set of shifted positions for the first set of objects at the second time, the second shift being determined based on a distance between a first intermediate position of a first object in the set of intermediate positions and a second actual position of a second object at the second time; identifying tracked objects included in the first set of objects; Specifying a search area; selecting from the second set of actual positions an actual position of the tracked object from the second set of objects in the second image at the second time, wherein the actual position of the tracked object in the second image at the second time is within the search area; determining a vector displacement of the tracked object between the first time and the second time from an actual position of the tracked object in the first image at the first time and an actual position of the tracked object in the second image at the second time; The method comprising:

7. The method of claim 6 , wherein an intermediate position of the tracked object from the set of intermediate positions is within the search area.

8. The method of claim 6 , comprising specifying the search area around the intermediate position of the tracked object.

9. The method of claim 7 or claim 8, comprising specifying the search area around the actual position of the tracked object.

10. 1. A method for predicting a position of a tracked object relative to a moving vehicle, comprising: determining a first set of actual positions of a first set of objects relative to the moving vehicle in a first image at a first time; determining a second set of actual positions of a second set of objects relative to the moving vehicle in a second image at a second time, the second set of objects including one or more of the same objects as the first set of objects; identifying tracked objects included in the first set of objects; applying a predicted shift to an actual position of the tracked object in the first image at the first time to generate a shifted position of the tracked object at the second time, the predicted shift being based on a vector velocity of the moving vehicle and a time difference between the first time and the second time; Specifying a search area; selecting from the second set of actual positions an actual position of the tracked object from the second set of objects in the second image at the second time, wherein the actual position of the tracked object in the second image at the second time is within the search area; determining a vector displacement of the tracked object between the first time and the second time from an actual position of the tracked object in the first image at the first time and an actual position of the tracked object in the second image at the second time; The method comprising:

11. The method of claim 10, comprising applying the predicted shift to each position of the first set of actual positions to generate a set of shifted positions.

12. 12. The method of claim 11, further comprising applying a second shift to the shift positions to generate modified shift positions of the tracked object, the second shift being based on a distance between a shift position in the set of shift positions and an actual position in the second set of actual positions.

13. The method of claim 12 , comprising specifying the search area around the modified shift position of the tracked object.

14. The method of claim 12 or claim 13, wherein the modified shift position of the tracked object is within the search area.

15. The method of any one of claims 5 to 8 and claims 10 to 13, further comprising determining a trajectory of the tracked object over time based on the vector displacement of the tracked object and the vector velocity of the moving vehicle between the first time and the second time.

16. The method of any one of claims 3, 6 to 8 and 10 to 13, further comprising determining a predicted position of the tracked object at a third time based on the vector displacement of the tracked object between the first time and the second time and the elapsed time between the second time and the third time.

17. 1. A method for predicting a position of a tracked object relative to a moving vehicle, comprising: determining a first set of actual positions of a first set of objects relative to the moving vehicle in a first image at a first time; identifying tracked objects included in the first set of objects; determining a vector velocity of the moving vehicle; determining a predicted position of the tracked object at a second time based on the velocity vector, a time difference between the first time and the second time, and an actual position of the tracked object in the first image at the first time; The method comprising:

18. The method of any one of claims 5 to 8, claims 10 to 13 and claim 17, wherein the vector velocity is determined using optical flow, a rotary encoder, a global positioning system, or a combination thereof.

19. 18. The method of any one of claims 1 to 3, claims 5 to 8, claims 10 to 13 and claim 17, wherein the first set of actual positions, the second set of actual positions, the shifted set of positions, the predicted positions, or a combination thereof comprises one dimension and two dimensions.

20. The method of claim 19 , wherein the first dimension and the second dimension are parallel to an image plane containing the first image or the second image.

21. 20. The method of claim 19, wherein the first set of actual positions, the second set of actual positions, the shifted set of positions, the predicted positions, or a combination thereof further comprises three dimensions.

22. The method of claim 21 , wherein the third dimension is perpendicular to an image plane containing the first image or the second image.

23. The method of claim 21 , wherein the three dimensions are determined based on the vector velocity of the moving vehicle and the vector displacement of the tracked object.

24. The method of claim 19 , wherein the test shift, the prediction shift, or the refinement shift is applied along the one dimension.

25. The method of claim 19 , wherein the test shift, the prediction shift, or the refinement shift is applied within the first dimension and the second dimension.

26. 22. The method of claim 21, wherein the test shift, the prediction shift, or the refinement shift is applied within the first dimension, the second dimension, and the third dimension.

27. The method of any one of claims 1 to 3, claims 5 to 8, claims 10 to 13 and claim 17, further comprising determining parameters of the tracked object.

28. 28. The method of claim 27, wherein the parameters include size, shape, plant category, orientation, or plant type.

29. 28. The method of claim 27, further comprising determining a confidence value for the parameter.

30. The method of any one of claims 1 to 3, claims 5 to 8, claims 10 to 13 and claim 17, further comprising determining a second predicted position based on the predicted position and the actual position of the tracked object in the first image at the first time, the actual position of the tracked object in the second image at the second time, or both.

31. The method of any one of claims 1 to 3, claims 5 to 8, claims 10 to 13 and claim 17, further comprising targeting the tracked object at the predicted position.

32. 32. The method of claim 31 , wherein targeting the tracked object includes correcting for parallax effects, distortion of the first image, distortion of the second image, targeting distortion, or a combination thereof.

33. The method of claim 31 , wherein targeting the tracked object includes aiming a targeting sensor, an instrument, or both, at the predicted location.

34. 34. The method of claim 33, wherein targeting the tracked object further comprises dynamically tracking the object with the targeting sensor, the instrument, or both.

35. 35. The method of claim 34, wherein dynamically tracking the object includes moving the targeting sensor, the instrument, or both, to match the vector velocity of the moving vehicle so that the targeting sensor, the instrument, or both remain aimed toward the predicted location while the moving vehicle is moving.

36. 36. The method of claim 35, wherein manipulating the tracked object includes illuminating the tracked object with electromagnetic radiation, moving the tracked object, spraying the tracked object, or a combination thereof.

37. 37. The method of claim 36, wherein the electromagnetic radiation is infrared light.

38. The method of any one of claims 1 to 3, claims 5 to 8, claims 10 to 13 and claim 17, wherein the first set of objects, the second set of objects, the tracked objects, or a combination thereof includes plants.

39. The method according to any one of claims 1 to 3, claims 5 to 8, claims 10 to 13 and claim 17, wherein the mobile vehicle is a trailer or an autonomous vehicle.

40. a predictive system including a sensor configured to collect a first image at a first time and a second image at a second time, the sensor coupled to a vehicle; a targeting system having an instrument; 1. An object tracking system comprising: The prediction system comprises: determining a first set of actual positions of a first set of objects relative to the vehicle in the first image at the first time; determining a second set of actual positions of a second set of objects relative to the vehicle in the second image at the second time, the second set of objects including one or more of the same objects as the first set of objects; applying a plurality of test shifts to the first set of actual positions of the first set of objects to obtain a set of test positions of the first set of objects at the second time, each test shift of the plurality of test shifts being determined based on a distance between a first actual position of a first object at the first time and a second actual position of a second object at the second time; determining a plurality of offsets between each of the test locations in the set of test locations for the first set of objects at the second time and each of the second set of actual locations for the second set of objects at the second time, each offset of the plurality of offsets corresponding to a test shift in the plurality of test shifts; selecting a refinement shift from the plurality of test shifts based on the plurality of offsets; identifying tracked objects included in both the first object set and the second object set; It is configured as follows: The object tracking system, wherein the targeting system is configured to target the tracked object with the instrument.

41. 41. The object tracking system of claim 40, wherein the vehicle is configured to move relative to a surface.

42. 42. The object tracking system of claim 41, wherein the first set of objects, the second set of objects, the tracked object, or a combination thereof, are positioned on the surface.

43. The object tracking system of any one of claims 40 to 42, wherein the first object set, the second object set, the tracked object, or a combination thereof includes plants.

44. 43. The object tracking system of any one of claims 40 to 42, wherein the vehicle comprises an autonomous car or trailer.

45. An object tracking system according to any one of claims 40 to 42, wherein the sensor is fixed relative to the vehicle.

46. 43. The object tracking system of any one of claims 40 to 42, wherein the instrument is configured to point at and manipulate the tracked object.

47. The object tracking system of any one of claims 40 to 42, wherein the targeting system further comprises a targeting sensor.

48. 48. The object tracking system of claim 47, wherein the targeting sensor is configured to point toward and image the tracked object.

49. 48. The object tracking system of claim 47, wherein the targeting system is configured to aim the instrument at the tracked object based on a position of the tracked object within a targeting image collected by the targeting sensor.

50. An object tracking system according to any one of claims 40 to 42, wherein the instrument is a laser.

Images