System and method for point-to-point matching and targeting of objects

Abstract

To provide a system and method for point-to-point matching and targeting of suitable objects. [Solution] This specification discloses methods, devices, modules, and systems that can be used for precise targeting of objects. Point-to-point targeting methods may be implemented by a system having two or more sensors, where the two or more sensors can locate objects and coordinate handoffs between the two or more sensors. These methods, devices, modules, and systems may be used for automated crop cultivation or maintenance. Devices disclosed herein may be configured to locate, identify, and autonomously target weeds with a beam, such as a laser beam, that can burn or irradiate the weeds. These methods, devices, modules, and systems may be used for crop management or weed control at home.

Description

Background Art

[0001] Cross-reference This application claims the benefit of U.S. Non-Provisional Application No. 17 / 576,814, filed on January 14, 2022, and entitled "SYSTEMS AND METHODS FOR POINT TO POINT OBJECT MATCHING AND TARGETING", and U.S. Provisional Application No. 63 / 162,285, filed on March 17, 2021, and entitled "SYSTEMS AND METHODS FOR POINT TO POINT OBJECT MATCHING AND TARGETING", the entire contents of which are incorporated herein by reference for all purposes.

[0002] With the progress of technology, the work that humans have been doing has been increasingly automated. Jobs performed in highly controlled environments, such as factory assembly lines, can be automated by instructing machines to perform the work in the same way every time. However, jobs performed in unpredictable environments, such as driving in the city or vacuuming a cluttered room, rely on dynamic feedback and adaptation to perform the work. Autonomous systems often struggle to identify and locate objects in unpredictable environments. Improvements in object detection, location, and targeting methods have advanced automation technology and improved the ability of autonomous systems to react and adapt to unpredictable environments.

Summary of the Invention

Means for Solving the Problems

[0003] In various embodiments, the Disclosure provides a method for targeting an object, which includes: collecting a predictive image using a predictive sensor; identifying a target object in the predictive image; determining a predicted location of the object based on the predictive image; orienting a targeting sensor to the predicted location; collecting a target image of the predicted location using the targeting sensor; identifying an object in the target image; and determining a target location of the object based on the target image.

[0004] In some embodiments, the method further includes orienting the targeting sensor to a target location. In some embodiments, orienting the targeting sensor to a target location includes determining an offset between a first position and a second position of the targeting sensor. In some embodiments, when the targeting sensor is positioned at the second position, the targeting sensor is orienting to the target location. In some embodiments, the target location is closer to the object than the predicted location. In some embodiments, the targeting sensor is associated with an instrument.

[0005] In some embodiments, the method further includes orienting the instrument to a target position. In some embodiments, orienting the instrument to a target position includes determining an offset between a first position of the instrument and a second position of the instrument. In some embodiments, when the instrument is positioned at the second position, the instrument is oriented to the target position. In some embodiments, the targeting sensor and the instrument are coupled to each other. In some embodiments, the orientation of the targeting sensor and the orientation of the instrument are correlated.

[0006] In some embodiments, the method further includes manipulating an object using an instrument. In some embodiments, manipulating an object includes irradiating the object with electromagnetic radiation. In some embodiments, the electromagnetic radiation is infrared light. In some embodiments, manipulating an object includes moving the object. In some embodiments, manipulating an object includes spraying the object.

[0007] In some embodiments, the predictive sensor differs from the targeting sensor in one or more parameters selected from the group consisting of sensor type, sensor resolution, magnification, field of view, color balance, and color sensitivity. In some embodiments, the predictive sensor is positioned at a different angle, distance, or both relative to the object than the targeting sensor. In some embodiments, the predictive sensor and targeting sensor are coupled to a vehicle. In some embodiments, the vehicle is in motion. In some embodiments, the predictive position takes into account the vehicle's motion relative to the object between the time the predictive image is collected and the time the target image is collected. In some embodiments, the target position takes into account the vehicle's motion relative to the object between the time the target image is collected and the time the operation is performed.

[0008] In some embodiments, the object is located on, above, or below the surface. In some embodiments, the surface is not flat. In some embodiments, the surface is an agricultural surface. In some embodiments, the surface is a construction surface. In some embodiments, the offset takes into account variations in the depth of the surface. In some embodiments, the target position is 50 mm or less, 25 mm or less, 10 mm or less, 5 mm or less, 3 mm or less, 2 mm or less, or 1 mm or less from the object.

[0009] In some embodiments, the predictive sensor is selected from the group consisting of a camera, a LiDAR sensor, a photodetector, an active pixel sensor, a semiconductor detector, an ultrasonic sensor, a RADAR detector, a sonar sensor, and a photodiode array. In some embodiments, the targeting sensor is selected from the group consisting of a camera, a LiDAR sensor, a photodetector, an active pixel sensor, a semiconductor detector, an ultrasonic sensor, a RADAR detector, a sonar sensor, and a photodiode array.

[0010] In some embodiments, identifying an object in a target image involves matching the object in the target image with an object in a predicted image. In some embodiments, the matching involves using a trained machine learning model to match the object in the target image with an object in the predicted image. In some embodiments, identifying an object in a predicted image involves using a trained machine learning model to identify the object. In some embodiments, the trained machine learning model is a deep learning model. In some embodiments, the object is selected from a group consisting of weeds, plants, and obstacles.

[0011] In some embodiments, the predicted position includes a position in the predicted image, a position of a targeting sensor, a position of an instrument, a position of a predictive sensor, a position of an object on a surface, a position of a vehicle, or a combination thereof. In some embodiments, the target position includes a position in the target image, a position of a targeting sensor, a position of an instrument, a position of a predictive sensor, a position of an object on a surface, a position of a vehicle, or a combination thereof.

[0012] In various embodiments, the Disclosure provides a system for targeting an object, the system comprising a prediction system having a prediction sensor and an object identification module, and a targeting system having a targeting sensor, an actuator configured to control the targeting sensor, and an object matching module, wherein the prediction sensor is configured to collect a prediction image, the object identification module is configured to identify a target object in the prediction image and to determine the predicted location of the object based on the prediction image, the targeting module is configured to receive the predicted location of the object from the object identification module, the actuator is configured to orient the targeting sensor to the predicted location of the object, the targeting sensor is configured to collect a target image of the predicted location, and the object matching module is configured to locate an object in the target image and to determine the target location of the object based on the target image.

[0013] In some embodiments, the actuator is configured to orient the targeting sensor to the target position. In some embodiments, the target position is closer to the object than the predicted position. In some embodiments, the actuator is configured to rotate or translate a mirror. In some embodiments, the rotation or translation of the mirror orients the targeting sensor.

[0014] In some embodiments, the targeting module further includes an instrument directed by an actuator. In some embodiments, the orientation of the targeting sensor and the orientation of the instrument are fixed relative to each other. In some embodiments, the instrument is configured to manipulate an object. In some embodiments, the instrument includes a laser. In some embodiments, the laser is an infrared laser. In some embodiments, the instrument includes a grabber. In some embodiments, the instrument includes a sprayer.

[0015] In some embodiments, the predictive sensor differs from the targeting sensor in one or more parameters selected from the group consisting of sensor type, sensor resolution, magnification, field of view, color balance, and color sensitivity. In some embodiments, the predictive sensor is positioned at a different angle, distance, or both relative to the object than the targeting sensor. In some embodiments, the object recognition module includes a trained machine learning model. In some embodiments, the object matching module includes a trained machine learning model. In some embodiments, the trained machine learning model is a deep learning model.

[0016] In some embodiments, the object is selected from a group consisting of weeds, plants, and obstacles. In some embodiments, the system is configured to perform the method of the present disclosure.

[0017] In various embodiments, the Disclosure provides a method for locating an object, which includes locating a first object in an image collected by a first sensor, locating a second object in an image collected by a second sensor, and determining that the second object is the same as the first object using an object-matching deep learning model trained with a first set of training images and a second set of training images.

[0018] In some embodiments, a first set of training images is acquired by a first sensor, and a second set of training images is acquired by a second sensor. In some embodiments, the first sensor differs from the second sensor in one or more parameters selected from the group consisting of sensor type, sensor resolution, magnification, field of view, color balance, and color sensitivity. In some embodiments, the first sensor is positioned at a different angle, distance, or both relative to the object than the second sensor. In some embodiments, the method further includes matching the second object with the first object. This specification also provides, for example, the following: (Item 1) A method for targeting objects, To provide predictive representations, Identifying the target object within the aforementioned predictive representation, Determining the predicted position of the object within the prediction representation, To provide a targeting representation of the aforementioned predicted position, Identifying the object within the targeting representation, The target position of the object is determined based on the targeting representation, The method, including the method described above. (Item 2) The method according to item 1, further comprising collecting the prediction representation using a prediction sensor, collecting the targeting representation using a targeting sensor, or both. (Item 3) The method according to item 2, further comprising orienting the targeting sensor to the predicted position. (Item 4) The method according to item 2, further comprising orienting the targeting sensor to the target position. (Item 5) The method of item 2, wherein orienting the targeting sensor to the target position includes determining the offset between a first position and a second position of the targeting sensor. (Item 6) The method according to item 2, wherein when the targeting sensor is positioned at the second position of the targeting sensor, the targeting sensor is directed towards the target position. (Item 7) The method according to item 1, wherein the target position is closer to the object than the predicted position. (Item 8) The method according to item 1, further comprising orienting the instrument to the target position. (Item 9) The method according to item 8, wherein the orientation of the device is correlated with the orientation of the targeting sensor. (Item 10) The method of item 8, wherein orienting the instrument to the target position includes determining an offset between a first position of the instrument and a second position of the instrument, and when the instrument is positioned at the second position of the instrument, the instrument is oriented to the target position. (Item 11) The method of item 8, further comprising manipulating the object using the aforementioned apparatus. (Item 12) The method according to item 11, wherein manipulating the object is selected from the group consisting of irradiating the object with electromagnetic radiation, moving the object, spraying the object, and combinations thereof. (Item 13) The predictive sensor differs from the targeting sensor in that the positioning includes an angle, distance, or both relative to the object, with respect to one or more parameters selected from the group consisting of sensor type, sensor resolution, magnification, field of view, color balance, color sensitivity, and positioning, as described in item 2. (Item 14) The method according to item 2, wherein the predictive sensor and the targeting sensor are coupled to a vehicle. (Item 15) The method of item 14, wherein the predicted position takes into account the motion of the vehicle relative to the object between the time the predicted representation is collected and the time the targeting representation is collected. (Item 16) The method of item 14, wherein the target position takes into account the motion of the vehicle relative to the object between the time the targeting representation is collected and the time the operation is performed. (Item 17) The method according to item 1, wherein the object is located on, above, or below the surface. (Item 18) The offset is the method described in item 17, which takes into account the variation in the depth of the surface. (Item 19) The method according to item 1, wherein the target position is 50 mm or less, 25 mm or less, 10 mm or less, 5 mm or less, 3 mm or less, 2 mm or less, or 1 mm or less from the object. (Item 20) The method according to item 2, wherein the predictive sensor, the targeting sensor, or both are selected from the group consisting of a camera, a LiDAR sensor, a photodetector, an active pixel sensor, a semiconductor detector, an ultrasonic sensor, a radar detector, a sonar sensor, and a photodiode array. (Item 21) Using a pre-trained machine learning model, Identifying the object within the targeting representation, Identifying the object in the aforementioned prediction representation, Matching the object in the targeting representation with the object in the prediction representation, The method described in item 1, including the method described in item 1. (Item 22) The method according to item 1, wherein the object is selected from the group consisting of weeds, plants, and obstacles. (Item 23) The method of item 2, wherein the predicted position, the target position, or both include the position in the prediction representation, the position of the targeting sensor, the position of the device, the position of the prediction sensor, the position of the object on the surface, the position of the vehicle, or any combination thereof. (Item 24) A method for targeting objects, Collecting predictive representations using predictive sensors, Identifying the target object within the aforementioned predictive representation, Determining the predicted position of the object within the prediction representation, The targeting sensor is directed towards the predicted position, Using the aforementioned targeting sensor, collect the targeting representation of the predicted position, Identifying the object within the targeting representation, The target position of the object is determined based on the targeting representation, The targeting sensor is directed towards the target position, Orienting the device towards the target position, Manipulating the object using the aforementioned device, The method, including the method described above.

[0019] Novel features of the present invention are described in particular in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by referring to the following detailed description showing exemplary embodiments in which the principles of the present invention are utilized, and to the following accompanying drawings. [Brief explanation of the drawing]

[0020] [Figure 1] This document shows isometric views of laser optical systems, including laser and visible light paths, according to one or more embodiments of this specification. [Figure 2] The image shows a top view of a laser optical system in which a laser path and a visible light path are shown, according to one or more embodiments of this specification. [Figure 3] The following are side cross-sectional views of laser optical systems, one or more embodiments of this specification, showing a clean air path. [Figure 4A] The following are side views of a targeting laser and the targeting coverage area of ​​a targeting laser according to one or more embodiments of this specification. [Figure 4B] The following are front views of a targeting laser and the targeting coverage area of ​​the targeting laser according to one or more embodiments of this specification. [Figure 5] The following are isometric views of a predictive camera, a plurality of targeting lasers, the predictive view area of ​​the predictive camera, and the targeting coverage area of ​​the targeting lasers according to one or more embodiments of this specification. [Figure 6] The following are front views of the coverage areas of an autonomous laser weeding robot, a predictive camera, and multiple targeting lasers according to one or more embodiments of this specification. [Figure 7] This specification shows isometric views of the coverage areas of an autonomous laser weeding robot, a predictive camera, and multiple targeting lasers according to one or more embodiments of this specification. [Figure 8] This specification describes methods for identifying, assigning, and targeting objects according to one or more embodiments thereof. [Figure 9] This specification describes a method for identifying, allocating, targeting, and removing weeds in a field, according to one or more embodiments thereof. [Figure 10A] This specification describes one or more embodiments of a system for identifying, locating, targeting, and manipulating objects. [Figure 10B]The crop region (inset) shown here is a target image of a weed on a surface and a predicted image of the area surrounding the weed, according to one or more embodiments of this specification. [Figure 11] This specification describes methods for identifying, locating, and targeting objects according to one or more embodiments thereof. [Figure 12] One or more embodiments of this specification show a calibration grid for calibrating a predictive sensor or a targeting sensor. [Figure 13] This is a block diagram showing components of a detection terminal according to an embodiment of the present disclosure. [Figure 14] This flowchart shows the procedure for determining the detection terminal position according to an embodiment of the present disclosure. [Figure 15] This is an exemplary block diagram of a computing device architecture for a computing device that can implement the various technologies described herein. [Modes for carrying out the invention]

[0021] Various exemplary embodiments of this disclosure are described in detail below. While specific implementations are described, it should be understood that this description is for illustrative purposes only. Those skilled in the art will recognize that other components and configurations can be used without departing from the spirit and scope of this disclosure. Therefore, the following descriptions and drawings are illustrative and should not be construed as limiting. Many specific details are described in order to fully understand this disclosure. However, in certain examples, well-known or conventional details are omitted to avoid obscuring the description. Any reference in this disclosure to one or an embodiment may refer to the same embodiment or any embodiment, and such references mean at least one of the exemplary embodiments.

[0022] References to “one embodiment” or “an embodiment” mean that certain features, structures, or characteristics described in relation to that embodiment are included in at least one embodiment of this disclosure. The phrase “in one embodiment” appearing in various places in this specification does not necessarily refer to all identical embodiments, nor is it the case that separate or alternative exemplary embodiments are mutually exclusive with other exemplary embodiments. Furthermore, various features are described that may be shown by some exemplary embodiments but not by others. Any feature of one example can be integrated with or used in conjunction with any other feature of any other example.

[0023] The terms used herein generally have the common meaning in the art within the context of this disclosure and in the specific context in which each term is used. Alternative languages ​​and synonyms may be used for one or more terms discussed herein, and no special meaning should be placed on whether a term is detailed or discussed herein. In some cases, synonyms for certain terms are provided. The explanation of one or more synonyms does not preclude the use of other synonyms. The use of examples anywhere in this specification, including examples of any term discussed herein, is illustrative only and is not intended to further limit the scope and meaning of this disclosure or any exemplary term. Similarly, this disclosure is not limited to the various exemplary embodiments given herein.

[0024] Without intending to limit the scope of this disclosure, examples of apparatus, apparatus, methods, and their associated results in exemplary embodiments of this disclosure are given below. Note that titles or subtitles may be used illustratively for the convenience of the reader and are not intended to limit the scope of this disclosure. Unless otherwise defined, technical and scientific terms used herein have the meanings generally understood by those skilled in the art to which this disclosure pertains. In case of any conflict, this specification, including its definitions, shall prevail.

[0025] Further features and advantages of this disclosure are set forth in the following description and may become apparent in part from that description or by practicing the principles disclosed herein. These features and advantages may be realized in combination with the instructions specifically set forth in the appended claims. These and other features of this disclosure may become more apparent from the following description and the appended claims or by practicing the principles described herein.

[0026] To clarify the explanation, in some cases, this technology may be presented as including individual functional blocks representing devices, device components, steps, or routines in a manner embodied in software or a combination of hardware and software.

[0027] In drawings, certain structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, such features may be arranged in a different way and / or order than those shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular drawing does not imply that such features are required in all embodiments; in some embodiments, they may be omitted or combined with other features.

[0028] While the concepts of this disclosure can take on various modifications and alternative forms, specific embodiments are shown as examples in the drawings and described in detail herein. However, it should be understood that there is no intention to limit the concepts of this disclosure to any specific form disclosed, but rather to cover all modifications, equivalents, and alternatives that are consistent with this disclosure and the appended claims.

[0029] This disclosure provides a system and method for point-to-point matching and targeting of objects to autonomously coordinate object detection across multiple sensors. Autonomous systems, such as self-driving cars, often rely on electromagnetic sensors to detect and identify objects, interact with objects, or avoid interacting with objects. Autonomous systems can use multiple sensors to expand the detection angle, widen the field of view, or improve detection resolution. However, coordinating multiple sensors can be difficult, for example, if the sensors have particularly different resolutions, fields of view, color sensitivities, positioning, or sensor types.

[0030] This specification describes systems and methods for coordinating object detection, localization, and targeting between two or more sensors in a detection system. The systems and methods of this disclosure may be used to localize an object with a first sensor, identify the same object with a second sensor, and manipulate that object. The first sensor may be used to determine the predicted location of the object, and the second sensor may be used to refine the object's location and determine its target location. The target location of the object may be more accurate than the predicted location of the object.

[0031] The systems and methods of this disclosure may have a wide range of applications involving the identification, detection, and precise targeting of objects located in irregular or unpredictable environments. For example, the systems and methods of this disclosure may be used to precisely locate and manipulate objects on uneven or non-flat surfaces, such as vegetation in a field or debris at a construction site. Alternatively, or in addition, the systems and methods of this disclosure may be used to precisely locate and target objects or areas within a larger area, such as a field, or within a smaller area within a field, in satellite imagery collected from orbit.

[0032] In some embodiments, by coordinating object detection between two or more sensors, the accuracy of object localization can be improved while maintaining a wide field of view by first locating an object using a wide-field sensor and then identifying the same object in a sensor with a narrower field of view. In some embodiments, by coordinating object detection between two or more sensors, the speed of object detection and targeting can be increased by enabling parallel targeting of multiple objects. For example, multiple objects may be located by a wide-field sensor, a first object may be assigned to a first targeting module, a second object to a second targeting module, and a third object to a third targeting module. Each of the first, second, and third objects can be located and targeted by the first, second, and third targeting modules, respectively.

[0033] Coordinating object detection between two or more sensors can be challenging for objects in unpredictable environments or on uneven or non-flat surfaces. For example, irregularities in the three-dimensional position of an object due to its positioning on an uneven surface can reduce the accuracy of object localization, potentially leading to discrepancies between the localization of the object in the image collected by the first sensor and the localization of the same object in the image collected by the second sensor. For instance, an object on a non-flat surface may appear to be in a different position in the image collected by the first sensor compared to the image collected by the second sensor. The method described herein enables accurate object localization between two or more sensors by using machine learning models, such as deep learning models, to account for irregularities in the object or environment.

[0034] As used herein, “image” may refer to a representation of a region or object. For example, an image may be a visual representation of a region or object formed by electromagnetic radiation (e.g., light, X-rays, microwaves, or radio waves) scattered from the region or object. In another example, an image may be a point cloud model formed by a light-detection ranging (LIDAR) or radio-detection ranging (RADAR) sensor. In yet another example, an image may be a sonogram produced by detecting sound waves, very low frequencies, or ultrasound reflected from a region or object. As used herein, “imaging” may be used to describe the process of collecting or generating a representation (e.g., an image) of a region or object.

[0035] Where used herein, positions such as the position of an object or the position of a sensor may be expressed relative to a reference frame. Exemplary reference frames include surface reference frames, vehicle reference frames, sensor reference frames, or actuator reference frames. Positions can be easily converted between reference frames, for example, by using conversion coefficients or calibration models. It should be understood that a position, position change, or offset may be represented in one reference frame, but may also be represented in any reference frame, or may be easily converted between reference frames.

[0036] Detection system In some embodiments, a detection system of the Disclosure configured to perform the method of the Disclosure may have a prediction system and a targeting system. The prediction system may include a prediction sensor configured to image a region of interest, and the targeting system may include a targeting sensor configured to image a portion of the region of interest. Image acquisition may include collecting a representation (e.g., an image) of the region of interest or a portion of the region of interest. In some embodiments, the prediction system may include multiple prediction sensors to enable coverage of a larger region of interest. In some embodiments, the targeting system may include multiple targeting sensors.

[0037] The target region may correspond to the overlapping area between the field of view of the targeting sensor and the field of view of the predictive sensor. Such overlap may be simultaneous or temporally separated. For example, the field of view of the predictive sensor may encompass the target region at the first time step, while the field of view of the targeting sensor may encompass the target region at the second time step but not at the first time step. Optionally, the detection system may move relative to the target region between the first and second time steps to facilitate the temporal separation of the overlapping fields of view of the predictive sensor and the targeting sensor.

[0038] In some embodiments, the predictive sensor may have a wider field of view than the targeting sensor. The predictive system may further include an object identification module for identifying target objects in the predictive image or representation collected by the predictive sensor. The object identification module can distinguish target objects in the predictive image from other objects.

[0039] The prediction module can determine the predicted location of the target object and transmit that predicted location to the targeting system.

[0040] The targeting system can direct a targeting sensor to a desired portion of a target area predicted to contain an object, based on a predicted position received from a prediction system. The targeting system may include an object matching module for determining whether a target object identified by the prediction system is present in a target image or representation collected by the targeting sensor. If the object matching module determines that the target object is present in the target image, the targeting module can determine a target position for the target object. The target position for an object may be closer to the object's actual position than its predicted position. In some embodiments, the targeting module can use the target position for an object to direct an instrument towards the object. In some embodiments, the instrument may perform an action on the object or manipulate the object.

[0041] The detection systems disclosed herein can be used to target objects on surfaces such as ground, floors, walls, fields, lawns, roads, embankments, stakes, or depressions. In some embodiments, the surface may be a non-flat surface such as uneven ground, uneven terrain, or a rough-textured floor. For example, the surface may be uneven ground in a construction site, field, or mine tunnel, or the surface may be uneven terrain including fields, roads, forests, hills, mountains, houses, or buildings. The detection systems described herein can locate objects on non-flat surfaces over a wider area more accurately, faster, or over a wider area than single-sensor systems or systems lacking an object matching module.

[0042] Alternatively, or in addition, the detection system may be used to target objects that can be separated from the surface on which the object is placed, such as the top of a tree away from its point of contact with the ground, and / or objects that can be located relative to the surface, for example, in the air or on the Earth's surface. Furthermore, the detection system may be used to target objects that can move relative to the surface, such as vehicles, animals, people, or flying objects.

[0043] Optical control system The methods described herein can be implemented by an optical control system, such as a laser optical system, to target a target object. For example, an optical system can be used to target a target object identified in an image or representation collected by a first sensor, such as a predictive sensor, and to locate the same object in an image or representation collected by a second sensor, such as a targeting sensor. In some embodiments, the first sensor is a predictive camera and the second sensor is a targeting camera. Targeting an object may include precisely locating the object using a targeting sensor and targeting the object with an instrument.

[0044] This specification describes optical control systems for directing a beam, such as a light beam, to a target location on a surface, such as the location of an object. Illustrative systems for object identification and point-to-point targeting are described with reference to Figures 1-7. In the illustrated embodiments, the instrument is a laser. However, other instruments, including but not limited to gripping instruments, spraying instruments, planting instruments, harvesting instruments, pollinating instruments, marking instruments, spraying instruments, or depositing instruments, are also within the scope of this disclosure.

[0045] Figure 1 shows an isometric view of one embodiment of the optical control system 100 disclosed herein. The emitter 101 is configured to direct a beam along an optical path, for example, a laser path 102. In some embodiments, the beam includes, for example, light, radio waves, microwaves, or electromagnetic radiation such as X-rays. In some embodiments, the light is visible light, infrared light, or ultraviolet light. The beam may be coherent. In one embodiment, the emitter is a laser, such as an infrared laser.

[0046] In some embodiments, the emitter emits a beam having a wavelength of approximately 1 m, approximately 100 mm, approximately 10 mm, approximately 1 mm, approximately 100 μm, approximately 10 μm, approximately 1.5 μm, approximately 1 μm, approximately 900 nm, approximately 800 nm, approximately 700 nm, approximately 600 nm, approximately 500 nm, approximately 400 nm, approximately 300 nm, approximately 100 nm, approximately 10 nm, or approximately 1 nm. In some embodiments, the emitter emits a beam having wavelengths of approximately 1 m to 100 mm, 100 mm to 10 mm, 10 mm to 1 mm, 1 mm to 100 μm, 100 μm to 10 μm, 10 μm to 1.5 μm, 1.5 μm to 1 μm, 1 μm to 900 nm, 900 nm to 800 nm, 800 nm to 700 nm, 700 nm to 600 nm, 600 nm to 500 nm, 500 nm to 400 nm, 400 nm to 300 nm, 300 nm to 100 nm, 100 nm to 10 nm, or 10 nm to 1 nm.

[0047] In some embodiments, the emitter may emit electromagnetic radiation of up to 10 mW, up to 100 mW, up to 1 W, up to 10 W, up to 100 W, up to 1 kW, or up to 10 kW. In some embodiments, the emitter may emit electromagnetic radiation of 10 mW to 100 mW, 100 mW to 1 W, 1 W to 10 W, 10 W to 100 W, 100 W to 1 kW, or 1 kW to 10 kW.

[0048] One or more optical elements may be positioned within the beam path. The optical elements may comprise one or more of a beam coupler 103, a first reflector 105, and a second reflector 106. These elements may be arranged in the direction of the beam path in the order of beam coupler 103, then first reflector 105, then second reflector 106.

[0049] In another example, one or both of the first reflector 105 or the second reflector 106 may be configured before the beam coupler 103 in the direction of the beam path. In yet another example, the optical elements may be configured in the order of beam coupler 103 followed by the first reflector 105 in the direction of the beam path. In yet another example, one or both of the first reflector 105 or the second reflector 106 may be configured before the beam coupler 103 in the direction of the beam path. Any number of additional reflectors can be positioned in the beam path.

[0050] The beam coupler 103 is sometimes called a beam coupling element. In some embodiments, the beam coupler 103 may be a zinc selenide (ZnSe), zinc sulfide (ZnS), or germanium (Ge) beam coupler. For example, the beam coupler 103 may be configured to transmit infrared light and reflect visible light. In some embodiments, the beam coupler 103 may be dichroic. In some embodiments, the beam coupler 103 may be configured to allow electromagnetic radiation having wavelengths longer than the cutoff wavelength to pass through and to reflect electromagnetic radiation having wavelengths shorter than the cutoff wavelength. In some embodiments, the beam coupler may be configured to allow electromagnetic radiation having wavelengths shorter than the cutoff wavelength to pass through and to reflect electromagnetic radiation having wavelengths longer than the cutoff wavelength.

[0051] In some embodiments, the cutoff wavelength may be approximately 1 m, approximately 100 mm, approximately 10 mm, approximately 1 mm, approximately 100 μm, approximately 10 μm, approximately 1.5 μm, approximately 1 μm, approximately 900 nm, approximately 800 nm, approximately 700 nm, approximately 600 nm, approximately 500 nm, approximately 400 nm, approximately 300 nm, approximately 100 nm, approximately 10 nm, or approximately 1 nm. In some embodiments, the cutoff wavelength may be approximately 1 m to 100 mm, approximately 100 mm to 10 mm, approximately 10 mm to 1 mm, approximately 1 mm to 100 μm, approximately 100 μm to 10 μm, approximately 10 μm to 1.5 μm, approximately 1.5 μm to 1 μm, approximately 1 μm to 900 nm, approximately 900 nm to 800 nm, approximately 800 nm to 700 nm, approximately 700 nm to 600 nm, approximately 600 nm to 500 nm, approximately 500 nm to 400 nm, approximately 400 nm to 300 nm, approximately 300 nm to 100 nm, approximately 100 nm to 10 nm, or approximately 10 nm to 1 nm. In other embodiments, the beam coupler may be a polarizing beam splitter, a long-pass filter, a short-pass filter, or a band-pass filter.

[0052] The optical control system of this disclosure may further include a lens positioned in the optical path. In some embodiments, the lens may be a focusing lens, which may be positioned to focus a beam, scattered light, or both. For example, the focusing lens may be positioned in the visible light path to focus scattered light onto a targeting camera. In some embodiments, the lens may be a defocusing lens, which may be positioned to defocus a beam, scattered light, or both. In some embodiments, the lens may be a collimating lens, which may be positioned to sight a beam, scattered light, or both. In some embodiments, two or more lenses may be positioned in the optical path. For example, two lenses may be positioned in series in the optical path to expand or narrow the beam.

[0053] The position and orientation of one or both of the first reflector element 105 and the second reflector element 106 can be controlled by an actuator. In some embodiments, the actuator may be a motor, solenoid, galvanometer, or servo. For example, the position of the first reflector element 105 may be controlled by a first actuator, and the position and orientation of the second reflector element 106 may be controlled by a second actuator. In some embodiments, a single reflector element may be controlled by multiple actuators. For example, the first reflector element 105 may be controlled by a first actuator along a first axis and a second actuator along a second axis. In some embodiments, a single actuator can control reflectors along multiple axes.

[0054] An actuator can change the angle of incidence of a beam striking a reflector by changing the position of the reflector by rotating the reflector. Changing the angle of incidence causes translational motion of the position where the beam strikes the surface. In some embodiments, the angle of incidence can be adjusted so that the position where the beam strikes the surface is maintained while the optical system moves relative to the surface. In some embodiments, a first actuator rotates a first reflector around a first rotation axis, thereby translating the position where the beam strikes the surface along a first translation axis, and a second actuator rotates a second reflector around a second rotation axis, thereby translating the position where the beam strikes the surface along a second translation axis. In some embodiments, the first and second actuators rotate the first reflector around a first and second rotation axis, respectively, thereby translating the position where the beam strikes the surface of the first reflector along a first and second translation axis. For example, a single reflector can be controlled by a first and a second actuator, and a single reflector controlled by two actuators can be used to translate the position where the beam strikes the surface along a first and a second translation axis.

[0055] The first and second translation axes may be orthogonal. The coverage area on the surface may be defined by the maximum translational motion along the first translation axis and the maximum translational motion along the second translation axis. One or both of the first and second actuators may be servo-controlled, piezoelectric, piezo-inertial, stepping motor-controlled, galvanometer-driven, linear actuator-controlled, or any combination thereof. One or both of the first and second reflectors may be mirrors, e.g., dichroic mirrors or dielectric mirrors, prisms, beam splitters, or any combination thereof. In some embodiments, one or both of the first and second reflectors may be any element capable of deflecting the beam.

[0056] Figure 2 shows a top view of one embodiment of the optical control system 100 shown in Figure 1. As seen in Figure 1, the targeting camera 104 can be positioned to capture light, such as visible light, traveling along the visible light path 152 in the opposite direction to the beam path, such as the laser path 102. The light may be scattered by a surface, such as a surface having a target object, or by an object, such as a target object, and travel along the visible light path 152 toward the targeting camera 104. In some embodiments, the targeting camera 104 is positioned to capture light reflected from the beam coupler 103. In other embodiments, the targeting camera 104 is positioned to capture light transmitted through the beam coupler 103. By capturing such light, the targeting camera 104 can be configured to image a target field of view on a surface. The targeting camera 104 may be coupled to the beam coupler 103, or it may be coupled to a support structure supporting the beam coupler 103. In one embodiment, the targeting camera 104 does not move relative to the beam coupler 103 so that it maintains a fixed position relative to the beam coupler 103.

[0057] Figure 3 shows a cross-sectional view of one embodiment of an optical control device disclosed herein. Figure 3 shows a mechanism for preventing the accumulation of dust and debris on the optical elements of the optical control device 100 shown in Figures 1 and 2. In some embodiments, the optical elements may be provided with hard stops 351 on mirrors to prevent the beam from striking areas of the optical control device outside a predetermined boundary on the surface. One or both of the optical elements, such as the beam coupling element 103 and reflective elements such as the first reflective element 105 and the second reflective element 106, may be protected by a housing. The optical elements may be surrounded by a housing. In some embodiments, the housing is sealed to prevent dust, debris, water, or any combination thereof from coming into contact with the optical elements.

[0058] The housing may include a laser-proof window 107, as shown in Figure 3. In some embodiments, the laser-proof window 107 is positioned to intersect the beam after a second reflector 106 in the beam path, such as the laser path 102, or after a first reflector 105 in the beam path. In some embodiments, the laser-proof window 107 is the last element in the beam path. The laser-proof window 107 can prevent dust, debris, water, or any combination thereof from reaching the optical element. In some embodiments, the laser-proof window 107 includes a material that is substantially transparent to electromagnetic radiation such as light. For example, the laser-proof window 107 may include glass, quartz, quartz glass, zinc selenide, a transparent polymer, or a combination thereof.

[0059] The housing may further include a self-cleaning device configured to prevent dust or debris from accumulating on the surface of the laser evacuation window 107, or to remove dust or debris that has accumulated on the surface of the laser evacuation window 107. In some embodiments, the self-cleaning device includes an aperture 352 within the outer surface of the housing, configured to release clean air into an airflow 353. The clean airflow 353 can prevent debris from damaging the laser evacuation window 107. In some embodiments, the clean air may be filtered. The aperture 352 may be configured to direct the airflow 353 to the outer surface of the evacuation window. The aperture 352 may be configured to direct the clean air across the surface of the laser evacuation window 107. In some embodiments, the housing is configured to guide the clean airflow 353 without obstructing the beam path 102. For example, the housing may include an opening 354 after the laser evacuation window 107 in the beam path, having a gap through which the beam can pass without obstruction. In some embodiments, the opening comprises a wall facing the aperture 352. The wall may be configured to control the direction of the airflow 353 and reduce turbulence without obstructing the beam. The opening may surround the laser evasion window 107 and the beam path 102, and in the direction of the beam path 102, the opening may be configured to narrow as it approaches the laser evasion window 107 and widen as it moves away from the laser evasion window 107. In some embodiments, the opening has smooth corners 355 to allow the passage of clean air while preventing turbulence.

[0060] After exiting the optical control system 100, the beam can be directed along the beam path 102 towards a surface, as shown in Figures 4A and 4B. In some embodiments, the surface includes a target object, such as weeds. Rotational motion of one or both of the reflectors 105 and 106, as shown in Figure 2, can produce a laser sweep along the first translation axis 401 and a laser sweep along the second translation axis 402, as shown in views 400 and 450 of Figures 4A and 4B, respectively. Rotational motion of one or both of the reflectors 105 and 106 can control the position where the beam strikes the surface. For example, rotational motion of one or both of the reflectors 105 and 106 can move the position where the beam strikes the surface to the position of the target object on the surface. In some embodiments, the beam is configured to damage the target object. For example, the beam may contain electromagnetic radiation, and the beam may irradiate an object. In another example, the beam may contain infrared light, and the beam may incinerate an object. In some embodiments, one or both of the reflecting elements can be rotated to surround an object, so that the beam scans the area containing the object.

[0061] A predictive camera or predictive sensor can work in conjunction with an optical control system, such as an optical control system 100, to identify and locate a target object. The predictive camera may have a field of view that surrounds the coverage area of ​​the optical control system, which is covered by amiable laser sweeps 401 and 402. The predictive camera may be configured to capture an image or representation of the region including the coverage area in order to identify and select a target object. The selected object can be assigned to the optical control system. In some embodiments, the field of view of the predictive camera and the coverage area of ​​the optical control system may be temporally separated such that the field of view of the predictive camera surrounds the target at a first time step and the coverage area of ​​the optical control system surrounds the target at a second time step. Optionally, the predictive camera, the optical control system, or both may move relative to the target between the first and second time steps.

[0062] In some embodiments, multiple optical control systems can be combined to increase the coverage area on a surface. Figure 5 shows a composite system 500 comprising multiple optical control systems 100. The multiple optical control systems are configured such that the laser sweep along the translation axis 402 of each optical control system overlaps with the laser sweep along the translation axis of an adjacent optical control system. The combined laser sweeps 401 and 402 define a coverage area 503 that can be reached by at least one beam from multiple beams from the multiple optical control systems. The predictive camera 501 may be positioned so that the field of view 502 of the predictive camera completely encloses the coverage area 503. In some embodiments, the detection system may comprise two or more predictive cameras, each having a field of view. The fields of view of the predictive cameras can be combined to form a predictive field of view that completely encloses the coverage area. In some embodiments, the predictive field of view may not completely enclose the coverage area at a single point in time, but may enclose the coverage area over two or more point in time (e.g., image frames). Optionally, one or more predictive cameras may be moved relative to the coverage area over two or more points in time to enable temporal coverage of the coverage area. The predictive cameras or predictive sensors may be configured to capture an image or representation of the region containing the coverage area 503 in order to identify and select an object to target. The selected object may be assigned to one of a plurality of optical control systems based on the object's position and the area covered by the laser sweeps 401 and 402 of the individual optical control systems.

[0063] As shown in view 600 of Figure 6 and view 700 of Figure 7, multiple optical control systems can be configured on the vehicle 601. For example, the vehicle may be an autonomous vehicle. The autonomous vehicle may be a robot. In some embodiments, the vehicle may be controlled by a human. For example, the vehicle may be driven by a human driver. In some embodiments, the vehicle may be coupled to a second vehicle driven by a human driver, and may be towed from behind or pushed by the second vehicle, for example. The vehicle may be remotely controlled by a human, for example by a remote control. In some embodiments, the vehicle may be remotely controlled via longwave signals, optical signals, satellites, or any other telecommunication method. Multiple optical control systems may be configured on the vehicle such that the coverage area overlaps with the underside, rear, front, or surrounding surfaces of the vehicle.

[0064] The vehicle 601 may be configured to travel on a surface containing multiple objects, such as a crop field containing one or more target objects, for example, multiple plants and one or more weeds. The vehicle 601 may comprise one or more of multiple wheels, a power source, a motor, a predictive camera 501, or any combination thereof. In some embodiments, the vehicle 601 has sufficient clearance above the surface to move over plants, such as crops, without damaging them. In some embodiments, the space between the inner edge of the left wheel and the inner edge of the right wheel is wide enough to pass over rows of plants without damaging them. In some embodiments, the distance between the outer edge of the left wheel and the outer edge of the right wheel is narrow enough for the vehicle to pass between two rows of plants, such as two rows of crops, without damaging them. In one embodiment, a vehicle comprising multiple wheels, multiple optical control systems, and a predictive camera can travel over rows of crops and burn or irradiate weeds by radiating one of multiple beams toward a target, such as weeds.

[0065] Autonomous weed control system The methods described herein can be implemented by an autonomous weed control system for targeting and removing weeds. For example, the autonomous weed control system may target a target weed identified in an image or representation collected by a first sensor, such as a predictive sensor, and use a second sensor, such as a targeting sensor, to locate the same weed in an image or representation collected by a second sensor. In some embodiments, the first sensor is a predictive camera and the second sensor is a targeting camera. Targeting weeds may include precisely locating the weed using the targeting sensor, targeting the weed with a laser, and removing the weed by burning it with laser light, such as infrared light. The predictive sensor may be part of a predictive module configured to determine the predicted location of a target object, and the targeting sensor may be part of a targeting module configured to refine the predicted location of the target object to determine a target location, and then target the target object at that target location using a laser. The predictive module may be configured to communicate with the targeting module to coordinate camera handoffs using point-to-point targeting, as described herein.

[0066] Prediction module The prediction modules of this disclosure may be configured to locate objects on a surface. Figure 8 shows a prediction module 810 configured to identify, assign, and target a target object. In some embodiments, a target prediction system 811 is configured to capture an image of a region of the surface using a prediction camera 501 or prediction sensor, identify a target object in the image, and determine the predicted location of the object.

[0067] The target prediction system 811 may include an object identification module configured to identify target objects and distinguish them from other objects in the predicted image. In some embodiments, the target prediction system 811 uses a machine learning model to identify and distinguish objects based on features extracted from a training dataset containing labeled images of objects. For example, the target prediction system 811 may be trained to identify weeds and distinguish weeds from other plants such as crops. In another example, the target prediction system 811 may be trained to identify debris and distinguish debris from other objects. The object identification module may be configured to identify plants and distinguish them from different plants, such as between crops and weeds. In some embodiments, the machine learning model may be a deep learning model, such as a deep learning neural network.

[0068] In some embodiments, the object recognition module includes using a discriminative machine learning model, such as a convolutional neural network. The discriminative machine learning model can be trained using many images, such as high-resolution images, of a surface with or without a target object. For example, the machine learning model can be trained using images of a field with or without weeds. Once trained, the machine learning model may be configured to identify regions in an image that contain a target object. The region may be defined by a polygon, such as a rectangle. In some embodiments, the region is a bounding box. In some embodiments, the region is a polygon mask covering the identified region. In some embodiments, the discriminative machine learning model may be trained to determine the location of a target object, such as its pixel location in a predicted image.

[0069] The camera control conversion system 812 may be configured to convert the position of an object in the predicted image to a position on a surface, or to a surface position relative to a reference frame of the detection system. For example, the camera control conversion system 812 can construct a plurality of interpolation functions that provide a conversion from the position in the predicted image to the position of one or more actuators controlling one or more reflective elements, such as the reflective elements 105 and 106 of the optical control system 100 shown in Figures 1 to 3, e.g., pan and tilt positions.

[0070] The prediction module 810 shown in Figure 8 may further include an attitude and motion correction system 813. The attitude and motion correction system 813 may include a positioning system, such as a wheel encoder or rotary encoder, an inertial measurement unit (IMU), a global positioning system (GPS), a distance measuring sensor (e.g., laser, SONAR, or RADAR), or an internal navigation system (INS). The attitude and motion correction system may utilize an inertial measurement unit (IMU) that can be directly or indirectly coupled to the prediction sensor. For example, the prediction sensor and IMU may be mounted on the vehicle. The IMU can collect readings of the IMU's motion and anything directly or indirectly coupled to the IMU, such as the prediction sensor. For example, the IMU can collect readings including three-dimensional acceleration and three-dimensional rotation information that can be used to determine the magnitude and direction of motion over time. The attitude and motion correction system may include a global positioning system (GPS). The GPS may be directly or indirectly coupled to the prediction sensor of the prediction module or the targeting sensor of the targeting module. For example, GPS can communicate with a satellite-based wireless navigation system to measure a first position of a predictive sensor at a first time point and a second position of the predictive sensor at a second time point. The attitude and motion correction system may include a wheel encoder that communicates with the vehicle's wheels. The wheel encoder can estimate speed or distance traveled based on angular frequency, rotational frequency, rotational angle, or wheel rotation speed. In some embodiments, the positioning system and the detection system may be positioned on the vehicle. Alternatively or additionally, the positioning system may be positioned on a vehicle that is spatially coupled to the detection system. For example, the positioning system may be installed on a vehicle that tows the detection system.

[0071] The attitude and motion correction system 813 may include an internal navigation system (INS). The INS may be directly or indirectly coupled to the targeting sensor. For example, the INS may include motion sensors such as accelerometers and rotation sensors such as gyroscopes to measure the position, orientation, and velocity of the targeting camera. The attitude and motion correction system 813 may or may not use an external reference to determine the change in the position of the targeting sensor. The attitude and motion correction system can determine the change in the position of the targeting sensor from a first position and a second position. In some embodiments, after the target prediction system has located the target object in the image, the attitude and motion correction system 813 determines the amount of time that has elapsed since the image was captured, and the magnitude and direction of the motion of the prediction camera that occurred during the elapsed time. The attitude and motion correction system 813 can integrate the object's position, elapsed time, and the magnitude and direction of motion to determine the adjusted position of the object on the surface.

[0072] Based on the object's position, the target assignment system 814 can assign the object to a targeting module 820. In some embodiments, the targeting module may be one of several targeting modules. The prediction module 810 can transmit the predicted position of the target object to the assigned targeting module 820. The predicted position of the object may be adjusted based on the magnitude and direction of motion over time, or its position may be within a region defined by a polygon, or both. The future predicted object position may be determined based on the magnitude and direction of motion predicted during a future period. The target assignment module 814 can assign a target to a targeting module that has a coverage area overlapping with the predicted position, the adjusted predicted position, or the future predicted position.

[0073] The prediction module 810 may include a system controller, such as a system computer having storage, random access memory (RAM), a central processing unit (CPU), and a graphics processing unit (GPU). The system computer may also include a tensor processing unit (TPU). The system computer must have sufficient RAM, storage space, CPU power, and GPU power to perform operations for detecting and identifying targets. The prediction sensor must provide an image with sufficient resolution to perform operations for detecting and identifying objects. In some embodiments, the prediction sensor may be a camera such as a charge-coupled device (CCD) camera or a complementary metal-oxide-semiconductor (CMOS) camera, a LiDAR detector, an infrared sensor, an ultraviolet sensor, an X-ray detector, or any other sensor capable of generating an image.

[0074] Targeting module The targeting module 820 of this disclosure may be configured to target an object identified by the prediction module 810. In some embodiments, the targeting module may direct an instrument towards the object to manipulate it. For example, the targeting module 820 may be configured to direct a laser beam towards a weed to burn it. In another example, the targeting module 820 may be configured to instruct a gripping tool to grasp an object. In yet another example, the targeting module may instruct a spraying tool to spray a fluid onto an object. In some embodiments, the object may be a weed, a plant, a field, debris, an obstacle, a surface area, or any other object that can be manipulated. Figure 8 shows a targeting module 820 configured to receive a predicted position of a target object from the prediction module 810 and direct a targeting camera 104 or a targeting sensor towards the predicted position. In some embodiments, the targeting module 820 may direct an instrument, such as a laser, towards the predicted position. In the embodiment shown in Figure 8, the position of the targeting sensor and the position of the instrument can be coupled. In some embodiments, multiple targeting modules 820 communicate with a prediction module 810.

[0075] The targeting module 820 includes and can communicate with an optical control system as described herein. For example, as shown for the optical control system 100 in Figures 1 to 3, the targeting module may include an emitter 101 that emits a beam along an optical path such as a laser path 102, a beam coupling element 103, a targeting camera 104, a first reflector 105 configured to deflect the beam controlled by a first actuator, and optionally a second reflector 106 configured to deflect the beam controlled by a second actuator positioned in the optical path. One or both actuators may be configured to change the deflection of the beam path by rotating one or both of the reflectors 105 or 106 about a first rotation axis and optionally a second rotation axis, thereby translating the position where the beam strikes a surface along a first translation axis and optionally along a second translation axis. In some embodiments, the first and second actuators can rotate a single reflector around a first and second rotation axis, respectively, to provide translational motion along the first and second translation axes at the point where the beam strikes the surface. In some embodiments, the first reflector 105, the second reflector 106, or both also control the direction of the targeting camera 104 or targeting sensor.

[0076] As shown in Figure 8, the target prediction system 821 can receive the predicted position of the target object from the prediction module 810 and orient the targeting camera 104 or targeting sensor to the predicted position of the object. The targeting camera 104 or targeting sensor can collect a target image of the area predicted to contain the target object. In some embodiments, the targeting module 820 includes an object matching module, which is configured to adjust the camera handoff using point-to-point targeting by determining whether the target image contains the target object identified by the prediction module. The object matching module can take into account differences in the appearance of the object in the predicted image and the target image, which may be due to differences between the prediction sensor and the targeting sensor such as sensor type, resolution, magnification, field of view or color balance and sensitivity, differences in imaging angle or position, motion of the detection system, variations in non-flat surfaces, differences in imaging frequency, or changes in the object between the time the prediction image was collected and the time the target image was collected. In some embodiments, the object matching module can take into account distortions introduced by the optical system, such as lens distortion, distortion from the ZnSe optical system, spherical aberration, or chromatic aberration.

[0077] In some embodiments, the object matching module may use an object matching machine learning module trained to identify the same object in different images, taking into account differences in the image sensor such as sensor type, resolution, magnification, field of view, or color balance and sensitivity, differences in the imaging angle or position, motion of the detection system, variations in non-flat surfaces, or differences between two images caused by changes in the object between the time the two images were collected.

[0078] If the object matching module identifies a target object in the target image, it can determine the target position of the object. The object matching module can determine an offset between the predicted position of the object and the target position of the object. Based on the offset, the camera control conversion system 822 can adjust the orientation of a targeting sensor, such as a targeting camera 104, by adjusting, for example, the position of the first reflector 105 and optionally the position of the second reflector 106. The positions of the reflectors may be controlled by actuators, as described herein. For example, the camera control conversion system 822 can convert the pixel position of the target in the target image into pan or tilt values ​​of one or both actuators corresponding to the mirror position predicted to deflect the beam to the target position. In some embodiments, the position of an instrument, such as a laser, is adjusted to point the instrument at the target position of the object. In some embodiments, the motion of the targeting sensor and the instrument are coupled. If the object matching module does not identify a target object in the target image, the camera control conversion system can adjust the position of the targeting sensor and acquire a second target image. Alternatively or additionally, if the object matching module fails to identify the target object in the target image, a different object may be selected from the predicted image, and a new predicted position may be determined. Reasons why the object matching module may fail to identify the target object in the target image may include improper motion correction or obstacles to objects in the target image.

[0079] The target position of an object can be further corrected using the attitude and motion correction system 823. The attitude and motion correction system 823 can determine the magnitude and direction of motion of a targeting camera using a positioning system such as a wheel encoder, IMU, GPS, distance sensor, or INS. In some embodiments, the magnitude and direction of motion are determined using acceleration and rotation readings from an IMU directly or indirectly coupled to the targeting sensor. For example, the targeting sensor and IMU may be mounted on a vehicle. The IMU can collect IMU motion readings and anything directly or indirectly coupled to it, such as the targeting sensor. For example, the IMU can collect readings including three-dimensional acceleration and three-dimensional rotation information that can be used to determine the magnitude and direction of motion over time. In some embodiments, the attitude and motion correction system can use a wheel encoder to determine the distance and velocity of motion of a targeting sensor, such as a targeting camera 104. In some embodiments, the attitude and motion correction system can use a GPS to determine the magnitude and direction of motion of a targeting sensor, such as a targeting camera 104. A wheel encoder can estimate speed or distance traveled based on angular frequency, rotational frequency, rotational angle, or wheel rotation speed. The speed or distance traveled can be used to determine the position of a vehicle relative to a surface, such as a vehicle directly or indirectly coupled to a targeting sensor. In some embodiments, the positioning system and detection system may be positioned on the vehicle. Alternatively or additionally, the positioning system may be positioned on a vehicle spatially coupled to the detection system. For example, the positioning system may be installed on a vehicle towing the detection system.

[0080] For example, a GPS may be mounted on the vehicle. The GPS can communicate with a satellite-based wireless navigation system to measure a first position of a targeting sensor, such as a targeting camera 104, at a first time point, and a second position of the targeting sensor at a second time point. In some embodiments, the attitude and motion correction system 823 can use an INS to determine the magnitude and direction of the targeting sensor's motion. For example, the INS can measure the position, orientation, and velocity of the targeting sensor. In some embodiments, after the target prediction system 821 has located the target object in the image, the attitude and motion correction system 823 determines the amount of time that has elapsed since the image was captured, and the magnitude and direction of the targeting sensor's motion that occurred during that time. The attitude and motion correction system 823 can integrate the object's position, elapsed time, and the magnitude and direction of its motion to determine the object's corrected target position. In some embodiments, the positioning system used by the attitude and motion correction system of the targeting module 823 is the same as the positioning system used by the attitude and motion correction system of the prediction module 813. The future target position of an object may be determined based on the predicted magnitude and direction of its motion during a future period. In some embodiments, the positioning system used by the attitude and motion correction system 823 of the targeting module 820 is different from the positioning system used by the attitude and motion correction system 823 of the prediction module.

[0081] The motor control system 824 may include a software-driven electrical component that provides signals to a first actuator and optionally a second actuator, and can control the position, orientation, or direction of a targeting sensor such as a targeting camera 104, a device such as a laser, or both. In some embodiments, the actuator can control a first reflector 105 and optionally a second reflector 106. For example, the actuator control system can transmit signals to the first and second actuators, including the pan-tilt values ​​of the actuators. The actuators may adopt signaled pan-tilt positions and move the first reflector 105 and the second reflector 106 around the first and second rotation axes to positions such that the beam emitted by the laser is deflected to the target position of the object, the corrected target position of the object, or the future target position of the object.

[0082] The targeting module 820 may include an instrument control system. In some embodiments, the instrument control system may be a laser control system 825. An instrument control system, such as the laser control system 825, may include a software-driven electrical component that can control the activation and deactivation of the instrument. Activation or deactivation may depend on the presence or absence of an object detected by the targeting camera 104. Activation or deactivation may depend on the position of the instrument relative to the position of the target object. In some embodiments, when an object is identified and located by the target prediction system, the instrument control system may activate an instrument, such as a laser emitter. In some embodiments, the instrument control system may activate an instrument when the range of the instrument, such as a beam path 102, is positioned to overlap with the position of the target object. In some embodiments, the instrument control system may activate an instrument when the range of the instrument is within a region of a surface containing an object defined by a polygon, for example, a bounding box or polygon mask covering an identified area.

[0083] The instrument control system can deactivate the instrument when an operation such as grasping, spraying, burning, or irradiating an object is performed on it, i.e., when an area containing the object is targeted by the instrument, when the object is no longer identified by the target prediction module, when a specified period of time has elapsed, or when any combination thereof occurs. For example, the laser control system 825 can deactivate the emitter when an area on a surface containing weeds is scanned by the beam, when the weeds are irradiated or burned, or when the beam has been activated for a predetermined period of time.

[0084] The prediction module and targeting module described herein may be used in combination to locate, identify, and target objects using instruments. The targeting control module may include an optical control system as described herein. The prediction module and targeting module may communicate, for example, with telecommunications or digital communications. In some embodiments, the prediction module and targeting module are coupled directly or indirectly. For example, the prediction module and targeting module may be coupled to a support structure. In some embodiments, the prediction module and targeting module are configured on a vehicle, for example, vehicle 601, as shown in Figures 6 and 7.

[0085] The targeting module may include a system computer having a system controller, such as storage, random access memory (RAM), a central processing unit (CPU), and a graphics processing unit (GPU). The system computer may also include a tensor processing unit (TPU). The system computer must have sufficient RAM, storage space, CPU power, and GPU power to perform operations for detecting and identifying targets. The targeting sensor must provide images with sufficient resolution to perform operations for matching objects with objects identified in the predicted image.

[0086] Weed targeting and weed control Figure 9 shows a process 900 of one embodiment of the devices and methods disclosed herein. The following examples are illustrative and do not limit the scope of the devices, systems, and methods described herein. This process includes identifying, assigning, matching, targeting, and weeding a field. In this example, the weeding system includes a predictive module 810 that communicates with a plurality of targeting modules 820. The predictive modules 810 and targeting modules 820 are controlled by a system controller, a computer including, for example, storage, RAM, CPU, and GPU. Each targeting module includes an optical control system 100, as shown in Figures 1 to 3. The predictive modules and targeting modules are coupled to a solid support. The solid support is positioned on a vehicle 601, as shown in Figures 6 and 7.

[0087] As shown in Figure 9, operations 920, 930, 940, 950, and 960 are repeated until the target field, e.g., a field containing crops, is completely scanned or until another endpoint is reached (910). First, the prediction module performs operation 920. The prediction camera 501 collects images of the field surface in an area around or in front of the vehicle 601. The system controller processes the images and identifies weeds in the images, for example using an object recognition machine learning model. In step 921, the prediction model predicts the location of one or more weeds identified in the images using the method described herein. In step 922, the camera control transformation system 812 transforms the pixel coordinates of the weeds in the images into ground positions. In step 923, the system controller instructs the prediction module to update the predicted position based on the motion of the vehicle 601, for example measured in 922 by a wheel encoder, IMU, rangefinder, or GPS. In step 924, each of one or more weeds is assigned to a targeting module based on the weed's ground position and the coverage area of ​​the targeting module 820.

[0088] In step 925, operations 930, 940, 950, and 960 are repeated for each targeting module. Operations 940, 950, and 960 are repeated for each weed. Targeting module 820, one of the multiple targeting modules, performs operation 940. The pan and tilt values ​​of the actuators of the mirrors that control the reflective elements that control the field of view of the targeting camera 104 may be set to point towards the predicted location of the weed based on the predicted location of the weed determined by the prediction system 810. In step 941, the targeting camera 104 captures a target image of the field, and the system controller identifies the weed in the target image. Weed identification may include matching the objects identified in the target image with the target objects identified in the prediction image using, for example, an object matching machine learning model described herein. In some embodiments, the machine learning model may be a deep learning model such as a deep learning neural network.

[0089] The targeting system can determine the target position of the weed based on the object matching module. The targeting system determines the offset between the predicted position of the weed and the target position of the weed. In step 942, the system controller converts the offset into pan and tilt values ​​for each actuator that controls each reflector in the optical control system controlled by the targeting module, and instructs the targeting camera 104 and laser to point toward the target position of the weed. In step 943, the system controller applies attitude and motion corrections to the pan and tilt values ​​of the actuators based on the vehicle's motion, measured, for example by a wheel encoder, IMU, rangefinder, or GPS, and in step 944, plans the route of the radiated beam path controlled by the actuator's pan and tilt position. When the actuator reaches the predetermined position, in step 945 the emitter is activated.

[0090] Operation 950 is repeated while the route planned in 946 is implemented. In 951, weeds are identified in images collected by the targeting camera 104, and in 952, the route plan is updated based on the observed weed locations. In 953, the system controller applies attitude and motion corrections to the actuator's pan and tilt values ​​based on the vehicle's motion measured by the wheel encoder, IMU, or GPS. In 954, the actuator moves to its position based on the updated route plan. In 960, once the planned route is complete, the emitter is deactivated.

[0091] Detection system for object identification and point-to-point targeting In some embodiments, a detection system of the Disclosure having a prediction system and a targeting system may be configured to identify and target an object using a point-to-point targeting method. The prediction system may include a prediction sensor configured to image a region of interest, and the targeting system may include a targeting sensor configured to image a portion of the region of interest. Image acquisition may include collecting a representation (e.g., an image) of the region of interest or a portion of the region of interest.

[0092] Figure 10A schematically shows a detection system 1000 that may be used in a method for identifying, locating, and precisely targeting a target object O. In some embodiments, the detection system may include an optical control system, such as the optical control system 100 shown in Figures 1 to 3, as described herein. The detection system 1000, including a prediction module 1010 and a targeting module 1050, can image a target area 1091 using a prediction sensor 1020 via line 1021. The target area 1091 may be an area of ​​surface 1090, such as the ground, floor, or field. The image may be a visible light image, an infrared image, an ultraviolet image, a LiDAR image, an X-ray image, or any other electromagnetic image. The prediction sensor 1020 may be a camera such as a charge-coupled device (CCD) camera or a complementary metal-oxide-semiconductor (CMOS) camera, a LiDAR detector, an infrared sensor, an ultraviolet sensor, an X-ray detector, or any other sensor capable of detecting electromagnetic waves.

[0093] The object identification module 1030 can receive a predicted image from the prediction sensor 1020 via line 1022. The prediction module 1010 may use the object identification module 1030 to determine the presence or absence of a target object O in the predicted image of the target region 1091 collected by the prediction sensor 1020. The object identification module 1030 can identify the target object in the predicted image and distinguish it from other objects in the predicted image. In some embodiments, the object identification module 1030 includes a discriminative machine learning model trained to identify a target object based on features extracted from labeled images used to train the discriminative machine learning model. The machine learning model may be a deep learning model, such as a deep learning neural network. In some embodiments, the object identification module 1030 may implement a heuristic model, thresholding, or classical detection algorithm to identify objects. In some embodiments, the object identification module uses spectral data to identify objects.

[0094] The identified object can be communicated to the object position module 1040 via line 1031. The object position module 1040 can determine the object predicted position 1095 of the target object O identified by the object identification module 1030. The object predicted position 1095 may be based on the position of the target object O within the target area 1091, such as a pixel position in the predicted image. In some embodiments, determining the object predicted position 1095 may include using a calibration model to convert the pixel position to a position on the surface 1090. In some embodiments, the calibration model may be a mathematical model such as a trigonometric model, a geometric model, or a spline model. The calibration model can correlate the pixel position in the predicted image to a surface position, the position or orientation of a targeting sensor, the position of an instrument, or a combination thereof. In some embodiments, the object position module 1040 can determine the object predicted position 1095 based on a predetermined calibration coefficient that correlates the pixel position of object O to a position on the surface. In some embodiments, the predicted position 1095 may take into account the motion of the detection system relative to the object between the time the predicted image is collected and the time the target image is collected.

[0095] The object position module 1040 can transmit the predicted object position 1095 to the targeting module 1050 via line 1041. In some embodiments, the targeting module 1050 is one of several targeting modules, and the targeting module 1050 may be selected based on the availability of targeting modules or the proximity of the targeting module to the predicted object position 1095.

[0096] The targeting control module 1055 of the targeting module 1050 can control the position, orientation, or direction of the targeting sensor 1060 via line 1056. In some embodiments, the targeting control module 1055 can control the position, orientation, or direction of the targeting sensor 1060 by moving an actuator that adjusts the position or orientation of the targeting sensor 1060. In some embodiments, the targeting control module 1055 can control the position, orientation, or direction of the targeting sensor 1060 by moving an actuator that adjusts the position or orientation of a reflective surface that directs electromagnetic waves to or from the targeting sensor. The targeting control module 1055 can adjust the position, orientation, or direction of the targeting sensor 1060 based on the object prediction position 1095 using a calibration model. In some embodiments, the targeting control module 1055 can adjust the position, orientation, or direction of the targeting sensor 1060 based on a predetermined calibration coefficient that correlates the predicted object position with the position of an actuator, such as the position of an actuator that moves the targeting sensor, or the position of an actuator that moves an electromagnetic wave directed to the targeting sensor or a reflective surface directed from the targeting sensor.

[0097] The targeting sensor 1060 may be adjusted by the targeting control module 1055 so that its position, orientation, or direction is toward the predicted object position 1095, and it collects a target image of the predicted region 1092, which is predicted to contain the target object O, via line 1061. The predicted region 1092 may cover a portion of the target region 1091 captured by the predicting sensor 1020.

[0098] Figure 10B shows an example of a predictive image of a weed on a surface captured using a wide field of view, for example, a target region 1091 (inset) captured using a predictive sensor and planted around the target weed, and a target image of the same weed captured using a narrower field of view, for example, a predictive region 1092 including the object predictive position 1095, or a target region 1093 including the object target position 1096. The object matching module 1065 in Figure 10A can determine the presence or absence of the target object in the target image of the predictive region 1092 or target region 1093, received from the targeting sensor 1060 via line 1063. In some embodiments, the object matching module 1065 includes an object matching machine learning model trained to detect objects in the target image and match them with objects identified in the predictive image. The object matching module 1065 can take into account differences in the appearance of an object, which may be caused by differences in sensors such as sensor type, resolution, magnification, field of view, or color balance and sensitivity, differences in imaging angle or position, motion of the detection system, variations in non-flat surfaces, or changes in the object between the time the predicted image was acquired and the time the target image was acquired.

[0099] If the object matching module 1065 fails to locate the target object in the target image, the targeting control module 1055 may adjust the position, orientation, or direction of the targeting sensor, allowing the targeting sensor to collect a second target image. This process may be repeated until the object is located in the target image. Alternatively or additionally, if the object matching module 1065 fails to locate the target object in the target image, a different object may be selected from the predicted image, and a new predicted position may be determined. Reasons why the object matching module 1065 may fail to identify the target object in the target image may include improper motion correction or obstacles to the object in the target image.

[0100] If the object matching module 1065 determines that the target object is present in the target image, the position refinement module 1070 can communicate with the object matching module via line 1066 to determine the object target position 1096 of the target object O based on the object's position in the target image. Determining the object target position may include using a calibration model to convert pixel positions in the target image to positions on the surface 1090. The calibration model can correlate pixel positions in the target image to surface positions, the position or orientation of the targeting sensor, the position of the instrument, or a combination thereof. In some embodiments, the target position 1096 may take into account the motion of the detection system relative to the object between the time the target image was acquired and the time the operation is performed.

[0101] In some embodiments, the object matching module can determine an offset based on the target position. The offset may be the offset between the current actuator position and the actuator position for orienting the targeting sensor to the object target position 1096. The offset may be the offset between the actuator position for orienting the targeting sensor to the object prediction position 1095 and the actuator position for orienting the targeting sensor to the object target position 1096. The offset may be the offset between the current actuator position and the actuator position for orienting the instrument to the object target position 1096. The offset may be the offset between the actuator position for orienting the instrument to the object prediction position 1095 and the actuator position for orienting the instrument to the object target position 1096. The offset may be the offset between the prediction position 1095 on the surface and the object target position 1096 on the surface. The offset can be determined as a function of surface coordinates, pixel position, actuator position, or a combination thereof. In some embodiments, the position refinement module can refine the target position 1096 based on the motion of the detection system and the time since the target image was acquired. For example, motion can be determined using a wheel encoder, distance sensor, IMU, or GPS.

[0102] Accordingly, the targeting control module 1055 can adjust the position, orientation, or direction of the fixture based on the target position of the object determined by the position refinement module 1070 and communicated to the targeting control module 1055 via line 1071. In some embodiments, the targeting control module 1055 can adjust the position, orientation, or direction of the fixture 1080 via line 1057 by moving an actuator that adjusts the position or orientation of the fixture 1080. In some embodiments, the targeting control module 1055 can adjust the position, orientation, or direction of the targeting sensor 1060 via line 1056, and the targeting sensor can collect an image of the target area 1093 of the surface 1090 including the object target position 1096 of the object O via line 1062. In some embodiments, the targeting control module 1055 can control the position, orientation, or direction of the instrument 1080 by moving an actuator, which adjusts the position or orientation of a reflective surface that directs radiation, such as laser radiation, from the instrument towards the object target position 1096.

[0103] In some embodiments, the movement of the instrument 1080 is coupled to the movement of the targeting sensor 1060 so that the instrument 1080 is directed to a fixed position relative to the field of view of the targeting sensor 1060. In some embodiments, the movement of the instrument 1080 is controlled by the same actuator as the movement of the targeting sensor 1060.

[0104] The instrument 1080 can perform an action on a target object by directing the instrument towards the object target position 1096 via line 1081. For example, the instrument 1080 may be a laser that emits laser light toward object O at object target position 1096. In another example, the instrument 1080 may be a gripping tool that grasps object O at object target position 1096. In yet another example, the instrument 1080 may be a spraying tool that sprays fluid toward object O at object target position 1096. In some embodiments, the instrument 1080 may be a planting tool that plants a plant at target position 1096. In some embodiments, the instrument 1080 may be a harvesting tool that harvests object O at object target position 1096. In some embodiments, the instrument 1080 may be a pollination tool that pollinates object O at object target position 1096.

[0105] Methods for object identification and point-to-point targeting The methods described herein may be used to identify and target objects using the detection systems described herein. Object identification and targeting may include coordinating handoffs between two or more sensors by identifying the object in an image or representation collected by a first sensor, e.g., a predictive sensor, and identifying the same object in an image or representation collected by a second sensor, e.g., a targeting sensor. In some embodiments, identifying the same object in an image collected by a second sensor may include identifying the object in the image and determining whether that object is the same object identified in an image collected by a first sensor. Sensor handoffs may be complicated due to differences between the predictive and targeting sensors, such as sensor type, resolution, magnification, field of view, or color balance and sensitivity, differences in imaging angle or position, motion of the detection system, variations in non-flat surfaces, or changes in the object between the time the predictive image was collected and the time the target image was collected.

[0106] Figure 11 shows an example of method 1100 for coordinating sensor handoff using point-to-point targeting to locate and target an object. This method can be implemented using a detection system described herein, for example, detection system 1000 shown in Figure 10A. In some embodiments, method 1100 may be implemented by a detection system including a prediction module having a prediction sensor and an object identification module, and a targeting module having a targeting sensor and an object matching module. For example, the method may be implemented using the detection system shown in Figure 8 or Figure 10A.

[0107] In 1110 of Figure 11, the predictive sensor collects a predictive image using, for example, a predictive module 1010 shown in Figure 10A. In 1120, the target object is identified within the predictive image, such as a target region 1091 as shown in Figure 10A. In some embodiments, the target object is identified using an object identification module, such as an object identification machine learning model. In 1130, the predicted position of the target object is determined, for example, the object predictive position 1095 shown in Figure 10A. In some embodiments, the predicted position is determined based on the pixel position of the target object in the predictive image. In some embodiments, the predicted position takes into account the motion of the detection system relative to the object between the time the predictive image is collected and the time the target image is collected. The positions of actuators that control the position, orientation, or direction of the targeting sensor may be determined based on the predicted position. For example, the pan or tilt values ​​of one or more actuators that control one or more mirrors that reflect light toward the targeting sensor may be determined based on the predicted position. In another example, the position values ​​of one or more actuators that move, tilt, or rotate the targeting sensor may be determined based on the predicted position. In some embodiments, the actuator may be a motor, solenoid, galvanometer, or servo. Converting a predicted position to an actuator position may include applying a calibration coefficient to convert the predicted position or the position of an object in the predicted image to an actuator position.

[0108] In 1140, the targeting sensor is directed to the predicted location of the target object, and in 1150, the targeting sensor collects an image of the predicted location using, for example, the targeting module 1050 in Figure 10A. In 1160, the target object is identified within a target image, such as the predicted region 1092 or the target region 1093. The target object can be identified by matching the object identified in the target image with the target object identified in the predicted image of the target region 1091. In some embodiments, object matching is performed by an object matching module, such as an object matching machine learning model. To match the object identified in the target image with the target object identified in the predicted image, the object matching module may take into account differences between the predicting sensor and the targeting sensor, such as sensor type, resolution, magnification, field of view, or color balance and sensitivity, differences in imaging angle or position, motion of the detection system, variations in non-flat surfaces, or changes in the object between the time the predicted image was collected and the time the target image was collected. If the target object is not identified in the target image, the position, orientation, or direction of the targeting sensor may be adjusted, and a second target image may be collected. In some embodiments, this process can be repeated until the target object is identified in the target image.

[0109] Once the target object is identified in the target image, the 1170 determines the target position of the target object. In some embodiments, the target position of the target object may be determined based on the pixel position of the object in the target image. In some embodiments, the target position may be determined from the position of the object identified by the object matching module. In some embodiments, determining the target position may include using calibration coefficients to convert the pixel position in the target image to the position of the object on the surface. In some embodiments, the target position may take into account the motion of the detection system relative to the object between the time the target image was collected and the time the operation is performed.

[0110] In 1180, an offset is determined between the target position and the predicted position of the object. In some embodiments, the position of the targeting sensor may be adjusted based on the determined offset. In some embodiments, the instrument is positioned based on the target position of the object or based on the offset. In some embodiments, the position of the instrument is fixed relative to the targeting sensor. In some embodiments, the movement of the targeting sensor and the instrument is coordinated. The instrument may be directed to the target position of the object in order to perform an action on the object or to manipulate the object. For example, the instrument may be a laser that emits laser light toward the object. In another example, the instrument may be a gripping tool that grasps the object. In another example, the instrument may be a spraying tool that sprays a fluid toward the object. In another example, the instrument may be a pollinating tool that pollinates the object. In some embodiments, the instrument may be a planting tool that plants plants toward the object's location, a picking tool that picks up the object, an inspection tool that inspects the object, a soil sampling tool that samples soil toward the object's location, an manipulation tool that manipulates the object, a repair tool that repairs the object, or a welding tool that welds the object.

[0111] Machine learning models for object identification As described herein, methods for locating and targeting objects may include identifying objects in images collected by sensors, such as predictive sensors. In some embodiments, object identification may be performed using an object identification module, such as an identification machine learning model. In some embodiments, the machine learning model may be a deep learning model, such as a deep learning neural network. The object identification module is used by a predictive system to identify a target object in a predictive image collected by a predictive sensor and to distinguish that target object from other objects. For example, the object identification module may identify weeds in a predictive image and to distinguish those weeds from other plants, such as crops, in the predictive image. In another example, the object identification module may identify debris fragments in a predictive image and to distinguish that debris from other items in the predictive image.

[0112] Object recognition machine learning models can be trained using images of objects labeled by human users. These images may contain a variety of objects that correspond to the target object or other objects that are not the target object. For example, an image may be of different types of plants, which can be identified by a human user as weeds or non-weeds, and the image can be labeled accordingly. For instance, a human user might mark a plant as a crop such as an onion, strawberry, corn, or potato, or as a weed such as a dandelion, morning glory, thistle, ryegrass, or shepherd's purse. The recognition machine learning model can be trained using labeled images and can extract features from a variety of objects. For example, a deep learning model can extract features from various types of plants. In some embodiments, the training images are high-resolution images that facilitate feature extraction by the deep learning model. To validate the model, the object recognition module can identify objects in unlabeled images that were not used to train the model. Identifications from the model may be compared to identifications from human users.

[0113] Machine learning models for object matching As described herein, a method for locating and targeting an object may include matching an object identified in an image collected by a first sensor, such as a predictive sensor, with an object identified in an image collected by a second sensor, such as a targeting sensor. The first and second sensors may have different characteristics that make it difficult to identify the same object in images collected by the first and second sensors. For example, the first and second sensors may have different sensor types, resolutions, magnifications, fields of view, or color balances and sensitivities, or the sensors may be positioned separately relative to the object.

[0114] Differences between the first and second sensors may cause the same object to appear differently in images collected by the first and second sensors. Other factors that may cause the same object to appear differently in images collected by the first and second sensors may include variations in non-flat surfaces, sensor motion, or changes in the object between the time the image was collected by the first sensor and the time the image was collected by the second sensor. An object-matching machine learning model may be used to match objects identified in images collected by the first sensor, such as a predictive sensor, with objects identified in images collected by the second sensor, taking into account the differences in the appearance of objects between images collected by the first and second sensors. In some embodiments, the machine learning model may be a deep learning model, such as a deep learning neural network.

[0115] A matching machine learning model can be trained using images of objects collected by a predictive sensor and images of the same object, different objects, or no objects collected by a targeting sensor. In some embodiments, images containing the same object are collected from the predictive and targeting sensors within approximately 1 second, 10 seconds, 30 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes, 45 minutes, 1 hour, 2 hours, 6 hours, 12 hours, 24 hours, or 48 hours between each other to minimize changes in the object between the time the predictive image is collected and the time the target image is collected. In some embodiments, a human user is provided with images of objects collected by the predictive sensor, and the human user can manually identify the same object in the images collected by the targeting sensor and label the target image accordingly.

[0116] A matching machine learning model can be trained using labeled target images in combination with predicted images of the object. The matching machine learning model can be trained to identify the same object captured by sensors with varying resolutions, fields of view, and color sensitivities, collected from various angles and distances. To validate the model, the object matching module is provided with images of the object collected by predictive sensors not used in training, and the object matching module can identify the same object in unlabeled images collected by targeting sensors. Identification from the model may be compared to identification from a human user.

[0117] The accuracy of an object matching model can be evaluated by measuring the percentage of direct hits, the percentage of hits within a given distance from the object, and the percentage of misses when the object is outside a given distance or does not exist. Sensitivity and specificity are evaluated by determining the frequency at which an object is identified when it does not exist (false positive rate) and the frequency at which an object is not identified when it does exist (false negative rate).

[0118] Adjustment of point-to-point targeting and sensor handoff The object identification and object matching methods described herein may be implemented by a system comprising two or more image sensors to coordinate sensor handoffs and perform point-to-point targeting of objects. An object matching module can be used to identify and locate objects in images collected by a first sensor, such as a predictive sensor, and to locate the same objects more precisely in images collected by a second sensor, such as a targeting sensor. In some embodiments, objects are manually identified in images collected by the first sensor. In some embodiments, objects are identified in images collected by the first sensor using an object identification module that implements software for identifying objects. In some embodiments, the object identification module may implement a discrimination machine learning model. In some embodiments, the object identification module may implement a heuristic model, thresholding, or classical detection algorithm. In some embodiments, the object identification module uses spectral data to identify objects. In some embodiments, objects are identified in images collected by the first sensor based on their location, such as pixel position in the image.

[0119] The second sensor may have a different position, orientation, distance, or resolution than the first sensor. By using an object matching module to match objects in images collected by the second sensor with objects identified in images collected by the first sensor, handoff between the first and second sensors can be facilitated, enabling the localization of the same object by both sensors. The first image sensor may have a wider field of view than the second sensor, allowing for more precise localization of objects in images collected by the second sensor than in images collected by the first sensor. The system may have 3, 4, 5, 6, 7, 8, 9, 10, or more sensors, enabling the localization of 2, 3, 4, 5, 6, 7, 8, 9, 10, or more objects in one or more images collected by the first sensor. For example, the system may include 2, 3, 4, 5, 6, 7, 8, 9, 10, or more targeting sensors. Each of the 2, 3, 4, 5, 6, 7, 8, 9, 10 or more objects can be matched and localized in images collected by the second, third, fourth, fifth, sixth, seventh, eighth, ninth, tenth or more sensors, enabling simultaneous targeting of multiple objects by the 3, 4, 5, 6, 7, 8, 9, 10 or more sensors.

[0120] A first sensor, such as a predictive sensor, can image an area or region of a surface. The first sensor may have higher resolution, a wider field of view, or a different position, distance, or orientation than a second sensor, such as a targeting sensor. In some embodiments, the first sensor may be positioned on a vehicle, such as a moving, flying, or orbiting vehicle. For example, the first sensor may be positioned on an automobile, remotely controlled vehicle, autonomous vehicle, self-driving vehicle, autonomous weeder, tractor, combine harvester, harvester, planter, sprayer, agricultural vehicle, construction vehicle, bulldozer, backhoe, crane, airplane, helicopter, remotely operated aircraft, drone, or satellite. In some embodiments, the second sensor may be positioned on the same vehicle as the first sensor, for example, the same automobile, remotely controlled vehicle, autonomous vehicle, self-driving vehicle, airplane, helicopter, remotely operated aircraft, drone, or satellite.

[0121] In some embodiments, the second sensor may be positioned on a second vehicle such as an automobile, remotely controlled vehicle, autonomous vehicle, self-driving vehicle, airplane, helicopter, remotely controlled aircraft, drone, or satellite. The second vehicle equipped with the second sensor may be at a known or predetermined position relative to the first vehicle equipped with the first sensor. For example, the first and second vehicles may be at fixed positions relative to each other. In another example, the relative positions between the first and second vehicles may be determined by a Global Positioning System (GPS), radar, LiDAR, sonar, or Long Range Radio Navigation (LORAN). The second sensor may be positioned on the same type of vehicle or a different type of vehicle. For example, the first sensor may be positioned on an orbiting vehicle such as a satellite, and the second sensor may be positioned on a flying sensor such as an airplane, helicopter, remotely controlled aircraft, or drone. In another example, the first sensor could be positioned on a flight sensor such as an airplane, helicopter, remotely controlled aircraft, or drone, while the second sensor could be positioned on a moving vehicle such as a car, tractor, remotely controlled vehicle, autonomous vehicle, or self-driving car.

[0122] An object identification module may be used to identify a target object located within an image collected by a first sensor. In some embodiments, the object identification module can distinguish the target object from other objects. A prediction system communicating with the first sensor can select a target object from the identified target objects. The prediction system can determine the predicted location of the object. In some embodiments, the predicted location is determined using empirically determined calibration coefficients that adjust the pixel location in the image to a surface location, vehicle location, location, orientation or direction of the first sensor, location, orientation or direction of the second sensor, or a combination thereof. The predicted location of the object may be transmitted to a targeting system communicating with the second sensor.

[0123] The targeting system can adjust the position, orientation, or direction of the second sensor, or the position, orientation, or direction of the vehicle carrying the second sensor, so that the second sensor is directed towards the predicted location of an object. For example, the targeting system may adjust the position of one or more actuators that control the second sensor or one or more mirrors that reflect light toward the second sensor, or the targeting sensor may move the vehicle to adjust the position, orientation, or direction of the second sensor. In some embodiments, the motion of the second sensor may be linked to the motion of an instrument, such as a laser, so that the motion of the second sensor and the motion of the instrument are correlated. In some embodiments, the instrument may be controlled independently of the second sensor. In some embodiments, the position, orientation, or direction of the instrument relative to the position, orientation, or direction of the second sensor is known.

[0124] The second sensor can image an area or region of the surface at the predicted location of the target object. In some embodiments, as shown in Figure 10B, the region imaged by the second sensor may be smaller than the region imaged by the first sensor. The inset in Figure 10B shows an image of a surface region including the target object, imaged by the prediction sensor and planted around the target object. The enlarged image in Figure 10B shows an image of a surface region imaged by the targeting sensor, which includes the same target object as determined by an object-matching machine learning model.

[0125] The targeting system can use an object matching module, such as an object matching machine learning model, to determine whether a selected object is present in the image collected by the second sensor. If the object is present in the image, the object matching module can determine the object's target location. The target location of the object may be closer to the object's actual location than the predicted location determined from the image collected by the first sensor. In some embodiments, the target location is determined based on a calibration coefficient that correlates the pixel location of the object in the image collected by the second sensor to a surface location, vehicle location, the location, orientation or direction of the second sensor, or a combination thereof.

[0126] The targeting system can determine the offset between the predicted position of an object and the target position of the object. In some embodiments, the offset may be the offset between the predicted surface position of the object and the target surface position. In some embodiments, the offset may be the offset between the position, direction, or orientation of the second sensor at the predicted position of the object and the position, direction, or orientation of the second sensor at the target position of the object. In some embodiments, the offset may be the offset between the position, direction, or orientation of the second sensor at the predicted position of the object and the position, direction, or orientation of the instrument at the target position of the object. In some embodiments, the offset may be the offset between the position, direction, or orientation of the instrument at the predicted position of the object and the position, direction, or orientation of the instrument at the target position of the object. In some embodiments, the offset may be the offset between the position, direction, or orientation of the instrument and the position, direction, or orientation of the instrument directed at the target position of the object. In some embodiments, the offset may be the offset between the position, direction, or orientation of the second sensor at the predicted position of the object and the position of the vehicle at the target position of the object. In some embodiments, the offset may be the offset between the position of the vehicle at the predicted position of the object and the position of the vehicle at the target position of the object.

[0127] The position, orientation, or direction of a second sensor, instrument, vehicle, or combination thereof may be adjusted based on an offset toward the object's target position. This instrument may be used to manipulate the object at its target position. For example, the instrument may be a laser that emits laser light toward the object. In another example, the instrument may be a gripping tool that grasps the object. In yet another example, the instrument may be a spraying tool that sprays a fluid onto the object. In yet another example, the instrument may be a pollinating tool that pollinates the object.

[0128] For example, using point-to-point targeting to coordinate sensor handoff by locating a target object in an image collected by a first sensor using an object matching module and identifying the same object in an image collected by a second sensor may improve the accuracy of object targeting compared to methods that do not use point-to-point targeting. In some embodiments, using point-to-point targeting to coordinate sensor handoff can improve the accuracy of targeting objects positioned on or near uneven or irregular surfaces. In some embodiments, point-to-point targeting may enable the coordination of sensor handoff between sensors of different types or specifications.

[0129] For example, point-to-point targeting may enable sensor handoff between a LiDAR detector and a camera, between a high-resolution camera and a low-resolution camera, or between a color camera and a monochrome camera. In some embodiments, point-to-point targeting can be used to locate a target object using a higher-quality sensor, such as one with higher resolution, better color reproduction, or a wider field of view, and then target that object using a lower-quality sensor, thereby improving targeting compared to targeting the object with only the lower-quality sensor. In some embodiments, point-to-point targeting can be used to improve targeting of moving objects or objects from moving vehicles by taking into account the unknown time-based movement of the object.

[0130] A detection system implementing the point-to-point targeting method of this disclosure can be used to target an object with improved accuracy than a system that does not use point-to-point targeting. In some embodiments, the implementation of the detection system using point-to-point targeting is approximately 1 mm, 1.5 mm, 2 mm, 2.5 mm, 3 mm, 4 mm, 5 mm, 6 mm, 7 mm, 8 mm, 9 mm, 10 mm, 15 mm, 20 mm, 25 mm, 30 mm, 40 mm, 50 mm, 60 mm, 70 mm, 80 mm, and 80 mm from the actual position of the target object. You may target objects within approximately 90mm, 100mm, 150mm, 200mm, 300mm, 400mm, 500mm, 1m, 1.5m, 2m, 2.5m, 3m, 4m, 5m, 6m, 7m, 8m, 9m, 10m, 15m, 20m, 25m, 30m, 40m, 50m, 60m, 70m, 80m, 90m, or 100m.

[0131] In some embodiments, the target position of the target object, determined using point-to-point targeting, is approximately 1 mm, 1.5 mm, 2 mm, 2.5 mm, 3 mm, 4 mm, 5 mm, 6 mm, 7 mm, 8 mm, 9 mm, 10 mm, 15 mm, 20 mm, 25 mm, 30 mm, 40 mm, 50 mm, and 60 mm from the actual position of the target object. It may be approximately 70mm, 80mm, 90mm, 100mm, 150mm, 200mm, 300mm, 400mm, 500mm, 1m, 1.5m, 2m, 2.5m, 3m, 4m, 5m, 6m, 7m, 8m, 9m, 10m, 15m, 20m, 25m, 30m, 40m, 50m, 60m, 70m, 80m, 90m, or within approximately 100m.

[0132] In some embodiments, the predicted and target positions of the target object, determined using point-to-point targeting, are approximately 1 mm, 1.5 mm, 2 mm, 2.5 mm, 3 mm, 4 mm, 5 mm, 6 mm, 7 mm, 8 mm, 9 mm, 10 mm, 15 mm, 20 mm, 25 mm, 30 mm, 40 mm, 50 mm, and 60 mm from the actual position of the target object. mm, approximately 70 mm, approximately 80 mm, approximately 90 mm, approximately 100 mm, approximately 150 mm, approximately 200 mm, approximately 300 mm, approximately 400 mm, approximately 500 mm, approximately 1 m, approximately 1.5 m, approximately 2 m, approximately 2.5 m, approximately 3 m, approximately 4 m, approximately 5 m, approximately 6 m, approximately 7 m, approximately 8 m, approximately 9 m, approximately 10 m, approximately 15 m, approximately 20 m, approximately 25 m, approximately 30 m, approximately 40 m, approximately 50 m, approximately 60 m, approximately 70 m, approximately 80 m, approximately 90 m, or within approximately 100 m.

[0133] Methods for calibrating optical systems The systems and methods disclosed herein may further include one or more calibration steps for calibrating the position, orientation, or motion of a predictive sensor, targeting sensor, instrument, or a combination thereof. In some embodiments, calibration can be used to correlate a position, such as a pixel position in an image or representation collected by a predictive sensor or targeting sensor, to a position on a surface. In some embodiments, calibration may be used to correlate a position in an image or representation collected by a first sensor, such as a predictive sensor, to a position in an image or representation collected by a second sensor, such as a targeting sensor. In some embodiments, calibration can be used to correlate a position in an image collected by a predictive sensor to the position, orientation, or motion of a targeting sensor. In some embodiments, calibration can be used to correlate a position in an image collected by a targeting sensor to the position, orientation, or motion of a targeting sensor, instrument, or both. In some embodiments, calibration can be used to correlate the position, orientation, or motion of an instrument to a position on a surface. Calibration methods may use empirical measurements to correlate position, orientation, localization, or motion. Alternatively or additionally, the calibration method may use a mathematical model to correlate position, orientation, localization, or motion.

[0134] Calibration of predictive and targeting sensors The system of this disclosure can be calibrated to correlate the position or orientation of a targeting sensor with a region of a predicted image collected by a predictive sensor. The correlation between the orientation of the targeting sensor and the region of the predictive camera can be performed using a calibration surface having distinguishable features or distinct fiducial markers, such as the calibration grid shown in Figure 12. In some embodiments, the calibration surface may be a surface on which markings are made. In some embodiments, the calibration surface may be a surface on which distinguishable objects are made. In some embodiments, the calibration may be a surface with variations, such as variations in color, texture, or density.

[0135] A system including a predictive sensor and a targeting sensor is positioned in a fixed position relative to the calibration surface such that the calibration surface overlaps with the fields of view of the predictive sensor and the targeting sensor. For example, the system may be placed on the surface. The predictive sensor can collect images of the calibration surface. The targeting sensor may collect a series of images of the calibration surface from various targeting sensor positions or orientations. Each of the different positions or orientations of the targeting sensor may correspond to a separate set of positions of actuators that control the position or orientation of the targeting sensor. In some embodiments, a series of images may be collected at random actuator positions.

[0136] For each series of images containing a region of the calibration surface, distinguishable features of the calibration surface can be identified and mapped to regions of the predicted image containing these distinguishable features. The corresponding actuator position in the target image can be correlated to regions of the predicted image such that the position in the predicted image correlates with the actuator position and the targeting camera position. In some embodiments, the calibration model can be used to extrapolate the corresponding predicted image position and actuator position to a position within a range of empirically measured sensor positions or directions. In some embodiments, the calibration model is a mathematical model such as a spline model, geometric model, or trigonometric function model. In some embodiments, the accuracy of the calibration model may decrease for non-planar surfaces.

[0137] The optical control systems shown in Figures 1 to 3 can be calibrated using the methods described herein. In some embodiments, the camera control conversion system of the prediction module 812 in Figure 8 is calibrated. In some embodiments, a calibration surface, such as the calibration grid shown in Figure 12, is positioned within the field of view of the prediction camera. The calibration surface includes known marks at known locations. The prediction camera can collect multiple images of the calibration surface at different locations relative to the calibration surface. The prediction module can then correlate the pixel locations of the known marks with known locations on the surface. An interpolation function can be constructed from multiple correlated pixel locations and known surface locations. In some embodiments, the interpolation function can be stored on a hard drive and loaded from the hard drive by the prediction module.

[0138] In some embodiments, the camera control conversion system of the targeting module 822 in Figure 8 is calibrated. In some embodiments, the targeting control module 1055 in Figure 10A is calibrated. In some embodiments, the calibration surface is positioned within the field of view of the targeting camera. The calibration surface includes known marks at known locations. The targeting module can collect multiple images of the calibration surface and multiple actuator locations such that multiple images have different fields of view. For example, the targeting module can collect multiple images at randomly selected pan-tilt values ​​of a first actuator and a second actuator. A calibration map can be constructed from multiple sample points. Each sample point may be collected by identifying the pixel location of a known mark in the image collected at a known actuator location and correlating the known location with the actuator location and pixel location. In some embodiments, the map is fitted to a calibration model. For example, the map may be fitted to a mathematical model such as a spline smoothing algorithm to construct a smooth curve, thereby enabling accurate estimation of the positions between sample points. In some embodiments, the calibration model may be stored on a hard drive and loaded from the hard drive by the targeting module.

[0139] Calibration of instruments The systems of this disclosure can be calibrated so that the position or orientation of the instrument correlates to the pixel position in the target image collected by the targeting sensor. In some embodiments, the position or orientation of the instrument is fixed relative to the targeting sensor. For example, the laser instrument and the targeting camera may be controlled by the same mirror so that the direction of the laser is coupled to the direction of the sensor, as shown in Figures 1 to 3. In another example, the instrument may be rigidly coupled to the targeting sensor so that the instrument moves with the targeting sensor. In some embodiments, the position or orientation of the instrument relative to the targeting sensor may be adjusted based on calibration. In some embodiments, the instrument can be moved relative to the targeting sensor.

[0140] Calibration of the instrument and targeting sensor may be performed by performing an action with the instrument at a location on the surface. In some embodiments, the action may leave a mark on the surface at that location. For example, a laser instrument may burn a spot on the surface, and a spraying instrument may leave a wet spot on the surface. In some embodiments, the action may change the surface at that location. For example, a gripping instrument may create a hole in the surface at that location. In some embodiments, the position of the instrument may be determined by a targeting sensor. The targeting sensor can collect an image of the surface at the location of the action by the instrument. The location of the action or the position of the instrument in the target image can be determined. The pixel position in the target image may be correlated with the position of the instrument or the position of the action of the instrument. In some embodiments, the position of the instrument relative to the targeting sensor may be adjusted to a preferred position. For example, the position of the instrument relative to the targeting sensor may be adjusted so that the position of the action of the instrument is at or near the center of the target image.

[0141] In some embodiments, the optical systems of the present disclosure, for example, the optical systems shown in Figures 1 to 3, can be calibrated to correlate the pixel positions in the surface image collected by the targeting sensor with the positions of the instrument on the surface, such as the position where the laser beam strikes the surface or the position where the tool interacts with an object or surface. The orientation of the targeting sensor and the orientation of the instrument may be physically linked. For example, the orientation of the targeting sensor and the direction of the laser may be physically linked via optical elements that control both the direction of the laser and the direction of the targeting sensor, such that the relative position of the targeting camera and the position of the laser beam on the surface are fixed, as shown in Figures 1 to 3. The targeting sensor can collect an image of the surface, and the instrument can perform an action on the surface while at the same position as the sensor. The action point is identified in the sensor image, and the pixel position of the action point is determined. This process of collecting an image, performing an action on the surface, and identifying the pixel position of the action point can be repeated to determine a calibration model that describes the relationship between the targeting sensor and the action point on the surface. In some embodiments, the calibration model is a mathematical model. In some embodiments, the calibration model is calculated using a spline model, geometric model, or trigonometric model to extrapolate the positions of sensors and instruments within a range of empirically measured positions.

[0142] Fine motion calibration of targeting systems The fine motion of the sensors or devices of this disclosure can be calibrated against position or distance on a surface. Fine motion calibration of a sensor can be performed by acquiring a first image of the calibration surface using the sensor, adjusting the position of the sensor, and acquiring a second image of the calibration surface. In some embodiments, the adjustment is performed using an actuator. The positions of distinguishable features in the first and second images can be determined, and the distance and direction of motion between the two positions can be determined. The distance and direction of motion can be correlated with the magnitude and direction of the actuator's motion. This process can be repeated for the motion or position of many different actuators to determine a calibration model. In some embodiments, mathematical models such as spline models, geometric models, or trigonometric function models can be used to extrapolate the motion of the corresponding sensors and actuators, such as mirror pan and tilt values, to empirically determined positions between points. For example, the motion of the corresponding sensors and actuators can be extrapolated using a spline model. In some embodiments, the accuracy of the calibration model may decrease for non-flat surfaces.

[0143] Fine motion calibration of an instrument can be performed by orienting the instrument to a position on a surface and acquiring a first image of the calibration surface using a sensor such as a targeting sensor. The position of the instrument in the first image may be determined by the position of the instrument itself or by the position of an action performed by the instrument. In some embodiments, the action performed by the instrument may leave a mark on the surface or alter the surface. The position of the instrument can be adjusted, and a second image of the surface can be acquired by a sensor. In some embodiments, the adjustment is performed using an actuator. The position of the instrument or the position of an action performed by the instrument in the first and second images may be determined, and the distance and direction of motion between the two positions may be determined. The distance and direction of motion can be correlated with the magnitude and direction of the actuator's motion.

[0144] This process can be repeated for the motion or position of many different actuators to determine a calibration model. The calibration model can be used to extrapolate the motion of corresponding instruments and actuators, such as the pan and tilt values ​​of a mirror, to empirically determined positions within a given point. In some embodiments, the calibration model is a mathematical model, such as a spline model, geometric model, or trigonometric model. In some embodiments, the accuracy of the calibration may decrease for non-planar surfaces.

[0145] Computer systems and methods Object identification and targeting methods can be implemented using a computer system. In some embodiments, the detection system described herein includes a computer system. In some embodiments, the computer system can autonomously implement the object identification and targeting method without human input. In some embodiments, the computer system can implement the object identification and targeting method based on instructions provided by a human user via a detection terminal.

[0146] Figure 13 shows components in a block diagram of non-limiting, exemplary embodiments of the detection terminal 1400 according to various aspects of the present disclosure. In some embodiments, the detection terminal 1400 is a device that displays a user interface to provide access to a detection system. As shown, the detection terminal 1400 includes a detection interface 1420. The detection interface 1420 enables the detection terminal 1400 to communicate with a detection system, such as the detection system in Figure 8 or the detection system in Figure 10A. In some embodiments, the detection interface 1420 may include an antenna configured to communicate with the detection system, for example, by a remote control. In some embodiments, the detection terminal 1400 may also include a local communication interface, such as an Ethernet® interface, a Wi-Fi interface, or another interface that allows other devices associated with the detection system to connect to the detection system via the detection terminal 1400. For example, the detection terminal may be a handheld device, such as a mobile phone, which runs a graphical interface, allowing a user to remotely operate or monitor the detection system via Bluetooth®, Wi-Fi, or a mobile network.

[0147] The detection terminal 1400 further includes a detection engine 1410. The detection engine can receive information regarding the status of a detection system, such as the detection system in Figure 8 or the detection system in Figure 10A. The detection engine can receive information regarding the number of identified objects, the identity of the identified objects, the location of the identified objects, the number of targeted objects, the identity of the targeted objects, the location of the detection system, the elapsed time of tasks performed by the detection system, the area covered by the detection system, the battery charge of the detection system, or a combination thereof.

[0148] Actual embodiments of the illustrated devices include many more components known to those skilled in the art. For example, each illustrated device may have a power supply, one or more processors, a computer-readable medium for storing computer executable instructions, and so on. For clarity, these additional components are not shown herein.

[0149] Figure 14 is a flowchart showing non-limiting and exemplary embodiments of procedure 1500 for detecting objects according to various aspects of the present disclosure. Procedure 1500 is an example of a procedure suitable for use with detection terminal 1400 shown in Figure 13 for setting up a detection terminal for communicating with the detection system of the present disclosure. In some embodiments, procedure 1500 is performed recursively to coordinate changes in the detection system, objects, and detection terminal 1400.

[0150] In block 1520, parameters are selected to detect the target object.

[0151] In block 1530, the detection system is instructed to detect objects according to detection parameters.

[0152] While exemplary embodiments have been illustrated and described, it should be understood that various modifications can be made therein without departing from the spirit and scope of this disclosure.

[0153] In some examples, the procedures described herein (e.g., procedure 900 in Figure 9, 1100 in Figure 11, or 1500 in Figure 14, or other procedures described herein) may be executed by a computing device or apparatus, such as a computing device having the computing device architecture 1600 shown in Figure 15. In one example, the procedures described herein may be executed by a computing device having the computing device architecture 1600. The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device, a server (e.g., in a Software as a Service (SaaS) system or other server-based system), and / or any other computing device having the resource capacity to execute the processes described herein, including procedures 900, 1100, or 1500. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other components configured to execute the steps of the processes described herein. In some examples, a computing device may include a display (as an example of an output device, or in addition to an output device), a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other types of data.

[0154] Components of computing devices can be implemented in circuits. For example, a component may include an electronic circuit or other electronic hardware that can include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits) to perform the various operations described herein, and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using them.

[0155] Procedures 900, 1100, and 1500 are presented as logical flow diagrams, where the operations represent sequences of operations that can be implemented in hardware, computer instructions, or a combination thereof. From the perspective of computer instructions, the operations represent computer-executable instructions stored in one or more computer-readable storage media that, when executed by one or more processors, perform the enumerated operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform specific functions or implement specific data types. The order in which the operations are listed is not intended to be interpreted as a restriction, and any number of listed operations can be combined in any order and / or in parallel to realize a process.

[0156] Furthermore, the processes described herein may be executed under the control of one or more computer systems consisting of executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is collectively executed by one or more processors in hardware or a combination thereof. As described above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program containing multiple instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-temporary.

[0157] Figure 15 shows an exemplary computing device architecture 1600 that can implement various technologies described herein. For example, the computing device architecture 1600 can implement the detection system shown in Figure 10A. The components of the computing device architecture 1600 are shown to communicate with each other using a connection 1605 such as a bus. The exemplary computing device architecture 1600 includes a processing unit (which may include a CPU and / or GPU) 1610 and a computing device connection 1605 that connects various computing device components to the processor 1610, including computing device memory 1615 such as read-only memory (ROM) 1620 and random access memory (RAM) 1625. In some embodiments, the computing device may include a hardware accelerator.

[0158] The computing device architecture 1600 may include a cache of high-speed memory that is directly connected to, adjacent to, or integrated as part of the processor 1610. The computing device architecture 1600 may copy data from memory 1615 and / or storage device 1630 to cache 1612 for rapid access by the processor 1610. In this way, the cache can provide a performance boost by avoiding delays in the processor 1610 while waiting for data. These modules and other modules may control or be configured to control the processor 1610 to perform various actions. Other computing device memories 1615 may also be available for use. Memory 1615 may include multiple different types of memory with different performance characteristics. The processor 1610 may include any general-purpose processor, as well as dedicated processors in which software instructions, as well as hardware or software services such as services 1 1632, 2 1634, and 3 1636 stored in storage device 1630, are incorporated into the processor design and configured to control the processor 1610. The processor 1610 may be an independent system including multiple cores or processors, buses, memory controllers, caches, etc. The multicore processor may be symmetrical or asymmetrical.

[0159] To enable user interaction with the computing device architecture 1610, the input device 1645 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, or voice. The output device 1635 can also be one or more of several output mechanisms known to those skilled in the art, such as a display, projector, television, or speaker device. In some cases, a multimodal computing device can allow the user to provide multiple types of inputs for communication with the computing device architecture 1600. The communication interface 1640 can generally operate and manage user inputs and computing device outputs. Since there are no restrictions on operation with any particular hardware configuration, the basic features described herein can be easily replaced with improved hardware or firmware configurations developed.

[0160] The storage device 1630 is non-volatile memory and can be a hard disk or other type of computer-readable medium capable of storing computer-accessible data, such as a magnetic cassette, flash memory card, solid-state memory device, digital versatile disk, cartridge, random access memory (RAM) 1625, read-only memory (ROM) 1620, and hybrids thereof. The storage device 1630 may include services 1632, 1634, and 1636 for controlling the processor 1610. Other hardware or software modules are also conceivable. The storage device 1630 can be connected to the computing device connector 1605. In one embodiment, a hardware module performing a particular function may include software components stored on a computer-readable medium in relation to hardware components necessary to perform that function, such as the processor 1610, the connector 1605, and the output device 1635.

[0161] The term “computer-readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, contain, or transport instructions and / or data. Computer-readable medium can include non-transient media that can store data and do not contain carrier waves and / or transient electronic signals that propagate wirelessly or via wired connections. Examples of non-transient media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory, or memory devices. Computer-readable medium may store code and / or machine-executable instructions on which procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements may be represented. Code segments can be coupled to other code segments or hardware circuits by passing and / or receiving information, data, arguments, parameters, or memory content. Information, arguments, parameters, data, etc., may be passed, transferred, or transmitted via any appropriate means, including memory sharing, message passing, token passing, network transmission, etc.

[0162] In some embodiments, computer-readable storage devices, media, and memory may include cables or wireless signals having bitstreams, etc. However, when referring to non-temporary computer-readable storage media, media such as energy, carrier signals, electromagnetic waves, and the signals themselves are explicitly excluded.

[0163] Specific details are provided in the above description to provide a complete understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that embodiments may be carried out without these specific details. For clarity of the description, in some cases the Art may be presented as including individual functional blocks having functional blocks that include devices, device components, steps, or routines in a manner implemented in software or a combination of hardware and software. Additional components other than those shown in the drawings and / or described herein may also be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary details. In other examples, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments.

[0164] Individual embodiments may be described above as processes or methods represented as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While operations may be described as sequential processes in a flowchart, many operations can be performed in parallel or concurrently. Furthermore, the order of operations may be reordered. A process may terminate when its operations are complete, but may include additional steps not shown in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. If a process corresponds to a function, its termination may correspond to the function's return to the calling function or the main function.

[0165] The processes and methods described above can be implemented using computer-executable instructions stored on a computer-readable medium, or otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that configure a general-purpose computer, a dedicated computer, or a processing unit to perform, or otherwise perform, a particular function or set of functions. Some of the computer resources used may be accessible over a network. Computer-executable instructions may include, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created in the methods described in the examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, and network storage devices.

[0166] Devices implementing the processes and methods described herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may employ one of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segment (e.g., a computer program product) for performing the required tasks may be stored in computer-readable or machine-readable media. A processor(s) may perform the required tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, digital assistants, rack-mount devices, and standalone devices. The functions described herein may also be embodied in peripherals or add-in cards. As a further example, such functions may also be implemented on circuit boards with different chips or different processes running within a single device.

[0167] Instructions, a medium for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are examples of means for providing the functionality described herein.

[0168] The various exemplary logic blocks, modules, circuits, and algorithmic steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly demonstrate this hardware and software compatibility, the diverse exemplary components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or as software depends on the specific application and the design constraints imposed on the overall system. A person skilled in the art may implement the described functionality in a variable manner for each specific application, but such a decision on implementation form should not be construed as resulting in a departure from the scope of this application.

[0169] The technologies described herein may be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such technologies may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices having multiple applications, including applications in wireless communication device handsets and other devices. Any function described as a module or component may be implemented together within an integrated logical device, or separately as individual but interoperable logical devices. When implemented in software, the technology may be at least partially realized by a computer-readable data storage medium containing program code having instructions that perform one or more of the above methods at runtime. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media such as random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media. Furthermore, or alternatively, the technology may be at least partially realized by a computer-readable communication medium, such as signals or waves, that can be accessed, read, and / or executed by a computer, which carries or communicates program code in the form of instructions or data structures.

[0170] The program code may be executed by a processor which may include one or more processors such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other uniform integrated circuits or discrete logic circuits. Such a processor may be configured to perform any of the techniques described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a DSP and a combination of a microprocessor, multiple microprocessors, one or more microprocessors working in conjunction with a DSP core, or any other such combination of components. Thus, the term “processor” as used herein may refer to any of the aforementioned structures, any combination thereof, or any other structure or device suitable for implementing the techniques described herein.

[0171] In the foregoing description, aspects of this application have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that this application is not limited thereto. Therefore, while exemplary embodiments of this application have been described in detail herein, it should be understood that the concepts of the present invention can be embodied and used in various other ways, and that, unless limited by the prior art, the appended claims are intended to be interpreted as including such variations. The various features and aspects of the uses described above can be used individually or in combination. Furthermore, embodiments can be used in any number of environments and uses beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, this specification and the drawings should be considered illustrative, not restrictive. For illustrative purposes, the methods have been described in a specific order. It should be understood that in alternative embodiments, the methods may be performed in a different order than described.

[0172] Those skilled in the art will understand that the symbols or terms less than ("<") and greater than (">") used herein may be replaced with the symbols less than or equal to ("≦") and greater than or equal to ("≧"), respectively, without departing from the scope of this description.

[0173] Where a component is described as being "configured" to perform a particular operation, such configuration can be achieved, for example, by designing an electronic circuit or other hardware to perform that operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform that operation, or by any combination thereof.

[0174] The phrase "connected" refers to any component that is physically connected to another component, either directly or indirectly, and / or communicates with another component, either directly or indirectly (for example, connected to another component via a wired or wireless connection and / or other appropriate communication interface).

[0175] The wording of a claim that includes “at least one” and / or “one or more” of a set indicates that one member or multiple members (any combination) of a set satisfy the claim. For example, the wording of a claim that includes “at least one of A and B” means A, B, or A and B. In another example, the wording of a claim that includes “at least one of A, B, and C” means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The words “at least one” and / or “one or more” of a set do not limit a set to the articles listed in the set. For example, the wording of a claim that includes “at least one of A and B” could mean A, B, or A and B, and could include additional articles not listed in the set A and B.

[0176] Where used herein, the terms “about” and “approximately” with respect to numbers are used herein to include numbers that fall within 10%, 5%, or 1% in both directions (greater than or less than) unless otherwise specifically stated or made clear from the context (except when such numbers exceed 100% of a possible value).

[0177] Examples The present invention can be further illustrated by the following non-limiting embodiments.

[0178] Example 1 Weeding in crop fields This embodiment describes weed control in a crop field using the detection system and method of the present disclosure. An autonomous vehicle, as shown in Figures 6 and 7, equipped with a prediction system, a targeting system, and an infrared laser, was positioned in the crop field. The vehicle autonomously traveled along rows of crops, and a wide-angle predictive camera imaged the field. The predictive camera's field of view was 27.7 in parallel to the direction of travel of the autonomous vehicle and 20 in perpendicular to the direction of travel. The prediction system identified weeds in the images collected by the predictive camera and distinguished them from the crops using a discriminative machine learning model trained to identify weed features and differentiate them from the features of onion crops. The prediction system selected weeds to be controlled from the identified weeds and determined the predicted location of the weeds based on the pixel location of the weeds in the predicted image. The predicted location was transmitted to the targeting system.

[0179] The targeting system included a targeting camera and an infrared laser, whose direction was adjusted by a mirror controlled by an actuator. As shown in Figures 1-3, the mirror reflected visible light from the surface to the targeting camera and infrared light from the laser to the surface. Based on empirically determined calibration coefficients that correlated object position coordinates with actuator positions, the targeting system converted the predicted position received from the prediction system to the actuator position and directed the targeting camera and laser to the predicted position. The targeting system adjusted the actuator to direct the targeting camera and infrared laser beam to the predicted position of the selected weed. The targeting camera, which had a narrower field of view than the prediction camera, imaged the field at the predicted position of the weed. The field of view of the targeting camera was 6 in parallel to the direction of travel of the autonomous vehicle and 4.5 in perpendicular to the direction of travel. Using an object matching machine learning model, the targeting system determined whether weeds were present in the images collected by the targeting camera, and if so, the target position of the weeds. The object-matching machine learning model considered the movement of the autonomous vehicle between predictive image acquisition and weed targeting, differences in camera viewpoints and image characteristics between the predictive and targeting cameras, and variability in the 3D environment due to uneven terrain and weed height. Using the object-matching machine learning model, the targeting system determined the target location of the weed and the offset between the predicted weed location and the target weed location. Based on the target location of the weed, the targeting system adjusted the position of the targeting camera and the infrared laser beam and activated the infrared beam toward the weed location. This beam irradiated the weed with infrared light for a sufficient time to damage or kill the weed, while adjusting the laser beam position to account for the movement of the autonomous vehicle during irradiation. The system was calibrated so that the position irradiated by the laser was within 5 mm, preferably within 2 mm, of the target location of the weed.

[0180] As shown in Figures 5 and 6, this vehicle included four targeting systems, allowing the targeting process to run multiple times in parallel. Each targeting system covered a different but overlapping area of ​​the prediction system. The prediction camera identified the second weed in the images it collected. Based on the availability of the targeting systems and the range of the targeting systems that targeted the predicted location of the second weed, the prediction system transmitted the predicted location of the second weed to the second targeting system. This process was repeated until all weeds in the crop field were removed, until the autonomous vehicle finished driving through the field, or until it reached another endpoint.

[0181] Example 2 Automatic identification and removal of debris in uneven construction environments. This example describes a system and method for automated identification and removal of debris in an uneven environment. An autonomous vehicle equipped with a predictive camera, a targeting camera, and a debris collection device autonomously navigates a construction site. The construction site has an uneven terrain surface. The predictive camera images the surface area of ​​the construction site and detects objects in the image. A trained identification machine learning model identifies the objects and selects those identified as debris.

[0182] The prediction system uses a calibration model to determine the predicted location of the debris and transmits this predicted location to the targeting system. The targeting system is selected based on availability and proximity to the selected debris. The targeting system instructs actuators that control the targeting camera and debris collection device to orient the targeting camera and debris collection device towards the predicted location of the debris. The targeting camera images the terrain surface at the predicted location of the debris, and a trained matching machine learning model determines whether the debris is located in the image and the target location of the debris. The machine learning model determines the offset between the predicted location of the debris and the target location of the debris and transmits this offset to the targeting system. The targeting system instructs the actuators to orient the targeting camera and debris collection device towards the target location of the debris, and the debris collection device collects the debris.

[0183] Additional position and acceleration sensors detect the autonomous vehicle's motion during the prediction, targeting, and debris collection processes, providing the targeting system with updated coordinates that take into account the vehicle's motion between image acquisition and debris collection.

[0184] Experimental Example 3 Automatic Identification and Removal of Obstacles in a Piping Network This embodiment describes a system and method for the automatic identification and removal of obstacles in a piping network. An autonomous vehicle equipped with a predictive camera, a targeting camera, and an obstacle removal device autonomously travels through the piping system. The piping system has an irregular surface. The predictive camera images the surface area of ​​the piping and detects objects in the image. A trained identification machine learning model identifies objects and selects objects that are identified as obstacles.

[0185] The prediction system uses a calibration model to determine the predicted location of the obstacle and transmits this predicted location to the targeting system. The targeting system is selected based on availability and proximity to the selected obstacle. The targeting system instructs actuators controlling the targeting camera and obstacle removal device to orient the targeting camera and obstacle removal device to the predicted location of the obstacle. The targeting camera images the pipe surface at the predicted location of the obstacle, and a trained matching machine learning model determines whether the obstacle is located in the image and the target location of the obstacle. The matching machine learning model determines the offset between the predicted location of the obstacle and the target location of the obstacle and transmits this offset to the targeting system. The targeting system instructs the actuators to orient the targeting camera and obstacle removal device to the target location of the obstacle, and the obstacle removal device removes the obstacle.

[0186] Additional position and acceleration sensors detect the autonomous vehicle's motion during the prediction, targeting, and obstacle removal processes, and provide the targeting system with updated coordinates, taking into account the vehicle's motion between image acquisition and obstacle removal.

[0187] Example 4 Adjusting satellite imaging for crop management from the air. This embodiment describes a system and method for coordinating satellite imagery with drone imagery to automate aerial crop management. One or more aerial crop management drones, each equipped with a targeting camera and a pesticide dispenser, fly over crop fields. The crop management drones communicate with an imaging satellite. The satellite images the Earth region, including the crop fields, and transmits the images to the crop management drones along with the geographic location data of the images. A trained identification machine learning model identifies fields in the images and selects the fields to be targeted.

[0188] The prediction system uses GPS location to determine the predicted location of the field and transmits this predicted location to a selected drone. The drone is selected based on availability and proximity to the selected obstacle. The drone's targeting system instructs the actuators controlling the drone to position the targeting camera and pesticide dispenser over the predicted location of the field. The targeting camera images the ground at the predicted location of the field, and a trained matching machine learning model determines whether the field is located in the image and the target location of the field. The matching machine learning model determines the offset between the predicted location of the obstacle and the target location of the obstacle and transmits this offset to the targeting system. The targeting system instructs the drone to position the targeting camera and pesticide dispenser over the target location of the field, and the pesticide dispenser sprays pesticide on the field.

[0189] Additional position and acceleration sensors detect the drone's motion and GPS coordinates during the prediction, targeting, and obstacle removal processes, providing the drone's position information to image acquisition satellites.

[0190] While preferred embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided only as examples. Herein, those skilled in the art will anticipate numerous variations, modifications, and substitutions without departing from the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed in the practice of the present invention. The following claims define the scope of the present invention, and the methods and structures contained within these claims, as well as their equivalents, are intended to be encompassed thereby.

Claims

[Claim 1] The invention described herein.

Images