Spot weed detection and processing within a field of view following machine learning training

Abstract

A weed spot spraying system is described for performing a spot-based weed spraying method based on classification values ​​rendered from partial field of view images according to a machine learning-based trained model applied to the partial field of view images by a processor. The system includes a camera, a spray nozzle assembly including a spray nozzle, and a processor. The method performed by the system includes acquiring a full field of view image of a crop bed by the camera. The method further includes extracting from the full field of view image a partial field of view image corresponding to a spray nozzle positioned to provide a spray field across a portion of the crop bed depicted in the partial field of view image, rendering a classification of the partial field of view image according to the machine learning-based trained model, and selectively activating the spray nozzle according to the classification of the partial field of view image.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This PCT application is related to U.S. Provisional Application No. 63 / 433,101, filed December 16, 2022, entitled "SPOT WEED DETECTION AND TREATMENT WITHIN A FIELD OF VIEW IN ACCORDANCE WITH MACHINE LEARNING TRAINING," the contents of which are expressly incorporated by reference in their entirety, including any references contained therein.

[0002]

[0002] The present disclosure relates generally to agricultural spot spraying devices that rely on machine learning. More particularly, the present disclosure relates to a system for selectively spraying within a single spray nozzle field in response to machine learning-guided image field classification corresponding to the single spray nozzle field. [Background technology]

[0003]

[0003] Considerable effort has been put into reducing the amount of herbicide applied to crops. One way in which such reduction has been implemented is by using spot sprays, as opposed to broad-area sprays of herbicides. Additionally, automated spot spray systems are currently under development that involve training artificial intelligence to recognize when weeds are present within a field of view and then activating a spot spray nozzle to spray the detected weeds. Importantly, known artificial intelligence-based systems rely on identifying weed image patterns within the camera's field of view, which relies on complex graphic image pattern recognition, within a given image acquired by the camera. Such pattern recognition is potentially very complex and involves complex processing (e.g., rotation) of the captured image, as well as providing a variety of potential optical patterns corresponding to the various weed types to be detected.

[0004]

[0004] Automated / machine learning-based spot spray weed control systems face several challenges to operate at a high level of performance. The first challenge is training such systems to accurately detect any of various types and sizes of weeds under a wide variety of operating environments (e.g., time of day, wet / dry ground, light / dark soil, etc.). To effectively train such systems, they need to have sufficient examples. As a result, both a wide variety of viewpoints and multiple instances of even similar viewpoints are required to ensure accurate automated detection of weeds.

[0005] A further challenge is ensuring that the system operates in a coordinated manner between weed detection and the subsequent activation / release of herbicides against the detected weeds. For example, if the system takes too long to process a given image, the farm equipment equipped with a spray nozzle in which the herbicide is displayed must slow down or stop to prevent the weed from leaving the corresponding herbicide spray nozzle field before the system can activate the spray nozzle to treat the detected weed with the herbicide.

[0006] Providing / operating an automated weed detection and spot spraying system is a challenging task. The number of provided / classified images required to effectively train such a system to accurately identify weeds can exceed one million images. Furthermore, effective image data parameterization and analysis is required to render near-instantaneous decisions on input real-time image data. Summary of the Invention

[0007]

[0007] Described herein is a weed spot spraying system configured to perform a spot-based weed spraying method based on classification values ​​rendered from partial field of view images according to a machine learning-based trained model applied to the partial field of view images by a processor. The system includes a camera, a spray nozzle assembly including a spray nozzle, and a processor operating cooperatively to perform the spot-based weed spraying method. Such a method includes acquiring, by the camera, a full field of view image of a crop bed. The method further includes extracting, from the full field of view image, a partial field of view image corresponding to a spray nozzle positioned to provide a spray field across a portion of the crop bed depicted in the partial field of view image. The processor renders a classification of the partial field of view image according to the machine learning-based trained model. The method further includes selectively activating the spray nozzle according to the classification of the partial field of view image.

[0008] While the appended claims set forth the features of the present invention with particularity, the present invention and its advantages are best understood from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic block diagram of an exemplary system configuration for implementing exemplary examples of the present invention in accordance with the present disclosure. [Figure 2a] FIG. 1 is an illustrative diagram showing a field of view for performing machine learning-based classification on extracted and normalized partial field of view images according to the present disclosure. [Figure 2b] FIG. 1 is an illustrative diagram showing sub-fields extracted from a field of view to perform machine learning-based classification on the extracted and normalized sub-field images in accordance with the present disclosure. [Figure 3a] 1 shows an image corresponding to a classification of "crop" according to the present disclosure. [Figure 3b] 1 shows images corresponding to a "crops and weeds" classification in accordance with the present disclosure. [Figure 3c] 1 shows an image corresponding to a "no crop" classification in accordance with the present disclosure. [Figure 4] 1 is a schematic diagram of a collaborative machine learning environment in accordance with the present disclosure. [Figure 5] FIG. 1 is a schematic diagram of an exemplary implementation of machine learning-based image classification to guide spot weeding in accordance with the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010]

[0014] While the invention is susceptible to various modifications and alternative constructions, specific exemplary embodiments thereof are shown in the drawings and are described in detail below. It should be understood, however, that there is no intention to limit the invention to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions and equivalents falling within the spirit and scope of the invention.

[0011]

[0015] An illustrative example will now be described that addresses the need for robust, dynamic, machine learning-based weed identification in the context of a spot weed control application. Furthermore, the teachings of a machine learning-trained characterization processor that issues output values ​​for an entire field of interest—as opposed to identifying specific visual features within the field of interest—can be applied to any of a number of treatable crop field conditions, including, for example, insect infestation, nutrient deficiencies, etc.

[0012]

[0016] Referring to FIG. 1, an exemplary weed detection and spot spray system 100 is shown. According to an illustrative example, the system 100 includes a camera 102 and a set of nozzles 104a, 104b, 104c, and 104d having laterally overlapping spray fields 106a, 106b, 106c, and 106d within a field of view 108 of the camera 102. By way of example, the dimension of each spray field is the area of ​​ground covered over time as the corresponding spray nozzle moves along a line formed by a path along which the crop field spraying implement on which the spray nozzle is mounted is pulled / pushed / mounted / self-propelled. While not shown, it will be understood that each of the nozzles 104a-104d is coupled to a tank containing a fluid (e.g., herbicide, liquid fertilizer, pesticide, etc.) to be applied to the field in accordance with an activation signal individually issued by a processor 110.

[0013]

[0017] In the illustrative example, camera 102 is a high-definition RGB digital camera (e.g., 12 Mpixels). However, any of a number of camera types and quantities are contemplated, including multispectral cameras and multiple cameras mounted on a spray boom assembly. Digital camera 102 is coupled via data bus 109 to processor 110 and memory 112, which stores and processes each captured image according to a machine learning-based image field classification scheme implemented by a neural network, as described further below.

[0014]

[0018] Additionally, a display device (not shown in FIG. 1 ) is coupled to the processor 110 and can receive the captured / classified images. The display device presents the captured / classified images to the operator / farmer, allowing the farmer to review and confirm the accuracy of the machine learning trained neural network in classifying received images that may or may not contain weeds. In addition to reviewing the captured / classified images, the display device's graphical user interface supports the user / farmer's input of annotations to identify and correct incorrectly classified captured images. In an exemplary scenario, the processor 110 is configured to pass any classified images with an associated uncertainty below a threshold (e.g., 85%) to a designated review buffer for further review and confirmation by the user / reviewer / farmer. The resulting reviewed / confirmed classified images, in the exemplary system, are passed to a centralized (networked) machine learning training facility (see FIG. 4 , described later in this specification).

[0015]

[0019] During operation, the camera 102 generates full image data of the crop bed within the field of view 108, including sub-fields 108a, 108b, 108c, and 108d (shown in FIG. 2a) corresponding to laterally overlapping spray fields 106a, 106b, 106c, and 106d of the sets of nozzles 104a, 104b, 104c, and 104d. The digital camera 102 acquires the full image data (corresponding to the field of view 108), which is subsequently transferred to the processor 110 and memory 112 for processing, as described in the illustrative examples herein. More specifically, the processor 110 extracts and normalizes the sub-field image data corresponding to each of the four sub-fields 108a-108d from the full image data of the portion of the crop bed corresponding to the field of view 108, in accordance with FIGS. 2a and 2b described later in this specification.

[0016]

[0020] 2a, an image of an exemplary field of view 108 is presented with overlays corresponding to spray fields 106a, 106b, 106c, and 106d. Note, in particular, that the spray field overlays are scaled according to the magnification effect of the near and far sub-fields of view of the overall camera field of view 108. Importantly, from each full image (corresponding to the field of view 108) acquired by camera 102, a set of four sub-images (corresponding to sub-fields of view 108a, 108b, 108c, and 108d) is extracted that substantially correspond to at least the width dimension of spray fields 106a, 106b, 106c, and 106d. In the illustrative example, the sub-fields of view 108a-108d are rectangular in shape. However, the sub-fields of view 108a-108d may have different shapes (e.g., trapezoidal shapes according to optical distortion of the crop field image by the forward-looking camera lens) according to other embodiments of the present disclosure. In an illustrative example, camera 102 captures a "forward-looking" view of the crop field before it passes under the spray fields 106a-106d of nozzles 104a-104d. As a result, the partial field of view image width (in pixels) corresponding to one of the outer spray nozzles (e.g., nozzle 104a) is smaller (in captured camera image pixels) than the partial field of view image width (in pixels) corresponding to one of the inner nozzles (e.g., nozzle 104b). As an example, the image of field of view 108 has dimensions of 1080 x 1920 pixels. As shown in FIG. 2b, extracted partial fields of view 108a and 108d have dimensions of 500 x 500 pixels, and extracted partial fields of view 108b and 108c have dimensions of 700 x 700 pixels, even though the spray fields of each nozzle 104a-104d have the same width. For forward-looking camera lenses, the far field captures a larger actual width, so the input subfield of view, which corresponds to a physical rectangle, is represented as a trapezoid in the captured camera image. Thus, the rectangular pixel image in Figure 2b represents a graphical morphing of a trapezoidal image subfield of view (the far end is smaller than the near end) into a rectangular / square image of the type shown in Figure 2b, referred to herein as "rectangularizing" the trapezoidal raw subfield of view image. Such an image transformation is an example of a more generalized "image correction" operation that includes perspective transformation.Such a transformation, also known as "inverse perspective mapping" (IPM), produces a "bird's-eye view" (a true overhead view, as shown in Figure 2a) from a relatively frontal perspective (108, as shown in Figure 1).

[0017]

[0021] According to an illustrative example, processor 110 (after performing rectangularization of the raw sub-field-of-view images) reduces / normalizes the raw extracted pixel image data corresponding to sub-fields of view 108a, 108b, 108c, and 108d to a 200x200 pixel image. Reducing the pixel image dimensions from the raw input image dimensions (500x500, 700x700) can be done in any of a variety of ways, such as by mapping pixels to the nearest corresponding pixel on a reduced image grid and discarding pixel data for pixels that are not closest to a given grid point in the reduced (200x200) pixel image location map.

[0018]

[0022] Processor 110 then performs a full image classification on each partial field of view image data instance based on the previously trained neural network to render a classification for each partial field of view image. The operation of processor 110 to perform input image classification operations on each of the rendered / normalized (200x200 pixel) images is described below with reference to a specific illustrative example.

[0019]

[0023] According to an illustrative example, the overall image classification output of the trained neural network of processor 110 is a simple classification of the processed partial-field-of-view image data instance for each processed partial-field-of-view image. In the simplest machine learning-based trained neural network scenario, the output classification value for each processed partial-field-of-view image is simply one of two values. The first value ("spray") corresponds to "weeds present in the image," and the second value ("don't spray") corresponds to "weeds not present in the image." However, in a more complex (but still relatively simple) classifier, the neural network is trained to detect when crop plants are present in the field of view and to activate spraying only when both crop plants and weeds are present in the partial-field-of-view image. This configuration yields at least a third classification, "no crop present," in which weeds may be present, but spraying is not activated due to the absence of crop plants. Each of the three different "classes" is illustratively depicted in the images of Figures 3a, 3b, and 3c, which show images corresponding to the classifications "crop," "crop and weed," and "no crop," respectively.

[0020]

[0024] Referring to FIG. 4, an exemplary networked configuration for implementing a collaborative machine learning configuration is shown, in which a group of individual users collaborate to provide additional partial field of view images and corresponding classifications. Such additional images may result from individuals post-spraying audits of the series of captured partial field of view images and the corresponding classifications assigned by the neural network. If the neural network renders an incorrect classification, auditors / reviewers of previously classified images within the group of individual users provide each incorrectly classified partial field of view image with (by way of example) a copy of the partial field of view (or full field) image data, the incorrect classification rendered by the neural network, and (optionally) a correct classification (proposed by the user). Additionally, multiple full image frames are also saved—comprising a full image including the incorrectly classified partial field of view image and full images before and after the incorrectly classified partial field of view image. In the illustrative example of FIG. 4, a first farmer 400, a second farmer 410, and a third farmer 420 each operate an instance of the system shown in FIG. 1. The identified farmers represent a population of several thousand farmers operating the system shown in FIG. 1. Each of the farmers 400, 410 and 420 operates their respective systems to perform spot spraying according to a local instance of a neural network that was constructed according to global training performed in a centralized neural network training facility 450.

[0021]

[0025] According to an illustrative example, facility 450 includes database 460 containing millions of captured partial field-of-view images that constitute training image set 465. Training image set 465 is provided to model training pipeline 470. The output of model training pipeline 470 is trained model 480, which includes neural network settings for neural network instances incorporated into each of farmers 400, 410, and 420's individual automated spot weed spraying systems. As shown in the illustrative example, the trained model output from model training pipeline 470, stored in trained model settings library 480, is characterized for a particular type of crop (e.g., wheat, corn, potato, soybean, etc.) or any other suitable grouping for classifying the received image set. Creating different neural network settings for specific crops (or other characteristic characteristics, such as soil type) may, given a sufficient number of training images, improve the robustness of the resulting neural network classifications rendered during operation by the instances of system 100 operated by farmers 400, 410, and 420. Of particular note, robust machine learning-based training is performed to render a "foundation model" that is used to configure a neural network to classify input images from various crop fields having a wide variety of crops, soil types / conditions, etc. The rendering of the foundation model is aided by a multidimensional map (database) configured to keep a geographic record of the source of the training images (environmental parameters describing the characteristics of the crop fields from which the training partial field images corresponding to the saved training instances were acquired).

[0022]

[0026] The output of the machine learning system stage implemented in the model training pipeline 470, in addition to providing a current output characterization parameter value (or set of values), is also in the form of a trained model 480. Thus, according to an illustrative example, the model training pipeline 470 incorporates and utilizes an initial training setting for the machine learning system stage. Such training may include, for example, constructing the trained model 480 (e.g., setting the weights and / or coefficients of the neural network's nodes and layers) using an initial training set of images with known classifications. The initial configuration of the trained model 480 continues iteratively until a setting is reached that results in a set of differences between the generated outputs and the expected outputs that fall within a specified minimum difference threshold. The specified minimum difference threshold may be specified based on individual differences and / or aggregated total differences (deltas) based on known output points provided by the training set. The settings may then be updated based on additional training points or a different set of difference thresholds based on particular needs and / or experience using the initial / current settings of the machine learning stage. Following the illustrative example of FIG. 4 , such updates to the originally provided classification model are facilitated by the annotated results of the currently existing version of the trained model 480 currently installed / running on the systems of the farmers 400, 410, and 420.

[0023]

[0027] 4 is a supervised, dynamically configured (via user / farmer-provided feedback) machine learning system. More specifically, in the above illustrative example, a neural network is embedded in each processor 110 operating within an individual farmer-operated instance of system 100. However, a wide variety of alternative machine learning architectures / types are contemplated, including support vector machines, linear regression, logistic regression, naive Bayes, linear discriminant analysis, decision trees, k-nearest neighbor algorithms, and self-supervised methods such as similarity learning and contrastive learning methods.

[0024]

[0028] 5, an exemplary data / decision flow is summarized according to a detailed structural and operational description of system 100 corresponding to one of the farmer-operated systems (e.g., farmer 400 system). According to the illustrative example, the system includes a data collection aspect 500 consisting of a pixel image dataset corresponding to the full field of view 108 of camera 102. According to the illustrative example, the captured pixel image dataset is transferred to a machine learning-based trained image classifier subsystem 510.

[0025]

[0029] A partial field of view image extraction stage 512 of the classifier subsystem 510 extracts partial field of view image instances (which may overlap with other partial field of view image instances according to the spray field nozzle coverage of each spray nozzle corresponding to one of the partial field of view image instances) from the received instance of the pixel image dataset corresponding to the full field of view 108, each partial field of view image corresponding to an area of ​​coverage of a corresponding spray nozzle for applying herbicide to weeds, or generally any fluid (e.g., pesticide, fertilizer, etc.) applied to treat a characteristic field of view of an agricultural crop. Each resulting partial field of view image is then presented to a machine learning trained image classifier stage 515 of the classifier subsystem 510.

[0026]

[0030] The classifier stage 515, which includes the configured / trained neural network, then renders a classification (“spray” or “don't spray”) and associated uncertainty score for the provided partial field image instance, corresponding to a particular area of ​​the field where the corresponding activated spray nozzle will apply herbicide within a particular operating period, which may be determined by the image / spray activation synchronization subsystem 520 (described later in this specification).

[0027]

[0031] According to an illustrative example, the classifier stage 515 includes a preconditioning component 516 that performs digital data processing (e.g., filtering, parameterization, etc.) on the received input partial field of view image instances rendered by the partial field of view image extraction stage 512. According to an illustrative example, the preconditioning component 516 renders the acquired data in a form suitable for further processing by the machine learning trained neural network stage 517. In its simplest form, such preconditioning may simply pass the data in digitized form (but otherwise unchanged) to storage for subsequent processing by the machine learning trained neural network stage 517. The preconditioning performed by the preconditioning component 516 may be a simple arithmetic operation applied to each value (e.g., multiplying each pixel value by a scalar value) or it may be a complex algorithm based on multiple sensor data values ​​or previously acquired image data. For example, an image may be preconditioned by averaging it with several previously acquired images. Yet another example of preconditioning is performing a Fourier transform on the image data to render the received time-domain data in the frequency domain. There may be more than one preconditioning step applied to image data instances acquired over time.

[0028]

[0032] The output of preconditioning component 516 is provided in a pre-established form to machine learning trained neural network stage 517. As a specific example, camera 102 may render a subimage having 6 million pixels (or 2 million pixels), but neural network stage 517 is trained and operates on images with a smaller number of pixels (e.g., 1 million pixels, 500,000 pixels, 40,000 pixels, etc.). In such a case, preconditioning component 516 downsamples the dataset to satisfy the 1 million pixel constraint of neural network stage 517.

[0029]

[0033] The machine learning trained neural network stage 517, in a particular example, includes an artificial neural network (e.g., a convolutional neural network). The artificial neural network is composed of a collection of configurable / adjustable processing layers (each layer including a set of computational nodes). The neural network incorporates / integrates a set of algorithms designed to perform an overall image classification operation on a set of input parameters corresponding to the pre-conditioned output from the pre-conditioning component 516. As an example, the data input for the machine learning trained neural network stage 517 is an input array having dimensions (length x width x 3) corresponding to the color (red, green, blue) pixels of the normalized color (red, green, blue) input partial field of view image to be classified.

[0030]

[0034] The output of the neural network stage 517 of the classifier stage 515 includes a set of outputs that return a value for each input digital image. More specifically, each output of the set of outputs corresponds to a particular one of a set of different classifications. Furthermore, in an illustrative example, the classification of a particular classified input partial field of view image is determined based on one of the classification individual outputs having the highest value. Furthermore, the combination of output values ​​is analyzed by the neural network to render a confidence value for the determined classification. The confidence value represents the accuracy of the "winning" classification output among the classification individual output set of the neural network. For example, in a simple binary classification output configuration, the classification output set has individual outputs corresponding to "spray" and "don't spray." The confidence value corresponding to the classification rendered by the classification output value set is based, for example, on the relative (or absolute) magnitude of the classification output set values ​​issued by the neural network.

[0031]

[0035] As an example, the neural network stage 517 includes a combination of computational elements and data structures organized as an input layer, multiple internal / hidden layers, and an output layer. The output from the preconditioning component 516 provides at least a portion of the information content of the input layer. The topology of the neural network stage 517 of the classifier stage 515 includes the number of internal layers, the number of computational nodes within each internal layer, and the inter-layer connectivity between nodes in adjacent layers of the artificial neural network. As an example, during training of the neural network stage 517, the configuration of the machine learning stage includes specifying a set of coefficients (weights) for each of the internal layers of the artificial neural network. More specifically, each layer is filled by a weighted combination of elements from the previous layer. A nonlinear activation function is applied to the resulting value of each node to determine the output value to pass to the next layer.

[0032]

[0036] Within a layer of the artificial neural network in the neural network stage 517, elements from previous layers are combined, illustratively in a weighted manner. The output value of each element is a nonlinear operation acting on the combination. As will be readily understood by those familiar with artificial neural network topology, each internal layer consists of a set of nodes, each of which receives a weighted contribution from each node in the previous layer and provides a weighted output (specified for each node) to each node in the next internal layer of the artificial neural network. A nonlinear operation is performed within each node on the weighted input received from the previous layer.

[0033]

[0037] The set of weight values ​​518 provides the configuration for the nodes of the neural network of the neural network stage 517. In particular, the values ​​at each layer of the artificial neural network described above are given by weight values ​​518. Each individual value of weight values ​​518 is established during training of the classifier stage 515 using a set of input partial field of view images and the corresponding appropriate classification (spray / don't spray).

[0034]

[0038] An important aspect of the operation of the image classifier stage 515 (and more specifically, the neural network stage 517) is the training process. Training involves determining weight values ​​for the neural network stage 517 of the image classifier stage 515. Similar to the operational example, output classification values ​​are rendered by the neural network stage 517 for a set of pre-trained data. However, in this case, each output classification value is accompanied by a label containing one or more parameters associated with the classification value. Because the appropriate weights are not yet known, an initial starting guess is used. During training, the weights are adjusted until the calculated classifications rendered by the neural network stage 517 accurately track the actual (observed) classifications for the input partial field of view images, i.e., until the neural network stage 517 operates within a predetermined performance envelope. Retraining may also be performed for any of a variety of reasons. Data collected in storage (e.g., misclassified partial field of view images submitted by farmers / users) can be used as labeled data to refine the operation of the neural network stage 517.

[0035]

[0039] According to an illustrative example, retraining / updating of the trained spray image classification model is performed using annotated (classified) verified classification images provided by farmers (e.g., farmers 400, 410, 420, etc.) during operation of their respective systems. As a specific example, during operation of system 100 by farmer 400, for any partial field of view image whose rendered classification does not meet a pre-set certainty value threshold, the partial field of view image is stored in a review buffer for review / confirmation of the classification assigned to the partial field of view image, either directly / automatically by classifier stage 515 sending it to cloud-based training facility 450, or alternatively / additionally, by operation of photo annotation subsystem 525 under direction of a user / observer / farmer. The confirmed classification for a particular partial field of view image is used to complement the classification model trained by machine learning (in cloud-based training facility 450 described above in this specification).

[0036]

[0040] Turning to the synchronization subsystem 520, by way of example, during real-time operation of the system 100, various operational parameters are applied to render the timing of spray activation for a particular partial field of view image having a "spray" classification rendered by the classifier stage 515. Examples of operational parameters used by the synchronization subsystem include, but are not limited to, machine travel speed, nozzle height, linear distance between the nozzle spray field and the field location corresponding to the classified image, etc.

[0037]

[0041] Additionally, in accordance with the disclosure herein, annotation subsystem 525 is provided for users to audit classifications assigned to partial field of view image instances by classifier subsystem 510. A user can use the editing facilities of annotation subsystem 525 to search for partial field of view images and corresponding classifications, edit / modify the classification assigned to a particular image, and then report the modifications to a centralized training facility of the type proposed / described in connection with FIG.

[0038]

[0042] In summary of the disclosure herein, several technical problems and corresponding solutions are provided herein that result in an improved weed spot spray configuration based on real-time classification of partial field of view images by a machine learning based trained (neural network) processor as either having or not having weeds, where the partial field of view images substantially match the lateral (width) dimensions of corresponding spray nozzles configured to selectively activate to apply herbicide to detected weeds in the spot spray operation.

[0039]

[0043] As shown in this disclosure, a reliably trained system is highly beneficial. To that end, according to this disclosure, annotating training images (providing classifications) is a very simple task that simply involves the user assigning a classification value to each training instance.

[0040]

[0044] In an exemplary scenario, a farmer can quickly audit (and annotate with the correct classification) a large number of captured partial field of view images, which are then used in a collaborative training environment to improve a classification model run by a neural network-based processor with the updated classification model.

[0041]

[0045] Another technical challenge involves ensuring that the spray nozzle is properly activated to spray the area of ​​the crop field corresponding to the partial field image classified as containing weeds. The present disclosure presents a relatively simple solution in which the nozzle is activated before passing over the target crop field area and then remains activated (spraying herbicide) for a duration that ensures that the target field portion has been sprayed. In that regard, the duration can be increased or decreased depending on the nozzle's movement speed along the detected field. Importantly, instead of identifying the exact location of the weed (within the camera's field of view), the disclosed system only classifies the entire partial field of view corresponding to the spray field width perpendicular to the direction of spray nozzle movement (left and right across the crop bed). Furthermore, the camera 102 is mounted on the same boom as the spray nozzles 104a-104d, ensuring that the field of view of the camera 102 adjusts with changes in the orientation of the boom to which the spray nozzles 104a-104d are attached. The boom may be oriented horizontally relative to the ground; alternatively, the boom may be oriented vertically relative to the ground.

[0042]

[0046] Additionally, the overlap of the partial field of view images, as well as the overlap of the spray fields 106a-106d of the spray nozzles 104a-104d, ensures that if one nozzle misses a weed (e.g., due to a strong crosswind), the weed will be treated by an adjacent nozzle. Furthermore, if one partial field of view image is misclassified, the weed may still be detected by another partial field of view image positioned laterally on the same image. Furthermore, for any given area traversed by the system 100, several images are captured by the system. Thus, as the system 100 advances along the crop line in the field, several opportunities to detect the weed arise because the weed is depicted in multiple successively acquired images.

[0043]

[0047] Yet another challenge is the need to provide a robust model for classifying a wide variety of weed images in various crops and soil environments. To that end, instead of training a neural network during the development phase, the classification model is continuously trained and updated. The network is improved by collecting interesting images during farmers' field work. The acquisition of such additional training images and their corresponding classifications is managed by targeted updates guided by included metadata that characterizes the conditions / environments in which the training instances were acquired. Such metadata can include any of the following types: geographic location, date, weather (lighting), climate, soil condition / color, soil type, etc. This metadata facilitates the construction of a robust model by retraining on characteristics / conditions for which the model was not previously trained, ensuring the inclusion / addition of classified training images.

[0044]

[0048] Various input data sources, including yield maps, fertilizer maps, soil maps, etc., can be used in determining when and where to acquire additional training images. Additionally, metadata based on weather forecasts and models to predict where certain rare conditions may occur can be used. The central management facility can then create an image collection plan and send the plan to specific sprayers when farmers begin spot spraying crop fields. Acquisition of new classification datasets is performed by an automated management system that rapidly acquires new training images without user intervention and then deploys updated models for the neural network configuration executed by processor 110. As an example, indications of neural network configuration issues (e.g., incorrect classifications) can be sent to the central management facility according to the illustrative example provided in FIG. 4. The model is updated, and the updated neural network configuration is downloaded to the farmer's processor 110 via a network connection.

[0045]

[0049] While the above discussion has been directed to classifying partial fields of view that may or may not have weeds, the disclosed arrangements may alternatively be trained to classify partial field of view images as diseased / non-diseased based on machine learning trained neural network processing of the partial field of view images.

[0046]

[0050] All references cited in this specification, including publications, patent applications, and patents, are herein incorporated by reference to the same extent as if each reference was individually and specifically indicated to be incorporated by reference and was set forth in its entirety herein.

[0047]

[0051] Use of the terms "a," "an," "the," "at least one," and similar referents in the context of describing the present invention (particularly in the context of the claims below) should be interpreted to encompass both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The use of the term "at least one" followed by a list of one or more items (e.g., "at least one of A and B") should be interpreted to mean one item (A or B) selected from the listed items or any combination of two or more of the listed items (A and B), unless otherwise indicated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "including" should be interpreted as open-ended terms (i.e., meaning "including, but not limited to"), unless otherwise noted. Recitation of ranges of values ​​herein is merely intended to serve as a shorthand method of individually referring to each separate value falling within the range, and each separate value is incorporated herein as if set forth individually herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or clearly contradicted by context. Any and all examples provided herein, or the use of exemplary language (e.g., "such as"), are intended merely to better illustrate the invention and do not pose a limitation on the scope of the invention unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the invention.

[0048]

[0052] Preferred embodiments of this invention are described herein, including the best mode known to the inventors for carrying out the invention. Variations of these preferred embodiments may become apparent to those skilled in the art upon reading the foregoing description. The inventors expect that such variations will be utilized by those skilled in the art, and the inventors intend the invention to be practiced otherwise than as specifically described herein. Accordingly, this invention includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Furthermore, any combination of the above-described elements in all possible variations thereof is encompassed by this invention unless otherwise indicated herein or clearly contradicted by context.

Claims

1. A method for performing spraying based on classification values ​​rendered from a partial field image according to a machine learning-based trained model applied to the partial field image by a processor, The steps include acquiring a full-field image of the object using a camera, The steps include extracting a partial field image from the full field image of the object that corresponds to the target partial field of view of a target spray nozzle arranged to provide a spray field extending over a portion of the crop bed depicted in the partial field image, The processor renders the classification of the partial field images according to the machine learning-based trained model, A step of selectively activating the spray nozzle according to the classification of the partial field image. Methods that include...

2. The method according to claim 1, wherein the rendering is performed by a neural network incorporated in the processor.

3. The method according to claim 2, wherein the neural network has an input array having dimensions corresponding to the pixel dimensions of the partial field image.

4. The method according to claim 3, wherein the neural network has a number of outputs corresponding to a set of potential classifications of the partial field images.

5. The method according to claim 1, wherein the method is performed on a system including a plurality of spray nozzles having a spray field that overlaps between two adjacent spray nozzles among the plurality of spray nozzles.

6. The method according to claim 1, wherein the camera is attached to the same physical mounting structure as the spray nozzle.

7. The method according to claim 1, wherein the partial field image is rectangular.

8. The method according to claim 1, wherein the partial field of view image is a rectangle rendered from an initial trapezoidal image partial field of view captured by a forward-looking camera.

9. The method according to claim 1, wherein the step of selectively activating the spray nozzle includes the step of keeping the activated spray nozzle in the ON state for a period of time related to the relative velocity at which the object moves along the line with respect to the spray nozzle.

10. The method according to claim 1, further comprising the step of adapting the on-time of the spray nozzle according to the mechanical operating speed that causes the object to move along the line with respect to the spray nozzle.

11. The method according to claim 1, further comprising the step of providing a sample partial field image and a corresponding validated classification of the sample partial field image to a training facility for the machine learning-based trained model.

12. The method according to claim 11, wherein the sample partial field image is provided in real time during the spraying operation.

13. The method according to claim 12, wherein the step of providing the sample partial field images to the reviewer for verification of the assigned classification is performed on an automated basis.

14. The method according to claim 13, wherein the automated base includes the step of applying the confidence value for classification of the sample partial field image to a threshold confidence value.

15. The method according to claim 1, wherein the rendering is performed on a cloud-based server system.

16. The method according to claim 1, further comprising the step of providing the rendered classification and associated partial field images to a reviewer for annotation.

17. The method according to claim 1, wherein the rendering is performed according to the base model.

18. The method according to claim 17, wherein a relevant training coverage map is provided that indicates the range of a particular type of training image on which machine learning training has been performed on the base model.

19. A spraying system configured to perform a spraying method based on classification values ​​rendered from a partial field image according to a machine learning-based trained model applied to the partial field image by a processor, Camera and, A spray nozzle assembly including the target spray nozzle, Processor and Equipped with, The aforementioned spot-based weed spraying method The steps include acquiring a full-field image of the object using the aforementioned camera, The steps include extracting a partial field image from the full field image of the object, corresponding to the target partial field of view of the target spray nozzle, which is arranged to provide a spray field extending over a portion of the crop bed depicted in the partial field image; The processor renders the classification of the partial field images according to the machine learning-based trained model, A step of selectively activating the spray nozzle according to the classification of the partial field image. A spraying system, including

20. The system further comprises a network communication interface configured to communicate with networked equipment and provide training messages, and the network communication interface is A partial field image instance, The aforementioned partial field image instance includes metadata describing the environment acquired by the camera, The confirmed characteristics of the aforementioned partial field image and A weed spot spraying system according to claim 19, comprising the above.

Images