Fixed point weed detection and processing within a field of view

Abstract

The present invention relates to a fixed-point weed spraying system for implementing a fixed-point weed spraying method based on a classification value generated from a sub-region image by a processor applying a machine learning-based training model to the sub-region image, the system comprising: a camera; the nozzle assembly comprises a nozzle; and a processor. A method implemented by the system includes acquiring, by a camera, a full-view region image of a crop surface. The method further includes extracting, from the full-field-of-view region image, a sub-region image corresponding to a nozzle positioned to provide a spray region extending over a portion of the crop surface depicted in the sub-region image; generating, by a processor, a classification of the sub-region images according to a training model based on machine learning; and selectively activating the nozzle according to the classification of the sub-region image.

Description

[0001] Cross-reference to related applications

[0002] This PCT application claims the benefit of U.S. Provisional Application No. 63 / 433,101, filed on Dec. 16, 2022, entitled “Spot Weed Detection and Treatment within a Field of View Based on Machine Learning Training,” the entire content of which is hereby incorporated by reference in its entirety, including any references contained therein. Technical Field

[0003] The present invention generally relates to an agricultural spot spraying device that relies on machine learning. More specifically, the present invention relates to a system for selective spraying within a single nozzle area based on machine learning-guided classification of image regions corresponding to a single nozzle area. Background Art

[0004] Considerable efforts have been made to reduce the amount of herbicide used on crops. One way to implement such a reduction in herbicide is to use spot spraying rather than broad-area spraying of herbicides. In addition, automated spot spraying systems are currently being developed that involve training artificial intelligence to identify when weeds are in the field of view and then activating spot nozzles to spray the detected weeds. Importantly, known artificial intelligence-based systems rely on identifying weed image patterns within the camera's field of view, which relies on complex graphical image pattern recognition within a given image acquired by the camera. Such pattern recognition can be very complex, involving complex processing (such as rotation) of the acquired images and providing various potential optical patterns corresponding to the various weed types to be detected.

[0005] Automated / machine learning-based spot spraying weed control systems face many challenges in high-performance operation. The first challenge is to train such systems to accurately detect various types and sizes of weeds in various operating environments (such as different times of day, wet / dry ground, light / dark soil, etc.). Sufficient samples are needed to effectively train such a system. Therefore, a variety of views and even multiple instances of similar views are needed to ensure the accuracy of automatic weed detection.

[0006] Another challenge is to ensure that the system operates in a coordinated manner between weed detection and subsequent triggering / release of herbicide on the detected weeds. For example, if the system takes too long to process a given image, the agricultural machinery carrying the nozzle (through which the herbicide is released) must slow down or stop to ensure that the weeds do not leave the corresponding herbicide nozzle area before the system can activate the nozzle to treat the detected weeds with herbicide.

[0007] Providing / operating an automatic weed detection and spot spraying system is a challenging task. The number of images provided / classified required to effectively train such a system to accurately identify weeds can exceed one million. Additionally, the image data needs to be effectively parameterized and analyzed to make near-instantaneous decisions on the input real-time image data. Summary of the Invention

[0008] This document describes a weed spot spraying system configured to apply a machine learning training model to a sub-region image by a processor, generate a classification value from the sub-region image, and perform a spot weed spraying method based on the classification value. The system includes a camera, a nozzle assembly including a nozzle, and a processor that cooperate to perform the spot weed spraying method. This method includes acquiring an image of the full field of view of the crop surface by the camera. The method also includes extracting a sub-region image corresponding to the nozzle from the full field of view image, where the nozzle is positioned to provide a spraying area extending over a portion of the crop surface depicted in the sub-region image. The processor generates a classification of the sub-region image based on a machine learning-based training model. The method also includes selectively activating the nozzle according to the classification of the sub-region image. Brief Description of the Drawings

[0009] While the appended claims set forth the features of the invention in detail, the invention and its advantages are best understood from the following detailed description in conjunction with the accompanying drawings, in which:

[0010] Figure 1 is a schematic block diagram of an exemplary system arrangement for implementing embodiments of the present invention according to the disclosure of the present invention;

[0011] Figure 2a and Figure 2b schematically shows a field of view and a sub-region extracted therefrom according to the disclosure of the present invention for performing machine learning-based classification on the sub-region image extracted and normalized therefrom;

[0012] Figure 3a 、 Figure 3b and Figure 3c respectively depict images corresponding to the "crop", "crop and weed", and "no crop" classifications according to the disclosure of the present invention;

[0013] Figure 4 provides a schematic diagram of a collaborative machine learning environment according to the disclosure of the present invention; and

[0014] Figure 5 is a schematic diagram of an exemplary instance of machine learning-based image classification for guiding spot weeding according to the disclosure of the present invention. Detailed Description of the Embodiments

[0015] Although the present invention is susceptible to various modifications and alternative constructions, specific exemplary embodiments thereof are shown in the drawings and will be described in detail hereinafter. However, it is to be understood that there is no intention to limit the invention to the specific forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention.

[0016] Illustrative examples are now described that address the need to provide robust and dynamic machine - learning - based weed identification in spot - spraying applications. Additionally, a machine - learning - trained feature processor can generate output values for an entire field - of - view area of interest, rather than identifying specific visual features within the field - of - view area of interest, and its teachings can be applied to any of a number of manageable farmland conditions (including, for example, pest infestations, nutrient deficiencies, etc.).

[0017] Referring to Figure 1 , an exemplary weed detection and spot - spraying system 100 is shown. According to one illustrative example, system 100 includes a camera 102 and nozzle groups 104a, 104b, 104c, and 104d, which have laterally - superimposed spraying areas 106a, 106b, 106c, and 106d within the field - of - view area 108 of the camera 102. By way of example, the size of each spraying area is the ground area covered over time as the nozzle travels along a route defined by the path of a towed / pushed / hung / self - propelled farmland spraying tool, and the corresponding nozzle is mounted on that path. Although not shown in the figures, it will be understood that each nozzle 104a - d is connected to a tank containing a fluid (such as herbicide, liquid fertilizer, pesticide, etc.), and the fluid is applied to the field in response to an activation signal issued individually by the processor 110.

[0018] In one illustrative example, camera 102 is a high - definition RGB digital camera (e.g., 12 megapixels). However, any of a variety of camera types and quantities can be considered, including multispectral cameras and multiple cameras mounted on a boom assembly. The digital camera 102 is coupled to the processor 110 and the memory 112 via a data bus 109 for storing and processing each acquired image according to a machine - learning - based image field classification arrangement performed by a neural network, which will be described further below.

[0019] In addition, a display device ( Figure 1(not shown in the figure) may be coupled to and receive the processor 110. The display device presents the acquired / classified images to the operator / farmer, enabling the farmer to review and confirm the accuracy of the neural network trained by machine learning in classifying the received images (which may or may not include weeds). In addition to reviewing the acquired / classified images, the graphical user interface of the display device also supports the user / farmer to input annotations to identify and correct misclassified acquired images. In an exemplary scenario, the processor 110 is configured to transfer any classified images with an accompanying uncertainty below a threshold (e.g., 85%) to a designated review buffer for further review and confirmation by the user / reviewer / farmer. In an exemplary system, the resulting reviewed / confirmed classified images are transferred to a centralized (networked) machine learning training facility (see Figure 4 , which will be described below).

[0020] In operation, the camera 102 generates complete image data of the crop surface within the field of view 108, which includes sub-regions 108a, 108b, 108c, and 108d (as Figure 2a shown), and the sub-regions correspond to the laterally superimposed spraying areas 106a, 106b, 106c, and 106d of the nozzle groups 104a, 104b, 104c, and 104d. According to the description of the illustrative examples herein, the digital camera 102 acquires the complete image data (corresponding to the field of view 108), and then transmits it to and processes it by the processor 110 and the memory 112. More specifically, according to Figure 2a and 2b described below, the processor 110 extracts / normalizes the sub-region image data corresponding to each of the four sub-regions 108a-d from the complete image data of a part of the crop surface corresponding to the field of view 108.

[0021] Brief reference is made to Figure 2a, the exemplary field of view region 108 image presents an overlay corresponding to the spraying regions 106a, 106b, 106c, and 106d, with particular note that the spraying region overlay is scaled according to the magnification effects of the near and far sub-regions of the entire camera field of view region 108. Importantly, a set of four sub-images (corresponding to sub-regions 108a, 108b, 108c, and 108d) are extracted from each complete image (corresponding to the field of view region 108) obtained from the camera 102, and these sub-images essentially at least correspond to the width dimensions of the spraying regions 106a, 106b, 106c, and 106d. In an illustrative example, the shapes of the sub-regions 108a-d are rectangular. However, according to other embodiments of the present invention, the sub-regions 108a-d can have different shapes (e.g., a trapezoidal shape according to the optical distortion of the forward-looking camera lens for the crop field image). In an illustrative example, the camera 102 captures a "forward-looking" view of the crop field before the crop field passes under the spraying regions 106a-d of the nozzles 104a-d. Thus, the width of the sub-region image (in pixels) corresponding to one of the outer nozzles (e.g., nozzle 104a) is smaller than the width of the sub-region image (in the captured camera image pixels) corresponding to one of the inner nozzles (e.g., nozzle 104b). For example, the size of the field of view region 108 image is 1080×1920 pixels. As Figure 2b shown, the extracted sub-regions 108a and 108d have dimensions of 500×500 pixels, and the extracted sub-regions 108b and 108c have dimensions of 700×700 pixels (although the spraying regions of each nozzle 104a-d have the same width). In the case of a forward-looking camera lens, the actual width collected in the far region is larger, so the input sub-region corresponding to the physical rectangle is represented by a trapezoid in the captured camera image. Thus, Figure 2b the rectangular pixel image representation in Figure 2b transforms the graphical deformation of the trapezoidal image sub-region (where the far edge is smaller than the near edge) into a Figure 2b rectangular / square image of the type shown in Figure 1 , herein referred to as the "rectangularization" of the trapezoidal original sub-region image. This image transformation is an example of a more generalized "image correction" operation involving perspective transformation. This transformation is also known as "inverse perspective mapping" (IPM), which produces a "bird's-eye" (directly looking down - as Figure 1 shown in 108) view from a relatively forward-facing perspective (as Figure 2a shown).

[0022] According to an illustrative example, the processor 110 (after rectifying the original sub-region images) reduces / normalizes the original extracted pixel image data corresponding to sub-regions 108a, 108b, 108c, and 108d into a 200×200 pixel image. The way to reduce the pixel image size from the original input image size (500×500, 700×700) can be performed by any of a variety of methods, including mapping the pixels to the nearest corresponding pixels on the reduced image grid and discarding the pixel data of the pixels that are not the closest to a given grid point in the reduced (200×200) pixel image position map.

[0023] Thereafter, the processor 110 performs whole-image classification on each sub-region image data instance based on a previously trained neural network to classify the images within each sub-region. The operation of the processor 110 to perform input image classification operations on each generated / normalized (200×200 pixel) image will be described below with reference to a specific illustrative example.

[0024] According to an illustrative example, for each processed sub-region image, the whole-image classification output of the trained neural network of the processor 110 is a simple classification of the processed sub-region image data instance. In the simplest machine learning-based trained neural network scenario, the output classification value for each processed sub-region image is just one of two values. The first value ("spray") corresponds to "there are weeds in the image", and the second value ("no spray") corresponds to "there are no weeds in the image". However, in a more complex (but still relatively simple) classification arrangement, the neural network is trained to detect when there are crops in the field of view, and the spray is only activated when both crops and weeds are within the sub-region image. This arrangement results in at least a third classification of "no crops", that is, there may be weeds, but since there are no crops, the spray is not activated. Each of the three different "categories" is Figure 3a , 3b and the three images in 3c are illustratively described, which correspond to the images of the "crop", "crop and weed", and "no crop" classifications respectively.

[0025] Turn to Figure 4, which shows an exemplary networked arrangement for implementing a collaborative machine learning arrangement, where an individual user group collaboratively provides additional sub-region images and corresponding classifications. These additional images may come from personal reviews, post-spraying, a series of collected sub-region images, and the corresponding classifications assigned by the neural network. If the neural network presents an incorrect classification, then among the individual user group, the reviewer / examiner of the previously classified images will provide the following for each misclassified sub-region image (for example): a copy of the sub-region (or entire) image data, the incorrect classification presented by the neural network, and (optionally) the correct classification (proposed by the user). In addition, multiple entire image frames are saved, constituting the entire image containing the misclassified sub-region image, as well as the entire images before / after the misclassified sub-region image. In Figure 4 the illustrative example, the first farmer 400, the second farmer 410, and the third farmer 420 each operate Figure 1 an instance of the system shown. The identified farmers represent thousands of farmers operating Figure 1 the system shown. Each of the farmers 400, 410, and 420 operates its respective system to perform spot spraying based on a local instance of a neural network configured according to global training conducted at a centralized neural network training facility 450.

[0026] According to one illustrative example, the facility 450 includes a database 460 that includes millions of collected sub-region images that make up a training image set 465. The training image set 465 is provided to a model training pipeline 470. The output of the model training pipeline 470 is a trained model 480 that contains a neural network configuration for a neural network instance to be integrated into the respective automated spot weed spraying systems of the farmers 400, 410, and 420. As shown in one illustrative example, the training model output from the model training pipeline 470 stored in the training model configuration library 480 is characterized for a specific type of crop (e.g., wheat, corn, potato, soybean, etc.) or any other appropriate grouping for classifying the received image set. Creating different neural network configurations for specific crops (or any other differentiating feature, such as soil type) can improve the robustness of the resulting neural network classifications presented in operation by the instances of the system 100 operated by the farmers 400, 410, and 420. Of particular note is the execution of machine learning-based robust training to generate a "base model" that is used to configure the neural network to classify input images from different crop fields (with various crops, soil types / conditions, etc.). A multi-dimensional map (database) helps generate the base model, which is configured to maintain a geographical record of the source of the training images (as well as environmental parameters describing the characteristics of the crop fields under which the training sub-region images corresponding to the stored training instances were acquired).

[0027] In addition to providing the current output feature parameter value (or a set of values), the output at the machine learning system level is performed by the model training pipeline 470 in the form of the trained model 480. Thus, according to one illustrative example, the model training pipeline 470 combines and utilizes an initial training configuration at the machine learning system level. For example, such training includes using an initial training image set with known classifications to configure the trained model 480 (e.g., setting the weights and / or coefficients of neural network nodes and layers). The initial configuration of the trained model 480 continues on an iterative basis until the configuration exhibits a difference between a set of generated outputs and expected outputs that falls within a specified minimum difference threshold. Based on the known output points provided by the training set, the specified minimum difference threshold can be set based on individual differences and / or aggregate total differences (Δ). Thereafter, based on specific requirements and / or experience using the initial / current machine learning-level configuration, the configuration can be updated based on additional training points or different sets of difference thresholds. According to Figure 4 the illustrative example, the annotation results of the currently existing version of the trained model 480 currently installed / executed on the systems of farmers 400, 410, and 420 contribute to such an update of the initially provided classification model.

[0028] Thus, Figure 4 the illustrative example provided in

[0029] is a supervised, dynamically configured (through feedback provided by the user / farmer) machine learning system. More specifically, in the above illustrative example, each processor 110 running within an instance of the system 100 operated by an individual farmer contains a neural network. However, various alternative machine learning architectures / types can be considered, including: support vector machines, linear regression, logistic regression, naive Bayes, linear discriminant analysis, decision trees, k-nearest neighbor algorithms, and similarity learning, as well as self-supervised methods such as contrast learning methods.

[0029] As Figure 5 shown, an exemplary data / decision flow is summarized based on a detailed description of the structure and operation of the system 100 corresponding to one of the systems operated by the farmer (e.g., the farmer 400 system). According to one illustrative example, the system includes a data acquisition section 500 that includes a pixel image data set corresponding to the full field of view region 108 of the camera 102. According to one illustrative example, the acquired pixel image data set is transmitted to the machine learning-based training image classifier subsystem 510.

[0030] The sub-region image extraction stage 512 of the classifier subsystem 510 extracts sub-region image instances from the received instance of the pixel image dataset corresponding to the full field of view region 108 (these instances can overlap with other sub-region image instances according to the spray area nozzle coverage of each nozzle corresponding to one of the sub-region image examples), where each sub-region image corresponds to the coverage area of a corresponding nozzle, for applying herbicide to weeds - or generally any application liquid for treating the field crop characteristic field of view (such as pesticides, fertilizers, etc.). Thereafter, each resulting sub-region image is presented to the machine learning trained image classifier stage 515 of the classifier subsystem 510.

[0031] Thereafter, the classifier stage 515, which includes a configured / trained neural network, classifies the provided sub-region image instance ("spray" or "no spray") and associated uncertainty scores. The sub-region image instance corresponds to a specific area of the field where, during a specific activation period, the corresponding activated nozzle dispenses herbicide over the specific area of the field. The activation period can be determined by the image / spray activation synchronization subsystem 520 (described below).

[0032] According to an illustrative example, the classifier stage 515 includes a preprocessing component 516 that performs digital data processing (such as filtering, parameterization, etc.) on the received input sub-region image instance generated by the sub-region image extraction stage 512. According to an illustrative example, the preprocessing component 516 converts the acquired data into a form suitable for further processing by the neural network stage 517 of machine learning training. In the simplest form, this preprocessing can simply pass the sensor data in digital form (but otherwise unmodified) to memory for subsequent processing by the neural network stage 517 of machine learning training. The preprocessing performed by the preprocessing component 516 can be a simple arithmetic operation applied to each value (multiplying each pixel value by a scalar value), or a complex algorithm based on multiple sensor data values or even based on previously acquired image data. For example, an image can be preprocessed by averaging it with several previously acquired images. Another example of preprocessing is performing a Fourier transform on the image data to convert the received time-domain data into the frequency domain. There may be multiple preprocessing steps applied to the image data instances acquired over time.

[0033] The output of the preprocessing component 516 is provided to the neural network stage 517 of the machine learning training in a pre-established form. By way of a specific example, the camera 102 may generate a sub-image with 6 million pixels (or 2 million pixels), but the neural network stage 517 is trained and operates on images with a smaller number of pixels (e.g., 1 million pixels, 500,000 pixels, 40,000 pixels, etc.). In this case, the preprocessing component 516 downsamples the data set to meet the 1 million pixel limit of the neural network stage 517.

[0034] In a specific example, the neural network stage 517 of the machine learning training includes an artificial neural network (e.g., a convolutional neural network). The artificial neural network includes a set of configurable / adjustable processing layers (each layer includes a set of computing nodes). The neural network incorporates / embodies a set of algorithms that are designed to perform an entire image classification operation on a set of input parameters corresponding to the preprocessing output of the preprocessing component 516. For example, the data input of the neural network stage 517 of the machine learning training is an input array whose dimensions (length × width × 3) correspond to the color (red, green, and blue) pixels of the normalized color (red, green, and blue) input sub-region image to be classified.

[0035] The output of the neural network stage 517 of the classifier stage 515 includes a set of outputs that assign a value to each input digital image. More specifically, each output in the set of outputs corresponds to a specific one of a set of different classifications. Additionally, in an illustrative example, the classification of a specific classification input sub-region image is set according to one of the classification-specific outputs with the highest value. Furthermore, the neural network analyzes the combination of the output values to generate a confidence value for the determined classification. The confidence value represents the degree of certainty of the "winning" classification output of the set of classification-specific outputs of the neural network. For example, in a simple binary classification output arrangement, a set of classification outputs has separate outputs corresponding to "spray" and "not spray". The certainty value corresponding to the classification generated by the set of classification output values is based on, for example, the relative (or absolute) magnitudes of the values of the set of classification outputs issued by the neural network.

[0036] For example, neural network stage 517 includes a combination of computing units and data structures that are organized as: an input layer, a plurality of internal / hidden layers, and an output layer. The output of the preprocessing component 516 provides at least a portion of the information content of the input layer. The topology of the neural network stage 517 of the classifier stage 515 includes a plurality of internal layers as well as the number of computing nodes within each internal layer and the inter-layer connections between the nodes of adjacent layers of the artificial neural network. For example, during the training process of the neural network stage 517, the configuration of the machine learning stage includes specifying a set of coefficients (weights) for each internal layer of the artificial neural network. More specifically, each layer is populated by a weighted combination of the neurons of the previous layer. A non-linear activation function is applied to the resulting value in each node to determine the output value to be passed to the next layer.

[0037] Within a layer of the artificial neural network of the neural network stage 517, for example, the neurons of the previous layer are combined in a weighted manner. Then, the output value of each neuron is a non-linear operation applied to that combination. It is readily understood by those familiar with the topology of the artificial neural network that each internal layer includes a set of nodes, where each node receives a weighted contribution from each node of the previous layer and provides a weighted output (based on each node's specification) to each node of the next internal layer of the artificial neural network. A non-linear operation is performed on the weighted inputs received from the previous layer within each node.

[0038] A set of weight values 518 provides the configuration of the neural network nodes of the neural network stage 517. In particular, the values in each layer of the above artificial neural network are provided by the weight values 518. During the training of the classifier stage 515, each individual value of the weight values 518 is established using a set of input sub-region images and their corresponding correct classifications (sprayed / not sprayed).

[0039] An important aspect of the operation of the image classifier stage 515 (more specifically, the neural network stage 517) is the training process. Training includes determining the weight values of the neural network stage 517 of the image classifier stage 515. As in the operating case, the neural network stage 517 presents output classification values for a series of preprocessed data. But in this case, each output classification value is accompanied by a label that includes one or more parameters associated with that classification value. Since the correct weights are not yet known, an initial starting guess value is used. During the training process, the weights are adjusted until the computed classifications generated by the neural network stage 517 accurately track the actual (observed) classifications of the input sub-region images, i.e., the neural network stage 518 operates within a predetermined performance range. For various reasons, retraining is also performed. Data collected in memory (e.g., misclassified sub-region images submitted by farmers / users) can be used as labeled data to optimize the operation of the neural network stage 517.

[0040] According to an illustrative example, the trained spraying image classification model is retrained / updated with the labeled (classified) validation classification images provided by farmers (such as farmers 400, 410, 420, etc.) during the operation of the respective systems. By way of a specific example, during the operation of the farmer 400 on the operating system 100, for any sub-region image whose generated classification does not meet the preset deterministic value threshold, the sub-region image is stored in the review buffer to review / confirm the classification result assigned to the sub-region image in the following two ways: directly / automatically transmitted to the cloud-based training facility 450 through the classifier level 515, or alternatively / additionally by operating the picture annotation subsystem 525 under the instruction of the user / observer / farmer. The confirmed classification of the specific sub-region image is used to supplement the classification model for machine learning training (at the cloud-based training facility 450 described above).

[0041] Regarding the synchronization subsystem 520, for example, during the real-time operation of the system 100, for a specific sub-region image with a "spraying" classification generated by the classifier level 515, various operating parameters are applied to generate a spraying activation timing. Examples of the operating parameters used by the synchronization subsystem include but are not limited to the machine travel speed, nozzle height, linear distance between the nozzle spraying area and the area position corresponding to the classification image, etc.

[0042] In addition, according to the disclosure of the present invention, an annotation subsystem 525 is provided for the user to review the classification results assigned by the classifier subsystem 510 to the sub-region image instances. The user can use the editing function of the annotation subsystem 525 to retrieve the sub-region image and the corresponding classification result, edit / correct the assigned classification of a specific image, and then report the corrected result to the Figure 4 associated centralized training facility of the kind proposed / described.

[0043] In summary, the present invention provides a number of technical problems and corresponding solutions. These technical problems and solutions improve the weed spot spraying device by judging whether there are weeds in the sub-region images through machine learning-based training (neural network) processors and real-time classification of sub-region images, where the sub-region images are substantially consistent with the lateral (width) dimension of the corresponding nozzle, and the nozzle is configured to selectively activate to disperse herbicides when weeds are detected during the spot spraying operation.

[0044] As shown in the disclosure of the present invention, a well-trained system has significant advantages. For this reason, according to the present invention, annotating training images (providing classifications) is a very simple task that only requires the user to assign classification values to each training instance.

[0045] In an exemplary scenario, a farmer can quickly review (and label with the correct classification) a large number of sub-region images collected. Thereafter, in a collaborative training environment, the labeled sub-region images are used to improve the classification model - by performing training using the updated classification model by a neural network-based processor.

[0046] Another technical challenge involves ensuring that the nozzles can be correctly activated to spray the crop field areas corresponding to the sub-region images classified as having weeds. The present invention proposes a relatively straightforward solution, where the nozzles are activated before passing over the target crop field areas, and then the nozzles are kept in the activated state (spraying herbicide) for a period of time to ensure that the target field portions are completely sprayed. In this regard, the duration can be extended / shortened according to the detected travel speed of the nozzles along the field. Importantly, the disclosed system does not require precise localization of the weeds (within the camera's field of view), but only classifies the entire sub-field area corresponding to the width of the spraying area (the lateral extent across the crop surface) perpendicular to the travel direction of the nozzles. In addition, the camera 102 is mounted on the same boom as the nozzles 104a-d to ensure that the field of view of the camera 102 is adjusted as the direction of the boom with the nozzles 104a-d mounted thereon changes. The boom can be oriented horizontally with respect to the ground, or the boom can be oriented perpendicular to the ground.

[0047] In addition, the overlapping sub-region images (such as the overlap of the spraying areas 106a-d of the spraying nozzles 104a-d) ensure that if one nozzle misses a weed (e.g., due to strong crosswinds), the weed will be treated by an adjacent nozzle. In addition, if a sub-region image is misclassified, the weed may still be detected by another laterally positioned sub-region image on the same image. In addition, for any given area passed by the system 100, the system acquires multiple images. Therefore, when the system 100 advances along the crop line in the field, there are several opportunities to detect weeds when depicting weeds in multiple continuously acquired images.

[0048] Another challenge is the need to provide a robust model for classifying various weed images in various crop and soil environments. To this end, rather than training the neural network during the development phase, it is continuously trained and updated. The network is improved by collecting interesting images during the farmer's field operations. The acquisition of such further training images and corresponding classifications is managed by targeted updates, guided by the included metadata that characterizes the conditions / environments for acquiring the training instances. Such metadata may include any of the following types: geographical location, date, weather (lighting), climate, soil conditions / color, soil type, etc. The metadata helps to build a robust model by retraining features / conditions that have not been previously model-trained, ensuring the addition / incorporation of classified training images.

[0049] When determining when and where to obtain additional training images, various input data sources can be used, including: yield maps, fertilizer maps, soil maps, etc. In addition, metadata based on weather forecasts and models can be used to predict areas where specific rare situations may occur. Thereafter, the central management facility can create an image acquisition plan and send the plan to specific sprayers when the farmer starts the spot spraying process on the farmland. The acquisition of the new classification dataset is achieved through an automated management system to quickly obtain new training images without user intervention, and then configure and deploy an updated model for the neural network executed by the processor 110. For example, according to Figure 4 In the illustrative example provided, an indication problem (e.g., misclassification) of the neural network configuration can be sent to the central management facility. Update the model and download the updated neural network configuration to the farmer's processor 110 via a network connection.

[0050] Although the above discussion is directed to classifying sub-regions where there may or may not be weeds, the disclosed technical solution can alternatively be trained to classify sub-region images as diseased / non-diseased through machine learning-based neural network image processing.

[0051] All references cited herein, including publications, patent applications, and patents, are incorporated herein by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and set forth in its entirety.

[0052] Unless otherwise specified herein or clearly contradicted by the context, the terms "a", "an", "the", and "at least one" and similar expressions used in the context of describing the present invention (particularly in the context of the claims) shall be construed to cover both the singular and plural forms. Unless otherwise specified herein or clearly contradicted by the context, the phrase "at least one", followed by a list of one or more items (e.g., "at least one of A and B"), shall be construed to mean either one item selected from the listed items (A or B) or any combination of two or more of the listed items (A and B). Unless otherwise specified, the terms "comprising", "having", "including", and "containing" shall be construed as open-ended terms (i.e., meaning "including but not limited to"). Unless otherwise specified herein, the recitation of a numerical range herein is merely a shorthand method for individually referring to each separate numerical value within that range, and each separate numerical value is deemed to be incorporated into the specification as if it were individually recited. Unless otherwise specified herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. The use of any and all examples or exemplary language (e.g., "such as") provided herein is merely intended to better illustrate the invention and, unless otherwise required, does not limit the scope of the invention. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the invention.

[0053] Preferred embodiments of the present invention are described herein, including the best mode known to the inventors for practicing the invention. Variations of these preferred embodiments may become apparent to those of ordinary skill in the art after reading the above description. The inventors expect skilled artisans to appropriately employ such variations, and the inventors desire that the invention be practiced in a manner different from that specifically described herein. Accordingly, within the scope permitted by applicable law, the present invention encompasses all modifications and equivalents of the subject matter recited in the appended claims. Additionally, unless otherwise specified herein or clearly contradicted by the context, any combination of the above elements in all possible variations thereof is included within the scope of the present invention.

Claims

1. A method for targeted weed spraying based on classification values, where the classification values are generated from sub-region images by a processor applying a machine learning-based training model to the sub-region images, the method comprising: Obtaining a full-field region image of the crop ground surface by a camera; Extracting a sub-region image corresponding to a nozzle from the full-field region image, the nozzle being positioned to provide a spraying area extending over a portion of the crop ground surface depicted in the sub-region image; Generating a classification of the sub-region image by the processor according to a machine learning-based training model; And Selectively activating the nozzle according to the classification of the sub-region image.

2. The method according to claim 1, wherein the generating of the classification is performed by a neural network integrated in the processor.

3. The method according to claim 2, wherein the neural network has an input array, the size of the input array corresponding to the pixel size of the sub-region image.

4. The method according to claim 3, wherein the neural network has an output quantity corresponding to a set of potential classifications of the sub-region image.

5. The method according to claim 1, wherein the method is performed on a system including a plurality of nozzles, the plurality of nozzles having overlapping spraying areas between two adjacent nozzles among the plurality of nozzles.

6. The method according to claim 1, wherein the camera is mounted on the same physical mounting structure as the nozzle.

7. The method according to claim 1, wherein the sub-region image is rectangular.

8. The method according to claim 1, wherein the sub-region image is a rectangle generated from a sub-region of an initial trapezoidal image acquired by a forward-looking camera.

9. The method according to claim 1, wherein, Selectively activating the nozzle includes keeping the activated nozzle in an open state for a period of time related to the speed of the nozzle along the travel route.

10. The method according to claim 1, the method further comprising adjusting the opening time of the nozzle according to the machine speed.

11. The method according to claim 1, the method further comprising providing sample sub-region images and corresponding verification classifications for the sample sub-region images to a training facility for the machine learning-based training model.

12. The method according to claim 11, wherein the sample sub-region images are provided in real time during the targeted spraying operation.

13. The method according to claim 12, wherein, Providing the sample sub-region images to an examiner to verify that the assigned classification is performed on an automated basis.

14. The method according to claim 13, wherein the automated basis includes comparing a confidence value for classifying the sample sub-region image with a threshold confidence value.

15. The method according to claim 1, wherein the generating of the classification is performed in a cloud-based server system.

16. The method according to claim 1, the method further comprising providing the generated classification and the associated sub-region image to an examiner for annotation.

17. The method according to claim 1, wherein the generating of the classification is performed according to a base model.

18. The method according to claim 17, wherein, The base model is provided with an associated training coverage map that indicates the extent of a particular type of training image for which machine learning training has been performed.

19. A weed spot spraying system configured to perform a weed spot spraying method based on a classification value that is generated by a processor applying a machine learning-based training model to a sub-region image from the sub-region image, the system comprising: A camera; A nozzle assembly that includes a nozzle; And A processor; Wherein the weed spot spraying method comprises: Obtaining, by the camera, an image of a full field of view of the crop surface; Extracting, from the full field of view image, a sub-region image corresponding to the nozzle, the nozzle being positioned to provide a spraying area extending over a portion of the crop surface depicted in the sub-region image; Generating, by the processor, a classification of the sub-region image according to a machine learning-based training model; and Selectively activating the nozzle according to the classification of the sub-region image.

20. The weed spot spraying system according to claim 19, wherein the system further comprises a network communication interface configured to communicate with a networked facility to provide training messages, the training messages comprising: Sub-region image instances; Metadata describing the environment in which the camera obtained the sub-region image instances; And Confirmed features of the sub-region images.

Images