Plant group recognition

Abstract

An agricultural machine moves through a field, the agricultural machine including an image sensor that captures images of plants in the field. A control system accesses the captured images and applies the images to a machine learning plant recognition model. The plant recognition model identifies pixels that represent plants and classifies the plants into plant groups (e.g., plant species). The identified pixels are labeled as the plant groups, and the locations of the pixels are determined. The control system actuates a treatment mechanism based on the identified plant groups and locations. Further, the images from the image sensor and the plant recognition model can be used to generate a plant recognition map. A plant recognition map is a map of the field that indicates the locations of the plant groups identified by the plant recognition model.

Description

Technical Field

[0001] This disclosure relates to identifying and processing plants in a field, and more specifically, to identifying a set of pixels in an image to represent plants in a group of plants and processing the plants based on the identified group of plants. Background Technology

[0002] It is difficult to apply treatments to individual plants rather than large areas of a field. To treat plants individually, farmers may apply treatments manually, for example, but this has proven labor-intensive and costly when carried out on an industrial scale. In some cases, agricultural systems use imaging techniques (e.g., satellite imaging, color imaging, thermal imaging, etc.) to identify and treat plants in the field. These systems have proven to be less effective at correctly identifying individual plants from plant groups comprising multiple species and treating those individual plants accordingly. Summary of the Invention

[0003] Agricultural machines are configured to move through fields and selectively process individual plants within the fields using various processing mechanisms. The agricultural machine processes individual plants by identifying their species. For this purpose, the agricultural machine includes image sensors that capture images of the plants in the field. The control system of the agricultural machine can execute a plant recognition model configured to identify pixels representing one or more species in the image. The location of the identified species in the image is also determined. Based on the identified species and their location in the image, the agricultural machine selectively processes the plants as it moves through the field.

[0004] In addition to identifying plants by their species, plant identification models can also identify plants based on other groupings (such as genus, family, plant characteristics (e.g., leaf shape, size, or color)) or the treatment to be applied. In some implementations, these plant groupings are customizable. This allows users of agricultural machinery to create plant groupings suitable for specific plants growing in the field. For example, if a user wants to treat hogweed, they can instruct the plant identification model to identify and classify the plant as "hogweed" or "non-hogweed." In some implementations, plant identification models are specifically trained to identify various types of weeds.

[0005] In some implementations, plant identification maps of fields are generated using images from image sensors and plant identification models. A plant identification map is a map of a field indicating the location of plant groups identified by the plant identification model. This map may include additional data providing insights into the cultivation and maintenance of the field, such as the total area covered by each plant group, the total area of ​​the field, the number of plants identified in each plant group, and the areas of the field treated by agricultural machinery. Attached Figure Description

[0006] Figure 1A An isometric view of an agricultural machine according to an example embodiment is shown.

[0007] Figure 1B A top view of an agricultural machine according to an example embodiment is shown.

[0008] Figure 1C An isometric view of an agricultural machine according to a second example embodiment is shown.

[0009] Figure 2 A cross-sectional view of an agricultural machine including a sensor according to a first example embodiment is shown, the sensor being configured to capture images of one or more plants.

[0010] Figure 3A An image captured according to an example implementation is shown.

[0011] Figure 3B A group of plant images generated based on captured images, according to a first example implementation, is shown.

[0012] Figure 3C A group of plant images generated based on captured images, according to a second example implementation, is shown.

[0013] Figure 3D A group of plant images generated based on captured images, according to a third example implementation, is shown.

[0014] Figure 4 A representation of a plant identification model according to an example implementation is shown.

[0015] Figure 5 A table showing a set of training images according to an example implementation is presented.

[0016] Figure 6A and Figure 6B The performance metrics of a plant recognition model trained using a set of training images, according to an example implementation, are shown.

[0017] Figures 7A to 7D Performance metrics of a plant identification model that identifies different plant groups according to instructions of an example implementation are shown.

[0018] Figure 8 This is a flowchart illustrating a method for processing plants using a plant identification model according to an example implementation.

[0019] Figure 9 This is a flowchart illustrating a method for training a plant recognition model according to an example implementation.

[0020] Figure 10 A plant identification map according to an example implementation is shown.

[0021] Figure 11 This is a schematic diagram illustrating a control system according to an example embodiment.

[0022] The accompanying drawings depict various embodiments for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein can be employed without departing from the principles described herein. Detailed Implementation

[0023] I. Introduction

[0024] Agricultural machinery includes one or more sensors that capture information about plants as the machine moves through a field. The machine includes a control system that processes the information acquired by the sensors to identify the plants. Numerous examples exist of agricultural machines processing visual information acquired by image sensors coupled to the machine to identify and treat plants. For example, as described in U.S. Patent Application 16 / 126,842, filed September 10, 2018, entitled “Semantic Segmentation to identify and treat Plants in a field and verify the Plant Treatments”.

[0025] II. Plant Treatment System

[0026] Agricultural machines for identifying and processing plants can have a variety of configurations, some of which are described in more detail below. For example, Figure 1A It is an isometric view of agricultural machinery. Figure 1B yes Figure 1A A top view of agricultural machinery. Figure 1C This is the second implementation of the agricultural machinery. Other implementations of the agricultural machinery are also possible. Figures 1A to 1C The agricultural machine 100 shown includes a detection mechanism 110, a processing mechanism 120, and a control system 130. The agricultural machine 100 may also include a mounting mechanism 140, a verification mechanism 150, a power supply, a digital memory, a communication device, or any other suitable components. The agricultural machine 100 may include more or fewer components than those described herein. Furthermore, the components of the agricultural machine 100 may have different or additional functions than those described below.

[0027] Agricultural machinery 100 is used to treat one or more plants 102 within a geographic area 104. Typically, the treatment is used to regulate plant growth. The treatment may be applied directly to a single plant 102 (e.g., a hygroscopic material), but may alternatively be applied directly to multiple plants, indirectly to one or more plants, applied to the plant's associated environment (e.g., soil, atmosphere, or other suitable parts of the plant's environment adjacent to or connected to environmental factors (e.g., wind), or otherwise applied to the plant. Treatments that can be applied include necrosis of the plant, necrosis of a portion of the plant (e.g., pruning), regulation of plant growth, or any other suitable plant treatment. Necrosis of the plant may include: removing the plant from a supporting substrate 106, burning a portion of the plant, applying a treatment concentration of working fluid (e.g., fertilizer, hormone, water, etc.) to the plant, or treating the plant in any other suitable manner. Regulation of plant growth may include: promoting plant growth, promoting the growth of plant parts, inhibiting (e.g., delaying) the growth of the plant or a portion of the plant, or otherwise controlling plant growth. Examples of regulating plant growth include: applying growth hormones to plants, applying fertilizers to plants or substrates, applying disease or insecticides to plants, electrically stimulating plants, watering plants, pruning plants, or otherwise treating plants. Plant growth can also be regulated by pruning, necrosis, or other treatments of adjacent plants.

[0028] Plant 102 can be a crop, but alternatively it can be weeds or any other suitable plant. The crop can be cotton, but alternatively it can be lettuce, soybeans, rice, carrots, tomatoes, corn, broccoli, cabbage, potatoes, wheat, or any other suitable cash crop. The field using the system is an outdoor field, but alternatively it can be a greenhouse, laboratory, growing room, container group, machine, or any other suitable environment. Plants can be grown in one or more plant rows (e.g., plant beds) where the plant rows are parallel, but alternatively they can be grown in a set of plant pots, where the plant pots can be arranged in rows, matrices, or random distribution, or in any other suitable configuration. Crop rows are typically spaced 2 inches to 45 inches apart (e.g., determined from the longitudinal row axis), but alternatively they can be spaced at any suitable distance, or have variable spacing between multiple rows.

[0029] Plants 102 within each field, row, or section typically comprise the same type of crop (e.g., the same genus, species, etc.), but alternatively may include multiple crops (e.g., a first crop and a second crop), both of which are to be treated. Each plant 102 may include a stem arranged above (e.g., above) the substrate 106, supporting the plant's branches, leaves, and fruits. Each plant may also include a root system connected to the stem, located below (e.g., underground) the substrate plane, supporting the plant's position and absorbing nutrients and water from the substrate 106. Plants may be vascular plants, non-vascular plants, woody plants, herbaceous plants, or any suitable type of plant. Plants may have a single stem, multiple stems, or any number of stems. Plants may have taproots or fibrous root systems. The substrate 106 is soil, but alternatively may be sponge or any other suitable substrate.

[0030] The detection mechanism 110 is configured to identify plants for treatment. Therefore, the detection mechanism 110 may include one or more sensors for identifying plants. For example, the detection mechanism 110 may include a multispectral imaging device, a stereo imaging device, a CCD imaging device, a single-lens imaging device, a CMOS imaging device, a hyperspectral imaging system, a LIDAR system (light detection and ranging system), a depth sensing system, a power meter, an IR imaging device, a thermal imager, a humidity sensor, a light sensor, a temperature sensor, or any other suitable sensor. In one embodiment, and described in more detail below, the detection mechanism 110 includes an array of image sensors configured to capture images of the plants. In some example systems, the detection mechanism 110 is mounted to the mounting mechanism 140 such that the detection mechanism 110 traverses the geographic location before the treatment mechanism 120 as the agricultural machine 100 moves through the geographic location. However, in some embodiments, the detection mechanism 110 traverses the geographic location substantially simultaneously with the treatment mechanism 120. In embodiments of the agricultural machine 100, the detection mechanism 110 is statically mounted to the mounting mechanism 140 relative to the direction of travel 115, close to the treatment mechanism 120. In other systems, the detection mechanism 110 can be integrated into any other component of the agricultural machine 100.

[0031] The treatment mechanism 120 is used to treat the identified plant 102. As the agricultural machine 100 moves in the direction of travel 115, the treatment mechanism 120 treats the treatment area 122. The effects of the treatment may include: plant necrosis, plant growth stimulation, partial plant necrosis or removal, partial plant growth stimulation, or any other suitable treatment effect as described above. The treatment may include: removing the plant 102 from the substrate 106, cutting the plant (e.g., slicing), burning the plant, electrically stimulating the plant, applying fertilizer or growth hormone to the plant, watering the plant, applying light or other radiation to the plant, injecting one or more working fluids into the substrate 106 adjacent to the plant (e.g., within a threshold distance from the plant), or otherwise treating the plant. In one embodiment, the treatment mechanism 120 is an array of spraying treatment mechanisms. The treatment mechanism 120 may be configured to spray one or more of the following: herbicides, fungicides, water, or insecticides. The processing unit 120 can operate between a standby mode where the processing unit 120 does not perform any processing and a processing mode where the processing unit 120 is controlled by the control system 130 to perform processing. However, the processing unit 120 can operate in any other suitable number of operating modes.

[0032] Agricultural machinery 100 may include one or more processing mechanisms 120. Processing mechanisms 120 may be fixed (e.g., statically coupled) to or attached to a mounting mechanism 140 relative to a detection mechanism 110. Alternatively, processing mechanisms 120 may be rotated or translated relative to the detection mechanism 110 and / or the mounting mechanism 140. In one variant, agricultural machinery 100 includes a single processing mechanism, wherein processing mechanism 120 is actuated or agricultural machinery 100 is moved to align the active area 122 of processing mechanism 120 with a target plant 102. In a second variant, agricultural machinery 100 includes an assembly of processing mechanisms, wherein a processing mechanism 120 (or a sub-component of processing mechanism 120) of this assembly is selected to process the identified plant 102 or a portion of a plant in response to plant identification and relative to the plant position of the assembly. Figures 1A to 1C In the third variant shown, the agricultural machine 100 includes an array of processing units 120, wherein the processing units 120 are actuated or the agricultural machine 100 is moved to align the active area 122 of the processing units 120 with the target plant 102 or plant segment.

[0033] Agricultural machinery 100 includes a control system 130 for controlling the operation of system components. The control system 130 can receive information from and / or provide input to the detection mechanism 110, verification mechanism 150, and processing mechanism 120. The control system 130 can be automated or user-operated. In some embodiments, the control system 130 can be configured to control operating parameters of the agricultural machinery 100 (e.g., speed, direction). The control system 130 also controls operating parameters of the detection mechanism 110. Operating parameters of the detection mechanism 110 may include processing time, position and / or angle, image capture interval, image capture settings, etc., of the detection mechanism 110. The control system 130 can be a computer, as described below. Figure 11 In more detail, the control system 130 may apply one or more models to identify one or more plants in a field. The control system 130 may be coupled to the agricultural machine 100, allowing a user (e.g., a driver) to interact with it. In other embodiments, the control system 130 is physically removed from the agricultural machine 100 and communicates wirelessly with system components (e.g., detection mechanism 110, processing mechanism 120, etc.). In some embodiments, the control system 130 is a general term encompassing multiple networked systems distributed in different locations (e.g., systems on the agricultural machine 100 and systems at remote locations). In some embodiments, one or more processes are performed by another control system. For example, the control system 130 receives plant processing instructions from another control system.

[0034] In some configurations, the agricultural machine 100 includes a mounting mechanism 140 for providing mounting points for system components. In one example, the mounting mechanism 140 statically holds and mechanically supports the positions of the detection mechanism 110, the processing mechanism 120, and the verification mechanism 150 relative to its longitudinal axis. The mounting mechanism 140 is a base frame or frame, but may alternatively be any other suitable mounting mechanism. Figures 1A to 1C In this embodiment, the mounting mechanism 140 is mounted on the main body of the agricultural machine 100 in the positive and negative x directions (in... Figures 1A to 1C The mounting mechanism 140 extends outward from the orientation shown, such that it is approximately perpendicular to the direction of travel 115. Figures 1A to 1C The mounting mechanism 140 includes an array of processing mechanisms 120 laterally positioned along the mounting mechanism 140. In an alternative configuration, the mounting mechanism 140 may be absent, may be alternatively positioned, or may be integrated into any other component of the agricultural machine 100.

[0035] Agricultural machinery 100 includes a first set of coaxial wheels and a second set of coaxial wheels, wherein the axis of rotation of the second set of wheels is parallel to the axis of rotation of the first set of wheels. In some embodiments, each wheel in each set is arranged along opposite sides of a mounting mechanism 140 such that the axis of rotation of the wheel is substantially perpendicular to the mounting mechanism 140. Figures 1A to 1C In this system, the axis of rotation of the wheels is generally parallel to the mounting mechanism 140. In alternative embodiments, the system may include any suitable number of wheels with any suitable configuration. The agricultural machine 100 may also include a coupling mechanism 142, such as a hook, for movably or statically coupling to a drive mechanism, such as a tractor, more preferably to the rear of the drive mechanism (so that the agricultural machine 100 can be towed behind the drive mechanism), but alternatively attached to the front of the drive mechanism or coupled to the side of the drive mechanism. Alternatively, the agricultural machine 100 may include a drive mechanism (e.g., a motor and transmission coupled to a first set of wheels and / or a second set of wheels). In other example systems, the system may have any other means of traversing the field.

[0036] In some configurations, the agricultural machine 100 also includes a verification mechanism 150 for recording measurements of the environment surrounding the agricultural machine 100. The agricultural machine can use these measurements to verify or determine the extent of plant treatment. The verification mechanism 150 records measurements of a geographic area previously measured by the detection mechanism 110. The verification mechanism 150 also records measurements of a geographic area containing plants treated by the treatment mechanism 120. The measurements from the verification mechanism 150 can also be used to empirically determine (e.g., calibrate) the operating parameters of the treatment mechanism to achieve the desired treatment effect. The verification mechanism 150 may be substantially similar to (e.g., of the same type) as the detection mechanism 110 or may be different from the detection mechanism 110. In some embodiments, the verification mechanism 150 is positioned away from the detection mechanism 110 relative to the direction of travel, with the treatment mechanism 120 positioned between the verification mechanism 150 and the detection mechanism 110, such that the verification mechanism 150 traverses the geographic location after the treatment mechanism 120 has traversed it. However, the mounting mechanism 140 can maintain the relative positions of the system components in any other suitable configuration. In other configurations of the agricultural machine 100, the verification mechanism 150 may be included in other components of the system.

[0037] In some configurations, the agricultural machine 100 may also include a power source for powering system components, including a detection mechanism 110, a control system 130, and a processing mechanism 120. The power source may be mounted to a mounting mechanism 140, removably coupled to a mounting mechanism 140, or may be separate from the system (e.g., located on a drive mechanism). The power source may be a rechargeable power source (e.g., a rechargeable battery pack), an energy harvesting power source (e.g., a solar system), a fuel-consuming power source (e.g., a fuel cell stack or internal combustion system), or any other suitable power source. In other configurations, the power source may be integrated into any other component of the agricultural machine 100.

[0038] In some configurations, the agricultural machine 100 may also include a communication device for communicating (e.g., sending and / or receiving) data between the control system 130 and a set of remote devices. The communication device may be a Wi-Fi communication system, a cellular communication system, a short-range communication system (e.g., Bluetooth, NFC, etc.), or any other suitable communication system.

[0039] Figure 2 A cross-sectional view of an agricultural machine including sensors, configured to capture images of one or more plants, is shown according to some example embodiments. The agricultural machine 200 may be similar to the one described above. Figures 1A to 1C Any agricultural machinery described. Figure 2 In this embodiment, the agricultural machine includes a sensor 210. Here, sensor 210 is a camera device (e.g., an RGB camera, a near-infrared camera, an ultraviolet camera, or a multispectral camera), but it can also be another type of image sensor suitable for capturing images of plants in a field. The agricultural machine 200 may include additional sensors mounted along the mounting mechanism 140. These additional sensors may be the same type as sensor 210 or a different type of sensor.

[0040] exist Figure 2 In this paper, sensor 210 has a field of view 215. In this context, field of view 215 is the angular range of the area captured by sensor 210. Therefore, the area captured by sensor 210 (e.g., field of view 215) may be affected by the properties (i.e., parameters) of sensor 210. For example, field of view 215 may be based on, for example, the size and focal length of a lens. Furthermore, field of view 215 may depend on the orientation of the sensor. For example, an image sensor with a tilted orientation can generate an image representing a trapezoidal region of a field, while an image sensor with a downward orientation can generate an image representing a rectangular region of a field. Other orientations are also possible.

[0041] exist Figure 2In this configuration, sensor 210 is tilted. More specifically, sensor 210 is mounted to the front region of mounting mechanism 140, and sensor 210 is tilted downwards toward the plant. As described herein, the downward tilt angle is defined as the angle between the z-axis and the negative y-axis. Field of view 215 includes plants 202a, 202b, 202c, and weeds 250. The distance between sensor 210 and each plant varies based on the plant's position and height. For example, plant 202c is farther from sensor 210 than plant 202a. Sensor 210 may be tilted in other directions.

[0042] Figure 2 A processing mechanism 120 of the agricultural machine is also shown. Here, the processing mechanism 120 is located behind the sensor 210 along the z-axis, but it could also be located in other positions. Regardless of orientation, the sensor 210 is positioned such that the processing mechanism 120 traverses the plant after the plant has passed through the field of view 215. More specifically, as the agricultural machine 100 moves toward the plant 202, the plant 202 will exit the field of view 205 at the edge 216 closest to the processing mechanism 120. The distance between the edge 216 and the processing mechanism 120 is the hysteresis distance. The hysteresis distance allows the control system 130 to capture and process an image of the plant before the processing mechanism 120 passes the plant. The hysteresis distance also corresponds to the hysteresis time. The hysteresis time is the amount of time that the agricultural machine has before the processing mechanism 120 passes the plant 202. The hysteresis time is the amount of time calculated based on the operating conditions of the agricultural machine (e.g., speed) and the hysteresis distance.

[0043] In some configurations, the processing mechanism 120 is positioned approximately in a straight line with the image sensor 210 along an axis parallel to the y-axis, but may deviate from this axis. In some configurations, the processing mechanism 120 is configured to move along the mounting mechanism 140 to process the identified plant. For example, the processing mechanism may move up and down along the y-axis to process the plant. Other similar examples are also possible. Furthermore, the processing mechanism 120 may tilt toward or away from the plant.

[0044] In various configurations, sensor 210 can have any suitable orientation for capturing images of plants. Furthermore, sensor 210 can be positioned at any suitable location along the mounting mechanism 140 so that it can capture images of plants as agricultural machinery travels through the field.

[0045] III. Plant Section Identification

[0046] As described above, the agricultural machine (e.g., agricultural machine 200) includes a sensor (e.g., sensor 210) configured to capture images of a portion of the field (e.g., field of view 215) as the agricultural machine moves through the field. In some embodiments, the image sensor is not coupled to the agricultural machine. The agricultural machine includes a control system (e.g., control system 130) configured to process the images and apply a plant recognition model to them. The plant recognition model identifies multiple groups of pixels representing plants and classifies these groups into plant groups (e.g., species). Plant recognition may additionally identify and classify pixels representing non-plant objects in the field (e.g., soil, rocks, debris in the field, etc.). The multiple groups of pixels are labeled as plant groups, and the location of the groups in the image is determined. The control system 130 may also be configured to generate and take processing actions for the identified plants based on the plant groups and locations.

[0047] A plant group comprises one or more plants and describes a characteristic or name common to those plants. Therefore, a plant recognition model not only identifies the presence of one or more plants in an image but can also classify each identified plant into the plant group that describes it. This allows agricultural machines to precisely perform agricultural actions targeting specific types and / or plant groups rather than a large number of different plants. Examples of plant groups include species, genus, family, plant characteristics (e.g., leaf shape, size, color, harmful and / or harmless), or corresponding plant treatments. In some implementations, a plant group includes subgroups. For example, if a plant group includes a weed group, the weed group may include weed subgroups, such as weed seed groups (e.g., hogweed and lambsquarters). In another example, a weed subgroup includes harmful weeds and harmless weeds (or less toxic weeds). Similar examples of crop groups are also possible. In these implementations, the plant recognition model may be instructed to classify plants into multiple groups (e.g., crops or weeds) and / or subgroups (e.g., hogweed or lambsquarters).

[0048] In some implementations, the plant recognition model is used to perform agricultural actions at a later point in time. For example, an image sensor (e.g., not on the agricultural machine) captures images of different parts of the field, and (e.g., using cloud processing) the plant recognition model is applied to the images to identify groups of plants in the field before the agricultural machine (e.g., machine 100) moves through the field and processes the plants. When it is time to process the plants in the field (e.g., later that day or another day), the plant groups and / or instructions for processing the identified plant groups can be provided to the agricultural machine. In other words, the image sensor may capture images of different parts of the field at a first point in time, and the agricultural machine may perform agricultural actions based on the images at a second point in time, which can occur at any time after the first point in time.

[0049] Figure 3A Example image 300 is accessed by a control system (e.g., captured by sensors on an agricultural machine). Image 300 includes pixels representing a first plant 305, a second plant 310, a third plant 315, and soil 320 in a field. Figure 3B This is an illustration of a plant group map 325A generated by applying a plant recognition model to an accessed image 300. A plant group map is an image that identifies the locations of one or more plant groups within an accessed image. Figure 3B The plant group image 325A in the image was generated by the plant recognition model using the bounding box method.

[0050] The bounding box method identifies multiple groups of pixels in an accessed image, including plant groups (e.g., a first plant group, a second plant group, and a third plant group), and places each group of pixels within a bounding box. For example, the plant recognition model identifies a group of pixels representing a first plant 305 and labels that group of pixels with a bounding box corresponding to the first plant group. Similarly, the plant recognition model surrounds the pixels of a second plant 310 with a second set of bounding boxes 335, and surrounds the pixels of a third plant 315 with a third set of bounding boxes 340. The group associated with the box depends on the grouping of the plant recognition model. Although Figure 3B The bounding box in the image is rectangular, but it can take other simple shapes, such as triangles or circles.

[0051] Since bounding boxes do not necessarily reflect the actual shape of the plants, the bounding box method can include pixels that do not represent plants (e.g., pixels representing soil 320, or pixels representing other plants). Because the treatment area can correspond to the bounding box area, the treatment selected for each plant group can be applied to unnecessary areas. For example, if a growth promoter is applied to the first plant group box 330, one of the second plants 310 may also be unintentionally treated with the growth promoter.

[0052] In other implementations, the plant recognition model performs pixel-by-pixel semantic segmentation to identify groups of plants in an image. Semantic segmentation can be faster and more accurate than bounding box methods. Figure 3C This is an example of a plant group map 325B generated using a semantic segmentation method to identify plant groups. Figure 3C Plant group image 325B shows a set of pixels 345 that may represent the first plant 305, multiple sets of pixels 350 that may represent the second plant 310, and a set of pixels 355 that may represent the third plant 315. Compared with bounding box methods, semantic segmentation can be more accurate because the identified multiple sets of pixels can take on any complex shape and are not limited to bounding boxes.

[0053] In other implementations, the plant recognition model performs instance segmentation to identify groups of plants in an image. Instance segmentation can be more accurate than semantic segmentation or bounding box methods. For example, instance segmentation can use a loss function that improves the detection of plants of various sizes. Furthermore, instance segmentation can provide data on the number of plants per unit area. Figure 3D This is an example of a plant group map 325C generated using a semantic segmentation method to identify plant groups. Figure 3D Plant group diagram 325C shows a set of pixels 360 that may represent the first plant 305, multiple sets of pixels 365 and 370 that may represent the second plant 310, and a set of pixels 375 that may represent the third plant 315. III.A Implementation of the Plant Recognition Model

[0054] There are several methods to determine plant group information in captured images. One approach is to use a plant recognition model running on a fully convolutional encoder-decoder network. For example, a plant recognition model can be implemented as a function in a neural network trained to determine plant group information from visual information encoded as pixels in an image. Plant recognition models can be similarly applied to pixel semantic segmentation models, where the class used to label the identified object is a plant group.

[0055] In this paper, the encoder-decoder network can be implemented as a plant recognition model 405 by the control system 130. The agricultural machine can execute the plant recognition model 405 to identify plant groups associated with pixels in the accessed image 400 and quickly generate an accurate plant group map 460. For illustration, Figure 4 This is a representation of a plant identification model based on an example implementation.

[0056] In the illustrated embodiment, the plant recognition model 405 is a convolutional neural network model with node layers, where the value at a node in the current layer is a transformation of the value at a node in the previous layer. The transformation in model 405 is determined by a set of weights and parameters connecting the current layer and the previous layer. For example, as... Figure 4 As shown, example model 405 includes five layers of nodes: layers 410, 420, 430, 440, and 450. Control system 130 applies function W1 to transform from layer 410 to layer 420, function W2 to transform from layer 420 to layer 430, function W3 to transform from layer 430 to layer 440, and function W4 to transform from layer 440 to layer 450. In some examples, the transformation can also be determined by a set of weights and parameters used to perform the transformations between the first few layers in the model. For example, the transformation W4 from layer 440 to layer 450 could be based on the parameters used to perform the transformation W1 from layer 410 to 420.

[0057] In the example process, the control system 130 inputs the accessed image 400 (e.g., accessed image 300) into the model 405 and encodes the image onto a convolutional layer 410. After processing by the control system 130, the model 405 outputs a plant group map 460 (e.g., 325A, 325B) decoded from the output layer 450. In the recognition layer 430, the control system 130 uses the model 405 to identify plant group information associated with pixels in the accessed image 400. Plant group information can indicate plants and other objects in a field and their locations in the accessed image 400. The control system 130 reduces the dimension of the convolutional layer 410 to the dimension of the recognition layer 430 to identify plant group information in the pixels of the accessed image, and then increases the dimension of the recognition layer 430 to generate the plant group map 460 (e.g., 325A, 325B). In some examples, the plant recognition model 405 can group pixels in the accessed image 400 based on plant group information identified in the recognition layer 430 when generating the plant group map 460.

[0058] As previously described, the control system 130 encodes the accessed image 400 into the convolutional layer 410. In one example, the captured image is directly encoded into the convolutional layer 410 because the dimensions of the convolutional layer 410 are the same as the pixel dimensions (e.g., the number of pixels) of the accessed image 400. In other examples, the captured image can be adjusted so that its pixel dimensions are the same as the dimensions of the convolutional layer 410. For example, the accessed image 400 can be cropped, reduced, scaled, etc.

[0059] Control system 130 applies model 405 to associate the accessed image 400 in convolutional layer 410 with plant group information in recognition layer 430. Control system 130 retrieves relevant information between these elements by applying a set of transformations (e.g., W1, W2, etc.) between corresponding layers. (Continue) Figure 4 For example, convolutional layer 410 of model 405 represents the accessed image 400, and recognition layer 430 of model 405 represents plant group information encoded in the image. Control system 130 identifies the plant group information corresponding to pixels in the accessed image 400 by applying transformations W1 and W2 to the pixel values ​​of the accessed image 400 in the space of convolutional layer 410. The weights and parameters used for the transformations can indicate the relationship between visual information contained in the accessed image and the inherent plant group information encoded in the accessed image 400. For example, the weights and parameters can be quantifications of shape, distance, ambiguity, etc., associated with the plant group information in the accessed image 400. Control system 130 can learn the weights and parameters using historical user interaction data and labeled images.

[0060] In recognition layer 430, the control system maps pixels in an image to associated plant group information based on latent information about objects represented by visual information in the captured image. The recognized plant group information can be used to generate a plant group map 460. To generate the plant group map 460, the control system 130 employs model 405 and applies transformations W3 and W4 to the plant group information recognized in recognition layer 430. The transformations result in a set of nodes in output layer 450. The weights and parameters of the transformations can indicate the relationship between image pixels in the accessed image 400 and plant groups in plant group map 460. In some cases, the control system 130 directly outputs plant group map 460 based on the nodes of output layer 450, while in other cases, the control system 130 decodes the nodes of output layer 450 into plant group map 460. That is, model 405 may include a transformation layer (not shown) that converts output layer 450 into plant group map 460.

[0061] For example, weights and parameters of the plant recognition model 405 can be collected and trained using data from previously captured visual images and the labeling process. The labeling process improves accuracy and reduces the amount of time required for the control system 130 to employ model 405 to identify plant group information associated with pixels in the image. See below. Figure 10 The labeling and training process will be described in more detail.

[0062] Additionally, model 405 may include layers called intermediate layers. Intermediate layers are layers that do not correspond to the convolutional layer 110 for the accessed image 400, the recognition layer 430 for plant group information, and the output layer 450 for the plant group map 460. For example, as... Figure 4 As shown, layer 420 is an intermediate encoder layer between convolutional layer 410 and recognition layer 430. Layer 440 is an intermediate decoder layer between recognition layer 430 and output layer 450. Hidden layers are latent representations of different aspects of the accessed image that are not observed in the data, but can control the relationships between elements of the image when recognizing plant groups associated with pixels in the image. For example, nodes in the hidden layer can have strong connections (e.g., large weight values) with input values ​​and values ​​of nodes in the recognition layer that share common plant group characteristics. Specifically, in Figure 4 In the example model, the nodes of hidden layers 420 and 440 can link inherent visual information of common characteristics shared by the accessed images 400 to help determine plant group information of one or more pixels.

[0063] Additionally, each intermediate layer can be a combination of functions such as residual blocks, convolutional layers, pooling operations, skip connections, and cascading. Any number of intermediate encoder layers 420 can be used to reduce the convolutional layer to the recognition layer, and any number of intermediate decoder layers 440 can be used to add the recognition layer 430 to the output layer 450. Alternatively, the encoder intermediate layers reduce the pixel dimension to the plant group recognition dimension, and the decoder intermediate layers add the recognition dimension of the plant group image dimension.

[0064] Furthermore, in various embodiments, model 405 can reduce the number of accessed images 400 and identify any number of objects in the field. The identified objects are represented in the recognition layer 430 as a data structure with recognition dimensions. In various other embodiments, the recognition layer can identify potential information representing other objects in the accessed images. For example, recognition layer 430 can identify the results of plant treatments, soil, obstacles, or any other objects in the field.

[0065] III.B Examples of Training Images

[0066] As mentioned above, a plant recognition model can be a machine learning model trained using images of plants in a field. The training image can be an accessed image or a portion of an accessed image (e.g., a bounding box surrounding the pixels representing the plants). In the former, the training image is larger and can provide more data for training the model. In the latter, the training image is localized to a portion of the image and can be labeled more quickly. In either case, the training image includes pixels representing plants from a plant group, as well as other objects in the field that can be used to train the plant recognition model. (See reference...) Figure 10 The generation of training images and the training of the plant recognition model are further described.

[0067] In some implementations, semantic segmentation labels with multiple plant groups can be generated from semantic segmentation labels with fewer plant groups and bounding box labels. For example, bounding box labels corresponding to multiple weed species can be combined with semantic segmentation labels having a single group for all weeds to generate semantic segmentation labels corresponding to multiple weed species. Initial labels can be combined by intersecting each bounding box with a semantic segmentation label and assigning the intersection portion of the semantic segmentation labels to the class of the bounding box. This approach can save time and money.

[0068] Figure 5A table describing example groups of training images is shown. The left column lists the plant groups labeled by bounding boxes within each group. In this example, the plant groups include species such as grasses, broadleaf weeds, cotton, soybean, hogweed, morning glory, horseweed / marestail, kochia, maize / corn, nutmeg, lambsquarters, and velvetleaf. The right column lists the total number of images including each species, and the middle column lists the total number of bounding boxes for each plant group (images may include multiple plant groups, and an image may include multiple plants from the same group).

[0069] III.C Example Recognition Model

[0070] Figure 6A It is a representation using reference Figure 5 This describes the confusion matrix of a plant recognition model trained on training images. Each axis lists plant groups. The x-axis lists the plant groups predicted by the model, and the y-axis lists the actual plant groups in a set of test images. Therefore, each row of the matrix represents the number of instances of the predicted plant group, and each column represents the number of instances of the plant group that appeared in the test image. In the confusion matrix, values ​​in the diagonal represent accurate predictions, while values ​​outside the diagonal represent prediction errors, such as false negatives and false positives. Figure 6B This is a table listing the additional performance metrics of the trained plant recognition model, including F-score, precision, and recall values. Precision, recall, and F-score are defined as follows:

[0071]

[0072] as well as

[0073]

[0074] TP represents the number of true positives, FP represents the number of false positives, and FN represents the number of false negatives.

[0075] Figure 6A and Figure 6B The performance metrics in the model indicate that the example plant prediction model is highly effective in identifying cotton, grass weeds, and soybean plants with high accuracy, and moderately effective in identifying broadleaf weeds and sedge plants. As previously stated, Figure 6A and Figure 6B The metric in the figure shows the use of reference. Figure 5 The performance of the example plant recognition model trained on the described training images is described. The plant recognition model described in this disclosure should not be limited to these performance metrics or those shown in subsequent images. For example, it can be improved by using additional training images that include these plant groups. Figure 6B The F-scores, precision, and recall values ​​for *Erigeron tigrinum*, *Kochia scoparia*, *Chenopodium album*, *Cyperus rotundus*, and *Vigna rotundifolia*.

[0076] III.D Example Plant Group

[0077] As previously mentioned, plant recognition models can identify and classify different plant groups. For example, plants can be grouped according to their species, genus, or family. Plant groups can be pre-defined (i.e., the plant recognition model is trained using images that include plant group labels), or groups can be provided to the plant recognition model after it has been trained (e.g., by a user). In the latter case, the plant recognition model can be trained to identify species (or another type of group), and after training, the model can be instructed to classify species into specific groups. See below. Figures 7A to 7D Describe example plant groupings. These figures include those based on references. Figure 5 The performance metrics of the plant recognition model trained on the described images.

[0078] The first example grouping categorizes plants as either "crops" or "weeds." Examples of this grouping are in... Figure 7A As shown in the image. Figure 7A The performance metric of a plant identification model is shown, which is instructed to classify plants as either "soybean" (i.e., crop) or "weed". In this case, species other than soybean (e.g., grass weeds, broadleaf weeds, pigweed, morning glory, tufted grass, kochia, nutgrass, lambsquarters, and velvetleaf) are classified as weeds. In this example, the metric indicates that the model can effectively identify soybean plants, weeds, and other plants.

[0079] The second example grouping separates weeds based on plant characteristics such as leaf shape, size, and / or color. Figure 7B An example of this grouping is shown in the figure. Figure 7B The performance metrics of a plant identification model are shown, which is instructed to classify plants as "soybean," "monocotyledonous weed," or "broadleaf weed." In this case, "monocotyledonous weed" includes weeds with slender leaves, while "broadleaf weed" includes weeds with broad leaves. In this example, the metrics indicate that the model is very effective in identifying soybean plants and monocotyledonous weeds, but of moderate effectiveness in identifying broadleaf weeds among other plants. "Monocotyledonous weed" and "broadleaf weed" can be considered as... Figure 7A A subgroup of the "weeds" group.

[0080] Figure 7CThe third example grouping is shown, where the plant identification model is instructed to classify plants as "soybean," "grass weeds," "broadleaf weeds," or "sedge weeds." In this case, "grass weeds" could include grass weeds and corn; "broadleaf weeds" could include cotton, kochia, lambsquarters, velvetleaf, morning glory, bulrush, sedge, and broadleaf weeds; and "sedge weeds" could include nutgrass. From a herbicide chemistry perspective, this could be useful because different families or species of weeds respond differently to different herbicide chemical mixtures. For example, chemicals used to treat grass weeds may differ from those used to treat broadleaf weeds. In this sense, one could use agricultural machinery with different chemical mixtures and target the delivery of chemicals based on weed type detection.

[0081] exist Figure 7C In the example, the metric indicates that the model is very effective at identifying soybean plants and grass weeds, but moderately effective at identifying broadleaf weeds. The model's ability to identify sedges can be improved by including more images containing nutgrass in the training images. In some implementations, "grass weeds," "broadleaf weeds," and "sedge weeds" can be considered as... Figure 7A A subgroup of the "weeds" group.

[0082] Pigweed can be classified into separate groups, for example, because pigweed is a common weed on many crops in the United States, such as cotton and soybeans. Figure 7D The example grouping shown illustrates this operation. In this example, the metric indicates that the model is highly effective at identifying soybean plants and grass weeds, and moderately effective at identifying broadleaf weeds and hogweed. In some cases, if the user is only interested in identifying hogweed (or any other plant), the plant identification model can be instructed to classify the plant as "hogweed" or "non-hogweed," where "non-hogweed" includes all other crops and weeds.

[0083] In another example of grouping, the plant identification model groups plants based on the plant treatments that should be applied to them. For example, plants can be grouped according to the herbicide, insecticide, fungicide, or fertilizer treatments that should be applied to the plants in that group. Using data from... Figure 5 For example, the first group of herbicides may include weeds that should be treated with herbicides (e.g., grasses and lambsquarters), the second group of herbicides may include weeds that should be treated with dicamba herbicide (e.g., broadleaf weed, tufted clover, morning glory, and velvet leaf), and the third group of herbicides may include weeds that should be treated with glufosinate herbicide (e.g., hogweed, kochia, and nutgrass).

[0084] III.E Application of Plant Identification Model

[0085] Figure 8A method for treating vegetation in a field according to one or more embodiments is illustrated. The method can be performed by agricultural machinery moving through the field. The agricultural machinery includes multiple processing mechanisms. Method 800 can be performed from the perspective of control system 130. Method 800 may include more or fewer steps than described herein. Furthermore, these steps may be performed in a different order or by different components than those described herein.

[0086] The control system receives 810 information describing plant groups to be identified in the field by the plant recognition model. This information may be based on input from a user of the agricultural machine (e.g., agricultural machine 100) or from one or more sensors (e.g., sensor 210). Each plant group includes one or more plants, and a plant group may correspond to one or more plants planted in the field. The plant group may describe the family, genus, or species of the plants in the plant group. In some embodiments, the plant group describes a plant treatment to be applied to the plants in the plant group. For example, each plant group describes a herbicide, insecticide, fungicide, or fertilizer treatment to be applied to the plants in the plant group. In other embodiments, the plant group describes plant characteristics common to the plants in each plant group (e.g., leaf shape, size, or leaf color).

[0087] The control system accesses an image of the field from an image sensor (820). The image sensor can be coupled to the agricultural machine as it moves through the field. The image includes a set of pixels representing plants. The control system applies a plant recognition model (830) to the image. The plant recognition model determines that the set of pixels representing plants is a plant in a plant group, classifies the set of pixels representing plants into a plant group, and determines the representation location of the classified set of pixels. Based on the classified plant group and representation location, the control system generates a plant treatment instruction (840) for processing the plants with a processing mechanism. The control system uses the plant treatment instruction to actuate a plant treatment mechanism (850) such that the plants are processed by the plant treatment mechanism as the agricultural machine moves through the plants in the field. In some embodiments, an image of the field is captured by the image sensor at a first time point, and the plant treatment mechanism is actuated at a second time point after the first time point. The second time point can be any time after the first time point.

[0088] III.F Training Plant Recognition Model

[0089] Figure 9 A method for training a plant recognition model according to one or more embodiments is illustrated. Method 900 can be executed from the perspective of control system 130. Method 900 may include more or fewer steps than described herein. Furthermore, these steps may be performed in a different order or by different components than those described herein.

[0090] The control system 130 accesses 910 an image having a set of pixels representing one or more plants. The image has a field of view as seen from the image sensor. The image sensor can be attached to the agricultural machine as it travels through the plants in the field. The control system 130 identifies multiple groups of pixels within the image 920. Each group of pixels represents one or more plants and indicates the representative location of that one or more plants in the image.

[0091] For each image in the image group, the control system 130 generates one or more labeled images by assigning plant groups to each pixel group in the image. For example, bounding boxes are placed around the pixel groups, and plant group labels are assigned to these boxes. In another example (e.g., for a pixel segmentation model), individual pixels in the pixel group are identified and assigned plant group labels. To label the images, the control system 130 may receive input from one or more users who are viewing the images and identifying plant groups in the bounding boxes. For example, an agronomically trained user identifies the species of each plant represented by the pixel groups in the bounding boxes. In some embodiments, the labeled images include labels for multiple sets of pixels representing non-plant objects in the field (e.g., soil, rocks, debris in the field, etc.). For example, non-plant objects are labeled by assigning “non-plant objects” to the pixels representing themselves.

[0092] The control system 130 uses the set of labeled images to train a plant recognition model 940 to determine (e.g., identify) plant groups and to determine the location of plants in individual images. The plant recognition model is trained by associating images with labeled images. For example, a neural network is trained to associate the label of a labeled image with a set of pixels in a corresponding unlabeled image. As previously described, the plant recognition model can be instructed to identify specific plant groups during the operation of the agricultural machinery. In some cases, the plant recognition model is instructed to identify plant groups different from those in the labeled images used to train the model. For example, the model may be instructed to identify genus, but is trained to identify species. In these cases, the plant recognition model can form clusters of plant groups corresponding to those specified in the instructions. Continuing with the previous example, the model can form groups of species corresponding to genus. By doing so, the plant recognition model can identify species based on its training and then group the identified species into multiple genus groups.

[0093] In some cases, if a user prioritizes the identification of one plant group over another, the plant identification model can be instructed to provide higher performance for a certain metric for a specific plant group. For example, when using a semantic segmentation plant identification model to classify pixels in an image into “pigweed,” “weeds other than pigweed,” and “non-weed,” a user might want to prioritize recalling “pigweed” over recalling “weeds other than weeds” and recalling “non-weed.” In this example, the plant identification model could be trained using a loss function such as asymmetric loss and parameters that prioritize recalling “pigweed,” for example, by using a higher β value for “pigweed” than for “weeds other than pigweed” and “non-weed.” In some implementations, identifying harmful weed species may be important. In these implementations, the loss function can be adjusted to penalize errors in identifying harmful weeds more severely than errors in identifying low-toxicity or non-toxic weeds. Therefore, the plant identification model can be tuned to handle harmful weeds more accurately, which can reduce variations in competition between harmful weeds and crops.

[0094] The control system 130 can periodically train the plant recognition model during agricultural machinery operation, at defined times, or before implementing the plant recognition model on the agricultural machinery. Furthermore, the plant recognition model can be trained by another system, allowing it to be implemented as an independent model on the agricultural machinery's control system.

[0095] IV. Plant Identification Diagram

[0096] In some implementations, a plant identification map of the field is generated. The plant identification map can be a spatially indexed geographic map of the field, indicating the location of plant groups identified by the plant identification model. Among other advantages, the plant identification map provides insights into cultivating and maintaining the field. Figure 10 An example plant identification diagram 1000 is shown. Figure 1000 provides a top-down real-world view of a field including three rows of crops (first plant group 1005) and weeds scattered between the rows (second plant group 1010 and third plant group 1015). Although not shown in Figure 10 The map provides general information, but may highlight one or more groups to help users identify the location of plant groups in the field. The plant identification map can also indicate, for example, the type of treatment action applied to an area of ​​the field, based on the classified plant groups and their locations. Figure 10 The example indicates treatment area 1020A where a first herbicide is applied to a second plant group 1010 and treatment area 1020B where a second herbicide is applied to a third plant group 1015. The plant identification map 1000 thus allows users to visualize the locations where plant treatment actions are applied in the field.

[0097] The control system 130 can generate Figure 1000 by combining accessed images or plant group images from a plant recognition model. Figure 1000 can be generated after the agricultural machine passes through the field and applies processing actions to the plants in the field. The images can be combined as follows. The external parameters (e.g., position and orientation) and internal parameters (e.g., sensor parameters and distortion parameters) of each image sensor can be known to the GPS receiver. Given these parameters, pixels in the images can be associated with or mapped to geographic locations on a ground plane. Each image can be mapped to a ground plane and the geographic coordinates of each image can be calculated. Once the geospatial coordinates of each image are known, the pixels can be placed on a digital map representing the geographic area imaged by the agricultural machine.

[0098] Since agricultural machinery can include different image sensors (e.g., visible wavelength cameras and IR cameras), plant identification maps can include different layers formed from images from each sensor (e.g., a visible wavelength layer and an IR layer). Figure 10 (Not shown in the image). Through the user interface, users can view one or more layers at a time, for example, overlaying an IR layer onto a visible wavelength layer. Other layers may include plant group identification from a plant identification model, sprayed areas of agricultural treatments applied by agricultural machinery (e.g., area 1020), or layers derived from the fact that the actual location of crop and weed species is identified, or other layers related to machine operation (e.g., dust images or other factors that may affect machine operation).

[0099] A color scheme can be applied to the identification map 1000 to highlight plant groups in the field. For example, the soil can be grayed out, and each plant group can be highlighted in a different color. In another example, only a single plant group can be highlighted for quick identification. In some implementations, the plant identification map is a heatmap, where the colors of the map indicate the spatial density of plant groups in the field.

[0100] like Figure 10 As shown, information such as agricultural machinery information, field data, and processing facility measurements can be used (in... Figure 10 Measures such as "spray geometry," area measurement, and timing measurement are overlaid on the plant identification map 1000. Agricultural machine information describes information associated with the agricultural machine, such as its type and model. Field data information describes information associated with the crops in the field, such as the crop (e.g., species) and its size (e.g., height). Processing mechanism measurements describe the orientation and location of the machine's processing mechanism. Figure 10 In the example, the agricultural machine includes nozzles oriented at a 40-degree angle. Area measures describe field and plant group measurements, such as the total area imaged by the agricultural machine, the total number of plants identified, etc. Figure 10 (not shown in the image) The total number of identified plant groups ( Figure 10 (Not shown in the image), the total area of ​​weed groups identified by the plant recognition model, and the total area sprayed by the treatment facility. Timing metrics describe the delays of the agricultural machinery (e.g., averages and maximums), such as the time spent capturing images of plants, classifying plants into plant groups, determining treatments, and applying treatments to the plants. Another metric of interest could be the accuracy of the agricultural machinery's weed spraying, calculated, for example, by comparing the spray locations identified by the machine with the weed locations identified in a "live" layer. The live layer is formed by a labeling machine that provides labels to the plant groups (this is typically not done during agricultural machinery operation).

[0101] V. Using plant group treatments

[0102] As described above, the agricultural machine can employ a plant recognition model executed by the control system 130 to classify plants into multiple plant groups and determine the location of the plant groups. The agricultural machine can then process the plants based on their plant groups and locations. When processing the plants, the control system 130 can determine one or more processing actions for the identified plant groups. As previously mentioned, processing actions may include, for example, actuating processing mechanisms, modifying processing parameters, modifying operating parameters, and modifying sensor parameters.

[0103] In some implementations, determining the processing action includes generating a processing map. A processing map is a data structure that associates plant groups in a plant group map (e.g., Figure 325) with processing actions. For example, a processing map includes plant group segments (e.g., 345, 350, and 355) selected according to predetermined processing actions. Those processing actions can be performed by a processing agency capable only of processing specific areas of the field. Thus, a specific area is associated with a specific processing agency.

[0104] In some implementations, the processing map has a field of view corresponding to the field of view of the accessed image (e.g., image 300), and portions of the field of view correspond to processing mechanisms that perform processing actions. Therefore, when a portion of the field of view includes a plant to be processed by a processing action, the corresponding processing mechanism will be used to process that plant.

[0105] When employing method 800, control system 130 can generate a treatment map. Control system 130 interprets and translates the data structure of the treatment map into machine signals necessary to complete the treatment actions at the appropriate time. Therefore, control system 130 can implement treatment actions to treat plants (or other objects) in a field. Examples of generating treatment maps to treat identified plants are disclosed in their entirety in U.S. Patent Application 16 / 126,842, filed September 10, 2018, entitled “Semantic Segmentation to identify and Treat Plants in a Field and Verify the Plant Treatments,” but other methods for generating treatment maps are also possible.

[0106] VIII. Control System

[0107] Figure 11 This is a block diagram illustrating components of an example machine used to read and execute instructions from a machine-readable medium. Specifically, Figure 11 A diagram illustrating a control system 130 in an example form of computer system 1100 is shown. Computer system 1100 can be used to execute instructions 1124 (e.g., program code or software) to cause the machine to perform any or more of the methods (or processes) described herein. In alternative embodiments, the machine operates as a standalone device or a connected (e.g., networked) device linking other machines. In a networked deployment, the machine can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0108] The machine can be a server computer, client computer, personal computer (PC), tablet PC, set-top box (STB), smartphone, Internet of Things (IoT) device, network router, switch or bridge, or any machine capable of (sequentially or otherwise) executing instructions 1124 specifying actions to be taken by that machine. Furthermore, although only a single machine is shown, the term "machine" should also be considered to include any set of machines that individually or collectively execute instructions 1124 to perform any one or more of the methods discussed herein.

[0109] Example computer system 1100 includes one or more processing units (typically processor 1102). Processor 1102 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a controller, a state machine, one or more application-specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof. Computer system 1100 also includes main memory 1104. The computer system may include storage units 1116. Processor 1102, memory 1104, and storage units 1116 communicate via bus 1108.

[0110] Additionally, the computer system 1100 may include static memory 1106, a graphics display 1110 (e.g., for driving a plasma display panel (PDP), liquid crystal display (LCD), or projector). The computer system 1100 may also include an alphanumeric input device 1112 (e.g., a keyboard), a cursor control device 1114 (e.g., a mouse, trackball, joystick, motion sensor, or other pointing instrument), a signal generation device 1118 (e.g., a speaker), and a network interface device 1120, all of which are configured to communicate via a bus 1108.

[0111] Storage unit 1116 includes a machine-readable medium 1122 on which instructions 1124 (e.g., software) embodying any one or more of the methods or functions described herein are stored. For example, instructions 1124 may include... Figure 2 The instructions 1124 are functions of the modules of system 130 described herein. Instructions 1124 may also reside wholly or at least partially within main memory 1104 or processor 1102 (e.g., within the processor's cache memory) during execution by computer system 1100, both of which constitute machine-readable media. Instructions 1124 may be sent or received on network 1126 via network interface device 1120.

[0112] IX. Other considerations

[0113] In the above description, for illustrative purposes, numerous specific details have been set forth to provide a thorough understanding of the illustrated system and its operation. However, it will be apparent to those skilled in the art that the system can be operated without these specific details. In other instances, to avoid obscuring the system, the structure and equipment are shown in block diagram form.

[0114] In this specification, "one embodiment" or "implementation" means that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one implementation of the system. The phrase "in one embodiment" appearing throughout this specification does not necessarily refer to the same embodiment in all instances.

[0115] This section presents certain parts of a detailed description based on algorithms or models and symbolic representations of operations on data bits within computer memory. Algorithms are generally conceived here as steps that produce desired results. These steps involve physical transformations or processing of physical quantities. Typically, though not always necessary, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated. It has been shown that these signals are sometimes referred to as bits, values, elements, symbols, characters, terms, numbers, etc., primarily due to their widespread use.

[0116] However, it should be remembered that all such terms should be associated with appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise stated, as will be apparent from the following discussion, it can be understood that throughout the description, discussions using terms such as “processing” or “calculating” or “operating” or “determining” or “displaying” refer to the actions and processes of a computer system or similar electronic computing device that process data represented as physical (electronic) quantities in the registers and memories of the computer system and convert that data into other data similarly represented as physical quantities in the computer system’s memory or registers or other such information storage, transmission or display devices.

[0117] Some of the operations described herein are performed by a computer physically installed within the machine. This computer may be specially constructed for the desired purpose, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored on the computer. Such a computer program may be stored in a computer-readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs and magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic cards or optical cards, or any type of non-transitory computer-readable storage medium suitable for storing electronic instructions.

[0118] The above figures and descriptions are illustrated by way of example only and represent various embodiments. It should be noted that, based on the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily considered as feasible alternatives without departing from the claimed principles.

[0119] One or more embodiments have been described above, examples of which are shown in the accompanying drawings. Note that, where feasible, similar or identical reference numerals may be used in the drawings and may indicate similar or identical functions. These drawings depict embodiments of the disclosed system (or method) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein.

[0120] The terms "coupled" and "connected," and their derivatives, can be used to describe some implementations. It should be understood that these terms are not intended to be synonymous. For example, the term "connected" may be used to describe some implementations to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, the term "coupled" may be used to describe some implementations to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also mean that two or more elements are not in direct physical or electrical contact with each other, but still cooperate or interact with each other. Implementations are not limited to this context.

[0121] As used herein, the terms “comprises,” “includes,” “having,” or any other variation thereof are intended to cover non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such a process, method, article, or apparatus. Furthermore, unless explicitly stated otherwise, “or” refers to an inclusive “or” rather than an exclusive “or.” For example, any of the following satisfies the condition “A or B”: A is true (or exists) and B is false (or does not exist); A is false (or does not exist) and B is true (or exists); and both A and B are true (or exist).

[0122] Furthermore, the terms "a" or "an" without quantitative limitation are used to describe elements and components in the embodiments described herein. This is merely for convenience and to give a general meaning to the system. This description should be understood to include one or at least one, and the singular includes the plural unless clearly implied otherwise.

[0123] Upon reading this disclosure, those skilled in the art will understand that alternative structural and functional designs are possible for systems and processes used to identify and process plants using agricultural machinery that incorporates a control system that performs a semantic segmentation model. Therefore, although specific embodiments and applications have been shown and described, it should be understood that the embodiments disclosed herein are not limited to the exact constructions and components disclosed herein. Various modifications, alterations, and variations that will be apparent to those skilled in the art may be made to the arrangement, operation, and details of the methods and apparatus disclosed herein without departing from the spirit and scope defined in the appended claims.

Claims

1. A method for treating plants in a field by an agricultural machine that moves through the field, the agricultural machine comprising a plurality of plant treatment mechanisms, the method comprising: Receive information describing multiple plant groups to be identified in the field by a plant identification model, each plant group including one or more plants and describing plant treatments to be applied to the plants in the plant groups; Access an image of the field captured by an image sensor, the image including a set of pixels representing the plants; The plant recognition model is applied to the image, and the plant recognition model is configured to: The plant is determined to be a plant in one of the multiple plant groups described by the received information based on the set of pixels representing the plant. The group of pixels representing the plant is classified as the plant group; and Determine the representative location of the classified group of pixels in the image; Based on the plant group classified and the representative location, generate plant treatment instructions for treating the plants using one of the plurality of plant treatment facilities; and The plant treatment command is used to actuate the plant treatment mechanism so that the plants are treated by the plant treatment mechanism as the agricultural machine moves through the plants in the field.

2. The method according to claim 1, wherein, The received plant groups include multiple weed groups corresponding to one or more plants in the field.

3. The method according to claim 2, wherein, The multiple weed groups describe the families of the plants in the weed groups.

4. The method according to claim 2, wherein, The multiple weed groups describe the genera of plants within the weed groups.

5. The method according to claim 2, wherein, The multiple weed groups describe the species of plants in the weed groups.

6. The method according to claim 1, wherein, Each plant group includes a weed group, and each weed group describes the plant characteristics of the plants in that weed group.

7. The method according to claim 6, wherein, The plant characteristics include at least one of the following: leaf shape, size, or leaf color.

8. The method according to claim 1, wherein, Each plant group includes a weed group, and each weed group describes one or more plant treatment mechanisms to be applied to the plants in the weed group.

9. The method according to claim 1, wherein, The corresponding plant treatments include at least one of the following: herbicide treatment, insecticide treatment, fungicide treatment, or fertilizer treatment.

10. The method according to claim 1, further comprising: Receive crop groups to be identified in the field by the plant recognition model.

11. The method according to claim 1, wherein, The image of the field is captured by the image sensor at a first time, and the plant treatment mechanism is actuated at a second time after the first time.

12. The method according to claim 1, wherein, The image sensor is coupled to the agricultural machine, and the image sensor captures images of the field as the agricultural machine moves through the field.

13. The method according to claim 1, wherein, The information received describing the plurality of plant groups is received after the plant recognition model has been trained.

14. The method according to claim 1, wherein, The information received describing the multiple plant groups is based on user input.

15. An agricultural machine, comprising: Multiple plant treatment units are used to treat the plants as the agricultural machinery travels through the plants in the field. The control system is configured to: Receive information describing multiple plant groups to be identified in the field by a plant identification model, each plant group including one or more plants and describing plant treatments to be applied to the plants in the plant groups; Access an image of the plant captured by an image sensor, the image comprising a set of pixels representing the plant; The plant recognition model is applied to the image, and the plant recognition model is configured to: The plant is determined to be a plant in one of the multiple plant groups described by the received information based on the set of pixels representing the plant. The group of pixels representing the plant is classified as the plant group; and Determine the representative location of the marked group of pixels; Based on the plant group classified and the representative location, generate plant treatment instructions for treating the plants using one of the plurality of plant treatment facilities; and The plant treatment command is used to actuate the plant treatment mechanism so that the plants are treated by the plant treatment mechanism as the agricultural machine moves through the plants in the field.

16. The agricultural machinery according to claim 15, wherein, The received plant groups include multiple weed groups corresponding to one or more plants planted in the field.

17. The agricultural machinery according to claim 16, wherein, The multiple weed groups describe the species of plants in the weed groups.

18. The agricultural machinery according to claim 16, wherein, The multiple weed groups describe the genera of plants within the weed groups.

19. The agricultural machinery according to claim 15, wherein, Each plant group includes a weed group, and each weed group describes the corresponding plant characteristics of the plants in the weed group.

20. The agricultural machinery according to claim 19, wherein, The plant characteristics include at least one of the following: leaf shape, size, or leaf color.

Images