Image segmentation and related training system for row tracking

Abstract

Methods and systems related to computer vision for agricultural applications are disclosed herein. A method for navigating a robot along a crop row wherein each step is computer-implemented by a navigation system for the robot, the method comprising capturing an image of at least a portion of the crop row, tagging the portion of the image with a tag using a segmentation network, deriving a navigation path from the portion of the image and the tag, and navigating the robot along the crop row, a control signal is generated for the autonomous navigation system to track the navigation path, and the robot is navigated along the crop row using the control signal.

Description

Background Art

[0001] Computer vision systems can be used to guide automated robotic agricultural processes, such as robotic manipulation (e.g., picking cherry tomatoes) or robotic navigation (e.g., navigating along crop rows). Computer vision systems can be based on machine learning systems that can be trained to perform certain tasks. Computer vision machine learning systems can be trained with unsupervised training routines, where labeled training data is not required. However, in agricultural applications, robots need to operate within a low error range to avoid damaging crops. In addition, field robots need to operate in a variety of environments, such as fields with different crops, fields with different crops at different growth stages, fields with different planting configurations, fields in different biomes, seasons and climates, and indoor greenhouses and open fields. These requirements for high accuracy and wide generalizability tend to make unsupervised training routines unsuitable for computer vision machine learning systems in agricultural applications. However, supervised training of computer vision machine learning systems requires large training data sets to produce systems that can be promoted in many environments. Training data samples are usually provided in the form of labeled data, but such labeled data are difficult to obtain due to the manual work of human annotators that are usually required to generate labels for data sets. Summary of the invention

[0002] Methods and systems for agricultural applications related to computer vision are disclosed. Methods and systems including a navigation system for navigating a robot along crop rows are disclosed. The navigation system can utilize a trained computer vision machine learning system, such as a trained segmentation network, which is used to segment image data into one or more labeled segments. In a specific embodiment of the present invention disclosed herein, one of the labeled segments can be an inter-row path, and the navigation system can use the labeled segments to derive a navigation path along the crop row and generate control signals to navigate the robot along the navigation path. The segmentation network can be a trained machine intelligence system that is trained using a supervised training routine.

[0003] In a specific embodiment of the present invention, a method and system for effectively training a navigation system for a field robot is provided. The method is used to train a navigation system to navigate a robot in a new environment (where the field robot has little or no prior knowledge of the new environment) without requiring a large amount of manually annotated training data.

[0004] In a specific embodiment of the present invention, in contrast to traditional training methods, a trained computer vision machine learning system can be trained directly on the data on which the system will be deployed to operate. For example, a trained computer vision machine learning system can be trained on representative crop rows in a crop row group in a single field or a single farm, and then deployed to navigate a robot along a specific crop row group. Although such a trained computer vision machine learning system may not be generalizable to other applications (such as crop row tracking on other farms), the amount of training data required to prepare the system for deployment is several orders of magnitude less than that of a more generalized system. For example, using the methods disclosed herein, the labeled training data set required to provide sufficient row tracking navigation performance (including avoiding collisions with obstacles such as people or field irrigation equipment) can be as small as 10 to 15 frames of labeled data. In addition, in a specific embodiment of the present invention disclosed herein, an intuitive interface is provided for a human operator to easily and efficiently generate labels for this small labeled data set. In addition, in a specific embodiment of the present invention disclosed herein, the labeled training data can be obtained entirely without label input from a human operator.

[0005] In a specific embodiment of the invention disclosed herein, a method for navigating a robot along a crop row is provided. Each step of the method can be implemented by a navigation system for the robot using a computer. The method can include capturing an image of at least a portion of the crop row, labeling the portion of the image with a label using a segmentation network, deriving a navigation path from the portion of the image and the label, generating a control signal for an autonomous navigation system to track the navigation path, and navigating the robot along the crop row using the control signal.

[0006] In a specific embodiment of the invention disclosed herein, a method for training a navigation system to navigate a robot along crop rows in a crop row group is provided. The method includes capturing an image set of at least a portion of the crop row group, displaying the image set on a user interface, accepting a label input set about the image set on the user interface, and training a segmentation network using the label input set and the image set.

[0007] In an alternative specific embodiment of the invention disclosed herein, a method for training a navigation system to navigate a robot along crop rows is provided, the method not requiring human labeling input. The method includes capturing a set of images of at least a portion of the set of crop rows while navigating the robot along the crop rows, and performing photogrammetric analysis on the set of images to generate a set of labeled inputs on the set of images of a user interface. The photogrammetric analysis includes solving for the location of the path in a first image based on an analysis of a subsequently captured second image. The method also includes training a segmentation network using the labeled input set and the set of images.

[0008] In a specific embodiment of the invention disclosed herein, a method for navigating a robot along crop rows in a crop row group includes training a navigation system according to any of the training methods described in the previous paragraphs and then, after training a segmentation network, navigating the robot along the crop rows using the segmentation network. The crop row group may be a crop row group in a single field or a single farm. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A flow chart showing a set of methods according to specific embodiments of the invention disclosed herein;

[0010] Figure 2 shows crop row paths annotated with navigation paths according to specific embodiments of the invention disclosed herein;

[0011] Figure 3 shows an example of the output of a segmentation network designed to segment an input image into multiple segments with different labels according to a specific embodiment of the invention disclosed herein;

[0012] Figure 4 shows two images taken at two different times by the same imager on a robot, one of which is used as input to a segmentation network to ultimately obtain a navigation path, which is then projected onto the other image, according to a specific embodiment of the invention disclosed herein;

[0013] Figure 5 shows two images with labeled portions for checking labels of portions of the images using expected geometry principles in accordance with specific embodiments of the invention disclosed herein;

[0014] Figure 6 A flow chart showing a set of methods for training a network and navigating a robot using the trained network according to specific embodiments of the invention disclosed herein;

[0015] Figure 7 A user interface is shown on which an image has been displayed to receive a plurality of label inputs from a user to train a segmentation network according to a specific embodiment of the invention disclosed herein. DETAILED DESCRIPTION

[0016] Methods and systems related to the field of computer vision for agricultural applications according to the above invention content are disclosed in detail herein. The methods and systems disclosed in this section are non-limiting embodiments of the present invention, are used for explanation purposes only, and should not be used to limit the full scope of the present invention. It should be understood that the disclosed embodiments may overlap with each other or may not overlap. Therefore, a portion of an embodiment or a specific embodiment of the embodiment may fall within or not fall within the scope of another embodiment or a specific embodiment of another embodiment, and vice versa. Different embodiments from different aspects may be combined or implemented separately. Many different combinations and sub-combinations of the representative embodiments shown within the broad framework of the present invention may be obvious to those skilled in the art, but they are not explicitly shown or described and should not be interpreted as being excluded.

[0017] In a specific embodiment of the present invention, a method for navigating a robot along a crop row includes a set of steps implemented by a navigation system for the robot using a computer. The navigation system can be used to guide the robot along the crop row and can be applied to a robot whose task is to perform agricultural tasks. The navigation system can be designed to guide the robot from the starting point to the end along the crop row while interacting with or avoiding objects encountered in the path, for example, when the robot reaches the end of the crop row along the path, guiding the robot to stop and wait for a person or not collide with a cart. The robot can then be manually turned at the end of the crop row, and the navigation system can be reengaged to start the robot tracking the next row. Alternatively, the navigation system can also turn the robot at the end of the crop row and continue to the next crop row.

[0018] The navigation system can be implemented by a non-transitory computer-readable medium storing instructions to perform the methods disclosed herein. The non-transitory computer-readable medium can be one or more memories, such as non-volatile memory on the robot. However, in a specific embodiment, the non-transitory computer-readable medium can include a memory located in a data center or a local server remote from the robot and accessible to the robot via a network. The robot can include one or more processors, such as a microprocessor, a machine learning accelerator, a microcontroller, or other elements that can execute stored instructions and send control signals to actuators, sensors, and other systems on or outside the robot to perform the methods disclosed herein.

[0019] The robot can be an agricultural robot, which is intended to perform one or more agricultural tasks, such as evaluating, picking, harvesting, watering, weeding and fertilizing crops. The robot can be a ground vehicle that moves partially along the ground, such as a legged, wheeled or tracked vehicle. The portion that moves along the ground can be connected to a platform for performing the robot's agricultural tasks. The robot can be designed to move along a single crop row path, or straddle a crop row and move its wheels or tracks along two adjacent crop row paths. In this way, navigating the robot along the crop row can include moving the robot along the crop row while ensuring that the portion that contacts the ground remains on the path, or moving in a manner to avoid crushing the crop or disturbing the soil near the delicate crop. Navigation of the robot can also include detecting obstacles (e.g., people, animals, irrigation equipment or other agricultural equipment) in the path of the robot, and avoiding obstacles or stopping completely while continuing to move along the crop row. Navigation of the robot can also include detecting the end of the row, so that the robot can stop and manually turn to the next row, or a separate navigation program can be started to automatically align itself with the next row. Although ground vehicles are used as examples throughout this disclosure, specific embodiments disclosed herein are applicable to any aerial robot, such as a floating, fixed-wing, or rotary vehicle designed to navigate in an agricultural environment.

[0020] In a specific embodiment of the present invention, a method for navigating a robot along a crop row includes the step of capturing an image of at least a portion of the crop row. The image can be captured by one or more sensors (e.g., one or more imagers). The sensor can be on the robot. The image can be captured by at least one imager on the robot. The image can be captured using at least two imagers on the robot. The image can include depth information captured by more than two imagers operating in combination to capture stereo depth data, or depth information captured by a dedicated depth sensor. The image can include image data from one or more electromagnetic spectra (e.g., SWIR, IR, UV, visible light, etc.). The image can be black and white, grayscale, or color. The image can be a 1, 2, 2.5, or 3 dimensional image.

[0021] In particular embodiments of the invention, images may be captured by one or more sensors. For example, images may be captured by a single sensor in the form of a visible light camera that captures a two-dimensional grayscale image of a portion of a crop row. As another example, images may be captured by a sensor pair in the form of two visible light cameras that capture a 2.5-dimensional grayscale image of a portion of a crop row by using stereo vision to determine the depth of pixels in the image. As another example, the sensor pair in the previous example may be enhanced with a third sensor in the form of a color visible light camera to capture a 2.5-dimensional color image of a portion of a crop row. As another example, images may be captured by a visible light color camera paired with a dedicated depth sensor to capture a 2.5-dimensional color image of a portion of a crop row.

[0022] In specific embodiments of the invention, the sensor that captures the image can be positioned in various ways relative to the robot. For example, the imager can be attached to the robot and locked in a fixed position relative to the robot, or it can be located on a separate vehicle, such as an aerial drone or another ground robot that moves in front of the robot and captures the image. The sensor can be in a fixed position and attitude relative to the robot throughout the operation, or it can be designed to change its position during operation. In specific embodiments of the invention, the sensor is attached to the robot and registered relative to the body of the robot in the navigation system so that the evaluation of the image inherently includes an evaluation of the position of certain parts of the robot relative to the image content. In specific embodiments, the attitude of the imager can be adjusted by the control system so that the imager is registered relative to the body of the robot regardless of the attitude of the imager selected by the control system.

[0023] The portion of the crop row captured in the image may exhibit various features. The portion of the crop row may include at least one crop row path. The portion of the crop row may include crops in the crop row. The portion of the crop row may be the portion in front of the vehicle and is generally aligned with the portion of the robot that contacts the ground (e.g., the wheels of the robot). In specific embodiments, the portion of the crop row may be an area surrounding the robot and include a portion of the robot, and the image may be a bird's-eye view image of the area. Such an image may be captured by a sensor located on a separate platform above the robot or attached to the robot via an accessory and suspended above the portion of the robot.

[0024] Figure 1A flowchart 100 of a set of methods for navigating a robot 110 along crop rows is shown, wherein each step is implemented by a computer for a navigation system 111 for the robot 110. The flowchart includes a loop path because it can be executed continuously while the navigation system is operating. For example, the loop can be performed every 0.1 seconds, every 1 second, or more or less frequently, based on a compromise between the computational and energy requirements associated with rapid execution and the degree of accuracy and safety required, and the fact that the robot may move relatively slowly compared to the rapid execution of the loop. In the illustrated case, the navigation system 111 is implemented entirely by one or more processors and non-transitory computer-readable media on the robot 110. The segmentation network can be part of the navigation system and can be implemented by a computer on the robot 110 (i.e., by one or more processors and non-transitory computer-readable media on the robot 110).

[0025] Flowchart 100 may begin with step 101 of capturing an image of at least a portion of a crop row. The image may be captured by at least one imager on the robot, such as imager 114. Imager 114 may be a color camera. Images may also be captured using at least two imagers, such as imager 112 and imager 113. Imager 112 and imager 113 may be grayscale imagers. All three imagers may be used in combination to capture a single image. Alternatively, imager 112 and imager 114 may operate to capture a single image of one crop row path, while imager 113 and imager 114 may operate to capture a second image of an adjacent crop row path. Images captured in accordance with the present disclosure may be, for example, captured by a robot. Figure 2 200 are those images represented by image 200 in which images are captured by a grayscale camera located on the robot having a view in the robot's intended direction of travel and approximately aligned with the robot's wheels or tracks. Image 200 includes three images 200A, 200B, 200C, which are the same captured images with different annotations. As seen in image 200A, the image includes a portion of a crop row, which portion of the crop row includes a crop row path 201. The crop row path 201 can be used to move a device (e.g., a robot) along the crop row without disturbing the crops. Therefore, a method for navigating a robot along crop rows can include attempting to keep the portion of the robot contacting the ground within the range of the crop row path 201.

[0026] In a specific embodiment of the present invention, a method for navigating a robot along a crop row includes the step of marking portions of an image with labels using a segmentation network. The image may be an image of at least a portion of a crop row and may have features of the above-mentioned image. The marking step may include providing the image as an input to a segmentation network. The step may include providing the image as an input to a segmentation network with or without preprocessing for preparing the image as an input to the segmentation network. The segmentation network may be modeled according to the architecture of SegNet, U-Net, M-net, Mask-RCNN, PspNet, GSCNN, etc. In a specific embodiment, the segmentation network may be implemented by a computer on the robot. For example, in a specific embodiment, the segmentation network may be a lightweight segmentation network that may be deployed and executed on an embedded controller on the robot. The segmentation network may be a lightweight segmentation network that may be trained and / or used to generate inferences on a small embedded system rather than on a cloud-based server. For example, the network may be embedded in a simple Arduino or Raspberry PI system.

[0027] In a particular embodiment of the invention, the input to the segmentation network may include additional data used in conjunction with the image data. For example, the input may include metadata about the image, such as the time the image was taken, the weather when the image was taken, the ambient light conditions when the image was taken, or other metadata. As another example, the input may include additional sensor data acquired along with the image data through additional sensor data, such as depth data of the object, radar data, stereo image data of the object, odometer data of the robot, inertial motion unit data of the robot, gravity data, image data from multiple electromagnetic spectra (e.g., SWIR, IR, UV, visible light, etc.), and others. The additional sensor data may be collected by additional sensors on the device.

[0028] In a specific embodiment of the invention, the robot may include an onboard machine learning accelerator to assist in the training or execution of the segmentation network. The image may be provided as an input to the segmentation network together with additional data or alone. The output of the segmentation network may have the same dimensionality as the input and provide a label value for each pixel or voxel in the input image. As described below, the label may be a single binary label value, or a label value selected from a label set.

[0029] The segmentation network may be a trained machine intelligence system, such as an artificial neural network (ANN). An alternative trained machine intelligence system may be used to replace the segmentation network. The trained machine intelligence system that can be used according to the embodiments of the present invention disclosed herein may be an ANN, a support vector machine, or any type of function approximator or an equivalent algorithmic system that can be iteratively adjusted using image data. The image data used to train the trained machine intelligence system may include real images or simulated images. Training may include supervised learning or unsupervised learning using labeled image data. In the case of ANN, various forms of ANN may be utilized, including convolutional neural networks, adversarial networks, attention networks, recurrent neural networks (RNNs), and various other networks. ANN may include multiple layers, such as convolutional layers, fully connected layers, pooling layers, upsampling layers, random inactivation layers, and other layers. ANN may include one or more encoders and one or more decoders. ANN may be feedforward only, or include recursive paths (e.g., in the case of RNN). One or more of linear regression, support vector regression, random forest, decision tree, or k nearest neighbor analysis may be used to train the trained machine intelligence system.

[0030] In a specific embodiment of the present invention, labeling can be performed in a variety of ways. For example, a segmentation network can analyze pixels, voxels, or other elements of an image, and apply labels to pixels in the form of an output of the segmentation network. For example, the network can determine which elements are part of a crop row path and which are not, and mark those parts that are part of a crop row path accordingly. Therefore, the output can be as simple as a binary assignment to an image element, indicating whether the element is part of a crop row path. As another example, the network can determine those specific pixels or voxels that are part of a left crop row, a right crop row, irrigation equipment, an animal, a person, a horizon, the end of a row, and other potential categories, and mark them as such. In a specific embodiment of the present invention, the label can be user-defined and specified by the user using training data labeled by a text string and a user-defined label. The output of the segmentation network can be a label of one or more elements of the image. A continuous set of elements to which a common label is applied can be referred to as a fragment of the image here. For example, a common label of "person" on a continuous set of portions of an image can be referred to as a fragment attributable to a person detected in the image.

[0031] In specific embodiments of the present invention where the segmentation network outputs multiple outputs, the segmentation network can provide more information than required for the row tracking task, and certain benefits can be achieved. For example, such an embodiment may not require a separate collision detection system in addition to the navigation system, thereby reducing the complexity required for the entire system. In addition, the ability to mark and thereby identify additional features that are elevated above the field and visible from a distance can allow the navigation system to locate itself within a crop row, field, or farm. Moreover, as described below, the addition of additional markings increases the degree of expected geometric and logical consistency between multiple segments, and thus the ability to mark additional features provides additional possibilities for geometric cross-checking of image markings. Therefore, the ability to perform more than two of these actions with a single system can improve the performance of the system and can alternatively or in combination reduce the computational complexity and resource requirements of the system.

[0032] Flowchart 100 continues with step 102, labeling portions of the image with labels using a segmentation network. This step can be performed automatically by the segmentation network, taking the image as input and producing a labeled image as output. For example, the image can be from Figure 2 The portion of the image may include crop row paths 201, and the marker may mark crop row paths 201 with matching tags. Figure 2 As shown, image 200B has been segmented by a segmentation network to produce segments 202, which are sets of pixels from portions of image 200B that have been labeled as paths. If the segmentation network is operating correctly, segment 202 should align with crop row path 201. In a specific embodiment of the invention, step 102 may include labeling a second portion of the image with a second label using the segmentation network. Step 102 may also include segmenting any number "X" of portions of the image, where "X" is the number of labels that the segmentation network has been designed to apply to the image elements.

[0033] Figure 3 An example of the output of a segmentation network used in a step such as step 102 is provided, where the segmentation network is designed to segment an input image into multiple segments with different labels. Thus, Figure 3 The step 310 of labeling X portions of the image with X labels using a segmentation network is included. For example, step 310 may include labeling a second portion of the image with a second label using a segmentation network. As shown, a single input image 300 of a portion of a crop row is segmented into five segments. These segments include segments labeled by a left crop row label 301, a right crop row label 302, a crop row path label 303, a sky label 304, and a person label 305. Figure 3The multiple labels shown in 300 can help with obstacle avoidance by avoiding collisions with portions of the image having human labels 305. For example, partially recognizing a human label 305 or recognizing a human label 305 surrounded by crop row path labels 303 can indicate that the robot should stop until manual input is provided to continue, or automatically continue after the assumed human obstacle no longer continues to appear in the output of the segmentation network. In addition, multiple labels applied to a single input image 300 can help with automatic geometric cross-checking of the segmentation network performance. For example, if part of the left row appears to the right of the right row, a defect in the segmentation can be detected.

[0034] In a specific embodiment of the present invention, the navigation system can derive a navigation path from the portion and label of the image generated as above. The navigation path can be the desired path that the robot is to take. For example, the path can be a set of specific positions where the portion of the robot that contacts the ground should contact the ground. For example, the path can be the path that the wheels of a wheeled robot are expected to roll along the ground. The process can include analyzing the portion and label (e.g., fragment) of the image associated with the inter-crop spacing path of the image, finding the centroid of the fragment in a set of transverse slices of the image from bottom to top, and linking the centroid of those fragments to the path. The derivation can be performed with various goals in mind, including avoiding obstacles, keeping the portion of the robot that contacts the ground as far away from the crops in the crop row as possible, minimizing changes in direction, and various other goals that are alternative or combined. These derivations can also utilize more than one label (e.g., a label for a left crop row and a right crop row) for deriving to maximize the path distance from two rows, or a label for deriving to avoid collisions between the robot and obstacles. Many alternative methods are possible, and the result is a navigation path for the robot in the image reference system.

[0035] Flowchart 100 continues with the step of deriving a navigation path from the portion of the image and the label. For example, the portion of the image may be Figure 2 The portion of the image marked as segment 202. Using this portion of the image, various mathematical analyses of this portion of the image and the labels, a navigation path can be derived. Figure 2 As shown in image 200C in , a derivation has been performed to guide the robot along the center of the path between crops, and thus a navigation path 203 is generated. The navigation path 203 can be defined with reference to the pixels of the image 200. For example, the navigation path can be a data structure that defines a set of pixel coordinates on the image 200C.

[0036] In a specific embodiment of the present invention, the navigation system may generate a control signal to cause the navigation system to track a navigation path generated according to the method disclosed herein, and then the control signal may be used to navigate the robot along the crop row. Therefore, the flowchart 100 includes a step 104 of generating a control signal for tracking by the autonomous navigation system, and a step 105 of navigating the robot to track the navigation path. The control signal may be generated to ensure that the robot tracks the navigation path. The process of generating the control signal may include a process of converting the navigation path in image coordinates into an executable control signal provided to the actuator of the robot. The control signal may include a carrot that the robot is designed to track, which is projected onto the navigation path at a specified distance from the robot. The controller may be designed to align the robot with the carrot and be able to reach the carrot. The update frequency of the carrot may be lower than the derivation frequency of the navigation path to prevent excessive noise in the control signal. However, in specific embodiments, since the robot may be designed to slow down before reaching the target, the update of the carrot should be faster than the speed at which the robot can approach the carrot. In these embodiments, the carrot should be updated fairly frequently to effectively keep it out of the reach of the robot. The control signals may include commands to the motors, gears and actuators of the robot to steer the robot, stop the robot or propel the robot forward or backward. The characteristics of the control signals will depend on the characteristics of the robot.

[0037] In a specific embodiment of the invention disclosed herein, the navigation path can provide a set point for the navigation system. For example, since the navigation path is continuously derived for additional images captured as the robot moves along the path, the generation of the control signal can include a feedback system that ensures that the lowest pixel or pixel group associated with the navigation path remains in the center of the bottom row of the image. The navigation path can provide a set point for a proportional-integral-derivative (PID) controller. Flowchart 100 includes a step 104 of generating a control signal for tracking by an autonomous navigation system, and a step 105 of navigating the robot to track the navigation path. The control signal can be a signal to move various actuators (e.g., motors, gears, and other actuators of the robot) to manipulate or otherwise navigate the robot. The generation of the control signal can be performed using a PID controller, wherein the navigation path provides a set point for the PID controller.

[0038] In a specific embodiment of the present invention, the navigation path can be used to determine whether there is an obstacle in the path of the robot. As described above, if the tags include tags for general obstacles, or tags for specific obstacles such as people, other agricultural equipment, etc., and the condition of the collision can be detected directly from the analysis of the image itself, the navigation system can stop the robot and wait for the obstacle to be removed or wait for human input to restart the robot. Alternatively, the navigation path can be used to determine whether the tag associated with the obstacle is in the path that the robot intends to travel. In these embodiments, the step of navigating the robot using the control signal can include at least temporarily stopping the robot to avoid the collision. Temporarily stopping the robot can include stopping movement until the obstacle is detected to be removed, or until the human operator restarts the navigation system.

[0039] In a specific embodiment of the invention, the navigation path can be used in the reference frame of the image, and in an alternative embodiment, the navigation path is converted to a different reference frame to navigate the robot along the path. For example, the navigation path can be converted into an earth-based coordinate system centered on the robot, or an earth-based coordinate system aligned with the crop row, the field in which the crop row is located, or the farm in which the crop row is located. The conversion of the navigation path to a different reference frame can be achieved by pre-registering one or more imagers used to generate the image with the reference frame. For example, the position of the robot in the reference frame can be monitored, and the position of the imager relative to the robot can also be monitored or fixed. The conversion can be combined or alternatively utilize sensors (such as gravity sensors, magnetometers, gyroscopes, inertial motion units, odometer sensors, etc.) to track the position and posture of the robot and / or imager relative to the reference frame.

[0040] Figure 4An image 400 and an image 420 are included that are captured by the same imager on a robot at two different times, wherein the image 400 is used as an input to a segmentation network to ultimately derive a navigation path 401. For example, the image 400 may be the image captured in step 101, and the navigation path 401 may be the navigation path derived in step 103. The flowchart 410 includes a set of methods that begin with a step 411 of transforming the navigation path to a reference frame. The navigation path may be the navigation path 401, and the reference frame may be an earth-based reference frame. The flowchart continues with step 412, where a second image of the crop row is captured, where the second image is registered in the reference frame. The second image may be image 420, and the second image may be registered relative to the earth-based reference frame using the same method and / or system associated with transforming the navigation path 401 to the reference frame. The flowchart 410 continues with step 413, where the navigation path is projected onto the second image. As shown projected onto the image 420, the projected navigation path 421 is no longer aligned with the center of the image. In this way, a controller using the projected navigation path 421 as part of the feedback control signal will guide the robot back in the counteracting direction to counteract the misalignment.

[0041] As described above, in specific embodiments of the invention disclosed herein, the segmentation of an image by a segmentation network may be enhanced using geometric reasoning and / or additional sensor data to confirm the performance of the segmentation network. If a deviation is detected between the geometric reasoning or additional sensor data and the output of the segmentation network, various steps may be taken in response.

[0042] In applications of navigating crop rows, there are many geometric factors available, regardless of the crop type and other factors faced by the specific robot. As mentioned earlier, with the available additional labels, many geometric reasoning cross-checks can be applied based on the expected geometric principles of various segments. However, even if the segmentation network only outputs a binary label for the crop row path, a geometric cross-check can still be applied to check the status of the segmentation. For example, a geometric reasoning cross-check can consider a pair of edges of a crop row segment. In this example, the geometric reasoning cross-check can be that in a "bird's eye view" view, the segmented crop row path should include two approximately parallel lines, and when the imager faces the direction of travel and is aligned with the crop row, the edges of the segmented crop row path should converge to a point. If these principles are violated, the navigation system can determine that the segmentation is flawed. As an example of additional geometric reasoning cross-checks available for more labels, the horizon line edge of the sky label should generally be flat and perpendicular to the reference frame of the forward-facing imager, if the sky label is available, the segmented crop row path should converge to a point at the horizon, and the left crop row segment and the right crop row segment should be on their respective sides of the image and separated by the crop row path label. The robot can be augmented with additional sensors that can be used to derive the geometry of the environment. For example, the robot can generate odometry data, gravity data, inertial motion unit data, depth data derived from active depth sensors or stereo image capture sensors, and a variety of other information sources. All of these information sources can be thought of as double checks on the segmentation network. Furthermore, the additional information can be combined with geometric reasoning to serve as a cross-check on the segmentation. For example, depth information can be used to cross-check with geometric reasoning to check labels, such as by comparing the segmented width of a crop row path to the distance to the center point where the width is measured.

[0043] In specific embodiments of the present invention, the navigation system may take various actions in response to detecting a deviation or defective segmentation. For example, in response to detecting a deviation, the navigation system may ignore the label and not generate a control signal based on the label. In embodiments where the control loop is fast enough, multiple images in a row may discard their labels in this manner without affecting the performance of the navigation system. However, in alternative methods, deviations may be handled in the same manner as obstacle detection as described above. In these embodiments, detecting a deviation may cause the robot to stop and wait for manual input to proceed, or stop until the robot captures an image with a label that does not violate the cross-check. Alternatively or in combination, detecting a deviation may prompt the system (e.g., by notifying a human operator and presenting a user interface for receiving label input from the operator) to request additional training data. In specific embodiments, the image presented to the operator for annotation will be the image in which the deviation is detected. The deviated segmentation may be overlaid on the image and presented to the user to allow them to modify it by providing a correction or by providing a completely new segmentation label input. In response to detecting a potential obstacle, a similar response may be taken.

[0044] Figure 5 An image 500 is shown having a labeled portion 501 of the image. Flowchart 510 includes a step 511 of checking the label of the portion of the image using an expected geometric principle associated with navigating multiple crop row groups. In this case, the principle is that the two edges of the labeled portion 501 should converge. As shown, the principle is violated and therefore the check will fail. As such, flowchart 510 continues with step 513, overwriting the label. Thus, the robot will skip generating control signals based on image 500 and will only begin generating control signals again when image 520 is used to generate a segment that does not violate the geometric check. The robot can also be designed to pause and wait for manual input before continuing when a difference is detected. Flowchart 510 may alternatively or in combination include a step 514 of capturing depth information to be used in iterations of step 511.

[0045] In a specific embodiment of the invention, the segmentation network will be a trained machine intelligence system trained using a supervised training routine. In a specific embodiment of the invention, training can be performed in various ways and will depend on the characteristics of the machine intelligence system being trained. For the avoidance of doubt, as used herein, the term "trained machine intelligence system" includes a system that has been configured but is still in its default state (i.e., before applying training data). The training process typically includes collecting training data, inputting the training data into the trained machine intelligence system, and adjusting the characteristics of the trained machine intelligence system based on the output generated in response to the training data input. The adjustment step may include using matching output training data associated with the input training data to see if the output generated by the trained machine intelligence system matches the expected output. This training of input data for which output training data is available can be referred to as labeled training data.

[0046] Typically, the training data will match the data format that the trained machine intelligence system will receive as input when deployed. For example, if the deployed system includes other data in addition to image data (e.g., the odometry data mentioned above), the training data may include that data as well. As another example, if the deployed system includes a visible light camera that will capture RGB images of objects, the training data may be RGB images of objects.

[0047] The training data may be collected (e.g., using a camera to capture images of portions of a crop row) or may be synthesized (e.g., as described below, by augmenting captured images or using a three-dimensional model of a crop row). Synthesized training data may be referred to as synthetic data. If the training data is collected, the benchmark true training output data may be captured by presenting the images to a human operator on a user interface and receiving labeled input from the human operator. Regardless of how the data is captured, the input training data may be modified with differences to improve the performance of the trained machine intelligence system. In addition, the original capture and synthesis of the device may be performed in a manner that intentionally introduces differences. These differences may include differences in lighting, color, adding atmospheric or weather effects, changing the surface material of an object, dust effects, slight perturbations of the sensor pose, noise, and other distortions to ensure that the trained machine intelligence system will operate regardless of errors in the sensors or changes in conditions during system operation. The training data may also be generated using differences generated by a difference model of sensors that will obtain input data when the trained machine intelligence system is deployed. In these embodiments, the trained machine intelligence system will be combined with a set of images of known objects through its training, where the known objects exhibit samples with the above-mentioned differences (e.g., the known objects are off-center in the image).

[0048] In specific embodiments of the present invention, in contrast to conventional training methods, the segmentation network can be specifically trained or tuned for a specific set of crop rows in a single field or a single farm. In these embodiments, the navigation system can include a segmentation network in an initial state that is subsequently tuned or trained on images from a specific set of crop rows in which the navigation system will be deployed. This is somewhat counterintuitive in the field of machine intelligence, since the goal is generally to generalize systems across a wide range of operating environments (e.g., systems that can recognize all types of rows even without additional training). The problem with such general systems is that for complex environments, they may require a large number of labeled images, perhaps billions of images before the row detector of a general robot learns. In contrast, a specific farmer only cares about their specific crop rows. In this case, it is possible to overtrain on only a few examples of a specific set of labeled images. For example, using the methods disclosed herein, the labeled training data set required to provide sufficient row tracking navigation performance (including avoiding collisions with obstacles such as people or field irrigation equipment) can be as small as 10 to 15 frames of labeled data. The system will then perform very well on data or scenes that closely match the small set of labeled images, but at the expense of being less general on images that may be encountered in different environments. This tradeoff between generality and focus can be extended so that the segmentation network is trained on groups of crop rows that share a single crop type. However, while such a network is more generalizable, it may require more training data before reaching a point where it can perform reliably.

[0049] The training process of the segmentation network according to specific embodiments of the present invention can include manually (e.g., using a joystick and a human driver) navigating the robot along the crop rows while capturing the image sets. In specific embodiments, the human driver can guide the robot along the crop rows to obtain these initial images. Typically, when the robot is used autonomously, the same sensors that will guide the navigation system are used to capture the training images. Using these methods, as an added benefit of training on the specific set of crop rows that the robot will operate, the navigation system will also be trained for any error distortions of the dedicated sensors that will be used to guide the robot.

[0050] Flowchart 600 illustrates a set of methods for training a segmentation network according to a specific embodiment of the invention disclosed herein. Flowchart 600 begins with step 601 of capturing an image of at least a portion of a crop row group. As shown, the flowchart includes loops because a set of images can be captured through multiple iterations of step 601. The sets of images captured through these iterations can all be images of at least a portion of a crop row group. They can be from the same row or from different rows. The process can also include instructing a human operator to capture images from adjacent rows. The images can be captured by the same sensor used to capture image 200 and other sensors described above. Figure 7 An image 700 that can be obtained in this step is provided. Flowchart 600 also includes training a segmentation network using an image or set of images. In a specific embodiment, before the image set can be used to train a segmentation network, they must be labeled using manual input or an alternative method as described below.

[0051] In a specific embodiment of the present invention, an intuitive interface is provided for a human operator to easily and efficiently generate labels for an image set that will be used as training data for a segmentation network. The interface may display images to a user, for example, on a touch screen, and accept input from the user to label one or more portions of the image. For example, the image may be displayed on a tablet computer to a user who is with the robot on site. Alternatively, the image may be displayed on an integrated display on the robot. Flowchart 600 includes a step 602 of displaying the image on a user interface. Step 602 may be repeated throughout the process loop so that it includes displaying the image set on the user interface. In a specific embodiment, multiple images may be captured in step 601, and the image set may be provided to the user in batches in a single iteration of step 602. For example, the captured images may be from a video sequence obtained when the robot is manually navigated to a crop row, and the automated system may select still images from the video feed to display in step 602. The automated system may select still images based on detected differences in image content, and preferentially select images with greater differences for labeling. Alternatively, the automated system may select still images from the video feed at random times or at fixed intervals.

[0052] In a specific embodiment of the present invention, a human operator may provide label input to an image in a variety of ways. Label input may be provided directly on a displayed image or with reference to a displayed image. Thus, flowchart 600 includes a step 603 of accepting a label input set on an image set of a user interface, wherein the image set may be an image set displayed in one or more iterations of step 602. For example, the label input set may be a set of sliding inputs provided on a touch screen user interface to label certain portions of an image as crop row paths and certain portions of an image as non-paths. The sliding input may point to an area of ​​an image and be used directly as a label input for training data. Alternatively, a standard computer vision processing system may utilize the sliding input to expand and / or reduce the sliding input (e.g., using edge analysis to expand an approximate sliding on a left crop row to select all pixels adjacent to the sliding and having pixel values ​​within the median range of the pixel values ​​selected by the sliding) to fit the image region identified as continuous by the processing system. As another example, the label input set may be a set of polygon inputs provided on a touch screen user interface. The polygon input may be provided in a series of taps, with the user interface automatically connecting consecutive tap points on the image until a closed loop path is formed. Alternatively or in combination, a polygonal template may be provided for the user to drag onto the display and resize appropriately. The polygonal template may be selected from a group of general shapes that conform to the marked portion (e.g., crop row path markers for a forward-looking sensor may be general triangles). Standard computer vision processing systems may also be applied to estimate the location of specific regions in the image and provide a best guess label set for the image along with a user interface option to allow the user to adjust the polygon provided by the user.

[0053] In a specific embodiment of the present invention, a human operator may provide a label input to an image with respect to more than one label. For example, a label input set may point to a set with at least three different labels. Different labels may be selected individually from a menu so that a user may select a label and then provide an associated label input on an image with the selected associated label or on an image associated with the selected associated label. Alternatively, different labels may be presented to the user in sequence, and the user may provide associated label inputs for these labels when prompted. In a specific embodiment of the present invention, a label may be applied to at least one user-defined label. In these embodiments, a user may be provided with the option of using a keyboard or voice input to specify a text string representing a label, and then provide an associated label input associated with the label they specified themselves. For example, a user may identify different kinds of crops, or repetitive features specific to their fields or farms, which may help guide a robot along crop rows and prevent the robot from colliding with obstacles. In a specific embodiment, a user may specify whether a label is associated with a landmark or an obstacle so that the navigation system may utilize segmentation appropriately.

[0054] Figure 7 A user interface is shown on which an image 700 is displayed to receive multiple label inputs from a user. As shown, the label input is in the form of a polygon input with which the user specifies a crop row path polygon 712 and a non-crop row path polygon 710. In the illustrated case, the polygon input is provided by the user tapping the corners of the polygon in a pattern that returns to the original tap point. The user specifies two different polygons, and the user interface requires two different labels. As shown, the polygons are not fully aligned with the identified areas. However, the inventors have found that even these rough inputs are sufficient to generate training data to enable the robot to fully perform a navigation program in a crop row. Likewise, in specific embodiments, the rough inputs are used directly as training data, while in other methods, the rough inputs are processed by a standard computer vision routine to more specifically select image regions associated with a given label. In these embodiments, the user may be given the option of viewing the outputs of the standard computer vision routines and accepting them or overwriting them in whole or in part. The method disclosed in this paragraph can generate a labeled training image 720, wherein the image 700 is used as input and the label 721 is used as the expected output of the segmentation network. The labeled training images can then be used to train the segmentation network 730 .

[0055] In a specific embodiment of the present invention, the set of images that need to be annotated by a human user can be a small number, such as 15-30 frames, while still generating enough training data to fully segment the path from the image set of crop rows on the same farm. This is beneficial because it reduces the time for human operators to set up the robot to work on the farm. Combined with the fact that in a specific embodiment, the labeled area does not need to correspond exactly to the labeled area, this produces an extremely effective method for generating the required training data. Due to the minimum performance required to fully segment the path for the purpose of guiding the robot to track the crop row, this low number of frames and low accuracy are sufficient. This low performance requirement is due to the usually clearly visible difference between a row of green plants and the brown soil between the crowded rows, and the relatively low speed of agricultural robots compared to other automatic vehicles (such as cars on the highway). In addition, since the labeled training data can be expanded using training data synthesis (in training data synthesis, the original labeled image is modified as described below), a low number of frames can still provide a large amount of training data. Using these techniques, thousands of labeled training inputs can be generated from 15-30 frames of labeled training data, and this larger set of thousands of training inputs can be used to train the segmentation network.

[0056] In addition, in specific embodiments of the invention disclosed herein, labeled training data can be obtained entirely without label input from a human operator. For example, a method can include capturing a set of images of at least a portion of the crop row group while navigating the robot along the crop row, and performing photogrammetric analysis or optical flow analysis on the image set to generate a set of label inputs on the image set. The photogrammetric analysis includes solving for the path position in the first image based on an analysis of a second image captured subsequently. The analysis of the second image captured subsequently will include analyzing a common set of one or more features of the two images and determining how the robot moves between the two images. This movement can then be projected onto the first image, with the projection thereby serving as an identification of at least a portion of the path between the crop rows. In these embodiments, the user may only be required to manually manipulate the robot along the crop row, and the system will capture a series of more than two images required for such photogrammetric analysis. Photogrammetric analysis or optical flow can include the derivation of a large set of images and paths through the images to generate labeled training data images required for training a segmentation network. For example, such a process can automatically generate labeled training images 720, where image 700 is used as input and label 721 is used as the expected output of the segmentation network. The labeled training images can then be used to train the segmentation network 730 .

[0057] In specific embodiments of the invention, labeled training data may be used to train a segmentation network. In these embodiments, labeled images may be provided as input to the segmentation network, and the labels generated by the segmentation network in response thereto may be compared to labels collected using any of the processes described above. The difference between the two may then be used to update the weights, filters, or other values ​​of the segmentation network based on the characteristics of the segmentation network. This step may be iterative, as the accuracy of the segmentation network may be monitored by an automated system or a human operator, and additional training data may be synthesized or obtained to bring the performance of the segmentation network to a desired level. Once trained, the segmentation network may be used by a navigation system to navigate the robot in any manner. For example, the segmentation network may be used to perform the above-referenced method. Figure 1 Navigate the robot in the manner described. Examples of these steps are given in Figure 6 The method of claim 6 is provided in steps 605 and 607. Step 605 includes training a segmentation network using a label input set such as obtained in step 605 and an image set such as obtained in step 601. Step 607 includes navigating the robot along the crop row using the segmentation network after training the segmentation network (e.g., in step 605).

[0058] In a particular embodiment of the invention, the progress of the training may be displayed to a human operator. For example, a simple progress bar may be presented to the user to indicate how many more images need to be collected or how long they need to manually navigate the robot. The progress indicator may also include prompts to instruct the user to capture images of different portions of a crop row or different crop rows. The training progress indicator may also include displaying the performance of the segmentation network in labeling images captured by the robot's sensors. For example, a user may view additional images collected by the robot's sensors while the navigation system labels the images in real time as it is being trained. Alternatively, the system may play back a video captured by the robot with labels overlaid on the images. In either case, the user may review the segmentation performance and determine whether it is acceptable. The user may provide input to the system to indicate that the training is complete when this determination is made. Alternatively, the system may determine that the training is complete by evaluating the performance of the segmentation network against additional training data that was not used to train the segmentation network.

[0059] Flowchart 600 includes a step 606 of displaying a segmentation network training progress indicator on a user interface. The segmentation network training progress indicator may include a visual depiction of the labeling of the segmentation network on a test image. For example, image 700 may be presented to the user with label 721 overlaid on the image, as shown by labeled training image 720. In this example, the training image is a test image because it is not used for training images, but is used to test the performance of the segmentation network and present it to the user. In this embodiment, the training progress indicator is provided by a user who manually assesses how well the segmentation network performs on the image (e.g., if performance is low, not much progress has been made). In a specific embodiment, the segmentation network training progress indicator may include an indication that more images are needed. The indication may be a public indication in text form output with the message, or a progress bar indicating the number of images that the system expects to be required to complete the training of the segmentation network.

[0060] In a specific embodiment of the present invention, the training process of the segmentation network is enhanced by the step of generating synthetic data to train the segmentation network. This step is represented as step 604 in the flowchart 600, using the image set captured in a single or multiple iterations of step 601, to generate a larger set of synthetic training images. This step may include taking images and distorting images, modifying their geometry, adding noise, changing the lighting by re-illumination, blurring images, recoloring images, and various other techniques to generate a larger set of images. In a specific embodiment of the present invention, the image set may be less than 100 or less than 20 images, and the larger set of synthetic training images may be greater than 500 images. In a specific embodiment of the present invention, this step may include creating a 3D model of the crop row using photogrammetry, so that the image can be truly enhanced by the detected actual lighting conditions and the real surface reflection model of the elements in the crop row (such as plants or soil). The larger set of images generated according to any of the methods disclosed above can then be used in step 605 to train the segmentation network to produce a segmentation network with a higher degree of generality and performance than the network trained on the original images captured only by one or more iterations of step 601.

[0061] In a specific embodiment of the present invention, a navigation system for navigating a robot along a crop row includes sensors and actuators on the robot. The system may also include a device for capturing an image of at least a portion of the crop row using a sensor. The device for capturing an image may be any sensor capable of capturing an image as mentioned in the above disclosure. The device for capturing an image may also include instructions stored in a non-transitory computer-readable medium to control those sensors so that the sensors capture sensor data and store data generated by the sensors. The system may also include a segmentation network such as described herein to mark portions of an image captured by the device for capturing an image. The system may also include a device for deriving a navigation path from a portion of an image having a label generated by the segmentation network. The device for deriving a navigation path may include instructions stored in a non-transitory computer-readable medium to perform step 103 described herein. The system may also include a device for generating a control signal for an actuator to cause the robot to track the navigation path. The device for generating a control signal may include instructions stored in a non-transitory computer-readable medium to perform step 104 described herein.

[0062] The system may also include means for checking labels of image portions using expected geometric principles associated with the navigation crop row group. The means for checking labels of image portions using expected geometric principles associated with the navigation crop row group may include instructions stored in a non-transitory computer readable medium to perform a comparison of relative portions of the labeled image as described above, such as storing geometric principles and evaluating geometric principles of segment edges, evaluating the relative position of one or more segments, evaluating the overlap of one or more segments, or evaluating the relative position of one or more segments in the image to detect differences from the geometric principles. The system may also include means for covering labels if the expected geometric principles are violated. The means for covering labels if the expected geometric principles are violated may include instructions stored in a non-transitory computer readable medium to cause the system to interrupt reference to the geometric principles. Figure 1 The control flow of the navigation control loop described above is described, and the segmentation is discarded. Therefore, covering labels need not include presenting alternative labels in place of the labels that caused the system to detect a difference.

[0063] The system may also include means for capturing depth information from the crop rows. The means for capturing depth information from the crop rows may be a dedicated depth sensor, such as an active or passive depth sensor (e.g., an ultraviolet projected depth sensor), a LIDAR system, a radar, or any other form of depth sensor including one or more visible light sensors that derives depth information using stereo vision processing techniques.

[0064] The system may also include means for converting the navigation path to the reference system. The means for converting the navigation path to the reference system may include instructions stored in a non-transitory computer readable medium to evaluate an image in which the navigation path is defined, access a registration of a sensor that captured the image with the reference system, and apply the registration to transform the navigation path into coordinates in the reference system.

[0065] The system may also include means for capturing a second image of the crop row.The means for generating the second image may include any of the sensors and other means described with reference to the means for capturing an image as described above.

[0066] The system may also include means for projecting the navigation path onto the second image. The means for projecting the navigation path onto the second image may include instructions stored in a non-transitory computer readable medium to evaluate the second image, access a registration of a sensor that captured the second image to the reference system, and reversely apply the registration to change coordinates of the navigation path in the reference system onto elements of the second image.

[0067] The system may also include means for capturing a set of images of at least a portion of the crop row set.The means for capturing a set of images may include any of the sensors and other means described with reference to the means for capturing images described above.

[0068] The system may also include means for generating a larger set of synthetic training images using the image set. The means for generating a larger set of synthetic training images using the image set may include instructions stored in a non-transitory computer readable medium to perform the method described above with reference to step 604, and may include a random number generator, an image filter, a model of ambient lighting when the image set was captured, and a three-dimensional model of one or more crop rows from which the images were obtained.

[0069] The system may also include means for training a segmentation network using the image set. The means for training a segmentation network using the image set may include instructions stored in a non-transitory computer-readable medium to perform the above step 605, including a system for applying a training input to the segmentation network, a system for evaluating a resulting output of the segmentation network by comparing with a matching expected training output associated with the training input, and a system for adjusting a value defining the segmentation network based on the evaluation.

[0070] Segmentation networks trained using specific embodiments of the invention disclosed herein, and training data obtained using specific embodiments of the invention disclosed herein, can, over time, be used to produce a general network that can operate on any farm having any type of crop rows without the need for pre-captured images at that farm. Even in this case, however, certain benefits may still arise from allowing the network to be overtrained on specific crop rows or on farm-specific structures, such as hydroponic support poles or irrigation pipes in or around which the network will operate. Given the minimal amount of work required to train a navigation system, it is more likely that such a process will be undertaken no matter how general or accurate the network becomes.

[0071] There are many applications for the various embodiments of the invention disclosed herein. For example, once a path is segmented from a portion of an image, the path can be passed to a robotic system, a navigation system for a cart or drone, a human or vehicle tracker, a landing or docking system for an aircraft, or an "assisted teleoperation" system. In the example of assisted teleoperation, a vehicle or robot can travel or move primarily by itself (respectively), thereby alleviating the burden of a human operator who directs the vehicle or robot but does not need to spend too much time. This allows one operator to potentially command many vehicles or robots. For example, in assisted teleoperation, a farmer can direct several robots to the general starting point of a row in several fields. Once the robot arrives there, it can notify the operator. The operator can then release the robot and start weeding the crop row, and the robot knows how to move along the plant row without running over the plants. It can then allow the operator to teleoperate and let the robot turn around, or the robot itself can fully know how to turn around and do another row. This allows the operator to have more freedom to perform high-level operations, rather than having to specifically drive each vehicle or robot.

[0072] Although the specification has been described in detail for specific embodiments of the present invention, it should be understood that those skilled in the art can easily conceive of changes, variations and equivalents of these embodiments based on an understanding of the foregoing. For example, although examples of agricultural applications with field robots are used throughout the specification, the specific embodiments of the present invention disclosed herein are more broadly applicable to any robot navigation or robot manipulation application. In addition, although examples of field robots tracking crop rows are used throughout the specification, the specific embodiments of the present invention disclosed herein are more broadly applicable to various other autonomous navigation applications in which the robot is designed to track a path. These modifications and variations of the present invention may be implemented by those skilled in the art without departing from the scope of the present invention, and the scope of the present invention is more specifically set forth in the appended claims.

Claims

1. A method (100) for navigating a robot (110) along a crop row, wherein each step is implemented by a navigation system (111) for the robot (110) using a computer, the method (100) comprising: capturing (101, 412) an image of at least a portion of the crop row; labeling (102, 310) portions of the image (200) with labels using a segmentation network; deriving (103) a navigation path (203, 401) from the portion of the image (200) and the label; generating (104) a control signal for the navigation system (111) to track the navigation path (203, 401); and The robot (110) is navigated (105) along the crop row using the control signal.

2. The method (100) according to claim 1, further comprising: checking labels (511) of portions of the image (200) using expected geometry associated with navigation crop row groups; as well as If the expected geometry is violated, the label is covered (513).

3. The method (100) according to claim 2, further comprising: capturing depth information from the crop row (514); Wherein checking the tag (511) using the expected geometry comprises using the depth information.

4. The method (100) of claim 2, wherein: The tags are used for crop row paths (201); and The expected geometry associated with navigating crop row groups involves edge pairs of portions of the image (200).

5. The method (100) of claim 1, wherein: The control signal is generated using a proportional-integral-derivative controller (104); and The navigation path (203, 401) provides a set point for the proportional-integral-derivative controller.

6. The method (100) of claim 1, wherein generating (104) the control signal comprises: converting (411) the navigation path (203, 401) to a reference system; capturing (412) a second image (420) of the crop row, wherein the second image (420) is registered in the reference system; as well as The navigation path (203, 401) is projected (413) onto the second image (420).

7. The method (100) according to claim 1, before capturing (101) the image (200), further comprising: capturing (601) a set of images (200) of at least a portion of a crop row group; Displaying (602) the set of images (200) on a user interface; Accepting (603) a set of label inputs on the image set (200) of the user interface; as well as The segmentation network is trained (605) using the labeled input set and the set of images (200).

8. The method (100) of claim 7, wherein: The crop rows are in the crop row groups; and The crop row groups are one of: (i) on a single farm; (ii) sharing a single crop type.

9. The method (100) of claim 1, before capturing (101) the image (200), further comprising: capturing (601) a set of images of at least a portion of a crop row group; as well as training (605) the segmentation network using the image set; The crop rows are in the crop row group, and the crop row group is one of: (i) on a single farm; or (ii) shares a single crop type.

10. The method (100) according to claim 9, before capturing (101) the image (200), further comprising: generating (604) a larger set of synthetic training images using the set of images; as well as wherein: (i) training (605) the segmentation network using the image set comprises training (605) the segmentation network using the larger synthetic training image set; and (ii) the image set is smaller than 100 and the larger synthetic training image set is larger than 500.

11. The method (100) according to claim 10, further comprising: The generating (604) of the larger set of synthetic training images includes performing at least one operation selected from the group consisting of distorting, blurring, relighting, and recoloring on the set of images.

12. The method (100) of claim 1, further comprising: labeling (102, 310) a second portion of the image (200) with a second label using the segmentation network; wherein: (i) the tags are for crop row paths (201); (ii) the second tags are for potential obstacles; and (iii) navigating the robot (110) along the crop rows using the control signals includes at least temporarily stopping the robot (110) to avoid collisions.

13. The method (100) of claim 1, wherein: The image (200) is captured by at least one imager (112, 113, 114) on the robot (110); and The segmentation network is implemented by a computer on the robot (110).

14. The method (100) of claim 1, wherein: The robot (110) includes at least two imagers (112, 113, 114); capturing the image (200) using the at least two imagers (112, 113, 114); and The image (200) includes depth information.

15. A method (100) for navigating (105) a robot (110) along crop rows in a crop row group, the method (100) comprising: capturing (601) a set of images of at least a portion of the crop row group; displaying (602) the image set on a user interface; accepting (603) a set of label inputs on the set of images in the user interface; Using the labeled input set and the image set, training (605) a segmentation network; as well as After training the segmentation network, the robot (110) is navigated (607) along the crop rows using the segmentation network.

16. The method (100) of claim 15, further comprising: generating (604) a larger set of synthetic training images using the set of images; wherein: (i) training (605) the segmentation network using the labeled input set and the image set includes training the segmentation network using the larger synthetic training image set; and (ii) the image set is less than 100 and the larger synthetic training image set is greater than 500.

17. The method (100) of claim 16, wherein: The generating (604) of the larger set of synthetic training images includes performing at least one operation selected from the group consisting of distorting, blurring, relighting, and recoloring on the set of images.

18. The method (100) of claim 15, further comprising: A segmentation network training progress indicator is displayed (606) on the user interface.

19. The method (100) of claim 18, wherein: The segmentation network training progress indicator includes a visual depiction of a marker (102, 310) of the segmentation network on a test image.

20. The method (100) of claim 15, wherein: The training (605) of the segmentation network includes displaying an indication that more images are needed.

21. The method (100) of claim 15, wherein: The tab input set is a set of sliding inputs on the user interface.

22. The method (600) of claim 15, wherein: The label input set is a set of polygon inputs on the user interface.

23. The method (100) of claim 15, wherein: The tag input set points to a set of at least three different tags.

24. The method (100) of claim 23, wherein: The set of at least three different tags includes at least one user-defined tag.

25. The method (100) of claim 23, wherein: The set of at least three different labels includes labels associated with humans.

26. A navigation system (111) for navigating a robot (110) along a crop row, comprising: sensor; an actuator on the robot (110); means for capturing (101, 412) an image (200) of at least a portion of the crop row using the sensor; a segmentation network for labeling (102, 310) portions of the image (200) with labels; means for deriving (103) a navigation path (203, 401) from a portion of said image (200) with labels; as well as Means for generating (104) control signals for the actuator to cause the robot (110) to follow the navigation path (203, 401).

27. The navigation system (111) of claim 26, further comprising: means for inspecting (511) a label of a portion of the image (200) using an expected geometry associated with a navigation crop row group; as well as Means for covering (513) the tag if the expected geometry is violated.

28. The navigation system (111) of claim 27, further comprising: means for capturing (514) depth information from said crop row; Wherein checking (511) the tag using the expected geometry comprises using the depth information.

29. The navigation system (111) of claim 26, further comprising: for converting the navigation path (203; 401) means for converting (411) to a reference frame; means for capturing (412) a second image (420) of the crop row, wherein the second image (420) is registered in the reference system; as well as Means for projecting (413) the navigation path (203, 401) onto the second image (420).

30. The navigation system (111) of claim 1, before capturing (101) the image (200), further comprising: means for capturing (601) a set of images of at least a portion of a crop row group; means for generating (604) a larger set of synthetic training images using the set of images; as well as means for training (605) said segmentation network using said image set; The crop rows are in the crop row group, the crop row group is one of: (i) on a single farm; or (ii) shares a single crop type, and the image set is less than 100 and the larger synthetic training image set is greater than 500.

Images