System and method for autonomous field navigation
The system enables precise autonomous navigation of agricultural vehicles along furrows using sensor-based detection and machine learning, addressing inefficiencies and crop damage in farming.
Patent Information
- Application Number
- CN202380036891.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-29
- Filing Date
- 2023-04-21
- Publication Date
- 2024-12-06
AI Technical Summary
Agricultural navigation systems lack autonomy and precision, leading to inefficiencies in farming and potential crop damage from vehicles navigating outside designated furrows.
A system and method for autonomously navigating agricultural vehicles using sensors and machine learning to detect furrows and adjust vehicle paths within specified turning zones, enabling precise field entry and exit without damaging crops.
Enhances farming efficiency by allowing vehicles to navigate along furrows with high precision, reducing crop damage and improving labor efficiency.
Smart Images

Figure CN119095481A_ABST
Abstract
Description
[0001] Cross-references
[0002] This patent application claims the benefit of U.S. Provisional Application No. 63 / 336,782, filed on April 29, 2022, entitled “SYSTEMS AND METHODS FOR AUTONOMOUS FIELD NAVIGATION,” which is incorporated herein by reference in its entirety for all purposes. Background Art
[0003] Agriculture is a multi-trillion dollar industry worldwide. Agriculture is a fundamental component of food production and includes the cultivation of livestock and plants. Population growth and declining crop yields due to climate change threaten global food security. Methods to increase agricultural production by improving crop yields and increasing labor efficiency can help alleviate food shortages. Navigation systems with greater autonomy and precision are needed to improve farming efficiency. Summary of the invention
[0004] In various aspects, the present disclosure provides a system for autonomously driving a vehicle, the system comprising: a steering controller for driving the vehicle; a processor coupled to the steering controller, wherein the processor is configured to determine a first turning zone, the first turning zone corresponding to a first field exit path having a first field exit position and a first field re-entry path and having a first field re-entry position, wherein the first turning zone is bounded by an outer boundary and an inner boundary, wherein the outer boundary encloses the inner boundary, wherein the inner boundary extends through the first field exit position and the first field re-entry position, and wherein the inner boundary encloses a planting area field; a position sensor coupled to the processor, the position sensor for identifying a position of the vehicle; and a position sensor coupled to the processor. A memory of the processor storing instructions that, when executed by the processor, cause the steering controller to perform a first turn by driving the vehicle along the first field exit path at the inner boundary toward the first field exit position, driving the vehicle toward a first forward position when encountering the inner boundary at the first field exit position, reversing the vehicle toward a first correction position when encountering the outer boundary, driving the vehicle toward the first forward position when reaching the first correction position, reversing the vehicle toward a first reverse position when reaching the first forward position, and driving the vehicle forward at the inner boundary toward the first field re-entry position when reaching the first reverse position or encountering the outer boundary.
[0005] In some aspects, the first forward position is positioned relative to the first field re-entry position based on a vehicle turning radius and a vehicle width. In some aspects, the first forward position is positioned at least one vehicle width from the first field re-entry position in a direction opposite to the first field re-entry path and at least one vehicle turning radius from the first field re-entry position in a direction opposite to the first field exit position. In some aspects, the first forward heading is parallel to a line connecting the first field exit position to the first field re-entry position. In some aspects, the first correction position is positioned relative to the first field exit position based on a vehicle length. In some aspects, the first correction position is positioned at least one vehicle length from the first field exit position in a direction opposite to the first field re-entry position. In some aspects, the first reverse position is positioned relative to the first field re-entry position based on a vehicle length. In some aspects, the first reverse position is positioned at least one vehicle length from the first field re-entry position in a direction opposite to the first field re-entry path. In some aspects, the first field re-entry position is positioned along the inner boundary. In some aspects, the first field exit path, the first field re-entry path, or both are linear. In some aspects, the first field exit path is parallel to the first field re-entry path.
[0006] In some aspects, the inner boundary is curvilinear. In some aspects, the outer boundary is curvilinear. In some aspects, the area of the inner boundary between the first field exit location and the first field re-entry location is not perpendicular to the first field exit path, the first field re-entry path, or both. In some aspects, the inner boundary encloses a field. In some aspects, the outer boundary is enclosed by an outer geofence. In some aspects, the outer boundary is separated from the outer geofence by at least one vehicle length. In some aspects, the first field exit path is separated from the first field re-entry path by a furrow spacing distance. In some aspects, the first field exit path, the first field re-entry path, or both are parallel to furrows in the field.
[0007] In some aspects, the first turn is further performed by: reversing the vehicle toward the first field re-entry position before reaching the first reverse position and encountering the outer boundary; and reversing the vehicle toward the first reverse position when reaching the first field re-entry position oriented in a direction other than along the first field re-entry path. In some aspects, the first turn is determined in real time, adaptively determined, or both. In some aspects, the first turn is updated as the first turn is performed.
[0008] In some aspects, the processor is further configured to determine a second turning zone, the second turning zone corresponding to a second field exit path having a second field exit position and a second field re-entry path and having a second field re-entry position, wherein the second turning zone is bounded by the outer boundary and the inner boundary, wherein the inner boundary extends through the second field exit position and the second field re-entry position; and wherein the instructions further cause the steering controller to perform a second turn by: driving the vehicle along the second field exit path toward the second field exit position; when encountering the inner boundary at the second field exit position, driving the vehicle toward a second forward position; when reaching the second forward position, reversing the vehicle toward a second reverse position; when reaching the second reverse position or encountering the outer boundary, reversing the vehicle toward the second field re-entry position at the inner boundary. In some aspects, the first turn and the second turn are different. In some aspects, the second turn is determined in real time, adaptively determined, or both. In some aspects, the second turn is updated as the first turn is performed.
[0009] In some aspects, the processor is further configured to detect a furrow between the first field re-entry position and the second field exit position. In some aspects, the process is configured to identify a furrow in a depth image of a field and a visual image of the field, determine a furrow position, and determine a furrow direction. In some aspects, identifying the furrow further comprises generating a segmented map from the visual image. In some aspects, determining the furrow direction further comprises determining a center of the furrow from the depth image. In some aspects, determining the furrow position and determining the furrow direction further comprises determining a position value and a direction value of a pixel of the visual image corresponding to the furrow. In some aspects, the processor is further configured to determine a representative position and a representative direction based on the position value and the direction value, wherein the representative position corresponds to the furrow position and the representative direction corresponds to the furrow direction.
[0010] In some aspects, the instructions, when executed by the processor, further cause the steering controller to travel along the furrow to the second field exit position. In some aspects, the first field exit position, the first field exit path, the first field re-entry position, the first field re-entry path, the first forward position, the first correction position, the first reverse position, the second field exit position, the second field exit path, the second field re-entry position, the second field re-entry path, the second forward position, the second reverse position, or a combination thereof, are determined based on sensor input. In some aspects, the first field exit position, the first field exit path, the first field re-entry position, the first field re-entry path, the first forward position, the first correction position, the first reverse position, the second field exit position, the second field exit path, the second field re-entry position, the second field re-entry path, the second forward position, the second reverse position, or a combination thereof, are determined in real time. In some aspects, the first field exit position, the first field exit path, the first field re-entry position, the first field re-entry path, the first forward position, the first correction position, the first reverse position, the second field exit position, the second field exit path, the second field re-entry position, the second field re-entry path, the second forward position, the second reverse position, or a combination thereof are adaptively determined. In some aspects, the first field exit position, the first field exit path, the first field re-entry position, the first field re-entry path, the first forward position, the first correction position, the first reverse position, the second field exit position, the second field exit path, the second field re-entry position, the second field re-entry path, the second forward position, the second reverse position, or a combination thereof are independent of a speed of the vehicle. In some aspects, the first field exit position, the first field exit path, the first field re-entry position, the first field re-entry path, the first forward position, the first correction position, the first reverse position, the second field exit position, the second field exit path, the second field re-entry position, the second field re-entry path, the second forward position, the second reverse position, or a combination thereof is independent of a turning rate of the vehicle.
[0011] In some aspects, the processor is further configured to generate an alarm to request human intervention. In some aspects, the alarm is generated when the vehicle fails to reach the first field exit position, the first correction position, the first forward position, the first reverse position, the first field re-entry position, the second field exit position, the second correction position, the second forward position, the second reverse position, or the second field re-entry position after a predetermined number of attempts. In some aspects, the predetermined number of attempts is 2 to 10 attempts, 3 to 8 attempts, or 4 to 6 attempts. In some aspects, the predetermined number of attempts is about 5 attempts. In some aspects, the alarm is generated when the vehicle crosses the outer boundary or exceeds the outer boundary by a predetermined distance. In some aspects, the predetermined distance is 0.5 to 1.5 vehicle lengths. In some aspects, the predetermined distance is about one vehicle length. In some aspects, the alarm is generated when the system fails to detect a furrow at the first field re-entry position or between the first field re-entry position and the second field exit position. In some aspects, the alarm is generated when the system detects an obstacle. In some aspects, the alert is configured to be transmitted via a network. In some aspects, the alert includes an image collected from the vehicle, a location of the vehicle, a location of the steering controller, a satellite image of the vehicle, or a combination thereof.
[0012] In some aspects, the processor is further configured to receive instructions from a person after generating the alert. In some aspects, the instructions are provided remotely. In some aspects, the processor is further configured to continue to perform the first turn, perform the second turn, or travel along the furrow after the instructions have been provided. In some aspects, the system also includes an obstacle detection sensor configured to detect obstacles. In some aspects, the obstacle detection sensor includes a lidar sensor. In some aspects, the position sensor includes a global positioning system sensor. In some aspects, the position sensor includes a camera.
[0013] In some aspects, the vehicle is an agricultural vehicle. In some aspects, the vehicle is a tractor, a weeder, a sprayer or a seed drill.
[0014] In various aspects, the present disclosure provides a method for autonomously driving a vehicle in a field, the method comprising: determining a turning zone, the turning zone corresponding to a field exit path having a first field exit position and a first field re-entry path and having a first field re-entry position, wherein the first turning zone is bounded by an outer boundary and an inner boundary, wherein the outer boundary encloses the inner boundary, wherein the inner boundary extends through the first field exit position and the first field re-entry position, and wherein the inner boundary encloses a planting area of the field; driving the vehicle along the first field exit path toward the first field exit position at the inner boundary; when the inner boundary is encountered at the first field exit position, driving the vehicle toward a first forward position; when the outer boundary is encountered, reversing the vehicle toward a first correction position; when the first correction position is reached, driving the vehicle toward the first forward position; when the first forward position is reached, reversing the vehicle toward a first reverse position; when the first reverse position is reached or the outer boundary is encountered, driving the vehicle forward at the inner boundary toward the first field re-entry position.
[0015] In some aspects, the method further comprises driving the vehicle at a variable speed. In some aspects, the method further comprises: reversing the vehicle toward the first field re-entry position before reaching the first reverse position and upon encountering the outer boundary; and reversing the vehicle toward the first reverse position upon reaching the first field re-entry position oriented in a direction other than along the first field re-entry path.
[0016] In some aspects, the method further includes: determining a second turning zone, the second turning zone corresponding to a second field exit path having a second field exit position and a second field re-entry path and having a second field re-entry position, wherein the second turning zone is bounded by the outer boundary and the inner boundary, wherein the inner boundary extends through the second field exit position and the second field re-entry position; driving the vehicle along the second field exit path toward the second field exit position; when encountering the inner boundary at the second field exit position, driving the vehicle toward a second forward position; when reaching the second forward position, reversing the vehicle toward a second reverse position; when reaching the second reverse position or encountering the outer boundary, reversing the vehicle toward the second field re-entry position at the inner boundary.
[0017] In some aspects, the method further comprises travelling along a furrow between the first field re-entry location and the second field exit location. In some aspects, travelling along the furrow comprises identifying the furrow in a depth image of the field, a visual image of the field, or both, and determining a furrow position, a furrow direction, or both. In some aspects, identifying the furrow further comprises generating a segmented map from the visual image. In some aspects, determining the furrow direction further comprises determining a center of the furrow from the depth image. In some aspects, determining the furrow position and determining the furrow direction further comprises determining position values and direction values of pixels of the visual image corresponding to the furrow.
[0018] In some aspects, the method further comprises determining a representative position and a representative direction based on the position value and the direction value, wherein the representative position corresponds to the furrow position and the representative direction corresponds to the furrow direction. In some aspects, the method is determined in real time, adaptively determined, or both. In some aspects, the method is updated as the method is performed.
[0019] In some aspects, the method further comprises determining the first field exit position, the first field exit path, the first field re-entry position, the first field re-entry path, the first forward position, the first correction position, the first reverse position, the second field exit position, the second field exit path, the second field re-entry position, the second field re-entry path, the second forward position, the second reverse position, or a combination thereof based on sensor input. In some aspects, the method further comprises determining the first field exit position, the first field exit path, the first field re-entry position, the first field re-entry path, the first forward position, the first correction position, the first reverse position, the second field exit position, the second field exit path, the second field re-entry position, the second field re-entry path, the second forward position, the second reverse position, or a combination thereof in real time. In some aspects, the method further comprises adaptively determining the first field exit position, the first field exit path, the first field re-entry position, the first field re-entry path, the first forward position, the first correction position, the first reverse position, the second field exit position, the second field exit path, the second field re-entry position, the second field re-entry path, the second forward position, the second reverse position, or a combination thereof.
[0020] In some aspects, the method further comprises generating an alert to request human intervention. In some aspects, the method comprises generating the alert when the vehicle fails to reach the first field exit position, the first correction position, the first forward position, the first reverse position, the first field re-entry position, the second field exit position, the second correction position, the second forward position, the second reverse position, or the second field re-entry position after a predetermined number of attempts. In some aspects, the method comprises attempting to reach the first field exit position, the first correction position, the first forward position, the first reverse position, the first field re-entry position, the second field exit position, the second correction position, the second forward position, the second reverse position, or the second field re-entry position 2 to 10 times, 3 to 8 times, or 4 to 6 times before generating the alert. In some aspects, the method comprises attempting to reach the first field exit position, the first correction position, the first forward position, the first reverse position, the first field re-entry position, the second field exit position, the second correction position, the second forward position, the second reverse position, or the second field re-entry position about 5 times before generating the alert.
[0021] In some aspects, the method includes generating the alert when the vehicle crosses the outer boundary or exceeds the outer boundary by a predetermined distance. In some aspects, the method includes generating the alert when the vehicle exceeds the outer boundary by a distance of 0.5 to 1.5 vehicle lengths. In some aspects, the method includes generating the alert when the vehicle exceeds the outer boundary by a distance of about one vehicle length. In some aspects, the method includes generating the alert when the system fails to detect a furrow at the first field re-entry position or between the first field re-entry position and the second field exit position. In some aspects, the method includes generating the alert when the system detects an obstacle. In some aspects, the method includes transmitting the alert over a network.
[0022] In some aspects, the method further comprises transmitting, with the alert, an image collected from the vehicle, a position of the vehicle, a position of the steering controller, a satellite image of the vehicle, or a combination thereof. In some aspects, the method further comprises receiving an instruction from a person after generating the alert. In some aspects, the instruction is received over a network. In some aspects, the method comprises continuing to perform the first turn, perform the second turn, or follow the furrow after the instruction has been received.
[0023] In some aspects, the method further comprises detecting an obstacle using an obstacle detection sensor. In some aspects, the obstacle detection sensor comprises a lidar sensor.
[0024] In some aspects, the methods are performed by the systems described herein.
[0025] In various aspects, the present disclosure provides a computer-implemented method for detecting furrows in a field, the computer-implemented method comprising: receiving a depth image of the field and a visual image of the field; obtaining labeled image data, the labeled image data comprising furrows, furrow positions, and furrow directions in images of similar furrows; using the labeled image data to train a machine learning model to identify furrows, determine furrow positions, and determine furrow directions; identifying furrows by using the depth image and the visual image as inputs to the machine learning model; determining the furrow positions and furrow directions of the furrows by using the depth image and the visual image as inputs to the machine learning model.
[0026] In some aspects, identifying the furrow further comprises generating a segment map from the visual image. In some aspects, determining the furrow direction further comprises determining the center of the furrow from the depth image. In some aspects, determining the furrow position and the furrow direction further comprises determining a position value and a direction value of a pixel of the visual image corresponding to the furrow.
[0027] In some aspects, the method further includes determining a representative position and a representative direction based on the position value and the direction value, wherein the representative position corresponds to the furrow position and the representative direction corresponds to the furrow direction.
[0028] In various aspects, the present disclosure provides a method for navigating in a field, the method comprising detecting a first furrow using a method as described herein, guiding a vehicle along the first furrow, exiting the field along a field exit path, maneuvering the vehicle using a system as described herein or a method as described herein, re-entering the field along a field re-entry path at a second furrow detected using a method as described herein, or a combination thereof.
[0029] Incorporated by Reference
[0030] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The novel features of the invention are particularly set forth in the appended claims. A better understanding of the features and advantages of the invention will be obtained by reference to the following detailed description which sets forth an illustrative embodiment in which the principles of the invention are utilized, and in the accompanying drawings:
[0032] Figure 1 shows an isometric view of an autonomous laser weeding vehicle according to one or more embodiments herein;
[0033] Figure 2 shows a top view of an autonomous laser weeding vehicle navigating a crop field while implementing various techniques described herein;
[0034] Figure 3 shows a top view of a field in which various techniques described herein may be implemented;
[0035] Figure 4 schematically illustrates an exemplary end-of-row turn according to one or more embodiments herein;
[0036] Figure 5 schematically illustrates an exemplary end-of-row turn according to one or more embodiments herein;
[0037] Figure 6 schematically illustrates an exemplary end-of-row turn according to one or more embodiments herein;
[0038] Figure 7 schematically illustrates an exemplary end-of-row turn according to one or more embodiments herein;
[0039] Figure 8 schematically illustrates an exemplary end-of-row turn according to one or more embodiments herein;
[0040] Fig. 9 schematically illustrates a furrow detection method according to one or more embodiments herein;
[0041] Fig.10 shows a Hough map with detected furrows generated using a furrow detection method according to one or more embodiments herein;
[0042] Fig.11 A furrow segmentation mask containing gaps and incomplete furrows is shown;
[0043] Fig.12 Schematically illustrates a machine learning model architecture for furrow detection according to one or more embodiments herein;
[0044] Fig.13 is an exemplary block diagram of a computing device architecture of a computing device that can implement the various techniques described herein;
[0045] Fig.14 is a flow chart illustrating a turn-by-turn navigation method according to one or more embodiments of the present invention;
[0046] Fig.15is a flow chart illustrating a method of requesting human intervention according to one or more embodiments herein. DETAILED DESCRIPTION
[0047] Various exemplary embodiments of the present disclosure are discussed in detail below. Although specific implementations are discussed, it should be understood that this description is for illustrative purposes only. Those skilled in the relevant art will recognize that other components and configurations can be used without departing from the spirit and scope of the present disclosure. Therefore, the following description and drawings are illustrative and should not be construed as limiting. Many specific details are described to provide a thorough understanding of the present disclosure. However, in some cases, well-known or conventional details are not described in order to avoid making the description unclear. References to an embodiment or embodiments in the present disclosure may be references to the same embodiment or any embodiment; and such references are intended to refer to at least one exemplary embodiment.
[0048] Reference to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. The phrase "in one embodiment" appearing in various places in the specification does not necessarily all refer to the same embodiment, nor is it a separate or alternative exemplary embodiment that is mutually exclusive of other exemplary embodiments. In addition, various features are described that may be exhibited by some exemplary embodiments but not by others. Any feature of one example may be integrated or used together with any other feature of any other example.
[0049] In the context of the present disclosure and in the specific context in which each term is used, the terms used in this specification generally have their ordinary meaning in the art. Alternative language and synonyms may be used for any one or more terms discussed herein, and there is no special meaning as to whether the terms are elaborated or discussed herein. In some cases, synonyms for certain terms are provided. The description of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification (including examples of any terms discussed herein) is merely exemplary, and is not intended to further limit the scope and meaning of the present disclosure or any exemplary term. Likewise, the present disclosure is not limited to the various exemplary embodiments given in this specification.
[0050] Without intending to limit the scope of the present disclosure, examples of instruments, devices, methods and related results thereof according to exemplary embodiments of the present disclosure are given below. Note that, for the convenience of the reader, titles or subtitles may be used in the examples, which should never limit the scope of the present disclosure. Unless otherwise defined, the technical and scientific terms used herein have the meanings commonly understood by those of ordinary skill in the art to which the present disclosure belongs. In the event of a conflict, the present document including the definition shall prevail.
[0051] Additional features and advantages of the present disclosure will be set forth in the following description and will be partially apparent from the description, or may be learned by practicing the principles disclosed herein. The features and advantages of the present disclosure may be realized and obtained by the instruments and combinations specifically pointed out in the appended claims. These and other features of the present disclosure may become more apparent from the following description and the appended claims, or may be learned by practicing the principles set forth herein.
[0052] For clarity of explanation, in some cases, the present technology may be presented as including various functional blocks, which represent devices, device components, steps or routines in methods embodied in software or a combination of hardware and software.
[0053] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such specific arrangement and / or order may not be required. On the contrary, in some embodiments, such features may be arranged in a different manner and / or order than shown in the exemplary drawings. In addition, including structural or method features in a specific drawing does not mean that such features are required in all embodiments, and in some embodiments, it may not be included or may be combined with other features.
[0054] Although the concepts of the present disclosure may have various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and will be described in detail herein. However, it should be understood that the concepts of the present disclosure are not intended to be limited to the specific forms disclosed, but on the contrary, the present invention is intended to cover all modifications, equivalents and alternatives consistent with the present disclosure and the appended claims.
[0055] Systems and methods for detecting and locating furrows in fields (such as farmland) are described herein. The systems and methods disclosed herein can be used to determine multiple parameters of detected furrows, such as position and angle, while autonomously navigating along the furrows. For example, an autonomous agricultural vehicle (such as a tractor, weeder, seeder, harvester, or sprayer) can navigate along crop furrows using the systems and methods described herein. The furrow detection methods as described herein can be used to identify furrows within images captured by sensors on an autonomous vehicle, and determine the position and direction of the furrows so that the autonomous vehicle can navigate along the furrows. These furrow detection methods can be implemented in real time and used to adjust the heading of the autonomous vehicle to continue traveling along the furrows.
[0056] Also described herein is a system and method for autonomously turning a vehicle at the end of a furrow within a specified turning zone to begin navigation along a second furrow. The autonomous turning method described herein can enable turning of an autonomous vehicle within a narrow turning zone, such as an autonomous agricultural vehicle (e.g., a tractor, a weeder, a planter, a harvester, or a sprayer). In addition, the autonomous turning method can facilitate navigation of non-rectangular or irregularly shaped fields such as center-pivot farms, in which the furrows may not be positioned perpendicular to the field boundaries, or the relative position of the furrow ends is variable. The navigation point (also referred to herein as a "position") within the turning zone can be determined based on field features, such as an inner boundary (e.g., an inner geofence or a boundary spaced a predetermined distance from the inner geofence), an outer boundary (e.g., an outer geofence or a boundary spaced a predetermined distance from the outer geofence), the position of the first furrow (e.g., the furrow that the autonomous vehicle is leaving), the position of the second furrow (e.g., the furrow that the autonomous vehicle is entering), the length of the autonomous vehicle, the width of the autonomous vehicle, the turning radius of the autonomous vehicle, or a combination thereof. In some embodiments, the navigation point can be determined in real time, such as when the vehicle leaves or approaches the exit of the first furrow. Field features can be detected using sensors located on or associated with the vehicle (e.g., cameras, GPS sensors, rotary encoders, etc.). The navigation point can be adjusted during the execution of the turn, for example, to account for terrain irregularities or deviations from an expected path or position. In some embodiments, the path to the navigation point can be determined, the path to the navigation point can be adjusted, or both can be determined in response to the vehicle position, speed, heading or trajectory, deviations from an expected position, speed, heading or trajectory, or a combination thereof.
[0057] As used herein, an "image" may refer to a representation of an area or object. For example, an image may be a visual representation of an area or object formed by electromagnetic radiation (e.g., light, x-rays, microwaves, or radio waves) scattered from an area or object. In another example, an image may be a point cloud model formed by a light detection and ranging (lidar) or radio detection and ranging (radar) sensor. In another example, an image may be a sonic map generated by detecting sound waves, infrasound waves, or ultrasound waves reflected from an area or object. As used herein, "imaging" may be used to describe the process of acquiring or generating a representation (e.g., an image) of an area or object.
[0058] As used herein, a position (such as the position of an object or the position of a sensor) can be expressed relative to a reference system. Exemplary reference systems include a surface reference system, a vehicle reference system, a sensor reference system or an actuator reference system, a three-dimensional geographic coordinate reference system, a two-dimensional surface coordinate reference system. Positions can be easily converted between reference systems, for example, by using conversion factors or calibration models. Although positions, position changes, or deviations can be expressed in one reference system, it should be understood that positions, position changes, or deviations can be expressed in any reference system or can be easily converted between reference systems.
[0059] Autonomous weed control system
[0060] The detection and navigation methods described herein can be implemented by an autonomous vehicle (also referred to herein as an "autonomous vehicle") that navigates a field. For example, the detection and navigation methods can be implemented by an autonomous weeding system that navigates a crop field while autonomously targeting and removing weeds. Such detection and navigation methods can facilitate maneuvering the vehicle along a furrow to avoid running over and damaging the crop. For example, an autonomous weeding system can navigate along a crop furrow and, upon reaching the end of the furrow, turn and begin navigating along a second furrow.
[0061] The autonomous weed control system can identify, target and eliminate weeds without human input. Optionally, the autonomous weed control system can be located on an autonomous vehicle or a manned vehicle or can be towed by a vehicle such as a tractor. Figure 1 As shown, the autonomous weeding system can be part of or coupled to a vehicle 100, such as a tractor or an autonomous vehicle. The vehicle 100 can travel through a crop field 200, such as Figure 2 As shown. When the vehicle 100 travels through the field 200, the vehicle can identify, aim at and eliminate the weeds in the un-weeded portion 210 of the field, leaving behind the weeded field 220. In some embodiments, the speed of the vehicle can be adjusted based on one or more properties of the field (e.g., weed density, weed size, terrain texture, crop type, furrow arrangement, etc.). For example, the autonomous driving vehicle of the automatic weeding system can travel at a slower speed when the weed density is high, and travel at a faster speed when the weed density is low. Alternatively or in addition, the autonomous driving vehicle of the autonomous weeding system can travel at a slower speed when aiming and eliminating larger weeds, and travel at a faster speed when aiming and eliminating smaller weeds. The detection and navigation methods described herein can be implemented by an autonomous driving vehicle (such as an autonomous weeding system) or a vehicle connected to an autonomous weeding system to navigate along the furrows in the crop field. The high precision of such detection and navigation methods enables accurate field navigation such as using a laser, thereby facilitating accurate aiming of weeds to eliminate weeds without damaging nearby crops.
[0062] Furrow detection
[0063] Methods for detecting furrows (e.g., long, narrow ditches on the ground) are described herein. Furrows can be used to navigate in fields (such as crop fields) by positioning the wheels of a vehicle in the furrows to avoid damaging crops planted between the furrows. The furrow detection method disclosed herein can be used to detect furrows in real time in images collected by sensors connected to an autonomous vehicle to navigate the vehicle along the furrows without pre-mapping the positions of crop rows. The autonomous vehicle may include one or more forward navigation sensors located at the front of the vehicle. In some embodiments, the autonomous vehicle may also include one or more rear navigation sensors located at the rear of the vehicle. In some embodiments, the forward sensor or the reverse sensor may include a depth sensor (e.g., a depth camera or a laser radar sensor) configured to generate a depth image. In some embodiments, the forward sensor or the reverse sensor may be a two-dimensional sensor (e.g., a two-dimensional camera), and the vehicle may also include one or more depth sensors (e.g., a depth camera or a laser radar sensor) located at the front or rear of the vehicle. The ditch-like feature may be used to detect the furrow in the depth map by using the furrow shape to identify the furrow center.
[0064] In some embodiments, a combination of machine learning (e.g., deep learning) and post-processing can be used to identify and locate furrows in an image. Machine learning can be implemented to generate segmentation masks and directional masks. The segmentation mask can include pixel allocation data, wherein pixels in the segmentation mask are designated as "furrows" or "background". The directional mask can include pixel values corresponding to the directional information of the corresponding furrows. For example, the pixels in the directional mask can include cos(2θ) and sin(2θ) values, where θ represents the angle of the furrow relative to the reference coordinate system. In another example, the pixels in the directional mask can include cos(θ) and sin(θ) values. In some embodiments, cos(2θ) and sin(2θ) values can be converted to cos(θ) and sin(θ) values. In some embodiments, the θ value can be limited to a semicircle. For example, the θ value can be 0 to π (i.e., 0° to 180°). In another example, the θ value can be -π / 2 to π / 2 (i.e., -90° to 90°).
[0065] In some embodiments, the furrow parameters may be expressed in polar coordinates relative to a reference point (e.g., an origin) and a reference coordinate system. Fig. 9As shown, the furrow position and direction can be expressed by ρ and θ values, respectively. Furrow lines 401, 402, 403, 404, 405, and 406 can be used to represent the first furrow, the second furrow, the third furrow, the fourth furrow, the fifth furrow, and the sixth furrow, respectively. Lines 411, 412, 413, 414, 415, and 416 representing the ρ values of the first furrow, the second furrow, the third furrow, the fourth furrow, the fifth furrow, and the sixth furrow can be drawn from the origin 410 of the coordinate system to intersect the furrow lines 401, 402, 403, 404, 405, and 406 at right angles. The lengths of the lines 411, 412, 413, 414, 415, and 416 represent the ρ values of the corresponding furrows. Angles 421, 422, 423, 424, 425, and 426 representing the values of θ for the first furrow, the second furrow, the third furrow, the fourth furrow, the fifth furrow, and the sixth furrow, respectively, can be determined relative to a coordinate system. In some embodiments, θ can be from 0 to π. In some embodiments, ρ can be positive or negative, depending on the position of the furrow relative to the origin. Fig. 9 , the upper left corner of the image is used as the origin 410. However, the origin can be any point. For example, the origin 410 can be the upper right corner, the lower right corner, the lower left corner, the center, or any other point. In some embodiments, the p and θ values, the cos(2θ) and sin(2θ) values, or the cos(θ) and sin(θ) values can be determined using a machine learning method.
[0066] The Hough transform can be used to generate a Hough map based on the cos(θ) and sin(θ) values and the x and y coordinates of the furrow relative to a reference point (e.g., the origin). The cos(θ), sin(θ), x, and y values can be determined based on the directional map, the segmented map, or a combination thereof. In some embodiments, the Hough transform can include identifying the minimum and maximum p values and the minimum and maximum θ values. In some embodiments, cos(θ) and sin(θ) to p and θ are plotted on the Hough map. An example of a Hough map containing data from furrow lines 402, 403, and 404 is shown in FIG. Fig.10 . Rho (ρ) values 412, 413, and 414 can be plotted on a first axis, and theta (θ) values 422, 423, and 424 can be plotted on a second axis. Data from the Hoff diagram can be clustered, fuzzified, or otherwise grouped to identify and group components corresponding to the same furrow. For example, Gaussian blurring can be applied, or clustering techniques such as DBSCAN can be used. Representative values of the grouped components can be used to determine the ρ and θ values for the corresponding furrow. For example, the representative value can be a mean, median, centroid, maximum, or other representative value.
[0067] Furrow detection methods using segmented masks alone may result in incomplete furrow detection, such as gaps or truncation in furrows. Fig.11 As shown, the furrow segmentation mask may include gaps 436 or early terminations 433. In some embodiments, the furrow detection method may use furrow line clustering to generate representative furrows and generate furrow predictions along the length of the furrow. Representative furrows may be generated by predicting directional values of the furrows, applying a Hough transform, and identifying representative values (e.g., by blurring or clustering the Hough transform data). In some embodiments, furrow line clustering may be used in conjunction with the segmentation mask to improve furrow detection. Furrow predictions may be verified using the mask in conjunction with predictions of the sin(2θ) and cos(2θ) of each furrow pixel in the image. Fig.11 The segmentation mask (shaded area) of furrow lines 401, 402, 403, 404, 405 and 406 is shown overlaid with examples of unit vectors (arrows) generated by prediction of individual pixels. The value corresponding to the shaded area for each furrow pixel can be determined. Fig.11 It can be seen that the determined values of pixels within the same furrow may be inconsistent.
[0068] Parameters such as ρ and θ values can be assigned to each furrow in an image acquired by one sensor or a pair of sensors (e.g., a pair of forward sensors or a pair of rearward sensors). The furrow parameters can be used to navigate the autonomous vehicle along the furrows. For example, the autonomous vehicle can navigate along the central furrow or the furrow located closest to one of the sensors. In some embodiments, images can be acquired continuously to detect furrows and navigate in furrows in real time.
[0069] Machine learning model for furrow detection
[0070] The furrow detection method can be implemented by a furrow detection module, which is configured to identify and locate furrows in an image (e.g., an image collected by a sensor connected to an autonomous vehicle). In some embodiments, the furrow detection module can communicate with a vehicle navigation system to navigate the vehicle along the furrow. The furrow detection module can implement one or more machine learning algorithms or networks, which are implemented and dynamically trained to identify and locate furrows in one or more images (e.g., a first image, a second image, a depth image, etc.). The one or more machine learning algorithms or networks can include neural networks (e.g., convolutional neural networks (CNNs), deep neural networks (DNNs), etc.), geometric recognition algorithms, photometric recognition algorithms, principal component analysis using feature vectors, linear discriminant analysis, look-only-once (YOLO) algorithms, hidden Markov modeling, multilinear subspace learning using tensor representations, neuron-stimulated dynamic link matching, support vector machines (SVMs), or any other suitable machine learning techniques. If the furrow detection module implements one or more neural networks for furrow detection, these one or more neural networks can include one or more convolutional layers, visual transformer layers, or a combination thereof. The furrow detection module may include a system controller, such as a system computer having a memory, a random access memory (RAM), a central processing unit (CPU), and a graphics processing unit (GPU). The system computer may include a tensor processing unit (TPU). The system computer should include sufficient RAM, storage space, CPU power, and GPU power to perform operations for identifying and locating furrows.
[0071] The furrow detection machine learning model can be trained using a sample training dataset of images (e.g., images of a field with one or more furrows). The training images can be labeled with one or more furrow parameters, such as furrow bodies (e.g., pixels corresponding to furrows, pixels corresponding to background, groups of pixels corresponding to the same furrow, etc.), furrow positions (e.g., positions of furrows relative to a reference point), furrow directions (e.g., angles of furrows relative to a reference coordinate system), or combinations thereof. In some embodiments, one or more machine learning algorithms implemented by the furrow detection module can be trained end-to-end (e.g., by training multiple parameters in combination). Alternatively or in addition, subnetworks (e.g., direction networks) can be trained independently. For example, the subnetworks can be trained using supervised, unsupervised, reinforcement, or other such training techniques as described above. An example of a model architecture for a furrow detection module is provided in FIG. Fig.12 As provided in Fig.12 As shown, furrow detection may include two processes: a first process of generating a segmentation mask, and a second process of generating a direction mask.
[0072] In the first process, an image (e.g., an image collected by a sensor connected to an autonomous vehicle) can be received by a backbone network. In some embodiments, the image can be an RGB color image of a surface (e.g., a field). The network can be a CNN, including any number of nodes (e.g., neurons) organized in any number of layers, or the network can be constructed using a visual transformer or a visual transformer. In some embodiments, a convolutional neural network may include an input layer configured to receive an image, a recognition layer configured to identify furrows in an image, and an output layer configured to output data (e.g., feature maps, segmentation masks, number of furrows, furrow positions, etc.). In some embodiments, a convolutional network may include an input layer receiving an image, a series of hidden layers, and one or more output layers. The output layer may output the result of the network, such as furrow positions. Each layer of a convolutional neural network may be connected by any number of additional hidden layers. For example, a convolutional network may include an input layer receiving an image, a series of hidden layers, and one or more output layers. Each hidden layer may perform convolution on an image and output a feature map. The feature map may be passed from a hidden layer to the next convolutional layer. The output layer may output the result of the network, such as furrow positions. In some embodiments, the output may be a multi-resolution output.
[0073] The backbone network can receive images via an input layer. In some embodiments, the backbone can include a pre-trained network (e.g., ResNet50, MobileNet, CBNetV2, etc.) or a custom trained network. The backbone can include a series of convolutional layers, discards, activation functions, pooling, batch normalization, visual transformers, visual transformers, other deep learning mechanisms, or combinations thereof, which can be organized into residual blocks. The backbone network can process images as input to generate outputs that can be fed into the rest of the machine learning network. For example, the output can include one or more feature maps that include features of the input image via the output layer.
[0074] The output of the backbone network (e.g., feature map) can be received by one or more additional networks or layers, which are configured to identify one or more furrow parameters, such as the number of furrows, furrow bodies, furrow positions, or combinations thereof. In some embodiments, a network or layer can be configured to evaluate a single parameter. In some embodiments, a network or layer can be configured to evaluate two or more parameters. In some embodiments, a parameter can be evaluated by a single network or layer. In some examples, the output of the backbone network (e.g., feature map) can be received by a dilated spatial pyramid pooling (ASPP) layer. The ASPP layer can apply a series of dilated convolutions to the output of the backbone network. In some embodiments, the output of the dilated convolution can be pooled and provided to subsequent network layers of a point detection model. In some embodiments, the output can be a segmentation mask, a directional mask, or both.
[0075] Interpolation can be applied to the output of the additional network (e.g., ASPP layer) to upsample the output to match the height and width of the transition layer from the second process. In some embodiments, the interpolation can be bilinear interpolation or nearest neighbor interpolation. The upsampled output from the first process can be spliced with the output of the second process in a splicing layer. The splicing layer can stack the output tensors of the upsampled layers from the first process and the second process.
[0076] The spliced output can be received by a convolution and upsampling network. The convolution part of the convolution and upsampling network can include a series of one or more convolutional neural networks running in parallel. In some embodiments, each convolutional neural network can include a series of convolutional layers, discards, activation functions, pooling, batch normalization, visual transformers, visual transformers, other deep learning mechanisms or combinations thereof, which can be organized into residual blocks. The output of the convolutional network can include slices of each ontology category (furrow, background, etc.), and can be upsampled using interpolation (e.g., bilinear interpolation) to match the height and width of the input image. Activation functions (e.g., sigmoid activation function, softmax activation function, step activation function, linear activation function, hyperbolic tangent activation function, rectified linear unit (ReLU) activation function, swish activation function, etc.) can be applied to each pixel of this output upsampled image.
[0077] In the second process, a depth image (e.g., an image containing three-dimensional information) can be received by a convolutional network. In some embodiments, a depth image can be generated by two images of a surface (e.g., a field) collected from different angles. The network can be a backbone network. The network can be a CNN, including any number of nodes (e.g., neurons) organized in any number of layers, or the network can be constructed using a visual transformer or a visual transformer. In some embodiments, a convolutional neural network may include an input layer configured to receive an image, a recognition layer configured to identify furrows in an image, and an output layer configured to output data (e.g., feature maps, segmentation masks, furrow numbers, furrow positions, etc.). Each layer of the convolutional neural network can be connected by any number of additional hidden layers. For example, a convolutional network may include an input layer receiving an image, a series of hidden layers, and one or more output layers. Each hidden layer can perform convolution on an image and output a feature map. The feature map can be passed from a hidden layer to the next convolutional layer. The output layer can output the result of the network, such as a directional mask. In some embodiments, the output can be a multi-resolution output.
[0078] The convolutional network can receive an image via an input layer. In some embodiments, the convolutional network may include a pre-trained network (e.g., ResNet50, MobileNet, CBNetV2, etc.) or a custom trained network. The convolutional network may include a series of convolutional layers, discards, activation functions, pooling, batch normalization, visual transformers, visual transformers, other deep learning mechanisms, or combinations thereof, which may be organized into residual blocks. The convolutional network may process a depth image as input to generate an output that may be fed into the rest of the machine learning network. For example, the output may include one or more feature maps that include features of the input image via the output layer.
[0079] Interpolation can be applied to the output of the backbone network (e.g., feature map) to upsample the output to match the height and width of the transition layer from the first process. In some embodiments, the interpolation can be bilinear interpolation. The upsampled output from the second process can be spliced with the output of the first process in a concatenation layer. The concatenation layer can stack the output tensors of the upsampled layers from the first process and the second process.
[0080] The spliced output can be received by a convolution and upsampling network. The convolution part of the convolution and upsampling network can include a series of one or more convolutional neural networks running in parallel. In some embodiments, each convolutional neural network can include a series of convolutional layers, discards, activation functions, pooling, batch normalization, visual transformers, visual transformers, other deep learning mechanisms or combinations thereof, which can be organized into residual blocks. The output of the convolutional network can include two slices of each ontology category (furrow, background, etc.). The first slice of each category can predict a first value corresponding to the position and direction of the furrow, and the second slice of each category can predict a second value corresponding to the position and direction of the furrow. In some embodiments, the first value and the second value can be related to the position and direction of the furrow in a polar coordinate system. For example, the first value can be a cos(2θ) value, and the second value can be a sin(2θ) value, where θ corresponds to the angle of the furrow relative to the reference coordinate frame. The slice can be upsampled using interpolation (e.g., bilinear interpolation) to match the height and width of the input image. An activation function (e.g., a sigmoid activation function, a softmax activation function, a step activation function, a linear activation function, a hyperbolic tangent activation function, a rectified linear unit (ReLU) activation function, a swish activation function, etc.) can be applied to each pixel of this output upsampled image. The output of the function can have a range of -1 to 1, including -1 and 1. In some embodiments, the pixel values can be normalized across the first slice and the second slice of each class so that the cos(2θ) of each pixel is 2 and sin(2θ) 2 The sum of is 1. In some embodiments, the furrow position and direction can be expressed as Cartesian coordinates (e.g., x, y, and / or z coordinates) relative to a reference point in the image (e.g., an edge of the image, the center of the image, or a grid line within the image). In some embodiments, the point position can be expressed as polar, spherical, or cylindrical coordinates (e.g., θ, ρ, and / or θ) relative to a reference point in the image (e.g., an edge of the image, the center of the image, or a polar grid line within the image). (spherical) or z (cylindrical) coordinates).
[0081] The machine learning process described herein can be used to detect furrows in images collected by sensors coupled to an autonomous vehicle and determine the position and orientation of the furrows relative to the vehicle. The autonomous vehicle can use the output of the machine learning process to navigate along the furrows.
[0082] End of line turn
[0083] Navigation systems and methods for turning a vehicle within a specified turning zone are described herein. The navigation systems and methods disclosed herein can be used to perform an end-of-row turn of a vehicle exiting a field along a first path (e.g., a first furrow) to position the vehicle to re-enter the field along a second path (e.g., a second furrow). These navigation systems and methods can be useful for driving an autonomous agricultural vehicle (e.g., a tractor, weeder, sprayer, or seeding machine) up and down between crop rows, and can be used to turn an autonomous vehicle within a restricted turning zone.
[0084] Figure 3 An example of a field 200 is shown, and the systems and methods of the present disclosure can be used to navigate the autonomous vehicle 100. The inner boundary 202 can be specified around the field (e.g., a crop field). In some embodiments, the inner boundary 202 can be specified around a certain area in the field (such as a planted area or an area to be planted). The outer boundary 201 can represent an available navigation area. The outer boundary can include the inner boundary 202 and a turning area 203 around the inner boundary 202. The outer boundary may be limited by terrain features such as cliffs, trees, hedges, roads, or walls, which may damage the vehicle if the vehicle crosses the outer boundary 201. In some embodiments, the inner boundary, the outer boundary, or both can be a geo-fence including GPS coordinates representing the boundary. In some embodiments, the inner boundary, the outer boundary, or both can be a boundary at a predetermined distance from the geo-fence including the GPS coordinates. For example, the outer boundary can be separated from the outer geo-fence by about one vehicle length or separated from the outer geo-fence by at least one vehicle length to prevent exceeding the outer geo-fence. Due to irregular field shapes (such as center pivot farms), the inner boundary, the outer boundary, or both may include curved shapes.
[0085] Figure 4 Shows features that can be used to autonomously perform end-of-row turns. Vehicles can leave the path along the field Leaving the field 200, entering a turning zone 203 bounded by an outer boundary 201 and an inner boundary 202. Field leaving path The first furrow 204 may be followed, or the field may leave the path The vehicle may be parallel to the first furrow 204. The vehicle may be turned so that it re-enters the path along the field. Re-enter the field 200. Field re-entry path The second furrow 205 may be followed, or the field may leave the path It may be parallel to the second furrow 205. The navigation target position may be based on the field leaving the path Field re-entry path The forward target position F can be determined based on the vehicle width W, the vehicle turning radius TR and the field exit path. Re-entry path to the field For example, the forward position F can be positioned at a position corresponding to the field reentry path The field re-entry position EN is at least one vehicle width W away from the field re-entry position EN in the opposite direction and at least one vehicle turning radius TR away from the field re-entry position EN in a direction parallel to the offset trajectory O. Forward heading The forward position F may be parallel to the offset trajectory O. If there is not enough space to reach the forward position F without hitting the outer boundary 201, the correction position C may be used. The correction position C may be specified based on the vehicle length L. For example, the correction position C may be located at least one vehicle length L away from the field exit position EX in the direction opposite to the offset trajectory O. The reverse target position R may be specified based on the vehicle length L. For example, the reverse target position R may be located at least one vehicle length L away from the field re-entry path. In the opposite direction, the field re-entry position EN is at least one vehicle length L away. The reverse position R may be within the turning area 203 , or the reverse position R may be outside the outer boundary 201 .
[0086] Examples of turns are described herein. Although specific examples are provided, multiple variations can be performed. For example, a turn can be made to the left or right. Multiple turns can be performed continuously to navigate in a field. When a vehicle is navigating in a field, turns and corresponding reference points can be determined based on furrow and boundary positions. For example, a reference point can be determined when the vehicle is leaving a furrow. In some embodiments, a reference point can be adjusted when a turn is being performed. In some embodiments, a reference point can be determined using field features detected by a sensor (e.g., a camera, a GPS sensor, a rotary encoder, etc.) located on or associated with the vehicle. A reference point can be determined based on one or more vehicle parameters (e.g., width, length, turning radius, etc.), and additional margins can be included to allow for errors. For example, a parameter can include additional margins to account for oversteering or understeering.
[0087] A turning path can be determined to reach a target position, a target heading, or both. Determining a turning path can include calculating a path (also referred to as a "trajectory") to a target position or heading and determining the direction of the wheels to follow the path. In some embodiments, the turning path can be determined using a Bezier curve or other curve model. In some embodiments, the curve can be determined when a turn is performed. In some embodiments, the curve can be updated as the turn is performed, and the wheel angle can be adjusted. In some embodiments, if the vehicle deviates from the target position or heading, a turning path can be determined to adjust the position of the vehicle. The wheel direction can be determined based on an initial position (e.g., the current position of the vehicle), a final position (e.g., a target position), and an angle between the initial heading (e.g., the current heading of the vehicle) and the final heading (e.g., the target heading at the target position). For example, the wheel position can be calculated as follows:
[0088] P3=0.5625(P1)+0.375(P2)+0.0625(P2-(V2·D·0.5·sinΘ))
[0089] Where P3 is the point the wheel is pointing to, P1 is the initial position, P2 is the target position, V1 is the initial heading, V2 is the final heading, θ is the angle between the initial heading and the final heading, and D is the distance between the initial position and the final position.
[0090] Figure 5 The turn is shown being performed without the correction position C. The vehicle is traveling along or parallel to the first furrow 204 and is leaving the path along the field. Exit the field 200. When reaching the field exit position EX located on the inner boundary 202, the vehicle turns along path (i) toward the forward position F. The vehicle reaches the forward heading The vehicle then reverses along path (ii) toward the reverse position R. When the reverse position R is reached, the vehicle advances along path (iii) toward the field re-entry position EN and continues along the field re-entry path. The pointed heading reaches the field re-entry position EN located on the inner boundary 202. The vehicle then continues to travel along or parallel to the second furrow 205.
[0091] Figure 6 A turn is shown being performed using the correction position C. The vehicle is traveling along or parallel to the first furrow 204 and is leaving the path along the field. The vehicle leaves the field 200. When reaching the field exit position EX on the inner boundary 202, the vehicle turns along the path (i) toward the forward position F. The vehicle hits the outer boundary 201 before reaching the forward position F. When hitting the outer boundary 201, the vehicle reverses along the path (ii) toward the correction position C. At the correction position C, the vehicle advances along the path (iii) toward the forward position F. The vehicle reaches the forward heading. The vehicle then reverses along path (iv) toward the reverse position R. When the reverse position R is reached, the vehicle moves forward along path (v) to the field re-entry position EN, and re-enters the field along the field re-entry path. The heading is directed to reach the field re-entry position EN located on the inner boundary 202. The vehicle then continues to travel along or parallel to the second furrow 205.
[0092] Figure 7 The turn is shown when the reverse position R is outside the outer boundary 201. The vehicle is traveling along or parallel to the first furrow 204 and leaves the path along the field. The vehicle leaves the field 200. When reaching the field exit position EX on the inner boundary 202, the vehicle turns along the path (i) toward the forward position F. The vehicle hits the outer boundary 201 before reaching the forward position F. When hitting the outer boundary 201, the vehicle reverses along the path (ii) toward the correction position C. At the correction position C, the vehicle advances along the path (iii) toward the forward position F. The vehicle reaches the forward heading. The vehicle then reverses along the path (iv) toward the reverse position R. The vehicle encounters the outer boundary 201 before reaching the reverse position R. When encountering the outer boundary 201, the vehicle advances along the path (v) to the field re-entry position EN, and does not collide with the field re-entry path. The aligned heading reaches the field re-entry position EN located on the inner boundary 202. The vehicle reverses along the path (vi) toward the reverse position R, and encounters the outer boundary 201 again before reaching the reverse position R. When reaching the outer boundary 201, the vehicle drives forward along the path (vii) to the field re-entry position EN, and continues along the field re-entry path. The heading is directed to reach the field re-entry position EN located on the inner boundary 202. The reverse to position R may be repeated as needed (e.g., 1, 2, 3, 4, 5, or more times) to move along the field re-entry path. The pointed heading reaches the field re-entry position EN located on the inner boundary 202. The vehicle then continues to travel along or parallel to the second furrow 205.
[0093] Figure 8The turn is shown being performed in a turn zone 203 at the region of the inner boundary 202 which is at a smaller angle relative to the furrows 204 and 205. The vehicle is travelling along or parallel to the first furrow 204 and is leaving the path along the field. The vehicle leaves the field 200. When reaching the field exit position EX on the inner boundary 202, the vehicle turns along the path (i) toward the forward position F. The vehicle hits the outer boundary 201 before reaching the forward position F. When hitting the outer boundary 201, the vehicle reverses along the path (ii) toward the correction position C. At the correction position C, the vehicle advances along the path (iii) toward the forward position F. The vehicle reaches the forward heading. The vehicle then reverses along path (iv) toward the reverse position R. When the reverse position R is reached, the vehicle moves forward along path (v) to the field re-entry position EN, and re-enters the field along the field re-entry path. The pointed heading reaches the field re-entry position EN located on the inner boundary 202. The vehicle then continues to travel along or parallel to the second furrow 205. In this example, the vehicle is able to perform a turn in the narrow turning 203 area using the smaller angle of the inner boundary 202 than if the inner boundary was perpendicular to the furrows 204 and 205.
[0094] In some embodiments, the vehicle may miss or overtake the target location, or the vehicle may arrive at the target location on the wrong heading. In this case, the vehicle can reverse (e.g., straight or toward a different target location), adjust the route, and try to reach the target location again. This process can be repeated (e.g., 1, 2, 3, 4, 5 or more times) until the vehicle reaches the target location.
[0095] In some embodiments, reference points (e.g., forward position, correction position, reverse position, etc.), boundaries (e.g., inner boundary, outer boundary, etc.) can be determined based on GPS coordinates. One or more GPS antennas (e.g., 2 antennas) can be located on the autonomous vehicle to accurately determine the location of the vehicle. In some embodiments, a base station can be located near a field (e.g., within about 100 miles) to improve GPS accuracy. Position coordinates can be converted between a geographic coordinate reference system and a field reference plane. In some embodiments, the field reference plane can be expressed in two dimensions.
[0096] An example of method 1400 for autonomously performing a turn is shown in Fig.14 The vehicle may leave the field at step 1410 (e.g., Figure 4 The position EX shown in FIG. 14A leaves the field and at step 1420 moves toward a forward position (eg, Figure 4If the vehicle cannot reach the forward position 1423, the vehicle can move toward the correction position (e.g., Figure 4 The vehicle may then resume driving toward the forward position. At step 1430 (e.g., once the vehicle has reached the forward position), the vehicle may move toward the reverse position (e.g., Figure 4 At step 1440 (e.g., once the vehicle has reached the reverse position or has failed to reach the reverse position), the vehicle may travel toward the field re-entry position. If the vehicle cannot reach the field re-entry position 1445, the vehicle may reverse toward the reverse position. At step 1450 (e.g., once the vehicle has reached the field re-entry position), the vehicle may re-enter the field at the field re-entry position. In some embodiments, the vehicle may begin to travel along the furrows when re-entering the field.
[0097] Human intervention
[0098] In some embodiments, the autonomous vehicle system may encounter a situation where it cannot continue to drive autonomously. In the case of a fault condition, support can be requested from humans. Humans can intervene to correct the route of the vehicle, or humans can determine that intervention is not required. When the vehicle has tried to reach the target location many times but has not arrived, human intervention can be requested. In this case, humans can evaluate the vehicle or surrounding conditions and provide instructions to reach the target location or the vehicle can autonomously reach the location where the target location is located. Alternatively or in addition, if the vehicle exceeds the preset boundary or crosses or is a certain distance outside the current boundary, human intervention can be requested. Then, humans can evaluate the vehicle or surrounding conditions and provide instructions to re-enter the boundary or reach the target location. Alternatively or in addition, if the vehicle cannot detect a furrow at the field re-entry position or travels along the furrow between the field re-entry position and the field exit position, human intervention can be requested. Humans can evaluate the vehicle or surrounding conditions and manually identify the furrow or provide instructions to reach the furrow. Alternatively or in addition, if an obstacle is detected near the vehicle or in the path, human intervention can be requested. The human may assess the area around the vehicle and provide instructions to avoid obstacles, or the human may determine that no obstacles exist.
[0099] In some embodiments, if the vehicle cannot reach the target position or heading within a predetermined number of attempts, human intervention can be requested. When the vehicle does not reach the target position (e.g., field exit position, correction position, forward position, reverse position, or field re-entry position) or target heading (e.g., field exit path, field re-entry path, or forward heading) within a predetermined number of attempts, an alarm can be generated by the autonomous vehicle system to request human intervention. For example, an alarm can be generated after 2 to 10 attempts, 3 to 8 attempts, 4 to 6 attempts, or about 5 attempts. When the alarm is generated, the system can pause the movement and wait for instructions from humans. Before generating the alarm, humans can receive information about the vehicle from the system, such as vehicle position (e.g., GPS position), steering position, wheel angle, distance from the boundary (e.g., distance from the outer boundary) or heading. The information received by humans can include sensor images collected by the system, satellite images of the vehicle, or proximity sensor data. Humans can use the information to determine whether human intervention is needed. In some embodiments, humans can determine that intervention is not needed and can instruct the system to resume autonomous operation. In some embodiments, a human can intervene by providing instructions to reach the target location or the vehicle can autonomously reach the location where the target location is located. For example, a human can provide the system with wheel angles, speed, distance, heading, or a combination thereof. The system can execute the instructions to reach the target location or the vehicle can autonomously reach the location where the target location is located. After executing the instructions, the system can resume autonomous operation.
[0100] In some embodiments, if the autonomous vehicle system cannot detect a furrow at the furrow of the field re-entry position or travel along the furrow between the field re-entry position and the field exit position, human intervention can be requested. When the system cannot detect a furrow when re-entering the field or during traveling along the furrow, an alarm can be generated by the system to request human intervention. When the alarm is generated, the system can pause the movement and wait for instructions from humans. Before generating the alarm, humans can receive information about the vehicle from the system, such as vehicle position (e.g., GPS position), steering position, wheel angle, distance from the boundary (e.g., distance from the outer boundary) or heading. The information received by humans can include sensor images collected by the system, satellite images of the vehicle, or proximity sensor data. Humans can use the information to determine whether human intervention is needed. In some embodiments, humans can determine that intervention is not needed and can instruct the system to resume autonomous operation. In some embodiments, humans can intervene by providing instructions to reach the furrow. For example, humans can provide wheel angles, speeds, distances, headings, or combinations thereof to the system. The system can execute instructions to reach the furrow. After executing the instructions, the system can resume autonomous operation.
[0101] In some embodiments, if the vehicle exceeds the boundary or crosses or is at a certain distance from the boundary, human intervention can be requested. The boundary can be an outer boundary or an outer geo-fence around the turning zone. When the vehicle exceeds the boundary or crosses or is at a certain distance from the boundary, an alarm can be generated by the autonomous driving vehicle system to request human intervention. For example, when the vehicle exceeds the boundary by 0.5 to 1.5 vehicle lengths or a distance of about one vehicle length, an alarm can be sent. When the alarm is generated, the system can pause the movement and wait for instructions from humans. Before generating the alarm, humans can receive information about the vehicle from the system, such as vehicle position (e.g., GPS position), steering position, wheel angle, distance from the boundary (e.g., distance from the outer boundary) or heading. The information received by humans can include sensor images collected by the system, satellite images of the vehicle, or proximity sensor data. Humans can use the information to determine whether human intervention is needed. In some embodiments, humans can determine that intervention is not needed and can instruct the system to resume autonomous operation. In some embodiments, humans can intervene by providing instructions to re-enter the turning zone. For example, humans can provide wheel angles, speeds, distances, headings, or combinations thereof to the system. The system can execute instructions to re-enter the turning zone. After executing the command, the system can resume autonomous operation.
[0102] In some embodiments, if the autonomous vehicle system detects an obstacle, human intervention may be requested. When an obstacle is detected, an alarm may be generated by the system to request human intervention. In some embodiments, the object may be detected by a sensor located on the vehicle (e.g., a lidar sensor, a radar sensor, or a camera). When an alarm is generated, the system may pause motion and wait for instructions from humans. Before generating an alarm, humans may receive information about the vehicle from the system, such as vehicle position (e.g., GPS position), steering position, wheel angle, distance from a boundary (e.g., distance from an outer boundary), or heading. The information received by humans may include sensor images collected by the system, satellite images of the vehicle, or proximity sensor data. Humans may use the information to determine whether human intervention is required. In some embodiments, humans may determine that intervention is not required and may instruct the system to resume autonomous operation. For example, humans may determine that there is no obstacle. In some embodiments, humans may intervene by instructing the vehicle to wait for the obstacle to move away. In some embodiments, humans may intervene by providing instructions to bypass the obstacle. For example, humans may provide the system with wheel angles, speeds, distances, headings, or a combination thereof. The system may execute instructions to bypass the obstacle or wait for the obstacle to move. After executing the command, the system can resume autonomous operation.
[0103] In some embodiments, human intervention can be provided remotely. The autonomous vehicle system can send alerts, vehicle information, sensor data, image data, or a combination thereof over a network. In some embodiments, the network can be a wi-fi network, a satellite network, a local area network, or a mobile data network. In some embodiments, the alert can be sent as a text, an email alert, or a mobile phone application notification. A human can view vehicle information, sensor data, image data, or a combination thereof at a remote location. A human can send instructions to the autonomous vehicle system over a network, and the system can execute the instructions. In some embodiments, a human can remotely control the vehicle over a network.
[0104] In some embodiments, human intervention can be provided on-site (e.g., at the vehicle). For example, a human can be notified by a signal from the autonomous vehicle system (e.g., sound or light) or a notification sent over a network (e.g., as a text, email alert, or mobile phone application notification). Once the alert is issued, the human can control or move the vehicle to correct the situation. For example, the human can manually control the vehicle to reposition the vehicle to a furrow re-entry position. In another example, the human can manually drive the vehicle along a portion of the furrow that a furrow detection system on the vehicle failed to detect until reaching a portion of the furrow that the furrow detection system can detect.
[0105] Example method 1500 for requesting human intervention Fig.15Provided in. At step 1510, a failure condition (e.g., an obstacle is detected, a furrow is not detected, a target location is not reached, or a boundary is exceeded) is met. When the failure condition is met, an alarm is generated at step 1520, and at step 1530, the vehicle suspends movement. In some embodiments, the alarm can be sent via a network (e.g., as a text, email alert, or mobile phone application notification). In some embodiments, the system can generate an audio or visual alarm from the vehicle. For example, the vehicle can be equipped with an indicator light indicating whether the vehicle is operating in manual mode or automatic mode. At step 1540, the autonomous vehicle system can send information to the human, such as vehicle position, steering position, wheel angle, distance from the boundary (e.g., distance from the outer boundary), heading before suspending movement, sensor images collected by the system, satellite images of the vehicle, or proximity sensor data. The information can be sent via a network (e.g., a wi-fi network, a satellite network, a local area network, or a mobile data network). The system can wait for instructions from a human. At step 1550, the system can receive instructions from a human. In some embodiments, the instructions can include wheel angles, speed, distance, heading, or a combination thereof. In some embodiments, the instruction may be that intervention is not required. At step 1560, the system may execute the instruction. For example, the system may operate the vehicle in accordance with human instructions. Upon executing the instruction or upon receiving an instruction that does not require intervention, at step 1570, the system may resume autonomous function.
[0106] Computer system and method
[0107] The detection and navigation methods described herein can be implemented using a computer system. In some embodiments, the detection and navigation systems described herein include computer systems. In some embodiments, the computer system can autonomously implement the detection and navigation methods without human input. In some embodiments, the computer system can implement the detection and navigation methods based on instructions provided by a human user through a detection terminal.
[0108] The actual implementation of the device shown will include more components known to those of ordinary skill in the art. For example, each of the devices shown will have a power supply, one or more processors, a computer-readable medium for storing computer-executable instructions, etc. For clarity, these additional components are not shown herein.
[0109] In some examples, the procedures described herein (e.g., Fig.12 , or other programs described herein) can be performed by, for example, Fig.13The processes described herein may be performed by a computing device or apparatus having computing device architecture 1600 as shown. In one example, the processes described herein may be performed by a computing device having computing device architecture 1600. The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device, a server (e.g., a software as a service (SaaS) system or other server-based system), and / or a computer having the ability to perform the processes described herein (including Fig.12 In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other components configured to perform the steps of the processes described herein. In some examples, a computing device may include a display (as an example of an output device or in addition to an output device), a network interface configured to transmit and / or receive data, any combination thereof, and / or other components. The network interface may be configured to transmit and / or receive data based on the Internet Protocol (IP) or other types of data.
[0110] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.
[0111] Program in Fig.12 As shown in the figure, the operations of the program represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media, which perform the described operations when executed by one or more processors. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.
[0112] In addition, the processes described herein may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed jointly on one or more processors, executed by hardware, or a combination thereof. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.
[0113] Fig.13 An example computing device architecture 1600 is shown of an example computing device that can implement the various techniques described herein. For example, the computing device architecture 1600 can implement Fig.12 The program shown, or control Figure 1 and Figure 2 The components of the computing device architecture 1600 are shown as being in electrical communication with each other using a connection 1605, such as a bus. The example computing device architecture 1600 includes a processing unit (which may include a CPU and / or a GPU) 1610 and a computing device connection 1605 that couples various computing device components including computing device memory 1615, such as a read-only memory (ROM) 1620 and a random access memory (RAM) 1625, to the processor 1610. In some embodiments, the computing device may include a hardware accelerator.
[0114] The computing device architecture 1600 may include a cache of high-speed memory directly connected to, in close proximity to, or integrated as part of the processor 1610. The computing device architecture 1600 may copy data from the memory 1615 and / or the storage device 1630 to the cache 1612 for fast access by the processor 1610. In this way, the cache can provide a performance boost, which avoids the processor 1610 from delaying while waiting for data. These and other modules can control or be configured to control the processor 1610 to perform various actions. Other computing device memories 1615 may also be used. The memory 1615 may include a variety of different types of memory with different performance characteristics. The processor 1610 may include any general-purpose processor and hardware or software services (such as service 1 1632, service 2 1634, and service 3 1636 stored in the storage device 1630) configured to control the processor 1610, as well as a dedicated processor in which software instructions are incorporated into the processor design. The processor 1610 may be an independent system that includes multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric.
[0115] To enable a user to interact with the computing device architecture 1610, the input device 1645 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, and the like. The output device 1635 can also be one or more of a variety of output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, and the like. In some cases, a multimodal computing device can enable a user to provide multiple types of input to communicate with the computing device architecture 1600. The communication interface 1640 can generally govern and manage user input and computing device output. There is no restriction on operating on any particular hardware arrangement, and therefore, the basic features herein can be easily replaced with improved hardware or firmware arrangements as they are developed.
[0116] The storage device 1630 is a non-volatile memory and can be a hard disk or other type of computer-readable medium that can store computer-accessible data, such as a magnetic tape, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette tape, a random access memory (RAM) 1625, a read-only memory (ROM) 1620, and a mixture thereof. The storage device 1630 may include services 1632, 1634, 1636 for controlling the processor 1610. Other hardware or software modules are contemplated. The storage device 1630 may be connected to the computing device connection 1605. In one aspect, a hardware module that performs a particular function may include a combination of software components stored in a computer-readable medium and necessary hardware components (such as the processor 1610, the connection 1605, the output device 1635, etc.) to perform the function.
[0117] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and do not include carrier waves and / or temporary electronic signals that are propagated wirelessly or through a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media such as compact discs (CDs) or digital versatile disks (DVDs), flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent any combination of programs, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, or the like.
[0118] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams and the like. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.
[0119] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be appreciated by those skilled in the art that these embodiments can be put into practice without these specific details. For clarity of explanation, in some cases, the present technology can be presented as including various functional blocks, which include functional blocks comprising devices, device components, steps in the method embodied in software or a combination of hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein can be used. For example, circuits, systems, networks, processes and other components can be shown as components in block diagram form, so as not to confuse the embodiments in unnecessary details. In other cases, well-known circuits, processes, algorithms, structures and techniques can be shown without unnecessary details so as not to confuse the embodiments.
[0120] Each embodiment can be described as a process or method above, and the process or method is depicted as a flow chart, a flow diagram, a data flow diagram, a structure diagram or a block diagram. Although the flow chart can describe the operation as a sequential process, many operations can be performed in parallel or simultaneously. In addition, the order of operations can be rearranged. The process terminates when its operation is completed, but may have additional steps not included in the figure. The process can correspond to a method, a function, a program, a subroutine, a subprogram, etc. When the process corresponds to a function, its termination can correspond to the function returning to the calling function or the main function.
[0121] The processes and methods according to the above examples can be implemented using computer executable instructions stored in or available from computer readable media. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a specific function or group of functions. Portions of the computer resources used may be accessed over a network. Computer executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language), firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, and the like.
[0122] The apparatus for implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may adopt any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments for performing necessary tasks (e.g., computer program products) may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc., of laptop computers, smart phones, mobile phones, tablet devices, or other small form factors. The functions described herein may also be embodied in peripheral devices or expansion cards. By further example, such functionality may also be implemented on circuit boards between different chips or different processes executed in a single device.
[0123] The instructions, the medium for communicating such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.
[0124] Various illustrative logic blocks, modules, circuits and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above in terms of their functions. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the entire system. The technician can implement the described functionality in different ways for each specific application, but such implementation decision should not be interpreted as causing a departure from the scope of the application.
[0125] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handheld terminals, or integrated circuit devices with multiple uses, including applications in wireless communication device handheld terminals and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or implemented separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions for executing one or more of the above methods when executed. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the techniques may be implemented at least in part by a computer-readable communication medium (such as a propagated signal or wave) that carries or communicates code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.
[0126] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in an alternative example, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.
[0127] While illustrative embodiments have been shown and described, it will be appreciated that various changes can be made therein without departing from the spirit and scope of the disclosure.
[0128] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative embodiments of the present application have been described in detail herein, it should be understood that the inventive concept can be implemented and adopted differently in other ways, and the appended claims are intended to be interpreted as including such variations except for the variations limited by the prior art. The various features and aspects of the above-mentioned application can be used alone or in combination. In addition, without departing from the broader spirit and scope of this specification, the embodiment can be used in any number of environments and applications beyond those described herein. Therefore, the description and the accompanying drawings are considered to be illustrative rather than restrictive. For illustrative purposes, the method is described in a specific order. It should be understood that in an alternative embodiment, the method can be performed in an order different from the described order.
[0129] One of ordinary skill in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.
[0130] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor, or other suitable electronic circuits) to perform the operations, or any combination thereof.
[0131] The phrase "coupled to" refers to any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).
[0132] Claim language or other language reciting "at least one" of a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one" of a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.
[0133] As used herein, the terms "about" and "approximately" when referring to a number are used herein to include numbers that fall within 10%, 5% or 1% in either direction (greater or less than the number) of the number unless otherwise stated or obvious from the context (unless such number would exceed 100% of the possible value).
[0134] Although preferred embodiments of the present invention have been shown and described herein, it will be appreciated by those skilled in the art that such embodiments are provided only by way of example. Without departing from the present invention, it will now be appreciated by those skilled in the art that many variations, changes and substitutions are present. It should be understood that various substitutions of the embodiments of the present invention described herein may be used to practice the present invention. The accompanying claims are intended to define the scope of the present invention, and thus encompass methods and structures within the scope of these claims and their equivalents.
Claims
1. A system for autonomously driving a vehicle, the system comprising: a steering controller for driving the vehicle; a processor coupled to the steering controller, wherein the processor is configured to determine a first steering zone corresponding to a first field exit path having a first field exit location and a first field re-entry path and having a first field re-entry location, wherein the first steering zone is bounded by the outer boundary and the inner boundary, wherein the outer boundary encloses the inner boundary, wherein the inner boundary extends through the first field exit location and the first field re-entry location, and wherein the inner boundary encloses a planting area field; a position sensor coupled to the processor, the position sensor for identifying a position of the vehicle; as well as A memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the steering controller to perform a first turn by: driving the vehicle along the first field exit path at the inner boundary toward the first field exit location, When the inner boundary is encountered at the first field exit position, driving the vehicle toward a first forward position, When the outer boundary is encountered, the vehicle is reversed toward a first correction position, When the first calibration position is reached, driving the vehicle toward the first forward position, When the first forward position is reached, reversing the vehicle toward a first reverse position, and When the first reverse position is reached or the outer boundary is encountered, the vehicle is driven forward at the inner boundary toward the first field re-entry position.
2. The system of claim 1, wherein the first advance position is located relative to the first field re-entry position based on a vehicle turning radius and a vehicle width.
3. The system of claim 1 or claim 2, wherein the first advance position is positioned at least one vehicle width from the first field re-entry position in a direction opposite to the first field re-entry path and at least one vehicle turning radius from the first field re-entry position in a direction opposite to the first field exit position.
4. The system of any one of claims 1 to 3, wherein a first forward heading is parallel to a line connecting the first field exit position to the first field re-entry position.
5. The system of any one of claims 1 to 4, wherein the first correction position is located relative to the first field exit position based on vehicle length.
6. The system of any one of claims 1 to 5, wherein the first correction position is located at least one vehicle length away from the first field exit position in a direction opposite to the first field re-entry position.
7. The system of any one of claims 1 to 6, wherein the first reverse position is located relative to the first field re-entry position based on a vehicle length.
8. The system of any one of claims 1 to 7, wherein the first reverse position is located at least one vehicle length away from the first field re-entry position in a direction opposite to the first field re-entry path.
9. The system of any one of claims 1 to 8, wherein the first field re-entry location is located along the inner boundary.
10. The system of any one of claims 1 to 9, wherein the first field exit path, the first field re-entry path, or both are linear.
11. The system of any one of claims 1 to 10, wherein the first field exit path is parallel to the first field re-entry path.
12. The system of any one of claims 1 to 11, wherein the inner boundary is curvilinear.
13. The system of any one of claims 1 to 12, wherein the outer boundary is curvilinear.
14. The system of any one of claims 1 to 13, wherein a region of the inner boundary between the first field exit location and the first field re-entry location is not perpendicular to the first field exit path, the first field re-entry path, or both.
15. The system of any one of claims 1 to 14, wherein the inner boundary encloses a field.
16. The system of any one of claims 1 to 15, wherein the outer boundary is enclosed by an outer geo-fence.
17. The system of claim 16, wherein the outer boundary is separated from the outer geofence by at least one vehicle length.
18. The system of any one of claims 1 to 17, wherein the first field exit path is separated from the first field re-entry path by a furrow spacing distance.
19. The system of any one of claims 1 to 18, wherein the first field exit path, the first field re-entry path, or both are parallel to furrows in the field.
20. The system of any one of claims 1 to 19, wherein the first turn is further performed by: Prior to reaching the first reverse position and upon encountering the outer boundary, reversing the vehicle toward the first field re-entry position; and When the first field re-entry position oriented in a direction other than along the first field re-entry path is reached, the vehicle is reversed toward the first reverse position.
21. The system of any one of claims 1 to 20, wherein the first turn is determined in real time, adaptively, or both.
22. The system of any one of claims 1 to 21, wherein the first turn is updated as the first turn is performed.
23. The system of any one of claims 1 to 22, wherein the processor is further configured to determine a second turning zone corresponding to a second field exit path and a second field re-entry path having a second field exit location and having a second field re-entry location, wherein the second turning zone is bounded by the outer boundary and the inner boundary, wherein the inner boundary extends through the second field exit location and the second field re-entry location; and wherein the instructions further cause the steering controller to perform a second turn by: driving the vehicle along the second field exit path toward a second field exit location; driving the vehicle toward a second forward position upon encountering the inner boundary at the second field exit position; When the second forward position is reached, reversing the vehicle toward a second reverse position; as well as When the second reverse position is reached or the outer boundary is encountered, the vehicle is reversed toward a second field re-entry position at the inner boundary.
24. The system of claim 23, wherein the first turn and the second turn are different.
25. The system of claim 23 or claim 24, wherein the second turn is determined in real time, adaptively, or both.
26. The system of any one of claims 23 to 25, wherein the second turn is updated as the first turn is performed.
27. The system of any one of claims 23 to 26, wherein the processor is further configured to detect a furrow between the first field re-entry location and the second field exit location.
28. The system of claim 27, wherein the process is configured to identify the furrow in a depth image of a field and a visual image of the field, determine a furrow location, and determine a furrow direction.
29. The system of claim 28, wherein identifying the furrow further comprises generating a segmented map from the visual image.
30. The system of claim 28 or claim 29, wherein determining the furrow direction further comprises determining the center of the furrow from the depth image.
31. The system of any one of claims 28 to 30, wherein determining the furrow position and determining the furrow direction further comprises determining position values and direction values of pixels of the visual image corresponding to the furrow.
32. The system of claim 31 , wherein the processor is further configured to determine a representative position and a representative direction based on the position value and the direction value, wherein the representative position corresponds to the furrow position and the representative direction corresponds to the furrow direction.
33. The system of any one of claims 27 to 32, wherein the instructions, when executed by the processor, further cause the steering controller to travel along the furrow to the second field exit position.
34. The system of any one of claims 1 to 33, wherein the first field exit position, the first field exit path, the first field re-entry position, the first field re-entry path, the first forward position, the first correction position, the first reverse position, the second field exit position, the second field exit path, the second field re-entry position, the second field re-entry path, the second forward position, the second reverse position, or a combination thereof is determined based on sensor input.
35. The system of any one of claims 1 to 34, wherein the first field exit position, the first field exit path, the first field re-entry position, the first field re-entry path, the first forward position, the first correction position, the first reverse position, the second field exit position, the second field exit path, the second field re-entry position, the second field re-entry path, the second forward position, the second reverse position, or a combination thereof is determined in real time.
36. The system of any one of claims 1 to 35, wherein the first field exit position, the first field exit path, the first field re-entry position, the first field re-entry path, the first forward position, the first correction position, the first reverse position, the second field exit position, the second field exit path, the second field re-entry position, the second field re-entry path, the second forward position, the second reverse position, or a combination thereof is adaptively determined.
37. The system of any one of claims 1 to 36, wherein the first field exit position, the first field exit path, the first field re-entry position, the first field re-entry path, the first forward position, the first correction position, the first reverse position, the second field exit position, the second field exit path, the second field re-entry position, the second field re-entry path, the second forward position, the second reverse position, or a combination thereof is independent of a speed of the vehicle.
38. The system of any one of claims 1 to 37, wherein the first field exit position, the first field exit path, the first field re-entry position, the first field re-entry path, the first forward position, the first correction position, the first reverse position, the second field exit position, the second field exit path, the second field re-entry position, the second field re-entry path, the second forward position, the second reverse position, or a combination thereof is independent of a steering rate of the vehicle.
39. The system of any one of claims 1 to 38, wherein the processor is further configured to generate an alarm to request human intervention.
40. The system of claim 39, wherein the alert is generated when the vehicle fails to reach the first field exit position, the first correction position, the first forward position, the first reverse position, the first field re-entry position, the second field exit position, the second correction position, the second forward position, the second reverse position, or the second field re-entry position after a predetermined number of attempts.
41. The system of claim 40, wherein the predetermined number of attempts is 2 to 10 attempts, 3 to 8 attempts, or 4 to 6 attempts.
42. The system of claim 40, wherein the predetermined number of attempts is approximately 5 attempts.
43. A system as claimed in any one of claims 39 to 42, wherein the alert is generated when the vehicle crosses the outer boundary or exceeds the outer boundary by a predetermined distance.
44. The system of claim 43, wherein the predetermined distance is 0.5 to 1.5 vehicle lengths.
45. The system of claim 43, wherein the predetermined distance is approximately one vehicle length.
46. The system of any one of claims 39 to 45, wherein the alarm is generated when the system fails to detect a furrow at the first field re-entry position or between the first field re-entry position and the second field exit position.
47. A system as claimed in any one of claims 39 to 46, wherein the alarm is generated when the system detects an obstacle.
48. A system as claimed in any one of claims 39 to 47, wherein the alarm is configured to be transmitted via a network.
49. The system of any one of claims 39 to 48, wherein the alert comprises an image acquired from the vehicle, a position of the vehicle, a position of the steering controller, a satellite image of the vehicle, or a combination thereof.
50. The system of any one of claims 39 to 49, wherein the processor is further configured to receive instructions from a human after generating the alarm.
51. The system of claim 50, wherein the instructions are provided remotely.
52. The system of claim 50 or claim 51, wherein the processor is further configured to continue executing the first turn, executing the second turn, or proceeding along the furrow after the instruction has been provided.
53. The system of any one of claims 1 to 52, wherein the system further comprises an obstacle detection sensor configured to detect an obstacle.
54. The system of claim 53, wherein the obstacle detection sensor comprises a lidar sensor.
55. The system of any one of claims 1 to 54, wherein the position sensor comprises a global positioning system sensor.
56. The system of any one of claims 1 to 55, wherein the position sensor comprises a camera.
57. The system of any one of claims 1 to 56, wherein the vehicle is an agricultural vehicle.
58. The system of any one of claims 1 to 57, wherein the vehicle is a tractor, a weeder, a sprayer or a seed drill.
59. A method of autonomously driving a vehicle in a field, the method comprising: determining a turning zone corresponding to a field exit path having a first field exit location and a first field re-entry path and having a first field re-entry location, wherein the first turning zone is bounded by an outer boundary and an inner boundary, wherein the outer boundary encloses the inner boundary, wherein the inner boundary extends through the first field exit location and the first field re-entry location, and wherein the inner boundary encloses a planting area of the field; driving the vehicle along the first field exit path at the inner boundary toward the first field exit location; driving the vehicle toward a first forward position upon encountering the inner boundary at the first field exit position; When the outer boundary is encountered, the vehicle is reversed toward a first correction position; When the first calibration position is reached, driving the vehicle toward the first forward position; When the first forward position is reached, reversing the vehicle toward a first reverse position; as well as When the first reverse position is reached or the outer boundary is encountered, the vehicle is driven forward at the inner boundary toward the first field re-entry position.
60. The method of claim 59 further comprising driving the vehicle at a variable speed.
61. The method of claim 59 or claim 60, further comprising: before reaching the first reverse position and upon encountering the outer boundary, reversing the vehicle toward the first field re-entry position; and when the first field re-entry position oriented in a direction other than along the first field re-entry path is reached, backing the vehicle toward the first reverse position.
62. The method of any one of claims 59 to 61, further comprising: determining a second turning zone corresponding to a second field exit path having a second field exit location and a second field re-entry path and having a second field re-entry location, wherein the second turning zone is bounded by the outer boundary and the inner boundary, wherein the inner boundary extends through the second field exit location and the second field re-entry location; driving the vehicle along the second field exit path toward a second field exit location; driving the vehicle toward a second forward position upon encountering the inner boundary at the second field exit position; When the second forward position is reached, reversing the vehicle toward a second reverse position; as well as When the second reverse position is reached or the outer boundary is encountered, the vehicle is reversed toward a second field re-entry position at the inner boundary.
63. The method of claim 62, further comprising traveling along a furrow between the first field re-entry location and the second field exit location.
64. The method of claim 63, wherein traveling along the furrow includes identifying the furrow in a depth image of the field, a visual image of the field, or both, and determining furrow location, furrow direction, or both.
65. The method of claim 64, wherein identifying the furrow further comprises generating a segmented map from the visual image.
66. The method of claim 64 or claim 65, wherein determining the furrow direction further comprises determining the center of the furrow from the depth image.
67. The method of any one of claims 64 to 66, wherein determining the furrow position and determining the furrow direction further comprises determining position values and direction values of pixels of the visual image corresponding to the furrow.
68. The method of claim 67, further comprising determining a representative position and a representative direction based on the position value and the direction value, wherein the representative position corresponds to the furrow position and the representative direction corresponds to the furrow direction.
69. The method of any one of claims 59 to 68, wherein the method is determined in real time, adaptively determined, or both.
70. A method as claimed in any one of claims 59 to 69, wherein the method is updated as the method is performed.
71. The method of any one of claims 59 to 70, further comprising determining the first field exit position, the first field exit path, the first field re-entry position, the first field re-entry path, the first forward position, the first correction position, the first reverse position, the second field exit position, the second field exit path, the second field re-entry position, the second field re-entry path, the second forward position, the second reverse position, or a combination thereof based on sensor input.
72. The method of any one of claims 59 to 71, further comprising determining in real time the first field exit position, the first field exit path, the first field re-entry position, the first field re-entry path, the first forward position, the first correction position, the first reverse position, the second field exit position, the second field exit path, the second field re-entry position, the second field re-entry path, the second forward position, the second reverse position, or a combination thereof.
73. The method of any one of claims 59 to 72, further comprising adaptively determining the first field exit position, the first field exit path, the first field re-entry position, the first field re-entry path, the first forward position, the first correction position, the first reverse position, the second field exit position, the second field exit path, the second field re-entry position, the second field re-entry path, the second forward position, the second reverse position, or a combination thereof.
74. The method of any one of claims 59 to 73, further comprising generating an alert to request human intervention.
75. The method of claim 74, comprising generating the alert when the vehicle fails to reach the first field exit position, the first correction position, the first forward position, the first reverse position, the first field re-entry position, the second field exit position, the second correction position, the second forward position, the second reverse position, or the second field re-entry position after a predetermined number of attempts.
76. The method of claim 74 or claim 75, comprising attempting to reach the first field exit position, the first correction position, the first forward position, the first reverse position, the first field re-entry position, the second field exit position, the second correction position, the second forward position, the second reverse position or the second field re-entry position 2 to 10 times, 3 to 8 times or 4 to 6 times before generating the alarm.
77. The method of claim 74 or claim 75, comprising attempting to reach the first field exit position, the first correction position, the first forward position, the first reverse position, the first field re-entry position, the second field exit position, the second correction position, the second forward position, the second reverse position, or the second field re-entry position approximately 5 times before generating the alert.
78. A method as claimed in any one of claims 74 to 77, comprising generating the alert when the vehicle crosses the outer boundary or exceeds the outer boundary by a predetermined distance.
79. A method as claimed in any one of claims 74 to 78, comprising generating the alert when the vehicle exceeds the outer boundary by a distance of 0.5 to 1.5 vehicle lengths.
80. The method of any one of claims 74 to 78, comprising generating the alert when the vehicle exceeds the outer boundary by a distance of approximately one vehicle length.
81. The method of any one of claims 74 to 80, comprising generating the alert when the system fails to detect a furrow at the first field re-entry position or between the first field re-entry position and the second field exit position.
82. A method as claimed in any one of claims 74 to 81, comprising generating the alarm when the system detects an obstacle.
83. A method as claimed in any one of claims 74 to 82, comprising transmitting the alarm via a network.
84. The method of claim 83, further comprising transmitting, along with the alert, images captured from the vehicle, the location of the vehicle, the location of the steering controller, satellite images of the vehicle, or a combination thereof.
85. The method of any one of claims 74 to 84, further comprising receiving an instruction from a human after generating the alarm.
86. The method of claim 85, wherein the instruction is received via a network.
87. A method as claimed in claim 85 or claim 86, comprising continuing to perform the first turn, performing the second turn or following the furrow after the instruction has been received.
88. The method of any one of claims 59 to 87, further comprising detecting an obstacle using an obstacle detection sensor.
89. The method of claim 88, wherein the obstacle detection sensor comprises a lidar sensor.
90. The method of any one of claims 59 to 89, wherein the method is performed by the system of any one of claims 1 to 58.
91. A computer-implemented method for detecting pear grooves in a field, the computer-implemented method comprising: receiving a depth image of a field and a visual image of said field; obtaining labeled image data, the labeled image data including furrows, furrow positions, and furrow directions in an image of similar furrows; using the labeled image data to train a machine learning model to identify furrows, determine furrow locations, and determine furrow directions; identifying furrows by using the depth image and the visual image as inputs to the machine learning model; as well as The furrow position and furrow direction of the furrow are determined by using the depth image and the visual image as inputs to the machine learning model.
92. The computer-implemented method of claim 91, wherein identifying the furrows further comprises generating a segmented map from the visual image.
93. The computer-implemented method of claim 91 or claim 92, wherein determining the furrow direction further comprises determining a center of the furrow from the depth image.
94. A computer-implemented method as described in any one of claims 91 to 93, wherein determining the furrow position and the furrow direction also includes determining position values and direction values of pixels of the visual image corresponding to the furrow.
95. The computer-implemented method of claim 94, further comprising determining a representative position and a representative direction based on the position value and the direction value, wherein the representative position corresponds to the furrow position and the representative direction corresponds to the furrow direction.
96. A method of navigating in a field, the method comprising: Using a computer-implemented method as described in any one of claims 91 to 95 to detect a first furrow, guiding a vehicle along the first furrow, exiting the field along a field exit path, maneuvering the vehicle using a system as described in any one of claims 1 to 58 or a method as described in any one of claims 59 to 90, re-entering the field along a field re-entry path at a second furrow detected using a computer-implemented method as described in any one of claims 91 to 95, or a combination thereof.
Citation Information
Patent Citations
Early-stage ribbing ridge line recognition algorithm based on one-dimensional Hough transform and expert system
CN103186773A
Farmland machine leading line extraction algorithm based on machine vision
CN103914836A
Apparatus and method for controlling head-in perpendicular parking of vehicle, and system for head-in perpendicular parking of vehicle with the apparatus
CN104670328A
Agricultural machine operation method and system for automatically generating working path without using map
CN108205322A
Method and system for removing weeds from farmland
CN108776803A
Cited By
Irregular plot internal boundary planning method
CN122636660A
