Method for obstacle detection, obstacle classification and obstacle tracking

Abstract

A method for identifying, classifying, and tracking obstacles includes scanning an area using a scanner (e.g., a stereoscopic camera) to generate a point cloud of the area. Determining a ground plane of the area based on the point cloud, and identifying an obstacle in the area based on the point cloud. Obstacles are classified based on the point clouds and tracked based on a series of point clouds generated by a scanner over time.

Description

Technical Field

[0001] This disclosure generally relates to methods for imaging, and more specifically, to methods for obstacle detection, obstacle classification, and obstacle tracking. Background Technology

[0002] Operating machines capable of moving within an environment requires operators or controllers to perceive obstacles so that the machines can operate automatically. When operating machines manually, the operator is responsible for observing and avoiding obstacles. However, accidents can occur for a variety of reasons, including fatigue, poor visibility, and inattention. Autonomous vehicles need to detect and avoid obstacles to prevent collisions. A method for detecting and tracking obstacles is needed, which can be used both to assist human operators in operating equipment and to facilitate the automated operation of vehicles and / or equipment. Summary of the Invention

[0003] A method for obstacle detection, obstacle classification, and obstacle tracking includes scanning an area using a scanner to generate a point cloud. A ground plane is determined based on the point cloud, and obstacles in the area are identified based on points in the point cloud that are associated with the ground plane. The determination may include: dividing an approximate ground plane into multiple cells; determining the maximum height of points in each of the multiple cells above the approximate ground plane; and classifying the cells as either ground planes or non-ground planes based on the maximum height of points in each of the multiple cells. The scanner may be mounted on a machine, and the approximate ground plane is based on the distance from the point where the machine contacts the ground to the scanner. Obstacle identification may include identifying cells with heights exceeding a threshold, where the height value is the height of points in the cell above the approximate ground plane. In one embodiment, obstacle identification in the area further includes forming segments by combining cells with height values ​​exceeding the threshold, and may further include determining that the segment contains multiple cells exceeding the threshold and identifying the segment as an obstacle in response to determining that the segment contains multiple cells exceeding the threshold. Obstacles can be classified based on the point cloud. The method may include generating a series of point clouds by a scanner over time, and tracking obstacles based on the series of point clouds. In one embodiment, obstacles are classified based on the output of a convolutional neural network, and this output is generated based on identified segments of one of the obstacles.

[0004] This document also describes an apparatus for obstacle detection, obstacle classification, and obstacle tracking, and a computer-readable medium storing computer program instructions for obstacle detection, obstacle classification, and obstacle tracking, the apparatus including a scanner and a processor. Attached Figure Description

[0005] FIG. 1A A stereoscopic camera device for scanning an area is shown according to one embodiment; FIG. 1B It shows that it is installed FIG. 1A Lawn mowers equipped with 3D camera systems; FIG. 2A The image captured using the image sensor of the stereo camera device shown in Figure 1 is illustrated. FIG. 2B It shows FIG. 2A 3D point cloud of the area shown in the image; FIG. 3 A cell map constructed using a cell height algorithm according to one implementation is shown; FIG. 4 The use according to one embodiment is shown. FIG. 3 Morphological filtering of cell maps FIG. 2A The map shows the area depicted in the image; FIG. 5 An embodiment is shown. FIG. 2A The image shows a map of the region that has a single connected component; FIG. 6 A map of the Earth's surface within the field of view of the stereoscopic camera device in Figure 1 is shown according to one embodiment; FIG. 7A An embodiment is shown. FIG. 2A The image shows a map of the area with identified obstacle points; FIG. 7B A method according to one embodiment is shown. FIG. 7A The resulting map after segmenting the map; FIG. 8 An obstacle identified in the field of view of the stereoscopic camera device in FIG1 according to one embodiment is shown; FIG. 9A The results of a recognition convolutional neural network (“CNN”) for obstacle recognition according to one embodiment are shown; FIG. 9B The results of a classification CNN for classifying obstacles according to one embodiment are shown; FIG. 10The results of an object classifier according to one embodiment are shown, in which obstacles are colored; FIG. 11A The image shows obstacles in a region at the first timestamp; FIG. 11B The obstacles in a region at the second timestamp are shown; FIG. 11C The obstacles in a region are shown at the third timestamp; FIG. 12 A flowchart illustrating a method for identifying, classifying, and tracking obstacles according to one embodiment is shown; and FIG. 13 A high-level block diagram of a computer according to one embodiment is shown. Detailed Implementation

[0006] Methods for obstacle detection, obstacle classification, and obstacle tracking involve scanning an area surrounding a machine using sensors (e.g., 3D sensors) to generate a point cloud. The point cloud is then analyzed using various techniques to detect objects in the area surrounding the machine (e.g., obstacles can be identified based on points in the point cloud). The detected objects are then classified and tracked. The results of detection, classification, and tracking can be used to facilitate the automated operation of the machine on which sensors are mounted, or to provide obstacle information to the machine's operator. This method is described below in conjunction with an autonomous lawnmower ("the machine"), but it can also be used with other types of machines.

[0007] In one embodiment, the method begins by scanning an area surrounding the machine using a scanner mounted at a known location on the machine. In one embodiment, the scanner is a sensor (i.e., a 3D sensor), such as, for example, a stereo camera, LiDAR, or a time-of-flight (“ToF”) camera. The sensor can be oriented in various positions such that it faces forward, backward, or sideways relative to the front of the machine and / or the direction of travel during machine operation. In one embodiment, the sensor is tilted downwards such that the surface the machine is traveling on is within the sensor's field of view. The sensor generates information based on the scan of the area surrounding the machine. In one embodiment, the sensor generates a point cloud and may also determine color (i.e., RGB values), black and white (“B&W”), and / or brightness information for each point in the cloud.

[0008] FIG. 1AA stereo imaging device 100 with lenses 102A and 102B is shown, each lens focusing an image onto one of image sensors 104A and 104B, respectively. In one embodiment, a 2D B&W image is obtained using one of the image sensors (e.g., image sensor 104A), and a 3D point cloud is generated using information from both image sensors 104A and 104B. The pixels of the 2D B&W image and the 3D point cloud are associated with a shared local coordinate system of the imaging device 100. In one embodiment, a Cartesian coordinate system is used as the shared local coordinate system of the two image sensors 104A and 104B. The shared local coordinate system has an origin O located at the upper left corner of image sensor 104A. The shared local coordinate system has three axes, where the X-axis is horizontal, the Y-axis is vertical, and the Z-axis is perpendicular to both the X-axis and Y-axis. FIG. 1B A stereoscopic camera device 100 is shown located on top of a lawnmower 214. The stereoscopic camera device 100 communicates with a processing unit 106, shown as being located in the cab of the lawnmower 214. In one embodiment, the processing unit 106 performs the methods, steps, and / or operations described herein based on data received from the stereoscopic camera device 100. In one embodiment, the stereoscopic camera device 100 is mounted on top of the lawnmower 214 in a downwardly tilted manner, such that the stereoscopic camera device 100 captures images of the area in front of the lawnmower 214. It should be noted that the vertical distance H from the bottom of the wheels 216 of the lawnmower 214 to the image sensors 104A, 104B of the stereoscopic camera device 100 is known.

[0009] Sensor 100 is used to acquire images of areas in which obstacles may be located. In one embodiment, a recognition convolutional neural network (“CNN”) is used to identify obstacles in the image, and a segmentation or detection CNN is used to classify the obstacles. The recognition CNN uses 2D frames (i.e., images) from the image sensor 104A of the stereo camera device 100 to obtain initial information.

[0010] FIG. 2A Image 200 of B&W captured using image sensor 104A is shown. Image 200 shows a person 210A near a golf cart 212A, and the shadow 214A of a lawnmower 214 on which the stereoscopic camera device 100 is located. FIG. 2B A 3D point cloud 202 captured using image sensors 104A and 104B is shown, which illustrates pixelated versions of the objects shown in image 200. Specifically, a pixelated person 210B is shown near the shadows 214B of a pixelated golf cart 212B and a pixelated lawnmower.

[0011] After acquiring the 3D point cloud, an approximate ground plane is determined based on the 3D point cloud and information about the position and orientation of the stereo camera device 100 on the lawn mower. In one embodiment, the ground plane is the ground on which the lawn mower can travel. The ground plane does not include plants or vegetation that may obstruct the view of the ground plane, such as tall grass or shrubs.

[0012] In one implementation, 3D point cloud data and the Random Sample Consensus (RANSAC) algorithm are used to determine the ground plane. Since the location of the stereo camera 100 mounted on the lawnmower is known, the approximate location of the ground plane is also known. This is because the vertical distance H from the bottom of the lawnmower's wheels to the image sensing element of the stereo camera 100 is known. Therefore, it can be assumed that the location of the ground plane in any specific area of ​​the 3D point cloud is within a specific distance above or below the ground plane height, which is determined using distances from the bottom of the lawnmower's wheels (e.g., from the bottom of the lawnmower's wheels). FIG. 1B The known vertical distance H from the image sensors 104A and 104B of the stereo camera device 100 (as shown) is estimated.

[0013] In one embodiment, based on the height, roll angle, and pitch angle of the stereo camera device 100, the 3D point cloud is transformed by translation and rotation, so that the determined ground plane is located in the xOz plane of the local coordinate system of the stereo camera device 100.

[0014] In one implementation, the Cartesian coordinate system (x, y, z) used for the stereo camera device 100 coordinate system is converted to a cylindrical coordinate system (r, a, h):

[0015] The h-coordinate represents the vertical distance from a point in the 3D point cloud to the determined ground plane. As described herein, obstacles can be identified based on points in the point cloud that are associated with the ground plane. In one embodiment, the determined ground plane is divided into cells in a cylindrical coordinate system, and the height of each cell is determined based on information from the 3D point cloud. The height of each cell can be calculated using all points in the cloud located within that cell. A point is considered to be within a cell if its projection onto the rOa plane lies within the cell's boundary. In one embodiment, the following cell height algorithm is used to calculate the cell height. First, all points within the cell are sorted by their height. Next, a sliding window is used to vertically group the points in the cell. In one embodiment, the points in the cell are vertically grouped from one point to a higher point. The window value is calculated by subtracting the number of points below the window from the number of points within the window. The position of the window with the largest window value is used as an estimate of the cell's height.

[0016] FIG. 3 The image shows a cell map 300 constructed using a cell height algorithm. The plane rOa in cylindrical coordinates is shown as coinciding with the plane xOz in Cartesian coordinates. The shading of a cell indicates its height. Higher cells have less shading. Black-shaded cells indicate that the cell does not contain any points from the point cloud.

[0017] Use a filter to FIG. 3 The cell map 300 shown is converted to FIG. 4 Map 400 in the image. In one embodiment, a morphological filtering operator is used to identify cells in the map with heights above a threshold. Cells with heights above the threshold are excluded from the ground plane, and the remaining cells are considered cells of the ground plane (i.e., the Earth's surface). In one embodiment, a sequence of erosion and dilation (i.e., opening operations) is used as the filtering operator.

[0018] In one implementation, each cell is indexed using the values ​​r and a, and each cell uniquely corresponds to a pair of indices in r and a. In one implementation, the erosion operation is defined using a radius w, and for each cell: 1) all cells are considered where the difference between the two indices and the corresponding indices of the original cell does not exceed w; 2) the minimum height in the considered cells is identified; and 3) the new height of the source cell is set to be equal to that minimum height.

[0019] In one implementation, the dilation operation is defined using a radius w, and is similar to the erosion operation, except that the dilation operation identifies the maximum height instead of the minimum height. In another implementation, the opening operation using radius w is defined as sequentially executing an erosion operation and a dilation operation, both using radius w.

[0020] In one implementation, a minimum and a maximum value for w are set. For each value of w, the height threshold dH(w) is calculated as w. dh1 + dh0, where dh0 and dh1 are user-defined parameters. In one implementation, the opening operation is performed while gradually increasing w from its minimum to its maximum value. After each opening operation, cells whose height has decreased by more than dH(w) are identified for subsequent filtering.

[0021] In one implementation, the height of all cells is restored to its original value before the open operation is applied. Cells stored during the open operation are identified as "non-Earth," and the remaining cells are identified as "Earth."

[0022] FIG. 4 It shows FIG. 2A The map 400 showing the area in the image is a map created using the methods described above. FIG. 3 The cell map 300 is generated by morphological filtering. Black cells 402 are classified as "Earth," while shaded cells 404 are classified as "Non-Earth." In one implementation, cells classified as "Earth" can be represented as a graph, where the vertices of the graph are Earth cells, and cells are connected (i.e., have shared nodes) if the height difference between cells is small and the cells are adjacent. In one implementation, "Earth" cells are grouped into connectivity components (i.e., contiguous regions).

[0023] FIG. 5 It shows FIG. 2A The image shows a map 500 of the region, which has a single connected component 502. The single connected component 502 is a single contiguous region of cells formed by all the "Earth" cells using morphological filtering. (See image 500.) FIG. 5 As shown, the boundary of connected component 502 is indicated by a thick line 504. Similarly, as... FIG. 5 As shown, the thick line 504 contrasts with the shadows of the cells contained in the connected component 502.

[0024] In one implementation, the height of the Earth's surface in cells not included in the connected component 502 (i.e., cells marked "non-Earth") is determined by interpolation or extrapolation based on information from cells located in the connected component 502.

[0025] FIG. 6 Map 600 is shown, which is a complete map of the Earth's surface within the field of view of the stereo camera device 100. Map 600 includes all cells identified as part of the Earth's surface, and the boundaries of connected component 602 and other cells with interpolated / extrapolated heights are marked with thick lines. The cells on map 600 have varying heights, with cells of higher heights shown as having a lighter shade than those of lower heights. In this embodiment, a flat plane is not used as the surface beneath the cells. The actual variable height of the area within the field of view of the stereo camera device 100 is used, which is important in applications where the stereo camera device 100 observes off-road conditions. Not FIG. 6 All cells shown have height information. In one case, cells have no height information because no points in the 3D point cloud are associated with these cells. Black hole 604 consists of a series of adjacent cells with no usable height information. These regions do not contain information that can be analyzed to identify obstacles. Based on data from the stereo camera device 100, these regions cannot be observed and pose a hazard as much as obstacles.

[0026] Next, all points in the 3D point cloud that exceed a certain threshold on the Earth's surface map are identified and marked as obstacle points. FIG. 7A It shows FIG. 2A The image shows a map of the area 700A, where obstacle points are marked in black, forming obstacles 702A and 704A.

[0027] In one embodiment, the points marked as obstacle points are grouped into segments in the coordinate system of the spherical camera device, wherein... It's longitude. It is latitude, and These are radii, which are determined using the following equation:

[0028] In one implementation, the spherical coordinate system is based on angular coordinates. and With constant increment and The system is divided into multiple segments. In this embodiment, each segment contains all the markers associated with that segment (i.e., the angular coordinates of each marker are located within that segment). If a segment contains fewer than a specified number of points, that segment will be excluded from the analysis due to insufficient representativeness. Next, a third [analysis method] is determined for each segment. Coordinates. In one implementation, this is achieved by using coordinates. This is performed by ranking all points belonging to that segment. After ranking, points with the minimum and maximum values ​​are excluded. 10% of the points. This allows for the exclusion of outliers. In one implementation, for this segment... The coordinates are those closest to the camera device (with the smallest distance). ) point (from the remainder set) Therefore, we obtain multiple segments, each with its own coordinates. , and . FIG. 7B It shows FIG. 7A Map 700B is a segmentation result of the map. The boxes show the segments of obstacles 702B and 704B in the spherical coordinate system.

[0029] In one implementation, obstacles are formed based on the vertices of a graph, which are considered segments of the obstacle. In another implementation, vertices are the segments remaining after excluding segments with a smaller number of points. If the segments are based on coordinates... and Adjacent and positioned relative to If the two vertices are close to each other, they are connected by nodes. This results in an undirected, unweighted graph with no cycles and multiple edges.

[0030] In one implementation, connected components in the graph are isolated by deep traversal. In another implementation, a standard component isolation algorithm is used to isolate connected components. All connected components are identified, where a connected component is a set of connected vertices in the graph (each vertex uniquely corresponding to a segment). In one implementation, a set of segments (connected components) is identified as potential obstacles.

[0031] FIG. 8 The obstacles 802 and 804 (i.e., segmental connected components) identified using the method described above are shown based on the field of view of the stereo camera device 100.

[0032] After completing the obstacle search and identification, obstacle classification begins. In one embodiment, a segmentation or detection CNN is used to classify obstacles. In a first embodiment, the segmentation neural network assigns each pixel of the camera image to a category in a predefined set (e.g., "person," "vehicle," "golf equipment"), or, if the neural network does not classify the pixel, to the background. In a specific embodiment, the detection neural network generates a set of bounding rectangles (bounding boxes) corresponding to the objects, each rectangle being assigned a category from a predefined set. These categories are indexed by priority. For example: person = 1, vehicle = 2, golf course personnel = 3. The smaller the index, the higher the priority. In one embodiment, people are more important than vehicles, and vehicles are more important than golf equipment.

[0033] In one implementation, the category of each pixel is defined as follows: 1) If a pixel is not within any bounding box, it is identified as background; 2) If a pixel falls within one or more bounding boxes of the same category, the pixel is considered to be of that same category; 3) If a pixel falls within several different bounding boxes, the pixel belongs to the pixel category with the smallest index (i.e., the most important). In one implementation, for both options (i.e., segmentation or detection CNN), the category of each pixel in the image is identified from a predefined set or background.

[0034] FIG. 9A The results of CNN segmentation of the image from the stereo camera device 100 are shown, in which obstacles 902A and 904A have been identified. FIG. 9B The results of the CNN classification are shown, where obstacles 902B and 904B are depicted as being surrounded by bounding boxes 902C and 904C. Although not shown in the figure, in one embodiment, objects can be colored according to their classification. For example, vehicles could be shown in red, while people could be shown in yellow. In one embodiment, the bounding boxes surrounding the classified obstacles include text recognition of the obstacle category.

[0035] In one implementation, by and The range defines each segment, and each pixel of the image corresponds to... and Certain values. These values ​​depend only on the pixel location in the image and the camera device parameters. Therefore, each segment corresponds to a specific group of pixels in the image.

[0036] Previously, obstacles were defined as a set of connected segments. In one implementation, it was assumed that all pixels corresponding to a segment were associated with an obstacle. To determine the obstacle category, the number of pixels of all categories and background pixels corresponding to the obstacle was counted. The resulting value was multiplied by a user-specified category weight. The weight value reflects the importance of the category. The category with the highest value was selected. If it is background, no category is assigned to the obstacle (the obstacle remains unclassified).

[0037] FIG. 10 The results of the obstacle classifier are shown, where vehicle 1004 is colored with a specific color (e.g., red), person 1002 is colored with a different color (e.g., yellow), and black hole 1006 (i.e., the unidentifiable area) is colored with another color (e.g., green). It should be noted that although the obstacles in the figure are not colored, they can be colored using various implementations described herein.

[0038] In one implementation, the obstacle identification and classification described above are performed for each time epoch. A series of timestamps can be used to track obstacles over time. Each frame captured by the stereo camera device 100 is associated with a timestamp. Obstacles in the current frame are matched with obstacles found in previous frames to track obstacles over time.

[0039] In one implementation, a Kalman filter is used to track obstacles. In one implementation, the Kalman filter has seven steps, beginning with the Kalman filter's prediction of all past obstacles in the previous frame. Next, for all possible pairs of current / past obstacles, the matching cost is calculated using the distance between the obstacle cores and the IOU metric. In one implementation, the IOU metric is the area of ​​the intersecting sectors of two obstacles divided by the area of ​​the total sectors of those obstacles. The matrix is ​​then compiled based on the cost of matching obstacles with each other. Then, using the Hungarian algorithm, the optimal solution is selected that minimizes the total cost of matching obstacles. The Kalman filter is then updated based on past obstacles that have already matched the current obstacle. Kalman filters for past obstacles that have not matched the current obstacle for a sufficient number of frames are removed. Kalman filters are created for obstacles that have not yet matched past obstacles, and this process is repeated.

[0040] FIG. 11A , FIG. 11B and FIG. 11C Three frames are shown, captured by stereoscopic camera device 100 during the movement of a lawnmower on a golf course. FIG. 11B and FIG. 11C As shown, FIG. 11AThe person shown has moved within the time frame during which the frame was captured. In one implementation, FIG. 11A , FIG. 11B and FIG. 11C The frame shown may contain information about obstacles located within the frame. In one implementation, FIG. 11A , FIG. 11B and FIG. 11C Each frame in the diagram contains a legend. In one implementation, the legend contains the following information: A stamp identifies the frame number. Black indicates the percentage of pixels in the 2D frame with a brightness of 0 to 25. White indicates the percentage of pixels in the 2D frame with a brightness of 230 to 255. Contamination indicates the degree of contamination of the camera lens as a percentage. DepthV is the percentage of points in the 3D cloud obtained from the camera device relative to the maximum possible number of points. In one implementation, a frame may include a legend for one or more obstacles and contain the following information: tID is a unique number associated with the obstacle. Type identifies the category of the obstacle (e.g., -2 / -1 / 0 / 1 / 2 / 3 / 4 represent black hole / unclassified / person / vehicle / animal / club / golf equipment, respectively). dC is the confidence level of obstacle detection as a percentage. cC is the confidence level of obstacle classification as a percentage. oID is the obstacle sequence number of the current stamp (frame).

[0041] FIG. 12 A flowchart 1200 of a method for identifying, classifying, and tracking obstacles according to one embodiment is shown. In one embodiment, in step 1202, a region is scanned using a scanner such as the stereo camera device 100 in FIG. 1 to generate a point cloud of the region, as described above. FIG. 2B As described above. In one embodiment, the point cloud is transmitted to... FIG. 2B The processing unit 106 shown here executes steps 1204 to 1210. In step 1204, based on the point cloud, the ground plane of the area is determined, as described above. FIG. 2B and FIG. 3 As described above. In step 1206, based on the point cloud, obstacles in the region are identified, as described above. FIG. 4 to FIG. 8 As described above. In step 1208, obstacles are classified based on the point cloud, as per the above description. FIG. 9A to FIG. 10 As described above. In step 1210, obstacles are tracked based on a series of point clouds generated by the scanner over time, as described above. FIG. 11A to FIG. 11C As stated above.

[0042] In one embodiment, determining the ground plane of the area includes dividing an approximate ground plane into multiple cells. Then, the maximum height of a point in each of the multiple cells above the approximate ground plane is determined, and based on the maximum height of the point in each of the multiple cells, the corresponding cells in the multiple cells are classified as either ground planes or non-ground planes. In one embodiment, the approximate ground plane is based on the distance from the location where the machine with the scanner mounted contacts the ground to the scanner. In one embodiment, identifying obstacles in the area includes identifying cells with height values ​​exceeding a threshold, wherein the height value is the height of the cell above the approximate ground plane. In one embodiment, identifying obstacles in the area further includes forming segments by combining cells with height values ​​exceeding the threshold. In one embodiment, identifying obstacles in the area further includes identifying segments of obstacles in response to determining that the segment contains multiple cells exceeding the threshold. In one embodiment, obstacle classification employs a convolutional neural network to classify obstacles based on identified segments of obstacles.

[0043] It is important to note that the "ground plane" does not necessarily need to be flat. The ground plane can be any arbitrary surface. For example, a golf course can have an uneven surface (i.e., a surface comprising one or more low mounds, ridges, or hills). The methods and algorithms described herein can be used to identify obstacles on surfaces with unstable / uniform heights. In one implementation, obstacles above the surface are first identified in a 3D point cloud to isolate them. Then, a neural network based on camera frame data is used to classify the obstacles. This sequence avoids the neural network erroneously detecting obstacles. For example, the neural network might process the shadow of an object in an image, even if the object casting the shadow is not in the image. For instance, a person can cast a shadow on a surface, and the neural network might classify that shadow as a person. Following the above sequence, a person's shadow would not be identified as an obstacle because the shadow would not be recognized as being above the surface and would therefore be considered part of the ground plane.

[0044] The location where the systems and methods described herein are used may influence how obstacle information is used. In one embodiment, the systems and methods described herein are used in conjunction with autonomous lawnmowers used on golf courses. In one embodiment, the type of obstacle determines how a vehicle such as an autonomous lawnmower will respond. If the obstacle is a person, the autonomous lawnmower will not move until the person leaves the area. If the obstacle is determined to be a vehicle, the autonomous lawnmower will reduce its speed and attempt to go around the vehicle. If the obstacle is determined to be golf clubs, and assuming the golf clubs are stationary, the autonomous lawnmower will go around the golf clubs.

[0045] The scanner 100 and processing unit 106, as well as the methods and techniques described herein, can be implemented using one or more computers. For example, they can be implemented using a computer. FIG. 12 The method shown. FIG. 13 A high-level block diagram of such a computer is shown. Computer 1302 includes processor 1304, which controls the overall operation of computer 1302 by executing computer program instructions that define such operations. The computer program instructions may be stored in storage device 1312 or other computer-readable media (e.g., disk, CD-ROM, etc.) and loaded into memory 1310 when execution of the computer program instructions is desired. Therefore, FIG. 12 The method steps can be defined by computer program instructions stored in memory 1310 and / or storage device 1312, and controlled by processor 1304 executing the computer program instructions. For example, the computer program instructions can be implemented as computer executable code programmed by those skilled in the art to perform... FIG. 12 The algorithm is defined by the method steps. Therefore, by executing computer program instructions, processor 1304 executes... FIG. 12 The method steps define the algorithm. Computer 1302 also includes one or more network interfaces 1306 for communicating with other devices over a network. Computer 1302 also includes input / output devices 1308 that enable a user to interact with computer 1302 (e.g., a monitor, keyboard, mouse, speakers, buttons, etc.). Those skilled in the art will recognize that actual computer implementations may also include other components, and FIG. 13 This is a high-level representation of some components of this computer for illustrative purposes.

[0046] The foregoing detailed description should be understood as illustrative and exemplary in all respects, and not restrictive, and the scope of the inventive concept disclosed herein should be interpreted in the full breadth permitted by patent law. It should be understood that the embodiments shown and described herein are merely for illustrating the principles of the inventive concept, and various modifications can be made by those skilled in the art without departing from the scope and spirit of the inventive concept. Various other combinations of features can be implemented by those skilled in the art without departing from the scope and spirit of the inventive concept.

Claims

1. A method, the method comprising: Use a scanner to scan an area to generate a point cloud; Based on the point cloud, the ground plane of the region is determined; as well as Obstacles in the region are identified based on points in the point cloud that are related to the ground plane.

2. The method according to claim 1, wherein, Determining the ground plane of the area includes: Divide the approximate ground plane into multiple cells; The maximum height of a point in each of the plurality of cells above the approximate ground plane is determined; and The cells are classified as either ground planes or non-ground planes based on the maximum height of the points in each of the plurality of cells.

3. The method according to claim 2, wherein, The scanner is mounted on the machine, and the approximate ground plane is based on the distance between the machine and the ground at the point of contact with the ground and the scanner.

4. The method according to claim 2, wherein, Obstacle identification includes: Cells with height values ​​exceeding a threshold are identified, where the height value is the height of a point in the cell above the approximate ground plane.

5. The method according to claim 4, wherein, Identifying obstacles in the area also includes: Segments are formed by combining cells with height values ​​exceeding the threshold.

6. The method according to claim 5, wherein, Identifying obstacles in the area also includes: The segment is determined to contain multiple cells exceeding a threshold; and In response to determining that the segment contains more than the threshold of cells, the segment is identified as an obstacle.

7. The method according to claim 1, further comprising: The obstacles are classified based on the point cloud.

8. The method according to claim 1, further comprising: The scanner generates a series of point clouds over time; as well as The obstacle is tracked based on the aforementioned point cloud.

9. The method according to claim 1, further comprising: The obstacles are classified based on the output of a convolutional neural network, which is generated based on the segment identified from one of the obstacles.

10. An apparatus comprising: A scanner configured to scan a region and generate a point cloud; and A processor, which communicates with the scanner, is configured to perform operations including the following: Based on the point cloud, the ground plane of the region is determined; and Obstacles in the region are identified based on points in the point cloud that are related to the ground plane.

11. The apparatus according to claim 10, wherein, Determining the ground plane of the area includes: Divide the approximate ground plane into multiple cells; The maximum height of a point in each of the plurality of cells above the approximate ground plane is determined; and The cells are classified as either ground planes or non-ground planes based on the maximum height of the points in each of the plurality of cells.

12. The apparatus according to claim 11, wherein, The scanner is mounted on the machine, and the approximate ground plane is based on the distance between the machine and the ground at the point of contact with the ground and the scanner.

13. The apparatus according to claim 11, wherein, Obstacle identification includes: Cells with height values ​​exceeding a threshold are identified, where the height value is the height of a point in the cell above the approximate ground plane.

14. The apparatus according to claim 13, wherein, Identifying obstacles in the area also includes: Segments are formed by combining cells with height values ​​exceeding the threshold.

15. The apparatus according to claim 14, wherein, Identifying obstacles in the area also includes: The segment is determined to contain multiple cells exceeding a threshold; and In response to determining that the segment contains more than the threshold of cells, the segment is identified as an obstacle.

16. The apparatus of claim 10, wherein the operation further comprises: The obstacles are classified based on the point cloud.

17. The apparatus of claim 10, further comprising: The scanner generates a series of point clouds over time; as well as The obstacle is tracked based on the aforementioned point cloud.

18. The apparatus of claim 10, wherein the operation further comprises: The obstacles are classified based on the output of a convolutional neural network, which is generated based on the segment identified from one of the obstacles.

19. A computer-readable medium storing computer program instructions that, when executed on a processor, cause the processor to perform operations including: Receive point cloud data from the scanner based on a scan of a region; Based on the point cloud, the ground plane of the region is determined; and Obstacles in the region are identified based on points in the point cloud that are related to the ground plane.

20. The computer-readable medium of claim 19, wherein, Determining the ground plane of the area includes: Divide the approximate ground plane into multiple cells; The maximum height of a point in each of the plurality of cells above the approximate ground plane is determined; and The cells are classified as either ground planes or non-ground planes based on the maximum height of the points in each of the plurality of cells.

21. The computer-readable medium of claim 20, wherein, The scanner is mounted on the machine, and the approximate ground plane is based on the distance between the machine and the ground at the point of contact with the ground and the scanner.

22. The computer-readable medium of claim 20, wherein, Obstacle identification includes: Cells with height values ​​exceeding a threshold are identified, where the height value is the height of a point in the cell above the approximate ground plane.

23. The computer-readable medium according to claim 22, wherein, Identifying obstacles in the area also includes: Segments are formed by combining cells with height values ​​exceeding the threshold.

24. The computer-readable medium of claim 23, wherein, Identifying obstacles in the area also includes: The segment is determined to contain multiple cells exceeding a threshold; and In response to determining that the segment contains more than the threshold of cells, the segment is identified as an obstacle.

25. The computer-readable medium of claim 19, further comprising: The obstacles are classified based on the point cloud.

26. The computer-readable medium of claim 19, further comprising: The scanner generates a series of point clouds over time; as well as The obstacle is tracked based on the aforementioned point cloud.

27. The computer-readable medium of claim 19, further comprising: The obstacles are classified based on the output of a convolutional neural network, which is generated based on the segment identified from one of the obstacles.

Images