Method for object detection for a mobile work machine

Abstract

The present invention relates to a method for object detection for a mobile work machine, comprising the steps of: receiving and / or capturing at least a first image element and a second image element, each comprising a three-dimensional image, determining a first plurality of point elements in the first image element and a second plurality of point elements in the second image element, assigning the first plurality of point elements to a predetermined number of cell elements, adapting the cell elements comprising the first plurality of point elements using the second plurality of point elements, forming a map element comprising a three-dimensional map based on the adapted cell elements in order to detect an object based on the formed map element.

Description

Prior ArtThe present invention relates to a method for object detection for a mobile working machine and to a mobile working machine.At present, there are a large number of different solutions for object detection in the robot area. As a result of the increasing number of application scenarios for travel robots and as a result of the increased autonomous requirements of the travel robots, the need for innovative and robust methods for object recognition is continually increasing.The constantly increasing detection accuracy in the area of the moving robot and the increasing competition ensure cost pressure, so that more favorable and more efficient components for moving robots are demanded more strongly.US 11093759 B2 discloses methods for automatically generating object polylines which represent roadside objects in a region of a roadway.US 2019163958 A1 discloses a method for planning routes for a moving device.US 10733320 B2 discloses a method for identifying the free space between objects.Disclosure of the InventionThe method according to the invention for object detection for a mobile work machine having the features of claim 1 has the advantage over the known ones that one or more point clouds can be fused in a 3D map. By means of the 3D map, not only a two-dimensional position can be determined, but also a third dimension can be determined, such as the height, so that a mobile working machine, in particular a travel robot, for example an industrial cleaning robot, can determine whether or not it can pass under an obstacle. A further advantage is that the 3D map moves with the mobile working machine or the robot and thus possible conversion steps can be avoided.This is achieved according to the invention in that the method for object detection for a mobile working machine has the steps:receiving and / or detecting at least one first image element and one second image element, each of which comprises a three-dimensional image,determining a first plurality of point elements in the first picture element and a second plurality of point elements in the second picture element,assigning the first plurality of point elements to a predetermined number of cell elements,adapting the cell elements, which have the first plurality of point elements, on the basis of the second plurality of point elements,forming a map element, which comprises a three-dimensional map, based on the adapted cell elements, in order to detect an object based on the formed map element.In other words, a three-dimensional map can be created by means of a plurality of points or a three-dimensional point cloud, in order to be able to identify possible objects or obstacles for the mobile work machine. In order to increase the detection accuracy, in particular a plurality of image elements of both a recording device and a plurality of recording devices can be fused in order to be able to avoid possible false detections. In a first step, all detected or determined points can be distributed or assigned to the cell elements. This can be repeated for each pixel so that the number of points in a cell can be continuously updated. Thus, the number of points assigned to a cell can change over the history. Based on the number of points in a cell, it can be decided whether an object has been detected. More preferably, there may be situations where cells have not been assigned points so that they take to the value zero. Thus, in particular already occupied cells can become free again.Based on the change in the number of points in a cell, the cell can be classified, for example, into static or dynamic objects. A static object is thereby detected if the number of points in the cell is substantially the same over a plurality of image elements. In cells which have a large variation in the number of points, a dynamic object such as a moving person or the like can be assumed. In this case, in particular individual cell elements can be indicative of an obstacle or the like. For example, when a mobile work machine such as a traveling robot has a plurality of cameras and each of the cameras detects an obstacle, they may be merged into a three-dimensional map. The stability of the object detection can be increased in that a sufficient number of points have been linked to a cell, and a high probability of an object at this location can be assumed. In this case, for example, the field of view of a camera which records the first image element can be simulated by means of a truncated cone or the like. Further, only cells that are in the truncated cone may be updated. For example, an object in a first image element is detected by means of the assignment of a plurality of points to a cell. In a second picture element, which is offset in time with respect to the first picture element, there is also an increased number of points in a cell. It can thus be assumed that an object or obstacle is present in this cell.The dependent claims show preferred developments of the invention.Preferably, the first picture element has been detected at a first point in time and the second picture element has been detected at a second point in time, the method further comprising the step of:detecting a temporal change in the map element based on the adapted cell elements.An advantage of this embodiment is that the change in the number of point elements in the cell elements can be used to detect a change in the map element. In this case, a history can preferably be generated in order to be able to extract additional information from the map element.More preferably, the method further comprises the step of:determining a degree of the temporal change in the map element.An advantage of this embodiment is that if the change over time exceeds a predetermined limit value, a fault detection can be deduced.More preferably, the method comprises the step of:combining a dot element of the first picture element with a dot element of the second picture element if both dot elements are assigned to the same cell element.An advantage of this embodiment is that the comparativeity of different point elements can be increased. In this case, the combining of the point elements can comprise, in particular, the formation of a checksum or the like.The method further preferably has the steps:determining a delta of a number of point elements in a cell element between the first picture element and the second picture element and / or a further second picture element of the cell element,determining a property of the cell element on the basis of the delta,associating said property with a dynamic object when said delta exceeds a predetermined threshold.An advantage of this embodiment is that it is possible to detect, on the basis of a change in the number of assigned points on a cell element, whether the detected object is a static or a dynamic object. The delta of the number of point elements can in particular be indicative of a change in the number of the respective cell element, in particular between a first image element and a second image element. If the delta is relatively high or higher than a predetermined limit value for the delta, it can be assumed that the object is a dynamic object and not a static object.Preferably, the determination of the first plurality of points comprises the steps of:determining a point element, in particular the first plurality of point elements, based on a depth estimate of a point in the first image element,associating the one point element with the first plurality of point elements when the one point element lies in a predetermined region of a height above a ground, in particular a placement plane, of the mobile working machine and / or when the one point element is free of an outer contour of the mobile working machine and / or when the one point element lies outside a possible roadway of the mobile working machine.An advantage of this embodiment is that only points in the plurality of point elements are taken into account which can represent potential obstacles. For example, a point may be outside a predetermined height above a placement plane / driving plane of the mobile work machine, so that the mobile work machine cannot present an obstacle, since the height of the mobile work machine is lower. It can further preferably be ensured that the point element is not a self-reflection of the mobile working machine, since all outer contours of the mobile working machine are filtered out of the plurality of point elements. Further preferably, all point elements are excluded which are part of a possible roadway of the mobile work machine and thus likewise cannot represent obstacles.Preferably, the step of associating the first plurality of point elements with the predetermined number of cell elements comprises the step of:incrementing a counter of a cell element when the point element has a position that is in a space of the cell element.An advantage of this embodiment is that if sufficient points are associated with a cell element, an object is detected and thus the probability of misdetection is reduced.More preferably, the cell element has a first state and a second state, wherein the method further comprises the step of:assigning the second state to the cell element when the counter reaches a predetermined value.An advantage of this embodiment is that the processability of the information in the cell elements can be significantly increased, since only the states for the respective cell or the like can be considered. For example, the first state may be free and the second state may be busy. For example, the first state may describe a cell which is trafficable and the second state may describe a cell which is not trafficable, because it may have an obstacle, for example.More preferably, the method further comprises the step of:adjusting the predetermined value based on a size of the cell element, a minimum size of the object to be detected, a density of the plurality of point elements and / or a distance between a point element and an origin.An advantage of this embodiment is that, depending on the respective detection situation, the predetermined value for defining the second or occupied state of a cell element can be flexibly adapted, in order to thus further increase the detection accuracy. The adaptation of the predetermined value can take place in particular depending on the size of the cell element or on the basis of a space which depicts the cell element. Further preferably, a minimum variable can be decisive for which predetermined value is to be aligned. Preferably, the density of the plurality of dot elements may decide how to adjust the predetermined value. Further preferably, the distance between a point element and an origin, such as the coordinate system, the camera system and the detected point, can be decisive for how the predetermined value can be adjusted.The method further preferably has the steps:receiving and / or detecting a third image element which comprises a three-dimensional image, wherein the first image element and the third image element differ in their recording orientation,determining a third plurality of dot elements in the third picture element,assigning the third plurality of point elements to the predetermined number of cell elements.An advantage of this embodiment is that the three-dimensional map can be further refined by means of the cell elements by using a plurality of camera sensors or the like in order to be able to write point elements to the respective cell elements. In this case, in particular a first image element can be recorded with a first camera element and a third image element can be recorded with a second camera element. In this case, the orientation of the two camera elements can be different from one another in particular. For example, the first camera element can be a front camera and the second camera element can be a rear camera.An advantage of this embodiment is that it is possible to prevent the mobile working machine from coming too close to possible objects or the like.A further aspect of the invention relates to a computer program which is configured to carry out the method as described above and below.A further aspect of the invention relates to a machine-readable storage medium on which the computer program as described above and below is stored.A further aspect of the invention relates to a mobile working machine, in particular a travel robot, which(s) is configured to carry out steps of the method as described above and below. For this purpose, the mobile work machine can have the storage medium as described above and below.Brief Description of the DrawingsHereinafter, embodiments of the invention will be described in detail with reference to the accompanying drawings. It shows: FIGS. 1 and 2 show a flow chart to illustrate steps of the method according to one specific embodiment. FIG. 3 shows a travel robot according to an embodiment.Embodiments of the InventionPreferably, all the same elements and / or steps are provided with the same reference numerals in all figures.FIG. 1 shows a flow chart to illustrate steps of method 100, according to one embodiment. The method 100 for object detection for a mobile working machine 200 preferably has the steps:receiving and / or acquiring S 1 at least one first image element and one second image element, each of which comprises a three-dimensional image,determining S2 a first plurality of point elements in the first picture element and a second plurality of point elements in the second picture element,mapping S3 of the first plurality of point elements to a predetermined number of cell elements,adapting the cell elements S4, which have the first plurality of point elements, on the basis of the second plurality of point elements,forming S 5 a map element, which comprises a three-dimensional map, based on the adapted cell elements, in order to detect an object based on the formed map element.FIG. 2 shows a flow chart to illustrate steps of method 100 according to one embodiment. The method 100 has the same steps S 1 to S 5 as already explained with reference to FIG. 1. Further preferably, the method 100 has the step of detecting a change over time S 6. Further preferably, the method 100 comprises the step of determining a degree S 7. The method 100 preferably comprises the step of combining S 8 a point element of the first image element with a point element of the second image element. Further preferably, the method 100 has the steps of determining S 9 a delta, determining S 10 a property and assigning S 11 the property. In addition, the method 100 can have the steps of ascertaining S 12 a point element and assigning S 13 the one point element. The method 100 can preferably have the step of incrementing S 14 a counter of a cell element. Further preferably, the method 100 comprises the step of assigning S 15 the second state to the cell element. Further preferably, the method 100 comprises the step of adapting S 16 the predetermined value. The method 100 preferably comprises the steps of receiving and / or detecting S 17 a third image element, ascertaining S 18 a third plurality of point elements, assigning S 19 the third plurality of point elements. Furthermore, the method 100 comprises a step of detecting one or more objects in a surrounding area of the mobile work machine on the basis of the map element formed.FIG. 3 shows a travel robot 200 according to an embodiment. The travel robot 200 may be configured to perform steps of the method 100 as described above and below. Further preferably, the travel robot 200 is configured to store a computer program which is configured to carry out steps of the method 100 as described above and below. The travel robot 200 further preferably has a logic unit 202, which is configured to carry out steps of the method 100 as described above and below. Further preferably, the travel robot 200 has an interface 204 which is configured to establish a connection, in particular to a computer network structure or the like. The travel robot 200 further preferably has a storage medium 206 which is configured to store a computer program which is configured to carry out steps of the method 100 as described above and below.References included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Patent Literature citedUS 11093759 B2

[0004] US 20119163958 A1

[0005] US 10733320 B2

[0006]

Claims

Method (100) for object detection for a mobile work machine (200), comprising the steps of: - receiving and / or detecting (S1) at least one first image element and one second image element, which each comprise a three-dimensional image, - determining (S2) a first plurality of point elements in the first image element and a second plurality of point elements in the second image element, - associating (S3) the first plurality of point elements with a predetermined number of cell elements, - adapting the cell elements (S4), which comprise the first plurality of point elements, on the basis of the second plurality of point elements, - forming (S5) a map element, which comprises a three-dimensional map, based on the adapted cell elements, in order to detect an object based on the formed map element.The method (100) according to claim 1, wherein the first picture element was detected at a first time and the second picture element was detected at a second time, further comprising the step of: - detecting a temporal change (S6) in the map element based on the adjusted cell elements.Method (100) according to claim 2, further comprising: - determining a degree (S7) of the temporal change in the map element.Method (100) according to one of the preceding claims, further comprising the step of: - combining (S8) a point element of the first picture element with a point element of the second picture element if both point elements are assigned to the same cell element.Method (100) according to claim 4, further comprising the steps of: - determining (S9) a delta of a number of point elements in a cell element between the first picture element and the second picture element and / or a further second picture element of the cell element, - determining (S10) a property of the cell element on the basis of the delta, - associating (S11) the property with a dynamic object if the delta exceeds a predetermined limit value.Method (100) according to one of the preceding claims, wherein the determination of the first plurality of points comprises the steps of: - determining (S12) a point element based on a depth estimate of a point in the first image element, - assigning (S13) the one point element to the first plurality of point elements if the one point element lies in a predetermined range of a height above a ground of the mobile work machine and / or if the one point element is free of an outer contour of the mobile work machine and / or if the one point element lies outside a possible roadway of the mobile work machine.The method (100) according to any one of the preceding claims, wherein the associating of the first plurality of point elements with the predetermined number of cell elements comprises the steps of: - incrementing (S14) a counter of a cell element if the point element has a position that lies in a space of the cell element.The method (100) according to claim 7, wherein the cell element has a first state and a second state, further comprising the step of: - associating (S15) the second state with the cell element when the counter reaches a predetermined value.The method (100) according to claim 8, further comprising the step of: - adjusting (S16) the predetermined value based on a size of the cell element, a minimum size of the object to be detected, a density of the plurality of point elements and / or a distance between a point element and an origin.Method (100) according to one of the preceding claims, further comprising the steps of: - receiving and / or acquiring (S17) a third picture element comprising a three-dimensional image, wherein the first picture element and the third picture element differ in their recording orientation, - determining (S18) a third plurality of point elements in the third picture element, - associating (S19) the third plurality of point elements with the predetermined number of cell elements.Computing unit which is configured to carry out the method according to one of the preceding claims.Mobile working machine (200) which is configured to carry out the steps of the method (100) according to one of Claims 1 to 10.A computer program comprising instructions for causing the computing unit according to claim 12 and / or the mobile work machine according to claim 12 to execute the method (100) according to any one of claims 1 to 10.A machine readable storage medium having stored thereon the computer program of claim 13.

Images