SENSOR-BASED OBJECT CHARACTERIZATION AS STATIC OR DYNAMIC

Abstract

Device for detecting a dynamic object, comprising a processor configured to determine a first point density of a first volume around a first point in a first image, wherein the first image is an image of a robot's environment, the image containing image data corresponding to three dimensions; determines one or more second point densities, each second point density of the one or more second point densities being a point density of a second volume around a second point; the second point corresponding to the first point in each of one or more second images, the one or more second images of the environment being an image or images taken prior to the first image, and containing image data corresponding to three dimensions;and classifies the first point as dynamic or static, based on a comparison of the first point density and the one or more second point densities.

Description

Technical area

[0001] Various aspects of this description generally concern the use of image data containing depth information (e.g., a depth image, Light Detection and Ranging (LiDAR) data, etc.) to characterize an object as static or dynamic. background

[0002] Strict safety requirements apply to autonomous mobile robots or stationary robots with autonomously moving limbs (hereinafter referred to as mobile robots), for example, when they are used near people, collaborate directly with people, or have the potential to cause damage to infrastructure. Current efforts to protect people working directly or near robots often rely on Light Detection and Ranging (LiDAR) sensors mounted either on or in the vicinity of the mobile robot. As soon as these sensors detect an object in a safety-relevant proximity to the robot, the robot is prompted to check for safety-relevant or hazardous aspects and, if necessary, take appropriate action, e.g., reduce its speed or even stop completely.

[0003] One challenge with such solutions is that they have difficulty distinguishing between a moving object (e.g., a human) in the immediate vicinity of the mobile robot and a stationary object (e.g., a wall, a table, etc.). Given this shortcoming, existing safety solutions assume a worst-case scenario for each detected object to ensure comprehensive protection of humans or infrastructure. Specifically, the robots assume that the object is moving towards the robot at a defined speed. This defined speed can be based, for example, on the American National Institute of Standards (ANSI) RIA 15.06 safety standard, which may be too conservative for primarily static environments (e.g., environments with few dynamic objects) and therefore relatively impractical.In particular, current practices and / or standards may unduly restrict, e.g., slow down, the approach of mobile robots to static objects. Brief description of the drawings

[0004] In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being generally placed upon illustrating the exemplary principles of the disclosure. In the following description, various exemplary embodiments of the disclosure are described with reference to the following drawings, in which: Fig. 1 and Fig. 2 show considerations for movement near dynamic objects compared to static objects; Fig. 3 shows a security system that detects whether an object is static or dynamic; Fig. 4 shows the occlusion of a point or group of points by a moving object; Fig. 5A and Fig. 5B show a first object obscuring the view of a second object; Fig. 6A, Fig. 6B and Fig. 6C show the change of a volume due to the movement of a device; Fig. 7A, Fig. 7B and Fig. 7C show the change of a volume due to the movement of an object; Fig. 8A and Fig. 8B shows an image of an object in three-dimensional space being converted into an image of an object in two-dimensional space; Fig. Figure 9 shows an occupancy grid with four states; Fig. 10 shows another occupancy grid with fewer states; and Fig. 11 shows a device for detecting a dynamic object. Description

[0005] The following detailed description refers to the accompanying drawings, which show, by way of illustration, exemplary details and embodiments in which aspects of the present disclosure may be practiced.

[0006] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0007] In the drawings, it should be noted that like reference numerals are used to represent the same or similar elements, features and structures unless otherwise indicated.

[0008] The phrase "at least one" and "one or more" may be understood to include a numerical quantity greater than or equal to one (e.g., one, two, three, four, [...], etc.). The phrase "at least one of" with reference to a group of elements may be used herein to mean at least one element from the group consisting of the elements. For example, the phrase "at least one of" with reference to a group of elements may be used herein to mean a selection of the following: one of the listed items, a plurality of one of the listed items, a plurality of individual listed items or a plurality of several individual listed items.

[0009] The words "plurality" and "multiple" in the description and claims expressly refer to a set of more than one. Accordingly, any phrases explicitly using the above-mentioned words (e.g., "plurality of [elements]," "multiple [elements]") that refer to a set of elements expressly refer to more than one of the elements. For example, the phrase "a plurality" may be understood to include a numerical quantity greater than or equal to two (e.g., two, three, four, five, [...], etc.).

[0010] The phrases "group of", "set of", "collection of", "series of", "sequence of", "grouping of", etc., when used in the description and claims, refer to a set equal to or greater than one, i.e., one or more. The phrases "proper subset", "reduced subset", and "smaller subset" refer to a subset of a set that is unequal to the set, illustratively referring to a subset of a set that contains fewer elements than the set.

[0011] The term "data" as used herein may be understood to include information in any suitable analog or digital form, provided, for example, as a file, a portion of a file, a set of files, a signal or stream, a portion of a signal or stream, a set of signals or streams, and the like. Furthermore, the term "data" may also be used to mean a reference to information, for example, in the form of a pointer. However, the term "data" is not limited to the aforementioned examples and may take various forms and represent any information as understood in the art.

[0012] The terms "processor" or "controller," for example, as used herein, may be understood as any type of technological entity that allows for the handling of data. The data may be processed according to one or more specific functions performed by the processor or controller. Furthermore, a processor or controller, as used herein, may be understood as any type of circuit, e.g., any type of analog or digital circuit.A processor or controller may thus be or include an analog circuit, a digital circuit, a mixed-signal circuit, a logic circuit, a processor, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), an integrated circuit, an application-specific integrated circuit (ASIC), etc., or any combination thereof. Any other type of implementation of the respective functions, which are described in more detail below, may also be considered a processor, controller, or logic circuit.It is to be understood that any two (or more) of the processors, controllers, or logic circuits described in detail herein may be implemented as a single entity with equivalent functionality or the like, and conversely, that any individual processor, controller, or logic circuit embodied herein may be implemented as two (or more) separate entities with equivalent functionality or the like.

[0013] As used herein, "memory" means a storage element or computer-readable medium (e.g., a non-transitory computer-readable medium) in which data or information can be stored for retrieval. References herein to "memory" may thus be construed to refer to volatile or non-volatile memory, including, but not limited to, random access memory (RAM), read-only memory (ROM), flash memory, semiconductor memory, magnetic tape, a hard disk drive, an optical drive, 3D-XPoint™, or any combination thereof. Also included within the term memory herein are, but are not limited to, registers, shift registers, processor registers, data buffers, etc. Memory may be local memory, where the memory is electrically connected to a processor that reads data from and / or stores data to the memory.Alternatively or additionally, memory may be remote memory, such as memory accessed by a processor via a communications protocol (e.g., via an internet protocol, a wireless communications protocol, etc.). Remote memory may include cloud storage. The term "software" refers to any type of executable instructions, including firmware.

[0014] Unless explicitly stated, the term "transmit" includes both direct transmission (point-to-point) and indirect transmission (via one or more intermediate points). Similarly, the term "receive" includes both direct and indirect reception. Furthermore, the terms "transmit," "receive," "communicate," and other similar expressions include both physical transmission (e.g., the transmission of radio signals) and logical transmission (e.g., the transmission of digital data over a software-level logical connection).For example, a processor or controller may transmit or receive data with another processor or controller in the form of radio signals over a software-level connection, with the physical transmission and reception being handled by radio-layer components such as RF transceivers and antennas, and the logical transmission and reception over the software-level connection being performed by the processors or controllers. The term "communicate" includes transmitting and / or receiving, i.e., unidirectional or bidirectional communication in one or both of the inbound and outbound directions. The term "compute" includes both "direct" computations via a mathematical expression / formula / relationship and "indirect" computations via lookup or hash tables and other array indexing or searching operations.

[0015] Unless explicitly stated, the term "image" includes image data containing both 2D and 3D information. Such "images" according to this definition may include, among others, depth information images (e.g., distance information images) or point clouds (e.g., 3D point clouds). This particularly applies to images containing image data corresponding to three or four (3D plus time) dimensions.

[0016] Mobile robots generally have difficulty distinguishing between dynamic and stationary objects. This applies both to LiDAR-based robots (which are unable to distinguish between static and dynamic objects based on a single scan) and to robots relying on certain modern safety systems (where such a distinction between static and dynamic objects is not part of the safety concept / algorithms). Since mobile robots do not determine whether the detected objects are dynamic or static, and to ensure safe operation under all conditions, current mobile robot safety systems assume that all objects are potentially dynamic and therefore potentially moving towards the robot at a certain speed (worst case).

[0017] For example, ANSI / RIA 15.06 states that a human should be expected to be able to move toward a mobile device at 1.5 meters / second. Assuming a hypothetical robot stopping time of τ=0.5 seconds and a stopping distance of d_robot=1 meter, the required safety distance is 1.75 meters (robot stopping distance + human travel during the stopping time). Therefore, the robot must maintain an additional (unnecessary) safety distance of 0.75 meters for each stationary object (e.g., a wall). Depending on the robot's environment, this may significantly impair the robot's usability or efficiency, or require additional, expensive countermeasures, such as restricting human access to certain areas.

[0018] Fig. 1 and Fig. 2 illustrate the above considerations regarding dynamic and static objects. In Fig. 1, a robot 102 approaches a static obstacle 104. The distance between the robot 102 and the static obstacle 104 is represented by 106. Based on the values ​​described in the previous section, the robot 102 has a braking distance of 1 meter and can therefore approach any static object to within 1 meter while still maintaining an appropriate safety distance to avoid collisions. Fig. 2, however, the robot 202 approaches a human 204. Since the human 204 is a dynamic object or is assumed to be such, an additional safety distance is required. That is, the Fig. 1 as 106 is represented here as 206. In addition, the distance traveled by the human 204 during the stopping time of the robot 202 is represented as 208. As can be seen from this figure, the safety distance of the robot from a dynamic object is greater than the safety distance of the robot from a static object.

[0019] Many approaches that could address the problem of static / dynamic object segmentation ignore safety requirements or do not adequately consider safety requirements. Such approaches may involve learning-based solutions such as object detectors (e.g., Yolo, PointPillars, etc.), which are not approved by safety certification bodies. Object trackers relying on Kalman filters are also not applicable, as these require, for example, tracking 3D bounding boxes, which might not be available within the safety path (such 3D bounding boxes are more suitable for detecting if something is too close than how the object is shaped), or providing an estimate of object positions and orientation. 3D bounding boxes typically reduce efficiency because the actual distance to actual objects may be greater than to the bounding box.Finally, there are dynamic occupancy grid approaches that use, for example, particle filters to identify moving scene elements; however, these are extremely computationally intensive and therefore may be difficult to implement for performing high-frequency safety tasks in mobile robots.

[0020] To mitigate the aforementioned shortcomings of modern safety systems for mobile robots, a solution or system is described that can distinguish between dynamic and static environmental elements. For this purpose, the system can use a high-resolution 3D depth sensor (a depth camera or a modern LiDAR sensor) as an input source. Alternative input devices can be conventional image sensors (e.g., spaced-apart cameras), for which one or more photogrammetry methods can be implemented to obtain 3D data from the multiple images. The system can then convert the data (if necessary, as with the depth sensor data) into a 3D point cloud. The system can then compare the density of this cloud for any region with the regional density of point clouds from previous images.Any significant increase / decrease in density can be interpreted as an indication that something has moved in the scene. In a subsequent analysis, the dynamic elements can be identified and separated from the static elements (which can be done on a 2D or 3D occupancy grid). Finally, the system can adjust the required safety distances, allowing the robot to move closer to static scene elements (e.g., walls) while maintaining larger safety distances from dynamic elements. The required safety distance can also be derived from characteristics or properties of the objects, e.g., a maximum acceleration or maximum speed of a mobile robot.

[0021] Fig. Figure 3 shows a safety system that detects whether an object is static or dynamic. In this system, the robot 302 is equipped with one or more sensors to obtain information about the robot's environment. As described above, these sensors may be a LiDAR device, a depth camera, or depth information obtained from 2D images using one or more photogrammetry techniques. The acquisition or generation of depth information is represented as 306. At 308, the system converts the depth information into a 3D point cloud. At 310, the 3D point cloud is further processed to remove traversable areas from the 3D point cloud (e.g., areas that are or are intended to be traversed by the robot). At 312, the resulting point cloud, from which the traversable areas have been removed, is projected onto a grid.In the next line, multiple point clouds from 310 are aligned using localization information 320 (GPS information, triangulation information, or other information for localizing the robot 302). That is, a current point cloud from 310 is aligned with one or more previous point clouds from 310 using the localization information 320. For a given point on the current 3D point cloud, the system compares the point density in a volume (e.g., a three-dimensional space) around the given point with a point density of the volume around a corresponding point on the one or more previous 3D point clouds. If the point density changes in the current volume and the previous volume, this indicates movement within that volume, or otherwise, that the volume corresponds to a dynamic object.Using this technique, the system can label one or more points of the current 3D point cloud as dynamic or static in step 326. The system can then convert the labeled 3D point cloud into a grid in which regions of the grid are labeled as corresponding to a static or dynamic object. This grid can then be received by the safety controller 314. This grid from step 328 is largely similar to the grid from step 312, except that the grid from step 328 includes a label as static or dynamic, whereas this would not be possible in the grid from step 312. Using the labeled grid, the safety controller 314 determines a safe behavior (e.g., a safe stopping distance, a stopping routine, etc.) depending on whether the respective object in the vicinity of the robot is detected as dynamic or static.

[0022] The conversion of 3D image data into a 3D point cloud is known and can be achieved using a variety of methods. Those skilled in the art will understand that various such methods are available, and that any such method for converting 3D information into a point cloud can be used without limitation.

[0023] While not strictly required for the methods disclosed herein, filtering the 3D point cloud to remove traversable areas, as illustrated in step 310, can simplify the categorization of objects as dynamic or static, as the resulting image will only contain points related to obstacles and not to paths along which the robot can move. Various methods for removing motion paths from the 3D point cloud are known, and those skilled in the art will understand that any of these methods may be used for the purposes of this disclosure.

[0024] Projecting the image onto a grid (also known as grid extraction), as illustrated in steps 312 and 328, can optionally be achieved by ignoring the Z value of each point within the 3D point cloud. Ignoring the Z value results in a two-dimensional image, which can significantly simplify subsequent processing of the image data.

[0025] The methods described here may require the system to determine whether a cell (e.g., a cell of the grid during or after grid extraction) is occupied or unoccupied. A cell in the grid can be defined as occupied if a number of projected points per cell lies outside a range. Similarly, the cell in the grid can be defined as unoccupied if the number of projected points per cell lies within the range. Alternatively, this can be done using a threshold, so that the cell is defined as occupied if the number of points within the cell exceeds the threshold and the cell is defined as unoccupied if the number of points within the cell is below the threshold.

[0026] If the occupied cell is within the robot's specified safety zone, the robot must decelerate or stop. However, the safety zone itself depends on whether the cell is defined for a dynamic or static object.

[0027] Regarding the alignment of the point clouds in step 324, it may be necessary to analyze the environment across consecutive images (e.g., sometimes multiple consecutive images) to identify an object as dynamic or static. For such an analysis, all point clouds must be located in the same coordinate system with the same reference point. This can be, for example, a global coordinate system (e.g., a target coordinate system) similar to a map frame, such as that used in a SLAM (Simultaneous Localization and Mapping) approach for localization. This can be achieved, for example, using the robot's position sensor. For each time point corresponding to a particular point cloud, a position corresponding to the robot's position sensor is recorded. The point clouds are then resolved relative to each other.

[0028] It should be noted that this method can use both the global coordinate system and the robot's position (e.g., via the localization sensor). The global coordinate system is static and therefore does not change depending on the robot's movement. To align the images (e.g., the point clouds) relative to the global coordinate system, it may be necessary to determine the robot's position relative to the global coordinate system at each time point corresponding to a 3D point cloud. Therefore, if a predetermined global coordinate system is available, the robot's position can be determined based on a localization sensor relative to the predetermined global coordinate system, and this relationship can be used to align the 3D point clouds.

[0029] In some configurations, the robot may be configured with multiple image sensors, such as multiple LiDAR devices or even multiple depth cameras or multiple image sensors from which depth information is obtained using one or more photogrammetry techniques. In such configurations, the system can fuse the various resulting point clouds as described above, but in such a way that for each time step, exactly one uniform (and filtered) point cloud is generated for the given global coordinate system.

[0030] In step 326, the system classifies points (e.g., all points, some points, or less than all points) within a given point cloud as either static or dynamic. Regions in three-dimensional space where measurements are taken that cannot be explained by previously obtained point clouds are assumed to represent either newly identified areas or dynamic objects.

[0031] For this analysis, the system can limit its investigation to the points (e.g., all points, some points, or at least less than all points) of a specific point cloud. For example, the point cloud can be the point cloud of 310 / 324, with the drivable areas omitted. The analysis can be performed point by point for a specific point p of the current point cloud P 0be performed, i.e., for a given point p, a radius in three-dimensional space (e.g., a volume) around the point p is evaluated in the current point cloud and in one or more previous point clouds. For example, if F is the set of sensor data or point clouds, then a simple definition of the state s of p is given by: s(p)={static, if 1|F|∑f∈F|rf(p)−r0(p)|<δotherwise dynamic

[0032] That is, if the average of the differences between the number p of points within a volume containing a reference point on one or more previous point clouds r f (p) and the number of points within a volume that has a corresponding reference point on a current point cloud r o(p) is smaller than a predetermined threshold δ, then the point in the current point cloud is said to be static, otherwise the point in the current point cloud is said to be dynamic.

[0033] Otherwise, a point is considered part of a static object, meaning that the density around the point does not change, or at least does not change significantly, between the current point cloud and one or more newer point clouds. However, if the density around the point changes significantly between the current point cloud and one or more newer point clouds, the point is said to be dynamic.

[0034] Unfortunately, this approach can have certain undesirable weaknesses in practice. For example, a moving robot may perceive objects as larger or smaller depending on how close it is to the object, which may need to be compensated for. Furthermore, a particular object may be temporarily obscured by one or more other objects and / or suddenly reappear in the robot's line of sight once the obscuring object is no longer in the way. This can easily erroneously result in a static object being temporarily marked as dynamic. These potential weaknesses are addressed below with one or more optional techniques or features to eliminate them.

[0035] In a first optional configuration, the system can take into account the point visibility of a moving robot. This means that a certain point (e.g., a point on a wall) may be visible in one image due to the movement of the robot (or a corresponding object), but obscured in another image. In such a case, equation (1) would indicate a dynamic object, since the point density for a volume around a certain point would change between a current image and a previous image. However, this might be incorrect, since a point on a wall, for example, may always remain static even if the moving robot detects a change in the point density around the point on the wall (e.g., due to an object temporarily obscuring the point on the wall). To solve this problem, the visibility of the point should be taken into account.

[0036] A particular point may be invisible to the robot, either because it lies outside the field of view of the sensor (e.g., image sensor, LiDAR, etc.) or because it is occluded by a closer object. To determine whether either case applies, the system can analyze the point cloud P for each sensor position within F 0 onto the sensor frame (e.g., a two-dimensional image plane). An underlying assumption is that a point that was outside the sensor frame for a particular shot was not visible to the robot in that image. However, if a point lies within the sensor frame, e.g., on pixel (i,j), the processor can compare an estimated distance with the distance measured on the pixel for that image. If the estimated distance is greater than the measured distance, the object was obscured by another object in that particular image. This results in a visibility function that can be described as follows: v(p,f)={invisible,p∉image plane for image planes fvisible, measured distance <gescha¨tzter Abstandsonst sichtbar

[0037] Otherwise, a given point p is invisible to the robot if it is not represented in this sensor frame (if it is outside the sensor frame) or if the measured distance to point p is less than an estimated distance to point P (meaning, for example, that another object is between the robot and the expected object at point p).

[0038] Fig. Figure 4 shows the occlusion of a point or group of points by a moving object. In this figure, a surface is divided into three sections: section a 402, section b 403, and section c 404. The robot's image sensors 406 are aligned with the surface containing sections a, b, and c. An object 408 moves from left to right between the surface and the image sensors 406. On the right, a graph shows the number of points in sections a, b, and c that are visible to the robot. Initially, all points in section a 402, section b 403, and section c 404 are visible to the image sensors 406. As the object 408 moves from left to right, the object begins to occlude the image sensor 406's view of section a 402, thereby reducing the number of points visible to the image sensor.Gradually, the moving object 408 obscures less of section a 402 and begins to obscure section b 403. In this case, the number of points visible for area a increases, while the number of points visible for area b decreases. Gradually, the moving object 408 obscures less of section b 403 and more of section c 404. Accordingly, the number of visible points for section b increases and the number of visible points for section c decreases.

[0039] To correct this phenomenon, the state function can be updated so that the amount of sensor data to be analyzed F is replaced by F', where: F'={f|f∈F and v(p,f)==visible}

[0040] Since this function can lead to an empty set, a point may only be considered static if |F'| > m where m is a user-defined threshold. In other words, s(p)={static, if |F'|>m∧1|F'|∑f∈F'|rf(p)−r0(p)|<δ for F'={f|F∧v(p,f)==visible}otherwise dynamic

[0041] Fig. 5A shows a first object 502 obscuring the view of a second object 504. Due to the arrangement of the first object 502 with respect to the image sensor and the second object 504, the first object 502 and the second object 504 appear to the image sensor as in Fig. 5B, where only a portion of the second object 504 is visible, since most of the second object 504 is obscured by the first object 502. Since the images captured with the corresponding image sensors, depth cameras, LiDAR, etc., contain depth information, the system can determine the depth information of each visible portion of the second object 504. In this way, the system can determine a relative distance between the image sensors (depth cameras, LiDAR, etc.) and the second object 504. Using simple geometric principles, the robot can also estimate the distance between itself and each obscured portion of the second object 504. Since the first object 502 obscures a large portion of the surface of the second object 504, much of the depth information associated with the direction of the second object 504 corresponds to the distance between the image sensors and the first object 502.Since the first object 502 is closer to the image sensors than the second object 504, the robot can determine whether the depth information associated with a particular point is less than the estimated distance between the robot (e.g., the image sensors) and the second device 504 (this estimated distance may also be referred to as the projected distance). If the distance associated with the depth information is less than the estimated distance / projected distance, it can be assumed that the point in question (the relevant part of the view of the second object 504) is occluded by the first object 502.

[0042] In an optional second configuration, the data to be analyzed can be modified based on changes in the distance between the robot and the corresponding object. As the robot approaches or moves away from a specific (stationary) object, the number of corresponding detection points changes (e.g., increases or decreases) because the object appears larger or smaller in the robot's sensor image. Under certain circumstances, this can lead to a stationary object being falsely detected as dynamic, as the change in the object's apparent size can lead to a change in the corresponding point density.

[0043] In the Fig. 6A, Fig. 6B and Fig. 6C shows changes in the dot volume according to this optional aspect of the description. In Fig. 6A, an object 602 is viewed by one or more image sensors of the robot 604, and a volume corresponding to 606a (a two-dimensional region is shown, but a three-dimensional volume is intended) is evaluated according to the methods described herein. As the robot moves (in this case, toward the object 602), changes may be made, as shown in the Fig. 6B and Fig. 6C. In Fig. 6B, the robot further evaluates a volume corresponding to a portion of the surface 602. In this case, it can be seen that the distance between the robot and the surface 602 in Fig. 6B is smaller than in Fig. 6A. Since the robot is now closer to surface 602, the surface appears larger in the robot's sensor field. To compensate for this, the volume to be examined 606b is reduced in size. The result is that the same number (or a comparable number) of points is present in 606b as in 606a, since 606b is larger in the robot's field of view. Similarly, in Fig. 6C, the robot has moved even closer to the object 602, so that the volume 606c to be examined is even smaller, so that the same number of points are present in each of 606a, 606b, and 606c, taking into account the relative change in size of each of 606a, 606b, and 606c from the robot's perspective. In this way, the object is classified as static despite the robot's movement (and therefore despite the otherwise ever-increasing volume).

[0044] Fig. 7A, Fig. 7B and Fig. 7C illustrate this procedure for a dynamic object. In Fig. 7A, the image sensors of robot 702 capture an image of an object 704, wherein the image is processed according to the steps described herein and a corresponding volume 706 is analyzed for point density. In Fig. 7A, volume 706 overlaps object 704 only slightly, so that only a few points correspond to object 704. In Fig. 7B, both the robot and the object 704 have moved, and as the robot has moved closer to the object 704, the size of the volume 706 has decreased to compensate for the otherwise larger-appearing object. The change in the size of the volume compensates for the movement (the change in position) of the robot compared to its previous position in Fig. 7A. However, since the object 704 is dynamic and also moves closer to the robot, a larger number of points corresponding to the object 704 are detected in the volume 706. To further illustrate this process, Fig. 7C shows that the robot moves even closer to the object 704, and that this reduced distance between the robot and the object 704 results from both the movement of the robot and the movement of the object 704. The size of the volume 706 is further reduced due to the movements of the robot. However, due to the movements of the object, the number of points detected within the volume 706 is greater than in Fig. 7B. In this way, object 704 is classified as dynamic.

[0045] In other words, as the robot approaches or moves away from a particular (stationary) object, the number of detected points within the volume will naturally increase (or decrease) as the object appears larger (or smaller) in the sensor image. This must be taken into account, as a static object could otherwise be classified as dynamic simply because it appears larger (e.g., because it results in a higher point density). Therefore, the system can rely on the projection of the sensor field, with the search radius r f (which corresponds to a size of the volume, a radius of the volume) for the frame (time step) f is defined as follows: rf(p)=df(p)×R

[0046] In this formula, R can be a fixed (user-defined or predefined) search radius in points (e.g., on the sensor frame), and d(p) can be the distance of point p to the sensor plane. Consequently, the search radius can be larger for more distant points and smaller for closer points. This is shown in the Fig. 7A, Fig. 7B and Fig. 7C, which show that closer objects use a smaller effective search radius in three dimensions, while more distant objects use a larger radius.

[0047] Although those skilled in the art will know that this can be implemented in various ways, one possible method for implementing this procedure is shown in the following pseudocode:

[0048] According to a third optional implementation, the process for detecting motion within an object to classify the object as dynamic or static can be simplified by performing a neighborhood search in a two-dimensional space. A radius search (a search within a volume) within a three-dimensional point cloud is computationally intensive. The associated computational cost can be O(n 2 ), where n is the number of points. To reduce this complexity, it may be desirable to perform the neighborhood search in the two-dimensional image space, since every neighbor in 3 dimensions is also a neighbor in 2 dimensions. This is at least the case in the Fig. 8A and Fig. 8B shown.

[0049] In Fig. 8A, a representation of an object in three-dimensional space 802 is converted into a representation of an object in two-dimensional space 804, e.g., by ignoring the Z-access information within the point cloud. A neighborhood search, as indicated by the dashed circle, leads to comparable results in a two-dimensional environment. In other words, instead of searching a three-dimensional volume, only a two-dimensional area is searched to determine the point density. The two-dimensional area search is less computationally expensive than the three-dimensional volume search. Similarly, Fig. 8B shows how a three-dimensional image 806 is converted into a two-dimensional image 808 and the neighborhood search of the three-dimensional volume (represented by the dashed circle on the right) is performed as a two-dimensional region (represented by the dashed circle on the left).

[0050] In a fourth optional implementation, the system can be configured to use an Unknown state. In this description, only two states have been used so far: static or dynamic. However, to enable a more meaningful and safe state transition, a third state (e.g., an Unknown state) can be introduced. In this way, a newly discovered point (or region or grouping of points) can be marked as unknown (e.g., it is known whether the object is dynamic or static) until enough measurements are available to reliably classify the point as static or dynamic. This allows for a faster transition (e.g., after a few images) from an Unknown state to a static or dynamic state, without the need to enable a similarly fast transition between dynamic and static states.In this context, it should be noted that rapid transitions between dynamic and static states may generally be undesirable, as such rapid transitions may result in a human worker who stops briefly being classified as static, whereas a longer classification as dynamic may be considered safer and / or necessary to meet certain safety regulations. Furthermore, without the Unknown state, all newly discovered points would have to be classified as dynamic, leading to the problem that a long transition from dynamic to static would eventually result in an environment where everything is labeled as dynamic.

[0051] This configuration could be solved as follows: s(p)={static, if |F'|>m∧1|F'|∑f∈F'|rf(p)−r0(p)|<δ for F'={f|F∧v(p,f)==visible}unknown, if |F'|≤motherwise dynamic

[0052] After creating the labeled point cloud, the system can create an occupancy grid as shown in Fig. 3, item 328. In this way, the point cloud can be projected onto the grid. However, since the principles and methods described here result in a labeled point cloud, the labels must now be taken into account. For example, a grid cell can be labeled dynamic if the number of unknown or dynamic points within the grid is above a threshold. Conversely, the grid cell can be labeled static if the number of unknown or dynamic points is below the threshold. Of course, the lower this threshold, the more safely the robot operates. Conversely, the higher the threshold, the smaller the resulting safety margins.

[0053] Fig. Figure 9 shows an occupancy grid with 4 states (dynamic 902, free 904, static 906, and unknown 908), as described above. Thus, cells that are occupied and dynamic (e.g., with at least k unknown or dynamic points) are denoted as 902. The free cells (known to be free and / or containing no corresponding point measurements) are denoted as 904. The occupied and static cells (with statically marked points) are denoted as 906. The unknown cells are denoted as 908.

[0054] As can be seen from the figure, grid cells containing only static elements are correctly identified as static, while cells corresponding to a human worker are correctly labeled as dynamic. For safety purposes, it is important to know where a static or dynamic cell or object is located and what the required safety distance to the respective cell is. In this respect, not all static cells are treated the same way. For example, a wall cannot be walked on or driven over, whereas a small object (e.g., a package, a tool, etc.) can be crossed or climbed over by a human. Since there is an unknown space behind every static object, some static objects with a low height may require additional steps. For example, the robot may move very close to a wall with an opening (e.g.,an open door) where a dynamic object may suddenly appear.

[0055] To overcome these challenges, each occupied cell can optionally be assigned the height of a detected obstacle (e.g., from the 3D point cloud data). This essentially makes the resulting grid a 2.5D grid. A 3D grid can also be used. Furthermore, all cells that can potentially be reached during the robot's dwell time, whether by visible or potentially obscured dynamic objects, can be marked as dangerous. This is Fig. 10, where unshaded (white) cells are passable by the robot, black-shaded cells represent objects or obstacles, and gray-shaded cells indicate danger zones where the cell is potentially accessible to both the robot and humans during the robot's dwell time. Potentially obscured dynamic objects are also displayed as danger zones to comply with safety standards.

[0056] As from Fig. As can be seen in Figure 10, the robot can travel close to walls but must maintain a significant distance from the human and the package (since the package could be traversed by an obscured dynamic object). The resulting grid can then be provided to the robot's motion limiter, which evaluates the proximity of each occupied cell (whether static or dynamic) to the robot. Static cells may require a smaller safety distance than dynamic cells. If a violation of the safety distance is detected (i.e., the cell is too close to the robot), a safety action (e.g., braking, deceleration) can be enforced.

[0057] Fig.11 shows an apparatus 1100 for detecting a dynamic object, comprising a processor 1102 configured to determine a first point density of a first volume around a first point in a first image, wherein the first image is an image of an environment of a robot, the image containing image data corresponding to three dimensions; determine one or more second point densities, wherein each second point density of the one or more second point densities is a point density of a second volume around a second point; wherein the second point corresponds to the first point in each of one or more second images, wherein the one or more second images of the environment is or are one or more images taken before the first image and containing image data corresponding to the three dimensions;and classifies the first point as dynamic or static based on a comparison of the first point density and the one or more second point densities;

[0058] In an optional configuration, the dynamic object detection device may further include one or more image sensors 1104 configured to generate a plurality of images or depth images. In this way, the processor 1102 may be further configured to generate the point cloud images by resolving the plurality of images or depth images with a position of the one or more image sensors when the plurality of images or depth images are acquired. The one or more image sensors 1104 may be a stereo camera or a depth camera, a LiDAR device, or another device capable of generating image data related to three dimensions. Additionally or alternatively, the one or more image sensors 1104 may be two-dimensional camera sensors configured to generate two-dimensional images.In this case, the one or more processors 1102 may be further configured to generate the point cloud images by resolving the plurality of two-dimensional images using one or more photogrammetry techniques.

[0059] In an optional configuration, the processor may be configured to determine first depth information of the first point within the first capture (e.g., in the first depth information image) and second depth information for each second point in the one or more second images (e.g., one or more second captures). The processor may further be configured to generate (or alternatively select) a modified image set using the first depth information and the second depth information as an image set including the one or more second images in which the first depth information corresponds to a greater depth than a depth of the second depth information of the corresponding second image.In this way, classifying the first point as dynamic or static based on the comparison of the first point density and the one or more second point densities may include classifying the first point as dynamic or static based on a comparison of the first point density and the one or more second point densities of the modified second image set.

[0060] In another optional configuration, the processor may also be configured to modify the second volume based on a comparison of the depth information of the first point and the depth information of the second point. In this way, the volume to be analyzed can be enlarged or reduced based on the depth information. In this way, the volume to be analyzed can be modified based on the relative movement of the dynamic object detection device (optionally configured as a robot) towards or away from the object. This adjustment helps correct for the fact that an object closer to the robot occupies a larger portion of the robot's field of view and therefore generates more corresponding points in the point cloud. Therefore, the volume to be analyzed can be reduced to correct for this phenomenon.Conversely, the volume can be increased as the relative distance between the device and the object increases to accommodate the change in size within the robot's field of view.

[0061] In this way, the change of the volume surrounding the second point based on the depth information of the first point compared to the depth information of the second point may include an enlargement of the second volume when the first depth information corresponds to a depth that is less than a depth of the second depth information, and a reduction of the second volume when the first depth information corresponds to a depth that is greater than a depth of the second depth information.

[0062] This document makes several references to the measurement of a volume surrounding a point. The term "volume" as used herein is intended to describe a three-dimensional space. The volume described here can optionally be spherical. A spherical volume can alternatively be understood as a three-dimensional region defined by all points with a predetermined radius from a central point. Although a spherical volume is referred to several times here, the volume to be measured can alternatively be defined by shapes other than a sphere, and the spherical volume is used here merely for convenience.

[0063] In an alternative configuration, a third category, "unknown," may be used, allowing the processor to optionally be configured to characterize a point as static, dynamic, or unknown. The introduction of this third category (e.g., the "unknown" category) may simplify future calculations and improve the overall operation of the device. It may be undesirable to allow a rapid transition from static to dynamic or from dynamic to static; however, this can be somewhat mitigated by requiring a slightly longer transition between static and dynamic, but allowing a somewhat rapid transition between static and unknown, and between dynamic and unknown. One effect of using the "unknown" category is that newly discovered points may be classified as unknown rather than dynamic.This avoids having to classify all newly discovered points as dynamic, as such a classification would impose significant and possibly even impractical restrictions on the movement of the device.

[0064] In this disclosure, the captured images with depth information are described as a point cloud that is rendered. The point cloud may contain a discrete set of data points in space. These data points, in combination with each other, may represent one or more three-dimensional shapes or objects. The points may be positioned or rendered within a set of Cartesian coordinates along three axes.

[0065] In an optional configuration, the device may be configured to operate in one or more modes of operation depending on whether an object is classified as static or dynamic. In particular, the processor may be configured to operate in a first mode of operation when the distance between the robot and an object corresponding to a static point is within a predetermined range and to operate in a second mode of operation when the distance between the robot and the object corresponding to a dynamic point is within the predetermined range. The first mode of operation may include the processor not sending a command to slow down or stop the robot; and the second mode of operation may include the processor sending a command to slow down or stop the robot.

[0066] The dynamic object detection device described herein can optionally be configured as an autonomous robot. In this way, the autonomous robot can also include one or more positioning sensors for determining the position of the autonomous robot relative to one or more reference points and a motion system for initiating movement or locomotion of the robot.

[0067] This description describes, among other things, a comparison between a first volume and a second volume. The first volume and the second volume may alternatively be understood as a first area and a second area or a first region and a second region.

[0068] In one possible configuration, the center of the first volume has the same x, y, and z coordinates as the center of the second volume. In this configuration, a first point density around the center of the first volume is compared with a second point density around the center of the second volume, where the first center and the second center have the same position but in different images. As described above, the size of the first volume can be equal to the size of the second volume. Alternatively, the size of the first volume can be larger or smaller than the size of the second volume.

[0069] In an alternative configuration, the center of the first volume has different x, y, or z coordinates than the center of the second volume. In this configuration, for example, the center of the first volume is adjusted based on a change in location or movement of the sensor (e.g., the device performing this procedure) or even based on a change in location or movement of an object. At this point, again, the size of the first volume can be equal to the size of the second volume, or the size of the first volume can be larger or smaller than the size of the second volume.

[0070] Further aspects of this description are disclosed by the following examples:

[0071] Example 1 is an apparatus for detecting a dynamic object, comprising a processor configured to determine a first point density of a first volume around a first point in a first image, the first image being an image of an environment of a robot, the image including image data corresponding to three dimensions; determine one or more second point densities, each second point density of the one or more second point densities being a point density of a second volume around a second point; the second point corresponding to the first point in each of one or more second images, the one or more second images of the environment being one or more images taken prior to the first image and including image data corresponding to the three dimensions;and classifies the first point as dynamic or static based on a comparison of the first point density and the one or more second point densities;

[0072] In Example 2, the subject matter of Example 1 can optionally include the processor being further configured to: determine first depth information of the first point within the first image; determine second depth information for each second point in the one or more second images; and a modified image set as an image set including the one or more second images, wherein the first depth information corresponds to a greater depth than a depth of the second depth information of the corresponding second image; wherein classifying the first point as dynamic or static based on comparing the first point density and the one or more second point densities comprises classifying the first point as dynamic or static based on a comparison of the first point density and the one or more second point densities of the modified second image set.

[0073] In Example 3, the subject matter of any of Examples 1 or 2 can optionally include the processor being further configured to change the second point density based on a comparison of the depth information of the first point and the depth information of the second point.

[0074] In Example 4, the subject matter of Example 3 can optionally include that changing the point density surrounding the second point based on depth information of the first point compared to depth information of the second point comprises increasing the second point density when the first depth information corresponds to a depth less than a depth of the second depth information, and decreasing the second point density when the first depth information corresponds to a depth greater than a depth of the second depth information.

[0075] In Example 5, the article of any of Examples 1 to 4 can optionally include the first volume and the second volume being spherical.

[0076] In Example 6, the subject matter of any of Examples 1 to 5 can optionally include the processor being further configured to classify the first point as dynamic, static, or unknown based on the comparison of the first point density and the one or more second point densities.

[0077] In Example 7, the subject matter of any of Examples 1 to 6 can optionally include the first image of the environment and the one or more second images of the environment being configured as point cloud images.

[0078] In Example 8, the subject matter of any of Examples 1 to 7 can optionally include the device further comprising one or more image sensors configured to generate a plurality of images or depth images; wherein the processor is further configured to generate the point cloud images by resolving the plurality of images or depth images with a position of the one or more image sensors when the plurality of images or depth images were captured.

[0079] In Example 9, the subject matter of Example 8 can optionally include the one or more image sensors comprising a stereo camera or a depth camera.

[0080] In Example 10, the subject matter of Example 8 can optionally include that the image generator is a LiDAR, Light Detection and Ranging, device.

[0081] In Example 11, the subject matter of Example 8 can optionally include the device further comprising one or more image sensors configured to generate a plurality of images; wherein the processor is further configured to generate the point cloud images by resolving the plurality of images using one or more photogrammetry methods. In an optional configuration, the image sensor or sensors can be depth or distance image sensors.

[0082] In Example 12, the subject matter of any of Examples 1 to 11 can optionally include the processor being further configured to generate a modified data set by removing from the first image and the one or more second images data corresponding to a surface traversable by the dynamic object detection device; and wherein classifying the first point as dynamic or static comprises classifying the first point as dynamic or static based on a comparison of the density of the first point and the density of the one or more second points within the modified data set.

[0083] In Example 13, the subject matter of any of Examples 1 to 12 can optionally include the processor being further configured to generate a two-dimensional grid of the environment and to label portions of the two-dimensional grid as dynamic or static based on the classification of the first point as dynamic or static based on the comparison of the first point density and the one or more second point densities.

[0084] In Example 14, the subject matter of any of Examples 1 to 13 can optionally include the processor being configured to operate in a first mode of operation when a distance between the robot and an object corresponding to a static point is within a predetermined range, and to operate in a second mode of operation when the distance between the robot and the object corresponding to a dynamic point is within the predetermined range.

[0085] In Example 15, the subject matter of Example 14 can optionally include wherein the first mode of operation comprises the processor not sending a command to decelerate or stop the robot; and wherein the second mode of operation comprises the processor sending a command to decelerate or stop the robot.

[0086] In Example 16, the subject matter of any of Examples 1 to 15 can optionally include that the device for detecting dynamic objects is configured as an autonomous robot.

[0087] Example 17 is a method for detecting a dynamic object, comprising: determining a first point density of a first volume around a first point in a first image, the first image being an image of an environment of a robot, the image including image data corresponding to three dimensions; determining one or more second point densities, each second point density of the one or more second point densities being a point density of a second volume around a second point; the second point corresponding to the first point in each of one or more second images, the one or more second images of the environment being one or more images taken before the first image and including image data corresponding to the three dimensions; and classifying the first point as dynamic or static based on a comparison of the first point density and the one or more second point densities.

[0088] In Example 18, the subject matter of Example 17 can optionally include the method further comprising: determining first depth information of the first point within the first image; determining second depth information for each second point in the one or more second images; and determining a modified image set as an image set comprising the one or more second images, in which the first depth information corresponds to a greater depth than a depth of the second depth information of the corresponding second image; wherein classifying the first point as dynamic or static based on the comparison of the first point density and the one or more second point densities comprises classifying the first point as dynamic or static based on a comparison of the first point density and the one or more second point densities of the modified second image set.

[0089] In Example 19, the subject matter of any of Examples 17 or 18 can optionally include the method further comprising: changing the second point density based on a comparison of the depth information of the first point and the depth information of the second point.

[0090] In Example 20, the subject matter of Example 19 can optionally include that changing the point density surrounding the second point based on depth information of the first point compared to depth information of the second point comprises increasing the second point density when the first depth information corresponds to a depth less than a depth of the second depth information, and decreasing the second volume when the first depth information corresponds to a depth greater than a depth of the second depth information.

[0091] In Example 21, the subject matter of any of Examples 17 to 20 can optionally include the method further comprising: classifying the first point as dynamic, static, or unknown based on the comparison of the first point density and the one or more second point densities.

[0092] In Example 22, the subject matter of any of Examples 17 to 21 can optionally include the first image of the environment and the one or more second images of the environment being configured as point cloud images.

[0093] In Example 23, the subject matter of Example 22 can optionally include the method further comprising: generating the point cloud images by resolving the plurality of images or depth images with a position of one or more image sensors when the plurality of images or depth images were captured.

[0094] In Example 24, the subject matter of any of Examples 17 to 23 can optionally include the method further comprising: generating a modified data set by removing from the first image and the one or more second images data corresponding to a surface traversable by the dynamic object detection device; and wherein classifying the first point as dynamic or static comprises classifying the first point as dynamic or static based on a comparison of the density of the first point and the density of the one or more second points within the modified data set.

[0095] In Example 25, the subject matter of any of Examples 17 to 24 can optionally include the method further comprising: generating a two-dimensional grid of the environment; and labeling portions of the two-dimensional grid as dynamic or static based on classifying the first point as dynamic or static based on comparing the first point density and the one or more second point densities.

[0096] In Example 26, the subject matter of any of Examples 17 to 25 can optionally include the method further comprising: operating in a first mode of operation when a distance between the robot and an object corresponding to a static point is within a predetermined range; and operating in a second mode of operation when the distance between the robot and the object corresponding to a dynamic point is within the predetermined range.

[0097] In Example 27, the subject matter of Example 26 can optionally include wherein the first mode of operation comprises sending no command to decelerate or stop the robot; and wherein the second mode of operation comprises sending a command to decelerate or stop the robot.

[0098] Example 28 is a non-transitory computer-readable medium comprising instructions that, if executed by a processor, cause the processor to perform the method of any of Examples 17-27.

[0099] Example 29, the subject matter of any one of Examples 1 to 15 can optionally include that the device is a security device for distinguishing dynamic from static objects.

[0100] Example 30, the subject matter of any one of Examples 3 to 15 can optionally include, wherein the processor is further configured to enlarge the second volume when the depth information corresponds to a greater distance to the first point than to a second point, or to decrease the second volume when the depth information corresponds to a smaller distance to the first point than to a second point.

[0101] Example 31, the subject matter of Example 14 can optionally include, wherein the processor is further configured to associate height information with at least one cell of the grid, and if the height information is within a range, calculate a first minimum distance to be maintained from an object in the cell, and if the height information is outside the range, calculate a second minimum distance to be maintained from an object in the cell, wherein the first minimum distance is greater than the second minimum distance.

[0102] Example 32, the subject matter of any one of Examples 1 to 17 can optionally include, wherein the processor is configured to compare depth information for a point with estimated depth information for the point and mark the point as occluded when a difference between the depth information and the estimated depth information is outside a range and mark the point as visible when a difference between the depth information and the estimated depth information is within a range.

[0103] Example 30, the subject matter of any one of Examples 1 to 32 can optionally include, wherein the second point corresponds to the first point in each of one or more second images.

[0104] Example 31, the subject matter of any one of Examples 1 to 16 can optionally include that the device for detecting a dynamic object is configured as an autonomous or semi-autonomous robot.

[0105] In Example 32, the subject matter of any one of Examples 1 to 16 can optionally include the dynamic object detection device being configured as a vehicle. The dynamic object detection device of any one of Claims 0 to 1, wherein the dynamic object detection device is configured as an autonomous robot.

[0106] Although the above descriptions and the associated figures may depict components as separate elements, those skilled in the art will understand the various ways to combine or integrate discrete elements into a single element. This may include combining two or more circuits to form a single circuit, mounting two or more circuits on a common chip or chassis to form an integrated element, executing discrete software components on a common processor core, etc. Conversely, one skilled in the art will recognize the possibility of separating a single element into two or more discrete elements, such as splitting a single circuit into two or more separate circuits, separating a chip or chassis into discrete elements originally provided thereon, separating a software component into two or more sections and executing each on a separate processor core, etc.

[0107] It is understood that implementations of methods described in detail herein are demonstrative and thus are understood to be capable of being implemented in a corresponding device. Likewise, it is understood that implementations of devices described in detail herein are understood to be capable of being implemented as a corresponding method. Therefore, it is understood that a device corresponding to a method described in detail herein may include one or more components configured to perform each aspect of the associated method.

[0108] All acronyms defined in the above description also apply in all claims contained herein.

Claims

[1] An apparatus for detecting a dynamic object, comprising: a processor configured to: a first point density of a first volume around a first point in a first image is determined, wherein the first image is an image of an environment of a robot, wherein the image contains image data corresponding to three dimensions; determines one or more second point densities, wherein each second point density of the one or more second point densities is a point density of a second volume around a second point; wherein the one or more second images of the environment is or are one or more images taken before the first image and containing image data corresponding to the three dimensions; and classifies the first point as dynamic or static based on a comparison of the first point density and the one or more second point densities. [2] The dynamic object detection apparatus of claim 1, wherein the processor is further configured to: a first depth information of the first point within the first image is determined; determines a second depth information for each second point in the one second image or in the plurality of second images; and determines a modified image set as an image set comprising the one or more second images in which the first depth information corresponds to a greater depth than a depth of the second depth information of the corresponding second image; wherein classifying the first point as dynamic or static based on the comparison of the first point density and the one or more second point densities comprises classifying the first point as dynamic or static based on a comparison of the first point density and the one or more second point densities of the modified second image set. [3] The dynamic object detecting apparatus according to claim 1 or 2, wherein the processor is further configured to change the second point density based on a comparison of the depth information of the first point and the depth information of the second point. [4] The dynamic object detection apparatus of claim 3, wherein the processor is further configured to increase the second point density when the depth information corresponds to a greater distance to the first point than to a second point, or to decrease the second volume when the depth information corresponds to a smaller distance to the first point than to a second point. [5] The dynamic object detecting apparatus according to claim 4, wherein changing the point density surrounding the second point based on depth information of the first point compared to depth information of the second point comprises increasing the second point density when the first depth information corresponds to a depth less than a depth of the second depth information, and decreasing the second point density when the first depth information corresponds to a depth greater than a depth of the second depth information. [6] Apparatus for detecting a dynamic object according to any one of claims 1 to 5, wherein the first volume and the second volume are spherical. [7] The apparatus for detecting a dynamic object according to any one of claims 1 to 6, wherein the processor is further configured to classify the first point as dynamic, static, or unknown based on the comparison of the first point density and the one or more second point densities. [8] The dynamic object detecting apparatus according to any one of claims 1 to 7, wherein the first image of the environment and the one or more second images of the environment are configured as point cloud images. [9] The dynamic object detection apparatus of claim 8, further comprising one or more image sensors configured to generate a plurality of images or depth images; wherein the processor is further configured to generate the point cloud images by resolving the plurality of images or depth images with a position of the one or more image sensors when the plurality of images or depth images were captured. [10] The dynamic object detecting apparatus of claim 9, wherein the one or more image sensors comprise a stereo camera or a depth camera. [11] Apparatus for detecting a dynamic object according to claim 9, wherein the image generator is a LiDAR, Light Detection and Ranging, device. [12] The dynamic object detection apparatus of claim 9, further comprising one or more image sensors configured to generate a plurality of images; wherein the processor is further configured to generate the point cloud images by resolving the plurality of images using one or more photogrammetry techniques. [13] The dynamic object detection device of any one of claims 1 to 12, wherein the processor is further configured to generate a modified data set by removing from the first image and the one or more second images data corresponding to a surface traversable by the dynamic object detection device; and wherein classifying the first point as dynamic or static comprises classifying the first point as dynamic or static based on a comparison of the density of the first point and the density of the one or more second points within the modified data set. [14] Apparatus for detecting a dynamic object according to any one of claims 1 to 13, wherein the processor is further configured to generate a two-dimensional grid of the environment and to label portions of the two-dimensional grid as dynamic or static based on the classification of the first point as dynamic or static based on the comparison of the first point density and the one or more second point densities. [15] The dynamic object detection apparatus of claim 14, wherein the processor is further configured to associate height information with at least one cell of the grid, and if the height information is within a range, calculate a first minimum distance to be maintained from an object in the cell, and if the height information is outside the range, calculate a second minimum distance to be maintained from an object in the cell, wherein the first minimum distance is greater than the second minimum distance. [16] The dynamic object detecting apparatus according to any one of claims 1 to 15, wherein the processor is configured to operate in a first operation mode when a distance between the robot and an object corresponding to a static point is within a predetermined range, and to operate in a second operation mode when the distance between the robot and the object corresponding to a dynamic point is within the predetermined range. [17] The dynamic object detecting apparatus of claim 16, wherein the first operation mode comprises the processor not sending a command to decelerate or stop or execute an evasive maneuver of the robot; and wherein the second operation mode comprises the processor sending a command to decelerate or stop or execute an evasive maneuver of the robot. [18] The dynamic object detection device of any one of claims 1 to 17, wherein the processor is configured to compare the depth information for a point with an estimated depth information for the point and to mark the point as hidden if a difference between the depth information and the estimated depth information is outside a range and to mark the point as visible if a difference between the depth information and the estimated depth information is within a range. [19] A dynamic object detecting apparatus according to any one of claims 1 to 18, wherein the dynamic object detecting apparatus is configured as an autonomous robot. [20] A method for detecting a dynamic object, comprising: Determining a first point density of a first volume around a first point in a first image, the first image being an image of an environment of a robot, the image containing image data corresponding to three dimensions; Determining one or more second point densities, wherein each second point density of the one or more second point densities is a point density of a second volume around a second point; wherein the second point corresponds to the first point in each of one or more second images, wherein the one or more second images of the environment is or are one or more images taken before the first image and containing image data corresponding to the three dimensions; and Classifying the first point as dynamic or static based on a comparison of the first point density and the one or more second point densities.

Images