System and method for controlling a mobile industrial robot using a probabilistic occupancy grid

Abstract

A robot (210) movable on a substrate (220) is controlled based on an occupancy grid (230) of cells (231), wherein each cell is associated with an occupancy probability that a physical object (240, 241, 242) is present in the cell. Based on the elevation map and the chromaticity map, the occupancy probabilities can be assigned as follows: a first partial occupancy probability px is derived from the average elevation of each cell; a distribution of chromaticity is estimated and the dominant chromaticity values ​​are identified; a rule is applied: if the chromaticity of the cell is one of the dominant chromaticity values ​​in the chromaticity map, a lower second partial occupancy probability p2 is assigned to the cell when the chromaticity of the cell is a dominant chromaticity value; and the occupancy probability is obtained by combining the first partial occupancy probability and the second partial occupancy probability.

Description

Technical Field

[0001] The present disclosure relates to the field of robotic navigation, and in particular, to methods and apparatus for controlling a mobile industrial robot based on a probabilistic occupancy grid representing substrates on which the robot can move. Background Art

[0002] A great deal of research has been and is still being done on the problem of developing perception mechanisms for supporting control systems in mobile industrial robots. A satisfactory perception mechanism should enable the robot to navigate efficiently in confined and manned environments while avoiding destructive collisions. Objects that are mixed with other objects at different elevations, objects with small size, transparent or reflective objects, dark or black objects are known to cause difficulties for perception mechanisms that attempt to discover and correctly evaluate them.

[0003] So-called occupancy grids, or probability maps, are a common tool for modeling navigable environments for mobile robots. The cells of the occupancy grid are initialized with neutral probability values ​​and subsequently refined based on acquired sensor data. The problem of updating such a probability map based on a sequence of sensor measurements taken from a mobile platform is discussed in a preprint by Chr. Robbiano et al. (Bayesian Learning of Occupancy Grid, retrieved from arxiv.org / abs / 1911.07915).

[0004] The present disclosure is directed to improving the prior art, particularly for indoor mobile robotic applications where navigable environments are addressed by a two-dimensional occupancy grid. Summary of the invention

[0005] One object is to provide a usable method and apparatus for providing an occupancy grid to support the control of a mobile industrial robot. Another object is to enable maintenance and continuous improvement of such an occupancy grid based on continuous measurements of a color depth sensor, in particular a robot mounted color depth sensor. A particular object is to propose efficient data processing techniques suitable for use cases where RGB-D (Red-Green-Blue plus Depth, i.e., color image and depth map) measurements are available.

[0006] These and other objects are achieved by a method and a robot controller according to the invention as defined by the independent claims. The dependent claims are directed to embodiments of the invention.

[0007] In a first aspect, a method for controlling an industrial robot movable on a substrate comprises: initializing an occupancy grid of cells, each cell representing a portion of the substrate and associated with an occupancy probability of some physical object being present in the cell; and obtaining an elevation map and a chromaticity map of the substrate. Based on the elevation map, the average elevation of each cell is calculated, and a first partial occupancy probability is derived from the average elevation in a manner such that a higher first partial occupancy probability corresponds to a larger average elevation. Based on the chromaticity map, the distribution of chromaticity is estimated and a dominant value is identified, and a second occupancy probability is calculated based on the chromaticity of the cell in a manner such that if the chromaticity of the cell is one of the dominant chromaticity values, a lower second partial occupancy probability is calculated. For each cell, the first partial probability and the second partial probability are merged, and the result constitutes the occupancy probability assigned to the cell in the occupancy grid. Thus, the provided occupancy grid can be used as a basis for controlling an industrial robot.

[0008] The inventors have recognized that, in order to determine whether a cell is free or occupied, it is advantageous to identify the global dominant chromaticity value and search for cells in which the chromaticity value of the cell simultaneously belongs to the global dominant chromaticity value and the elevation value is relatively small. Criteria based on this combination are therefore a useful tool for constructing algorithms that converge in a reasonable time, produce highly accurate occupancy grids, or both. Thus, potential production clock time spent by the robot waiting to generate a new occupancy grid (or for updates to an existing occupancy grid) is reduced. Likewise, robot navigation can become safer and more accurate.

[0009] In a second aspect, a robot controller for controlling at least one mobile industrial robot is provided. The robot controller comprises an input interface, a processing circuit arrangement and an output interface, and is configured to perform the above method.

[0010] The invention also relates to a computer program comprising instructions for causing a computer or in particular a robot controller to perform the above method. The computer program may be stored or distributed on a data carrier. As used herein, a "data carrier" may be a transient data carrier such as a modulated electromagnetic or light wave, or a non-transient data carrier. Non-transient data carriers include volatile and non-volatile memories, such as permanent and non-permanent memories of the magnetic, optical or solid-state type. Still within the scope of a "data carrier", such a memory may be fixedly mounted or portable.

[0011] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless otherwise explicitly defined herein. All references to "a / an / the element, device, component, part, step, etc." will be openly interpreted as referring to at least one instance of an element, device, component, part, step, etc., unless otherwise explicitly stated. Unless explicitly stated, the steps of any method disclosed herein do not have to be performed in the exact order disclosed. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Aspects and embodiments will now be described by way of example with reference to the accompanying drawings, in which:

[0013] Figure 1 is a flow chart of a method according to an embodiment;

[0014] Figure 2 is a perspective view of an industrial robot adapted to move on a substrate, the substrate being modeled as a two-dimensional occupancy grid, and objects on the substrate representing potential obstacles to the robot's movement;

[0015] Figure 3 is a block diagram of a robot controller according to an embodiment;

[0016] Figure 4 shows a two-dimensional color space parameterized by normalized (intensity normalized) green and red components of pixel intensity values ​​according to a color model;

[0017] Figure 5 and Figure 6 Shows the division into Figure 4 An example of numbered baskets of a color space of ; and

[0018] Figure 7 It is a reference Figure 6 Histogram of an example chromaticity map for the partitions of . DETAILED DESCRIPTION

[0019] Aspects of the present disclosure will now be described more fully below with reference to the accompanying drawings, on which certain embodiments of the invention are shown. However, these aspects may be implemented in many different forms and should not be construed as limiting; rather, embodiments are provided by way of example so that the present disclosure will be comprehensive and complete and fully convey the scope of all aspects of the invention to those skilled in the art. Throughout the specification, the same reference numerals represent the same elements.

[0020] Figure 2An environment in which the present invention can be applied is described. Here, an industrial robot 210 can move on a substrate 220 via wheels 214, belts, claws, movable suction cups, or other propulsion and / or attachment devices. The substrate 220 can be a flat or curved surface with a horizontal, inclined, or vertical orientation; it can optionally be equipped with tracks or other movement guides suitable for cooperating with the robot 210. Example substrates 220 include indoor floors, outdoor floors, surfaces pretreated to facilitate travel, etc. In addition, there are physical objects that can act as obstacles of varying severity to the movement of the robot 210 on the substrate 220. As an example, Figure 2 Shown are a box-shaped object 240, a liquid pool 241, and a medium-sized pile of powdered material 242 that is comparable in height to the robot 210. Of these, the liquid pool 241 does not generally hinder the movement of the robot 210, and is therefore intended to be considered a non-obstacle for the algorithm described below.

[0021] The bookkeeping of obstacles relies on an occupancy grid 230, i.e., a data structure that associates each of a plurality of cells 231 of the grid with an occupancy probability p. The occupancy probability of a cell at grid index (x, y) may be denoted as p(x, y). The resolution of the grid may be denoted as the cell size parameter w res , which may be of the order of 0.01 m to 1 m, such as 0.05 m, 0.10 m, or 0.20 m. For navigation of relatively slow-moving and sensitive robots 210, smaller cell sizes are preferred. In this disclosure, unless otherwise specified, grid indices (x, y) refer to the grid at resolution w. res A discrete grid at . A grid index can, for example, be represented as an integer pair, and a real-valued Cartesian coordinate (X, Y, Z) (which can be read from a sensor) can be converted to a grid index (x, y) by an orthogonal projection onto a plane with Z=0 (corresponding to substrate 220) combined with a rounding operation. The fact that the grid index corresponds to the converted coordinate (X, Y, Z) can be written as (X, Y, Z)~(x, y); note that this is a rank-deficient operation that is generally irreversible. The grid index can optionally be expanded into integer triplets (x, y, z) with a height index z, each triplet referring to a cubic grid cell (voxel). In the present disclosure, uppercase letters are preferred for coordinates and lowercase letters are preferred for grid and height indices.

[0022] The present disclosure will rely on common terminology and the theoretical basis generally involved in occupancy grids. In particular, the occupancy probability of a cell can be understood as the posterior probability that a physical object exists in the cell. The highest probability of the presence of a physical object corresponds to the value p=1. Cells with occupancy probability values ​​in the range of (0.5, 1.0] can be considered occupied cells, and cells with values ​​in the range of [0.0, 0.5) can be considered idle. The remaining value 0.5 represents an unknown occupancy state. The occupancy grid 230 is a probabilistic representation that goes beyond the binary distinction of occupied / idle, thereby allowing sensor fusion and fictitious combinations of multiple measurements from a single sensor. In some embodiments, an alternative occupancy grid is used, in which each cell is associated with a probability p′ (“non-occupancy probability”) that the cell is idle (i.e., no physical object is present). This means that the highest probability of the presence of a physical object corresponds to the value p′=0.

[0023] The robot 210 shown is equipped with a movable arm 213, an end effector 212 and a sensor 211. The sensor can be elevated, that is, located a certain distance Z above the horizontal plane of the substrate 220. a The sensor 211 may be configured to employ at least one measurement principle including sensing incident electromagnetic waves reflected or emitted by an object 240, 241, 242 on the substrate 230 or the substrate 230 itself. As an example, Figure 2 , the incident light from box 240 is shown. In particular, sensor 211 has the ability to provide elevation and chromaticity maps of substrate 220 and any physical objects present on substrate 220. For example, an RGB-D sensor can produce a two-dimensional color bitmap in which at least some pixels are annotated with depth values, representing an image plane orthogonal to the direction of the incident angle S. Such depth-annotated pixels can be converted into a point cloud, in which each point is represented by three-dimensional coordinates (e.g., Cartesian coordinates, spherical coordinates, radial coordinates) and additionally carries a color or chromaticity value. Alternatively, the RGB-D sensor can generate a point cloud directly. A library with general software routines for analyzing point cloud data and performing operations thereon can be obtained, for example, from the Point Cloud Library (https: / / pointclouds.org).

[0024] Figure 3 Suitable for control Figure 2Schematic diagram of a robot controller 300 of a robot 210 of FIG. The robot controller 300 may be integrated with the robot 210, or may be implemented as a separate, fixed or mobile unit. It includes: an input interface 310 configured to receive occupancy-related measurements acquired by sensors 211 on the robot; a processing circuit device 320 configured to assign occupancy probabilities based on the received occupancy-related measurements; an output interface 330 for providing commands to the robot 210; and an operator interface 340. The interfaces 320, 340 and the processing circuit device 320 may be local resources or networked (or cloud) resources; in particular, the processing circuit device 320 may include a processor or multiple processors. The operator interface 340 may be a text-based, graphical, auditory, and / or tactile interface through which the robot controller 300 may receive instructions that indicate, at a high level, commands to be fed to the robot 210, and through which it may output human-understandable information indicating the state or condition of the robot 210. Figure 3 Also shown is a set of exemplary communication connections (data connections) between the internal functional components 310, 320, 330, 340 of the robot controller 300. Alternatively, these connections may be implemented as a central bus providing multi-point connections.

[0025] To establish and maintain the occupancy grid, the robot controller 300 may execute Figure 1 Method 100 is shown.

[0026] In a first step 110, the occupancy grid is initialized. It may take into account the geometry of the substrate 220 and the cell size w res Initialization may also include assigning neutral occupancy probabilities (e.g., p=0.5 corresponding to unknown occupancy states) to all cells 231 of the occupancy grid 230. Alternatively, specific placeholder values ​​may be assigned to indicate that the occupancy probabilities have not yet been updated taking into account occupancy-related measurements.

[0027] In a second step 120 , an elevation map and a chromaticity map of the substrate 220 are acquired.

[0028] As used herein, the elevation map compares each point of the substrate 220 to a point above a reference level Z=Z min The reference level may coincide with the base plate 220, or may be a dedicated reference plane; the latter option may simplify calculations in the case where the base plate 220 is curved. The resolution of the elevation map may be one value per cell 231 of the occupancy grid 230 or finer resolution; the elevation values ​​may be continuous or may be proportional to the grid resolution w res is discretized consistently. Figure 2In the example shown, physical objects 240, 241, 242 have respective elevations h 24 0,h 241 ,h 242 .

[0029] The chromaticity diagram associates points of the substrate 220 with corresponding chromaticity values. Similar to the elevation map, it is sufficient for the chromaticity diagram to have one chromaticity value for each cell 231 of the occupancy grid 230. Chromaticity can refer to any suitable color model, such as RGB, HSL, HSV, Munsell, NCS, etc. For the purpose of illustration and not limitation, RGB representation will be used in this specification. If multiple points within a cell 231 carry chromaticity values, their average value or the value obtained by global smoothing (e.g., fitting a function similar to the function h discussed below) can be applied as the chromaticity value of the cell. Alternatively, multiple chromaticity values ​​remain unchanged, the main criteria are evaluated for each chromaticity value, and all results of these evaluations in the cell are merged into one.

[0030] The data source of the two figures may be a sensor 211. As indicated, the sensor 211 may be an RGB-D sensor. The output of the sensor 211 may be (or may be converted into) three separate pixel intensity components (X i , Y i , Z i , R i , G i , B i ), or the pixel intensity components concatenated into a binary value (X i , Y i , Z i , RGB i ). Then, starting from a tuple of this form, the second step 120 may comprise:

[0031] - Discretization operation, in which the horizontal coordinates (X i , Y i ) is converted (rounded) to grid coordinates (x, y);

[0032] -Elevation extraction, where elevation Z i is associated with the cell at grid coordinate (x, y); and

[0033] - Chroma extraction, where the chroma data (R i , G i , B i ) or RGB i Associated with the cell at grid coordinate (x, y).

[0034] The elevation extraction may optionally include a pre-processing operation 130 that eliminates point cloud points that are located in remote areas compared to the sensor and / or points that are physically inaccessible. In this sense, remote areas that are difficult to see from the sensor 211 may correspond to those points that meet the following criteria (X i , Y i , Z i ):

[0035] C1:|(X i , Y i )-(X a , Y a )|≥d1

[0036] where |·| represents the Euclidean distance, (X a , Y a , Z a ) is the position of the sensor 211 and d1 is a constant (cutoff point). The constant d1 depends on the sensor hardware and the difficulty of the navigable environment; for example, in indoor use, it can be of the order of tens of meters. Although the above expression involves two-dimensional distances (i.e., measured parallel to the substrate 220), variations of the criterion C1 can involve three-dimensional distances. The physically unreachable points (X1) to be removed in the pre-processing operation are i , Y i , Z i ) are for those Z i <Z min , which may be due to sensor artifacts.

[0037] The preprocessing operation 130 may also include eliminating point cloud points that represent outliers of elevation. The fact that an elevation value is significantly different from other observations may indicate that its correctness is questionable, so it may be considered an outlier. Again, reference may be made to software routines available from the Point Cloud Library (https: / / pointclouds.org).

[0038] Chroma extraction can optionally include intensity normalization, where a pixel intensity triplet (R, G, B) is mapped to a pair (r, g) such that

[0039]

[0040] Each of R, G, and B can be a number in the range [0, 255] that is linearly related to the corresponding intensity of the color component. Figure 4 is a graphical representation of the resulting two-dimensional color space. The reference colors green, blue, and red are indicated along with white, corresponding to a flat spectrum where R=G=B. Intensity normalization can help eliminate measurement artifacts associated with uneven illumination of different areas of the substrate 220.

[0041] In a fourth step 140, the average elevation Z of each cell is calculated. ave (x, y), and derive the first partial occupancy probability p1. The execution of the fourth step 140 may be restricted to those cells 231 of the occupancy grid 230 for which the elevation map contains elevation data. The average elevation may be calculated as follows:

[0042]

[0043] where N(x, y) is the sensor value in the grid cell at (x, y). i , Y i , Z i ) is the number of points in the point cloud. Another way to calculate the average elevation is to use a function h so that the surface ε = {(X, Y, Z): Z = h(X, Y)} approximates the point cloud data {(X i , Y i , Z i Here, the function h may have a predetermined smoothness (eg, continuously differentiable n times), the smoothness being defined on the substrate 220 and being fitted to the point cloud data in such a way that the surface constitutes L for some point cloud data with q≥0 q Approximately, or in another appropriate sense "close" to the point cloud data, the average elevation is obtained by sampling the surface ε at each grid cell as follows:

[0044] Z ave (x, y) = h(x, y)

[0045] This alternative way of calculating the mean elevation can result in more uniform data accuracy because it reduces the random effects of rounding / quantization into the grid cells of the sensor data.

[0046] Based on the mean elevation, the first part of the probability can be calculated as follows:

[0047]

[0048] Where Z max is the maximum height value. Maximum height Z max may be equal to the height at which the sensor 211 is set, as this may correspond to the height of the highest object visible to the sensor 211 without obstruction. Alternatively, the maximum height Z max Set to a value corresponding to the height of what is believed for other reasons to be the highest physical object on substrate 220 , the substrate can be reliably found by sensor 211 under normal operating conditions.

[0049] A subsequent step 150 of the method 100 comprises a plurality of sub-steps 152 , 154 , in which the chrominance data are processed with the ultimate goal of determining a second partial occupancy probability p2 of at least some cells 231 of the occupancy grid 230 .

[0050] In a first sub-step 152, the distribution of chromaticity is estimated to identify dominant chromaticity values. The estimation may include forming a histogram for a set of predefined bins or color categories. Figure 5 An example of partitioning (gridding) the color space (r, g) into approximately uniformly sized rectangles that have been sequentially numbered or provided with other identifiers is shown. The definition of the partitions is equivalent to a function c(r, g) that maps a pair of normalized pixel intensity values ​​(r, g) to the sequential number of the bins. Figure 6 An alternative partitioning is shown that reflects the fact that perceived chromaticity changes relatively slowly near the corner (0,0), (1,0), (0,1), and can therefore be reliably captured by a smaller number of bins.

[0051] exist Figure 5 The partitions shown in and to some extent in Figure 6 The partitions shown in can be said to represent world-agnostic partitions, more precisely in the sense that the density of the baskets is approximately uniform over the color space. In use cases where it is known or expected that certain chromaticity values ​​are absent or very unusual, this a priori knowledge can be exploited to form color space partitions with locally decreasing and / or locally increasing density. Figure 7 A reference to an exemplary chromaticity diagram is shown. Figure 6 The vertical axis represents the chromaticity frequency f of the corresponding basket. The frequency f can be an occurrence rate, such as the number or normalized number of observations falling into each basket. For example, one can define

[0052] f(c)=Card{i:C(r i , g i )=c}, 1≤c≤c max ,

[0053] Where Card represents the number of elements in the collection, and c max is the number of baskets (here: 11).

[0054] As an alternative to using a histogram, the first sub-step 152 can start with the parameterization of a probability density function (ansatz) and vary its parameters such that it matches the sensor data.

[0055] exist Figure 7 In the histogram plotted in , the primary chromaticity value corresponds to the chromaticity value of bin 2. The second layer of primary chromaticity values ​​may include bins 1 and 10.

[0056] The second sub-step 154 ​​utilizes knowledge gained from the chromaticity distribution, such as can be visualized in the form of a histogram. As explained, the inventors have recognized that chromaticity values ​​for a cell that are equal to or close to the dominant chromaticity value (referenced to the global chromaticity map) and appear as non-significant elevation values ​​in the same grid cell can indicate that the cell is unoccupied, i.e., there is no physical object. Therefore, for the cell at the grid coordinate (x, y) associated with the chromaticity (r, g) = (r(x, y), g(x, y)), the second partial occupancy probability can be assigned as follows:

[0057]

[0058] Here, the relative dominance of the chromaticity bin to which the cell belongs is quantified as the fraction of the most filled chromaticity bin. If the chromaticity value belongs to the most filled (dominant) bin, then p2(x, y) = 0. Alternative definitions of p2 include

[0059]

[0060] For some constant 0<k<1, and

[0061]

[0062] In the next step 160, the first partial occupancy probability p1(x,y) and the second partial occupancy probability p2(x,y) of the cell at the grid coordinate (x,y) are combined to obtain the occupancy probability p(x,y) to be assigned to the cell in the occupancy grid 230. The first partial occupancy probability and the second partial occupancy probability may be combined linearly or nonlinearly. For example,

[0063]

[0064] The coefficients α1, α2 are the same or different and are greater than 0. Appropriate coefficient values ​​can be determined by empirical evaluation, where the initial guess can reflect the expected data quality of the elevation map and chromaticity map, or its value can serve as a guide to determine whether there is a physical object in the cell.

[0065] The fact that each of the two partial occupancy probabilities contributes reflects the inventors' recognition that the combination of dominant chromaticity values ​​and low elevation is important for indicating that a cell is likely to be idle, i.e., to be assigned a low occupancy probability. Figure 2 As can be recalled from the example, liquid pool 241 has a different color than substrate 220, which alone indicates that it is an obstacle, but at the same time the elevation h 241 ≈0 can be neglected, which would tend to disqualify it as an obstacle.

[0066] In some embodiments, the first partial occupancy probability p1(x, y) and the second partial occupancy probability p2(x, y) are also related to the pre-existing occupancy probability p old (x, y) merge. To this end, a recursive rule such as Bayes' rule can be applied. The merge can be performed as follows:

[0067]

[0068] As a final step 170 of method 100, the robotic controller 300 may control the movement of the industrial robot 310 based on the occupancy grid. Such control may include local motion planning, tactical path planning, collision avoidance, speed regulation, and similar tasks.

[0069] Aspects of the disclosure have mainly been described above with reference to a few embodiments. However, as readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims.

Claims

1. A method (100) for controlling an industrial robot (210), the industrial robot being movable on a substrate (220), comprising: Initializing (110) an occupancy grid (230) of cells (231), each of said cells (231) representing a portion of said substrate and associated with an occupancy probability that a certain physical object (240) is present in said cell; Acquiring (120) an elevation map and a chromaticity map of the substrate; calculating (140) an average elevation of each cell based on the elevation map, and deriving a first portion occupancy probability, wherein a higher first portion occupancy probability corresponds to a larger average elevation; Based on the chromaticity diagram, - estimating (152) the distribution of said chromaticity, - identifies the primary chromaticity value, and - calculating (154) a second partial occupancy probability based on the chrominance of each cell, wherein if the chrominance of the cell is one of the dominant chrominance values, a lower second partial occupancy probability is assigned to the cell; for each cell, combining (160) the first portion of occupancy probabilities and the second portion of occupancy probabilities into an occupancy probability for the cell; and The industrial robot is controlled (170) based on the occupancy grid.

2. The method according to claim 1, wherein the elevation map and the chromaticity map are acquired from an RGB-D sensor (211).

3. The method according to claim 1 or 2, further comprising: Prior to calculating the average elevation, outliers of the elevation and / or physically unreachable values ​​of the elevation and / or values ​​of elevation of remote areas of the substrate are eliminated (130).

4. The method of claim 1 or 2, wherein said estimating the distribution of said chromaticity comprises forming a histogram with predefined bins.

5. The method of claim 1 or 2, wherein the chromaticity diagram comprises values ​​in a two-dimensional chromaticity space obtained by normalizing triplets of pixel intensity values.

6. The method according to claim 1 or 2, wherein said combining (160) the first portion occupancy probability and the second portion occupancy probability comprises calculating a weighted average.

7. The method of claim 1 or 2, wherein the merging (160) further comprises: For each cell, the first partial occupancy probability and the second partial occupancy probability are combined with a pre-existing occupancy probability for the cell.

8. A robot controller (300) configured to control at least one industrial robot (210) capable of moving on a substrate (220), the robot controller comprising: an input interface (310) for receiving data representing an elevation map and a chromaticity map of the substrate from at least one sensor (211); Processing circuitry (320) configured to perform the steps of any one of the preceding claims; and An output interface (330) is used to provide commands to the industrial robot.

9. A computer program product, comprising instructions for causing the robot controller according to claim 8 to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing the computer program product according to claim 9.

Images