A method, apparatus and mobile robot for obstacle detection based on depth images

Abstract

The application discloses a kind of obstacle detection methods based on depth image, which comprises: obtaining depth image, eliminating the depth point in depth image with the height less than the set height threshold of mobile robot bearing plane, to obtain effective image;The effective image in depth image is projected to the bearing plane, to obtain the projection point in the bearing plane, based on the projection point in the bearing plane, the projection point less than the set distance threshold is extracted from mobile robot, to obtain the first boundary of obstacle.Effectively avoid the depth point in depth image, effectively reduce the depth map quality requirement of depth camera.

Description

Technical Field

[0001] This invention relates to the field of machine vision, and in particular, to an obstacle detection method based on depth images. Background Technology

[0002] With the maturity of depth camera technology, depth cameras are increasingly used in machine vision, especially in obstacle avoidance in robots. However, when using depth cameras for obstacle avoidance, high-quality depth images must be output. For example, the image space reconstructed based on the depth image must have a high degree of matching with the actual physical space so that small obstacles can be detected based on high-quality depth images, while avoiding false detections. Otherwise, obstacle detection based on depth images will be inaccurate or even erroneous, thus hindering the application of machine vision in obstacle detection. Summary of the Invention

[0003] This invention provides an obstacle detection method based on depth images to reduce the dependence of depth image-based obstacle detection on the quality of the depth images.

[0004] In a first aspect, the present invention provides an obstacle detection method based on depth images, the method comprising:

[0005] Acquire depth image,

[0006] Depth points in the depth image whose height from the mobile robot's bearing plane is less than a set height threshold are removed to obtain valid images;

[0007] The effective image is projected onto the bearing plane to obtain the projection points in the bearing plane.

[0008] Based on the projection points in the bearing plane, projection points that are less than a set distance threshold from the mobile robot are extracted to obtain the first boundary of the obstacle.

[0009] Preferably, the method further includes at least one of the following processes:

[0010] Processing for deleting isolated projection points in the bearing plane;

[0011] This is used to cluster and filter the projection points in the first boundary, remove erroneous projection points, and obtain the processing of the second boundary;

[0012] This is used to correct the current boundary when the distance between the mobile robot and the current boundary is less than a set boundary distance threshold.

[0013] Preferably, the process of removing depth points in the depth image whose height from the mobile robot's bearing plane is less than a set height threshold to obtain a valid image includes:

[0014] Calculate the height gradient of depth points in the depth image along the camera's optical axis.

[0015] Based on the set height gradient threshold

[0016] Height gradients greater than a threshold are selected and the depth points used to calculate those gradients are retained. Height gradients less than or equal to the threshold are removed, resulting in a valid image containing all valid depth points.

[0017] The step of projecting the effective image from the depth image onto the bearing plane to obtain the projection points in the bearing plane includes:

[0018] The coordinate information of the effective depth points in the effective image is transformed to the bearing plane in the coordinate system of the mobile robot to obtain the projection points.

[0019] Preferably, the calculation of the height gradient of depth points in the depth image along the camera optical axis includes:

[0020] Based on the depth image, determine the three-dimensional coordinate information of the first spatial point corresponding to the depth point in the depth image in the world coordinate system;

[0021] Based on the three-dimensional coordinate information of the first spatial point, the first spatial point is converted into two-dimensional coordinate information in a first plane that is parallel to the camera optical axis and perpendicular to the bearing plane, and the second spatial point information is obtained.

[0022] For any second spatial point, calculate the height difference between the second spatial point and another second spatial point, as well as the distance difference along the camera optical axis, and use the ratio of the height difference to the distance difference as the height gradient of the second spatial point.

[0023] The process of converting the coordinate information of effective depth points in the effective image to the bearing plane in the coordinate system of the mobile robot to obtain the projection points includes:

[0024] For any effective depth point, the three-dimensional coordinate information of the first spatial point corresponding to the effective depth point is converted into two-dimensional coordinate information in the bearing plane under the coordinate system of the mobile robot to obtain the coordinate information of the projection point of the effective depth point;

[0025] or

[0026] For any valid depth point, based on the coordinate information of the valid depth point, the valid depth point is transformed into two-dimensional coordinate information in the bearing plane under the coordinate system of the mobile robot, and the coordinate information of the projection point of the valid depth point is obtained.

[0027] Preferably, determining the three-dimensional coordinate information of the first spatial point corresponding to the depth point in the depth image in the world coordinate system based on the depth image includes:

[0028] Based on the depth image, the cancellation line in the depth image is identified, and depth points below the cancellation line are extracted.

[0029] Determine the spatial information of the first spatial point corresponding to the extracted depth point;

[0030] The step of calculating the height difference between any second spatial point and another second spatial point, as well as the distance difference along the camera's optical axis, further includes:

[0031] Select a second spatial point that has a set first distance threshold along the camera's optical axis as another spatial point.

[0032] Preferably, the step of extracting projection points that are less than a set distance threshold from the mobile robot based on the projection points in the bearing plane includes:

[0033] Based on the coordinate information of the projection points, the projection points closest to the mobile robot are selected, and the selected projection points are used as the first boundary.

[0034] The process for clustering and filtering the projection points in the first boundary to remove erroneous projection points and obtain the second boundary includes:

[0035] Detect discontinuous and continuous boundary intervals within the first boundary;

[0036] Perform a correctness check on the projection points in the discontinuous boundary interval.

[0037] Correct projection points should be retained.

[0038] For incorrect projection points, starting from the incorrect projection point, proceed sequentially along the current depth direction to filter each projection point. When a correct projection point is found, use it as the projection point in the boundary.

[0039] The projection points in the continuous boundary interval, the retained correct projection points, and the selected correct projection points are used as the second boundary.

[0040] The process for deleting isolated projection points includes:

[0041] Iterate through all projection points.

[0042] If the number of consecutive projection points in the neighborhood of the first projection point of the current projection point being traversed is less than the set threshold for the number of first projection points, then the current projection point is determined to be an isolated projection point and is deleted.

[0043] Preferably, the detection of discontinuous boundary intervals and continuous boundary intervals in the first boundary includes:

[0044] When adjacent projection points in the first boundary are not continuous, and / or the length of the boundary with continuous projection points is not greater than the set threshold for the length of the first projection point, then the adjacent projection points and / or the boundary with continuous projection points are the non-continuous boundary intervals in the first boundary.

[0045] When adjacent projection points in the first boundary are continuous and the length of the boundary with continuous projection points is greater than the first projection point length threshold, then the boundary with continuous projection points is a continuous boundary interval in the first boundary.

[0046] in,

[0047] Discontinuity between adjacent projection points is defined as follows: the distance between adjacent projection points is not less than a set projection point distance threshold.

[0048] Continuity between adjacent projection points is defined as follows: the distance between adjacent projection points is less than a set projection point distance threshold.

[0049] The correctness detection of projection points in discontinuous boundary intervals includes:

[0050] For any projection point within a discontinuous boundary interval, the correctness of the projection point is determined based on the projection points within the neighborhood of the second projection point.

[0051] If the number of projection points in the neighborhood of the second projection point is greater than the set threshold for the number of second projection points, then the projection point is determined to be a correct projection point; otherwise, the projection point is determined to be an incorrect projection point.

[0052] Preferably, the projection point distance threshold includes:

[0053] When the distance between the mobile robot and the first boundary is less than the set second distance threshold, the projection point distance threshold becomes the first projection point distance threshold.

[0054] When the distance between the mobile robot and the first boundary is not less than the set second distance threshold, then the projection point distance threshold is the second projection point distance threshold.

[0055] Among them, the distance threshold of the first projection point is greater than the distance threshold of the first projection point.

[0056] Preferably, the process for correcting the current boundary when the distance between the mobile robot and the current boundary is less than a set boundary distance threshold includes:

[0057] If the distance between the mobile robot and the current boundary is less than the set third distance threshold, count the number and / or length of the projected points in the current boundary.

[0058] When the number of projection points counted in the current boundary is less than the set third projection point number threshold, and / or the length of projection points counted in the current boundary is less than the set second projection point length threshold, delete the boundary interval where the counted projection points are located.

[0059] The current boundary is at least one of the first boundary and the second boundary.

[0060] Preferably, after projecting the effective image from the depth image onto the bearing plane to obtain the projection points on the bearing plane, the process further includes:

[0061] A two-dimensional grid is constructed, and the projection points in the bearing plane are distributed in each grid to obtain a grid map that represents the distribution of projection points; the size of the grid with the smallest region is set as needed.

[0062] For any given grid cell, label it according to the projection points within the grid cell:

[0063] If there are no projection points in a grid, the grid is marked as an empty grid.

[0064] If a raster has a projection point, and the adjacent boundary raster in the same column and / or row has no projection points, then the raster is identified as a boundary raster.

[0065] If a grid has a projection point, and the adjacent boundary grid in the same column and / or row of the grid also has a projection point, then the grid is identified as an obstacle grid.

[0066] The process for deleting isolated projection points includes:

[0067] Traverse the non-empty cells in the raster graph.

[0068] If the number of empty grid cells within the first grid cell neighborhood of the current grid cell being traversed is less than the set first grid cell number threshold, or the number of projected points is less than the set first projection point number threshold, then the grid cell is marked as an empty grid cell.

[0069] The step of extracting projection points that are less than a set distance threshold from the mobile robot based on the projection points in the bearing plane includes:

[0070] Extract the boundary grid cell in the grid map that is closest to the mobile robot, and use it as the first boundary;

[0071] The process for clustering and filtering the projection points in the first boundary to remove erroneous projection points and obtain the second boundary includes:

[0072] Detect discontinuous and continuous boundary intervals within the first boundary.

[0073] Perform correctness checks on boundary grids within discontinuous boundary regions.

[0074] Correct boundary grid cells should be preserved.

[0075] For erroneous boundary grids, starting from the erroneous boundary grid, the grids in the grid map are filtered sequentially along the current depth direction. When a correct grid is found, the grid is updated as a boundary grid.

[0076] The boundary grid in the continuous boundary interval, the retained boundary grid, and the updated boundary grid are used as the second boundary.

[0077] Preferably, in the raster image, the column grids are arranged along the current depth direction, and the row grids are arranged perpendicular to the column grids.

[0078] The step of identifying any grid cell based on the projection points within the grid cell further includes:

[0079] If a grid has a projection point and there is no projection point in the adjacent boundary grid in the same column, then the grid is identified as a boundary grid. Alternatively, if in the same column of grids adjacent to the grid, there is no projection point in the grid on the side closer to the mobile robot and / or there is a projection point in the grid on the side farther from the mobile robot, then the grid is identified as a boundary grid. The distance closer to the mobile robot and the distance farther from the mobile robot are set as needed.

[0080] If a grid has a projection point, and the adjacent boundary grid in the same column of the grid also has a projection point, then the grid is identified as an obstacle grid.

[0081] The extraction of the boundary grid cell in the grid map that is closest to the mobile robot, as the first boundary, includes:

[0082] Extract the boundary grid cell closest to the mobile robot from each column of grid cells, and use it as the first boundary.

[0083] The detection of discontinuous boundary intervals and continuous boundary intervals in the first boundary includes:

[0084] When adjacent boundary grids in the first boundary are not continuous, and / or the boundary length of a continuous boundary grid is not greater than the set first grid length threshold, then the adjacent boundary grid and / or the boundary of the continuous boundary grid is a non-continuous boundary interval.

[0085] When adjacent boundary grids in the first boundary are continuous and the length of the boundary with continuous boundary grids is greater than the first grid length threshold, then the boundary with continuous boundary grids is a continuous boundary interval.

[0086] in,

[0087] Discontinuity between adjacent boundary grids is defined as follows: the distance between adjacent boundary grids is not less than a set grid distance threshold, or the number of grids between adjacent boundary grids is not less than a set boundary grid number threshold.

[0088] Continuity between adjacent boundary grids is defined as follows: the distance between adjacent boundary grids is less than a set grid distance threshold, or the number of adjacent boundary grids is less than a set boundary grid number threshold.

[0089] The correctness detection of boundary grids in discontinuous boundary intervals includes:

[0090] For any boundary grid cell within a discontinuous boundary interval, the correctness of the projection points in the boundary grid cell and / or the boundary grid cell itself is determined based on the projection points within the neighborhood of the first grid cell of that boundary grid cell.

[0091] If the number of non-empty grids in the neighborhood of the first grid is greater than the set second grid number threshold, or if the number of projection points in the grids in the neighborhood of the first grid is greater than the set second projection point number threshold, then the projection points in the boundary grid and / or the boundary grid are determined to be correct; otherwise, they are determined to be incorrect.

[0092] In the case of erroneous graticles, starting from the erroneous graticle, the graticles are sequentially filtered along the current depth direction. When a correct graticle is found, it is used as the boundary graticle. This includes:

[0093] Starting from the erroneous raster, sequentially filter the rasters in the same column as the erroneous raster along the current depth direction. When a correct raster is found, update the correct raster as the boundary raster.

[0094] Repeat this process until every faulty grid cell is updated.

[0095] Preferably, the determination that the distance between adjacent boundary grids is not less than a set grid distance threshold includes: if the distance between two adjacent boundary grids in the column direction is not less than the grid distance threshold, and the column grids to which the two adjacent boundary grids belong do not include other boundary grids that make the distance in the column direction less than the grid distance threshold, then the distance between the two adjacent boundary grids is not less than the set grid distance threshold.

[0096] The determination that the number of grids between adjacent boundary grids is less than the set boundary grid number threshold includes: if the number of grids in the column direction of the two adjacent boundary grids is not less than the boundary grid number threshold, and the column grids to which the two adjacent boundary grids belong do not include other boundary grids that would make the number of grids in the column direction less than the boundary grid number threshold, then it is determined that the number of grids between adjacent boundary grids is less than the set boundary grid number threshold.

[0097] The determination that the distance between adjacent boundary grids is less than a set grid distance threshold includes: if the distance between two adjacent boundary grids in the column direction is less than the grid distance threshold, or if the column grids to which the two adjacent boundary grids belong include other boundary grids that make the distance in the column direction less than the grid distance threshold, then the distance between the two adjacent boundary grids is less than the set grid distance threshold.

[0098] The determination that the number of grids between adjacent boundary grids is less than the set boundary grid number threshold includes: if the number of grids in the column direction of two adjacent boundary grids is less than the boundary grid number threshold, or if the column grids to which the two adjacent boundary grids belong include other boundary grids that make the number of grids in the column direction less than the boundary grid number threshold, then the number of grids between adjacent boundary grids is less than the set boundary grid number threshold.

[0099] Wherein, when the distance between the mobile robot and the first boundary is less than the set second distance threshold, the grid distance threshold is the first grid distance threshold, and the boundary grid number threshold is the first boundary grid number threshold;

[0100] When the distance between the mobile robot and the first boundary is not less than the set second distance threshold, the grid distance threshold is the second grid distance threshold, and the boundary grid number threshold is the second boundary grid number threshold.

[0101] Among them, the first grid distance threshold is greater than the second grid distance threshold; the first boundary grid number threshold is greater than the second boundary grid number threshold.

[0102] The process for correcting the current boundary when the distance between the mobile robot and the current boundary is less than a set boundary distance threshold includes:

[0103] If the distance between the mobile robot and the current boundary is less than the set third distance threshold, count the number and / or length of the boundary grid cells in the current boundary.

[0104] If the number of boundary grid cells counted is less than the set third grid cell count threshold, and / or the length of the boundary grid cells counted is less than the set second grid cell length threshold, then the boundary containing the counted boundary grid cells is deleted.

[0105] In a second aspect, the present invention provides a mobile robot, wherein a computer program is stored in the storage medium, and the computer program, when executed by a processor, implements any of the steps of obstacle detection based on depth images.

[0106] A third aspect of the present invention provides an obstacle detection device based on depth images, the device comprising:

[0107] The effective depth point extraction module is used to remove depth points in the depth image whose height from the mobile robot's bearing plane is less than a set height threshold, thus obtaining an effective image.

[0108] The projection module is used to project the effective image from the depth image onto the bearing plane to obtain the projection points on the bearing plane.

[0109] The boundary extraction module is used to extract projection points that are less than a set distance threshold from the mobile robot based on the projection points in the bearing plane, thereby obtaining the first boundary of the obstacle.

[0110] Preferably, the device further includes at least one of the following modules:

[0111] An isolated point processing module is used to delete isolated projection points in the bearing plane;

[0112] The first filtering module is used to cluster and filter the projection points in the first boundary, remove erroneous projection points, and obtain the second boundary.

[0113] The second filtering module is used to correct the current boundary if the distance between the mobile robot and the current boundary is less than a set boundary distance threshold.

[0114] Preferably, the effective depth point extraction module is configured to calculate the height gradient of depth points in the depth image along the camera optical axis, and based on a set height gradient threshold, filter out height gradients greater than the threshold and retain the depth points used to calculate those height gradients; discard height gradients not greater than the threshold and remove the depth points used to calculate those height gradients, thus obtaining an effective image containing effective depth points.

[0115] The projection module includes a projection submodule, which is configured to convert the coordinate information of effective depth points in the effective image to the bearing plane under the coordinates of the mobile robot to obtain the projection point.

[0116] Preferably, the effective depth point extraction module is configured to, based on the depth image, determine the three-dimensional coordinate information of the first spatial point corresponding to the depth point in the depth image in the world coordinate system; based on the three-dimensional coordinate information of the first spatial point, determine the two-dimensional coordinate information of the first spatial point in a first plane that is parallel to the camera optical axis and perpendicular to the bearing plane, to obtain the second spatial point information; for any second spatial point, calculate the height difference between the second spatial point and another second spatial point, and the distance difference in the direction of the camera optical axis, and use the ratio of the height difference to the distance difference as the height gradient of the second spatial point;

[0117] The projection submodule is configured to, for any effective depth point, convert the three-dimensional coordinate information of the first spatial point corresponding to the effective depth point into two-dimensional coordinate information in the bearing plane under the coordinate system of the mobile robot, thereby obtaining the coordinate information of the projection point of the effective depth point; or

[0118] For any valid depth point, based on the coordinate information of the valid depth point, the valid depth point is transformed into two-dimensional coordinate information in the bearing plane under the coordinate system of the mobile robot, and the coordinate information of the projection point of the valid depth point is obtained.

[0119] Preferably, the effective depth point extraction module is configured to determine the disappearance line in the depth image based on the depth image, extract the depth points below the disappearance line, and determine the spatial information of the first spatial point corresponding to the extracted depth point;

[0120] The effective depth point extraction module is further configured to select a second spatial point that has a set first distance threshold along the camera optical axis as another spatial point.

[0121] Preferably, the projection submodule is configured to filter out the projection point closest to the mobile robot based on the coordinate information of the projection point, and use the filtered projection point as the first boundary.

[0122] The first filtering module is configured to detect discontinuous boundary intervals and continuous boundary intervals in the first boundary; to perform correctness checks on the projection points in the discontinuous boundary intervals, and to retain correct projection points; for incorrect projection points, to filter the projection points one by one along the current depth direction starting from the incorrect projection point, and to use the correct projection point as the projection point in the boundary when it is selected; and to use the projection points in the continuous boundary intervals, the retained correct projection points, and the selected correct projection points as the second boundary.

[0123] The isolated point processing module is configured to traverse all projection points. If the number of consecutive projection points in the neighborhood of the first projection point of the current projection point is less than the set threshold for the number of first projection points, then the current projection point is determined to be an isolated projection point and the isolated projection point is deleted.

[0124] Preferably, the boundary extraction module is configured such that when adjacent projection points in the first boundary are discontinuous, and / or the length of the boundary with continuous projection points is not greater than a set threshold for the length of the first projection point, then the adjacent projection points and / or the boundary with continuous projection points are the discontinuous boundary intervals in the first boundary.

[0125] When adjacent projection points in the first boundary are continuous and the length of the boundary with continuous projection points is greater than the first projection point length threshold, then the boundary with continuous projection points is a continuous boundary interval in the first boundary.

[0126] in,

[0127] Discontinuity between adjacent projection points is defined as follows: the distance between adjacent projection points is not less than a set projection point distance threshold.

[0128] Continuity between adjacent projection points is defined as follows: the distance between adjacent projection points is less than a set projection point distance threshold.

[0129] The correctness detection of projection points in discontinuous boundary intervals includes:

[0130] For any projection point within a discontinuous boundary interval, the correctness of the projection point is determined based on the projection points within the neighborhood of the second projection point.

[0131] If the number of projection points in the neighborhood of the second projection point is greater than the set threshold for the number of second projection points, then the projection point is determined to be a correct projection point; otherwise, the projection point is determined to be an incorrect projection point.

[0132] Preferably, the projection point distance threshold includes:

[0133] When the distance between the mobile robot and the first boundary is less than the set second distance threshold, the projection point distance threshold becomes the first projection point distance threshold.

[0134] When the distance between the mobile robot and the first boundary is not less than the set second distance threshold, then the projection point distance threshold is the second projection point distance threshold.

[0135] Among them, the distance threshold of the first projection point is greater than the distance threshold of the first projection point.

[0136] Preferably, the second filtering module is configured to count the number and / or length of projected points in the current boundary when the distance between the mobile robot and the current boundary is less than a set third distance threshold.

[0137] When the number of projection points counted in the current boundary is less than the set third projection point number threshold, and / or the length of projection points counted in the current boundary is less than the set second projection point length threshold, delete the boundary interval where the counted projection points are located.

[0138] The current boundary is at least one of the first boundary and the second boundary.

[0139] Preferably, the projection module further includes,

[0140] The rasterization submodule is used to project the effective image from the depth image onto the carrier plane, obtain the projection points in the carrier plane, construct a two-dimensional raster, and distribute the projection points in the carrier plane into each raster to obtain a raster map that characterizes the distribution of projection points; wherein, the size of the region of the raster with the smallest region is set as needed.

[0141] For any given grid cell, label it according to the projection points within the grid cell:

[0142] If there are no projection points in a grid, the grid is marked as an empty grid.

[0143] If a raster has a projection point, and the adjacent boundary raster in the same column and / or row has no projection points, then the raster is identified as a boundary raster.

[0144] If a grid has a projection point, and the adjacent boundary grid in the same column and / or row of the grid also has a projection point, then the grid is identified as an obstacle grid.

[0145] The process for deleting isolated projection points includes:

[0146] Traverse the non-empty cells in the raster graph.

[0147] If the number of empty grid cells within the first grid cell neighborhood of the current grid cell being traversed is less than the set first grid cell number threshold, or the number of projected points is less than the set first projection point number threshold, then the grid cell is marked as an empty grid cell.

[0148] The boundary extraction module is configured to extract the boundary grid cell in the grid map that is closest to the mobile robot, and use it as the first boundary.

[0149] The first filtering module is configured to detect discontinuous boundary intervals and continuous boundary intervals in the first boundary, perform correctness checks on the boundary grids in the discontinuous boundary intervals, retain correct boundary grids, and filter the grids in the grid map sequentially along the current depth direction, starting from the incorrect boundary grid. When a correct grid is filtered out, the grid is updated as a boundary grid. The boundary grids in the continuous boundary intervals, the retained boundary grids, and the updated boundary grids are used as the second boundary.

[0150] Preferably, in the raster image, the column grids are arranged along the current depth direction, and the row grids are arranged perpendicular to the column grids.

[0151] The rasterization submodule is further configured as follows:

[0152] If a grid has a projection point and there is no projection point in the adjacent boundary grid in the same column, then the grid is identified as a boundary grid. Alternatively, if in the same column of grids adjacent to the grid, there is no projection point in the grid on the side closer to the mobile robot and / or there is a projection point in the grid on the side farther from the mobile robot, then the grid is identified as a boundary grid. The distance closer to the mobile robot and the distance farther from the mobile robot are set as needed.

[0153] If a grid has a projection point, and the adjacent boundary grid in the same column of the grid also has a projection point, then the grid is identified as an obstacle grid.

[0154] The boundary extraction module is configured to extract the boundary grid cell in each column that is closest to the mobile robot, and use it as the first boundary.

[0155] When adjacent boundary grids in the first boundary are not continuous, and / or the boundary length of a continuous boundary grid is not greater than the set first grid length threshold, then the adjacent boundary grid and / or the boundary of the continuous boundary grid is a non-continuous boundary interval.

[0156] When adjacent boundary grids in the first boundary are continuous and the length of the boundary with continuous boundary grids is greater than the first grid length threshold, then the boundary with continuous boundary grids is a continuous boundary interval.

[0157] in,

[0158] Discontinuity between adjacent boundary grids is defined as follows: the distance between adjacent boundary grids is not less than a set grid distance threshold, or the number of grids between adjacent boundary grids is not less than a set boundary grid number threshold.

[0159] Continuity between adjacent boundary grids is defined as follows: the distance between adjacent boundary grids is less than a set grid distance threshold, or the number of adjacent boundary grids is less than a set boundary grid number threshold.

[0160] The boundary extraction module is configured as follows:

[0161] For any boundary grid cell within a discontinuous boundary interval, the correctness of the projection points in the boundary grid cell and / or the boundary grid cell itself is determined based on the projection points within the neighborhood of the first grid cell of that boundary grid cell.

[0162] If the number of non-empty grids in the neighborhood of the first grid is greater than the set second grid number threshold, or if the number of projection points in the grids in the neighborhood of the first grid is greater than the set second projection point number threshold, then the projection points in the boundary grid and / or the boundary grid are determined to be correct; otherwise, they are determined to be incorrect.

[0163] In the case of erroneous graticles, starting from the erroneous graticle, the graticles are sequentially filtered along the current depth direction. When a correct graticle is found, it is used as the boundary graticle. This includes:

[0164] Starting from the erroneous raster, sequentially filter the rasters in the same column as the erroneous raster along the current depth direction. When a correct raster is found, update the correct raster as the boundary raster.

[0165] Repeat this process until every faulty grid cell is updated.

[0166] The determination that the distance between adjacent boundary grids is not less than the set grid distance threshold includes: if the distance between two adjacent boundary grids in the column direction is not less than the grid distance threshold, and the column grids to which the two adjacent boundary grids belong do not include other boundary grids that make the distance in the column direction less than the grid distance threshold, then the distance between the two adjacent boundary grids is not less than the set grid distance threshold.

[0167] The determination that the number of grids between adjacent boundary grids is less than the set boundary grid number threshold includes: if the number of grids in the column direction of the two adjacent boundary grids is not less than the boundary grid number threshold, and the column grids to which the two adjacent boundary grids belong do not include other boundary grids that would make the number of grids in the column direction less than the boundary grid number threshold, then it is determined that the number of grids between adjacent boundary grids is less than the set boundary grid number threshold.

[0168] The determination that the distance between adjacent boundary grids is less than a set grid distance threshold includes: if the distance between two adjacent boundary grids in the column direction is less than the grid distance threshold, or if the column grids to which the two adjacent boundary grids belong include other boundary grids that make the distance in the column direction less than the grid distance threshold, then the distance between the two adjacent boundary grids is less than the set grid distance threshold.

[0169] The determination that the number of grids between adjacent boundary grids is less than the set boundary grid number threshold includes: if the number of grids in the column direction of two adjacent boundary grids is less than the boundary grid number threshold, or if the column grids to which the two adjacent boundary grids belong include other boundary grids that make the number of grids in the column direction less than the boundary grid number threshold, then the number of grids between adjacent boundary grids is less than the set boundary grid number threshold.

[0170] Wherein, when the distance between the mobile robot and the first boundary is less than the set second distance threshold, the grid distance threshold is the first grid distance threshold, and the boundary grid number threshold is the first boundary grid number threshold;

[0171] When the distance between the mobile robot and the first boundary is not less than the set second distance threshold, the grid distance threshold is the second grid distance threshold, and the boundary grid number threshold is the second boundary grid number threshold.

[0172] Among them, the first grid distance threshold is greater than the second grid distance threshold; the first boundary grid number threshold is greater than the second boundary grid number threshold.

[0173] The second filtering module is configured to count the number and / or length of boundary grids in the current boundary when the distance between the mobile robot and the current boundary is less than a set third distance threshold. If the counted number of boundary grids is less than a set third grid count threshold and / or the counted length of the boundary grids is less than a set second grid length threshold, then the boundary containing the counted boundary grids is deleted.

[0174] This invention eliminates depth points in the depth image whose height from the bearing plane is less than a set height threshold as non-obstacle depth points, thereby obtaining valid depth points and effectively avoiding erroneous depth points in the depth image. The remaining depth points after elimination are projected onto the bearing plane, and the boundary is extracted based on the projected points on the bearing plane. Since the projected points only retain depth information, the adverse effects of erroneous depth points on obstacle detection are avoided, and obstacle detection can be performed in a two-dimensional plane, reducing the quality requirements of the depth image. Furthermore, by rasterizing the projected points, erroneous points can be efficiently filtered, achieving a downsampling effect on the depth points in the depth image, effectively reducing the quality requirements of the depth image from the depth camera, and obstacle detection can still be performed on low-quality depth images. Attached Figure Description

[0175] Figure 1 This is a schematic diagram of an obstacle detection method based on depth images according to this application.

[0176] Figure 2 This is a schematic flowchart of an obstacle detection method based on depth images according to Embodiment 1 of this application.

[0177] Figure 3 A schematic diagram for acquiring depth images for a mobile robot.

[0178] Figure 4 for Figure 3 A side view and a schematic diagram of the height gradient.

[0179] Figure 5 This is a schematic flowchart of an obstacle detection method based on depth images, according to Embodiment 2 of this application.

[0180] Figure 6 This is a schematic diagram of a projection point raster map.

[0181] Figure 7 This is a schematic diagram of an isolated grid.

[0182] Figure 8 This is a schematic diagram of an obstacle detection device based on depth images according to an embodiment of this application. Detailed Implementation

[0183] To make the objectives, technical means, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings.

[0184] Depth cameras are increasingly used in obstacle avoidance for robots, but this method also presents several challenges. For example, depth images from depth cameras often contain erroneous depth points, which can interfere with the robot's perception of obstacles. Furthermore, while dense depth images facilitate complete obstacle perception and detection, even when reconstructing depth images using feature points, rich textures are typically required, and even with rich textures, it may be impossible to reconstruct a dense depth image. These factors contribute to the dependence of depth image-based obstacle detection on the quality of the depth image itself.

[0185] In view of this, this application removes depth points in the depth image whose height from the bearing plane is less than a set height threshold as non-obstacle depth points, projects the remaining depth points onto the bearing plane, extracts the boundary based on the projection points in the bearing plane, and removes erroneous depth points by filtering the projection points through various filtering strategies, thereby performing obstacle detection.

[0186] See Figure 1 As shown, Figure 1 This is a schematic diagram of the obstacle detection method based on depth images according to this application. On the mobile robot side, the method includes:

[0187] Step 101: Obtain the depth image.

[0188] The depth images come from cameras that acquire the depth of objects, including stereo vision-based depth cameras and Time-of-Flight (ToF) based depth cameras. The depth cameras can be mounted on a mobile robot to acquire depth images along the robot's direction of travel.

[0189] In a depth image, depth points can be understood as pixels, and each depth point corresponds to both positional and depth information in the pixel coordinate system.

[0190] Step 102: Remove depth points in the depth image whose height from the mobile robot's bearing plane is less than a set height threshold to obtain a valid image.

[0191] This step removes non-obstacle depth points from the carrying plane. For example, if the carrying plane of the mobile robot is the ground, ground depth points can be removed. This prevents ground depth points from interfering with the projection point when the effective image in the depth image is projected onto the carrying plane, thus preserving the depth information of obstacles.

[0192] Step 103: Project the effective image onto the bearing plane to obtain the projection point in the bearing plane.

[0193] This step reduces obstacle detection in three-dimensional space to obstacle detection in two-dimensional plane on the bearing plane. Since depth information is more important for obstacle detection than height information, projecting the effective image onto the bearing plane helps reduce the requirements for depth image quality.

[0194] Step 104: Based on the projection points in the bearing plane, extract the projection points that are less than a set distance threshold from the mobile robot to obtain the first boundary.

[0195] Furthermore,

[0196] At least one of the following screening strategies can be used for processing:

[0197] 1) After obtaining the projection points in the bearing plane, delete the isolated projection points;

[0198] 2) After obtaining the first boundary, cluster the projection points in the first boundary to remove erroneous projection points and obtain the second boundary;

[0199] 3) Used to correct the current boundary when the distance between the mobile robot and the current boundary is less than a set boundary distance threshold. For example, after obtaining the first boundary and / or the second boundary, if the distance between the mobile robot and the current boundary is less than a set third distance threshold, the number and / or length of the projected points in the boundary are counted. If the number of projected points in the counted boundary is less than a set third projection point number threshold, and / or the length of the projected points in the counted boundary is less than a set second projection point length threshold, then the boundary interval containing the counted projected points is deleted.

[0200] The obstacle detection method proposed in this invention reduces three-dimensional spatial information to two-dimensional spatial information by projecting it onto a bearing plane. This allows the robot to avoid obstacles in a two-dimensional projection environment. It utilizes the three-dimensional perception information from a three-dimensional sensor while simultaneously performing rapid path planning in a two-dimensional environment, achieving obstacle avoidance effects similar to two-dimensional laser data. By filtering depth point clouds in depth images and eliminating erroneous depth points, correct obstacle information is obtained, effectively reducing the requirements for depth images and depth cameras. It also effectively solves the problem of potential false detections by mobile robots when even high-quality depth images contain erroneous depth points.

[0201] For ease of reference, the following is a description of the threshold parameters involved in the embodiments.

[0202] Height gradient threshold: Used for height gradient filtering;

[0203] First distance threshold: used to select the depth point in the depth direction when performing height gradient calculation;

[0204] Second distance threshold: used in steps 2051 and 5051 to detect the distance between the mobile robot and the first boundary when detecting continuous boundary intervals in the first boundary;

[0205] The third distance threshold is used to determine the distance between the mobile robot and the obstacle (current boundary) during close-range screening, such as in steps 206 and 506.

[0206] First projection point number threshold: used to detect isolated points in step 203;

[0207] The second projection point number threshold is used to determine the correctness of the projection points when detecting the correctness of the projection points in the first boundary in step 2052.

[0208] The third projection point number threshold is used in step 206 to determine whether to delete the boundary in order to correct the boundary.

[0209] Projection point distance threshold: Used to determine the continuity of adjacent projection points when detecting continuous boundary intervals in the first boundary in step 2051; includes a first projection point distance threshold and a second projection point distance threshold;

[0210] First projection point distance threshold: When detecting continuous boundary intervals in the first boundary in step 2051, if the distance between the mobile robot and the first boundary is close (less than the second distance threshold), it is used to determine the continuity of adjacent projection points.

[0211] Second projection point distance threshold: When detecting continuous boundary intervals in the first boundary in step 2051, if the distance between the mobile robot and the first boundary is relatively far (not less than the second distance threshold), it is used to determine the continuity of adjacent projection points.

[0212] First projection point length threshold: When detecting continuous boundary intervals in the first boundary in step 2051, it is used to determine whether the length of continuous projection points can be used as a continuous boundary.

[0213] Second projection point length threshold: used in step 206 to determine whether to delete the boundary in order to correct the boundary;

[0214] Grid distance threshold: Used to determine the continuity of adjacent boundary grids when detecting continuous boundary intervals in the first boundary in step 5051; includes a first grid distance threshold and a second grid distance threshold;

[0215] First grid distance threshold: When detecting continuous boundary intervals in the first boundary in step 5051, if the distance between the mobile robot and the first boundary is relatively close (less than the second distance threshold), it is used to determine the continuity of adjacent boundary grids.

[0216] Second grid distance threshold: When detecting continuous boundary intervals in the first boundary in step 5051, if the distance between the mobile robot and the first boundary is relatively far (not less than the second distance threshold), it is used to determine the continuity of adjacent boundary grids.

[0217] Boundary grid number threshold: When detecting continuous boundary intervals in the first boundary in step 5051, it is used to determine the continuity of adjacent boundary grids; it includes the first boundary grid number threshold and the second boundary grid number threshold.

[0218] First boundary grid number threshold: When detecting continuous boundary intervals in the first boundary in step 5051, if the distance between the mobile robot and the first boundary is relatively far (not less than the second distance threshold), it is used to determine the continuity of adjacent boundary grids.

[0219] Second boundary grid number threshold: When detecting continuous boundary intervals in the first boundary in step 5051, if the distance between the mobile robot and the first boundary is relatively far (not less than the second distance threshold), it is used to determine the continuity of adjacent boundary grids.

[0220] First grid length threshold: When detecting continuous boundary intervals in the first boundary in step 5051, it is used to determine whether the length of the continuous boundary grid can be used as a continuous boundary.

[0221] Second grid length threshold: used in step 506 to determine whether to delete the boundary in order to correct the boundary;

[0222] First grid number threshold: used in step 503 to detect isolated points;

[0223] Second grid number threshold: Used to determine the correctness of the boundary grid when detecting the correctness of the boundary grid in the first boundary in step 5052;

[0224] The third grid number threshold is used in step 506 to determine whether to delete the boundary in order to correct the boundary.

[0225] Example 1

[0226] See Figure 2 As shown, Figure 2 This is a schematic flowchart of an obstacle detection method based on depth images according to Embodiment 1 of this application. On the mobile robot side, the method includes, after acquiring a depth image, ...

[0227] Step 201: Remove depth points in the depth image that are less than a set height distance from the bearing plane where the mobile robot is located, and obtain a valid image.

[0228] Given that this application detects obstacles by projecting depth points from a depth image onto the bearing plane of the mobile robot body, and then using the projected points on the bearing plane, it is necessary to extract effective depth points to obtain an effective image that includes these effective depth points, in order to avoid a large number of ground depth points being mistaken for obstacles. The specific method is as follows:

[0229] Step 2011: Based on the depth image, obtain the spatial information of the spatial point P corresponding to the depth point p in the depth image, so as to obtain the three-dimensional coordinate information of the spatial point P. For ease of description, it will be referred to as the first spatial point below.

[0230] Taking a stereo camera as an example, in this step, for any depth point in the depth image, the coordinate information of the depth point p in the world coordinate system can be calculated based on the intrinsic and extrinsic parameters of the stereo camera and the baseline distance between the stereo cameras.

[0231] Since depth points below the vanishing line in a depth image better reflect physical spatial information, and in order to reduce computational load, depth points below the vanishing line in the depth image can optionally be obtained.

[0232] Step 2012: Based on the three-dimensional coordinate information of the first spatial point P, the three-dimensional coordinates of the first spatial point P are transformed into two-dimensional coordinates in a first plane that is parallel to the camera optical axis and perpendicular to the bearing plane, to obtain a two-dimensional spatial point P', which will be referred to as the second spatial point for ease of description.

[0233] See Figure 3 As shown, Figure 3 A schematic diagram for acquiring depth images for a mobile robot, see [link / reference] Figure 4 As shown, Figure 4 for Figure 3 A schematic diagram of a side view in a camera coordinate system, which is a view with the yczc plane as the cross section. Figure 3 In the coordinate system, the camera coordinate system is: the zc axis along the camera optical axis, perpendicular to the camera optical axis zc, and pointing towards the mobile robot's bearing plane. Figure 3The coordinate system of the mobile robot is as follows: the yc axis (vertically downward) is perpendicular to the yczc plane, and the xc axis is perpendicular to the yczc plane. The mobile robot coordinate system is as follows: the forward direction (depth) of the mobile robot body is the xb axis, the zb axis (height direction) is perpendicular to the mobile robot's bearing surface and vertically upward, and the yb axis (lateral direction) is perpendicular to the xbzb plane. The xb axis is parallel to the zc axis but in opposite directions; the zb axis is parallel to the yc axis but in opposite directions; and the yb axis is parallel to the xc axis but in opposite directions. Transforming the first spatial point P to the xbzb plane in the mobile robot coordinate system is equivalent to projecting it onto the xbzb plane in the mobile robot coordinate system. This xbzb plane is parallel to the camera's optical axis and perpendicular to the mobile robot's bearing surface (the xbyb plane in the mobile robot coordinate system).

[0234] Step 2013: Calculate the height gradient of the second spatial point along the camera optical axis to obtain the height gradient between the two second spatial points in the depth direction.

[0235] The camera's optical axis can also be understood as the depth direction, which is the forward and backward direction relative to the mobile robot. The height direction is the yc axis of the camera coordinate system, which is the height direction of the mobile robot body, i.e., the zb axis of the mobile robot coordinate system.

[0236] In this step, for any second spatial point, its height gradient along the camera's optical axis is calculated. For example, for any second spatial point, adjacent second spatial points are selected, and the ratio of their height difference to their distance difference is calculated, expressed mathematically as:

[0237]

[0238] Where dh represents the height difference between the two second spatial points, and dz represents the distance difference between the two second spatial points along the camera optical axis (zc axis direction).

[0239] The second spatial point adjacent to the second spatial point includes: a second spatial point located in the zc-axis direction at a position adjacent to the second spatial point in front of it; a second spatial point located in the zc-axis direction at a position adjacent to the second spatial point behind it; and a second spatial point located in the zc-axis direction in the height direction adjacent to the second spatial point in front of / behind it. For example... Figure 4 The second spatial points 4 and 5 in the diagram.

[0240] like Figure 4In the diagram, based on the distance and height distance between second spatial points 1 and 2 along the zc axis, the height gradient between them can be obtained. Similarly, the height gradients between second spatial points 3 and 4 and 5 along the zc axis can be obtained. It can be seen that when the distance between the two preceding and succeeding second spatial points along the zc axis is the same, a larger height gradient indicates a greater likelihood of the point being an obstacle. Based on this, ground depth points can be eliminated using the difference in height gradient along the zc axis.

[0241] To improve the reliability of using height gradient differences along the zc axis to eliminate spatial points on the ground, preferably, for any second spatial point, the height gradient between that second spatial point and a second spatial point having a set first distance threshold along the zc axis is calculated. This ensures that the height gradient calculation can be performed under the premise of the same current distance difference, thereby avoiding unreliable representation of obstacle information by the height gradient due to differences in distance difference. Thus, in step 2013, the distance between the two preceding and following second spatial points along the zc axis can satisfy the current first distance threshold.

[0242] Step 2014: Based on the set height gradient threshold, select height gradients that are greater than the height gradient threshold to obtain valid second spatial points. Based on the valid second spatial points, the valid depth points in the depth image can be obtained.

[0243] In this step, each calculated height gradient is compared with a height gradient threshold.

[0244] If the height gradient is greater than the height gradient threshold, then the two second spatial points used to calculate that height gradient are retained.

[0245] If the height gradient is not greater than the height gradient threshold, the two second spatial points used to calculate the height gradient are removed, thereby eliminating spatial points on the ground and preventing them from being identified as obstacles.

[0246] Since a valid second spatial point corresponds to a valid first spatial point, and a valid first spatial point corresponds to a depth point in the depth image, the valid depth points in the depth image can be obtained.

[0247] Step 2015: Determine whether the height gradients of all second spatial points have been calculated. If so, based on all valid depth points and the depth image, obtain the valid image of the depth image, and then proceed to step 202. Otherwise, update the first distance threshold, for example, by increasing or decreasing the first distance threshold according to a set step size, and return to step 2013.

[0248] Step 202: Obtain the projection of the effective depth points in the valid image onto the bearing plane of the mobile robot body to obtain the projection points;

[0249] In this step, the effective depth points in the depth image can be projected onto the bearing plane of the mobile robot body to obtain the effective depth points of the bearing plane after projection; alternatively, the effective first spatial point corresponding to the effective second spatial point can be projected onto the bearing plane of the mobile robot body to obtain the effective first spatial point of the bearing plane after projection. For ease of description, the effective depth points of the bearing plane after projection and the effective first spatial point of the bearing plane after projection can both be referred to as projection points.

[0250] Projection can be performed through coordinate system transformation. For example,

[0251] For any valid depth point, based on the coordinate information of the valid depth point, the valid depth point is transformed into two-dimensional coordinate information in the bearing plane under the coordinate system of the mobile robot, and the coordinate information of the projection point of the valid depth point is obtained.

[0252] or,

[0253] For any effective depth point, the three-dimensional coordinate information of the first spatial point corresponding to the effective depth point is converted into two-dimensional coordinate information in the bearing plane under the coordinate system of the mobile robot to obtain the coordinate information of the projection point of the effective depth point.

[0254] The bearing plane of the mobile robot body is the xbyb plane in the mobile robot coordinate system, which is usually the ground plane.

[0255] Step 203: Delete isolated projection points.

[0256] In this step, the projection points are traversed. If the number of projection points in the neighborhood of the current projection point is less than a set threshold for the number of first projection points, then the projection point is determined to be an invalid projection point, i.e., an isolated point, and is deleted. Preferably, if there are no other projection points in the neighborhood of the first projection point, then the projection point is determined to be an invalid projection point, i.e., an isolated point, and is deleted.

[0257] The neighborhood range of the first projection point can be set as needed.

[0258] Step 204: Select projection points that are less than a set distance threshold from the mobile robot, and use the selected projection points as the first boundary.

[0259] Preferably, based on the coordinate information of the projection points, the projection points closest to the mobile robot are selected, and the selected projection points are used as the first boundary.

[0260] Step 205: Cluster the projection points in the first boundary and remove erroneous projection points to optimize the boundary.

[0261] To improve obstacle detection accuracy due to various false detections, clustering filtering and boundary optimization are performed based on projection points. Since erroneous projection points are few and have small areas, filtering is performed by statistically analyzing the continuity of the first boundary projection points. Specifically:

[0262] Step 2051: Detect discontinuous and continuous boundary intervals within the first boundary.

[0263] In this step, the first boundary is divided into two types of boundary intervals: continuous and discontinuous. Once one type of boundary interval is detected, the other type of boundary interval can be determined. Thus, continuous projection points in the first boundary can be detected based on the distance between projection points being less than a set projection point distance threshold, and / or discontinuous projection points in the first boundary can be detected based on the distance between projection points being not less than a set projection point distance threshold. Continuous boundary intervals can be detected based on the length of the boundary formed by continuous projection points being greater than a set first projection point length threshold, and / or discontinuous boundary intervals can be detected based on the boundary interval containing discontinuous projection points in the first boundary or the length of the boundary formed by continuous projection points not being greater than a set first projection point length threshold, thereby determining the continuous and discontinuous boundary intervals in the first boundary.

[0264] in,

[0265] The method for detecting continuous boundary intervals in the first boundary can be:

[0266] When adjacent projection points in the first boundary are continuous and the length of the boundary with continuous projection points is greater than the first projection point length threshold, then the boundary with continuous projection points is a continuous boundary interval in the first boundary; continuous between adjacent projection points means that the distance between adjacent projection points is less than a set projection point distance threshold.

[0267] The method for detecting discontinuous boundary intervals in the first boundary can be:

[0268] When adjacent projection points in the first boundary are discontinuous, and / or the length of the boundary with continuous projection points is not greater than the set first projection point length threshold, then the adjacent projection points and / or the boundary with continuous projection points are the discontinuous boundary intervals in the first boundary; discontinuity between adjacent projection points means that the distance between adjacent projection points is not less than the set projection point distance threshold.

[0269] Preferably, when the distance between the mobile robot and the first boundary is less than the set second distance threshold, that is, when the distance between the mobile robot and the first boundary is relatively close, the projection point distance threshold is the first projection point distance threshold.

[0270] When the distance between the mobile robot and the first boundary is not less than the set second distance threshold, that is, when the distance between the mobile robot and the first boundary is relatively far, the projection point distance threshold is the second projection point distance threshold.

[0271] The first projection point distance threshold is greater than the second projection point distance threshold. Thus, when the mobile robot is far from the first boundary, the distance between consecutive projection points can be small. Conversely, when the mobile robot is close to the first boundary, the distance between consecutive projection points can be large as the accuracy of the projection points improves.

[0272] Preferably, the method for simultaneously detecting continuous and discontinuous boundary intervals within the first boundary includes:

[0273] Determine whether the distance between adjacent projected points in the first boundary is less than a projection point distance threshold. If so, determine that the adjacent projected points are continuous.

[0274] Determine whether the length of the boundary formed by consecutive adjacent projection points is greater than the first projection point length threshold. If so, determine that the boundary is a consecutive boundary interval in the first boundary.

[0275] If adjacent projection points are discontinuous, and / or the length of the boundary formed by adjacent projection points is not greater than the first projection point length threshold, then the boundary interval formed by the discontinuous adjacent projection points, and / or the boundary interval formed by adjacent projection points whose boundary length is not greater than the first projection point length threshold, is determined to be a discontinuous boundary interval in the first boundary.

[0276] Repeat the process until all projection points in the first boundary have been detected.

[0277] Step 2052: For the projected points in the discontinuous boundary interval, check their correctness. Determine whether the detected projected point is correct. If correct, retain the projected point; otherwise, proceed to step 2053.

[0278] In this step, for any projection point in a discontinuous boundary interval, the correctness of the projection point is determined by whether there is a projection point within the neighborhood of the second projection point.

[0279] If the number of projection points within the neighborhood of the second projection point is greater than the set threshold for the number of second projection points, then the projection point is determined to be a correct projection point.

[0280] Otherwise, the projection point is determined to be an incorrect projection point.

[0281] Step 2053: Starting from the erroneous projection point, sequentially filter the projection points along the depth direction to select a correct projection point. Use the selected correct projection point as the projection point in the boundary. Repeat this step until every erroneous projection point is filtered out. Use the continuous boundary interval, the projection point retained in step 2052, and the selected correct projection point as the boundary after the obstacle is updated, i.e., the second boundary.

[0282] Step 206: When the distance between the mobile robot and the second boundary is less than the set third distance threshold, the projection points in the second boundary are filtered in order to optimize the boundary in the case of close proximity.

[0283] In this embodiment, given that obstacles can be seen more clearly when the mobile robot is closer to them, optionally, if the distance between the mobile robot and the obstacle (i.e., the second boundary) is less than a set third distance threshold, the number and / or length of projection points in the second boundary are counted. If the number of projection points in the counted boundary is less than a set third projection point number threshold, and / or the length of projection points in the counted boundary is less than a set second projection point length threshold, then the boundary interval where the counted projection points are located is determined to be an incorrect boundary interval, and the boundary interval is deleted, thereby correcting the boundary.

[0284] When the distance between the mobile robot and the second boundary is not less than the third distance threshold, the second boundary obtained in step 205 is retained until the distance between the mobile robot and the second boundary is less than the third distance threshold.

[0285] It should be understood that step 206 can also be used to filter the first boundary, i.e., it can be performed after step 204. In addition, step 206 can also filter the first boundary after step 204 and the second boundary after step 205 to correct the current boundary.

[0286] Step 207: Take the current second boundary as the obstacle detection result and output it.

[0287] This embodiment removes ground depth points by using height gradients and deletes isolated projection points. It also uses clustering filtering of projection points in the boundary to remove erroneous projection points in the boundary and uses correct projection points as projection points in the boundary. Furthermore, by correcting the boundary interval in close-range cases, it reduces the dependence of obstacle detection on depth image quality and improves the accuracy of obstacle detection.

[0288] Example 2

[0289] See Figure 5 As shown, Figure 5This is a schematic diagram illustrating a process for determining obstacle boundaries in the obstacle detection method based on depth images of this application. The method includes:

[0290] Step 501: Remove depth points in the depth image whose distance from the bearing plane where the mobile robot is located is less than a set height distance, to obtain a valid image. This step is the same as step 201.

[0291] Step 502: Given the huge number of depth points in the depth image, the depth points can be represented as a point cloud, and the corresponding number of projection points will also be huge. In order to facilitate the analysis of the distribution of projection points and the processing of projection points, a two-dimensional grid is constructed to distribute the projection points in the bearing plane into each grid, so as to obtain a grid map used to characterize the distribution of projection points.

[0292] The size of the grid with the minimum area can be set according to the desired distribution of projection points, obstacle size, and other factors. Optionally, each projection point is located in a grid so that the grid and the projection point correspond. More preferably, a certain proportion of projection points are located in the grid, so that the grid may include more than one projection point.

[0293] To facilitate obstacle detection and avoidance by the mobile robot during its movement, the column grids of the two-dimensional grid are arranged along the current depth direction of the mobile robot, for example, along... Figure 3 In the coordinate system of the mobile robot, column grids are arranged along the xb direction (depth direction), and the row grids are arranged perpendicular to the column grids. For example, along... Figure 3 The yb direction of the mobile robot coordinate system. To facilitate the description of the positional information of each grid in the grid image, the grid image can be regarded as an image plane and described using an image coordinate system. That is, the upper left corner of the grid image is the origin, the horizontal direction to the right is the u-axis, and the vertical direction downward is the v-axis.

[0294] To facilitate the selection of projection points, the grid is labeled according to the distribution of projection points in the grid:

[0295] For any given grid,

[0296] If there are no projection points in a grid, it indicates that the grid belongs to an empty area, and a first label is assigned to the grid to identify the empty area attribute.

[0297] If a grid has a projection point, and the adjacent boundary grid in the same column has no projection point, it indicates that the grid may belong to an obstacle boundary. The grid is then assigned a second label to identify the boundary attribute. Preferably, if the grid in the same column adjacent to the grid has no projection point in the grid closer to the mobile robot (the grid with larger position coordinates in the image coordinate system) and / or has a projection point in the grid farther from the mobile robot (the grid with smaller position coordinates in the image coordinate system), then the grid is likely to belong to an obstacle boundary. The grid is then assigned the second label. The distance closer to the mobile robot and the distance farther from the mobile robot are set as needed.

[0298] If a grid cell contains a projection point, and an adjacent boundary grid cell in the same column also contains a projection point, then the grid cell may be an obstacle. Therefore, a third label is assigned to the grid cell to identify its obstacle attributes.

[0299] Thus, a raster map can include boundary rasters, obstacle rasters, and open rasters.

[0300] The identification can be done using pixel values. For example, setting the pixel values ​​of a raster. See also Figure 6 As shown, Figure 6 This is a schematic diagram of a projection point grid map. In the diagram, the diagonal grid lines represent open areas, the white grid lines represent obstacle boundaries, and the gray grid lines represent obstacles.

[0301] Step 503: Delete the projected points in isolated rasters, or mark isolated rasters in the raster map as empty rasters.

[0302] In this step, the grids in the grid map are traversed. If the number of empty grids in the neighborhood of the first grid of the currently traversed grid is less than the set threshold for the number of first grids, or the number of projection points is less than the set threshold for the number of first projection points, then the grid is determined to be an isolated grid and the projection points in the grid are invalid projection points. That is, the grid or the projection points in the grid are isolated points. The projection points in the grid are deleted, or the isolated grids in the grid map are marked as empty grids.

[0303] The neighborhood of the first grid cell can be set as needed, for example, setting m×n grid cells as the neighborhood. Optionally, if the neighborhood of the first grid cell of the currently traversed grid cell is entirely empty and there are no other projection points within the neighborhood of the first grid cell, then the grid cell is determined to be an isolated point, and its projection points are deleted.

[0304] To improve the efficiency of isolated point deletion, it is preferable to traverse all non-empty grid cells in the raster graph.

[0305] See Figure 7As shown in the figure, a grid is an isolated point if there are no other grids with projection points in the neighborhood of the grid.

[0306] Step 504: Extract the boundary grid cell with the maximum position coordinates and boundary marker in each column of the grid map, and use it as the first boundary.

[0307] Since all potentially valid obstacle points are projected onto the grid image, the side closer to the mobile robot is the projection point that is of greater concern during obstacle avoidance, i.e. the projection point that the mobile robot is likely to collide with first. Therefore, the obstacle boundary extracted from the grid image is the grid with the boundary marker that is closest to the mobile robot in each column of grids, which is the projection point with the largest position coordinates in the image coordinate system.

[0308] Because there may be several incorrect projection points in front of the mobile robot—locations where there are actually no obstacles, possibly due to incorrect detection of lights or the ground—the extracted first boundary is a coarse boundary. For example, Figure 6 The first boundary is composed of white boundary grids.

[0309] Step 505: Cluster the boundary grids in the first boundary for boundary optimization.

[0310] To improve obstacle detection accuracy due to various false detections, clustering filtering and boundary optimization based on the grid image are performed. Since erroneous projection points are few and have small areas, filtering is performed by statistically analyzing the continuity of the first boundary. Specifically:

[0311] Step 5051: Detect discontinuous and continuous boundary intervals within the first boundary.

[0312] In this step, given that the first boundary includes both continuous and discontinuous boundary intervals, detecting one type of boundary interval determines the other. Therefore, continuous boundary grids in the first boundary can be detected based on the distance between grids being less than a set grid distance threshold, and / or discontinuous boundary grids in the first boundary can be detected based on the distance between grids being not less than a set grid distance threshold. Continuous boundary intervals can be detected based on the boundary length formed by continuous boundary grids being greater than a set first grid length threshold, and / or discontinuous boundary intervals can be detected based on the boundary interval containing the discontinuous boundary grids in the first boundary, or the boundary length formed by continuous boundary grids being not greater than a set first grid length threshold, thereby determining the continuous and discontinuous boundary intervals in the first boundary.

[0313] The method for detecting continuous boundary intervals in the first boundary can be as follows:

[0314] When adjacent boundary grids in the first boundary are continuous and the length of the boundary with continuous boundary grids is greater than a length threshold, then the boundary with continuous boundary grids is a continuous boundary interval in the first boundary. Continuity between adjacent boundary grids is defined as follows: the distance between adjacent boundary grids is less than a set grid distance threshold, or the number of adjacent boundary grids is less than a set boundary grid number threshold. Preferably, if there are grids in the column grids containing two adjacent boundary grids in the first boundary whose distance in the column direction is less than the set grid distance threshold, or if the column grids to which the two adjacent boundary grids belong include other boundary grids whose distance in the column direction is less than the grid distance threshold, then the two adjacent boundary grids are continuous. Alternatively, if there are grids in the column grids containing two adjacent boundary grids in the first boundary whose distance in the column direction is less than the set boundary grid number threshold, or if the column grids to which the two adjacent boundary grids belong include other boundary grids whose distance in the column direction is less than the boundary grid number threshold, then the two adjacent boundary grids are continuous. The column grid to which the two adjacent boundary grids belong refers to all grids located in the same column as the two boundary grids. According to the above detection method, Figure 6 In the middle, grids 8 to 19 in the boundary are all continuous grids.

[0315] The method for detecting discontinuous boundary intervals in the first boundary can be:

[0316] When adjacent boundary grids in the first boundary are discontinuous, and / or the boundary length of a grid with continuous boundary grids is not greater than the set first grid length threshold, then the adjacent boundary grids and / or the boundary with continuous boundary grids are the discontinuous boundary intervals in the first boundary.

[0317] Discontinuity between adjacent boundary grids is defined as follows: the distance between adjacent boundary grids is not less than a set grid distance threshold, or the number of adjacent boundary grids is not less than a set boundary grid number threshold. Preferably, if the distance between two adjacent boundary grids in the column direction in the first boundary is not less than the set grid distance threshold, and the column grids to which the two adjacent boundary grids belong do not include other boundary grids whose distance in the column direction is less than the grid distance threshold, or the number of grids in the column direction of two adjacent boundary grids in the first boundary is not less than the set boundary grid number threshold, and the column grids to which the two adjacent boundary grids belong do not include other boundary grids whose number of grids in the column direction is less than the boundary grid number threshold, then the two adjacent boundary grids are discontinuous.

[0318] For ease of understanding, combined with Figure 6To illustrate, if grids 6 and 8 are adjacent in the row direction but significantly apart in the column direction, and there are no other boundary grids between the column grids containing grids 6 and 8 such that the distance in the column direction is less than the grid distance threshold, then grids 6 and 8 are discontinuous. However, if the column grids to which grid 6 belongs and / or grid 8 belongs include other boundary grids such that the distance in the column direction is less than the grid distance threshold, for example... Figure 6 If grid 8' is a boundary grid, it is located in the column grid to which grid 8 belongs. That is, grid 8' and grid 8 are located in the same column grid. The distance between grid 8' and grid 6 in the column direction is less than the grid distance threshold. That is, the column grid to which grid 8 belongs includes boundary grid 8' such that the distance in the column direction between the column grids to which two adjacent boundary grids belong is less than the grid distance threshold. In this case, grid 6 and grid 8 are continuous grids.

[0319] Preferably, when the distance between the mobile robot and the first boundary is less than the set second distance threshold, that is, when the distance between the mobile robot and the first boundary is relatively close, the grid distance threshold is the first grid distance threshold, and the boundary grid number threshold is the first boundary grid number threshold.

[0320] When the distance between the mobile robot and the first boundary is not less than the set second distance threshold, that is, when the distance between the mobile robot and the first boundary is relatively far, the projection point distance threshold is the second grid distance threshold, and the boundary grid number threshold is the second boundary grid number threshold.

[0321] The first grid distance threshold is greater than the second grid distance threshold, and the first boundary grid number threshold is greater than the second boundary grid number threshold. Thus, when the mobile robot is far from the first boundary, the distance between consecutive boundary grids can be small. Conversely, when the mobile robot is close to the first boundary, the distance between consecutive boundary grids can be large as the accuracy of the projection point improves.

[0322] Preferably, the method for simultaneously detecting continuous and discontinuous boundary intervals within the first boundary includes:

[0323] Determine whether the distance between adjacent boundary grids in the first boundary is less than a grid distance threshold, or whether the number of adjacent boundary grids in the first boundary is less than a boundary grid number threshold. If so, then the adjacent boundary grids are determined to be continuous.

[0324] Determine whether the length of the boundary composed of consecutive adjacent boundary grids is greater than the first grid length threshold. If so, determine that the boundary is a consecutive boundary interval in the first boundary.

[0325] If adjacent boundary grids are discontinuous, and / or the length of the boundary formed by adjacent boundary grids is not greater than the first grid length threshold, then the boundary interval formed by the discontinuous adjacent boundary grids, and / or the boundary interval formed by adjacent boundary grids whose boundary length is not greater than the first grid length threshold, is determined to be a discontinuous boundary interval in the first boundary.

[0326] Repeat the process until all boundary grids in the first boundary have been detected.

[0327] For example, when the distance between the mobile robot and the first boundary is less than the set second distance threshold, that is, when the distance between the mobile robot and the first boundary is relatively close, it is determined whether the distance between two adjacent boundary grids in the column direction is not less than the set first grid distance threshold, and whether there are other boundary grids in the column grids to which the two adjacent boundary grids belong, such that the distance in the column direction is less than the set first grid distance threshold. Alternatively, it is determined whether the number of grids in the column direction of two adjacent boundary grids in the first boundary is not less than the set first boundary grid number threshold, and whether there are other boundary grids in the column grids to which the two adjacent boundary grids belong, such that the number of grids in the column direction is less than the set first boundary grid number threshold. If so, the boundary grids in the two adjacent columns are determined to be continuous boundary grids; otherwise, they are determined to be non-continuous boundary grids. This process is repeated until all the boundary grids in all columns of the first boundary grids have been detected, and then a boundary composed of continuous boundary grids is obtained.

[0328] When the distance between the mobile robot and the first boundary is not less than the second distance threshold, that is, when the distance between the mobile robot and the first boundary is far, it is determined whether the distance between two adjacent boundary grids in the column direction is not less than the set second grid distance threshold, and whether there are other boundary grids in the column grids to which the two adjacent boundary grids belong, such that the distance in the column direction is less than the set second grid distance threshold. Alternatively, it is determined whether the number of grids in the column direction of two adjacent boundary grids in the first boundary is not less than the set first boundary grid number threshold, and whether there are other boundary grids in the column grids to which the two adjacent boundary grids belong, such that the number of grids in the column direction is less than the set second boundary grid number threshold. If so, the boundary grids in the two adjacent columns are determined to be continuous boundary grids; otherwise, they are determined to be non-continuous boundary grids. This process is repeated until all the boundary grids in all columns of the first boundary grids have been detected, and then a boundary composed of continuous boundary grids is obtained.

[0329] Wherein, the first grid distance threshold is greater than the second grid distance threshold, or the first boundary grid number threshold is greater than the second boundary grid number threshold. In this way, when the distance between the mobile robot and the first boundary is far, the column direction distance between consecutive boundary grids in the grid diagram can be small. Conversely, when the distance between the mobile robot and the first boundary is close, the column direction distance between consecutive boundary grids in the grid diagram can be large.

[0330] If the length of the boundary formed by continuous boundary grids is greater than the set first grid length threshold, the boundary interval is determined to be a continuous boundary interval; otherwise, it is determined to be a non-continuous boundary interval.

[0331] Follow these steps, as follows Figure 6 As shown in the figure, grids 8 to 19 are continuous boundary intervals, while grids 6, 20, and 21 are non-continuous boundary intervals.

[0332] Step 5052: For boundary grids in discontinuous boundary intervals, check the correctness of the boundary grids. Determine whether the checked boundary grid is correct. If correct, retain the boundary grid; otherwise, proceed to step 5053.

[0333] In this step, for any boundary grid cell within a discontinuous boundary interval, the correctness of the projection points in the boundary grid cell or the boundary grid cell itself is determined based on the projection points present in the grid cells within the neighborhood of the second grid cell of that boundary grid cell, or the number of non-empty grid cells present.

[0334] If the number of non-empty grid cells in the neighborhood of the second grid cell of the boundary grid exceeds a set threshold for the number of second grid cells, or if the number of projected points in the grid cells in the neighborhood of the boundary grid exceeds a set threshold for the number of boundary projected points, then the projected points in the boundary grid are determined to be correct, and the boundary grid is considered a correct grid.

[0335] Otherwise, the projection point in the boundary grid is determined to be incorrect, and the boundary grid is considered an erroneous grid.

[0336] For example, such as Figure 6 In this example, assuming the neighborhood of boundary grid 1 is 3×3 grids and the threshold for the number of second grids is 4, and there is only one grid 2 in the neighborhood of boundary grid 1, it does not meet the grid correctness judgment condition. Therefore, boundary grid 1 is judged as an incorrect grid. Similarly, boundary grid 3 will also be judged as an incorrect grid.

[0337] Step 5053: Starting from the erroneous raster, sequentially filter the rasters in the same column as the erroneous raster along the depth direction to select a correct raster, and update the selected correct raster as the boundary raster; repeat this step until each erroneous raster is updated; use the continuous boundary interval raster, the retained boundary raster, and the updated boundary raster as the second boundary.

[0338] In this step, when a boundary grid is an incorrect grid, it means that the correct boundary grid should be located along the current depth direction of the mobile robot in the grid map. Therefore, starting from this boundary grid, the grid correctness is checked sequentially for all grids in the same column as the boundary grid along the mobile robot's travel direction. When a checked grid is correct, it is updated as a boundary grid to serve as the boundary of that column of grids; or, starting from the projection point in the boundary grid, the projection point correctness is checked sequentially for all projection points in the grids in the same column as the boundary grid along the mobile robot's travel direction, i.e., the current depth direction. When a checked projection point is correct, the grid containing that projection point is updated as a boundary grid. Figure 6 As shown, when the mobile robot's current movement direction is upward, since boundary grid 1 is an incorrect grid, the correct boundary grid should be above the grid in the same column as boundary grid 1 in the grid diagram. Therefore, for the grids in the same column as boundary grid 1, the correctness of each grid is checked one by one upward, starting from boundary grid 1. According to the grid correctness judgment condition assumed above, if grid 5 is detected as a correct grid, then grid 5 is updated as a boundary grid. In this way, the two isolated boundary grids 1 and 3 will be deleted, and the boundary grid will move upward to the actual boundary. In step 506, when the distance between the mobile robot and the obstacle is less than the set third distance threshold, the current second boundary itself is filtered in order to perform boundary optimization in close-range situations.

[0339] Given that obstacles are more clearly visible when the mobile robot is closer to them, optionally, if the distance between the mobile robot and the obstacle is less than a set third distance threshold, the number and / or length of boundary grids in the second boundary are counted. If the counted number of boundary grids is less than a set third grid number threshold, and / or the counted length of the boundary grid is less than a set second length threshold, and / or the number of projection points in the boundary grid is less than a set third projection point number threshold, then the boundary grid is determined to be an erroneous grid, and the boundary grid is deleted, or the projection points in the boundary grid are deleted.

[0340] When the distance between the mobile robot and the obstacle is not less than the third distance threshold, the second boundary obtained in step 505 is retained until the distance between the mobile robot and the obstacle is less than the third distance threshold.

[0341] Step 507: Take the current second boundary as the obstacle detection result and output it.

[0342] In the above steps, it should be understood that the determination and processing of grids in the grid diagram can also be understood as the determination and processing of projection points in the grid; the first grid neighborhood range used for isolated point deletion in step 503 and the second grid neighborhood range used for grid correctness detection in step 5052 may be different.

[0343] This application addresses the problem of erroneous depth points in the depth image point cloud output by a depth camera. It projects these depth points onto a two-dimensional grid map of the mobile robot's carrying plane, reducing the spatial three-dimensional information to two-dimensional information. This allows for both three-dimensional perception information and rapid path planning in a two-dimensional environment. Furthermore, it enables efficient filtering of erroneous projection points through various filtering strategies, reducing the requirements for the depth image quality of the depth camera. Even with low-quality depth images, obstacle detection is still possible, which helps improve the reliability of obstacle avoidance for the mobile robot.

[0344] See Figure 8 As shown, Figure 8 This is a schematic diagram of a depth image-based obstacle detection device according to an embodiment of this application. The device includes,

[0345] The effective depth point extraction module is used to remove depth points in the depth image whose height from the mobile robot's bearing plane is less than a set height threshold, thus obtaining an effective image.

[0346] The projection module is used to project the effective image from the depth image onto the bearing plane to obtain the projection points on the bearing plane.

[0347] The boundary extraction module is used to extract projection points that are less than a set distance threshold from the mobile robot based on the projection points in the bearing plane, thereby obtaining the first boundary.

[0348] Furthermore, the device further includes at least one of the following modules:

[0349] The isolated point processing module is used to delete isolated projected points.

[0350] The first filtering module is used to cluster and filter the projection points in the first boundary, removing erroneous projection points to obtain the second boundary.

[0351] The second filtering module is used to correct the current boundary when the distance between the mobile robot and the current boundary is less than a set boundary distance threshold. For example, when the distance between the mobile robot and the obstacle is less than a set third distance threshold, the module counts the number and / or length of the projection points in the first boundary and / or the second boundary. When the number of projection points in the counted boundary is less than a set third projection point number threshold, and / or the length of the projection points in the counted boundary is less than a set second projection point length threshold, the boundary interval where the counted projection points are located is deleted.

[0352] The projection module also includes,

[0353] The projection submodule is used to project the effective image from the depth image onto the bearing plane to obtain the projection points on the bearing plane.

[0354] The rasterization submodule is used to construct a two-dimensional raster, distributing the projection points in the carrying plane into each raster to obtain a raster map that characterizes the distribution of projection points, and identifying each raster based on the projection points in the raster.

[0355] This invention also provides a mobile robot, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to implement any of the above-described steps of obstacle detection based on depth images.

[0356] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0357] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0358] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-described steps for obstacle detection based on depth images.

[0359] For the device / network-side equipment / storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0360] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0361] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An obstacle detection method based on depth images, characterized in that, The method includes: Acquire depth image, Depth points in the depth image whose height from the mobile robot's bearing plane is less than a set height threshold are removed to obtain valid images; The effective image is projected onto the bearing plane to obtain the projection points in the bearing plane. Based on the projection points in the bearing plane, the projection points that are less than a set distance threshold from the mobile robot are extracted to obtain the first boundary of the obstacle. in, The process of removing depth points in the depth image whose height relative to the mobile robot's bearing plane is less than a set height threshold yields a valid image, including: Calculate the height gradient of depth points in the depth image along the camera's optical axis. Based on the set height gradient threshold, height gradients greater than the height gradient threshold are filtered out, and the depth points used to calculate the height gradient are retained. Height gradients not greater than the height gradient threshold are removed, and the depth points used to calculate the height gradient are also removed, resulting in a valid image containing valid depth points.

2. The method as described in claim 1, characterized in that, The method further includes at least one of the following processes: Processing for deleting isolated projection points in the bearing plane; This is used to cluster and filter the projection points in the first boundary, remove erroneous projection points, and obtain the processing of the second boundary; This is used to correct the current boundary when the distance between the mobile robot and the current boundary is less than a set boundary distance threshold.

3. The method as described in claim 2, characterized in that, The step of projecting the effective image onto the bearing plane to obtain projection points in the bearing plane includes: The coordinate information of the effective depth points in the effective image is transformed to the bearing plane in the coordinate system of the mobile robot to obtain the projection points.

4. The method as described in claim 3, characterized in that, The calculation of the height gradient of depth points in the depth image along the camera optical axis includes: Based on the depth image, determine the three-dimensional coordinate information of the first spatial point corresponding to the depth point in the depth image in the world coordinate system; Based on the three-dimensional coordinate information of the first spatial point, the first spatial point is converted into two-dimensional coordinate information in a first plane that is parallel to the camera optical axis and perpendicular to the bearing plane, and the second spatial point information is obtained. For any second spatial point, calculate the height difference between the second spatial point and another second spatial point, as well as the distance difference along the camera optical axis, and use the ratio of the height difference to the distance difference as the height gradient of the second spatial point. The process of converting the coordinate information of effective depth points in the effective image to the bearing plane in the coordinate system of the mobile robot to obtain the projection points includes: For any effective depth point, the three-dimensional coordinate information of the first spatial point corresponding to the effective depth point is converted into two-dimensional coordinate information in the bearing plane under the coordinate system of the mobile robot to obtain the coordinate information of the projection point of the effective depth point; or For any valid depth point, based on the coordinate information of the valid depth point, the valid depth point is transformed into two-dimensional coordinate information in the bearing plane under the coordinate system of the mobile robot, and the coordinate information of the projection point of the valid depth point is obtained.

5. The method as described in claim 4, characterized in that, The determination of the three-dimensional coordinate information of the first spatial point corresponding to the depth point in the depth image in the world coordinate system based on the depth image includes: Based on the depth image, the cancellation line in the depth image is identified, and depth points below the cancellation line are extracted. Determine the spatial information of the first spatial point corresponding to the extracted depth point; The step of calculating the height difference between any second spatial point and another second spatial point, as well as the distance difference along the camera's optical axis, further includes: Select a second spatial point that has a set first distance threshold along the camera's optical axis as another spatial point.

6. The method as described in any one of claims 2 to 5, characterized in that, The step of extracting projection points that are less than a set distance threshold from the mobile robot based on the projection points in the bearing plane includes: Based on the coordinate information of the projection points, the projection points closest to the mobile robot are selected, and the selected projection points are used as the first boundary. The process for clustering and filtering the projection points in the first boundary to remove erroneous projection points and obtain the second boundary includes: Detect discontinuous and continuous boundary intervals within the first boundary; Perform a correctness check on the projection points in the discontinuous boundary interval. Correct projection points should be retained. For incorrect projection points, starting from the incorrect projection point, proceed sequentially along the current depth direction to filter each projection point. When a correct projection point is found, use it as the projection point in the boundary. The projection points in the continuous boundary interval, the retained correct projection points, and the selected correct projection points are used as the second boundary. The process for deleting isolated projection points includes: Traverse all projection points, If the number of consecutive projection points in the neighborhood of the first projection point of the current projection point being traversed is less than the set threshold for the number of first projection points, then the current projection point is determined to be an isolated projection point and is deleted.

7. The method as described in claim 6, characterized in that, The detection of discontinuous boundary intervals and continuous boundary intervals in the first boundary includes: When adjacent projection points in the first boundary are not continuous, and / or the length of the boundary with continuous projection points is not greater than the set threshold for the length of the first projection point, then the adjacent projection points and / or the boundary with continuous projection points are the non-continuous boundary intervals in the first boundary. When adjacent projection points in the first boundary are continuous and the length of the boundary with continuous projection points is greater than the first projection point length threshold, then the boundary with continuous projection points is a continuous boundary interval in the first boundary. in, Discontinuity between adjacent projection points is defined as follows: the distance between adjacent projection points is not less than a set projection point distance threshold. Continuity between adjacent projection points is defined as follows: the distance between adjacent projection points is less than a set projection point distance threshold. The correctness detection of projection points in discontinuous boundary intervals includes: For any projection point within a discontinuous boundary interval, the correctness of the projection point is determined based on the projection points within the neighborhood of the second projection point. If the number of projection points in the neighborhood of the second projection point is greater than the set threshold for the number of second projection points, then the projection point is determined to be a correct projection point; otherwise, the projection point is determined to be an incorrect projection point.

8. The method as described in claim 7, characterized in that, The projection point distance threshold includes: When the distance between the mobile robot and the first boundary is less than the set second distance threshold, the projection point distance threshold becomes the first projection point distance threshold. When the distance between the mobile robot and the first boundary is not less than the set second distance threshold, then the projection point distance threshold is the second projection point distance threshold. Among them, the distance threshold of the first projection point is greater than the distance threshold of the first projection point.

9. The method as described in claim 2, characterized in that, The process for correcting the current boundary when the distance between the mobile robot and the current boundary is less than a set boundary distance threshold includes: If the distance between the mobile robot and the current boundary is less than the set third distance threshold, count the number and / or length of the projected points in the current boundary. When the number of projection points counted in the current boundary is less than the set third projection point number threshold, and / or the length of projection points counted in the current boundary is less than the set second projection point length threshold, delete the boundary interval where the counted projection points are located. The current boundary is at least one of the first boundary and the second boundary.

10. The method as described in any one of claims 2 to 5, characterized in that, After projecting the effective image onto the bearing plane to obtain the projection points in the bearing plane, the process further includes: A two-dimensional grid is constructed, and the projection points in the bearing plane are distributed in each grid to obtain a grid map that represents the distribution of projection points; the size of the grid with the smallest region is set as needed. For any given grid cell, label it according to the projection points within the grid cell: If there are no projection points in a grid, the grid is marked as an empty grid. If a raster has a projection point, and the adjacent boundary raster in the same column and / or row has no projection points, then the raster is identified as a boundary raster. If a grid has a projection point, and the adjacent boundary grid in the same column and / or row of the grid also has a projection point, then the grid is identified as an obstacle grid. The process for deleting isolated projection points includes: Traverse the non-empty cells in the raster graph. If the number of empty grid cells within the first grid cell neighborhood of the current grid cell being traversed is less than the set first grid cell number threshold, or the number of projected points is less than the set first projection point number threshold, then the grid cell is marked as an empty grid cell. The step of extracting projection points that are less than a set distance threshold from the mobile robot based on the projection points in the bearing plane includes: Extract the boundary grid cell in the grid map that is closest to the mobile robot, and use it as the first boundary; The process for clustering and filtering the projection points in the first boundary to remove erroneous projection points and obtain the second boundary includes: Detect discontinuous and continuous boundary intervals within the first boundary. Perform correctness checks on boundary grids within discontinuous boundary regions. Correct boundary grid cells should be preserved. For erroneous boundary grids, starting from the erroneous boundary grid, the grids in the grid map are filtered sequentially along the current depth direction. When a correct grid is found, the grid is updated as a boundary grid. The boundary grid in the continuous boundary interval, the retained boundary grid, and the updated boundary grid are used as the second boundary.

11. The method as described in claim 10, characterized in that, In the raster graph, the column grids are arranged along the current depth direction, and the row grids are arranged perpendicular to the column grids. The step of identifying any grid cell based on the projection points within the grid cell further includes: If a grid has a projection point and there is no projection point in the adjacent boundary grid in the same column, then the grid is identified as a boundary grid. Alternatively, if in the same column of grids adjacent to the grid, there is no projection point in the grid on the side closer to the mobile robot and / or there is a projection point in the grid on the side farther from the mobile robot, then the grid is identified as a boundary grid. The distance closer to the mobile robot and the distance farther from the mobile robot are set as needed. If a grid has a projection point, and the adjacent boundary grid in the same column of the grid also has a projection point, then the grid is identified as an obstacle grid. The first boundary is defined as the boundary grid cell in the extracted grid that is closest to the mobile robot, including: Extract the boundary grid cell closest to the mobile robot from each column of grid cells, and use it as the first boundary. The detection of discontinuous boundary intervals and continuous boundary intervals in the first boundary includes: When adjacent boundary grids in the first boundary are not continuous, and / or the boundary length of a continuous boundary grid is not greater than the set first grid length threshold, then the adjacent boundary grid and / or the boundary of the continuous boundary grid is a non-continuous boundary interval. When adjacent boundary grids in the first boundary are continuous and the length of the boundary with continuous boundary grids is greater than the first grid length threshold, then the boundary with continuous boundary grids is a continuous boundary interval. in, Discontinuity between adjacent boundary grids is defined as follows: the distance between adjacent boundary grids is not less than a set grid distance threshold, or the number of grids between adjacent boundary grids is not less than a set boundary grid number threshold. Continuity between adjacent boundary grids is defined as follows: the distance between adjacent boundary grids is less than a set grid distance threshold, or the number of adjacent boundary grids is less than a set boundary grid number threshold. The correctness detection of boundary grids in discontinuous boundary intervals includes: For any boundary grid cell within a discontinuous boundary interval, the correctness of the projection points in the boundary grid cell and / or the boundary grid cell itself is determined based on the projection points within the neighborhood of the first grid cell of that boundary grid cell. If the number of non-empty grids in the neighborhood of the first grid is greater than the set second grid number threshold, or if the number of projection points in the grids in the neighborhood of the first grid is greater than the set second projection point number threshold, then the projection points in the boundary grid and / or the boundary grid are determined to be correct; otherwise, they are determined to be incorrect. For erroneous boundary grids, starting from the erroneous boundary grid, the grids in the grid map are sequentially filtered along the current depth direction. When a correct grid is found, it is updated as a boundary grid, including: Starting with the erroneous boundary grid, sequentially filter the grids in the same column as the erroneous boundary grid along the current depth direction. When a correct grid is found, update the correct grid as the boundary grid. Repeat this process until every erroneous boundary grid cell is updated.

12. The method as described in claim 11, characterized in that, The determination that the distance between adjacent boundary grids is not less than the set grid distance threshold includes: if the distance between two adjacent boundary grids in the column direction is not less than the grid distance threshold, and the column grids to which the two adjacent boundary grids belong do not include other boundary grids that make the distance in the column direction less than the grid distance threshold, then the distance between the two adjacent boundary grids is not less than the set grid distance threshold. The determination that the number of grids between adjacent boundary grids is less than the set boundary grid number threshold includes: if the number of grids in the column direction of the two adjacent boundary grids is not less than the boundary grid number threshold, and the column grids to which the two adjacent boundary grids belong do not include other boundary grids that would make the number of grids in the column direction less than the boundary grid number threshold, then it is determined that the number of grids between adjacent boundary grids is less than the set boundary grid number threshold. The determination that the distance between adjacent boundary grids is less than a set grid distance threshold includes: if the distance between two adjacent boundary grids in the column direction is less than the grid distance threshold, or if the column grids to which the two adjacent boundary grids belong include other boundary grids that make the distance in the column direction less than the grid distance threshold, then the distance between the two adjacent boundary grids is less than the set grid distance threshold. The determination that the number of grids between adjacent boundary grids is less than the set boundary grid number threshold includes: if the number of grids in the column direction of two adjacent boundary grids is less than the boundary grid number threshold, or if the column grids to which the two adjacent boundary grids belong include other boundary grids that make the number of grids in the column direction less than the boundary grid number threshold, then the number of grids between adjacent boundary grids is less than the set boundary grid number threshold. Wherein, when the distance between the mobile robot and the first boundary is less than the set second distance threshold, the grid distance threshold is the first grid distance threshold, and the boundary grid number threshold is the first boundary grid number threshold; When the distance between the mobile robot and the first boundary is not less than the set second distance threshold, the grid distance threshold is the second grid distance threshold, and the boundary grid number threshold is the second boundary grid number threshold. Among them, the first grid distance threshold is greater than the second grid distance threshold; the first boundary grid number threshold is greater than the second boundary grid number threshold. The process for correcting the current boundary when the distance between the mobile robot and the current boundary is less than a set boundary distance threshold includes: If the distance between the mobile robot and the current boundary is less than the set third distance threshold, count the number and / or length of the boundary grid cells in the current boundary. If the number of boundary grid cells counted is less than the set third grid cell count threshold, and / or the length of the boundary grid cells counted is less than the set second grid cell length threshold, then the boundary containing the counted boundary grid cells is deleted.

13. A mobile robot, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the steps of obstacle detection based on depth images as described in any one of claims 1 to 12.

14. An obstacle detection device based on depth images, characterized in that, The device includes, The effective depth point extraction module is used to remove depth points in the depth image whose height from the mobile robot's bearing plane is less than a set height threshold, thus obtaining an effective image. The projection module is used to project the effective image from the depth image onto the bearing plane to obtain the projection points on the bearing plane. The boundary extraction module is used to extract projection points that are less than a set distance threshold from the mobile robot based on the projection points in the bearing plane, so as to obtain the first boundary of the obstacle. in, The process of removing depth points in the depth image whose height relative to the mobile robot's bearing plane is less than a set height threshold yields a valid image, including: Calculate the height gradient of depth points in the depth image along the camera's optical axis. Based on the set height gradient threshold, height gradients greater than the height gradient threshold are filtered out, and the depth points used to calculate the height gradient are retained. Height gradients not greater than the height gradient threshold are removed, and the depth points used to calculate the height gradient are also removed, resulting in a valid image containing valid depth points.

Images