Real-time reflection removal method, apparatus, electronic device and readable storage medium

Abstract

This application discloses a real-time reflection removal method, apparatus, electronic device, and readable storage medium. The real-time reflection removal method includes: acquiring point cloud data collected in real time by a depth camera, and segmenting the point cloud data into positive point cloud, ground point cloud, and negative point cloud; determining that the negative point cloud is a first reflection point cloud and deleting it if the number of positive point clouds is zero and the number of negative point clouds is greater than zero, and if the number of negative point clouds is less than a first preset threshold; performing a deduplication operation on the negative point cloud if both the number of positive and negative point clouds are greater than zero; and returning to the step of determining whether the number of negative point clouds is less than the first preset threshold if the number of negative point clouds after deduplication is greater than or equal to a second preset threshold. This application achieves real-time and accurate removal of ground reflections by acquiring point cloud data in real time and classifying and segmenting it, and by judging the difference in the number of positive and negative point clouds.

Description

Technical Field

[0001] This application relates to the field of optical inspection technology, and in particular to real-time reflection removal methods, apparatus, electronic devices, and readable storage media. Background Technology

[0002] In the field of robot control, especially the motion control of mobile robots, when they operate on smooth surfaces such as tiles, objects on the ground will cast reflections on the smooth surface. This causes the images captured by the mobile robot's camera to show the reflection below the ground, leading to incorrect judgments of a risk of falling ahead and preventing proper navigation and motion planning. Therefore, real-time and accurate removal of ground reflections is crucial during robot operation.

[0003] In existing technologies, neural networks are primarily used to train detection models to identify and remove reflections in images. Reflections in the depth map can also be removed by mapping the identified reflection regions of the image. However, the reflection image is not necessarily perfectly symmetrical to the object, and the reflection image is often unclear, making neural network training insufficient for satisfactory results. Furthermore, this method requires significant computational resources, making it unsuitable for deployment on robots with limited computing resources. Summary of the Invention

[0004] This application provides at least a real-time reflection removal method, apparatus, electronic device, and readable storage medium to solve the aforementioned problems in the prior art.

[0005] The first aspect of this application provides a real-time reflection removal method, which includes:

[0006] Acquire point cloud data collected in real time by a depth camera, and segment the point cloud data into positive point cloud, ground point cloud, and negative point cloud;

[0007] Determine whether the number of points in both the positive and negative point clouds is greater than zero;

[0008] In response to the fact that the number of positive point clouds is zero and the number of negative point clouds is greater than zero, it is determined whether the number of negative point clouds is less than a first preset threshold.

[0009] If the number of negative point clouds is less than a first preset threshold, then the negative point cloud is determined to be the first reflection point cloud, and the first reflection point cloud is deleted.

[0010] If both the number of positive and negative point clouds are greater than zero, then deduplication is performed on the negative point cloud.

[0011] Determine whether the number of negative point clouds after deduplication is less than the second preset threshold;

[0012] If the number of negative point clouds after deduplication is greater than or equal to the second preset threshold, then return to the step of determining whether the number of negative point clouds is less than the first preset threshold.

[0013] Optionally, the step of segmenting the point cloud data into positive point cloud, ground point cloud, and negative point cloud includes:

[0014] Obtain the world coordinate system; the world coordinate system includes the positive height plane and the negative height plane;

[0015] Calculate the height of the point cloud data in the world coordinate system;

[0016] If the height of the point cloud data is greater than that of the forward height plane, then the point cloud data is defined as a forward point cloud.

[0017] If the height of the point cloud data is less than that of the negative height plane, then the point cloud data is defined as a negative point cloud.

[0018] If the height of the point cloud data is less than or equal to the positive height plane and greater than or equal to the negative height plane, then the point cloud data is defined as a ground point cloud.

[0019] Optionally, the step of determining whether the number of negative point clouds is less than a first preset threshold includes:

[0020] If the number of negative point clouds is greater than or equal to a first preset threshold, then radius filtering is performed on the negative point clouds;

[0021] Obtain the remaining negative point cloud and determine whether the number of remaining negative point clouds is less than a first preset threshold.

[0022] If the number of remaining negative point clouds is greater than or equal to a first preset threshold, then the remaining negative point clouds are determined to be true point clouds.

[0023] If the number of remaining negative point clouds is less than a first preset threshold, then the remaining negative point clouds are determined to be the second reflection point clouds, and the second reflection point clouds are deleted.

[0024] Optionally, the step of radius filtering the negative point cloud includes:

[0025] A corresponding detection area is generated based on the preset radius of each negative point cloud; wherein, the preset radii of all negative point clouds are equal;

[0026] Obtain the number of negative point clouds within the detection area and determine whether the number of negative point clouds within the detection area is less than a third preset threshold;

[0027] If the number of negative point clouds in the detection area is less than a third preset threshold, then the negative point cloud corresponding to the detection area is defined as a noise point and the noise point is deleted.

[0028] Optionally, before the step of deduplicating the negative point cloud, the following steps are included:

[0029] The preset resolution is determined based on the sampling density of point cloud data;

[0030] Voxel downsampling is performed on the positive and negative point clouds based on a preset resolution.

[0031] Optionally, the positive point cloud includes at least one positive point cloud point, and the negative point cloud includes at least one negative point cloud point; the step of voxel downsampling the positive and negative point clouds based on a preset resolution includes:

[0032] A three-dimensional voxel grid is established based on a preset resolution, such that each three-dimensional voxel grid includes at least one positive point cloud point and / or at least one negative point cloud point;

[0033] Calculate the weighted average of the coordinates of all positive point cloud points and / or all negative point cloud points in the 3D voxel raster, and use the weighted average of the coordinates as the point cloud coordinate value of the corresponding 3D voxel raster; among them, the point cloud points with point cloud coordinate values ​​are used as new positive point cloud points or negative point cloud points.

[0034] Optionally, the steps for deduplicating negative point clouds include:

[0035] Create a hash table and a hash function;

[0036] The hash addresses of all new positive point cloud points and all new negative point cloud points are calculated based on the hash function;

[0037] In response to the existence of a new positive point cloud point and a new negative point cloud point with the same hash address, the new negative point cloud point is defined as the third reflection point cloud, and the third reflection point cloud is deleted.

[0038] A second aspect of this application provides a real-time reflection removal apparatus, which includes:

[0039] The acquisition module is used to acquire point cloud data collected in real time by the depth camera and to segment the point cloud data into positive point cloud, ground point cloud and negative point cloud;

[0040] The execution module is used to determine whether the number of positive and negative point clouds is greater than zero; in response to the number of positive point clouds being zero and the number of negative point clouds being greater than zero, it determines whether the number of negative point clouds is less than a first preset threshold; in response to the number of negative point clouds being less than the first preset threshold, it determines that the negative point cloud is the first reflection point cloud and deletes the first reflection point cloud; in response to the number of both positive and negative point clouds being greater than zero, it performs a deduplication operation on the negative point cloud; it determines whether the number of negative point clouds after the deduplication operation is less than a second preset threshold; in response to the number of negative point clouds after the deduplication operation being greater than or equal to the second preset threshold, it returns the operation of determining whether the number of negative point clouds is less than the first preset threshold.

[0041] A third aspect of this application provides an electronic device including a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement the real-time reflection removal method as described above.

[0042] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the real-time reflection removal method described above.

[0043] The beneficial effects of this application are as follows: Unlike existing technologies, this application acquires point cloud data in real time using a depth camera and classifies and segments the point cloud data. Based on the different quantities of positive and negative point clouds, it can achieve real-time and accurate removal of ground reflections. Simultaneously, this application also sets a first preset threshold and a second preset threshold to accurately determine the quantity of negative point clouds, improving the accuracy of ground reflection judgment and thus achieving real-time and accurate removal of ground reflections.

[0044] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating an embodiment of the real-time reflection removal method of this application;

[0047] Figure 2 yes Figure 1 A detailed flowchart of step S11 is shown below;

[0048] Figure 3yes Figure 1 A detailed flowchart of step S13 is shown below;

[0049] Figure 4 yes Figure 1 A detailed flowchart of step S15 is shown below;

[0050] Figure 5 This is a schematic diagram of the structure of an embodiment of the real-time reflection removal device of this application;

[0051] Figure 6 This is a schematic diagram of the framework of an embodiment of the electronic device of this application;

[0052] Figure 7 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0053] To enable those skilled in the art to better understand the technical solutions of this application, the real-time reflection removal method, apparatus, electronic device, and readable storage medium provided in this application are described in further detail below with reference to the accompanying drawings and specific embodiments. It is understood that the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0054] The terms "first," "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0055] This application provides a real-time reflection removal method to accurately identify and delete object reflections. Please refer to [link / reference]. Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the real-time reflection removal method of this application.

[0056] The real-time reflection removal method of this application can be executed by a real-time reflection removal device. For example, the real-time reflection removal method can be executed by a terminal device, a server, or other processing device. The real-time reflection removal device may include user equipment (UE), mobile device, user terminal, terminal, cellular phone, wireless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the real-time reflection removal method can be implemented by a processor calling computer-readable instructions stored in memory.

[0057] Specifically, the real-time reflection removal method in this embodiment is implemented by a mobile robot equipped with a depth camera. The depth camera acquires depth images along the movement path of the mobile robot to plan the trajectory of the mobile robot based on the depth images.

[0058] Specifically, the real-time reflection removal method of this disclosure embodiment may include the following steps:

[0059] Step S11: Acquire point cloud data collected in real time by the depth camera, and segment the point cloud data into positive point cloud, ground point cloud and negative point cloud.

[0060] In this process, the mobile robot uses a depth camera to capture a depth map of the path in front of it, and then converts the captured depth map into point cloud data.

[0061] Specifically, converting a depth map into point cloud data is essentially a coordinate system transformation, specifically from the image coordinate system containing the depth map to the camera coordinate system. The constraints of this coordinate system transformation are the intrinsic parameters of the depth camera. Optionally, the coordinates in the camera coordinate system can be calculated using the intrinsic parameters of the depth camera and the coordinates in the image coordinate system. The intrinsic parameters of the depth camera typically include the principal point coordinates, focal length, and coordinate axis tilt parameters. Relative to the imaging plane, the principal point coordinates are x0 and y0, and the focal length can include the focal lengths fx and fy along the x-axis and y-axis, respectively. Generally, fx and fy are equal. Ideally, the coordinate axis tilt parameter is 0.

[0062] Optionally, for details on the process of segmenting point cloud data into positive point cloud, ground point cloud, and negative point cloud, please refer to [link to relevant documentation]. Figure 2 , Figure 2 yes Figure 1 A detailed flowchart of step S11 is provided. Specifically, it includes the following steps:

[0063] Step S111: Obtain the world coordinate system.

[0064] The world coordinate system is obtained by transforming the image coordinate system, and the constraints of this coordinate system transformation are the intrinsic and extrinsic parameters of the depth camera.

[0065] Specifically, the image coordinate system is transformed into the camera coordinate system based on the intrinsic parameters of the depth camera, and the camera coordinate system is further transformed into the world coordinate system based on the extrinsic parameters of the depth camera. The extrinsic parameters of the depth camera may include rotation and translation matrices. The rotation matrix is ​​used to describe the orientation of the coordinate axes of the world coordinate system relative to the camera coordinate system, and the translation matrix is ​​used to describe the position of the origin in the camera coordinate system.

[0066] Step S112: Calculate the height of the point cloud data in the world coordinate system.

[0067] Based on the acquired point cloud data and the world coordinate system, the specific coordinates of all point cloud data in the world coordinate system are calculated. The height of the point cloud data in the world coordinate system is the z-axis coordinate value of the point cloud data in the world coordinate system.

[0068] Specifically, the world coordinate system includes a positive height plane and a negative height plane, where the positive height plane is the coordinate plane with a z-axis coordinate value of m, and the negative height plane is the coordinate plane with a z-axis coordinate value of n. Optionally, generally, m equals -n.

[0069] Optionally, the height of all point cloud data in the world coordinate system is calculated, and the height of all point cloud data is compared with the z-axis coordinate values ​​of the positive height plane and the negative height plane. It is determined whether the height of the point cloud data is greater than the z-axis coordinate value of the positive height plane, or whether the height of the point cloud data is less than the z-axis coordinate value of the negative height plane, or whether the height of the point cloud data is less than or equal to the z-axis coordinate value of the positive height plane and greater than or equal to the z-axis coordinate value of the negative height plane. In response to the determination result, any one of the steps S113-S115 is executed respectively.

[0070] Step S113: In response to the point cloud data having a height greater than the forward height plane, the point cloud data is defined as a forward point cloud.

[0071] Specifically, point cloud data is defined as a forward point cloud when its height is greater than the z-axis coordinate value of the forward height plane. Generally, forward point clouds are formed by objects on the ground captured by a depth camera.

[0072] Step S114: In response to the point cloud data having a height less than the negative height plane, the point cloud data is defined as a negative point cloud.

[0073] Specifically, a point cloud is defined as a negative point cloud if its height is less than the z-axis coordinate of the negative height plane. Generally, negative point clouds are formed by the reflections of obstacles on the ground captured by a depth camera.

[0074] Step S115: In response to the point cloud data having a height less than or equal to the positive height plane and greater than or equal to the negative height plane, the point cloud data is defined as a ground point cloud.

[0075] Specifically, when the height of the point cloud data is less than or equal to the z-axis coordinate value of the positive height plane and greater than or equal to the z-axis coordinate value of the negative height plane, the point cloud data is defined as a ground point cloud.

[0076] Optionally, depending on the working environment of the mobile robot and the different depth maps captured by the depth camera, the number of positive point clouds, negative point clouds, and ground point clouds included in the point cloud data converted from the depth map will vary. For example, it may include only positive point clouds, or only negative point clouds, or only ground point clouds, or both positive and negative point clouds, or both positive and ground point clouds, or both negative and ground point clouds, or positive, negative, and ground point clouds, etc.

[0077] Step S12: Determine whether the number of positive point clouds and negative point clouds are both greater than zero.

[0078] In step S11, after obtaining the positive and negative point clouds after classification, it is further determined whether the number of both the positive and negative point clouds is greater than zero.

[0079] Specifically, when it is determined that the number of positive point clouds is zero and the number of negative point clouds is greater than zero, step S13 is executed further; when it is determined that both the number of positive point clouds and the number of negative point clouds are greater than zero, step S15 is executed further; when it is determined that the number of negative point clouds is zero, regardless of whether there are positive point clouds, it can be considered that there are no point clouds formed by the reflection of objects on the ground, that is, no reflection of objects is captured, and it can be considered that no reflection deletion operation is required.

[0080] Step S13: In response to the fact that the number of positive point clouds is zero and the number of negative point clouds is greater than zero, determine whether the number of negative point clouds is less than the first preset threshold.

[0081] When the number of positive point clouds is zero and the number of negative point clouds is greater than zero, it can be assumed that the negative point clouds may be formed by ground cliffs such as pits or stairs captured by the depth camera. The latter is formed by the reflection of objects on the ground. Therefore, it is necessary to further determine the number of negative point clouds.

[0082] Specifically, when the depth camera captures ground features such as pits or steep cliffs, it generates a large amount of negative point cloud data; when the depth camera captures the reflection of an object on the ground, it captures only a small portion of the reflection if the object is not captured at the same time, thus generating less negative point cloud data.

[0083] In this embodiment, multiple simulation experiments are conducted, and a first preset threshold is set based on the sampling density of the depth camera. Optionally, when the number of negative point clouds is determined to be greater than or equal to the first preset threshold, further actions are performed as follows: Figure 3 As shown in step S131; when it is determined that the number of negative point clouds is less than the first preset threshold, further execute step S14.

[0084] Step S131: In response to the number of negative point clouds being greater than or equal to a first preset threshold, radius filtering is performed on the negative point clouds.

[0085] In this embodiment, since image capture, coordinate system transformation, and image transmission are all necessary operations, there is a possibility of random signal interference causing image noise. Therefore, when the number of negative point clouds is determined to be greater than or equal to a first preset threshold, indicating the presence of a large number of negative point clouds, denoising of the classified negative point clouds is required. Specifically, this embodiment uses radius filtering to denoise the negative point clouds.

[0086] Optionally, the specific steps for radius filtering of the negative point cloud can be as shown in steps S1311-S1313. Specifically, it includes the following steps:

[0087] Step S1311: Generate the corresponding detection area based on the preset radius of each negative point cloud.

[0088] In this embodiment, all negative point clouds have the same preset radius. Specifically, the detection area is a sphere with the negative point cloud as its center and the preset radius as its detection radius, meaning that the range of each detection area is equal.

[0089] Step S1312: Obtain the number of negative point clouds in the detection area and determine whether the number of negative point clouds in the detection area is less than the third preset threshold.

[0090] Specifically, the number of negative point clouds within each corresponding detection area is obtained, and the number of negative point clouds in all detection areas is compared with a third preset threshold. The third preset threshold is used to calibrate noise points.

[0091] Specifically, when the number of negative point clouds within the detection area is greater than or equal to the third preset threshold, it proves that the negative point cloud corresponding to the detection area is closely adjacent to or at least in a nearby position with other negative point clouds, and is not in a discrete state. Therefore, the negative point cloud can be considered a non-discrete point, i.e., a noise point. When the number of negative point clouds within the detection area is less than the third preset threshold, step S1313 is further executed.

[0092] Step S1313: In response to the fact that the number of negative point clouds in the detection field is less than the third preset threshold, the negative point cloud corresponding to the detection field is defined as a noise point and the noise point is deleted.

[0093] Specifically, when the number of negative point clouds in the detection field is less than the third preset threshold, it proves that the negative point cloud corresponding to the detection field is in a discrete distribution state with other negative point clouds. The negative point cloud is defined as a discrete point, i.e. a noise point, and needs to be deleted to achieve noise reduction.

[0094] Optionally, in this embodiment, both the preset radius and the third preset threshold are related to the sampling density of the depth camera.

[0095] Step S132: Obtain the remaining negative point cloud and determine whether the number of remaining negative point clouds is less than the first preset threshold.

[0096] In this embodiment, step S131 obtains the negative point cloud after removing noise points, and further determines whether the number of remaining negative point clouds is less than a first preset threshold. The first preset threshold is used to calibrate the reflection, and can be specifically set based on the working environment of the mobile robot and the sampling density.

[0097] Optionally, if the number of remaining negative point clouds is greater than or equal to the first preset threshold, step S133 is further executed; if the number of remaining negative point clouds is less than the first preset threshold, step S134 is further executed.

[0098] Step S133: In response to the number of remaining negative point clouds being greater than or equal to the first preset threshold, the remaining negative point clouds are determined to be real point clouds.

[0099] Specifically, if the number of remaining negative point clouds is greater than or equal to the first preset threshold, then the remaining negative point clouds can be determined to be point clouds formed by ground cliffs such as pits or stairs, that is, the remaining negative point clouds are real point clouds.

[0100] Step S134: In response to the fact that the number of remaining negative point clouds is less than the first preset threshold, the remaining negative point clouds are determined to be the second reflection point clouds, and the second reflection point clouds are deleted.

[0101] Specifically, if the number of remaining negative point clouds is less than a first preset threshold, then the remaining negative point clouds are determined to be point clouds formed by the reflection of an object on a smooth plane, and the remaining negative point clouds are defined as the second reflection point cloud. To prevent the reflection point cloud from affecting the subsequent navigation and motion planning of the mobile robot, the second reflection point cloud is deleted.

[0102] Step S14: In response to the fact that the number of negative point clouds is less than the first preset threshold, the negative point cloud is determined to be the first reflection point cloud, and the first reflection point cloud is deleted.

[0103] Specifically, if the number of negative point clouds is less than a first preset threshold, the negative point cloud obtained through direct classification is determined to be a point cloud formed by the reflection of an object on a smooth plane, and this negative point cloud is defined as the first reflection point cloud. To prevent the reflection point cloud from affecting the subsequent navigation and motion planning of the mobile robot, the first reflection point cloud is deleted.

[0104] Step S15: In response to the fact that both the number of positive point clouds and the number of negative point clouds are greater than zero, a deduplication operation is performed on the negative point clouds.

[0105] If both the number of positive and negative point clouds are greater than zero, it can be considered that at least part of the object and its reflection on the smooth plane have been captured. By performing a deduplication operation on the negative point cloud, the point cloud formed by the reflection corresponding to the object can be removed, and further judgment can be made on the deduplicated negative point cloud.

[0106] Optionally, before performing deduplication on the negative point cloud, steps S21-S22 can be executed to downsample the point cloud data, reducing the amount of data required for matching and judgment, thereby improving detection efficiency. Specifically, this includes the following steps:

[0107] Step S21: Determine the preset resolution based on the sampling density of the point cloud data.

[0108] Specifically, the point cloud data is obtained by image conversion through depth maps. The resolution of the image conversion is based on the sampling density of the depth camera. In this embodiment, the preset resolution can be determined according to the sampling density of the point cloud data, that is, according to the sampling density of the depth camera. It can also be set according to empirical values.

[0109] Step S22: Perform voxel downsampling on the positive and negative point clouds based on a preset resolution.

[0110] In this embodiment, voxel downsampling is performed on the positive and negative point clouds according to the preset resolution determined in step S21. The specific voxel downsampling process is shown in steps S221-S222 below. Specifically, it includes the following steps:

[0111] Step S221: Establish a three-dimensional voxel grid based on a preset resolution, so that each three-dimensional voxel grid includes at least one positive point cloud point or at least one negative point cloud point.

[0112] In this embodiment, a three-dimensional voxel grid is established based on a preset resolution, wherein the length, width, and height of the three-dimensional voxel grid are the same. Optionally, the three-dimensional voxel grid can typically be set to 0.05*0.05*0.05.

[0113] Specifically, the positive point cloud includes at least one positive point cloud point, and the negative point cloud includes at least one negative point cloud point. In this embodiment, each three-dimensional voxel grid includes at least one positive point cloud point or at least one negative point cloud point.

[0114] Step S222: Calculate the weighted average of the coordinates of all positive point cloud points or all negative point cloud points in the 3D voxel grid, and use the weighted average of the coordinates as the point cloud coordinate value of the corresponding 3D voxel grid.

[0115] Specifically, the weighted average of the coordinates of all positive point cloud points or all negative point cloud points in each 3D voxel grid is calculated, and this weighted average is used as the point cloud coordinate value of the corresponding 3D voxel grid. Point cloud points with point cloud coordinate values ​​are then designated as new positive or negative point cloud points.

[0116] Specifically, each 3D voxel grid may include only positive point clouds, only negative point clouds, or both positive and negative point clouds. The calculated weighted average coordinate value can be positive or negative. When the weighted average coordinate value is positive, the point cloud point with the current point cloud coordinate value is taken as the new positive point cloud point; when the weighted average coordinate value is negative, the point cloud point with the current point cloud coordinate value is taken as the new negative point cloud point.

[0117] Optionally, after obtaining all new positive and negative point cloud points according to steps S21-S22, the process of further deduplicating the negative point cloud can be found in the following section. Figure 4 , Figure 4 yes Figure 1 A detailed flowchart of step S15 is provided. Specifically, it includes the following steps:

[0118] Step S151: Create a hash table and a hash function.

[0119] In this embodiment, the hash address of each positive and negative point cloud is calculated using a hash function and stored in a hash table.

[0120] Step S152: Calculate the hash addresses of all new positive point cloud points and all new negative point cloud points based on the hash function.

[0121] The hash function in this embodiment can be specifically represented by the following formula (1):

[0122] H(X, Y) = 137*X + 149*Y (1)

[0123] Where H(X, Y) is the hash address, and X and Y are the coordinates of the positive or negative point cloud points, respectively.

[0124] Step S153: In response to the existence of a new positive point cloud point and a new negative point cloud point with the same hash address, define the new negative point cloud point as the third reflection point cloud and delete the third reflection point cloud.

[0125] In this embodiment, the hash addresses of all positive point cloud points are first calculated and stored in a hash table. Then, the hash addresses of all negative point cloud points are calculated. When a hash address for a negative point cloud point already exists, the positive point cloud point corresponding to that hash address is considered to be the point cloud formed by the object, and the negative point cloud point corresponding to that hash address is considered to be the point cloud formed by the object's reflection. Therefore, this negative point cloud point is defined as the third reflection point cloud. To prevent the reflection point cloud from affecting the subsequent navigation and motion planning of the mobile robot, the third reflection point cloud is deleted.

[0126] Step S16: Determine whether the number of negative point clouds after deduplication is less than the second preset threshold.

[0127] In this embodiment, the negative point cloud is deduplicated in step S15, and the number of negative point clouds after deduplication is compared with a second preset threshold, wherein the second preset threshold is used to calibrate the reflection.

[0128] Specifically, if the number of negative point clouds after deduplication is greater than or equal to a second preset threshold, step S17 is further executed; if the number of negative point clouds after deduplication is less than the second preset threshold, the negative point cloud after deduplication can be considered as a point cloud formed by an object, and the negative point cloud point corresponding to the hash address is a point cloud formed by the reflection of the object. Therefore, this negative point cloud point is defined as the fourth reflection point cloud. To prevent the reflection point cloud from affecting the subsequent navigation motion planning of the mobile robot, the fourth reflection point cloud is deleted.

[0129] Step S17: In response to the fact that the number of negative point clouds after the deduplication operation is greater than or equal to the second preset threshold, return to the step of determining whether the number of negative point clouds is less than the first preset threshold.

[0130] Specifically, when the number of negative point clouds after deduplication is greater than or equal to a second preset threshold, it is further determined whether the number of negative point clouds is less than a first preset threshold. Then, the process returns to step S13 to execute the corresponding steps based on the determination result, until all negative point clouds have been determined. This embodiment uses the second and first preset thresholds to redundantly determine the number of negative point clouds, improving the accuracy of reflection determination.

[0131] This application acquires point cloud data in real time using a depth camera and classifies and segments the point cloud data. Based on the different quantities of positive and negative point clouds, it can achieve real-time and accurate removal of ground reflections without the need for supplementary judgment by other sensors, simplifying the judgment method and improving detection efficiency. Simultaneously, this application also sets a first preset threshold and a second preset threshold, which can accurately determine the quantity of negative point clouds, improve the accuracy of ground reflection judgment, and thus achieve real-time and accurate removal of ground reflections.

[0132] Furthermore, when it is determined that both positive and negative point clouds exist, this application optimizes point cloud processing steps such as solving the bounding boxes of positive and negative point clouds or performing plane fitting by establishing a hash table and calculating hash addresses, and by comparing hash addresses, thereby improving the computational efficiency of the algorithm.

[0133] This application also provides a real-time reflection removal device; please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of the structure of an embodiment of the real-time reflection removal device of this application. Figure 5 As shown, the real-time reflection removal device 50 includes an acquisition module 51 and a calculation module 52 connected to each other. Optionally, in this embodiment, the real-time reflection removal device 50 can be installed on a mobile robot, and the mobile robot is equipped with a depth camera that captures a depth map of the mobile robot's movement path.

[0134] Specifically, the acquisition module 51 is used to acquire point cloud data collected by the depth camera, and to segment the point cloud data into positive point cloud, ground point cloud and negative point cloud, and to extract target ground point cloud from ground point cloud using a random sampling method.

[0135] The execution module 52 is used to determine whether the number of positive point clouds and negative point clouds are both greater than zero; in response to the number of positive point clouds being zero and the number of negative point clouds being greater than zero, it determines whether the number of negative point clouds is less than a first preset threshold; in response to the number of negative point clouds being less than the first preset threshold, it determines that the negative point cloud is the first reflection point cloud and deletes the first reflection point cloud; in response to the number of both positive and negative point clouds being greater than zero, it performs a deduplication operation on the negative point cloud; it determines whether the number of negative point clouds after the deduplication operation is less than a second preset threshold; in response to the number of negative point clouds after the deduplication operation being greater than or equal to the second preset threshold, it returns to the operation of determining whether the number of negative point clouds is less than the first preset threshold.

[0136] This application also provides an electronic device, please refer to... Figure 6 , Figure 6 This is a schematic diagram of a framework of an embodiment of the electronic device of this application. The electronic device 60 includes a memory 61 and a processor 62 coupled to each other. The processor 62 is used to execute program instructions stored in the memory 61 to implement the steps in any of the above embodiments of the real-time reflection removal method. In a specific implementation scenario, the electronic device 60 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 60 may also include mobile devices such as laptops and tablets, which are not limited here.

[0137] Specifically, processor 62 controls itself and memory 61 to implement the steps in any of the above-described real-time reflection removal method embodiments. Processor 62 can also be referred to as a CPU (Central Processing Unit). Processor 62 may be an integrated circuit chip with signal processing capabilities. Processor 62 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 62 can be implemented using integrated circuit chips.

[0138] This application also provides a computer-readable storage medium; please refer to [link to relevant documentation]. Figure 7 , Figure 7This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 70 stores a computer program 71 that can be executed by a processor. The computer program 71 is used to implement the steps in any of the above embodiments of the real-time reflection removal method.

[0139] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0140] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0142] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0143] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A real-time reflection removal method, characterized in that, include: Acquire point cloud data collected in real time by a depth camera, and segment the point cloud data into positive point cloud, ground point cloud, and negative point cloud; Determine whether the number of both the positive point cloud and the negative point cloud is greater than zero; In response to the fact that the number of positive point clouds is zero and the number of negative point clouds is greater than zero, it is determined whether the number of negative point clouds is less than a first preset threshold. If the number of negative point clouds is less than a first preset threshold, then the negative point cloud is determined to be a first reflection point cloud, and the first reflection point cloud is deleted. If both the number of positive point clouds and the number of negative point clouds are greater than zero, then a deduplication operation is performed on the negative point clouds. Determine whether the number of negative point clouds after deduplication is less than a second preset threshold. If the number of negative point clouds after the deduplication operation is greater than or equal to the second preset threshold, then return to the step of determining whether the number of negative point clouds is less than the first preset threshold. The step of segmenting the point cloud data into positive point cloud, ground point cloud, and negative point cloud includes: Obtain the world coordinate system; wherein, the world coordinate system includes a positive height plane and a negative height plane; Calculate the height of the point cloud data in the world coordinate system; If the height of the point cloud data is greater than the forward height plane, then the point cloud data is defined as the forward point cloud; If the height of the point cloud data is less than the negative height plane, then the point cloud data is defined as the negative point cloud; If the height of the point cloud data is less than or equal to the positive height plane and greater than or equal to the negative height plane, then the point cloud data is defined as the ground point cloud.

2. The real-time reflection removal method according to claim 1, characterized in that, The step of determining whether the number of negative point clouds is less than a first preset threshold includes: If the number of negative point clouds is greater than or equal to the first preset threshold, then radius filtering is performed on the negative point clouds; Obtain the remaining negative point cloud and determine whether the number of remaining negative point clouds is less than the first preset threshold. If the number of remaining negative point clouds is greater than or equal to the first preset threshold, then the remaining negative point clouds are determined to be real point clouds. If the number of remaining negative point clouds is less than the first preset threshold, then the remaining negative point clouds are determined to be second reflection point clouds, and the second reflection point clouds are deleted.

3. The real-time reflection removal method according to claim 2, characterized in that, The step of performing radius filtering on the negative point cloud includes: A corresponding detection area is generated based on a preset radius for each of the negative point clouds; wherein, the preset radii of all the negative point clouds are equal; The number of negative point clouds within the detection area is obtained, and it is determined whether the number of negative point clouds within the detection area is less than a third preset threshold. If the number of negative point clouds in the detection area is less than the third preset threshold, then the negative point cloud corresponding to the detection area is defined as a noise point and the noise point is deleted.

4. The real-time reflection removal method according to claim 1, characterized in that, Before performing the deduplication operation on the negative point cloud, the following steps are included: The preset resolution is determined based on the sampling density of the point cloud data; Voxel downsampling is performed on the positive point cloud and the negative point cloud based on the preset resolution.

5. The real-time reflection removal method according to claim 4, characterized in that, The positive point cloud includes at least one positive point cloud point, and the negative point cloud includes at least one negative point cloud point; the step of voxel downsampling the positive point cloud and the negative point cloud based on the preset resolution includes: A three-dimensional voxel grid is established based on the preset resolution, such that each three-dimensional voxel grid includes at least one positive point cloud point and / or at least one negative point cloud point; Calculate the weighted average of the coordinates of all positive point cloud points and / or all negative point cloud points in the three-dimensional voxel grid, and use the weighted average of the coordinates as the point cloud coordinate value of the corresponding three-dimensional voxel grid; wherein, the point cloud point with the point cloud coordinate value is used as a new positive point cloud point or a new negative point cloud point.

6. The real-time reflection removal method according to claim 5, characterized in that, The step of performing deduplication on the negative point cloud includes: Create a hash table and a hash function; The hash addresses of all the new positive point cloud points and all the new negative point cloud points are calculated based on the hash function. In response to the existence of a new positive point cloud point and a new negative point cloud point with the same hash address, the new negative point cloud point is defined as the third reflection point cloud, and the third reflection point cloud is deleted.

7. A real-time reflection removal device, characterized in that, include: The acquisition module is used to acquire point cloud data collected in real time by the depth camera and to segment the point cloud data into positive point cloud, ground point cloud and negative point cloud; The execution module is used to determine whether the number of positive point clouds and negative point clouds are both greater than zero; in response to the number of positive point clouds being zero and the number of negative point clouds being greater than zero, it determines whether the number of negative point clouds is less than a first preset threshold. If the number of negative point clouds is less than a first preset threshold, then the negative point cloud is determined to be a first reflection point cloud, and the first reflection point cloud is deleted; if the number of positive point clouds and the number of negative point clouds are both greater than zero, then a deduplication operation is performed on the negative point cloud; then it is determined whether the number of negative point clouds after the deduplication operation is less than a second preset threshold. If the number of negative point clouds after the deduplication operation is greater than or equal to the second preset threshold, then the operation of determining whether the number of negative point clouds is less than the first preset threshold is returned. The acquisition module is also used for: Obtain the world coordinate system; wherein, the world coordinate system includes a positive height plane and a negative height plane; Calculate the height of the point cloud data in the world coordinate system; If the height of the point cloud data is greater than the forward height plane, then the point cloud data is defined as the forward point cloud; If the height of the point cloud data is less than the negative height plane, then the point cloud data is defined as the negative point cloud; If the height of the point cloud data is less than or equal to the positive height plane and greater than or equal to the negative height plane, then the point cloud data is defined as the ground point cloud.

8. An electronic device, characterized in that, It includes a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement the real-time reflection removal method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the real-time reflection removal method as described in any one of claims 1-6.

Images