Three-dimensional target detection method and device, computer readable storage medium and robot

Abstract

The application belongs to the technical field of robots, and particularly relates to a three-dimensional target detection method and device, a computer readable storage medium and a robot. The method comprises the following steps: acquiring a two-dimensional image and a depth image in a specified direction; performing two-dimensional target detection on a preset target object in the two-dimensional image to obtain a two-dimensional target detection frame corresponding to the target object; converting the depth image into point cloud data, and determining a point cloud set corresponding to the two-dimensional target detection frame in the point cloud data; performing three-dimensional target detection on the target object in the point cloud set to obtain a three-dimensional target detection frame corresponding to the target object; and determining the size and position of the target object according to the three-dimensional target detection frame. In the application, the result of two-dimensional target detection and the point cloud data obtained by converting the depth image are fused and used, so that fast and accurate three-dimensional target detection can be realized.

Description

Technical Field

[0001] This application belongs to the field of robotics technology, and in particular relates to a three-dimensional target detection method, device, computer-readable storage medium, and robot. Background Technology

[0002] In existing technologies, robots typically use RGB cameras or multi-line LiDAR for 3D object detection. However, when using RGB cameras for 3D object detection, it is impossible to obtain distance information between the target object and the robot. Even with monocular depth estimation, the estimated distance information is still not accurate enough, and achieving monocular depth estimation generally requires a certain amount of hardware computing power. When using multi-line LiDAR for 3D object detection, because the number of lines in a typical LiDAR is relatively small, the number of object types detected through point clouds is not as wide as that detected by RGB cameras. Furthermore, data acquisition and annotation are more labor-intensive and resource-intensive, resulting in a limited range of detectable objects, making it difficult to detect small-sized targets. Multi-line LiDAR is also more expensive, increasing hardware costs. Summary of the Invention

[0003] In view of this, embodiments of this application provide a three-dimensional target detection method, apparatus, computer-readable storage medium, and robot to solve the problems of inaccurate distance information, high hardware computing power requirements, difficulty in detecting small-sized target objects, and high cost in existing three-dimensional target detection methods.

[0004] A first aspect of this application provides a three-dimensional target detection method, which may include:

[0005] Acquire a 2D image and a depth image in a specified orientation;

[0006] Two-dimensional target detection is performed on a preset target object in the two-dimensional image to obtain a two-dimensional target detection box corresponding to the target object;

[0007] The depth image is converted into point cloud data, and a set of points corresponding to the two-dimensional target detection box is determined from the point cloud data.

[0008] The point cloud set is layered based on the distance information in the point cloud set to obtain each point cloud layer; the number of points in each point cloud layer is counted; and a 3D target detection box corresponding to the target object is determined based on the number of points in each point cloud layer.

[0009] The size and position of the target object are determined based on the three-dimensional target detection box.

[0010] In one specific implementation of the first aspect, the step of performing layering processing on the point cloud set based on distance information in the point cloud set to obtain various point cloud layers may include:

[0011] The distance information in the point cloud set is normalized to obtain normalized distance information;

[0012] The point cloud set is layered according to the normalized distance information to obtain each point cloud layer; wherein, each point cloud in the same point cloud layer has the same normalized distance information.

[0013] In one specific implementation of the first aspect, determining the 3D target detection box corresponding to the target object based on the number of points in each point cloud layer may include:

[0014] Point cloud layers with a number of points greater than a preset point cloud number threshold are selected as candidate point cloud layers.

[0015] Candidate point cloud layers with a consecutive number of layers greater than or equal to a preset number of layers are grouped into point cloud layer groups.

[0016] The point cloud layer groups and the interval layers with a layer number interval less than the preset interval threshold are combined into a large point cloud layer group.

[0017] Determine the bounding rectangles corresponding to each point cloud layer group, and take the point cloud layer group with the largest bounding rectangle area as the target point cloud layer group.

[0018] Calculate the area ratio between the bounding rectangle corresponding to the target point cloud group and the two-dimensional target detection box;

[0019] If the area ratio is greater than a preset ratio threshold, the detection box corresponding to the target point cloud group is determined as the three-dimensional target detection box corresponding to the target object.

[0020] In one specific implementation of the first aspect, after calculating the area ratio between the bounding rectangle corresponding to the target point cloud group and the two-dimensional target detection box, the method may further include:

[0021] If the area ratio is less than or equal to the ratio threshold, the interval threshold is increased, and the process returns to the step of combining the point cloud layer groups with layer intervals less than the preset interval threshold and the interval layers into a large point cloud layer group, and the subsequent steps, until the area ratio is greater than the ratio threshold.

[0022] In one specific implementation of the first aspect, determining the circumscribed rectangles corresponding to each large group of point cloud layers may include:

[0023] For each point cloud layer group, a set of pixels corresponding to the point cloud layer group is determined in the two-dimensional target detection box;

[0024] Determine the bounding rectangle of the pixel set, and then define the bounding rectangle of the pixel set as the bounding rectangle corresponding to the large group of point cloud layers.

[0025] In one specific implementation of the first aspect, before selecting point cloud layers with a number of point clouds greater than a preset point cloud number threshold as candidate point cloud layers, the method may further include:

[0026] Calculate the number of pixels in the two-dimensional target detection box;

[0027] If the number of pixels is less than a preset pixel number threshold, then the point cloud number threshold is determined based on the number of pixels.

[0028] If the number of pixels is greater than or equal to the number of pixels threshold, then pixel sampling is performed in the two-dimensional target detection box to obtain the sampled two-dimensional target detection box;

[0029] Calculate the number of sampled pixels in the sampled two-dimensional target detection box, and determine the point cloud number threshold based on the number of sampled pixels.

[0030] A second aspect of this application provides a three-dimensional target detection device, which may include:

[0031] The image acquisition module is used to acquire two-dimensional images and depth images in a specified direction;

[0032] A two-dimensional target detection module is used to perform two-dimensional target detection on a preset target object in the two-dimensional image and obtain a two-dimensional target detection box corresponding to the target object.

[0033] The point cloud set determination module is used to convert the depth image into point cloud data and determine the point cloud set corresponding to the two-dimensional target detection box in the point cloud data;

[0034] The 3D target detection module is used to perform layering processing on the point cloud set according to the distance information in the point cloud set to obtain each point cloud layer; count the number of points in each point cloud layer; and determine the 3D target detection box corresponding to the target object based on the number of points in each point cloud layer.

[0035] The target object information determination module is used to determine the size and position of the target object based on the three-dimensional target detection box.

[0036] In one specific implementation of the second aspect, the three-dimensional target detection module may include:

[0037] The layered processing submodule is used to perform layered processing on the point cloud set according to the distance information in the point cloud set to obtain each point cloud layer;

[0038] The point cloud count statistics submodule is used to count the number of points in each point cloud layer.

[0039] The 3D target detection submodule is used to determine the 3D target detection box corresponding to the target object based on the number of points in each point cloud layer.

[0040] In one specific implementation of the second aspect, the hierarchical processing submodule may include:

[0041] The normalization processing unit is used to normalize the distance information in the point cloud set to obtain normalized distance information;

[0042] The layering processing unit is used to perform layering processing on the point cloud set according to the normalized distance information to obtain each point cloud layer; wherein, each point cloud in the same point cloud layer has the same normalized distance information.

[0043] In one specific implementation of the second aspect, the three-dimensional target detection submodule may include:

[0044] The candidate point cloud layer determination unit is used to identify point cloud layers with a number of points greater than a preset point cloud number threshold as candidate point cloud layers.

[0045] The point cloud layer group determination unit is used to combine candidate point cloud layers with a consecutive number of layers greater than or equal to a preset number of layers into a point cloud layer group.

[0046] The point cloud layer group determination unit is used to combine the point cloud layer subgroups and the interval layers with a layer number interval less than a preset interval threshold into a point cloud layer group.

[0047] The outer rectangle determination unit is used to determine the outer rectangles corresponding to each point cloud layer group, and to take the point cloud layer group with the largest outer rectangle area as the target point cloud layer group.

[0048] An area ratio calculation unit is used to calculate the area ratio between the bounding rectangle corresponding to the target point cloud group and the two-dimensional target detection box.

[0049] A three-dimensional target detection box determination unit is used to determine the detection box corresponding to the target point cloud group as the three-dimensional target detection box corresponding to the target object if the area ratio is greater than a preset ratio threshold.

[0050] In one specific implementation of the second aspect, the three-dimensional target detection submodule may further include:

[0051] An interval threshold adjustment unit is used to increase the interval threshold if the area ratio is less than or equal to the ratio threshold, and return to the step of combining the point cloud layer groups with layer intervals less than the preset interval threshold and the interval layers into a large point cloud layer group and subsequent steps, until the area ratio is greater than the ratio threshold.

[0052] In one specific implementation of the second aspect, the circumscribed rectangle determination unit may be specifically used to: for each large group of point cloud layers, determine the set of pixels corresponding to the large group of point cloud layers in the two-dimensional target detection box; determine the circumscribed rectangle of the set of pixels, and determine the circumscribed rectangle of the set of pixels as the circumscribed rectangle corresponding to the large group of point cloud layers.

[0053] In one specific implementation of the second aspect, the three-dimensional target detection submodule may further include:

[0054] A point cloud number threshold determination unit is used to calculate the number of pixels in the two-dimensional target detection box; if the number of pixels is less than a preset pixel number threshold, the point cloud number threshold is determined based on the number of pixels; if the number of pixels is greater than or equal to the pixel number threshold, pixel sampling is performed in the two-dimensional target detection box to obtain a sampled two-dimensional target detection box; the number of sampled pixels in the sampled two-dimensional target detection box is calculated, and the point cloud number threshold is determined based on the number of sampled pixels.

[0055] A third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described three-dimensional target detection methods.

[0056] A fourth aspect of this application provides a robot including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described three-dimensional target detection methods.

[0057] The fifth aspect of this application provides a computer program product that, when run on a robot, causes the robot to perform the steps of any of the three-dimensional target detection methods described above.

[0058] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment acquires a two-dimensional image and a depth image in a specified direction; performs two-dimensional target detection on a preset target object in the two-dimensional image to obtain a two-dimensional target detection box corresponding to the target object; converts the depth image into point cloud data, and determines a point cloud set corresponding to the two-dimensional target detection box in the point cloud data; performs three-dimensional target detection on the target object in the point cloud set to obtain a three-dimensional target detection box corresponding to the target object; and determines the size and position of the target object based on the three-dimensional target detection box. In this application embodiment, to address the problems that a single RGB camera cannot obtain accurate distance information and that multi-line LiDAR is costly, depth image data using a depth camera is introduced, which can obtain more accurate object distance information and reduce hardware costs; to address the problem that multi-line LiDAR cannot detect small objects, the result of two-dimensional target detection is used as the basis for three-dimensional target detection, which can be applied to various objects and sizes; and by fusing the result of two-dimensional target detection with the point cloud data obtained from the depth image conversion, fast and accurate three-dimensional target detection can be achieved on a low-hardware computing platform, allowing the robot to obtain more accurate and refined environmental information. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a flowchart of one embodiment of a three-dimensional target detection method in this application.

[0061] Figure 2 A schematic flowchart illustrating the process of performing 3D target detection on a target object in a point cloud set and obtaining the 3D target detection box corresponding to the target object;

[0062] Figure 3 A schematic flowchart illustrating the process of determining the 3D target detection box corresponding to the target object based on the number of points in each point cloud layer;

[0063] Figure 4 This is a structural diagram of one embodiment of a three-dimensional target detection device according to the present application.

[0064] Figure 5 This is a schematic block diagram of a robot according to an embodiment of this application. Detailed Implementation

[0065] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0066] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0067] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0068] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0069] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0070] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0071] In this embodiment, to address the issues of a single RGB camera failing to acquire accurate distance information and the high cost of multi-line LiDAR, depth image data from a depth camera is introduced. This not only provides more accurate object distance information but also reduces hardware costs. To address the limitation of multi-line LiDAR in detecting small objects, the results of 2D object detection are used as the basis for 3D object detection, making it applicable to various object types and sizes. Furthermore, by fusing the results of 2D object detection with point cloud data obtained from depth image conversion, fast and accurate 3D object detection can be achieved on low-hardware computing platforms, enabling robots to acquire more accurate and refined environmental information.

[0072] Please see Figure 1 One embodiment of a three-dimensional target detection method in this application may include:

[0073] Step S101: Obtain a two-dimensional image and a depth image in the specified direction.

[0074] In this embodiment, an RGB camera and a depth camera can be pre-installed on the robot. The RGB camera is used to acquire RGB images, which are two-dimensional images, and the depth camera is used to acquire depth images. The RGB camera and the depth camera are oriented as similarly as possible so that they can acquire two-dimensional images and depth images in the same direction respectively.

[0075] After obtaining the 2D image and the depth image, the 2D image and the depth image can be pre-aligned so that they can be fused together later.

[0076] Step S102: Perform two-dimensional target detection on the preset target object in the two-dimensional image to obtain the two-dimensional target detection box corresponding to the target object.

[0077] The target object can be set according to the actual situation. For example, it can include, but is not limited to, people, vehicles, animals or any other kind of object.

[0078] In this embodiment, any existing two-dimensional object detection algorithm can be used for two-dimensional object detection, including but not limited to Faster-RCNN, YOLO, SSD, and Retina. These two-dimensional object detection algorithms can ultimately output two-dimensional object detection boxes corresponding to the target object. Here, the width of the two-dimensional object detection box is denoted as w, and the height of the two-dimensional object detection box is denoted as h, both in pixels.

[0079] Step S103: Convert the depth image into point cloud data, and determine the point cloud set corresponding to the two-dimensional target detection box in the point cloud data.

[0080] When converting point cloud data, the depth image can first be converted into point cloud data in the depth camera coordinate system, and then the point cloud data in the depth camera coordinate system can be converted into point cloud data in the robot coordinate system. Unless otherwise specified, the point cloud data used in the following process refers to the point cloud data in the robot coordinate system. Point cloud data can be represented as (x, y, z), where x, y, and z are the components of the point cloud on the three coordinate axes (X-axis, Y-axis, and Z-axis) of the robot coordinate system, and the unit is meters (m). The X-axis corresponds to the distance information of the point cloud, that is, the distance between the point cloud and the robot in the direction directly in front of the robot.

[0081] Based on the two-dimensional object detection boxes in the two-dimensional image, corresponding two-dimensional object detection boxes can also be found in the depth image. Furthermore, point clouds corresponding to the target pixels in the depth image can be found in the point cloud data. These point clouds together constitute a point cloud set. Here, the target pixel is the pixel within the region bounded by the two-dimensional object detection box in the depth image.

[0082] Step S104: Perform 3D target detection on the target object in the point cloud set to obtain the 3D target detection box corresponding to the target object.

[0083] like Figure 2 As shown, step S104 may specifically include the following process:

[0084] Step S1041: Perform layering processing on the point cloud set according to the distance information in the point cloud set to obtain each point cloud layer.

[0085] Specifically, the distance information in the point cloud set can first be normalized to obtain normalized distance information.

[0086] For each point cloud in the point cloud set, its distance information, i.e., its component on the X-axis (x), can be bounded. If x is greater than a preset upper boundary value, it is set as the upper boundary value; if x is less than a preset lower boundary value, it is set as the lower boundary value; if x is greater than or equal to the lower boundary value and less than or equal to the upper boundary value, it remains unchanged. The specific values ​​of the upper and lower boundary values ​​can be determined according to actual conditions, and this application embodiment does not impose specific limitations on them. For example, the upper boundary value can be 10.24m, 5.12m, or 2.56m, and the lower boundary value can be 0.01m.

[0087] After boundary definition, the distance information can be normalized. For example, for a point cloud set where the distance information ranges from [0.01m to 10.24m] after boundary definition, the distance information of each point cloud can be multiplied by 100 to normalize it to the integer range of [1, 1024]. The data is distributed across 1024 integers in the range [1, 1024]. Of course, any other normalization method in the prior art can be used depending on the actual situation, and this application embodiment does not specifically limit this.

[0088] After normalization, the point cloud set can be layered according to the normalized distance information to obtain various point cloud layers. Within the same point cloud layer, all point clouds share the same normalized distance information. All point clouds with a normalized distance of 1 form one point cloud layer, all point clouds with a normalized distance of 2 form another, all point clouds with a normalized distance of 3 form yet another, and so on.

[0089] Step S1042: Count the number of point clouds in each point cloud layer.

[0090] The statistical analysis of the number of points in each point cloud layer actually corresponds to the histogram of the distribution of the number of points at different distances from the robot, that is, the statistical histogram of the point cloud in the X-axis direction directly in front of the robot.

[0091] Step S1043: Determine the 3D target detection box corresponding to the target object based on the number of point clouds in each point cloud layer.

[0092] like Figure 3 As shown, step S1043 may specifically include the following process:

[0093] Step S10431: Select point cloud layers with a number of points greater than a preset point cloud number threshold as candidate point cloud layers.

[0094] The specific value of the point cloud number threshold can be set according to the actual situation, and this application embodiment does not impose a specific limitation on it.

[0095] In one specific implementation of this application, the number of pixels in the two-dimensional target detection box can be calculated first, and the number of pixels can be denoted as N. Then, N = w * h.

[0096] Then, the number of pixels is compared with a preset pixel number threshold. The specific value of the pixel number threshold can be set according to the actual situation, and this application embodiment does not specifically limit it.

[0097] If the number of pixels is less than the pixel number threshold, the point cloud number threshold can be determined based on the number of pixels. Let the point cloud number threshold be denoted as n, then: n = N * c. Where c is a preset coefficient, and its specific value can be set according to the actual situation. For example, it can be set to 0.005 or other values. This application embodiment does not specifically limit this.

[0098] If the number of pixels is greater than or equal to the pixel number threshold, pixel sampling is performed within the two-dimensional object detection bounding box to obtain a sampled two-dimensional object detection bounding box. The specific sampling method can be set according to the actual situation; for example, it can be sampling every other row, every other column, or every two rows and every two columns. This embodiment does not specifically limit this. After sampling, the number of sampled pixels in the sampled two-dimensional object detection bounding box can be calculated, and the point cloud number threshold can be determined based on the number of sampled pixels. Let the number of sampled pixels be N2, then: n = N2 * c.

[0099] Step S10432: Combine each candidate point cloud layer with a consecutive layer number greater than or equal to a preset layer number threshold into a point cloud layer group.

[0100] The specific value of the layer threshold can be set according to the actual situation, and this application embodiment does not impose a specific limitation on it. Generally, the layer threshold is positively correlated with the thickness of the target object. That is, for a target object with a larger thickness, a larger layer threshold should be set, and vice versa, for a target object with a smaller thickness, a smaller layer threshold should be set.

[0101] Here, we take a layer threshold of 3 as an example. The point cloud layers are numbered sequentially as: 1, 2, 3, 4, 5, ..., where the sequence number is the normalized distance information. If, after the screening in step S10431, the candidate point cloud layers obtained are as follows: 101, 102, 104, 105, 106, 108, 109, 110, 111, 114, 115, 116, 125, ..., among which, candidate point cloud layers 101 and 102 are continuous with a continuous layer count of 2, which is less than the layer count threshold, and are not combined into a point cloud layer group; candidate point cloud layers 104, 105, and 106 are continuous with a continuous layer count of 3, which is equal to the layer count threshold, and are combined into a point cloud layer group; candidate point cloud layers 108, 109, 110, and 111 are continuous with a continuous layer count of 4, which is greater than the layer count threshold, and are combined into a point cloud layer group; candidate point cloud layers 114, 115, and 116 are continuous with a continuous layer count of 3, which is equal to the layer count threshold, and are combined into a point cloud layer group, and so on.

[0102] Step S10433: Combine the point cloud layer groups and the interval layers with a layer number interval less than the preset interval threshold into a large point cloud layer group.

[0103] The specific value of the interval threshold can be set according to the actual situation, and this application embodiment does not impose specific limitations on it.

[0104] Here, we take an interval threshold of 3 as an example. Following the example in step S10432, the layer interval between point cloud group (104, 105, 106) and point cloud group (108, 109, 110, 111) is 1, meaning there is only one interval layer (107) in between, which is less than the interval threshold. The layer interval between point cloud group (108, 109, 110, 111) and point cloud group (114, 115, 116) is 2, meaning there are two interval layers (112 and 113) in between. If the interval layers are less than the interval threshold, then the point cloud layer groups (104, 105, 106), (108, 109, 110, 111), (114, 115, 116) and their corresponding interval layers 107, 112, and 113 can be combined into a large point cloud layer group, namely: (104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116), and so on.

[0105] Step S10434: Determine the bounding rectangles corresponding to each point cloud layer group, and take the point cloud layer group with the largest bounding rectangle area as the target point cloud layer group.

[0106] For each large group of point cloud layers, the set of pixels corresponding to the large group can first be determined within the 2D object detection bounding box. Then, the bounding rectangle of the pixel set is determined, and this bounding rectangle is set to the bounding rectangle corresponding to the large group of point cloud layers.

[0107] Step S10435: Calculate the area ratio between the bounding rectangle corresponding to the target point cloud group and the two-dimensional target detection box.

[0108] Step S10436: Determine whether the area ratio is greater than the preset ratio threshold.

[0109] The specific value of the ratio threshold can be set according to the actual situation, and this application embodiment does not impose a specific limitation on it.

[0110] If the area ratio is less than or equal to the ratio threshold, the interval threshold is increased, and the process returns to step S10433 and subsequent steps until the area ratio is greater than the ratio threshold; if the area ratio is greater than the ratio threshold, step S10437 is executed.

[0111] Step S10437: Determine the detection box corresponding to the target point cloud group as the three-dimensional target detection box corresponding to the target object.

[0112] Specifically, the bounding rectangle corresponding to the target point cloud group can be used to limit the range of the 3D target detection box in the Y and Z axis directions, and the distance information corresponding to each point cloud layer contained in the target point cloud group can be used to limit the range of the 3D target detection box in the X axis direction.

[0113] In the process of determining the 3D target detection box, the following assumption was used: the actual bounding rectangle area of ​​the target object can be close to the area of ​​the 2D target detection box, and the point cloud of the target object has a certain continuity or weak continuity in the X-axis direction.

[0114] Step S105: Determine the size and position of the target object based on the 3D target detection box.

[0115] Specifically, the projection length of the 3D target detection box on the X-axis can be used as the width of the target object, the projection length of the 3D target detection box on the Y-axis can be used as the length of the target object, the projection length of the 3D target detection box on the Z-axis can be used as the height of the target object, and the center position coordinates of the 3D target detection box can be used as the center position coordinates of the target object.

[0116] In summary, this application embodiment acquires a two-dimensional image and a depth image in a specified direction; performs two-dimensional target detection on a preset target object in the two-dimensional image to obtain a two-dimensional target detection box corresponding to the target object; converts the depth image into point cloud data, and determines a point cloud set corresponding to the two-dimensional target detection box in the point cloud data; performs three-dimensional target detection on the target object in the point cloud set to obtain a three-dimensional target detection box corresponding to the target object; and determines the size and position of the target object based on the three-dimensional target detection box. In this application embodiment, to address the problems of a single RGB camera not being able to obtain accurate distance information and the high cost of multi-line LiDAR, depth image data from a depth camera is introduced, which can obtain more accurate object distance information and reduce hardware costs; to address the problem that multi-line LiDAR cannot detect small objects, the result of two-dimensional target detection is used as the basis for three-dimensional target detection, which can be applied to various objects and sizes; and by fusing the result of two-dimensional target detection with the point cloud data obtained from the depth image conversion, fast and accurate three-dimensional target detection can be achieved on a low-hardware computing platform, allowing the robot to obtain more accurate and refined environmental information.

[0117] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0118] Corresponding to the three-dimensional target detection method described in the above embodiments, Figure 4 This diagram illustrates a structural diagram of an embodiment of a three-dimensional target detection device provided in this application.

[0119] In this embodiment, a three-dimensional target detection device may include:

[0120] Image acquisition module 401 is used to acquire a two-dimensional image and a depth image in a specified direction;

[0121] The two-dimensional target detection module 402 is used to perform two-dimensional target detection on a preset target object in the two-dimensional image to obtain a two-dimensional target detection box corresponding to the target object.

[0122] The point cloud set determination module 403 is used to convert the depth image into point cloud data and determine the point cloud set corresponding to the two-dimensional target detection box in the point cloud data;

[0123] The three-dimensional target detection module 404 is used to perform three-dimensional target detection on the target object in the point cloud set and obtain a three-dimensional target detection box corresponding to the target object.

[0124] The target object information determination module 405 is used to determine the size and position of the target object based on the three-dimensional target detection box.

[0125] In one specific implementation of this application embodiment, the three-dimensional target detection module may include:

[0126] The layered processing submodule is used to perform layered processing on the point cloud set according to the distance information in the point cloud set to obtain each point cloud layer;

[0127] The point cloud count statistics submodule is used to count the number of points in each point cloud layer.

[0128] The 3D target detection submodule is used to determine the 3D target detection box corresponding to the target object based on the number of points in each point cloud layer.

[0129] In one specific implementation of this application embodiment, the hierarchical processing submodule may include:

[0130] The normalization processing unit is used to normalize the distance information in the point cloud set to obtain normalized distance information;

[0131] The layering processing unit is used to perform layering processing on the point cloud set according to the normalized distance information to obtain each point cloud layer; wherein, each point cloud in the same point cloud layer has the same normalized distance information.

[0132] In one specific implementation of this application embodiment, the three-dimensional target detection submodule may include:

[0133] The candidate point cloud layer determination unit is used to identify point cloud layers with a number of points greater than a preset point cloud number threshold as candidate point cloud layers.

[0134] The point cloud layer group determination unit is used to combine candidate point cloud layers with a consecutive number of layers greater than or equal to a preset number of layers into a point cloud layer group.

[0135] The point cloud layer group determination unit is used to combine the point cloud layer subgroups and the interval layers with a layer number interval less than a preset interval threshold into a point cloud layer group.

[0136] The outer rectangle determination unit is used to determine the outer rectangles corresponding to each point cloud layer group, and to take the point cloud layer group with the largest outer rectangle area as the target point cloud layer group.

[0137] An area ratio calculation unit is used to calculate the area ratio between the bounding rectangle corresponding to the target point cloud group and the two-dimensional target detection box.

[0138] A three-dimensional target detection box determination unit is used to determine the detection box corresponding to the target point cloud group as the three-dimensional target detection box corresponding to the target object if the area ratio is greater than a preset ratio threshold.

[0139] In one specific implementation of this application embodiment, the three-dimensional target detection submodule may further include:

[0140] An interval threshold adjustment unit is used to increase the interval threshold if the area ratio is less than or equal to the ratio threshold, and return to the step of combining the point cloud layer groups with layer intervals less than the preset interval threshold and the interval layers into a large point cloud layer group and subsequent steps, until the area ratio is greater than the ratio threshold.

[0141] In one specific implementation of this application, the circumscribed rectangle determination unit can be specifically used to: for each large group of point cloud layers, determine a set of pixels corresponding to the large group of point cloud layers in the two-dimensional target detection box; determine the circumscribed rectangle of the pixel set, and determine the circumscribed rectangle of the pixel set as the circumscribed rectangle corresponding to the large group of point cloud layers.

[0142] In one specific implementation of this application embodiment, the three-dimensional target detection submodule may further include:

[0143] A point cloud number threshold determination unit is used to calculate the number of pixels in the two-dimensional target detection box; if the number of pixels is less than a preset pixel number threshold, the point cloud number threshold is determined based on the number of pixels; if the number of pixels is greater than or equal to the pixel number threshold, pixel sampling is performed in the two-dimensional target detection box to obtain a sampled two-dimensional target detection box; the number of sampled pixels in the sampled two-dimensional target detection box is calculated, and the point cloud number threshold is determined based on the number of sampled pixels.

[0144] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0145] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0146] Figure 5 A schematic block diagram of a robot provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0147] like Figure 5 As shown, the robot 5 in this embodiment includes a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. When the processor 50 executes the computer program 52, it implements the steps described in the various embodiments of the three-dimensional target detection methods above, for example... Figure 1 Steps S101 to S105 are shown. Alternatively, when the processor 50 executes the computer program 52, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 4 The functions of modules 401 to 405 are shown.

[0148] For example, the computer program 52 may be divided into one or more modules / units, which are stored in the memory 51 and executed by the processor 50 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 52 in the robot 5.

[0149] Those skilled in the art will understand that Figure 5 This is merely an example of robot 5 and does not constitute a limitation on robot 5. It may include more or fewer parts than shown, or combine certain parts, or different parts. For example, robot 5 may also include input / output devices, network access devices, buses, etc.

[0150] The processor 50 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0151] The memory 51 can be an internal storage unit of the robot 5, such as a hard drive or memory. The memory 51 can also be an external storage device of the robot 5, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 51 can include both internal and external storage units of the robot 5. The memory 51 is used to store the computer program and other programs and data required by the robot 5. The memory 51 can also be used to temporarily store data that has been output or will be output.

[0152] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0153] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0154] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0155] In the embodiments provided in this application, it should be understood that the disclosed devices / robots and methods can be implemented in other ways. For example, the device / robot embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0156] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0158] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0159] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A three-dimensional object detection method, characterized by, The method comprises the following steps: acquiring a two-dimensional image and a depth image in a specified direction; performing two-dimensional target detection on a preset target object in the two-dimensional image to obtain a two-dimensional target detection frame corresponding to the target object; converting the depth image into point cloud data and determining a point cloud set corresponding to the two-dimensional target detection frame in the point cloud data; performing hierarchical processing on the point cloud set according to distance information in the point cloud set to obtain each point cloud layer; and respectively counting the number of point clouds in each point cloud layer; taking a point cloud layer with a point cloud number greater than a preset point cloud number threshold as a candidate point cloud layer; combining each candidate point cloud layer with a number of layers greater than or equal to a preset layer number threshold into a point cloud layer group; combining each point cloud layer group and an interval layer with an interval less than a preset interval threshold into a point cloud layer group; determining an external rectangle corresponding to each point cloud layer group, respectively; and taking a point cloud layer group with the largest external rectangle area as a target point cloud layer group; calculating an area ratio between the external rectangle corresponding to the target point cloud layer group and the two-dimensional target detection frame; if the area ratio is greater than a preset ratio threshold, taking a detection frame corresponding to the target point cloud layer group as a three-dimensional target detection frame corresponding to the target object; determining the size and position of the target object according to the three-dimensional target detection frame.

2. The three-dimensional object detection method of claim 1, wherein, The hierarchical processing on the point cloud set according to the distance information in the point cloud set comprises the following steps: performing normalization processing on the distance information in the point cloud set to obtain normalized distance information; performing hierarchical processing on the point cloud set according to the normalized distance information to obtain each point cloud layer; wherein each point cloud in the same point cloud layer has the same normalized distance information.

3. The three-dimensional object detection method of claim 1, wherein, After calculating the area ratio between the external rectangle corresponding to the target point cloud layer group and the two-dimensional target detection frame, the method further comprises the following steps: if the area ratio is less than or equal to the ratio threshold, increasing the interval threshold, and returning to perform the step of combining each point cloud layer group and an interval layer with an interval less than a preset interval threshold into a point cloud layer group and the subsequent steps thereof until the area ratio is greater than the ratio threshold.

4. The three-dimensional object detection method of claim 1, wherein, The determination of the external rectangle corresponding to each point cloud layer group comprises the following steps: for each point cloud layer group, determining a set of pixel points corresponding to the point cloud layer group in the two-dimensional target detection frame; determining an external rectangle of the set of pixel points, and taking the external rectangle of the set of pixel points as the external rectangle corresponding to the point cloud layer group.

5. The three-dimensional object detection method of any one of claims 1 to 4, wherein, Before taking a point cloud layer with a point cloud number greater than a preset point cloud number threshold as a candidate point cloud layer, the method further comprises the following steps: calculating the number of pixels in the two-dimensional target detection frame; if the number of pixels is less than a preset pixel number threshold, determining the point cloud number threshold according to the number of pixels; if the number of pixels is greater than or equal to the pixel number threshold, performing pixel sampling in the two-dimensional target detection frame to obtain a sampled two-dimensional target detection frame; calculating the number of sampled pixels in the sampled two-dimensional target detection frame, and determining the point cloud number threshold according to the number of sampled pixels.

6. A three-dimensional object detection apparatus characterized by comprising: The method comprises the following steps: An image acquisition module is configured to acquire a two-dimensional image and a depth image in a specified direction; A two-dimensional target detection module is configured to perform two-dimensional target detection on a preset target object in the two-dimensional image to obtain a two-dimensional target detection frame corresponding to the target object; A point cloud set determination module is configured to convert the depth image into point cloud data and determine a point cloud set corresponding to the two-dimensional target detection frame in the point cloud data; A three-dimensional target detection module is configured to perform hierarchical processing on the point cloud set according to distance information in the point cloud set to obtain each point cloud layer, and count the number of point clouds in each point cloud layer; Point cloud layers with a number of point clouds greater than a preset point cloud number threshold are regarded as candidate point cloud layers; Point cloud layers with a number of layers greater than or equal to a preset layer number threshold are combined into a point cloud layer group, and point cloud layer groups and interval layers with a layer interval less than a preset interval threshold are combined into a point cloud layer group; an outer rectangle corresponding to each point cloud layer group is determined, and a point cloud layer group with the largest outer rectangle area is regarded as a target point cloud layer group; an area ratio between the outer rectangle corresponding to the target point cloud layer group and the two-dimensional target detection frame is calculated; if the area ratio is greater than a preset ratio threshold, a detection frame corresponding to the target point cloud layer group is determined as a three-dimensional target detection frame corresponding to the target object; A target object information determination module is configured to determine the size and position of the target object according to the three-dimensional target detection frame.

7. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 6. The computer program is executed by the processor to implement the steps of the three-dimensional target detection method according to any one of claims 1 to 5.

8. A robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the three-dimensional target detection method according to any one of claims 1 to 5.

Images