A high-precision depth camera

Abstract

A high-precision depth camera comprises a projector for projecting laser light to a target object, a first receiver for receiving a first reflection signal of the laser light, a second receiver for receiving a second reflection signal of the laser light, and a processor for obtaining TOF depth data according to a phase difference between the first reflection signal and the emission signal, generating binocular depth data according to parallax of the first reflection signal and the second reflection signal, generating binocular 3D point cloud and TOF 3D point cloud respectively according to the binocular depth data and the TOF depth data, correcting the TOF 3D point cloud by using the binocular 3D point cloud, and fusing the binocular 3D point cloud and the TOF 3D point cloud to obtain high-precision depth data. The application combines binocular technology and TOF technology to obtain two kinds of depth data of the same scene, processes point cloud data to obtain high-precision, complete and accurate depth data, and makes three-dimensional reconstruction better.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of depth cameras, in particular, to a high-precision depth camera. BACKGROUND

[0002] With the development of the "lazy economy", the market size of household sweeping machines is getting larger and larger, and their intelligence is also improving. Automatic navigation, automatic obstacle avoidance, and automatic cleaning are becoming more and more common. The automatic obstacle avoidance function mainly relies on 3D vision sensors, such as line structured light sensors, binocular stereo cameras, ToF cameras, and monocular speckle structured light depth sensors.

[0003] These sensors have their own advantages and disadvantages. The line structured light can completely reconstruct the scene depth at the laser bright stripe, but its depth field of view is small. The binocular stereo camera is prone to matching failure in weak texture areas, resulting in depth reconstruction failure, and the calculation amount is large. The ToF camera has a multipath effect, which causes depth distortion in corner scenes, affecting the ranging result, and there are many flying points, which interfere with obstacle segmentation. The monocular speckle structured light is prone to depth missing in small obstacles.

[0004] In order to make the obstacle avoidance effect of the sweeping machine meet various common scenes in the family as much as possible and improve the user experience, the various disadvantages brought by the single vision sensor need to be solved. Therefore, we have invented a depth fusion algorithm that can well fuse binocular depth data and ToF depth data, play their respective strengths, and avoid their respective shortcomings. SUMMARY

[0005] Therefore, the present application combines binocular technology and ToF technology to obtain two kinds of depth data of the same scene. Through processing of the point cloud data, high-precision, complete, and accurate depth data are obtained, and the three-dimensional reconstruction effect is better.

[0006] In a first aspect, the present application provides a high-precision depth camera, characterized in that it comprises:

[0007] a projector for projecting laser light towards a target object;

[0008] a first receiver for receiving a first reflection signal of the laser light;

[0009] a second receiver for receiving a second reflection signal of the laser light;

[0010] a processor configured to obtain TOF depth data according to a phase difference between the first reflected signal and the emitted signal, generate binocular depth data according to a parallax between the first reflected signal and the second reflected signal, generate binocular 3D point cloud and TOF 3D point cloud respectively according to the binocular depth data and the TOF depth data, correct the TOF 3D point cloud using the binocular 3D point cloud, and fuse the binocular 3D point cloud and the TOF 3D point cloud to obtain high-precision depth data.

[0011] Optionally, the high-precision depth camera comprises:

[0012] a obtaining module configured to obtain binocular depth data and TOF depth data of a same scene, and generate binocular 3D point cloud and TOF 3D point cloud respectively;

[0013] a binocular module configured to identify a plane in the binocular 3D point cloud, and perform plane fitting;

[0014] a TOF module configured to identify a ground surface near portion and a ground surface above near portion in the TOF 3D point cloud, and segment an obstacle according to normal vector information of a surface of the ground surface near portion; wherein the ground surface near portion refers to a point cloud portion within a first threshold from the ground surface;

[0015] a correction module configured to correct a depth value of the obstacle in the TOF depth data using the binocular depth data;

[0016] a fusion module configured to fuse the binocular depth data and the corrected TOF depth data to obtain high-precision depth data.

[0017] Optionally, the high-precision depth camera comprises:

[0018] a center point unit configured to identify a ground surface in the binocular 3D point cloud, and identify a segmentation line of adjacent ground surfaces;

[0019] a segmentation unit configured to divide the ground surface into a first portion and a second portion according to the segmentation line;

[0020] a fitting unit configured to perform plane fitting on the first portion and the second portion respectively.

[0021] Optionally, the high-precision depth camera comprises:

[0022] an inheritance unit configured to obtain a position of a segmentation line in a previous image;

[0023] a fine-tuning unit configured to, on a current image, expand outward according to the position of the segmentation line to obtain an expansion region, and determine the position of the segmentation line of the current image by a normal vector direction in the expansion region;

[0024] a segmentation unit configured to divide the ground into a first part and a second part according to the segmentation line;

[0025] a fitting unit configured to perform plane fitting on the first part and the second part respectively.

[0026] Optionally, the high-precision depth camera has the features that the TOF module comprises:

[0027] an alignment unit configured to align the binocular depth data and the TOF depth data;

[0028] a first marking unit configured to mark, in the TOF 3D point cloud, a pixel point with a distance from the plane not greater than a first threshold value as a ground vicinity part, and mark a pixel point with a distance from the plane greater than the first threshold value as a ground vicinity above part;

[0029] a normal vector unit configured to calculate a normal vector of each pixel point in the ground vicinity part, and calculate an included angle a between the normal vector and a ground normal vector;

[0030] a second marking unit configured to mark the pixel point as a first pixel when the included angle a is greater than a second threshold value;

[0031] a third marking unit configured to mark an object to which the first pixel belongs as an obstacle when a number of adjacent first pixels is greater than a third threshold value.

[0032] Optionally, the high-precision depth camera has the features that the correction module comprises:

[0033] an alignment unit configured to align the binocular depth data and the TOF depth data;

[0034] a first correction unit configured to correct, for the binocular depth data, corresponding TOF depth data of the obstacle;

[0035] a second correction unit configured to correct uncorrected TOF depth data according to the corrected TOF depth data.

[0036] Optionally, the high-precision depth camera has the features that the first receiver and the second receiver start exposure at the same time.

[0037] Optionally, the high-precision depth camera, characterized in that if the processor fails to partially reconstruct the binocular 3D point cloud, the TOF 3D point cloud data is directly used as the final data for the partial data.

[0038] In a second aspect, the application provides a robot, characterized in that it comprises the high-precision depth camera according to any one of the above.

[0039] In a third aspect, the application provides a vehicle, characterized in that it comprises the high-precision depth camera according to any one of the above.

[0040] Compared with the prior art, the application has the following beneficial effects:

[0041] The application simultaneously uses binocular technology and TOF technology, utilizes the respective characteristics of the two technologies, and makes up for each other's shortcomings, so that the quality of the final depth data is better than that of a single technology, and more accurate and comprehensive three-dimensional reconstruction effects can be obtained, thereby making the obstacle avoidance and navigation applications more accurate.

[0042] The application can better distinguish obstacles by dividing the 3D point cloud into a ground near part and a part above the ground near part, especially for small-volume and small-height obstacles, thereby making the target judgment more accurate, and making the judgment of target objects by devices such as a sweeping robot more accurate.

[0043] The application combines binocular technology and TOF technology, so that the problems of multipath effect, noise points, discontinuity and the like in three-dimensional reconstruction are very well solved, the continuity and consistency of data are improved, and the data is more consistent with the real scene, so that it can be used for more accurate detection and detection services.

[0044] The application obtains two types of depth data by using one projector and two receivers, compared with the existing system, only one receiver needs to be added, the scheme is simple, and only a small amount of cost needs to be added, so that the measurement distance can be greatly improved, and the commercial promotion value is high. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description only constitute the embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor based on the provided drawings. Other features, objects and advantages of the application will become more apparent through reading the following detailed description of the non-limiting embodiments with reference to the following drawings:

[0046] Figure 1 Figure 9 is a schematic diagram of a comparison between binocular depth data and TOF depth data according to an embodiment of the present application;

[0047] Figure 2 Figure 10 is a schematic diagram of a high-precision depth camera according to an embodiment of the present application;

[0048] Figure 3 Figure 11 is a schematic diagram of a processor according to an embodiment of the present application;

[0049] Figure 4 Figure 12 is a schematic diagram of a binocular module according to an embodiment of the present application;

[0050] Figure 5 Figure 13 is a schematic diagram of another binocular module according to an embodiment of the present application;

[0051] Figure 6 Figure 14 is a schematic diagram of a TOF module according to an embodiment of the present application;

[0052] Figure 7 Figure 15 is a schematic diagram of a calibration module according to an embodiment of the present application;

[0053] Figure 8 Figure 16 is a schematic diagram of a robot according to an embodiment of the present application;

[0054] Figure 9 Figure 17 is a schematic diagram of a vehicle according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] The present application will be further described with reference to the following examples. The following examples are provided to further illustrate the application and should not be construed as limiting the application in any way. It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Such modifications and variations are intended to be included within the scope of the application.

[0056] The terms "first", "second", "third", "fourth" and the like in the description and in the claims, if any, of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms is interchangeable under appropriate circumstances such that the descriptive

[0057] The technical solutions of the present application will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes can not be described in detail in some examples.

[0058] The high-precision depth camera provided in the embodiments of the present application aims to solve the problems in the prior art.

[0059] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes can not be described in detail in some examples. The embodiments of the present application will be described below with reference to the drawings.

[0060] Figure 1 This is a comparison diagram of binocular depth data and TOF depth data in the embodiments of the present application. This embodiment uses binocular depth data and TOF depth data of a sweeping robot for comparison. Binocular technology is more suitable for application in small-volume devices such as sweeping robots, but it is prone to reconstruction failure in weak texture areas. The TOF technology is not affected by the surface texture of the object due to its technical principle, so it can obtain complete depth data of the target area during three-dimensional reconstruction, but there are factors such as multipath interference, which cause partial data distortion in strong texture areas such as corners.

[0061] As shown in Figure 1 , in binocular depth data, the data density received at a position closer to the camera is higher, and the data density received at a position farther from the camera is lower, which is shown in the figure as more point cloud data at a close distance and lower point cloud density at a long distance. In TOF depth data, the data is relatively more uniform, but there are more noise points, and compared with binocular depth data, the depth data of the ground is "abnormal", that is, part of the target object is below the ground, which is mainly caused by multipath interference, and this part of data can be corrected by binocular data.

[0062] The present application combines the advantages of binocular technology and TOF technology, and corrects the measurement results of TOF, so that the final depth data is more accurate and reliable.

[0063] Figure 2 This is a structural diagram of a high-precision depth camera in the embodiments of the present application. As shown in Figure 2 , the high-precision depth camera in the embodiments of the present application comprises:

[0064] The projector 100 is used to project laser light to the target object.

[0065] Specifically, the projector 100 periodically projects laser, and good return signals can be received at both close and long distances. The laser has high irradiation intensity, and after being reflected by the target object, it can be easily received by the receiver, thereby ensuring that signals can still be obtained at a long distance. The projector 100 can be an edge laser emitter, a vertical cavity surface emitting laser (VCSEL), or any other type of laser emitter, and the present application does not limit this.

[0066] The first receiver 200 is configured to receive the first reflected signal of the laser.

[0067] Specifically, the first receiver 200 selects a corresponding sensor according to the type of laser. For example, the first receiver 200 can be an infrared sensor for receiving a laser signal, and of course, a sensor that can receive the projector 100 can also be used.

[0068] The second receiver 300 is configured to receive the second reflected signal of the laser.

[0069] Specifically, the second receiver 300 and the first receiver 200 are both configured to receive the signal of the projector 100. The second receiver 300 can be the same as or different from the first receiver 200. Preferably, the second receiver 300 is the same as the first receiver 200, such as the same model of infrared sensor. The first reflected signal and the second reflected signal are both reflected signals of the laser projected by the projector 100, and the difference is only that they are received by different receivers. The first receiver and the second receiver start exposure at the same time.

[0070] The processor 400 is configured to obtain TOF depth data according to the phase difference between the first reflected signal and the emitted signal, generate binocular depth data according to the parallax of the first reflected signal and the second reflected signal, generate binocular 3D point cloud and TOF 3D point cloud respectively according to the binocular depth data and the TOF depth data, correct the TOF 3D point cloud using the binocular 3D point cloud, and then fuse the binocular 3D point cloud and the TOF 3D point cloud to obtain high-precision depth data.

[0071] Specifically, the processor 400 analyzes the signal phase of the first signal according to the proportional relationship of the energy values collected by the first receiver 200 at different time windows, indirectly measures the time difference between the emitted signal and the received signal, and then obtains the TOF depth data. The binocular depth data is generated according to the parallax of the first reflected signal and the second reflected signal. The TOF depth data is obtained according to the ToF technology, and the binocular depth data is obtained according to the binocular technology. The technical principles of the two are different, and therefore, the binocular depth data can be used to correct the TOF depth data.

[0072] The first receiver 200 can be one receiver or two or more receivers. The ratio of the number of the projectors 100 to the number of the first receivers 200 can be 1:1, 1:2, 2:1 or other ratios. When the ratio of the number of the projectors 100 to the number of the first receivers 200 is not 1:1, the ratio can be adjusted according to the cooperation state of the multiple projectors 100 and the multiple first receivers 200. The description of the first receiver 200 in this paragraph also applies to the second receiver 300. The number of the second receivers 300 can be equal to or different from the number of the first receivers 200.

[0073] The processor 400 generates a binocular 3D point cloud and a TOF 3D point cloud according to the binocular depth data and the TOF depth data. The TOF 3D point cloud data is corrected by using the binocular 3D point cloud data, so that the TOF 3D point cloud data is more accurate. The binocular 3D point cloud may have a local reconstruction failure problem at a weak texture, and the TOF 3D point cloud has a multipath effect at a strong texture such as a corner. Therefore, the binocular 3D point cloud after reconstruction can be used to correct the TOF 3D point cloud. If the binocular 3D point cloud partially fails to reconstruct, the TOF 3D point cloud data of the part is directly used as the final data.

[0074] Compared with correcting the TOF 3D point cloud by using the binocular 3D point cloud, the binocular 3D point cloud and the TOF 3D point cloud are fused to obtain high-precision depth data.

[0075] The embodiment integrates the ToF technology and the binocular technology, uses the advantages of the ToF technology and the binocular technology, has higher precision, and is better than the measurement effect of a single system. The embodiment only needs to add one receiver to an existing ToF system, so that the measurement quality and effect are greatly improved, and the embodiment has very high cost performance and commercial promotion value.

[0076] Figure 3 FIG. 1 is a structural schematic diagram of a processor according to an embodiment of the present application. As shown in FIG. 1, the processor according to an embodiment of the present application comprises: Figure 3

[0077] The acquisition module 100 is configured to acquire binocular depth data and TOF depth data of the same scene, and generate a binocular 3D point cloud and a TOF 3D point cloud, respectively.

[0078] ​Specifically, the binocular depth data and the TOF depth data of the same scene are usually shot within a visual angle of no more than 10 degrees, facilitating subsequent alignment and various operations. Of course, the alignment operation can be performed in the present module or in any subsequent module, and the present embodiment does not limit this. The binocular 3D point cloud is obtained through binocular depth data reconstruction, and the TOF 3D point cloud is obtained through TOF depth data reconstruction. The 3D point cloud is a reproduction of a three-dimensional space, which depends on the accuracy of the data. The more accurate the depth data, the more accurate the three-dimensional space after reconstruction. When calibrating the depth data, the device is usually calibrated, but the deviation caused by the technical characteristics cannot be overcome. The present embodiment can cross-verify the data in the 3D point cloud to obtain high-precision depth data.

[0079] The binocular module 200 is configured to identify a plane in the binocular 3D point cloud and perform plane fitting.

[0080] Specifically, in the 3D point cloud, the ground is fixed and unchanging, and thus can serve as an important reference for correction. The present embodiment divides the data in the three-dimensional space into ground data and obstacle data. The ground data is the data of the ground obtained, and is represented as a continuous plane, so that the ground can be fitted. The plane in the present module refers to the plane on which the ground lies, rather than the plane of the camera visual angle. This is because the visual angle of the camera usually has a certain angle deviation from the ground. At the same time, taking the ground as the plane also overcomes the problem of inconsistent data caused by various disturbance factors in the implementation of the shooting process, so that the ground data after three-dimensional reconstruction has better consistency.

[0081] The binocular module 200 can correct the data through plane fitting of the ground, especially in scenes where the ground data accounts for a high proportion, such as various low-speed moving robots such as sweeping robots.

[0082] The TOF module 300 is configured to identify a ground vicinity part and a part above the ground vicinity in the TOF 3D point cloud, and segment the obstacle according to the normal vector information of the surface of the ground vicinity part.

[0083] Specifically, the ground vicinity part refers to the part of the point cloud within a first threshold from the ground. Unlike the division of three-dimensional data into ground and obstacle in the binocular 3D point cloud, the three-dimensional data in the TOF 3D point cloud is divided into a ground vicinity part and a part above the ground vicinity. Due to the characteristics of the TOF technology, the accuracy of the data obtained by the ground is not high, so the ground vicinity part needs to be corrected using binocular data. The first threshold is usually a fixed value, but can be adjusted according to different TOF technologies and application scenarios.

[0084] The normal vector of the ground is perpendicular to the ground, while the normal vector of the obstacle is usually at a large angle with the normal vector of the ground, even 90 degrees or other angles. Therefore, the normal vector information of the surface near the ground can be used to identify the obstacle. When the angle between the normal vector and the ground is less than a certain angle, it is determined to be an obstacle.

[0085] The correction module 400 is configured to correct the depth value of the obstacle in the TOF depth data by using the binocular depth data.

[0086] Specifically, since the binocular depth data is more accurate than the TOF depth data above the ground, the binocular depth data can be used to correct the depth data of the obstacle in the TOF. In this module, the binocular depth data and the TOF depth data are already aligned. Due to the existence of the TOF multipath effect, it is easy to be distorted at the junctions such as corners, and the influence of these distortions on small obstacles is very large, so it must be corrected by the binocular depth data. The depth value corrected by this module includes both small obstacles and large obstacles.

[0087] The fusion module 500 is configured to fuse the binocular depth data and the corrected TOF depth data to obtain high-precision depth data.

[0088] Specifically, the corrected TOF depth data has high precision and is continuous, while the binocular depth data has better precision. This module fuses the binocular depth data and the corrected TOF depth data to obtain high-precision depth data that combines the advantages of binocular technology and TOF technology. When fusing the binocular depth data and the corrected TOF depth data, a suitable fusion method can be selected according to the specific technical characteristics and application scenarios.

[0089] For example, the ground data in the binocular depth data is combined with the obstacle data in the corrected TOF depth data to generate high-precision depth data. The binocular depth data is used for the ground, while the corrected TOF depth data is used for the obstacle, thereby obtaining high-precision depth data. Since the multipath interference exists in the TOF data, the ground data in the TOF depth data is different from the ground data in the binocular depth data, and therefore the data of the obstacle adjacent to the ground also has errors. When correcting and fusing, the abnormal data such as unconfirmed points and flying points in the TOF depth data need to be processed. The flying points can be considered as noise and directly deleted. For the part of the obstacle in the TOF depth data that is lower than the ground in the binocular 3D point cloud, this part is also deleted, and the corresponding part is re-corrected.

[0090] For example, the ground data in the binocular depth data and the ground data in the TOF depth data are given different weights according to the distance to generate final ground data, and then combined with the obstacle data in the corrected TOF depth data to generate high-precision depth data. The weight value is in 【0, 1】, and 0 and 1 are both available values. Because the use range of binocular technology and TOF technology does not completely overlap, different values can be used in different distances. The change of weight value with distance is not linear.

[0091] For example, different weight values are given to the binocular depth data and the TOF depth data according to the distance to the obstacle, so as to generate final ground data, and then combined with the obstacle data in the corrected TOF depth data to generate high-precision depth data. Higher weight values are given to the ground data in the binocular depth data near the obstacle, and higher weight values are given to the TOF depth data far from the obstacle. The weight value is in 【0, 1】, and 0 and 1 are both available values. The change of weight value with distance is not linear.

[0092] The embodiment reconstructs the depth data in three dimensions and calibrates in three-dimensional space, so that the data with strong correlation between the data are well verified and correlated, and the characteristics of binocular technology and TOF technology are combined to overcome the defects of single technology, so that the data correction effect is better, and high-precision depth data is obtained.

[0093] Figure 4 A schematic diagram of a binocular module structure in an embodiment of the application. Compared with the previous embodiment, the binocular module 200 in the embodiment includes:

[0094] The center point unit 210 is used to identify the ground in the binocular 3D point cloud and identify the segmentation line of adjacent ground.

[0095] Specifically, an XYZ coordinate system is established in the binocular 3D point cloud, and the ground is identified according to the normal vector and the Z-axis horizontal angle. The ground is usually a continuous area, so the average of the normal vectors of the pixels in the area can be taken as the normal vector of the area. When the selection of the area does not exceed 1 / 10 of the total area of the ground. The normal vectors of adjacent areas are different, indicating that the directions of the ground are different. The place where the direction of the normal vector changes is the place where the segmentation line is located, that is, the angle change of the normal vector can be used to identify the segmentation line of adjacent ground.

[0096] The segmentation unit 220 is used to divide the ground into a first part and a second part according to the segmentation line.

[0097] Specifically, the first part is the part closer to the shooting position. The second part is the part farther from the shooting position. For example, when a depth camera is used for shooting, the first part is the ground part closest to the camera, and the farther part is the second part. The second part can include one plane or two or more planes. For example, when the ground includes three different slopes, the plane where the shooting position is located is the first part, and the other two planes are the second part.

[0098] The fitting unit 230 is configured to perform plane fitting on the first part and the second part respectively.

[0099] Specifically, plane fitting is performed on the first part. If the second part has only one plane, plane fitting is performed on it. If the second part has two or more planes, fitting is performed on each plane.

[0100] The embodiment divides the ground into different parts and performs fitting respectively, so that the plane fitting effect of the ground is better, and the subsequent navigation, path planning, obstacle recognition and other actions are more accurate, so that the application can be applied to more application scenarios.

[0101] Figure 5 Another schematic diagram of a binocular module structure in the embodiment of the application is shown in FIG. 2B, which is suitable for continuous processing of the same scene image. Compared with the previous embodiment, the binocular module 200 in this embodiment includes:

[0102] The inheritance unit 240 is configured to obtain the position of the segmentation line in the previous image.

[0103] Specifically, the positions of all segmentation lines in the previous image are obtained. If there is only one segmentation line in the previous image, one segmentation line is obtained; if there are two segmentation lines in the previous image, two segmentation lines are obtained.

[0104] The fine-tuning unit 250 is configured to expand outward according to the position of the segmentation line to obtain an expansion area in the current image, and determine the position of the segmentation line in the current image by the direction of the normal vector in the expansion area.

[0105] Specifically, the range of the expansion area is determined according to the speed of the change in the angle of view. For example, when shooting on a sweeping robot, different ranges can be selected according to the speed of the sweeping robot. When the speed is large, the range is also large, and when the speed is small, the range is also small. The position of the segmentation line in the current image can be roughly located by the position of the segmentation line, and the same method as in the previous embodiment is used to determine the position of the segmentation line in the current image, i.e., by the direction of the normal vector.

[0106] The segmentation unit 220 is configured to divide the ground into a first part and a second part according to the segmentation line.

[0107] The fitting unit 230 is configured to perform plane fitting on the first part and the second part respectively.

[0108] The embodiment utilizes the recognition result of the previous image, greatly reduces the data processing amount, and still can accurately obtain the information of the segmentation line, so that the information can be obtained in real time in continuous shooting, and the data processing capacity and timely response capacity are improved.

[0109] Figure 6 A schematic diagram of a TOF module structure in an embodiment of the present application is shown in FIG. 3. Compared with the previous embodiment, the TOF module 300 in the present embodiment comprises:

[0110] The alignment unit 310 is configured to align the binocular depth data and the TOF depth data.

[0111] Specifically, the alignment of the binocular depth data and the TOF depth data is the basis for subsequent operations. If the alignment has been performed in the foregoing module, the alignment unit 310 can be omitted.

[0112] The first marking unit 320 is configured to mark the pixel points with a distance from the plane not greater than the first threshold value as the ground surface nearby part, and mark the pixel points with a distance from the plane greater than the first threshold value as the ground surface above part in the TOF 3D point cloud.

[0113] Specifically, the data in the TOF 3D point cloud is marked according to the distance from the plane. The data of the ground surface nearby part needs to be corrected according to the binocular depth data and the obstacles are identified, and the data of the ground surface above part does not need to be corrected.

[0114] The normal vector unit 330 is configured to calculate the normal vector of each pixel point in the ground surface nearby part, and calculate the included angle a between the normal vector and the ground surface normal vector.

[0115] Specifically, the normal vector is calculated in units of pixel points. The ground surface in the present module refers to the ground surface region closest to the pixel points, and therefore the mean value of the normal vectors of the pixel points in the ground surface region can be taken as the ground surface normal vector. It should be noted that the region is not the set of all ground surface pixels, but part of all ground surface pixels.

[0116] The second marking unit 340 is configured to mark the pixel point as a first pixel when the included angle a is greater than a second threshold value.

[0117] Specifically, when the included angle a is greater than the second threshold value, it is known that the angle with the ground is larger, and it can be part of the obstacle. The value of the second threshold value can be selected according to different application scenarios. For example, the ground with ceramic tiles is relatively smooth, and the included angle a can be smaller to improve the identification ability for small obstacles. The rough road surface is relatively rough, and the included angle a can be larger to improve the stability of identification. Generally, the included angle a can be 15, 20, 25, 30, 35, 40 degrees or other angles.

[0118] The third marking unit 350 is configured to mark the object to which the first pixel belongs as an obstacle when the number of adjacent first pixels is greater than a third threshold value.

[0119] Specifically, a single pixel point can be a flying point. A small number of adjacent pixel points can be caused by uneven ground, so a third threshold value needs to be set to filter out the above two cases. When identifying the first pixel, the object also needs to be identified to obtain the part that can be the same object, such as judging according to the mutation of the pixel normal vector.

[0120] The embodiment can identify small obstacles according to the normal vector information, filter out the influence of flying points and surface defects, balance the recognition degree and stability, and achieve good recognition effect in various application places.

[0121] Figure 7 A schematic diagram of a correction module structure in an embodiment of the present application is shown. Compared with the previous embodiment, the correction module 400 in the present embodiment includes:

[0122] The alignment unit 310 is configured to align the binocular depth data and the TOF depth data.

[0123] Specifically, aligning the binocular depth data and the TOF depth data is the basis for the subsequent modules. If the binocular depth data and the TOF depth data have been aligned in the foregoing modules, the alignment unit 310 can no longer be set.

[0124] The first correction unit 420 is configured to correct the corresponding TOF depth data of the obstacle for the binocular depth data.

[0125] Specifically, since the binocular depth data has higher accuracy, the corresponding TOF depth data is corrected using the binocular depth data to make the TOF data more accurate. Since the binocular uses spot technology, the accurate data obtained is not continuous, so only part of the TOF depth data can be corrected. The second correction unit 430 is run after the first correction unit 420.

[0126] The second correction unit 430 is configured to correct the uncorrected TOF depth data according to the corrected TOF depth data.

[0127] Specifically, in the 3D point cloud, the corrected TOF depth data is used to correct the adjacent uncorrected TOF depth data. Since the corrected TOF depth data is independent of each other and there is uncorrected data between them, there are data mutations and the like. The uncorrected TOF depth data can be pulled back by using algorithms such as linear regression, local weighted regression, and the like.

[0128] The embodiment corrects the TOF depth data by using the binocular depth data, and uses the corrected data to regress the uncorrected data in the TOF depth data, so that the TOF depth data is more accurate and comprehensive, and high-precision depth data can be obtained.

[0129] Figure 8 A schematic diagram of a robot structure in an embodiment of the present application. Figure 6 The embodiment is still applicable to other types of robots such as humanoid robots, meal delivery robots, traffic directing robots, unmanned aerial vehicles, and the like, as understood by those skilled in the art.

[0130] As shown in FIG. 6, Figure 8 The robot includes a body 600 and a depth camera. The depth camera includes a first receiver 601, a projector 602, and a second receiver 603. The first receiver 601, the projector 602, and the second receiver 603 are adjacently arranged on the front side of the robot, and the projector 602 is located at the central position of the first receiver 601 and the second receiver 603, so that the signals received by the first receiver 601 and the second receiver 603 have symmetry, which is more convenient for later processing. Since the working environment of the robot is complex, the depth camera is arranged in a protective cover, which can play a protective role. The protective cover can prevent the depth camera from being damaged by external force, and also has a certain waterproof effect, so that the depth camera is not soaked by excessive water.

[0131] Figure 9 A schematic diagram of a vehicle structure in an embodiment of the present application. Figure 9 The embodiment is still applicable to other types of vehicles such as trucks, cars, trains, buses, and the like, as understood by those skilled in the art.

[0132] As shown in FIG. 6, Figure 9As shown, the vehicle includes a vehicle body 700 and a depth camera. The depth camera includes a first receiver 710, a projector 720 and a second receiver 730. The first receiver 710, the projector 720 and the second receiver 730 are adjacently arranged in front of the vehicle. The projector 720 is located at the central position of the first receiver 710 and the second receiver 730, and the projector 720 is located in the center of the vehicle, so that the signals received by the first receiver 710 and the second receiver 730 have symmetry, which is more convenient for later processing. Due to the characteristics of the vehicle, it needs to identify targets at a long distance during driving, and also needs to identify targets at a short distance when the vehicle speed is slow, and the depth camera currently installed in the vehicle cannot measure the distance in such a wide range. And due to the consideration of the appearance and cost of the vehicle, it is not possible to install too many probes, so the scheme in the present application can be better applied to vehicles. When the vehicle is driving in a long-distance scene such as a highway, ToF can be used to obtain data at a farther distance, and the measurement distance is farther than in the prior art. When the vehicle is driving in a congested road section, the binocular system can be used to obtain data at a short distance, and detailed data such as the other vehicle license plate can be obtained, which is a technical effect that cannot be achieved by dToF or distance sensors in the prior art.

[0133] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0134] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the specific embodiments described above, and various modifications or changes can be made by those skilled in the art within the scope of the claims, which do not affect the essential content of the present application.

Claims

1. A high-precision depth camera, characterized by, The application relates to a high-precision depth data acquisition method and device. The application comprises: a projector for projecting laser light to a target object; a first receiver for receiving a first reflected signal of the laser light; a second receiver for receiving a second reflected signal of the laser light; a processor for obtaining TOF depth data according to the phase difference between the first reflected signal and the emitted signal, generating binocular depth data according to the parallax of the first reflected signal and the second reflected signal, generating binocular 3D point cloud and TOF 3D point cloud respectively according to the binocular depth data and the TOF depth data, correcting the TOF 3D point cloud by using the binocular 3D point cloud, and fusing the binocular 3D point cloud and the TOF 3D point cloud to obtain high-precision depth data; the processor comprises: an acquisition module for acquiring binocular depth data and TOF depth data of the same scene, and generating binocular 3D point cloud and TOF 3D point cloud respectively; a binocular module for identifying a plane in the binocular 3D point cloud and performing plane fitting; a TOF module for identifying a ground surface near portion and a ground surface near above portion in the TOF 3D point cloud, and segmenting an obstacle according to the normal vector information of the surface of the ground surface near portion; wherein the ground surface near portion refers to the point cloud portion within a first threshold distance from the ground surface; a correction module for correcting the depth value of the obstacle in the TOF depth data by using the binocular depth data; 2. The high precision depth camera of claim 1, wherein, a fusion module for fusing the binocular depth data and the corrected TOF depth data to obtain high-precision depth data. The binocular module comprises: a center point unit for identifying a ground surface in the binocular 3D point cloud and identifying a segmentation line of adjacent ground surfaces; a segmentation unit for dividing the ground surface into a first portion and a second portion according to the segmentation line; 3. The high precision depth camera of claim 1, wherein, a fitting unit for performing plane fitting on the first portion and the second portion respectively. The binocular module comprises: an inheritance unit for acquiring the position of a segmentation line in a previous image; a fine-tuning unit for expanding outward to obtain an expansion area according to the position of the segmentation line in a current image, and determining the position of the segmentation line in the current image by the direction of the normal vector in the expansion area; a segmentation unit for dividing the ground surface into a first portion and a second portion according to the segmentation line; 4. The high precision depth camera of claim 1, wherein, a fitting unit for performing plane fitting on the first portion and the second portion respectively. The TOF module comprises: an alignment unit for aligning the binocular depth data and the TOF depth data; a first marking unit for marking a pixel point with a distance not greater than the first threshold from the plane as a ground surface near portion, and marking a pixel point with a distance greater than the first threshold from the plane as a ground surface near above portion in the TOF 3D point cloud; a normal vector unit for calculating the normal vector of each pixel point in the ground surface near portion, and calculating the included angle a between the normal vector and the ground surface normal vector; a second marking unit for marking the pixel point as a first pixel when the included angle a is greater than a second threshold. A third marking unit is configured to mark the object to which the first pixel belongs as an obstacle when the number of adjacent first pixels is greater than a third threshold.

5. The high precision depth camera of claim 1, wherein, The correction module comprises: An alignment unit is configured to align the binocular depth data and the TOF depth data; A first correction unit is configured to correct the corresponding TOF depth data of the obstacle for the binocular depth data; A second correction unit is configured to correct the uncorrected TOF depth data according to the corrected TOF depth data.

6. The high precision depth camera of claim 1, wherein, The first receiver and the second receiver start exposure at the same time.

7. The high precision depth camera of claim 1, wherein, If the processor fails to partially reconstruct the binocular 3D point cloud, the TOF 3D point cloud data is directly used as the final data for the partial data.

8. A robot, characterized in that A high-precision depth camera comprising any one of claims 1-7.

9. A vehicle characterized by comprising: A high-precision depth camera comprising any one of claims 1-7.

Images