A method for fusing binocular depth data and TOF depth data

Abstract

A method for deep fusion of binocular depth data and Time-of-Flight (TOF) depth data includes the following steps: Step S1: Acquire binocular depth data and TOF depth data of the same scene, and generate binocular 3D point clouds and TOF 3D point clouds respectively; Step S2: Identify planes in the binocular 3D point clouds and perform plane fitting; Step S3: Identify the portion near the ground and the portion above the ground in the TOF 3D point clouds, and segment obstacles based on the normal vector information of the surface of the portion near the ground; wherein, the portion near the ground refers to the point cloud portion within a first threshold distance from the ground; Step S4: Correct the depth value of the obstacles in the TOF depth data using the binocular depth data; Step S5: Fuse the binocular depth data and the corrected TOF depth data to obtain high-precision depth data. This invention combines binocular technology and TOF technology, resulting in better 3D reconstruction effects.

Description

Technical Field

[0001] This invention relates to the field of image data processing, and more specifically, to a method for deep fusion of binocular depth data and TOF depth data. Background Technology

[0002] With the development of the "lazy economy," the market size of household robotic vacuum cleaners is growing, and their intelligence level is constantly improving, with functions such as automatic navigation, automatic obstacle avoidance, and automatic cleaning becoming increasingly common. Among them, the automatic obstacle avoidance function mainly relies on 3D vision sensors, such as line structured light sensors, binocular stereo cameras, ToF cameras, and monocular speckle structured light depth sensors.

[0003] These sensors each have their own advantages and disadvantages. Line structured light can completely reconstruct the scene depth at the laser bright stripe, but its depth field of view is small. Binocular stereo cameras are prone to matching failures in weak texture areas, leading to depth reconstruction failures, and the computational load is large. ToF cameras have multipath effects, which can easily cause depth distortion in scenes such as corners, affecting the ranging results, and there are many flying points, which interfere with obstacle segmentation. Monocular speckle structured light is prone to depth loss at small obstacles.

[0004] To ensure that the obstacle avoidance performance of robotic vacuum cleaners can meet the needs of various common household scenarios and improve the user experience, it is necessary to address the various drawbacks of the single vision sensor. To this end, we have invented a deep fusion algorithm that can effectively integrate binocular depth data and ToF depth data, leveraging the strengths of each while avoiding their respective weaknesses. Summary of the Invention

[0005] Therefore, this invention combines binocular technology with TOF technology to acquire two types of depth data for the same scene. By processing the point cloud data, high-precision, complete, and accurate depth data is obtained, resulting in better 3D reconstruction.

[0006] In a first aspect, the present invention provides a method for deep fusion of binocular depth data and TOF depth data, characterized by comprising the following steps:

[0007] Step S1: Acquire binocular depth data and TOF depth data of the same scene, and generate binocular 3D point cloud and TOF 3D point cloud respectively;

[0008] Step S2: Identify the plane in the binocular 3D point cloud and perform plane fitting;

[0009] Step S3: Identify the portion near the ground and the portion above the ground in the TOF3D point cloud, and segment the obstacle based on the normal vector information of the surface of the portion near the ground; wherein, the portion near the ground refers to the point cloud portion within a first threshold distance from the ground;

[0010] Step S4: Use the binocular depth data to correct the depth value of the obstacle in the TOF depth data;

[0011] Step S5: Fuse the binocular depth data with the corrected TOF depth data to obtain high-precision depth data.

[0012] Optionally, the method for deep fusion of binocular depth data and TOF depth data is characterized in that step S2 includes:

[0013] S21: Identify the ground in the binocular 3D point cloud and identify the dividing lines between adjacent ground surfaces;

[0014] S22: Divide the ground into a first part and a second part according to the dividing line;

[0015] S23: Perform planar fitting on the first part and the second part respectively.

[0016] Optionally, the method for deep fusion of binocular depth data and TOF depth data is characterized in that step S2 includes:

[0017] S24: Obtain the position of the segmentation line in the previous image;

[0018] S25: On the current image, expand outwards according to the position of the segmentation line to obtain an extended region, and determine the position of the current image segmentation line within the extended region by the direction of the normal vector.

[0019] S22: Divide the ground into a first part and a second part according to the dividing line;

[0020] S23: Perform planar fitting on the first part and the second part respectively.

[0021] Optionally, the method for deep fusion of binocular depth data and TOF depth data is characterized in that step S3 includes:

[0022] Step S31: Align the binocular depth data with the TOF depth data;

[0023] Step S32: In the TOF3D point cloud, pixels whose distance from the plane is not greater than the first threshold are marked as the near-ground portion, and pixels whose distance from the plane is greater than the first threshold are marked as the above-ground portion;

[0024] Step S33: Calculate the normal vector of each pixel in the area near the ground, and calculate the angle α between the normal vector and the ground normal vector;

[0025] Step S34: When the included angle α is greater than the second threshold, mark the pixel as the first pixel;

[0026] Step S35: When the number of adjacent first pixels is greater than the third threshold, mark the object to which the first pixel belongs as an obstacle.

[0027] Optionally, the method for deep fusion of binocular depth data and TOF depth data is characterized in that step S4 includes:

[0028] S41: Align the binocular depth data and the TOF depth data;

[0029] S42: Using the binocular depth data, correct the corresponding TOF depth data of the obstacle;

[0030] S43: Correct the uncorrected TOF depth data based on the corrected TOF depth data.

[0031] Secondly, the present invention provides a device for deep fusion of binocular depth data and TOF depth data, characterized in that it comprises:

[0032] The acquisition module is used to acquire binocular depth data and TOF depth data of the same scene, and generate binocular 3D point cloud and TOF 3D point cloud respectively;

[0033] A binocular module is used to identify planes in the binocular 3D point cloud and perform plane fitting;

[0034] The TOF module is used to identify the near-ground portion and the portion above the ground in the TOF3D point cloud, and to segment obstacles based on the normal vector information of the surface of the near-ground portion; wherein, the near-ground portion refers to the point cloud portion within a first threshold distance from the ground.

[0035] A correction module is used to correct the depth value of the obstacle in the TOF depth data using the binocular depth data;

[0036] The fusion module is used to fuse the binocular depth data with the corrected TOF depth data to obtain high-precision depth data.

[0037] Optionally, the device for deep fusion of binocular depth data and TOF depth data is characterized in that the binocular module includes:

[0038] The center point unit is used to identify the ground in the binocular 3D point cloud and to identify the dividing lines between adjacent ground surfaces;

[0039] A segmentation unit is used to divide the ground into a first part and a second part according to the segmentation line;

[0040] The fitting unit is used to perform planar fitting on the first part and the second part respectively.

[0041] Optionally, the device for deep fusion of binocular depth data and TOF depth data is characterized in that the binocular module includes:

[0042] Inheritance unit, used to obtain the position of the segmentation line in the previous image;

[0043] The fine-tuning unit is used to expand outward on the current image according to the position of the segmentation line to obtain an extended region, and to determine the position of the current image segmentation line within the extended region by the direction of the normal vector.

[0044] A segmentation unit is used to divide the ground into a first part and a second part according to the segmentation line;

[0045] The fitting unit is used to perform planar fitting on the first part and the second part respectively.

[0046] Optionally, the device for deep fusion of binocular depth data and TOF depth data is characterized in that the TOF module includes:

[0047] An alignment unit is used to align the binocular depth data with the TOF depth data;

[0048] The first marking unit is used to mark pixels in the TOF3D point cloud whose distance from the plane is not greater than the first threshold as the part near the ground, and to mark pixels whose distance from the plane is greater than the first threshold as the part above the ground.

[0049] A normal vector unit is used to calculate the normal vector of each pixel in the area near the ground, and to calculate the angle α between the normal vector and the ground normal vector.

[0050] The second marking unit is used to mark the pixel as the first pixel when the included angle α is greater than the second threshold.

[0051] The third marking unit is used to mark the object to which the first pixel belongs as an obstacle when the number of adjacent first pixels is greater than a third threshold.

[0052] Optionally, the device for deep fusion of binocular depth data and TOF depth data is characterized in that the correction module includes:

[0053] An alignment unit is used to align the stereo depth data and the TOF depth data.

[0054] The first correction unit is used to correct the TOF depth data corresponding to the obstacle using the binocular depth data.

[0055] The second correction unit is used to correct the uncorrected TOF depth data based on the corrected TOF depth data.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] This invention employs both binocular and Time-of-Flight (TOF) technologies, leveraging the unique characteristics of each to compensate for the weaknesses of the other. This results in superior depth data quality compared to using a single technology, leading to more accurate and comprehensive 3D reconstruction. Consequently, applications such as obstacle avoidance and navigation become more precise.

[0058] This invention divides 3D point clouds into a portion near the ground and a portion above the ground, which can better identify obstacles, especially small obstacles with low height. This makes the judgment of targets more accurate, thus enabling devices such as robotic vacuum cleaners to make more accurate judgments of target objects.

[0059] This invention combines binocular and Time-of-Flight (TOF) technologies to effectively address issues such as multipath effects, noise, and discontinuities in 3D reconstruction, thereby improving data continuity and consistency. This makes the data more consistent with the real scene and can be used for more precise detection and inspection. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort. Other features, objects, and advantages of the present invention will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0061] Figure 1 This is a schematic diagram comparing binocular depth data and TOF depth data in an embodiment of the present invention;

[0062] Figure 2 This is a flowchart illustrating the steps of a method for deep fusion of binocular depth data and TOF depth data in an embodiment of the present invention.

[0063] Figure 3 This is a flowchart illustrating the steps of plane fitting in a binocular 3D point cloud according to an embodiment of the present invention;

[0064] Figure 4 This is a flowchart illustrating another step in planar fitting in a binocular 3D point cloud according to an embodiment of the present invention;

[0065] Figure 5 This is a flowchart illustrating the steps of TOF3D point cloud segmentation of obstacles in an embodiment of the present invention.

[0066] Figure 6 This is a flowchart illustrating a step in correcting TOF data using binocular data, as described in an embodiment of the present invention.

[0067] Figure 7 This is a schematic diagram of the structure of a device for deep fusion of binocular depth data and TOF depth data according to an embodiment of the present invention;

[0068] Figure 8 This is a schematic diagram of a binocular module structure in an embodiment of the present invention;

[0069] Figure 9 This is a schematic diagram of another binocular module structure in an embodiment of the present invention;

[0070] Figure 10 This is a schematic diagram of a TOF module structure in an embodiment of the present invention;

[0071] Figure 11 This is a schematic diagram of a correction module structure in an embodiment of the present invention. Detailed Implementation

[0072] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0073] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0074] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0075] This invention provides a method for deep fusion of binocular depth data and TOF depth data, aiming to solve the problems existing in the prior art.

[0076] The technical solutions of the present invention and how they solve the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0077] Figure 1 This is a schematic diagram comparing binocular depth data and Time-of-Flight (TOF) depth data in an embodiment of the present invention. This embodiment uses a comparison of binocular depth data and TOF depth data from a robotic vacuum cleaner for illustration. Binocular technology is more suitable for applications in small-sized devices such as robotic vacuum cleaners, but it is prone to reconstruction failure in areas with weak textures. TOF technology, due to its technical principle, is not affected by the surface texture of objects, and therefore can obtain complete depth data of the target area during 3D reconstruction. However, it is susceptible to multipath interference and other factors, leading to some data distortion in areas with strong textures, such as corners.

[0078] like Figure 1 As shown, in stereo depth data, the closer the location is to the camera, the higher the received data density, while the farther the location is from the camera, the lower the received data density. This is represented in the figure as more point cloud data at close range and lower point cloud density at distant range. In TOF depth data, the data is relatively more uniform, but there is more noise. Furthermore, compared to stereo depth data, the ground depth data shows an "anomaly," meaning that some target objects are located below the ground. This is mainly due to multipath interference, and this part of the data can be better corrected by stereo data.

[0079] This invention combines the advantages of both binocular and Time-of-Flight (ToF) technologies and corrects the ToF measurement results, making the final depth data more accurate and reliable.

[0080] Figure 2 This is a flowchart illustrating the steps of a method for deep fusion of binocular depth data and TOF depth data according to an embodiment of the present invention. Figure 2 As shown in the figure, an embodiment of the present invention provides a method for deep fusion of binocular depth data and TOF depth data, comprising the following steps:

[0081] Step S1: Obtain binocular depth data and TOF depth data of the same scene, and generate binocular 3D point cloud and TOF 3D point cloud respectively.

[0082] In this step, the binocular depth data and TOF depth data of the same scene are typically captured from a viewing angle of no more than 10 degrees to facilitate subsequent alignment and other operations. Of course, alignment can be performed in this step or any subsequent step; this embodiment does not impose any restrictions. A binocular 3D point cloud is obtained through binocular depth data reconstruction, and a TOF 3D point cloud is obtained through TOF depth data reconstruction. The 3D point cloud is a reproduction of three-dimensional space, which depends on the accuracy of the data. The more accurate the depth data, the more accurate the reconstructed 3D space. When calibrating depth data, the device is usually calibrated, but this cannot overcome the biases introduced by the technology. This embodiment can utilize the data in the 3D point cloud for cross-validation to obtain high-precision depth data. In this embodiment, TOF can be either iToF or dToF.

[0083] Step S2: Identify the plane in the binocular 3D point cloud and perform plane fitting.

[0084] In this step, the ground, being fixed in the 3D point cloud, serves as a crucial reference for correction. This embodiment divides the data in 3D space into ground data and obstacle data. Ground data, representing a continuous plane, allows for ground fitting. The plane in this step refers to the plane containing the ground, not the plane of the camera's viewpoint. This is because the camera's viewpoint typically has an angular deviation from the ground. Furthermore, using the ground as the plane overcomes data inconsistencies caused by various disturbances during the shooting process, resulting in better consistency in the reconstructed 3D ground data.

[0085] This step can correct the data by fitting the plane of the ground, especially in scenarios where ground data accounts for a large proportion, such as robotic vacuum cleaners and other low-speed moving robots.

[0086] Step S3: Identify the part near the ground and the part above the ground in the TOF3D point cloud, and segment the obstacle based on the normal vector information of the surface of the part near the ground.

[0087] In this step, unlike binocular 3D point clouds where 3D data is divided into ground and obstacles, TOF 3D point clouds are divided into a portion near the ground and a portion above the ground. The portion near the ground refers to the point cloud portion within a first threshold distance from the ground. Due to the characteristics of TOF technology, the accuracy of the ground data it acquires is not high; therefore, the portion near the ground needs to be corrected using binocular data. The first threshold is usually a fixed value, but it can be adjusted according to different TOF technologies and application scenarios.

[0088] The ground's normal vector points vertically upwards, while the angle between the obstacle's normal vector and the ground's normal vector is typically large, sometimes even 90 degrees or other angles. Therefore, obstacles can be identified using the normal vector information of a portion of the surface near the ground. When the angle between the normal vector and the ground is less than a certain angle, it is determined to be an obstacle.

[0089] Step S4: Use the binocular depth data to correct the depth value of the obstacle in the TOF depth data.

[0090] In this step, because binocular depth data is more accurate than Time-of-Flight (TOF) depth data above the ground, it can be used to correct the depth data of obstacles in the TOF dataset. In this step, the binocular depth data and TOF depth data are already aligned. Due to the multipath effect of TOF, it is prone to distortion at intersections such as corners, and this distortion has a significant impact on small obstacles; therefore, it must be corrected using binocular depth data. The depth values ​​corrected in this step include both small and large obstacles.

[0091] Step S5: Fuse the binocular depth data with the corrected TOF depth data to obtain high-precision depth data.

[0092] In this step, the calibrated TOF depth data has high accuracy and is continuous, while the binocular depth data has even better accuracy. This step fuses the binocular depth data with the calibrated TOF depth data to obtain high-precision depth data that combines the advantages of both binocular and TOF technologies. When fusing the binocular depth data with the calibrated TOF depth data, an appropriate fusion method can be selected based on the specific technical characteristics and application scenario.

[0093] For example, ground data from the binocular depth data is combined with obstacle data from the corrected Time-of-Flight (TOF) depth data to generate high-precision depth data. Binocular depth data is used for the ground, while corrected TOF depth data is used for obstacles, resulting in high-precision depth data. Due to multipath interference in the TOF data, its ground data differs from the ground data in the binocular depth data, thus causing errors in the data of obstacles adjacent to the ground. During correction and fusion, abnormal data such as unidentifiable points and flying spots in the TOF depth data need to be processed. Flying spots can be considered noise and directly deleted. For the portion of the obstacle in the TOF depth data that is lower than the ground in the binocular 3D point cloud, this portion is also deleted, and the corresponding portion is re-corrected.

[0094] For example, ground data from the binocular depth data and ground data from the TOF depth data are weighted differently according to distance to generate final ground data. This final ground data is then combined with obstacle data from the corrected TOF depth data to generate high-precision depth data. Weight values ​​range from [0,1], with both 0 and 1 being valid values. Since the application ranges of binocular and TOF technologies do not completely overlap, different values ​​can be used at different distances. The weight values ​​do not change linearly with distance.

[0095] For example, based on the distance to the obstacle, different weight values ​​are assigned to the binocular depth data and the TOF depth data to generate final ground data. This final ground data is then combined with the obstacle data in the corrected TOF depth data to generate high-precision depth data. Higher weight values ​​are assigned to the ground data in the binocular depth data near the obstacle, while higher weight values ​​are assigned to the TOF depth data at greater distances from the obstacle. Weight values ​​range from [0,1], and both 0 and 1 are valid values. The weight values ​​do not change linearly with distance.

[0096] This embodiment reconstructs depth data in three dimensions and calibrates it in three-dimensional space, which allows for the verification and correlation of data with strong interrelationships. By combining the characteristics of binocular and Time-of-Flight (TOF) technologies, it leverages the strengths of both to obtain more accurate point cloud and depth data, overcoming the shortcomings of single technologies and resulting in better data correction and high-precision depth data.

[0097] Figure 3 This is a flowchart illustrating the steps of planar fitting in a binocular 3D point cloud according to an embodiment of the present invention, adapted to scenarios with uneven ground. Figure 3 As shown in the figure, an embodiment of the present invention provides a method for plane fitting in a binocular 3D point cloud, comprising the following steps:

[0098] S21: Identify the ground in the binocular 3D point cloud and identify the dividing lines between adjacent ground surfaces.

[0099] In this step, an XYZ coordinate system is established on the binocular 3D point cloud, and the ground is identified based on the angle between the normal vector and the Z-axis. The ground is usually a continuous region, so the average of the pixel normal vectors within that region can be taken as the normal vector for that region. The selected region should not exceed 1 / 10 of the total ground area. Different normal vectors in adjacent regions indicate different ground directions. The points where the normal vector direction changes are the dividing lines; that is, the dividing lines between adjacent ground surfaces can be identified by the change in the angle of the normal vector.

[0100] S22: Divide the ground into a first part and a second part according to the dividing line.

[0101] In this step, the first part is the area closer to the shooting position. The second part is the area farther from the shooting position. For example, when using a depth camera, the first part is the area closest to the camera, and the farther part is the second part. The second part can contain one plane, or it can contain two or more planes. For example, when the ground has three different slopes, the plane where the shooting position is located is the first part, and the other two planes are the second part.

[0102] S23: Perform planar fitting on the first part and the second part respectively.

[0103] In this step, a plane fit is performed on the first part. If the second part has only one plane, a plane fit is performed on it. If the second part has two or more planes, a fit is performed on each plane.

[0104] This embodiment divides the ground into different parts and fits them separately, thereby improving the planar fitting effect of the ground and making subsequent navigation, path planning, obstacle recognition and other actions more accurate, so that the invention can be applied to more application scenarios.

[0105] Figure 4 This is a flowchart illustrating another step in planar fitting of binocular 3D point clouds in an embodiment of the present invention, applicable to continuous processing of images of the same scene. For example... Figure 4 As shown, another method for plane fitting in binocular 3D point clouds in this embodiment of the invention includes:

[0106] S24: Get the position of the dividing line in the previous image.

[0107] In this step, the positions of all dividing lines in the previous image are obtained. If there is only one dividing line in the previous image, then one dividing line is obtained; if there are two dividing lines in the previous image, then both dividing lines are obtained.

[0108] S25: On the current image, expand outwards according to the position of the segmentation line to obtain an extended region, and determine the position of the current image segmentation line within the extended region by the direction of the normal vector.

[0109] In this step, the extent of the expanded area is determined by the speed of the viewpoint change. For example, when shooting from a robot vacuum cleaner, different ranges can be selected based on the robot vacuum cleaner's movement speed; a larger speed results in a larger range, and a smaller speed results in a smaller range. The position of the dividing line can be roughly located in the current image using the position of the dividing line. Then, the same method as identifying the dividing line in the previous embodiment is used, i.e., the position of the dividing line in the current image is determined by the direction of the normal vector.

[0110] S22: Divide the ground into a first part and a second part according to the dividing line.

[0111] S23: Perform planar fitting on the first part and the second part respectively.

[0112] This embodiment utilizes the recognition results of the previous image to significantly reduce the amount of data processing while still accurately obtaining the segmentation line information. This enables real-time information acquisition during continuous shooting, improving data processing capabilities and timely response capabilities.

[0113] Figure 5 This is a flowchart illustrating the steps of obstacle segmentation using TOF3D point cloud technology in an embodiment of the present invention. Figure 5 As shown, a method for segmenting obstacles using TOF3D point clouds in an embodiment of the present invention includes the following steps:

[0114] Step S31: Align the binocular depth data with the TOF depth data.

[0115] In this step, aligning the binocular depth data with the TOF depth data is fundamental for subsequent operations. If alignment has already been performed in the preceding steps, this step can be skipped.

[0116] Step S32: In the TOF3D point cloud, pixels whose distance from the plane is not greater than the first threshold are marked as the near-ground portion, and pixels whose distance from the plane is greater than the first threshold are marked as the above-ground portion.

[0117] In this step, the data in the TOF3D point cloud are labeled according to their distance from the plane. The data near the ground needs to be corrected based on the binocular depth data and obstacles need to be identified, while the data above the ground does not need to be corrected.

[0118] Step S33: Calculate the normal vector of each pixel in the area near the ground, and calculate the angle α between the normal vector and the ground normal vector.

[0119] In this step, the normal vector is calculated on a pixel-by-pixel basis. The ground in this step refers to the ground region closest to the stated pixel; therefore, the average normal vector of the pixels in this ground region can be taken as the ground normal vector. It should be noted that the region is not the set of all ground pixels, but rather a portion of all ground pixels.

[0120] Step S34: When the included angle α is greater than the second threshold, mark the pixel as the first pixel.

[0121] In this step, when the included angle 'a' is greater than the second threshold, it indicates a large angle with the ground, suggesting it may be part of an obstacle. The value of the second threshold can be selected based on different application scenarios. For example, on a smooth, tiled surface, the included angle 'a' can be smaller to improve the ability to identify small obstacles. On a rough, worn surface, the included angle 'a' can be larger to improve the stability of the identification. Typically, the included angle 'a' can be 15, 20, 25, 30, 35, 40 degrees, or other angles.

[0122] Step S35: When the number of adjacent first pixels is greater than the third threshold, mark the object to which the first pixel belongs as an obstacle.

[0123] In this step, a single pixel may be a flying point. An insufficient number of adjacent pixels may be due to uneven ground surfaces; therefore, a third threshold is needed to filter out these two situations. When identifying the first pixel, it is also necessary to simultaneously identify objects to obtain parts that may belong to the same object, for example, by judging based on abrupt changes in the pixel normal vector.

[0124] This embodiment identifies obstacles based on normal vector information, enabling the identification of small obstacles while filtering out the influence of flying points and surface defects, achieving a balance between recognition accuracy and stability, and can achieve good recognition results in various application scenarios.

[0125] Figure 6 This is a flowchart illustrating the steps of correcting TOF data using binocular data in an embodiment of the present invention. Figure 6 As shown, a method for correcting TOF data using binocular data in an embodiment of the present invention includes the following steps:

[0126] S41: Align the binocular depth data and the TOF depth data.

[0127] Aligning the stereo depth data with the TOF depth data in this step is fundamental for subsequent steps. If the stereo depth data and TOF depth data have already been aligned in the preceding steps, this step can be skipped.

[0128] S42: Using the binocular depth data, correct the corresponding TOF depth data of the obstacle.

[0129] In this step, because binocular depth data has higher accuracy, it is used to correct the corresponding TOF depth data to make the TOF data more accurate. Since binocular systems use techniques such as light spot detection, the accurate data they obtain is not continuous; therefore, only a portion of the TOF depth data can be corrected. After this step is completed, step S43 needs to be executed.

[0130] S43: Correct the uncorrected TOF depth data based on the corrected TOF depth data.

[0131] In this step, the corrected Time-of-Flight (TOF) depth data is used to correct adjacent, uncorrected TOF depth data within the 3D point cloud. Since the corrected TOF depth data are independent of each other and contain uncorrected data, data abrupt changes are possible. Algorithms such as linear regression and locally weighted regression can be used to recover the uncorrected TOF depth data.

[0132] This embodiment corrects the TOF depth data using binocular depth data, and then uses the corrected data to regress the uncorrected data in the TOF depth data, thereby making the TOF depth data more accurate and comprehensive, and thus obtaining high-precision depth data.

[0133] Figure 7 This is a schematic diagram of a device for deep fusion of binocular depth data and TOF depth data according to an embodiment of the present invention. Figure 7 As shown, an apparatus for deep fusion of binocular depth data and TOF depth data in an embodiment of the present invention includes:

[0134] The acquisition module 100 is used to acquire binocular depth data and TOF depth data of the same scene, and generate binocular 3D point cloud and TOF 3D point cloud respectively;

[0135] The binocular module 200 is used to identify planes in the binocular 3D point cloud and perform plane fitting.

[0136] The TOF module 300 is used to identify the portion near the ground and the portion above the ground in the TOF3D point cloud, and to segment obstacles based on the normal vector information of the surface of the portion near the ground; wherein, the portion near the ground refers to the point cloud portion within a first threshold distance from the ground;

[0137] The correction module 400 is used to correct the depth value of the obstacle in the TOF depth data using the binocular depth data;

[0138] The fusion module 500 is used to fuse the binocular depth data with the corrected TOF depth data to obtain high-precision depth data.

[0139] This embodiment can be installed either inside or outside the depth camera. By processing and fusing depth data, it obtains high-precision depth data, improving accuracy, especially in identifying small obstacles. The parameters of this embodiment can also be adjusted according to different scenarios to achieve optimal performance in various situations.

[0140] Figure 8 This is a schematic diagram of a binocular module structure according to an embodiment of the present invention. Figure 8 As shown, compared to the previous embodiments, the binocular module 200 in this embodiment includes:

[0141] The center point unit 210 is used to identify the ground in the binocular 3D point cloud and to identify the dividing lines of adjacent ground.

[0142] The segmentation unit 220 is used to divide the ground into a first part and a second part according to the segmentation line;

[0143] The fitting unit 230 is used to perform planar fitting on the first part and the second part respectively.

[0144] Figure 9 This is a schematic diagram of another binocular module structure in an embodiment of the present invention. Figure 9 As shown, compared to the previous embodiments, the binocular module 200 in this embodiment includes:

[0145] Inheritance unit 240 is used to obtain the position of the segmentation line in the previous image;

[0146] The fine-tuning unit 250 is used to expand an extended region outward based on the position of the segmentation line on the current image, and to determine the position of the current image segmentation line within the extended region by the direction of the normal vector.

[0147] The segmentation unit 220 is used to divide the ground into a first part and a second part according to the segmentation line;

[0148] The fitting unit 230 is used to perform planar fitting on the first part and the second part respectively.

[0149] Figure 10 This is a schematic diagram of a TOF module structure according to an embodiment of the present invention. Figure 10 As shown, compared to the previous embodiments, the TOF module 300 in this embodiment includes:

[0150] Alignment unit 310 is used to align the binocular depth data with the TOF depth data;

[0151] The first marking unit 320 is used to mark pixels in the TOF3D point cloud whose distance from the plane is not greater than the first threshold as the part near the ground, and to mark pixels whose distance from the plane is greater than the first threshold as the part above the ground.

[0152] Normal vector unit 330 is used to calculate the normal vector of each pixel in the area near the ground, and to calculate the angle α between the normal vector and the ground normal vector;

[0153] The second marking unit 340 is used to mark the pixel as the first pixel when the included angle α is greater than the second threshold.

[0154] The third marking unit 350 is used to mark the object to which the first pixel belongs as an obstacle when the number of adjacent first pixels is greater than a third threshold.

[0155] Figure 11 This is a schematic diagram of a correction module structure according to an embodiment of the present invention. Figure 11 As shown, compared to the previous embodiments, the correction module 400 in this embodiment includes:

[0156] Alignment unit 310 is used to align the stereo depth data and the TOF depth data;

[0157] The first correction unit 420 is used to correct the TOF depth data corresponding to the obstacle using the binocular depth data.

[0158] The second correction unit 430 is used to correct the uncorrected TOF depth data based on the corrected TOF depth data.

[0159] The various embodiments described in this specification are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0160] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for deep fusion of binocular depth data and TOF depth data, characterized in that, Includes the following steps: Step S1: Acquire binocular depth data and TOF depth data of the same scene, and generate binocular 3D point cloud and TOF 3D point cloud respectively; Step S2: Identify the plane in the binocular 3D point cloud and perform plane fitting; Step S3: Identify the portion near the ground and the portion above the ground in the TOF3D point cloud, and segment the obstacle based on the normal vector information of the surface of the portion near the ground; wherein, the portion near the ground refers to the point cloud portion within a first threshold distance from the ground; Step S4: Use the binocular depth data to correct the depth value of the obstacle in the TOF depth data; Step S5: Fuse the binocular depth data with the corrected TOF depth data to obtain high-precision depth data; Step S3 includes: Step S31: Align the binocular depth data with the TOF depth data; Step S32: In the TOF3D point cloud, pixels whose distance from the plane is not greater than the first threshold are marked as the near-ground portion, and pixels whose distance from the plane is greater than the first threshold are marked as the above-ground portion; Step S33: Calculate the normal vector of each pixel in the area near the ground, and calculate the angle α between the normal vector and the ground normal vector; Step S34: When the included angle α is greater than the second threshold, mark the pixel as the first pixel; Step S35: When the number of adjacent first pixels is greater than the third threshold, mark the object to which the first pixel belongs as an obstacle.

2. The method for deep fusion of binocular depth data and TOF depth data according to claim 1, characterized in that, Step S2 includes: S21: Identify the ground in the binocular 3D point cloud and identify the dividing lines between adjacent ground surfaces; S22: Divide the ground into a first part and a second part according to the dividing line; S23: Perform planar fitting on the first part and the second part respectively.

3. The method for deep fusion of binocular depth data and TOF depth data according to claim 1, characterized in that, Step S2 includes: S24: Obtain the position of the segmentation line in the previous image; S25: On the current image, expand outwards according to the position of the segmentation line to obtain an extended region, and determine the position of the current image segmentation line within the extended region by the direction of the normal vector. S22: Divide the ground into a first part and a second part according to the dividing line; S23: Perform planar fitting on the first part and the second part respectively.

4. The method for deep fusion of binocular depth data and TOF depth data according to claim 1, characterized in that, Step S4 includes: S41: Align the binocular depth data and the TOF depth data; S42: Using the binocular depth data, correct the corresponding TOF depth data of the obstacle; S43: Correct the uncorrected TOF depth data based on the corrected TOF depth data.

5. A device for deep fusion of binocular depth data and TOF depth data, characterized in that, include: The acquisition module is used to acquire binocular depth data and TOF depth data of the same scene, and generate binocular 3D point cloud and TOF 3D point cloud respectively; A binocular module is used to identify planes in the binocular 3D point cloud and perform plane fitting; The TOF module is used to identify the near-ground portion and the portion above the ground in the TOF3D point cloud, and to segment obstacles based on the normal vector information of the surface of the near-ground portion; wherein, the near-ground portion refers to the point cloud portion within a first threshold distance from the ground. A correction module is used to correct the depth value of the obstacle in the TOF depth data using the binocular depth data; The fusion module is used to fuse the binocular depth data with the corrected TOF depth data to obtain high-precision depth data; The TOF module includes: An alignment unit is used to align the binocular depth data with the TOF depth data; The first marking unit is used to mark pixels in the TOF3D point cloud whose distance from the plane is not greater than the first threshold as the part near the ground, and to mark pixels whose distance from the plane is greater than the first threshold as the part above the ground. A normal vector unit is used to calculate the normal vector of each pixel in the area near the ground, and to calculate the angle α between the normal vector and the ground normal vector. The second marking unit is used to mark the pixel as the first pixel when the included angle α is greater than the second threshold. The third marking unit is used to mark the object to which the first pixel belongs as an obstacle when the number of adjacent first pixels is greater than a third threshold.

6. The device for deep fusion of binocular depth data and TOF depth data according to claim 5, characterized in that, The binocular module includes: The center point unit is used to identify the ground in the binocular 3D point cloud and to identify the dividing lines between adjacent ground surfaces; A segmentation unit is used to divide the ground into a first part and a second part according to the segmentation line; The fitting unit is used to perform planar fitting on the first part and the second part respectively.

7. The device for deep fusion of binocular depth data and TOF depth data according to claim 5, characterized in that, The binocular module includes: Inheritance unit, used to obtain the position of the segmentation line in the previous image; The fine-tuning unit is used to expand outward on the current image according to the position of the segmentation line to obtain an extended region, and to determine the position of the current image segmentation line within the extended region by the direction of the normal vector. A segmentation unit is used to divide the ground into a first part and a second part according to the segmentation line; The fitting unit is used to perform planar fitting on the first part and the second part respectively.

8. The device for deep fusion of binocular depth data and TOF depth data according to claim 5, characterized in that, The correction module includes: An alignment unit is used to align the stereo depth data and the TOF depth data. The first correction unit is used to correct the TOF depth data corresponding to the obstacle using the binocular depth data. The second correction unit is used to correct the uncorrected TOF depth data based on the corrected TOF depth data.

Images