A method for fusing structured light data and TOF data

Abstract

A kind of method of structural light data and TOF data depth fusion, comprising the following steps: step S1: obtaining the structural light depth data and TOF depth data of the same scene, and generating structural light 3D point cloud and TOF 3D point cloud respectively;Step S2: plane is identified in the structural light 3D point cloud, and plane fitting is carried out;Step S3: part near ground and part above ground near ground are identified in the TOF 3D point cloud, and according to the normal vector information of the surface of part near ground, the obstacle is segmented;Wherein, part near ground refers to the point cloud part within first threshold distance from ground;Step S4: the depth value of the obstacle in the TOF depth data is corrected using the structural light depth data;Step S5: the structural light depth data and the TOF depth data after correction are fused, and high-precision depth data is obtained.The structural light technology and TOF technology are combined in the present application, so that three-dimensional reconstruction effect is better.

Description

Technical Field

[0001] This invention relates to the field of image data processing, and more specifically, to a method for deep fusion of structured light data and TOF data. Background Technology

[0002] With the development of the "lazy economy," the market size of household robotic vacuum cleaners is growing, and their intelligence level is constantly improving, with functions such as automatic navigation, automatic obstacle avoidance, and automatic cleaning becoming increasingly common. Among them, the automatic obstacle avoidance function mainly relies on 3D vision sensors, such as line structured light sensors, binocular stereo cameras, ToF cameras, and monocular speckle structured light depth sensors.

[0003] These sensors each have their own advantages and disadvantages. Line structured light can completely reconstruct the scene depth at the laser bright stripe, but its depth field of view is small. Binocular stereo cameras are prone to matching failures in weak texture areas, leading to depth reconstruction failures, and the computational load is large. ToF cameras have multipath effects, which can easily cause depth distortion in scenes such as corners, affecting the ranging results, and there are many flying points, which interfere with obstacle segmentation. Monocular speckle structured light is prone to depth loss at small obstacles.

[0004] To ensure that the obstacle avoidance performance of robotic vacuum cleaners can meet the needs of various common household scenarios and improve the user experience, it is necessary to address the various drawbacks of the single vision sensor. To this end, we have invented a deep fusion algorithm that can effectively fuse monocular speckle structured light depth data with ToF depth data, leveraging the strengths of each while avoiding their respective weaknesses. Summary of the Invention

[0005] Therefore, this invention combines structured light technology with Time-of-Flight (TOF) technology to acquire two types of depth data for the same scene. By processing the point cloud data, high-precision, complete, and accurate depth data is obtained, resulting in better 3D reconstruction.

[0006] In a first aspect, the present invention provides a method for deep fusion of structured light data and TOF data, characterized by comprising the following steps:

[0007] Step S1: Acquire structured light depth data and TOF depth data of the same scene, and generate structured light 3D point cloud and TOF 3D point cloud respectively;

[0008] Step S2: Identify planes in the structured light 3D point cloud and perform plane fitting;

[0009] Step S3: Identify the portion near the ground and the portion above the ground in the TOF3D point cloud, and segment the obstacle based on the normal vector information of the surface of the portion near the ground; wherein, the portion near the ground refers to the point cloud portion within a first threshold distance from the ground;

[0010] Step S4: Use the structured light depth data to correct the depth value of the obstacle in the TOF depth data;

[0011] Step S5: Fuse the structured light depth data with the corrected TOF depth data to obtain high-precision depth data.

[0012] Optionally, the method for deep fusion of structured light data and TOF data is characterized in that step S2 includes:

[0013] S21: Identify the ground in the structured light 3D point cloud and identify the dividing lines between adjacent ground surfaces;

[0014] S22: Divide the ground into a first part and a second part according to the dividing line;

[0015] S23: Perform planar fitting on the first part and the second part respectively.

[0016] Optionally, the method for deep fusion of structured light data and TOF data is characterized in that step S2 includes:

[0017] S24: Obtain the position of the segmentation line in the previous image;

[0018] S25: On the current image, expand outwards according to the position of the segmentation line to obtain an extended region, and determine the position of the current image segmentation line within the extended region by the direction of the normal vector.

[0019] S22: Divide the ground into a first part and a second part according to the dividing line;

[0020] S23: Perform planar fitting on the first part and the second part respectively.

[0021] Optionally, the method for deep fusion of structured light data and TOF data is characterized in that step S3 includes:

[0022] Step S31: Align the structured light depth data with the TOF depth data;

[0023] Step S32: In the TOF3D point cloud, pixels whose distance from the plane is not greater than the first threshold are marked as the near-ground portion, and pixels whose distance from the plane is greater than the first threshold are marked as the above-ground portion;

[0024] Step S33: Calculate the normal vector of each pixel in the area near the ground, and calculate the angle α between the normal vector and the ground normal vector;

[0025] Step S34: When the included angle α is greater than the second threshold, mark the pixel as the first pixel;

[0026] Step S35: When the number of adjacent first pixels is greater than the third threshold, mark the object to which the first pixel belongs as an obstacle.

[0027] Optionally, the method for deep fusion of structured light data and TOF data is characterized in that step S4 includes:

[0028] S41: Align the structured light depth data and the TOF depth data;

[0029] S42: Using the structured light depth data, the corresponding TOF depth data of the obstacle is corrected;

[0030] S43: Correct the uncorrected TOF depth data based on the corrected TOF depth data.

[0031] Secondly, the present invention provides a device for deep fusion of structured light data and TOF data, characterized in that it comprises:

[0032] The acquisition module is used to acquire structured light depth data and TOF depth data of the same scene, and generate structured light 3D point cloud and TOF 3D point cloud respectively;

[0033] A structured light module is used to identify planes in the structured light 3D point cloud and perform plane fitting.

[0034] The TOF module is used to identify the near-ground portion and the portion above the ground in the TOF3D point cloud, and to segment obstacles based on the normal vector information of the surface of the near-ground portion; wherein, the near-ground portion refers to the point cloud portion within a first threshold distance from the ground.

[0035] A correction module is used to correct the depth value of the obstacle in the TOF depth data using the structured light depth data;

[0036] The fusion module is used to fuse the structured light depth data with the corrected TOF depth data to obtain high-precision depth data.

[0037] Optionally, the device for deep fusion of structured light data and TOF data is characterized in that the structured light module includes:

[0038] The center point unit is used to identify the ground in the structured light 3D point cloud and to identify the dividing lines between adjacent ground surfaces;

[0039] A segmentation unit is used to divide the ground into a first part and a second part according to the segmentation line;

[0040] The fitting unit is used to perform planar fitting on the first part and the second part respectively.

[0041] Optionally, the device for deep fusion of structured light data and TOF data is characterized in that the structured light module includes:

[0042] Inheritance unit, used to obtain the position of the segmentation line in the previous image;

[0043] The fine-tuning unit is used to expand outward on the current image according to the position of the segmentation line to obtain an extended region, and to determine the position of the current image segmentation line within the extended region by the direction of the normal vector.

[0044] A segmentation unit is used to divide the ground into a first part and a second part according to the segmentation line;

[0045] The fitting unit is used to perform planar fitting on the first part and the second part respectively.

[0046] Optionally, the device for deep fusion of structured light data and TOF data is characterized in that the TOF module includes:

[0047] An alignment unit is used to align the structured light depth data with the TOF depth data;

[0048] The first marking unit is used to mark pixels in the TOF3D point cloud whose distance from the plane is not greater than the first threshold as the part near the ground, and to mark pixels whose distance from the plane is greater than the first threshold as the part above the ground.

[0049] A normal vector unit is used to calculate the normal vector of each pixel in the area near the ground, and to calculate the angle α between the normal vector and the ground normal vector.

[0050] The second marking unit is used to mark the pixel as the first pixel when the included angle α is greater than the second threshold.

[0051] The third marking unit is used to mark the object to which the first pixel belongs as an obstacle when the number of adjacent first pixels is greater than a third threshold.

[0052] Optionally, the device for deep fusion of structured light data and TOF data is characterized in that the correction module includes:

[0053] An alignment unit is used to align the structured light depth data and the TOF depth data.

[0054] The first correction unit is used to correct the TOF depth data corresponding to the obstacle using the structured light depth data.

[0055] The second correction unit is used to correct the uncorrected TOF depth data based on the corrected TOF depth data.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] This invention employs both structured light and Time-of-Flight (TOF) technologies, leveraging the unique characteristics of each to complement each other. This results in superior depth data quality compared to using a single technology, leading to more accurate and comprehensive 3D reconstruction. Consequently, applications such as obstacle avoidance and navigation become more precise.

[0058] This invention divides 3D point clouds into a portion near the ground and a portion above the ground, which can better identify obstacles, especially small obstacles with low height. This makes the judgment of targets more accurate, thus enabling devices such as robotic vacuum cleaners to make more accurate judgments of target objects.

[0059] This invention combines structured light technology with Time-of-Flight (TOF) technology to effectively solve problems such as noise and discontinuity in 3D reconstruction, improving the continuity and consistency of data. This makes the data more consistent with the real scene and can be used for more accurate detection and inspection. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort. Other features, objects, and advantages of the present invention will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0061] Figure 1 This is a schematic diagram comparing structured light depth data and TOF depth data in an embodiment of the present invention;

[0062] Figure 2 This is a flowchart illustrating the steps of a method for deep fusion of structured light data and TOF data in an embodiment of the present invention.

[0063] Figure 3 This is a flowchart illustrating the steps of plane fitting in a structured light 3D point cloud according to an embodiment of the present invention.

[0064] Figure 4 This is a flowchart illustrating another step in planar fitting in a structured light 3D point cloud according to an embodiment of the present invention.

[0065] Figure 5 This is a flowchart illustrating the steps of TOF3D point cloud segmentation of obstacles in an embodiment of the present invention.

[0066] Figure 6 This is a flowchart illustrating a step in correcting TOF data using structured light data, as described in an embodiment of the present invention.

[0067] Figure 7 This is a schematic diagram of a device for deep fusion of structured light data and TOF data according to an embodiment of the present invention;

[0068] Figure 8 This is a schematic diagram of a structured optical module in an embodiment of the present invention;

[0069] Figure 9 This is a schematic diagram of another structured optical module in an embodiment of the present invention;

[0070] Figure 10 This is a schematic diagram of a TOF module structure in an embodiment of the present invention;

[0071] Figure 11 This is a schematic diagram of a correction module structure in an embodiment of the present invention. Detailed Implementation

[0072] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0073] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0074] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0075] This invention provides a method for deep fusion of structured light data and TOF data, which aims to solve the problems existing in the prior art.

[0076] The technical solutions of the present invention and how they solve the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0077] Figure 1 This is a schematic diagram comparing structured light depth data and Time-of-Flight (TOF) depth data in an embodiment of the present invention. This embodiment uses a comparison of monocular speckle structured light depth data and TOF depth data from a robotic vacuum cleaner. Monocular speckle structured light is more suitable for applications in small-sized devices such as robotic vacuum cleaners, but its data is prone to loss at small obstacles. TOF technology, when using floodlight illumination, obtains comprehensive data, thus providing complete depth data for the target area during 3D reconstruction. However, it is susceptible to multipath interference, leading to some data distortion.

[0078] like Figure 1 As shown, in structured light depth data, the data density is higher at locations closer to the camera and lower at locations farther away. This is represented in the figure as more point cloud data at closer distances and lower point cloud density at farther distances. In TOF depth data, the data is relatively more uniform, but there is more noise. Furthermore, compared to structured light depth data, the ground depth data exhibits an "anomaly," meaning some target objects are located below the ground surface. This is mainly due to multipath interference.

[0079] This invention combines the advantages of structured light technology and Time-of-Flight (ToF) technology, and corrects the measurement results of ToF, making the final depth data more accurate and reliable.

[0080] Figure 2 This is a flowchart illustrating the steps of a method for deep fusion of structured light data and TOF data according to an embodiment of the present invention. Figure 2 As shown in the figure, an embodiment of the present invention provides a method for deep fusion of structured light data and TOF data, which includes the following steps:

[0081] Step S1: Obtain structured light depth data and TOF depth data for the same scene, and generate structured light 3D point cloud and TOF 3D point cloud respectively.

[0082] In this step, the shooting angles of structured light depth data and TOF depth data for the same scene are typically no more than 10 degrees to facilitate subsequent alignment and other operations. Of course, alignment can be performed in this step or any subsequent step; this embodiment does not impose any restrictions. A structured light 3D point cloud is reconstructed from the structured light depth data, and a TOF 3D point cloud is reconstructed from the TOF depth data. The 3D point cloud is a reproduction of three-dimensional space, which depends on the accuracy of the data. The more accurate the depth data, the more accurate the reconstructed 3D space. When calibrating depth data, the equipment is usually calibrated, but this cannot overcome the biases introduced by the technology. This embodiment can utilize the data in the 3D point cloud for cross-validation to obtain high-precision depth data. In this embodiment, TOF can be either iToF or dToF.

[0083] Step S2: Identify the plane in the structured light 3D point cloud and perform plane fitting.

[0084] In this step, monocular speckle structured light offers high accuracy, but it is prone to depth loss at small obstacles, making it the most suitable structured light technology for this embodiment. However, this embodiment can also be applied to other structured light technologies. In 3D point clouds, the ground is fixed and can serve as an important reference for correction. In this embodiment, the data is divided into ground data and obstacle data in three-dimensional space. Ground data is the acquired ground data, which is represented as a continuous plane, thus allowing for ground fitting. The plane in this step refers to the plane containing the ground, not the plane of the camera's viewpoint. This is because the camera's viewpoint usually has a certain angular deviation from the ground. Furthermore, using the ground as the plane overcomes the data inconsistency caused by various disturbances during the shooting process, resulting in better consistency of the reconstructed 3D ground data.

[0085] This step can correct the data by fitting the plane of the ground, especially in scenarios where ground data accounts for a large proportion, such as robotic vacuum cleaners and other low-speed moving robots.

[0086] Step S3: Identify the part near the ground and the part above the ground in the TOF3D point cloud, and segment the obstacle based on the normal vector information of the surface of the part near the ground.

[0087] In this step, unlike structured light 3D point clouds where the 3D data is divided into ground and obstacles, TOF 3D point clouds are divided into a portion near the ground and a portion above the ground. The portion near the ground refers to the point cloud portion within a first threshold distance from the ground. Due to the characteristics of TOF technology, the accuracy of the ground data it acquires is not high; therefore, the portion near the ground needs to be corrected using structured light data. The first threshold is usually a fixed value, but it can be adjusted according to different TOF technologies and application scenarios.

[0088] The ground's normal vector points vertically upwards, while the angle between the obstacle's normal vector and the ground's normal vector is typically large, sometimes even 90 degrees or other angles. Therefore, obstacles can be identified using the normal vector information of a portion of the surface near the ground. When the angle between the normal vector and the ground is less than a certain angle, it is determined to be an obstacle.

[0089] Step S4: Use the structured light depth data to correct the depth value of the obstacle in the TOF depth data.

[0090] In this step, because structured light depth data is more accurate than Time-of-Flight (TOF) depth data above the ground, it can be used to correct the depth data of obstacles in the TOF dataset. In this step, the structured light and TOF depth data are already aligned. Due to the multipath effect of TOF, it is prone to distortion at intersections such as corners, and this distortion has a significant impact on small obstacles; therefore, it must be corrected using structured light depth data. The depth values ​​corrected in this step include both small and large obstacles.

[0091] Step S5: Fuse the structured light depth data with the corrected TOF depth data to obtain high-precision depth data.

[0092] In this step, the corrected TOF depth data has high accuracy and is continuous, while the structured light depth data has even better accuracy. This step fuses the structured light depth data with the corrected TOF depth data to obtain high-precision depth data that combines the advantages of both structured light and TOF technologies. When fusing the structured light depth data with the corrected TOF depth data, an appropriate fusion method can be selected based on the specific technical characteristics and application scenario.

[0093] For example, ground data from the structured light depth data is combined with obstacle data from the corrected Time-of-Flight (TOF) depth data to generate high-precision depth data. The structured light depth data is used for the ground, while the corrected TOF depth data is used for obstacles, resulting in high-precision depth data. Due to multipath interference in the TOF data, the ground data differs from the ground data in the structured light depth data, thus causing errors in the data of obstacles adjacent to the ground. During correction and fusion, abnormal data such as unidentifiable points and flying points in the TOF depth data need to be processed. Flying points can be considered noise and directly deleted. For the portion of the obstacle in the TOF depth data that is lower than the ground in the structured light 3D point cloud, this portion is also deleted, and the corresponding portion is re-corrected.

[0094] For example, ground data from the structured light depth data and ground data from the Time-of-Flight (TOF) depth data are weighted differently according to distance to generate final ground data. This final ground data is then combined with obstacle data from the corrected TOF depth data to generate high-precision depth data. Weight values ​​range from [0,1], and both 0 and 1 are valid values. Since the application ranges of structured light and TOF technologies do not completely overlap, different values ​​can be used at different distances. The weight values ​​do not change linearly with distance.

[0095] For example, based on the distance to the obstacle, different weight values ​​are assigned to the structured light depth data and the Time-of-Flight (TOF) depth data to generate final ground data. This final ground data is then combined with the obstacle data in the corrected TOF depth data to generate high-precision depth data. Higher weight values ​​are assigned to the ground data in the structured light depth data near the obstacle, while higher weight values ​​are assigned to the TOF depth data at greater distances from the obstacle. Weight values ​​range from [0,1], and both 0 and 1 are valid values. The weight values ​​do not change linearly with distance.

[0096] This embodiment reconstructs depth data in three dimensions and calibrates it in three-dimensional space, which allows for the verification and correlation of data with strong interrelationships. By combining the characteristics of structured light technology and TOF technology, the advantages of each are combined to obtain more accurate point cloud data and depth data, overcoming the shortcomings of single technologies, resulting in better data correction and high-precision depth data.

[0097] Figure 3 This is a flowchart illustrating the steps of planar fitting in a structured light 3D point cloud according to an embodiment of the present invention, suitable for scenarios with uneven ground. Figure 3 As shown in the figure, an embodiment of the present invention provides a method for plane fitting in structured light 3D point clouds, comprising the following steps:

[0098] S21: Identify the ground in the structured light 3D point cloud and identify the dividing lines between adjacent ground surfaces.

[0099] In this step, an XYZ coordinate system is established on the structured light 3D point cloud, and the ground is identified based on the angle between the normal vector and the Z-axis. The ground is usually a continuous region, so the average of the pixel normal vectors within that region can be taken as the normal vector for that region. The selected region should not exceed 1 / 10 of the total ground area. Different normal vectors in adjacent regions indicate different ground directions. The points where the normal vector direction changes are the dividing lines; that is, the dividing lines between adjacent ground surfaces can be identified by the change in the angle of the normal vector.

[0100] S22: Divide the ground into a first part and a second part according to the dividing line.

[0101] In this step, the first part is the area closer to the shooting position. The second part is the area farther from the shooting position. For example, when using a depth camera, the first part is the area closest to the camera, and the farther part is the second part. The second part can contain one plane, or it can contain two or more planes. For example, when the ground has three different slopes, the plane where the shooting position is located is the first part, and the other two planes are the second part.

[0102] S23: Perform planar fitting on the first part and the second part respectively.

[0103] In this step, a plane fit is performed on the first part. If the second part has only one plane, a plane fit is performed on it. If the second part has two or more planes, a fit is performed on each plane.

[0104] This embodiment divides the ground into different parts and fits them separately, thereby improving the planar fitting effect of the ground and making subsequent navigation, path planning, obstacle recognition and other actions more accurate, so that the invention can be applied to more application scenarios.

[0105] Figure 4 This is a flowchart illustrating another step in planar fitting of a structured light 3D point cloud in an embodiment of the present invention, applicable to continuous processing of images of the same scene. For example... Figure 4 As shown, another method for plane fitting in structured light 3D point clouds in this embodiment of the invention includes:

[0106] S24: Get the position of the dividing line in the previous image.

[0107] In this step, the positions of all dividing lines in the previous image are obtained. If there is only one dividing line in the previous image, then one dividing line is obtained; if there are two dividing lines in the previous image, then both dividing lines are obtained.

[0108] S25: On the current image, expand outwards according to the position of the segmentation line to obtain an extended region, and determine the position of the current image segmentation line within the extended region by the direction of the normal vector.

[0109] In this step, the extent of the expanded area is determined by the speed of the viewpoint change. For example, when shooting from a robot vacuum cleaner, different ranges can be selected based on the robot vacuum cleaner's movement speed; a larger speed results in a larger range, and a smaller speed results in a smaller range. The position of the dividing line can be roughly located in the current image using the position of the dividing line. Then, the same method as identifying the dividing line in the previous embodiment is used, i.e., the position of the dividing line in the current image is determined by the direction of the normal vector.

[0110] S22: Divide the ground into a first part and a second part according to the dividing line.

[0111] S23: Perform planar fitting on the first part and the second part respectively.

[0112] This embodiment utilizes the recognition results of the previous image to significantly reduce the amount of data processing while still accurately obtaining the segmentation line information. This enables real-time information acquisition during continuous shooting, improving data processing capabilities and timely response capabilities.

[0113] Figure 5 This is a flowchart illustrating the steps of obstacle segmentation using TOF3D point cloud technology in an embodiment of the present invention. Figure 5 As shown, a method for segmenting obstacles using TOF3D point clouds in an embodiment of the present invention includes the following steps:

[0114] Step S31: Align the structured light depth data with the TOF depth data.

[0115] In this step, aligning the structured light depth data with the TOF depth data is fundamental for subsequent operations. If the alignment has already been performed in the preceding steps, this step can be skipped.

[0116] Step S32: In the TOF3D point cloud, pixels whose distance from the plane is not greater than the first threshold are marked as the near-ground portion, and pixels whose distance from the plane is greater than the first threshold are marked as the above-ground portion.

[0117] In this step, the data in the TOF3D point cloud are labeled according to their distance from the plane. The data near the ground needs to be corrected based on the structured light depth data and obstacles need to be identified, while the data above the ground does not need to be corrected.

[0118] Step S33: Calculate the normal vector of each pixel in the area near the ground, and calculate the angle α between the normal vector and the ground normal vector.

[0119] In this step, the normal vector is calculated on a pixel-by-pixel basis. The ground in this step refers to the ground region closest to the stated pixel; therefore, the average normal vector of the pixels in this ground region can be taken as the ground normal vector. It should be noted that the region is not the set of all ground pixels, but rather a portion of all ground pixels.

[0120] Step S34: When the included angle α is greater than the second threshold, mark the pixel as the first pixel.

[0121] In this step, when the included angle 'a' is greater than the second threshold, it indicates a large angle with the ground, suggesting it may be part of an obstacle. The value of the second threshold can be selected based on different application scenarios. For example, on a smooth, tiled surface, the included angle 'a' can be smaller to improve the ability to identify small obstacles. On a rough, worn surface, the included angle 'a' can be larger to improve the stability of the identification. Typically, the included angle 'a' can be 15, 20, 25, 30, 35, 40 degrees, or other angles.

[0122] Step S35: When the number of adjacent first pixels is greater than the third threshold, mark the object to which the first pixel belongs as an obstacle.

[0123] In this step, a single pixel may be a flying point. An insufficient number of adjacent pixels may be due to uneven ground surfaces; therefore, a third threshold is needed to filter out these two situations. When identifying the first pixel, it is also necessary to simultaneously identify objects to obtain parts that may belong to the same object, for example, by judging based on abrupt changes in the pixel normal vector.

[0124] This embodiment identifies obstacles based on normal vector information, enabling the identification of small obstacles while filtering out the influence of flying points and surface defects, achieving a balance between recognition accuracy and stability, and can achieve good recognition results in various application scenarios.

[0125] Figure 6 This is a flowchart illustrating a step in correcting TOF data using structured light data, as described in an embodiment of the present invention. Figure 6 As shown, a method for correcting TOF data using structured light data in an embodiment of the present invention includes the following steps:

[0126] S41: Align the structured light depth data and the TOF depth data.

[0127] Aligning the structured light depth data with the TOF depth data in this step is fundamental for subsequent steps. If the structured light depth data and TOF depth data have already been aligned in the preceding steps, this step can be skipped.

[0128] S42: Using the structured light depth data, correct the corresponding TOF depth data of the obstacle.

[0129] In this step, because structured light depth data has higher accuracy, it is used to correct the corresponding Time-of-Flight (TOF) depth data to make the TOF data more accurate. Since structured light uses techniques such as light spots, the obtained accurate data is not continuous; therefore, only a portion of the TOF depth data can be corrected. After this step is completed, step S43 needs to be executed.

[0130] S43: Correct the uncorrected TOF depth data based on the corrected TOF depth data.

[0131] In this step, the corrected Time-of-Flight (TOF) depth data is used to correct adjacent, uncorrected TOF depth data within the 3D point cloud. Since the corrected TOF depth data are independent of each other and contain uncorrected data, data abrupt changes are possible. Algorithms such as linear regression and locally weighted regression can be used to recover the uncorrected TOF depth data.

[0132] This embodiment corrects the TOF depth data using structured light depth data, and then uses the corrected data to regress the uncorrected data in the TOF depth data, thereby making the TOF depth data more accurate and comprehensive, and thus obtaining high-precision depth data.

[0133] Figure 7 This is a schematic diagram of a device for deep fusion of structured light data and TOF data according to an embodiment of the present invention. Figure 7 As shown, an embodiment of the present invention provides a device for deep fusion of structured light data and TOF data, comprising:

[0134] The acquisition module 100 is used to acquire structured light depth data and TOF depth data of the same scene, and generate structured light 3D point cloud and TOF 3D point cloud respectively;

[0135] The structured light module 200 is used to identify planes in the structured light 3D point cloud and perform plane fitting.

[0136] The TOF module 300 is used to identify the portion near the ground and the portion above the ground in the TOF3D point cloud, and to segment obstacles based on the normal vector information of the surface of the portion near the ground; wherein, the portion near the ground refers to the point cloud portion within a first threshold distance from the ground;

[0137] The correction module 400 is used to correct the depth value of the obstacle in the TOF depth data using the structured light depth data;

[0138] The fusion module 500 is used to fuse the structured light depth data with the corrected TOF depth data to obtain high-precision depth data.

[0139] This embodiment can be installed either inside or outside the depth camera. By processing and fusing depth data, it obtains high-precision depth data, improving accuracy, especially in identifying small obstacles. The parameters of this embodiment can also be adjusted according to different scenarios to achieve optimal performance in various situations.

[0140] Figure 8 This is a schematic diagram of a structured light module according to an embodiment of the present invention. Figure 8 As shown, compared to the previous embodiments, the structured light module 200 in this embodiment includes:

[0141] The center point unit 210 is used to identify the ground in the structured light 3D point cloud and to identify the dividing lines of adjacent ground surfaces.

[0142] The segmentation unit 220 is used to divide the ground into a first part and a second part according to the segmentation line;

[0143] The fitting unit 230 is used to perform planar fitting on the first part and the second part respectively.

[0144] Figure 9 This is a schematic diagram of another structured light module in an embodiment of the present invention. Figure 9 As shown, compared to the previous embodiments, the structured light module 200 in this embodiment includes:

[0145] Inheritance unit 240 is used to obtain the position of the segmentation line in the previous image;

[0146] The fine-tuning unit 250 is used to expand an extended region outward based on the position of the segmentation line on the current image, and to determine the position of the current image segmentation line within the extended region by the direction of the normal vector.

[0147] The segmentation unit 220 is used to divide the ground into a first part and a second part according to the segmentation line;

[0148] The fitting unit 230 is used to perform planar fitting on the first part and the second part respectively.

[0149] Figure 10 This is a schematic diagram of a TOF module structure according to an embodiment of the present invention. Figure 10 As shown, compared to the previous embodiments, the TOF module 300 in this embodiment includes:

[0150] Alignment unit 310 is used to align the structured light depth data with the TOF depth data;

[0151] The first marking unit 320 is used to mark pixels in the TOF3D point cloud whose distance from the plane is not greater than the first threshold as the part near the ground, and to mark pixels whose distance from the plane is greater than the first threshold as the part above the ground.

[0152] Normal vector unit 330 is used to calculate the normal vector of each pixel in the area near the ground, and to calculate the angle α between the normal vector and the ground normal vector;

[0153] The second marking unit 340 is used to mark the pixel as the first pixel when the included angle α is greater than the second threshold.

[0154] The third marking unit 350 is used to mark the object to which the first pixel belongs as an obstacle when the number of adjacent first pixels is greater than a third threshold.

[0155] Figure 11 This is a schematic diagram of a correction module structure according to an embodiment of the present invention. Figure 11 As shown, compared to the previous embodiments, the correction module 400 in this embodiment includes:

[0156] Alignment unit 310 is used to align the structured light depth data and the TOF depth data;

[0157] The first correction unit 420 is used to correct the TOF depth data corresponding to the obstacle using the structured light depth data.

[0158] The second correction unit 430 is used to correct the uncorrected TOF depth data based on the corrected TOF depth data.

[0159] The various embodiments described in this specification are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0160] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for deep fusion of structured light data and TOF data, characterized in that, Includes the following steps: Step S1: Acquire structured light depth data and TOF depth data of the same scene, and generate structured light 3D point cloud and TOF 3D point cloud respectively; Step S2: Identify planes in the structured light 3D point cloud and perform plane fitting; Step S3: Identify the portion near the ground and the portion above the ground in the TOF3D point cloud, and segment the obstacle based on the normal vector information of the surface of the portion near the ground; wherein, the portion near the ground refers to the point cloud portion within a first threshold distance from the ground; when the angle between the normal vector and the ground is less than a certain angle, it is determined to be an obstacle; Step S4: Use the structured light depth data to correct the depth value of the obstacle in the TOF depth data; Step S5: Fuse the structured light depth data with the corrected TOF depth data to obtain high-precision depth data.

2. The method for deep fusion of structured light data and TOF data according to claim 1, characterized in that, Step S2 includes: S21: Identify the ground in the structured light 3D point cloud and identify the dividing lines between adjacent ground surfaces; S22: Divide the ground into a first part and a second part according to the dividing line; S23: Perform planar fitting on the first part and the second part respectively.

3. The method for deep fusion of structured light data and TOF data according to claim 1, characterized in that, Step S2 includes: S24: Obtain the position of the segmentation line in the previous image; S25: On the current image, expand outwards according to the position of the segmentation line to obtain an extended region, and determine the position of the current image segmentation line within the extended region by the direction of the normal vector. S22: Divide the ground into a first part and a second part according to the dividing line; S23: Perform planar fitting on the first part and the second part respectively.

4. The method for deep fusion of structured light data and TOF data according to claim 1, characterized in that, Step S3 includes: Step S31: Align the structured light depth data with the TOF depth data; Step S32: In the TOF3D point cloud, pixels whose distance from the plane is not greater than the first threshold are marked as the near-ground portion, and pixels whose distance from the plane is greater than the first threshold are marked as the above-ground portion; Step S33: Calculate the normal vector of each pixel in the area near the ground, and calculate the angle α between the normal vector and the ground normal vector; Step S34: When the included angle α is greater than the second threshold, mark the pixel as the first pixel; Step S35: When the number of adjacent first pixels is greater than the third threshold, mark the object to which the first pixel belongs as an obstacle.

5. The method for deep fusion of structured light data and TOF data according to claim 1, characterized in that, Step S4 includes: S41: Align the structured light depth data and the TOF depth data; S42: Using the structured light depth data, the corresponding TOF depth data of the obstacle is corrected; S43: Correct the uncorrected TOF depth data based on the corrected TOF depth data.

6. A device for deep fusion of structured light data and TOF data, characterized in that, include: The acquisition module is used to acquire structured light depth data and TOF depth data of the same scene, and generate structured light 3D point cloud and TOF 3D point cloud respectively; A structured light module is used to identify planes in the structured light 3D point cloud and perform plane fitting. The TOF module is used to identify the portion near the ground and the portion above the ground in the TOF3D point cloud, and to segment obstacles based on the normal vector information of the surface of the portion near the ground; wherein, the portion near the ground refers to the point cloud portion within a first threshold distance from the ground; when the angle between the normal vector and the ground is less than a certain angle, it is determined to be an obstacle; A correction module is used to correct the depth value of the obstacle in the TOF depth data using the structured light depth data; The fusion module is used to fuse the structured light depth data with the corrected TOF depth data to obtain high-precision depth data.

7. The device for deep fusion of structured light data and TOF data according to claim 6, characterized in that, The structured light module includes: The center point unit is used to identify the ground in the structured light 3D point cloud and to identify the dividing lines between adjacent ground surfaces; A segmentation unit is used to divide the ground into a first part and a second part according to the segmentation line; The fitting unit is used to perform planar fitting on the first part and the second part respectively.

8. The apparatus for deep fusion of structured light data and TOF data according to claim 6, characterized in that, The structured light module includes: Inheritance unit, used to obtain the position of the segmentation line in the previous image; The fine-tuning unit is used to expand outward on the current image according to the position of the segmentation line to obtain an extended region, and to determine the position of the current image segmentation line within the extended region by the direction of the normal vector. A segmentation unit is used to divide the ground into a first part and a second part according to the segmentation line; The fitting unit is used to perform planar fitting on the first part and the second part respectively.

9. The device for deep fusion of structured light data and TOF data according to claim 6, characterized in that, The TOF module includes: An alignment unit is used to align the structured light depth data with the TOF depth data; The first marking unit is used to mark pixels in the TOF3D point cloud whose distance from the plane is not greater than the first threshold as the part near the ground, and to mark pixels whose distance from the plane is greater than the first threshold as the part above the ground. A normal vector unit is used to calculate the normal vector of each pixel in the area near the ground, and to calculate the angle α between the normal vector and the ground normal vector. The second marking unit is used to mark the pixel as the first pixel when the included angle α is greater than the second threshold. The third marking unit is used to mark the object to which the first pixel belongs as an obstacle when the number of adjacent first pixels is greater than a third threshold.

10. The apparatus for deep fusion of structured light data and TOF data according to claim 6, characterized in that, The correction module includes: An alignment unit is used to align the structured light depth data and the TOF depth data. The first correction unit is used to correct the TOF depth data corresponding to the obstacle using the structured light depth data. The second correction unit is used to correct the uncorrected TOF depth data based on the corrected TOF depth data.

Images