RGBD camera

Abstract

The RGBD camera is characterized in that the RGBD camera comprises a projector used for projecting infrared light; the receiver is used for receiving a reflected signal of the infrared light and generating a sparse depth map; the RGB sensor is used for generating an RGB image; the processor is used for performing fusion according to the sparse depth map and the RGB map to obtain fusion features, decoding the fusion features through a feature decoder to obtain a dense depth map and a ground plane space coefficient, finding out an error point according to the ground plane space coefficient, and recalculating the depth of the error point according to an internal reference to obtain the depth of the error point; and correct three-dimensional coordinates are obtained. According to the invention, the binocular depth reconstruction task is realized with low calculation cost.

Description

Technical Field

[0001] The present invention relates to the technical field of depth cameras, and in particular, to an RGBD camera. Background Art

[0002] In recent years, with the rapid development of technology and the continuous growth of application requirements in many fields, the demand for image-based depth reconstruction has shown an increasingly high trend. In many fields such as autonomous driving, virtual reality, augmented reality, and intelligent security monitoring, accurate and efficient depth reconstruction technology plays a crucial role. It can provide rich and accurate spatial information for these fields, helping the system make more intelligent and accurate decisions and feedback. However, successfully achieving depth reconstruction in the target scene, strictly ensuring the flatness of the ground, and maintaining high stability during its time series is undoubtedly a very challenging and difficult task at the technical level.

[0003] Regarding the current technical situation, although the typical depth prediction model based on a depth camera has achieved a certain degree of improvement in overall accuracy, its prediction effect is unsatisfactory when dealing with some extremely common ground scenes in daily life. Taking CRE as an example, although this model can achieve relatively good results in the detailed presentation of depth prediction, once it encounters special scenes such as high highlights or specular reflections on the ground, serious defects will be exposed, and the phenomenon of ground bending is very likely to occur. This ground bending situation will greatly interfere with various subsequent analyses and applications based on depth information, resulting in a significant reduction in the accuracy and reliability of the data, thereby affecting the performance of the entire system and the correctness of the decision-making.

[0004] In addition, if we want to achieve the goal of real-time depth reconstruction in diverse scenarios, there are numerous three-dimensional depth reconstruction algorithms based on depth cameras on the current market. These algorithms theoretically provide various ways and methods for realizing real-time depth reconstruction, but they generally have a significant problem, that is, they have extremely high requirements for computing power. In actual application scenarios, this means that high-performance computing devices need to be equipped, such as powerful GPUs or professional computing clusters, etc. In this way, not only does it increase the hardware cost and energy consumption of the system, but also in some scenarios where requirements are placed on the volume, weight, and portability of the device, such as mobile robots, head-mounted virtual reality devices, etc., the application of these algorithms will be greatly restricted and it is difficult to meet the actual usage requirements.

[0005] The disclosure of the above background art content is only used to assist in understanding the inventive concept and technical solution of the present invention, and it does not necessarily belong to the prior art of this patent application. Without clear evidence indicating that the above content was publicly available on the filing date of this patent application, the above background art should not be used to evaluate the novelty and inventiveness of this application. Summary of the Invention

[0006] To this end, based on an RGBD camera, according to a sparse depth map, a high-resolution dense depth map is obtained using a deep learning network. Finally, through the smoothing of the ground plane coefficients in three-dimensional space, the ground plane is corrected, ensuring the flatness and stability of the ground during depth reconstruction, and achieving the binocular depth reconstruction task at a relatively low computational cost.

[0007] The present invention provides an RGBD camera, which is characterized by comprising:

[0008] A projector for projecting infrared light;

[0009] A receiver for receiving the reflected signal of the infrared light and generating a sparse depth map;

[0010] An RGB sensor for generating an RGB image;

[0011] A processor for fusing the sparse depth map and the RGB map to obtain a fused feature, then decoding the fused feature through a feature decoder to obtain a dense depth map and ground plane spatial coefficients, finding error points according to the ground plane spatial coefficients, and recalculating the depth of the error points according to the internal parameters to obtain correct three-dimensional coordinates.

[0012] Optionally, in the RGBD camera, the processor includes:

[0013] Step S1: Fusing the RGB map and the sparse depth map through a feature encoder to obtain a fused feature, and decoding the fused feature through a feature decoder to obtain a dense depth map and ground plane spatial coefficients; the depth value density of the dense depth map is higher than that of the sparse depth map;

[0014] Step S2: Converting the dense depth map into three-dimensional point cloud data according to the internal parameters;

[0015] Step S3: Fitting out the ground plane;

[0016] Step S4: Obtaining error points below the ground plane in the three-dimensional point cloud according to the ground plane;

[0017] Step S5: Finding the two-dimensional points (x, y) of the corresponding image coordinates according to the error points, taking the depth z as an unknown parameter, calculating the expressions of X and Y according to the internal parameters, substituting the expressions into the plane formula, and finding the depth z' to obtain the correct three-dimensional coordinates (X, Y, Z).

[0018] Optionally, the RGBD camera further includes:

[0019] Step S6: Project the correct three-dimensional coordinates (X, Y, Z) onto the two-dimensional plane through the internal parameters to obtain the final depth prediction result.

[0020] Optionally, for the described RGBD camera, the feature encoder and the feature decoder in step S1 are both convolutional neural networks.

[0021] Optionally, for the described RGBD camera, step S3 includes:

[0022] Step S31: Obtain the first ground plane according to the ground plane spatial coefficient;

[0023] Step S32: Fit the second ground plane according to the three-dimensional point cloud data and calculate the average deviation value between the point cloud and the second ground plane;

[0024] Step S33: Perform weighted averaging on the first ground plane and the second ground plane to obtain the third ground plane; wherein, the weight of the second ground plane is inversely proportional to the average deviation value.

[0025] Optionally, for the described RGBD camera, the third ground plane and the ground plane coefficient of the previous frame are weighted and added as the ground plane coefficient of the current frame.

[0026] Optionally, for the described RGBD camera, the weights of the third ground plane and the ground plane coefficient of the previous frame are proportional to the corresponding point cloud numbers.

[0027] Optionally, for the described RGBD camera, in step S3, the ground plane is fitted according to the ground plane spatial coefficient.

[0028] Optionally, for the described RGBD camera, the ground plane spatial coefficient and the ground plane coefficient of the previous frame are weighted and added as the ground plane coefficient of the current frame.

[0029] Optionally, for the described RGBD camera, in step S5, according to the formula the depth z' is obtained, and then

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] Based on the RGBD camera, according to the sparse depth map, the present invention uses a deep learning network to obtain a high-resolution dense depth map. Finally, through the smoothing of the ground plane coefficient in the three-dimensional space, the ground plane is corrected, ensuring the flatness and stability of the ground during depth reconstruction, and realizing the binocular depth reconstruction task at a low computational cost. Brief Description of the Drawings

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the provided drawings. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objectives, and advantages of the present invention will become more obvious:

[0033] Figure 1 It is a schematic structural diagram of an RGBD camera in an embodiment of the present invention;

[0034] Figure 2 It is a flowchart of the steps processed by a processor in an embodiment of the present invention;

[0035] Figure 3 It is another flowchart of the steps processed by a processor in an embodiment of the present invention;

[0036] Figure 4 It is a flowchart of the steps for fitting a ground plane in an embodiment of the present invention;

[0037] Figure 5 It is a network framework diagram combining time series in an embodiment of the present invention; Detailed Description of the Embodiments

[0038] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can be made. These all belong to the protection scope of the present invention.

[0039] The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims, and drawings of the present invention are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here, for example, can be implemented in an order other than those illustrated or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0040] An RGBD camera provided by an embodiment of the present invention aims to solve the problems existing in the prior art.

[0041] The following uses specific embodiments to elaborate in detail on the technical solutions of the present invention and how the technical solutions of the present application solve the above technical problems. These several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below with reference to the accompanying drawings.

[0042] Based on the RGBD camera, according to the sparse depth map, the present invention uses a deep learning network to obtain a high-resolution dense depth map. Finally, through the smoothing of the ground plane coefficients in the three-dimensional space, the ground plane is corrected, ensuring the flatness and stability of the ground during depth reconstruction while achieving the binocular depth reconstruction task at a relatively low computational cost.

[0043] Figure 1 It is a schematic structural diagram of an RGBD camera in an embodiment of the present invention. As Figure 1 shown, an RGBD camera in an embodiment of the present invention includes:

[0044] A projector for projecting infrared light.

[0045] Specifically, the core function of the projector is to project infrared light outward. Its operation is based on specific optical and electronic technologies. Through internal light-emitting elements (such as infrared light-emitting diodes, etc.), according to set parameters such as mode, angle, and intensity, the infrared light is emitted into the space environment facing the camera lens. These infrared lights will be reflected when they encounter the surface of an object, providing a basic condition for subsequent acquisition of depth information. For example, in an indoor environment, the projected infrared light will be reflected from the surfaces of objects such as tables, chairs, and walls, and then the corresponding reflection signals can be captured by the receiver.

[0046] A receiver for receiving the reflection signal of the infrared light and generating a sparse depth map.

[0047] Specifically, the receiver is responsible for receiving the infrared light reflection signals that are projected by the projector and reflected back from the object surface. The receiver can sense information such as the intensity and direction of the infrared light. When receiving the reflection signals, it will convert the optical signals into electrical signals according to the relevant photoelectric conversion principle, and then process them through the built-in signal processing module, such as amplifying and filtering the signals, removing noise interference, and enhancing the effective signals. Subsequently, based on these processed reflection signals received and the pre-set algorithms, it calculates information such as the distance of the reflection points at different positions relative to the camera, and then generates a sparse depth map. Due to the complexity of the object surface, the depth data often lacks or is incorrect. Therefore, the correct depth map obtained by the receiver is a sparse depth map.

[0048] An RGB sensor, for generating an RGB image.

[0049] Specifically, the RGB sensor is mainly used to capture the visible light information in the environment and then generate an RGB image. It is usually an array composed of multiple photosensitive elements (such as photodiodes, etc.), distributed according to a certain arrangement rule. These photosensitive elements have different response sensitivities to light of the three primary colors: red (R), green (G), and blue (B). When the external light irradiates the sensor surface, the photosensitive elements at different positions will generate corresponding electrical signals according to the intensity of the corresponding color light received. After analog-to-digital conversion and processing by the internal image synthesis algorithm, the numerical values of the red, green, and blue color components corresponding to each pixel point are combined, and finally an RGB image that is rich in color and can intuitively reflect the visual characteristics such as the appearance and texture of environmental objects is formed. For example, when shooting a colorful garden scene, the RGB sensor can accurately capture various bright colors of the flowers and the emerald green of the green leaves and other color information, and truly restore the visual appearance of the entire garden.

[0050] A processor, for fusing the sparse depth map and the RGB map to obtain a fused feature, then decoding the fused feature through a feature decoder to obtain a dense depth map and a ground plane space coefficient, and finding error points according to the ground plane space coefficient, and recalculating the depth of the error points according to the internal parameters to obtain the correct three-dimensional coordinates.

[0051] Specifically, the processor undertakes multiple key processing steps. First, it acquires the sparse depth map generated by the receiver and the RGB image generated by the RGB sensor, and then uses a specialized image fusion algorithm to fuse the two. During the fusion process, according to the corresponding relationships such as the depth information in the depth map and the pixel positions and colors in the RGB image, the features of the two are integrated to obtain the fused features. For example, the depth data at the contour of an object in the depth map is combined with the appearance texture features corresponding to the object in the RGB image to form a fused feature description that includes both spatial position and visual appearance features.

[0052] Next, a feature decoder is used to perform decoding operations on the fused features. The feature decoder is a module constructed based on deep learning or traditional image processing algorithms, which can extract more detailed and accurate depth information, ground plane spatial coefficients, etc. from the fused features. Among them, the dense depth map is obtained by supplementing and refining the original discrete depth data through processing such as fused feature decoding on the basis of the sparse depth map, so that it can cover more pixel positions in the entire image scene, forming a depth map with a corresponding depth value for almost every pixel, and more comprehensively reflecting the spatial structure of the environment. The ground plane spatial coefficients are some parameters related to the ground plane of the camera's environment, such as the tilt angle of the ground plane, the height relative to the camera, etc. By analyzing these coefficients, it is possible to determine which depth data points do not conform to the spatial rules of the ground plane, and then find the error points.

[0053] Finally, for the found error points, the processor will use the pre-calibrated internal parameters of the camera (such as internal optical parameters like focal length, optical center position, etc.) and apply corresponding geometric calculation and correction algorithms to recalculate the depth values of these error points, so as to correct them into correct depth information. Then, combined with other accurate depth information and corresponding pixel positions, etc., the correct three-dimensional coordinates of the object are calculated, and the three-dimensional spatial position and morphological information of the object in the scene are completely constructed.

[0054] This series of processing operations of the processor realizes the conversion from the original infrared reflection signal and visible light image to accurate three-dimensional coordinate information, which is the key to the high-precision and practical implementation of the entire RGBD camera. By fusing different types of image features, the complementary nature of depth information and visual appearance information can be fully utilized to improve the accuracy of scene understanding. The detection and correction of error points and the final calculation of three-dimensional coordinates can ensure the reliability of the output three-dimensional spatial data, enabling the camera to be better applied to many fields with high requirements for spatial position accuracy, such as three-dimensional modeling, virtual reality, robot vision, etc.

[0055] Figure 2 It is a flowchart of the steps of a processor processing in an embodiment of the present invention. AsFigure 2 As shown in Figure 2 , the steps of a processor processing in an embodiment of the present invention include:

[0056] Step S1: The RGB image and the sparse depth map are fused through a feature encoder to obtain a fused feature, and the fused feature is decoded through a feature decoder to obtain a dense depth map and a ground plane spatial coefficient; the depth value density of the dense depth map is higher than that of the sparse depth map.

[0057] In this step, the feature encoder plays a key role. It uses a specific fusion algorithm to fuse the RGB image from the RGB sensor and the sparse depth map generated by the receiver. The fusion process comprehensively considers the visual feature information such as the color and texture of each pixel in the RGB image and the depth data at the corresponding position (or related positions) in the sparse depth map, and integrates these two different types of information according to certain rules and weights to obtain a fused feature. For example, for an object in the scene, the appearance contour, color distribution and other features of the object presented in the RGB image are combined with the depth values of some key positions of the object in the sparse depth map, so that the fused feature contains both the visual appearance of the object and the characteristics of its spatial position depth.

[0058] Subsequently, the feature decoder performs a decoding operation on the obtained fused feature. The feature decoder is usually constructed based on a deep learning model (such as a convolutional neural network, etc.) or traditional image processing and analysis algorithms. It analyzes and extracts more detailed and comprehensive information from the fused feature. On the one hand, it generates a dense depth map, and the depth value density of the dense depth map is higher than that of the sparse depth map, which means that only some discrete positions in the original sparse depth map have depth data, while the decoded dense depth map can give corresponding depth values at more pixel positions, more finely reflecting the spatial depth of the entire scene, just like perfecting a sketch that only outlines the approximate depth of the object into a detailed map with depth markings at almost every detail; on the other hand, it can also calculate the ground plane spatial coefficients, which involve the spatial attitude information of the ground plane relative to the camera, such as the angles between the ground plane and the coordinate axes of the camera coordinate system, the height of the ground plane relative to the camera, etc., providing a basis for subsequent judgment of whether each point in the scene conforms to the ground plane law.

[0059] Through this series of operations of fusion and decoding, the depth mining and integration of the original RGB image and the sparse depth map are realized, and the information advantages of the two different dimensions are complementary, enriching the details of the depth information and extracting the key coefficients related to the ground plane, laying a foundation for subsequent further construction of an accurate three-dimensional scene model.

[0060] Step S2: Convert the dense depth map into three-dimensional point cloud data according to the internal parameters.

[0061] In this step, after obtaining the dense depth map, the processor converts it into three-dimensional point cloud data according to the pre-calibrated internal parameters of the camera. The internal parameters of the camera include important internal optical parameters such as focal length and the position of the optical center, which define the geometric relationship of camera imaging. Based on these internal parameters and the depth value corresponding to each pixel in the dense depth map, using the corresponding coordinate transformation and spatial geometry calculation algorithms, the pixel points on the two-dimensional image plane are transformed into the three-dimensional space, and each pixel point corresponds to a point in the three-dimensional space. The combination of numerous such points forms the three-dimensional point cloud data. For example, for a pixel point with coordinates (x, y) and a depth value of z in the dense depth map, through the geometric relationship determined by the internal parameters, its actual coordinates (X, Y, Z) in the three-dimensional space are calculated. The combination of a large number of such points can present the approximate shape of the entire scene in the three-dimensional space, and objects and the ground in the scene will be presented in the form of point clouds.

[0062] Step S3: Fit the ground plane.

[0063] In this step, mainly perform the operation of fitting the ground plane on the three-dimensional point cloud data. Usually, a suitable mathematical model (such as the plane equation model, commonly in the form of Ax + By + Cz + D = 0) and the corresponding fitting algorithm (such as the least squares method, etc.) are used to achieve this. The processor analyzes the spatial distribution of numerous points in the three-dimensional point cloud data, finds out the points that conform to the characteristics of the ground plane, and determines the parameters (A, B, C, D, etc.) of the ground plane equation through these points, thereby fitting the ground plane. For example, in an indoor scene, theoretically, numerous point cloud data points on the ground should be roughly on the same plane. By applying the fitting algorithm to these points, the plane equation representing the indoor ground can be obtained, accurately describing the position and attitude of the ground in the three-dimensional space.

[0064] The ground plane is an important reference benchmark in many scenes. After fitting the ground plane, it can provide a standard reference for subsequent judgment of which points do not conform to the ground plane law and belong to error points, which helps to improve the accuracy of the three-dimensional coordinate calculation of the entire scene and ensure that the constructed three-dimensional scene conforms to the actual spatial logic.

[0065] Step S4: According to the ground plane, obtain the error points below the ground plane in the three-dimensional point cloud.

[0066] In this step, based on the already fitted ground plane, the processor will screen the 3D point cloud data to find the points below the ground plane as error points. Because in a normal and reasonable scene space layout, most objects should be located above the ground plane or be part of the ground plane itself. Points below the ground plane are likely to be abnormal points caused by calculation errors in depth values due to reasons such as measurement errors and abnormal reflections. For example, in the 3D point cloud data of an outdoor road scene, if there are points significantly lower than the road plane (ground plane), these points probably do not conform to the actual situation and will be determined as error points that need to be corrected.

[0067] Accurately finding the error points can specifically correct these points with depth value problems, avoid these error data from affecting the accuracy of the entire 3D scene model, improve the reliability of the finally output 3D coordinate information, and ensure that the constructed 3D space scene conforms to the actual situation.

[0068] Step S5: Find the two-dimensional points (x, y) of the corresponding image coordinates according to the error points, take the depth z as an unknown parameter, calculate the expressions of X and Y according to the internal parameters, substitute the expressions into the plane formula, and find the depth z' to obtain the correct 3D coordinates (X, Y, Z).

[0069] In this step, for the detected error points, first find the two-dimensional points (x, y) of the corresponding image coordinates according to their positions in the 3D point cloud, and at this time take the depth z as an unknown parameter. Then, based on the geometric relationship determined by the camera internal parameters, through corresponding coordinate transformations and calculations, obtain the expressions of X and Y in terms of the depth z and the known internal parameters and image coordinates (x, y). For example, according to the principle of similar triangles and the camera imaging model, derive the mathematical expression of X related to the parameters such as the focal length in x, y, z and the internal parameters, and the same applies to Y.

[0070] Next, substitute these expressions into the already fitted plane formula (ground plane equation). Since the parameters (A, B, C, D, etc.) in the plane formula are known, the depth z' can be obtained by solving the equation. This new depth value z' is the correct depth after correction. Finally, combined with the previously derived expressions of X and Y and the obtained z', the correct 3D coordinates (X, Y, Z) can be obtained, thereby accurately correcting the coordinates of the error points and improving the spatial information of the entire 3D scene.

[0071] Through this series of calculation and correction operations, effective depth value correction can be performed on the detected error points, enabling each point in the three-dimensional point cloud data to have accurate three-dimensional coordinates, ensuring the accuracy and reliability of the entire three-dimensional scene model in terms of spatial position information, and allowing the three-dimensional spatial data output by the RGBD camera to be better applied to various fields requiring high-precision three-dimensional scene cognition, such as autonomous driving, three-dimensional modeling, etc.

[0072] Figure 3 This is the flowchart of the steps processed by another processor in the embodiment of the present invention. As Figure 3 shown, compared with the foregoing embodiment, the steps processed by another processor in the embodiment of the present invention further include:

[0073] Step S6: Project the correct three-dimensional coordinates (X, Y, Z) onto the two-dimensional plane through the internal parameters to obtain the final depth prediction result.

[0074] In this step, for the correct three-dimensional coordinates (X, Y, Z) that have been corrected, use the projection matrix determined by the camera internal parameters (this matrix can be constructed according to the internal parameter values, which reflects the conversion relationship from three-dimensional coordinates to two-dimensional image coordinates), and perform operations according to the corresponding projection calculation formula. Specifically, generally through linear algebra operations such as matrix multiplication, multiply the three-dimensional coordinates (X, Y, Z) by the projection matrix to obtain the corresponding coordinate values on the two-dimensional plane. The two-dimensional plane coordinates obtained after projection, combined with the corresponding depth value z’, jointly constitute the final depth prediction result. This result not only contains the position information of the object in the two-dimensional image but also carries accurate depth information (reflected by the depth value z’), and can more comprehensively and accurately describe the spatial position and depth distribution of the object in the scene relative to the camera, and can be used in many subsequent related application scenarios such as object detection, scene understanding, three-dimensional reconstruction visualization, etc., providing a high-quality data basis for further analysis and processing.

[0075] Reproject the three-dimensional points back onto the two-dimensional plane through the camera internal parameters to obtain the final depth map prediction result. The formula is as follows:

[0076] z = Z

[0077]

[0078] Figure 4 This is the flowchart of the steps for fitting out the ground plane in the embodiment of the present invention. As Figure 4 shown, the steps for fitting out the ground plane in the embodiment of the present invention include:

[0079] Step S31: Obtain the first ground plane according to the ground plane spatial coefficients.

[0080] In this step, based on the ground plane spatial coefficient, a theoretical ground plane, that is, the first ground plane mentioned here, can be directly determined. It has a definite position and attitude in the three-dimensional space and represents the ground plane situation preliminarily estimated according to the previous feature processing.

[0081] Step S32: Fit a second ground plane according to the three-dimensional point cloud data, and calculate the average deviation value between the point cloud and the second ground plane.

[0082] In this step, using the obtained three-dimensional point cloud data, select those points that may belong to the ground plane. This can be judged based on some prior knowledge or spatial distribution characteristics. For example, points in the bottom of the scene and relatively flat areas are likely to belong to the ground plane part. Then, use a suitable fitting algorithm, such as the least squares method commonly used. Substitute the coordinates of these selected points into the plane equation, and determine the parameters of the plane equation through an optimization solution method, thereby fitting out the second ground plane. This process is to re-determine the position and attitude of the ground plane based on the actual point cloud data, making it more conform to the ground plane situation in the real scene.

[0083] After fitting out the second ground plane, it is necessary to measure the degree of conformity between the point cloud data and this second ground plane, which is reflected by calculating the average deviation value. For each point in the point cloud data, substitute its coordinates into the equation of the second ground plane to calculate the corresponding deviation value (which can be calculated according to the distance formula from a point to a plane, etc.). Then sum up the deviation values of all points and divide by the total number of points to obtain the average deviation value. The smaller the average deviation value, the more consistent the point cloud data is with the fitted second ground plane; otherwise, it indicates a larger deviation.

[0084] Step S33: Perform weighted averaging on the first ground plane and the second ground plane to obtain a third ground plane; where the weight of the second ground plane is inversely proportional to the average deviation value.

[0085] In this step, considering that the first ground plane is determined based on the coefficients obtained from the previous feature decoder and has a certain theoretical basis; while the second ground plane is fitted according to the actual point cloud data and is closer to the actual scene situation, but there may be certain fitting errors (reflected by the average deviation value). In order to combine the advantages of both, a weighted averaging method is used to obtain a more accurate ground plane, that is, the third ground plane.

[0086] It is stipulated that the weight of the second ground plane is inversely proportional to the average deviation value. That is to say, the smaller the average deviation value, the more reliable the second ground plane is, and the greater its weight; conversely, the larger the average deviation value, the smaller its weight. In this embodiment, the equation of the third ground plane in the three-dimensional space is determined, and the final ground plane representation that is more in line with the actual situation and integrates various aspects of information is obtained, providing a more accurate basis for subsequent operations such as error point judgment based on the ground plane.

[0087] In some embodiments, in step S3, a ground plane is fitted according to the ground plane spatial coefficient and the three-dimensional point cloud data. In an ideal situation, a ground plane can be represented by a plane equation (for example, the general form in a three-dimensional Cartesian coordinate system is Ax + By + Cz + D = 0). The ground plane spatial coefficient obtained previously can yield a plane. Using those points in the three-dimensional point cloud data that belong to the ground plane part (it can be roughly judged which points may be on the ground plane through some prior knowledge or based on spatial distribution characteristics, such as points in a relatively flat area near the bottom, etc.), and adopting a suitable fitting algorithm (such as the least squares method, etc.), substituting the coordinates of these points into the plane equation, and through an optimization solution method, determining the plane equation parameters that best conform to the distribution law of these points, thereby obtaining another plane by fitting the point cloud. The two planes are fused to obtain the final ground plane.

[0088] In some embodiments, according to the formula the depth z' is obtained, and further After finding the error point, the two-dimensional point (x, y) corresponding to its image coordinates is found. Regarding the depth z as an unknown parameter, the expressions of X and Y are calculated according to the internal parameters. Substituting the expressions into the plane formula, the correct z is calculated inversely, and then the correct three-dimensional coordinates (X, Y, Z) are deduced according to z. The formula process is as follows:

[0089]

[0090] Z = z

[0091]

[0092] Figure 5 This is a network framework diagram combining time series in an embodiment of the present invention. As Figure 5 shown, this embodiment considers the association between the data of the previous frame and the data of the current frame. Figure 5 The model in

[0093] In some embodiments, in step S3, the third ground plane and the ground plane coefficient of the previous frame are weighted and added as the ground plane coefficient of the current frame. The predicted ground plane spatial coefficient (a t , bt , c t , d t ) and the ground plane coefficients (a m-1 , b m-1 , c m-1 , d m-1 ) stored in the memory are weighted and added together to serve as the ground plane coefficients (a m , b m , c m , d m ) of the current frame, and are stored in the memory for use in the next frame, as shown on the right side in Figure 4 .

[0094] In some embodiments, the ground plane spatial coefficients are weighted and added to the ground plane coefficients of the previous frame to serve as the ground plane coefficients of the current frame. The predicted ground plane spatial coefficients (a t , b t , c t , d t ) of the current frame and the ground plane coefficients (a m-1 , b m-1 , c m-1 , d m-1 ) stored in the memory are weighted and added together to serve as the ground plane coefficients (a m , b m , c m , d m ) of the current frame, and are stored in the memory for use in the next frame, as shown on the right side in Figure 4 .

[0095] In practical application scenarios, such as when a camera continuously captures images and performs depth reconstruction operations in a dynamic environment, there is often a certain continuity and correlation between the ground planes of adjacent frames. By weighted adding the ground plane spatial coefficients calculated for the current frame to the ground plane coefficients of the previous frame to determine the ground plane coefficients of the current frame, it helps to utilize historical information to stabilize the estimation of the ground plane, reduce the inaccurate fitting of the ground plane caused by factors such as noise and local anomalies that may exist in single-frame data, and make the determination of the ground plane smoother and more reliable in a continuous video sequence or dynamic scene.

[0096] Set appropriate weight values according to actual application requirements, experience, etc., which are the weights of the current frame coefficients and the previous frame coefficients respectively, and usually satisfy to ensure the rationality of the result after weighted summation. For example, the weights can be adjusted according to factors such as the expected speed of scene change and the stability of camera movement. If the scene is relatively stable and the camera movement is gentle, can be appropriately increased to make the coefficients of the previous frame play a greater role; on the contrary, if the scene changes rapidly, can be appropriately increased to focus more on the coefficients calculated from the current frame. Calculate according to the rule of weighted summation to obtain the final ground plane coefficient of the current frame.

[0097] Through such a weighted summation method, during the continuous image frame processing, the correlation of ground plane information between different frames can be fully utilized to improve the robustness and accuracy of the entire depth reconstruction and plane correction methods.

[0098] In some embodiments, the weight of the ground plane spatial coefficient and the ground plane coefficient of the previous frame is proportional to the corresponding number of point clouds. When fusing the ground plane coefficients using multi-frame data, the number of point clouds is a very crucial consideration factor. The number of point clouds to a certain extent reflects the reliability or the amount of information of the current frame and the previous frame data for ground plane estimation. Generally speaking, the more the number of point clouds, the richer the data for fitting the ground plane, and the more representative and accurate the calculated ground plane coefficients will be. Therefore, establishing a proportional relationship between the weight and the corresponding number of point clouds is to hope that when fusing the ground plane coefficients, the coefficients obtained based on a larger amount of point cloud data will play a more important role in finally determining the ground plane coefficient of the current frame.

[0099] First, it is necessary to separately count the number of point clouds used to fit the ground plane in the current frame and the number of point clouds used to fit the ground plane in the previous frame. This counting process is usually completed by counting the point clouds that are screened out and may belong to the ground area before performing the ground plane fitting operation.

[0100] According to the principle that the weight is proportional to the number of point clouds, calculate their respective weights. When calculating the weights, first find the sum of the number of point clouds in the two frames, and then determine the weight of the ground plane spatial coefficient of the current frame and the weight of the ground plane coefficient of the previous frame in the following way.

[0101] Through such a weight setting and weighted calculation process, the difference in data information content reflected by the number of point clouds in different frames is fully considered, making the fusion of ground plane coefficients more reasonable and scientific, further improving the accuracy and stability of ground plane estimation during continuous multi-frame processing in a dynamic scene, and providing a more reliable basis for subsequent operations such as depth correction based on the ground plane.

[0102] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0103] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various deformations or modifications within the scope of the claims, which does not affect the essence of the present invention.

Claims

1. An RGBD camera, characterized in that: include: A projector for projecting infrared light; A receiver, configured to receive a reflection signal of the infrared light and generate a sparse depth map; RGB sensor, used to generate RGB images; The processor is used to fuse the sparse depth map and the RGB map to obtain a fused feature, decode the fused feature through a feature decoder to obtain a dense depth map and a ground plane spatial coefficient, find the error point according to the ground plane spatial coefficient, recalculate the depth of the error point according to the internal parameter, and obtain the correct three-dimensional coordinate.

2. The RGBD camera according to claim 1, characterized in that: The processor includes: Step S1: fusing the RGB image and the sparse depth image through a feature encoder to obtain a fused feature, and decoding the fused feature through a feature decoder to obtain a dense depth image and a ground plane spatial coefficient; the depth value density of the dense depth image is higher than that of the sparse depth image; Step S2: converting the dense depth map into three-dimensional point cloud data according to the internal reference; Step S3: fitting the ground plane; Step S4: according to the ground plane, obtaining error points below the ground plane in the three-dimensional point cloud; Step S5: Find the corresponding two-point point (x, y) of the image coordinates according to the error point, take the depth z as the unknown parameter, calculate the expression of X and Y according to the internal parameter, substitute the expression into the plane formula, calculate the depth z', and obtain the correct three-dimensional coordinates (X, Y, Z).

3. The RGBD camera according to claim 2, characterized in that: Also includes: Step S6: Project the correct three-dimensional coordinates (X, Y, Z) onto a two-dimensional plane through internal references to obtain a final depth prediction result.

4. The RGBD camera according to claim 2, characterized in that: The feature encoder and the feature decoder in step S1 are both convolutional neural networks.

5. The RGBD camera according to claim 2, characterized in that: Step S3 includes: Step S31: obtaining a first ground plane according to the ground plane space coefficient; Step S32: fitting a second ground plane according to the three-dimensional point cloud data, and calculating an average deviation value between the point cloud and the second ground plane; Step S33: performing weighted averaging on the first ground plane and the second ground plane to obtain a third ground plane; wherein the weight of the second ground plane is inversely proportional to the average deviation value.

6. The RGBD camera according to claim 5, characterized in that: The third ground plane is weightedly added to the ground plane coefficient of the previous frame to obtain the ground plane coefficient of the current frame.

7. The RGBD camera according to claim 6, characterized in that: The weights of the third ground plane coefficients and the ground plane coefficients of the previous frame are proportional to the corresponding number of point clouds.

8. The RGBD camera according to claim 2, characterized in that: In step S3, a ground plane is fitted according to the ground plane spatial coefficient.

9. The RGBD camera according to claim 8, characterized in that: The ground plane spatial coefficient is weightedly added to the ground plane coefficient of the previous frame to obtain the ground plane coefficient of the current frame.

10. The RGBD camera according to claim 2, characterized in that: In step S5, according to the formula Obtain the depth z', and then obtain

Images