Method and device for detecting plane angular point coordinates, and computer device

Abstract

The invention discloses a plane angular point coordinate detection method and device and computer equipment. The method comprises the steps that RGB image information and depth image information of a plane are acquired; obtaining a two-dimensional coordinate of the plane angular point according to the RGB image information; obtaining a plane equation of the plane according to the depth image information; and according to the two-dimensional coordinates of the plane angular points and the plane equation of the plane, obtaining the three-dimensional coordinates of the plane angular points. According to the method, a visual method and a point cloud method for plane angle point detection are fused, so that the robustness of a three-dimensional coordinate detection algorithm for the plane angle points can be remarkably improved, and the complexity and the calculation power requirement of a calculation process are reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, and computer device for detecting plane corner point coordinates. Background Art

[0002] Currently, industrial robotic arms need to identify and locate their work areas with high precision when performing work surface manipulation tasks, which is crucial to improving the operational efficiency and safety of the robotic arms. Traditional work area recognition methods rely primarily on single-vision or point cloud data processing, but these methods all have significant limitations: 1. Single-vision methods are heavily dependent on environmental conditions and are easily affected by factors such as lighting changes, shadow interference, and reflections, resulting in unstable recognition results in complex environments; 2. Although single point cloud methods can obtain three-dimensional spatial information, they often suffer from insufficient accuracy when processing planar objects, especially in extracting object edges and detailed features. Existing algorithms have poor generalization capabilities when faced with work surfaces of different materials, colors, and shapes, making it difficult to adapt to diverse industrial application scenarios and requiring frequent recalibration and parameter adjustments. Summary of the Invention

[0003] The embodiments of the present application provide a method and device for detecting the coordinates of plane corner points to at least partially solve the above technical problems.

[0004] In order to achieve the above-mentioned object, according to a first aspect of the present application, a method for detecting coordinates of a plane corner point is provided, comprising:

[0005] Obtaining RGB image information and depth image information of the plane;

[0006] Obtaining the two-dimensional coordinates of the plane corner points according to the RGB image information;

[0007] Obtaining a plane equation of the plane according to the depth image information;

[0008] The three-dimensional coordinates of the plane corner point are obtained according to the two-dimensional coordinates of the plane corner point and the plane equation of the plane.

[0009] Optionally, obtaining the two-dimensional coordinates of the plane corner points according to the RGB image information includes:

[0010] The two-dimensional coordinates of the plane corner points are obtained according to the edge detection results and the line detection results of the RGB image information.

[0011] Optionally, obtaining the two-dimensional coordinates of the plane corner points according to the edge detection results and the line detection results of the RGB image information includes:

[0012] Obtaining edge information of the plane according to an edge detection result of the RGB image information;

[0013] The two-dimensional coordinates of the plane corner points are obtained according to the straight line detection result of the edge information.

[0014] Optionally, before obtaining the two-dimensional coordinates of the plane corner points based on the edge detection results and the line detection results of the RGB image information, the method further includes:

[0015] According to the blur detection result of the RGB image information, partially blurred RGB image information is eliminated.

[0016] Optionally, obtaining the plane equation of the plane according to the depth image information includes:

[0017] A plane equation of the plane is obtained according to the segmentation and clustering results of the depth image information and the plane fitting result.

[0018] Optionally, obtaining the plane equation of the plane according to the segmentation and clustering results of the depth image information and the plane fitting result includes:

[0019] Obtaining clustering information of the plane according to the segmentation and clustering results of the depth image information;

[0020] The plane equation of the plane is obtained according to the plane fitting result of the clustering information.

[0021] Optionally, obtaining the plane clustering result according to the segmentation and clustering result of the depth image information includes:

[0022] Obtaining a plurality of sub-depth image information according to the depth image information;

[0023] A clustering result of the plane is obtained according to the clustering analysis results of the plurality of sub-depth image information.

[0024] Optionally, after obtaining the plane equation of the plane based on the segmentation and clustering results of the depth image information and the plane fitting results, the method further includes:

[0025] Boundary information of the plane is obtained according to the plane equation and a calculation result of the convex hull of the plane.

[0026] Optionally, before obtaining the plane equation of the plane based on the segmentation and clustering results of the depth image information and the plane fitting results, the method further includes:

[0027] Get camera internal parameters;

[0028] The point cloud information of the plane is obtained according to the camera intrinsic parameters and the depth image information.

[0029] Optionally, before obtaining the plane equation of the plane based on the segmentation and clustering results of the depth image information and the plane fitting results, the method further includes:

[0030] The filtered depth image information is obtained according to the filtering result of the depth image information.

[0031] Optionally, before obtaining the three-dimensional coordinates of the plane corner point based on the two-dimensional coordinates of the plane corner point and the plane equation of the plane, the method includes:

[0032] Get camera internal parameters;

[0033] The step of obtaining the three-dimensional coordinates of the plane corner point based on the two-dimensional coordinates of the plane corner point and the plane equation of the plane comprises:

[0034] The three-dimensional coordinates of the plane corner point are obtained according to the camera intrinsic parameters, the plane equation and the two-dimensional coordinates.

[0035] Optionally, obtaining the three-dimensional coordinates of the plane corner point according to the camera intrinsic parameters, the plane equation and the two-dimensional coordinates includes:

[0036] Obtaining normalized coordinates of the plane corner point according to the camera intrinsic parameters and the two-dimensional coordinates;

[0037] The three-dimensional coordinates of the plane corner point are obtained according to the normalized coordinates and the plane equation.

[0038] Optionally, after obtaining the three-dimensional coordinates of the plane corner point based on the two-dimensional coordinates of the plane corner point and the plane equation of the plane, the method further includes:

[0039] The final three-dimensional coordinates of the plane corner point are determined according to the rectangle verification result of the three-dimensional coordinates of the plane corner point.

[0040] Optionally, obtaining edge information of the plane according to an edge detection result of the RGB image information includes:

[0041] The Canny edge detection algorithm is used to perform edge detection on the RGB image information, so as to obtain edge information of the plane according to the edge detection result of the RGB image information.

[0042] Optionally, obtaining the two-dimensional coordinates of the plane corner points according to the straight line detection result of the edge information includes:

[0043] A Hough line detection algorithm is used to perform line detection on the edge information, so as to obtain the two-dimensional coordinates of the plane corner point according to the line detection result of the edge information.

[0044] Optionally, obtaining the clustering information of the plane according to the segmentation and clustering result of the depth image information includes:

[0045] The depth image information is segmented and clustered using a DBSCAN clustering algorithm, so as to obtain the flatness clustering information according to the segmentation and clustering results of the depth image information.

[0046] Optionally, obtaining a plane equation of the plane according to a plane fitting result of the clustering information includes:

[0047] A random sampling consensus algorithm is used to perform plane fitting on the clustering information, so as to obtain a plane equation of the plane according to the plane fitting result of the clustering information.

[0048] Optionally, obtaining filtered depth image information according to a result of filtering the depth image information includes:

[0049] The depth image information is filtered using at least one of a time domain filtering algorithm, a spatial filtering algorithm, and a hole filling filtering algorithm, so as to obtain filtered depth image information according to a result of the filtering of the depth image information.

[0050] A device comprises a device for detecting coordinates of plane corner points, wherein the device is used to execute any of the methods described above.

[0051] A computer device, comprising:

[0052] at least one memory for storing a program;

[0053] At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute any one of the methods described above.

[0054] A computer storage medium stores instructions, which, when executed on a computer, enable the computer to execute any one of the methods described above.

[0055] The method of this application fuses the visual method and point cloud method for plane corner point detection, using different algorithms to process RGB image information and depth image information respectively. The two-dimensional coordinates of the plane corner points are obtained based on the RGB image information, and the plane equation of the plane is obtained based on the depth image information. The three-dimensional coordinates of the plane corner points are then obtained based on the two-dimensional coordinates of the plane corner points and the plane equation of the plane through a fusion algorithm. The above method can significantly improve the robustness of the three-dimensional coordinate detection algorithm for plane corner points, reducing the complexity of the calculation process and the computing power requirements.

[0056] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0058] In order to more completely understand the present application and its beneficial effects, the following description will be given in conjunction with the accompanying drawings, wherein the same drawing numbers represent the same parts in the following description.

[0059] Figure 1 is a flow chart of a method for detecting plane corner coordinates provided in an exemplary embodiment of the present disclosure;

[0060] Figure 2 FIG. 4 is a flow chart of another method for detecting plane corner coordinates provided in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0062] According to the first aspect of the present application, a method for detecting the coordinates of a plane corner point is provided. The method can be run in a system composed of one or more devices with relevant functions. Specifically, the method may include: obtaining RGB image information and depth image information of the plane; obtaining the two-dimensional coordinates of the plane corner point based on the RGB image information; obtaining the plane equation of the plane based on the depth image information; and obtaining the three-dimensional coordinates of the plane corner point based on the two-dimensional coordinates of the plane corner point and the plane equation of the plane.

[0063] See also Figure 1 , the method flow chart of this embodiment, the method may include the following specific steps:

[0064] S1: Acquire RGB image information and depth image information of the plane;

[0065] The method of this embodiment can be applied to devices or systems capable of acquiring RGB image information of a target plane, such as mobile phones, tablets, and other devices or systems equipped with camera modules. Furthermore, the device or system can also be capable of acquiring depth image information of the plane, such as smart cars, smartphones, and other devices or systems equipped with depth cameras. In certain embodiments, the device can also be any device or system equipped with an RGB-D camera, without specific limitations herein. All of the above devices and systems can acquire RGB and depth image information of the target plane. In certain embodiments, the device or system can also be an intelligent manipulator, an industrial robot, or a humanoid robot. In certain embodiments, the plane can be a tabletop or other protruding surface of an object, without specific limitations herein. Furthermore, it should be understood that the term "plane" is a general term and has no specific meaning. It simply indicates that the detection target is a surface of an object, and does not require that the detection target be a smooth surface without protrusions or depressions. Those skilled in the art should understand that: 1. The RGB image information can be a color model based on the three primary colors of red, green, and blue, which is widely used in the fields of digital imaging, display devices, and image processing. 2. Depth image information can be an image that records the distance (depth) information of objects in the scene from the camera. It is widely used in computer vision, robotics, 3D reconstruction, augmented reality and other fields.

[0066] S2: Obtain the two-dimensional coordinates of the plane corner points according to the RGB image information;

[0067] The RGB image information can be a color model based on the three primary colors of red, green, and blue, and is widely used in the fields of digital images, display devices, and image processing. The two-dimensional coordinates of the plane corner point are obtained based on the RGB image information. The RGB image information of the plane can be processed. Specifically, the RGB image information can be processed using a direct corner detection algorithm to detect the two-dimensional left side of the plane corner point. Alternatively, edge detection can be performed on the RGB image information first to obtain an edge detection result, and then line detection can be performed on the edge detection result to obtain a line detection result, and then the two-dimensional coordinates of the plane corner point can be calculated. In addition, edge detection can be performed on the RGB image information first to obtain an edge detection result, and then line detection can be performed on the RGB image information to obtain a line detection result. Based on the edge detection result and the line detection result, the two-dimensional coordinates of the plane corner point can be calculated. The specific implementation method is not limited in this embodiment.

[0068] S3: Obtaining a plane equation of the plane according to the depth image information;

[0069] The depth image (DepthImage) information can be an image that records the distance (depth) information of an object in a scene from a camera, and is widely used in fields such as computer vision, robotics, 3D reconstruction, and augmented reality. The plane equation of the plane obtained according to the depth image information can be: by calculating the depth image information to obtain the plane equation of the detection target plane, specifically, the depth image information can be calculated based on one or more algorithms in least squares plane fitting of selected points, RANSAC robust plane fitting, region growing and plane segmentation, combining corner detection and plane association, and deep learning plane prediction to obtain the plane equation of the plane by calculation. Specifically, when the plane is a table top of an operating table, the depth image information of the table top can be calculated using a region growing and plane segmentation algorithm to obtain the plane equation of the table top of the operating table. In addition, because the plane is a general technical name, the specific features of the plane are not limited. Therefore, the operating table top can be an uneven non-plane or a flat table top with other items placed on it, which is not specifically limited here.

[0070] S4: Obtain the three-dimensional coordinates of the plane corner point according to the two-dimensional coordinates of the plane corner point and the plane equation of the plane.

[0071] The two-dimensional coordinates of the plane corner points can be the two-dimensional coordinates of the plane in the plane coordinate system set by the system or device. Similarly, the three-dimensional coordinates of the plane corner points can be the three-dimensional coordinates of the plane in the plane coordinate system set by the system or device. Specifically, the two-dimensional coordinates of the plane corner points and the plane equation of the plane can be calculated by one or more algorithms selected from the group consisting of the inverse projection method, the homography matrix method, the least squares method, and the direct algebra method, so as to obtain the three-dimensional coordinates of the plane corner points. Specifically, when the plane is a table top of an operating table, the least squares method can be used to calculate the two-dimensional coordinates of the plane corner points and the plane equation of the plane, so as to obtain the three-dimensional coordinates of the four corner points of the table top of the operating table. In addition, it should be understood that the specific number of corner points may be determined by the function of the detection object and the specific algorithm used, and is not specifically limited here.

[0072] This method fuses visual and point cloud methods for detecting plane corners, using separate algorithms to process RGB image information and depth image information. The method then uses the fusion algorithm to obtain the 2D coordinates of the plane corners based on the RGB image information and the plane equation of the plane based on the depth image information. The fusion algorithm then uses the 2D coordinates of the plane corners and the plane equation to obtain the 3D coordinates of the plane corners. This method significantly improves the robustness of the 3D coordinate detection algorithm for plane corners, reducing the complexity and computing power required for the calculation process.

[0073] As an optional implementation, see Figure 2 , S21-S23, obtaining the two-dimensional coordinates of the plane corner point based on the RGB image information; including: obtaining the two-dimensional coordinates of the plane corner point based on the edge detection result and the line detection result of the RGB image information.

[0074] The edge detection result of the RGB image information may be edge detection mark information obtained after calculating the RGB image information through an edge detection algorithm, wherein the edge detection algorithm may be a first-order derivative operator such as a Sobel operator, a Prewitt operator, a Roberts operator, a Scharr operator, or a second-order derivative operator such as a Laplacian operator, LoG (Laplacian of Gaussian), or DoG (Difference of Gaussians), or an algorithm such as a Canny edge detection, a SUSAN operator, a Kirsch operator, a deep learning-based HED (Holistically-Nested Edge Detection), a CNN-Based method, a GAN-Based method, or a combination of multiple algorithms, and the like, is not specifically limited here. The edge detection result of the RGB image information can be the line detection mark information of the image obtained after the RGB image information is calculated by a line detection algorithm. The line detection algorithm can be a combination of one or more algorithms such as standard Hough transform, probabilistic Hough transform, LSD (Line Segment Detector), RANSAC (Random Sample Consensus), deep learning methods (such as LSTR, LineFormer) or DBSCAN clustering (density-based line segment detection), which is not specifically limited here. The two-dimensional coordinates of the plane corner points obtained based on the edge detection results and line detection results of the RGB image information can be a combination of one or more algorithms based on the line intersection method based on geometric relationships, the corner detection method based on curvature analysis, and the deep learning end-to-end method. Specifically, it can be a combination of one or more algorithms such as line intersection method, LSD line segment clustering, Harris corner detection, curvature-based corner detection, SuperPoint, DeepLabCut, etc., which is not specifically limited here.

[0075] As an optional implementation, see Figure 2 , S21-S23, obtaining the two-dimensional coordinates of the plane corner point based on the edge detection result and the straight line detection result of the RGB image information; including: obtaining the edge information of the plane based on the edge detection result of the RGB image information; obtaining the two-dimensional coordinates of the plane corner point based on the straight line detection result of the edge information.

[0076] The edge information of the plane is obtained based on the edge detection result of the RGB image information. The edge detection result can be obtained by calculating the RGB image information through any of the above-mentioned algorithms. In addition, the edge information of the plane can be image information with the edge mark information after edge detection. The two-dimensional coordinates of a series of points on the edge of the plane can be known through this image information. The two-dimensional coordinates of the plane corner point are obtained based on the line detection result of the edge information. It can be that line detection is performed on the image information of the plane with edge marks, and line detection is performed on the plane image information with edge marks using any of the above-mentioned line detection algorithms, so that the straight lines among these edge lines can be found, and then the two-dimensional coordinates of the plane corner point are obtained by calculating the intersection points between these straight lines. Finally, part of these intersection points can be determined as the two-dimensional coordinates of the corner point of the plane through rectangle check or triangle check. Specifically, the above is only an exemplary explanation. If there are certain algorithms that can directly obtain the two-dimensional coordinates of the plane intersection point, it should be understood that it is included in the scope of protection of this application.

[0077] As an optional implementation, see Figure 2 , S21-S23, before obtaining the two-dimensional coordinates of the plane corner point based on the edge detection results and the line detection results of the RGB image information, the method may further include: eliminating partially blurred RGB image information based on the blur detection results of the RGB image information.

[0078] The blur detection result can be RGB image information obtained after calculating or screening the RGB image information using a blur detection algorithm or blur threshold screening, and then removing part of the blurred RGB image information according to specific threshold screening requirements to obtain clear RGB image information, which is convenient for the use of this information in subsequent algorithms and processes.

[0079] As an optional implementation, see Figure 2 , S31-S34, obtaining the plane equation of the plane according to the depth image information; including: obtaining the plane equation of the plane according to the segmentation clustering result and the plane fitting result of the depth image information.

[0080] The separation and clustering results can be obtained by calculating and processing the depth image information of the plane using traditional geometric methods and deep learning methods. Specifically, it can be a joint calculation of one or more algorithms including Euclidean clustering, RANSAC plane segmentation, normal vector clustering, 3D CNN segmentation, multi-view projection segmentation, multi-view projection segmentation, and graph convolutional network (GCN). The plane fitting of the depth image information can be calculated by one or more algorithms including least squares method, principal component analysis, random sampling consensus (RANSAC), weighted least squares method, Hough transform, robust regression (M estimation), nonlinear optimization method, etc., which are not specifically limited here.

[0081] As an optional implementation, see Figure 2 , S31-S34, obtaining the plane equation of the plane based on the segmentation clustering results and plane fitting results of the depth image information; including: obtaining the clustering information of the plane based on the segmentation clustering results of the depth image information; obtaining the plane equation of the plane based on the plane fitting results of the clustering information.

[0082] Specifically, the plane segmentation based on region growing, multi-plane detection based on RANSAC, Euclidean clustering and plane verification, hierarchical clustering based on normal vector / curvature, plane segmentation based on deep learning (such as PointNet++), and optimization method based on graph cut can be used to jointly calculate one or more algorithms, and the segmentation and clustering results of the depth image information are used as the input of the algorithm to obtain the clustering information of the plane. In addition, the clustering information of the plane can be calculated using one or more algorithms such as least squares method, principal component analysis, random sampling consensus (RANSAC), weighted least squares method, Hough transform, robust regression (M estimation), nonlinear optimization method, etc., to obtain the plane equation of the plane.

[0083] As an optional implementation, see Figure 2 , S31-S34, the clustering result of the plane is obtained based on the segmentation clustering result of the depth image information; including: obtaining multiple sub-depth image information based on the depth image information; obtaining the clustering result of the plane based on the clustering analysis result of the multiple sub-depth image information.

[0084] Specifically, any one or more algorithms such as connected region analysis, cluster segmentation, threshold segmentation, region growing, deep learning segmentation, and multi-plane segmentation can be used to perform segmentation calculations on the depth image information to obtain multiple sub-depth image information after segmentation. After obtaining multiple sub-depth image information, K-means clustering, DBSCAN (density clustering), hierarchical clustering, spectral clustering and other feature vector-based clustering algorithms, or normal vector consistency clustering, spatial continuity clustering and other geometric constraint-based clustering algorithms, or one or more algorithms in a hybrid method of feature clustering combined with geometric verification can be used to perform cluster analysis on multiple sub-depth image information to obtain the clustering result of the plane.

[0085] As an optional implementation, see Figure 2 , S31-S34, after obtaining the plane equation of the plane based on the segmentation clustering result and the plane fitting result of the depth image information, the method also includes: obtaining the boundary information of the plane based on the plane equation and the convex hull calculation result of the plane.

[0086] Specifically, after obtaining the plane equation of the plane based on the depth image information, a convex hull calculation can be performed on the plane based on the plane equation to obtain the convex hull boundary points of the plane. Specifically, when the plane is an industrial robot operating table, the convex hull calculation can be used to obtain the approximate convex hull boundary points of the operating area of ​​the operating table.

[0087] As an optional implementation, see Figure 2 , S31-S34, before obtaining the plane equation of the plane based on the segmentation clustering result and the plane fitting result of the depth image information, the method also includes: obtaining camera intrinsic parameters; obtaining point cloud information of the plane based on the camera intrinsic parameters and the depth image information.

[0088] Specifically, after obtaining the plane equation of the plane based on the segmentation clustering results and plane fitting results of the depth image information, the intrinsic parameters of the camera can be obtained first. The intrinsic parameters of the camera are the imaging parameters of the camera itself, which can be used to convert the depth image information into point cloud information, thereby obtaining the point cloud information of the plane.

[0089] As an optional implementation, see Figure 2 , S31-S34, before obtaining the plane equation of the plane based on the segmentation clustering results and plane fitting results of the depth image information, the method also includes: obtaining filtered depth image information based on the filtering processing results of the depth image information.

[0090] Specifically, before processing and calculating the depth image information, one or more of a variety of filtering algorithms such as Gaussian filtering algorithm, Temporal Filter (time domain filter), Spatial Filter (spatial filter), Holes Filling Filter (hole filling filter) can be used to filter the depth image information to optimize the depth image information and ensure the integrity of the depth image information.

[0091] As an optional implementation, see Figure 2 , S41-S43, before obtaining the three-dimensional coordinates of the plane corner point according to the two-dimensional coordinates of the plane corner point and the plane equation of the plane; including: obtaining camera intrinsic parameters;

[0092] The method of obtaining the three-dimensional coordinates of the plane corner point according to the two-dimensional coordinates of the plane corner point and the plane equation of the plane includes: obtaining the three-dimensional coordinates of the plane corner point according to the camera intrinsic parameters, the plane equation and the two-dimensional coordinates.

[0093] Specifically, to obtain the three-dimensional coordinates of the plane corner point based on the two-dimensional coordinates of the plane corner point and the plane equation of the plane, the camera intrinsic parameters can be first obtained, and then one or more algorithms such as ray-plane intersection, extrinsic parameter transformation, projection matrix method, least squares optimization, etc. are used to calculate the camera intrinsic parameters, the plane equation, and the two-dimensional coordinates to obtain the three-dimensional coordinates of the plane corner point. More specifically, the three-dimensional coordinates of the operating table corner point can be obtained based on the two-dimensional coordinates of the operating table corner point, the camera intrinsic and extrinsic parameters, and the plane equation of the operating table plane.

[0094] As an optional implementation, see Figure 2 , S41-S43, obtaining the three-dimensional coordinates of the plane corner point according to the camera intrinsic parameters, the plane equation and the two-dimensional coordinates; including: obtaining the normalized coordinates of the plane corner point according to the camera intrinsic parameters and the two-dimensional coordinates; obtaining the three-dimensional coordinates of the plane corner point according to the normalized coordinates and the plane equation.

[0095] Specifically, according to the camera intrinsic parameters and the two-dimensional coordinates, the normalized coordinates of the plane corner point can be calculated by the following formula:

[0096]

[0097] Among them, the two-dimensional coordinates are (u, v), the camera intrinsic focal length f x 、f y and the principal point coordinates (c x， c y), x and y are the normalized coordinates of the plane corner point. In addition, based on the normalized coordinates and the plane equation, the three-dimensional coordinates of the plane corner point can be calculated by the following method:

[0098] Assume the plane equation is:

[0099] Ax+By+Cz+D=0

[0100] According to the plane equation and the normalized coordinates x, y, we can get (X, Y, Z) as follows:

[0101]

[0102] X=x*Z

[0103] Y=y*Z

[0104] The final three-dimensional coordinates (X, Y, Z) calculated are the coordinates of the point (u, v) in the RGB image in the camera coordinate system. The three-dimensional coordinates fuse the plane information under the point cloud feature and the position information under the RGB feature to obtain the four coplanar corner points in space, thereby determining the desktop range of the operating table.

[0105] As an optional implementation, see Figure 2 , S41-S43, after obtaining the three-dimensional coordinates of the plane corner point based on the two-dimensional coordinates of the plane corner point and the plane equation of the plane; the method also includes: determining the final three-dimensional coordinates of the plane corner point based on the rectangle verification result of the three-dimensional coordinates of the plane corner point.

[0106] Specifically, for the rectangle check of the 3D coordinates, the rectangular properties of the four corner points can be verified. By calculating the mutual distances between these four points and sorting the results, it can be determined whether there are two pairs of equal sides. In addition, a threshold can be set in this process to take into account the tolerance of errors.

[0107]

[0108] After calculating the distances between all points, we sort these distances from smallest to largest. Next, we need to determine whether there are two pairs of different distances whose difference is within the preset threshold ε:

[0109] S x -S y ≤ε

[0110] After careful judgment, when a rectangle is successfully formed, the four corner points of the rectangle should be output. This process not only demonstrates the rigor of the algorithm but also its inherent logical consistency. Therefore, the entire algorithm logic should be fully presented to ensure the effectiveness and accuracy of the operation.

[0111] As an optional implementation, see Figure 2 , S41-S43, obtaining the edge information of the plane according to the edge detection result of the RGB image information; including: using the Canny edge detection algorithm to perform edge detection on the RGB image information to obtain the edge information of the plane according to the edge detection result of the RGB image information.

[0112] Specifically, the roughly segmented image can be converted into a grayscale image first, and then the Canny edge detection algorithm (Canny operator) can be applied to the grayscale image for edge detection. To effectively improve the accuracy of edge detection, the image can also be Gaussian filtered to smooth the image and reduce noise interference. Gaussian filtering can use the following algorithms:

[0113]

[0114] After Gaussian filtering, we use the Sobel operator of the Canny operator to calculate the gradient strength and direction of the image. The Sobel operator is a discrete differential operator that is designed to approximate the gradient of the image. The gradient strength is defined as:

[0115]

[0116] The gradient direction is defined as:

[0117]

[0118] The sobel operator is defined as:

[0119]

[0120] Pixels are then meticulously inspected along the gradient direction to preserve local maxima, effectively suppressing non-maxima. Next, by setting high and low threshold hyperparameters, the gradient strength is classified into strong edges, weak edges, and non-edges. Finally, strong edges are connected to achieve holistic edge detection, ensuring the integrity and accuracy of edge features.

[0121] As an optional implementation, see Figure 2, S21-S23, obtaining the two-dimensional coordinates of the plane corner point according to the straight line detection result of the edge information; including: using the Hough line detection algorithm to perform straight line detection on the edge information to obtain the two-dimensional coordinates of the plane corner point according to the straight line detection result of the edge information.

[0122] Specifically, after obtaining the edge detection results, we can use the Hough line detection algorithm to identify the four edges of the desktop. This process allows us to clearly define the working area. In Hough space, the expression of a line can be accurately described as:

[0123] ρ=xcosθ+ysinθ

[0124] An accumulator array is used to record the votes in the parameter space, and the line is identified by detecting the peak value in the accumulator.

[0125] As an optional implementation, see Figure 2 , S31-S34, obtaining the clustering information of the plane according to the segmentation and clustering results of the depth image information; including: using the DBSCAN clustering algorithm to segment and cluster the depth image information to obtain the clustering information of the flatness according to the segmentation and clustering results of the depth image information.

[0126] Specifically, before using the clustering algorithm for calculation, a depth threshold can be set to effectively and roughly segment the point cloud of the desktop plane. Subsequently, the depth image information is restored to point cloud information through the camera intrinsic parameters, and then the segmented point cloud is clustered using the DBSCAN algorithm to further extract the desktop plane. Finally, the accurate desktop plane equation is obtained by plane fitting the clustering results. Specifically, the minimum number of points for DBSCAN can be set to 100, and for each point in the point cloud, its neighborhood is defined as:

[0127] N ε (p)=q∈D|distance≤ε

[0128] When the number of points in its neighborhood exceeds the minimum number of points set, the point is marked as a core point. After obtaining the plane equation, the convex hull calculation can be performed based on the plane to obtain the approximate enclosing boundary of the working area, which is used for verification with the corner points obtained from the RGB image features.

[0129] As an optional implementation, see Figure 2 , S31-S34, obtaining the plane equation of the plane according to the plane fitting result of the clustering information; including: using a random sampling consistency algorithm to perform plane fitting on the clustering information to obtain the plane equation of the plane according to the plane fitting result of the clustering information.

[0130] Specifically, the DBSCAN algorithm is a density-based clustering algorithm that identifies clusters of arbitrary shapes and distinguishes noise points by evaluating the distribution of data points in a neighborhood. Its core principle is density accessibility: first, point types are defined, including core points (the number of data points within their neighborhood radius Eps ≥ MinPts), boundary points (located in the neighborhood of core points but not meeting the core point conditions themselves), and noise points (isolated low-density areas). The algorithm process is divided into three steps:

[0131] 1. Density expansion: Randomly select an unvisited point. If it is a core point, recursively expand it around it and include all density-reachable points in the same cluster.

[0132] 2. Parameter-driven: Control the density sensitivity of clusters by adjusting Eps (neighborhood radius) and MinPts (minimum number of neighborhood points). For example, a smaller Eps tends to identify fine-grained clusters, while increasing MinPts enhances noise immunity.

[0133] 3. Dynamic clustering: It does not require a preset number of clusters and automatically handles non-spherical clusters and noise, but it is sensitive to parameters. The k-distance graph (the horizontal axis is points sorted by distance, and the vertical axis is the distance to the kth nearest neighbor) can be used to assist in determining Eps. Compared with K-means, DBSCAN can effectively handle complex distributions (such as rings and cross clusters) and is more robust, but the computational complexity is higher (O(n) 2 )) is suitable for scenarios such as geospatial hotspot analysis and local object extraction from images. For example, in urban traffic data, Eps = 500 meters and MinPts = 50 can cluster high-density congested areas, while scattered vehicles are marked as noise.

[0134] As an optional implementation, see Figure 2 , S31-S34, the depth image information after filtering is obtained according to the filtering processing result of the depth image information; including: using at least one algorithm of a time domain filtering algorithm, a spatial filtering algorithm, and a hole filling filtering algorithm to filter the depth image information to obtain the filtered depth image information according to the filtering processing result of the depth image information.

[0135] Specifically, in order to improve the stability and accuracy of the depth map, the present invention uses any one or more of the three filtering algorithms to process the depth map, including:

[0136] TemporalFilter (temporal filter): By comparing the depth information of the current depth frame with the depth information of the previous frames, it reduces the influence of random noise and realizes depth information smoothing. The current smoothing value S_t is determined by the following formula, where D_t is the current original depth value and α is the smoothing factor

[0137] S t =α*D t +(1-α)*S t-1

[0138] Spatial Filter: Performs spatial neighborhood filtering on each pixel in the depth map, using the depth information of surrounding pixels to improve the continuity and consistency of the depth map. Where f_r and f_s are Gaussian functions of pixel value and spatial distance respectively:

[0139]

[0140] Holes Filling Filter: When depth values ​​are missing or holes appear in the depth map, a hole filling algorithm is used to estimate and fill the missing areas to ensure the integrity of the depth map. In actual use, the three depth filtering algorithms can be used simultaneously to optimize the depth map.

[0141] The present application also provides a device, which includes a device for detecting coordinates of plane corner points, wherein the device is used to execute the method described in any of the above embodiments to detect the three-dimensional coordinates of the corner points of the plane.

[0142] The present application also provides a computer device, characterized in that it includes: at least one memory for storing programs; at least one processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method described in any of the above embodiments.

[0143] The present application also provides a computer storage medium, wherein the computer storage medium stores instructions. When the instructions are executed on a computer, the computer executes the method described in any of the above embodiments.

[0144] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0145] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0146] The embodiments, implementation methods and related technical features of the present application can be combined and replaced with each other without conflict.

[0147] The above are merely preferred embodiments of the present application and do not constitute any form of limitation to the present application. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A method for detecting plane corner coordinates, characterized in that: include: Obtaining RGB image information and depth image information of the plane; Obtaining the two-dimensional coordinates of the plane corner points according to the RGB image information; Obtaining a plane equation of the plane according to the depth image information; The three-dimensional coordinates of the plane corner point are obtained according to the two-dimensional coordinates of the plane corner point and the plane equation of the plane.

2. The detection method according to claim 1, wherein The step of obtaining the two-dimensional coordinates of the plane corner points according to the RGB image information comprises: The two-dimensional coordinates of the plane corner points are obtained according to the edge detection results and the line detection results of the RGB image information.

3. The detection method according to claim 2, characterized in that The method of obtaining the two-dimensional coordinates of the plane corner points according to the edge detection results and the line detection results of the RGB image information includes: Obtaining edge information of the plane according to an edge detection result of the RGB image information; The two-dimensional coordinates of the plane corner points are obtained according to the straight line detection result of the edge information.

4. The detection method according to claim 2, characterized in that Before obtaining the two-dimensional coordinates of the plane corner points based on the edge detection results and the line detection results of the RGB image information, the method includes: According to the blur detection result of the RGB image information, partially blurred RGB image information is eliminated.

5. The detection method according to claim 1, wherein The step of obtaining a plane equation of the plane according to the depth image information comprises: A plane equation of the plane is obtained according to the segmentation and clustering results of the depth image information and the plane fitting result.

6. The detection method according to claim 5, characterized in that The step of obtaining the plane equation of the plane based on the segmentation and clustering results of the depth image information and the plane fitting results comprises: Obtaining clustering information of the plane according to the segmentation and clustering results of the depth image information; The plane equation of the plane is obtained according to the plane fitting result of the clustering information.

7. The detection method according to claim 6, characterized in that The step of obtaining the plane clustering result based on the segmentation and clustering result of the depth image information comprises: Obtaining a plurality of sub-depth image information according to the depth image information; A clustering result of the plane is obtained according to the clustering analysis results of the plurality of sub-depth image information.

8. The detection method according to claim 5, characterized in that After obtaining the plane equation of the plane based on the segmentation and clustering results of the depth image information and the plane fitting results, the method further includes: Boundary information of the plane is obtained according to the plane equation and a calculation result of the convex hull of the plane.

9. The detection method according to claim 5, characterized in that Before obtaining the plane equation of the plane based on the segmentation and clustering results of the depth image information and the plane fitting results, the method further includes: Get camera internal parameters; The point cloud information of the plane is obtained according to the camera intrinsic parameters and the depth image information.

10. The detection method according to claim 5, characterized in that: Before obtaining the plane equation of the plane based on the segmentation and clustering results of the depth image information and the plane fitting results, the method further includes: The filtered depth image information is obtained according to the filtering result of the depth image information.

11. The detection method according to claim 1, characterized in that Before obtaining the three-dimensional coordinates of the plane corner point according to the two-dimensional coordinates of the plane corner point and the plane equation of the plane, the method includes: Get camera internal parameters; The step of obtaining the three-dimensional coordinates of the plane corner point based on the two-dimensional coordinates of the plane corner point and the plane equation of the plane comprises: The three-dimensional coordinates of the plane corner point are obtained according to the camera intrinsic parameters, the plane equation and the two-dimensional coordinates.

12. The detection method according to claim 11, characterized in that The method of obtaining the three-dimensional coordinates of the plane corner point according to the camera intrinsic parameters, the plane equation and the two-dimensional coordinates includes: Obtaining normalized coordinates of the plane corner point according to the camera intrinsic parameters and the two-dimensional coordinates; The three-dimensional coordinates of the plane corner point are obtained according to the normalized coordinates and the plane equation.

13. The detection method according to claim 1, characterized in that After obtaining the three-dimensional coordinates of the plane corner point according to the two-dimensional coordinates of the plane corner point and the plane equation of the plane; The method further comprises: The final three-dimensional coordinates of the plane corner point are determined according to the rectangle verification result of the three-dimensional coordinates of the plane corner point.

14. The detection method according to claim 3, characterized in that The step of obtaining edge information of the plane according to the edge detection result of the RGB image information comprises: The Canny edge detection algorithm is used to perform edge detection on the RGB image information, so as to obtain edge information of the plane according to the edge detection result of the RGB image information.

15. The detection method according to claim 3, characterized in that Obtaining the two-dimensional coordinates of the plane corner points according to the straight line detection results of the edge information; comprising: A Hough line detection algorithm is used to perform line detection on the edge information, so as to obtain the two-dimensional coordinates of the plane corner point according to the line detection result of the edge information.

16. The detection method according to claim 6, characterized in that The step of obtaining the clustering information of the plane according to the segmentation and clustering result of the depth image information comprises: The depth image information is segmented and clustered using a DBSCAN clustering algorithm, so as to obtain the flatness clustering information according to the segmentation and clustering results of the depth image information.

17. The detection method according to claim 6, characterized in that The step of obtaining a plane equation of the plane according to a plane fitting result of the clustering information comprises: A random sampling consensus algorithm is used to perform plane fitting on the clustering information, so as to obtain a plane equation of the plane according to the plane fitting result of the clustering information.

18. The detection method according to claim 10, characterized in that The step of obtaining filtered depth image information according to a result of filtering the depth image information comprises: The depth image information is filtered using at least one of a time domain filtering algorithm, a spatial filtering algorithm, and a hole filling filtering algorithm, so as to obtain filtered depth image information according to a result of the filtering of the depth image information.

19. A device, characterized in that The device comprises a device for detecting coordinates of plane corner points, wherein the device is used to execute the method according to any one of claims 1-18.

20. A computer device, characterized in that: include: at least one memory for storing a program; At least one processor is used to execute the program stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 18.

21. A computer storage medium, wherein instructions are stored in the computer storage medium, and when the instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 18.

Images