HHA encoding method, device, electronic device and storage medium for depth image
By performing direction clustering screening and normalization of deep image point cloud data, HHA images are generated, which solves the problems of poor feature extraction and slow encoding speed in the prior art, and achieves more efficient environmental information expression and feature extraction.
Patent Information
- Application Number
- CN202111672802.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-31
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-12-31
AI Technical Summary
When using depth images for feature extraction, the existing methods have poor results, cannot effectively express environmental information, and the HHA encoding speed is slow.
By obtaining point cloud data of depth images, performing direction clustering screening, horizontal point cloud categories and vertical point cloud categories are obtained, combining disparity map, height map, normalized processing is performed to generate HHA images .
It improves the HHA encoding speed, better expresses environmental information, facilitates subsequent feature extraction and three-dimensional information calculation, reduces comparison operations during the encoding process, and improves efficiency.
Smart Images

Figure CN114463448B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a HHA encoding method, device, electronic device and storage medium for depth images. Background Art
[0002] Saliency detection imitates the human visual mechanism and uses computers to detect the most prominent objects in the image in the human eye. The saliency map obtained by the saliency detection algorithm is a grayscale map, which uses grayscale values to represent the saliency of different objects in the image. The brighter the area in the image, the more prominent it is. Depth image is a single-channel data representation method that represents the depth information of the environment in the form of an image. Existing methods use depth information to generate saliency maps, but only use depth maps for feature extraction, which has poor feature extraction effect and cannot express more environmental information. Summary of the invention
[0003] The main technical problem solved by the present application is to provide a HHA encoding method, device, electronic device and storage medium for a depth image, which can improve the speed of HHA encoding of the depth image.
[0004] In order to solve the above problems, the first aspect of the present application provides a HHA encoding method for a depth image, and the HHA encoding method for a depth image includes: acquiring point cloud data of a depth image; performing directional clustering screening on the point cloud data to obtain point cloud data after clustering screening, and the point cloud data after clustering screening includes horizontal point cloud categories and vertical point cloud categories; obtaining a disparity map, a height map, and a normal vector and gravity angle map based on the point cloud data after clustering screening; normalizing the disparity map, the height map, and the normal vector and gravity angle map to obtain a HHA image.
[0005] The step of acquiring the point cloud data of the depth image includes: acquiring the depth image by using a detection camera, and performing coordinate transformation on the depth image according to an intrinsic parameter matrix of the detection camera to obtain the point cloud data.
[0006] Wherein, the point cloud data is subjected to directional clustering screening to obtain point cloud data after clustering screening, and the point cloud data after clustering screening includes horizontal point cloud categories and vertical point cloud categories, including: based on the point cloud data, obtaining point cloud normal vectors and point cloud surface curvature; based on the point cloud normal vectors and the point cloud surface curvature, the point cloud data is subjected to directional clustering screening to obtain horizontal point cloud categories and vertical point cloud categories; according to the horizontal point cloud categories and the vertical point cloud categories, the gravity direction is optimized to obtain the point cloud data after clustering screening.
[0007] Among them, the method of obtaining the point cloud normal vector and the point cloud surface curvature based on the point cloud data includes: calculating the coordinate mean of the point cloud in the xyz direction according to the point cloud data; calculating the covariance matrix of the point cloud based on the coordinate mean and the coordinates of each point in the point cloud; performing eigenvalue decomposition on the covariance matrix, and taking the eigenvector corresponding to the minimum eigenvalue as the point cloud normal vector; and performing singular value decomposition on the covariance matrix to obtain the normal vectors corresponding to the three xyz planes, and determining the point cloud surface curvature according to the normal vectors of the three xyz planes.
[0008] Among them, the point cloud data is subjected to directional clustering screening based on the point cloud normal vector and the point cloud surface curvature to obtain horizontal point cloud categories and vertical point cloud categories, including: after the point cloud data is subjected to straight-through filtering, all filtered points are arranged in ascending order according to the curvature to obtain a sorted point set; according to the point cloud data and the curvature value of each point, the sorted point set is divided into a plurality of category field point sets for regional growth; and the horizontal point cloud category and the vertical point cloud category are screened out from the plurality of category field point sets.
[0009] Wherein, the sorted point set is divided into a plurality of category domain point sets for regional growth according to the point cloud data and the curvature value of each point, including: from the sorted point set, a number of points with the smallest curvature value are selected as seed points, the seed points are added to the seed point set and the neighborhood point set, and the seed points are deleted from the sorted point set; the seed point set is traversed: according to the depth image, the neighboring point cloud information of the current seed point is obtained from the point cloud data, the neighboring points in the sorted point set are obtained according to the neighboring point cloud information, and the current seed point and the neighboring points are calculated. The normal vector angle and Euclidean distance between them are added to the current neighborhood point set, and the neighboring points whose normal vector angle meets the preset angle threshold and whose Euclidean distance meets the preset distance threshold are deleted from the sorted point set. In addition, when the curvature value of the neighboring point meets the curvature threshold, the neighboring point is added to the current seed point set; repeat the above steps of traversing the seed point set until the current seed point set is empty, and obtain a current neighborhood point set as a category domain point set; repeat the above steps until the sorted point set is empty, and obtain multiple category domain point sets for regional growth.
[0010] Wherein, the filtering out of the horizontal point cloud category and the vertical point cloud category from the multiple category domain point sets includes: calculating the average normal vector of each of the category domain point sets, merging any two of the category domain point sets whose angle between the average normal vectors is less than a first threshold into the same category, and obtaining a plurality of fused category domain point sets; taking any two of the fused category domain point sets whose angle between the average normal vectors satisfies a preset threshold range as a horizontal-vertical category pair; wherein the preset threshold range is 90°±α, and α is an angle tolerance threshold; for each of the horizontal-vertical category pairs, respectively calculating the average normal vector and the average normal vector of the two fused category domain point sets; The angle of the unit vector of the y-axis, the fused category domain point set with a small angle is used as the horizontal point cloud category, and the fused category domain point set with a large angle is used as the vertical point cloud category; the angle between the average normal vector of each category domain point set and the average normal vector of the horizontal point cloud category is calculated, and the category domain point set with an angle less than a second threshold is added to the horizontal point cloud category to obtain the filtered horizontal point cloud category; and the angle between the average normal vector of each category domain point set and the average normal vector of the vertical point cloud category is calculated, and the category domain point set with an angle less than the second threshold is added to the vertical point cloud category to obtain the filtered vertical point cloud category.
[0011] Among them, the optimization of the gravity direction according to the horizontal point cloud category and the vertical point cloud category to obtain the point cloud data after clustering screening includes: obtaining an optimized target matrix of the gravity direction according to a first angle between the average normal vector of the horizontal point cloud category and the gravity direction and a second angle between the average normal vector of the vertical point cloud category and the gravity direction; performing eigenvalue decomposition on the optimized target matrix of the gravity direction to obtain the current optimal gravity direction; repeating the above steps several times using the Gauss-Newton method to obtain the final optimal gravity direction as the optimized gravity direction, and obtaining the point cloud data after clustering screening based on the optimized gravity direction.
[0012] Among them, the disparity map, height map, and normal vector and gravity angle map are obtained according to the point cloud data after clustering screening, including: calculating the disparity map through the depth image and the fixed parameters of the detection camera; and calculating the corresponding rotation matrix through the optimized gravity direction and the y-axis unit vector, and obtaining the height map according to the point cloud data and the rotation matrix; and performing a dot product operation on the optimized gravity direction and the point cloud normal vector to obtain the normal vector and gravity angle map.
[0013] In order to solve the above problems, the second aspect of the present application provides a HHA encoding device for a depth image, and the HHA encoding device for a depth image includes: an acquisition module, the acquisition module is used to acquire point cloud data of the depth image; a clustering screening module, the clustering screening module is used to perform directional clustering screening on the point cloud data to obtain the point cloud data after clustering screening, and the point cloud data after clustering screening includes horizontal point cloud categories and vertical point cloud categories; a calculation module, the calculation module is used to obtain a disparity map, a height map, and a normal vector and gravity angle map according to the point cloud data after clustering screening; a normalization module, the normalization module is used to normalize the disparity map, the height map, and the normal vector and gravity angle map to obtain a HHA image.
[0014] To solve the above problems, the third aspect of the present application provides an electronic device, wherein the sound source orientation positioning electronic device comprises a processor and a memory connected to each other; the memory is used to store program instructions, and the processor is used to execute the program instructions to implement the HHA encoding method of the depth image of the first aspect above.
[0015] To solve the above problem, the fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, and when the program instructions are executed by a processor, the HHA encoding method for the depth image of the first aspect is implemented.
[0016] The beneficial effects of the present invention are as follows: Different from the prior art, after acquiring the point cloud data of the depth image, the present application can obtain horizontal point cloud categories and vertical point cloud categories by performing directional clustering and screening on the point cloud data, and can obtain a disparity map, a height map, and a normal vector and gravity angle map based on the point cloud data after clustering and screening. After normalizing the disparity map, the height map, and the normal vector and gravity angle map, an HHA image can be obtained. By performing HHA encoding on the depth image, an HHA image can be obtained. Since the HHA image can express more environmental information, it is convenient for subsequent feature extraction and three-dimensional information calculation; in addition, in the process of encoding the HHA image, the strategy of point cloud clustering and plane screening is adopted to obtain horizontal point cloud categories and vertical point cloud categories, so that there is no need to compare all point clouds, which can accelerate the HHA encoding process. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 It is a flowchart of an embodiment of a HHA encoding method for a depth image of the present application;
[0018] Figure 2 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step S12;
[0019] Figure 3 yes Figure 2A schematic diagram of a flow chart of an embodiment of step S121;
[0020] Figure 4 This is a schematic diagram of the visualization of point clouds and their normal vectors in an application scenario;
[0021] Figure 5 yes Figure 2 A flow chart of an embodiment of step S122;
[0022] Figure 6 yes Figure 5 A flow chart of an embodiment of step S1222 in FIG.
[0023] Figure 7 yes Figure 5 A flow chart of an embodiment of step S1223;
[0024] Figure 8 yes Figure 2 A schematic diagram of a flow chart of an embodiment of step S123;
[0025] Fig. 9 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step S13;
[0026] Fig.10 It is a structural schematic diagram of an embodiment of a HHA encoding device for depth images of the present application;
[0027] Fig.11 It is a structural schematic diagram of an embodiment of the electronic device of the present application;
[0028] Fig.12 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION
[0029] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.
[0030] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.
[0031] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.
[0032] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of a HHA encoding method for a depth image of the present application. The HHA encoding method for a depth image in this embodiment comprises the following steps:
[0033] Step S11: Acquire point cloud data of the depth image.
[0034] In this application, a depth image can be obtained by a detection camera. The detection camera can be a depth camera, which can collect color images (RGB data) and depth images (Depth data). Each pixel in the depth image stores the distance information of the object in the scene from the detection camera. Point cloud data is a general description of 3D data, which consists of three-dimensional coordinate points in space. Specifically, the depth image is obtained by using a detection camera. The pixel points in the depth image can be obtained according to the distance information of the pixel points from the detection camera, and the corresponding pixel points in the world coordinate system can be obtained, that is, the corresponding point cloud data.
[0035] In one embodiment, the step S11 specifically includes: acquiring the depth image by using a detection camera, and performing coordinate transformation on the depth image according to an intrinsic parameter matrix of the detection camera to obtain the point cloud data.
[0036] Specifically, the pixel coordinate system can be set as: The world coordinate system is The rotation and translation matrix between the world coordinate system and the camera coordinate system is [R|T], and the intrinsic parameter matrix of the detection camera is Where fx and fy represent the focal length of the detection camera, s represents the coordinate axis tilt parameter, which is generally 0, cx, cy represent the principal point coordinates (optical center coordinates); and the homogeneous transformation matrix from the world coordinate system to the pixel coordinate system is:
[0037]
[0038] Where Z c Represents the depth information in the z direction in the camera coordinate system.
[0039] Since the world coordinate system of the depth image and point cloud is the camera coordinate system, the rotation and translation matrix between the world coordinate system and the camera coordinate system is:
[0040]
[0041] Therefore, the above homogeneous transformation matrix can be simplified to:
[0042]
[0043] Therefore, the coordinates of the pixel points in the point cloud data after coordinate transformation can be obtained as:
[0044]
[0045] Step S12: performing directional clustering screening on the point cloud data to obtain point cloud data after clustering screening, wherein the point cloud data after clustering screening includes horizontal point cloud categories and vertical point cloud categories.
[0046] It can be understood that in the process of encoding HHA images, the point cloud clustering and plane screening strategies are adopted to perform directional clustering and screening on the point cloud data, so that horizontal point cloud categories and vertical point cloud categories can be obtained, so that there is no need to compare all point clouds later, which can accelerate the HHA encoding process.
[0047] Step S13: obtaining a disparity map, a height map, and a normal vector and gravity angle map according to the point cloud data after clustering and screening.
[0048] Step S14: normalizing the disparity map, the height map, and the normal vector and gravity angle map to obtain a HHA image.
[0049] After obtaining the point cloud data after clustering screening, the difference between the left and right images in binocular vision, the height above the ground, and the angle between the point cloud normal vector and the gravity direction can be calculated based on the depth image, point cloud data, and gravity direction, so as to obtain the corresponding disparity map, height map, and normal vector and gravity angle map. Then, the disparity map, height map, and normal vector and gravity angle map are used as the new image three channels for normalization processing. The image obtained based on the new image three channels is the HHA image.
[0050] The above scheme can obtain an HHA image by performing HHA encoding on the depth image. Since the HHA image can express more environmental information, it is convenient for subsequent feature extraction and three-dimensional information calculation. In addition, in the process of encoding the HHA image, the point cloud clustering and plane screening strategies are adopted to obtain horizontal point cloud categories and vertical point cloud categories, so that there is no need to compare all point clouds, which can accelerate the HHA encoding process.
[0051] Please combine Figure 2 , Figure 2 yes Figure 1 Schematic diagram of a flow chart of an embodiment of step S12. In one embodiment, the above step S12 specifically includes:
[0052] Step S121: Based on the point cloud data, obtain the point cloud normal vector and the point cloud surface curvature.
[0053] The point cloud normal vector is an attribute that indicates the geometric shape change of a point cloud at a certain point. It is not only related to the feature information of the point, but also can reflect the features of the point cloud blocks in the neighborhood of the point. The point cloud surface curvature is the characteristic value of the point cloud data surface to describe the degree of change of the point cloud surface. By acquiring the point cloud data and according to the coordinate information of the pixel points in the point cloud data, the point cloud normal vector and the point cloud surface curvature can be obtained.
[0054] Specifically, please combine Figure 3 , Figure 3 yes Figure 2 FIG. 1 is a flow chart of an embodiment of step S121 in FIG. 1. In one embodiment, the above step S121 specifically includes:
[0055] Step S1211: Calculate the coordinate mean of the point cloud in the xyz direction according to the point cloud data.
[0056] Since the point cloud data is composed of three-dimensional coordinate points in space, that is, the point cloud data has the coordinate information of each pixel point, the coordinate mean of the point cloud in the xyz direction can be calculated, where x mean represents the mean of the x coordinates, y mean represents the mean of the y coordinates, z mean Represents the mean of the z-coordinates.
[0057] Step S1212: Calculate the covariance matrix of the point cloud based on the coordinate mean and the coordinates of each point in the point cloud.
[0058] It can be understood that according to the mean value x of the point cloud in the x direction mean , the mean coordinate y of the point cloud in the y direction mean , the coordinate set X of each point in the point cloud in the x direction and the coordinate set Y of each point in the point cloud in the y direction, the corresponding covariance can be obtained as:
[0059]
[0060] Similarly, other covariances can be obtained. Thus, the covariance matrix of the point cloud can be calculated as:
[0061]
[0062] Where X, Y, and Z represent the coordinate sets of the point cloud in the xyz direction respectively.
[0063] Step S1213: performing eigenvalue decomposition on the covariance matrix, and taking the eigenvector corresponding to the minimum eigenvalue as the point cloud normal vector.
[0064] Please combine Figure 4 , Figure 4This is a schematic diagram of the visualization of point clouds and their normal vectors in an application scenario. After obtaining the covariance matrix of the point cloud, the covariance matrix can be decomposed into eigenvalues, and the eigenvector corresponding to the minimum eigenvalue can be used as the point cloud normal vector.
[0065] Step S1214: performing singular value decomposition on the covariance matrix to obtain normal vectors corresponding to the three xyz planes, and determining the surface curvature of the point cloud according to the normal vectors of the three xyz planes.
[0066] Perform singular value decomposition on the covariance matrix C of the point cloud and obtain C = UΣV T , where U and V are both unit orthogonal matrices, that is, UU T =I and VV T =I, U is called the left singular matrix, V is called the right singular matrix, Σ is a diagonal matrix, and only has values on the main diagonal, which are called singular values, and the other elements are all 0. Therefore, according to the diagonal matrix Σ, three singular values can be obtained, namely λ0, λ1 and λ2, and λ0, λ1 and λ2 correspond to the normal vectors of the point cloud in the three planes xyz respectively. Therefore, the curvature of the point cloud surface can be determined according to the three singular values. Specifically, if the three singular values satisfy: λ0<λ1<λ2, the calculation formula of the point cloud surface curvature is:
[0067] Step S122: performing directional clustering screening on the point cloud data based on the point cloud normal vector and the point cloud surface curvature to obtain horizontal point cloud categories and vertical point cloud categories.
[0068] Step S123: optimizing the gravity direction according to the horizontal point cloud category and the vertical point cloud category to obtain the point cloud data after clustering and screening.
[0069] Due to the light sensitivity of the detection camera and other reasons, the acquired depth image is prone to introduce noise. Therefore, the point cloud data is directional clustered and screened based on the point cloud normal vector and the point cloud surface curvature, and the qualified point clouds are clustered into horizontal point cloud categories and vertical point cloud categories. Then, the gravity direction can be optimized according to the horizontal point cloud category and the vertical point cloud category to obtain the optimized gravity direction. The influence of noise on the calculation results of the gravity direction can be eliminated, thereby improving the accuracy of gravity direction estimation. There is no need to compare all point clouds, which can accelerate the HHA encoding process.
[0070] Please combine Figure 5 , Figure 5 yes Figure 2 Schematic diagram of a flow chart of an embodiment of step S122. In one embodiment, the above step S122 specifically includes:
[0071] Step S1221: After performing straight-through filtering on the point cloud data, all filtered points are arranged in ascending order according to curvature to obtain a sorted point set.
[0072] After acquiring the point cloud data, filter-based methods such as straight-through filtering can be used to remove noise from the point cloud data. After filtering, all points can be arranged in ascending order according to the curvature to obtain a sorted point set.
[0073] Step S1222: Divide the sorted point set into a plurality of category domain point sets for region growing according to the point cloud data and the curvature value of each point.
[0074] Region growing is a method of classifying point clouds by taking regional connectivity into consideration. For example, any point in the point cloud data can be taken as a seed point, and then the curvature value and the connectivity conditions between the neighboring points are used to determine whether the neighboring points and the seed point are in the same region, thereby performing region growing segmentation and dividing the sorted point set into multiple category field point sets for region growing.
[0075] Please combine Figure 6 , Figure 6 yes Figure 5 Schematic diagram of a flow chart of an embodiment of step S1222. In one embodiment, the above step S1222 specifically includes:
[0076] Step S12221: From the sorted point set, select several points with the smallest curvature values as seed points, add the seed points to the seed point set and the neighborhood point set, and delete the seed points from the sorted point set.
[0077] After all points are arranged in ascending order according to curvature to obtain a sorted point set, points with smaller curvature values can be selected from the sorted point set as seed points, and these seed points are added to the seed point set and the neighborhood point set, thus obtaining the initial seed point set and the neighborhood point set, and these seed points are deleted from the sorted point set.
[0078] Step S12222: traverse the seed point set: according to the depth image, obtain the neighboring point cloud information of the current seed point from the point cloud data, obtain the neighboring points in the sorted point set according to the neighboring point cloud information, calculate the normal vector angle and Euclidean distance between the current seed point and the neighboring points, add the neighboring points whose normal vector angle meets the preset angle threshold and whose Euclidean distance meets the preset distance threshold to the current neighborhood point set, and delete them from the sorted point set, and in addition, when the curvature value of the neighboring point meets the curvature threshold, add the neighboring point to the current seed point set.
[0079] After obtaining the initial seed point set, each seed point in the seed point set can be traversed. For the current seed point, the neighboring point cloud information of the current seed point can be obtained from the point cloud data according to the index of the corresponding depth image, and then all neighboring points in the sorted point set are obtained according to the neighboring point cloud information, and the normal vector angle and Euclidean distance between the current seed point and each neighboring point are calculated, and then the neighboring points whose normal vector angle meets the preset angle threshold and whose Euclidean distance meets the preset distance threshold are added to the current neighborhood point set to update the neighborhood point set, and these neighboring points added to the neighborhood point set are deleted from the sorted point set. In addition, if there are neighboring points whose curvature values meet the curvature threshold among these neighboring points, the neighboring points whose curvature values meet the curvature threshold are added to the current seed point set; then the current seed point is deleted from the seed point set, and traversed to the next seed point.
[0080] Step S12223: Repeat the above steps of traversing the seed point set until the current seed point set is empty, and obtain a current neighborhood point set as a category domain point set.
[0081] It can be understood that, when the seed point set is traversed according to step S12222, after all the seed points are traversed, the seed point set is empty, and an updated current neighborhood point set is obtained, and the current neighborhood point set is used as a category domain point set.
[0082] Step S12224: Repeat the above steps until the sorted point set is empty, and obtain multiple category domain point sets for region growing.
[0083] It can be understood that after steps S12221 to S12223, a category domain point set can be obtained, and then the remaining points in the sorted point set continue to select several points with the smallest curvature values as seed points, and steps S12221 to S12223 are repeated, so that multiple category domain point sets for regional growth can be obtained.
[0084] Step S1223: Filter out the horizontal point cloud category and the vertical point cloud category from the multiple category domain point sets.
[0085] After all point clouds are divided into a plurality of category domain point sets for region growing, the horizontal point cloud category and the vertical point cloud category may be clustered and screened out from all category domain point sets.
[0086] Please combine Figure 7 , Figure 7 yes Figure 5 Schematic diagram of a flow chart of an embodiment of step S1223. In one embodiment, the above step S1223 specifically includes:
[0087] Step S12231: Calculate the average normal vector of each of the category domain point sets, merge any two of the category domain point sets whose angle between the average normal vectors is less than a first threshold into the same category, and obtain a plurality of fused category domain point sets.
[0088] After all points are divided into multiple category domain point sets for region growing, the average normal vector of each category domain point set is calculated, and then the angle between the average normal vectors of any two category domain point sets is calculated. If the angle is less than a first threshold, the two category domain point sets are merged into the same category. Thus, the multiple category domain point sets obtained in step S122 can be merged into several fused category domain point sets.
[0089] Step S12232: any two of the fused category domain point sets whose included angle between the average normal vectors satisfies a preset threshold range are taken as horizontal-vertical category pairs; wherein the preset threshold range is 90°±α, and α is the angle tolerance threshold.
[0090] After obtaining several fused category domain point sets, the angle between the average normal vectors of any two fused category domain point sets can be calculated. If the angle is within the range of 90°±α, the two fused category domain point sets can be used as horizontal-vertical category pairs. α represents the angle tolerance threshold, so when the roll angle in the depth direction of the detection camera is less than the threshold α, the gravity direction can be adaptively calculated without setting the angle threshold between the y direction of the detection camera and the gravity direction.
[0091] Step S12233: For each horizontal-vertical category pair, respectively calculate the angle between the average normal vector of the two fused category domain point sets and the y-axis unit vector, and use the fused category domain point set with a smaller angle as the horizontal point cloud category, and use the fused category domain point set with a larger angle as the vertical point cloud category.
[0092] After determining the above-mentioned horizontal-vertical category pairs, for each horizontal-vertical category pair, the angle between the average normal vector of the two fused category domain point sets and the y-axis unit vector (0,1,0) is calculated respectively, and then the two angles are compared, and the fused category domain point set with a smaller angle is taken as the horizontal point cloud category, and the fused category domain point set with a larger angle is taken as the vertical point cloud category.
[0093] Step S12234: Calculate the angle between the average normal vector of each of the category domain point sets and the average normal vector of the horizontal point cloud category, add the category domain point sets with an angle less than a second threshold to the horizontal point cloud category, and obtain a filtered horizontal point cloud category.
[0094] Step S12235: Calculate the angle between the average normal vector of each category domain point set and the average normal vector of the vertical point cloud category, add the category domain point sets with an angle less than a second threshold to the vertical point cloud category, and obtain the filtered vertical point cloud category.
[0095] In the above step S12233, the horizontal point cloud category and the vertical point cloud category are obtained. In addition to these two types of point clouds, there are other category domain point sets that are not used as horizontal-vertical category pairs. For these other category domain point sets, the average normal vectors of the horizontal point cloud category and the vertical point cloud category are used as the measurement standard to calculate the angle between the average normal vector of each category domain point set and the average normal vector of the horizontal point cloud category, and the angle between the average normal vector of the vertical point cloud category. Therefore, the category domain point set whose angle with the average normal vector of the horizontal point cloud category is less than the second threshold can be added to the horizontal point cloud category to obtain the filtered horizontal point cloud category, and the category domain point set whose angle with the average normal vector of the vertical point cloud category is less than the second threshold can be added to the vertical point cloud category to obtain the filtered vertical point cloud category, so that all point clouds that meet the conditions can be clustered and filtered out into the horizontal point cloud category and the vertical point cloud category. It can be understood that step S12234 and step S12235 can be performed simultaneously.
[0096] Please combine Figure 8 , Figure 8 yes Figure 2 Schematic diagram of a flow chart of an embodiment of step S123 in FIG. In one embodiment, the above step S123 specifically includes:
[0097] Step S1231: Obtain an optimized target matrix of the gravity direction according to a first angle between the average normal vector of the horizontal point cloud category and the gravity direction and a second angle between the average normal vector of the vertical point cloud category and the gravity direction.
[0098] Step S1232: performing eigenvalue decomposition on the optimization target matrix of the gravity direction to obtain the current optimal gravity direction.
[0099] Step S1233: Repeat the above steps several times using the Gauss-Newton method to obtain the final optimal gravity direction, and obtain the clustered and screened point cloud data based on the optimized gravity direction.
[0100] It can be understood that the optimization objective equation in the direction of gravity is expressed as:
[0101]
[0102] Among them, θ(n,g) represents the angle between vectors n and g, n∈μ ⊥Represents the average normal vector of the vertical point cloud category, n∈μ 口 represents the average normal vector of the horizontal point cloud category, and g is the gravity vector. The optimization target matrix M in the gravity direction can be expressed in matrix form as follows:
[0103] M=N ⊥ N ⊥ T -N || N || T , where N ⊥ is the matrix representation of the average normal vector of the vertical point cloud category, N || is the matrix representation of the average normal vector of the horizontal point cloud category, and M is a 3*3 matrix. By performing eigenvalue decomposition on the optimization target matrix M of the gravity direction, the current optimal gravity direction can be calculated; then the Gauss-Newton method is used to repeat the calculation N times to obtain the final optimal gravity direction, that is, the optimized gravity direction. After obtaining the optimized gravity direction, the point cloud data after the final clustering screening can be determined according to the optimized gravity direction.
[0104] Please combine Fig. 9 , Fig. 9 yes Figure 1 Schematic diagram of a flow chart of an embodiment of step S13. In one embodiment, the above step S13 specifically includes:
[0105] Step S131: Calculate the disparity map using the depth image and fixed parameters of the detection camera.
[0106] It can be understood that the binocular disparity calculation formula is:
[0107]
[0108] Among them, f represents the focal length of the detection camera, b represents the baseline distance between the two eyes, Disparity represents the disparity value, and d represents the depth. Therefore, by inputting the depth image, according to the depth information d in the depth image and the fixed parameters f and b of the detection camera, the disparity value of each pixel can be obtained, and then the disparity map can be obtained.
[0109] Step S132: Calculate the corresponding rotation matrix through the optimized gravity direction and the y-axis unit vector, and obtain the height map according to the point cloud data and the rotation matrix.
[0110] Specifically, the rotation matrix Rotate between the optimized gravity direction vector G and the y-axis unit vector of the camera coordinate system of the detection camera can be calculated through the optimized gravity direction vector G and the y-axis unit vector, and the rotation matrix is used for the original point cloud data:
[0111] pcRotate n×3 =pc n×3 Rotate 3×3 ;
[0112] Among them, pc is the point cloud and n is 3.
[0113] Then, the height map can be expressed as:
[0114] H=pcRotate n×3 [y]-minpcRotate n×3 [y].
[0115] Step S133: performing a dot product operation on the optimized gravity direction and the point cloud normal vector to obtain a graph of the normal vector and the gravity angle.
[0116] By performing a dot product operation on the optimized gravity direction and the point cloud normal vector, a graph of the normal vector and the gravity angle can be obtained.
[0117] The above steps S151, S152 and S153 may be performed simultaneously.
[0118] See also Fig.10 , Fig.10 It is a structural diagram of an embodiment of the HHA encoding device for depth images of the present application. The HHA encoding device 100 for depth images in this embodiment includes an acquisition module 1000, a clustering screening module 1002, a calculation module 1004 and a normalization module 1006 which are interconnected; the acquisition module 1000 is used to acquire point cloud data of the depth image; the clustering screening module 1002 is used to perform directional clustering screening on the point cloud data to obtain point cloud data after clustering screening, and the point cloud data after clustering screening includes horizontal point cloud categories and vertical point cloud categories; the calculation module 1004 is used to obtain a disparity map, a height map, and a normal vector and gravity angle map based on the point cloud data after clustering screening; the normalization module 1006 is used to perform normalization processing on the disparity map, the height map, and the normal vector and gravity angle map to obtain an HHA image.
[0119] In one embodiment, the acquisition module 1000 executes the step of acquiring point cloud data of a depth image, including: acquiring the depth image using a detection camera, and performing coordinate transformation on the depth image according to an intrinsic parameter matrix of the detection camera to obtain the point cloud data.
[0120] In one embodiment, the clustering screening module 1002 performs directional clustering screening on the point cloud data to obtain point cloud data after clustering screening, and the point cloud data after clustering screening includes the steps of horizontal point cloud categories and vertical point cloud categories, including: based on the point cloud data, obtaining the point cloud normal vector and the point cloud surface curvature; based on the point cloud normal vector and the point cloud surface curvature, performing directional clustering screening on the point cloud data to obtain horizontal point cloud categories and vertical point cloud categories; optimizing the gravity direction according to the horizontal point cloud categories and the vertical point cloud categories to obtain the point cloud data after clustering screening.
[0121] In one embodiment, the clustering screening module 1002 executes the steps of obtaining the point cloud normal vector and the point cloud surface curvature based on the point cloud data, specifically including: calculating the coordinate mean of the point cloud in the xyz direction according to the point cloud data; calculating the covariance matrix of the point cloud based on the coordinate mean and the coordinates of each point in the point cloud; performing eigenvalue decomposition on the covariance matrix, and taking the eigenvector corresponding to the minimum eigenvalue as the point cloud normal vector; and performing singular value decomposition on the covariance matrix to obtain the normal vectors corresponding to the three xyz planes, and determining the point cloud surface curvature according to the normal vectors of the three xyz planes.
[0122] In one embodiment, the clustering screening module 1002 performs directional clustering screening on the point cloud data based on the point cloud normal vector and the point cloud surface curvature to obtain the steps of horizontal point cloud categories and vertical point cloud categories, including: after straight-through filtering the point cloud data, all filtered points are arranged in ascending order according to the curvature to obtain a sorted point set; according to the point cloud data and the curvature value of each point, the sorted point set is divided into multiple category field point sets for regional growth; and the horizontal point cloud category and the vertical point cloud category are screened out from the multiple category field point sets.
[0123] In one embodiment, the clustering screening module 1002 performs the steps of dividing the sorted point set into a plurality of category domain point sets for regional growth according to the point cloud data and the curvature value of each point, specifically including: selecting a number of points with the smallest curvature value as seed points from the sorted point set, adding the seed points to the seed point set and the neighborhood point set, and deleting the seed points from the sorted point set; traversing the seed point set: obtaining the neighboring point cloud information of the current seed point from the point cloud data according to the depth image, obtaining the neighboring points located in the sorted point set according to the neighboring point cloud information, and calculating the current seed point. The normal vector angle and Euclidean distance between the previous seed point and the adjacent point, the adjacent points whose normal vector angle meets the preset angle threshold and whose Euclidean distance meets the preset distance threshold are added to the current neighborhood point set, and deleted from the sorted point set. In addition, when the curvature value of the adjacent point meets the curvature threshold, the adjacent point is added to the current seed point set; repeat the above steps of traversing the seed point set until the current seed point set is empty, and obtain a current neighborhood point set as a category domain point set; repeat the above steps until the sorted point set is empty, and obtain multiple category domain point sets for regional growth.
[0124] In one embodiment, the clustering screening module 1002 performs the steps of screening out the horizontal point cloud category and the vertical point cloud category from the multiple category domain point sets, specifically including: calculating the average normal vector of each category domain point set, merging any two category domain point sets whose angle between the average normal vectors is less than a first threshold into the same category, and obtaining a plurality of fused category domain point sets; taking any two fused category domain point sets whose angle between the average normal vectors satisfies a preset threshold range as a horizontal-vertical category pair; wherein the preset threshold range is 90°±α, and α is an angle tolerance threshold; for each horizontal-vertical category pair, respectively calculating the two fused category domain points; The angle between the average normal vector of the point set and the unit vector of the y-axis, the fused category domain point set with a small angle is used as the horizontal point cloud category, and the fused category domain point set with a large angle is used as the vertical point cloud category; the angle between the average normal vector of each category domain point set and the average normal vector of the horizontal point cloud category is calculated, and the category domain point set with an angle less than a second threshold is added to the horizontal point cloud category to obtain the filtered horizontal point cloud category; and the angle between the average normal vector of each category domain point set and the average normal vector of the vertical point cloud category is calculated, and the category domain point set with an angle less than the second threshold is added to the vertical point cloud category to obtain the filtered vertical point cloud category.
[0125] In one embodiment, the clustering screening module 1002 performs the steps of optimizing the gravity direction according to the horizontal point cloud category and the vertical point cloud category to obtain the point cloud data after clustering screening, including: obtaining an optimized target matrix of the gravity direction according to a first angle between the average normal vector of the horizontal point cloud category and the gravity direction and a second angle between the average normal vector of the vertical point cloud category and the gravity direction; performing eigenvalue decomposition on the optimized target matrix of the gravity direction to obtain the current optimal gravity direction; repeating the above steps several times using the Gauss-Newton method to obtain the final optimal gravity direction as the optimized gravity direction, and obtaining the point cloud data after clustering screening based on the optimized gravity direction.
[0126] In one embodiment, the calculation module 1004 executes the steps of obtaining a disparity map, a height map, and a normal vector and gravity angle map based on the point cloud data after clustering screening, including: calculating the disparity map through the depth image and the fixed parameters of the detection camera; and calculating the corresponding rotation matrix through the optimized gravity direction and the y-axis unit vector, and obtaining the height map based on the point cloud data and the rotation matrix; and performing a dot product operation on the optimized gravity direction and the point cloud normal vector to obtain the normal vector and gravity angle map.
[0127] For details about the HHA encoding method for depth images implemented by the HHA encoding device 100 of the present application, please refer to the contents in the above-mentioned HHA encoding method embodiment for depth images, which will not be described again here.
[0128] See also Fig.11 , Fig.11 1 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 110 in this embodiment includes a processor 1102 and a memory 1101 connected to each other; the memory 1101 is used to store program instructions, and the processor 1102 is used to execute the program instructions stored in the memory 1101 to implement the steps of any of the above-mentioned HHA encoding method embodiments of the depth image. In a specific implementation scenario, the electronic device 110 may include but is not limited to: a microcomputer, a server.
[0129] Specifically, the processor 1102 is used to control itself and the memory 1101 to implement the steps of any of the above-mentioned HHA encoding method embodiments for the depth image. The processor 1102 may also be referred to as a CPU (Central Processing Unit). The processor 1102 may be an integrated circuit chip having the ability to process signals. The processor 1102 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 1102 may be implemented by an integrated circuit chip.
[0130] See also Fig.12 , Fig.12 The computer-readable storage medium 120 of the present application stores program instructions 1200 thereon, and when the program instructions 1200 are executed by a processor, the steps in any of the above-mentioned HHA encoding method embodiments for depth images are implemented.
[0131] The computer-readable storage medium 120 may specifically be a medium that can store program instructions 1200, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or the like, or it may be a server that stores the program instructions 1200. The server may send the stored program instructions 1200 to other devices for execution, or it may also execute the stored program instructions 1200 by itself.
[0132] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and apparatuses can be implemented in other ways. For example, the above-described device and apparatus implementation methods are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0133] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0134] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0135] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.
Claims
1. A HHA encoding method for a depth image, characterized in that: The HHA encoding method of the depth image comprises: Get point cloud data of depth image; Performing directional clustering screening on the point cloud data to obtain point cloud data after clustering screening, wherein the point cloud data after clustering screening includes a horizontal point cloud category and a vertical point cloud category; Obtaining a disparity map, a height map, and a normal vector and gravity angle map according to the point cloud data after clustering and screening; Normalizing the disparity map, the height map, and the normal vector and gravity angle map to obtain a HHA image; The point cloud data is subjected to directional clustering screening to obtain point cloud data after clustering screening, and the point cloud data after clustering screening includes horizontal point cloud categories and vertical point cloud categories, including: Based on the point cloud data, obtaining a point cloud normal vector and a point cloud surface curvature; Performing directional clustering screening on the point cloud data based on the point cloud normal vector and the point cloud surface curvature to obtain horizontal point cloud categories and vertical point cloud categories; The gravity direction is optimized according to the horizontal point cloud category and the vertical point cloud category to obtain the point cloud data after clustering and screening.
2. The HHA encoding method of depth image according to claim 1, characterized in that: The step of obtaining the point cloud data of the depth image includes: The depth image is acquired by using a detection camera, and the coordinates of the depth image are transformed according to an intrinsic parameter matrix of the detection camera to obtain the point cloud data.
3. The HHA encoding method of depth image according to claim 1, characterized in that: The step of obtaining a point cloud normal vector and a point cloud surface curvature based on the point cloud data includes: According to the point cloud data, calculating the coordinate mean of the point cloud in the xyz direction; Calculate the covariance matrix of the point cloud based on the coordinate mean and the coordinates of each point in the point cloud; Performing eigenvalue decomposition on the covariance matrix, and taking the eigenvector corresponding to the minimum eigenvalue as the point cloud normal vector; and The covariance matrix is subjected to singular value decomposition to obtain normal vectors corresponding to the three xyz planes, and the surface curvature of the point cloud is determined according to the normal vectors of the three xyz planes.
4. The HHA encoding method of depth image according to claim 1, characterized in that: The step of performing directional clustering screening on the point cloud data based on the point cloud normal vector and the point cloud surface curvature to obtain horizontal point cloud categories and vertical point cloud categories includes: After performing straight-through filtering on the point cloud data, all filtered points are arranged in ascending order according to curvature to obtain a sorted point set; According to the point cloud data and the curvature value of each point, the sorted point set is divided into a plurality of category domain point sets for region growing; The horizontal point cloud category and the vertical point cloud category are screened out from the multiple category area point sets.
5. The HHA encoding method of depth image according to claim 4, characterized in that: The step of dividing the sorted point set into a plurality of category domain point sets for region growing according to the point cloud data and the curvature value of each point includes: From the sorted point set, select a number of points with the smallest curvature values as seed points, add the seed points to the seed point set and the neighborhood point set, and delete the seed points from the sorted point set; Traversing the seed point set: according to the depth image, obtaining the neighboring point cloud information of the current seed point from the point cloud data, obtaining the neighboring points in the sorted point set according to the neighboring point cloud information, calculating the normal vector angle and the Euclidean distance between the current seed point and the neighboring points, adding the neighboring points whose normal vector angle meets the preset angle threshold and whose Euclidean distance meets the preset distance threshold to the current neighborhood point set, and deleting them from the sorted point set, and in addition, when the curvature value of the neighboring point meets the curvature threshold, adding the neighboring point to the current seed point set; Repeat the above steps of traversing the seed point set until the current seed point set is empty, and obtain a current neighborhood point set as a category domain point set; Repeat the above steps until the sorted point set is empty, and obtain multiple category domain point sets for region growing.
6. The HHA encoding method of depth image according to claim 4, characterized in that: The step of selecting the horizontal point cloud category and the vertical point cloud category from the plurality of category field point sets comprises: Calculating the average normal vector of each of the category domain point sets, merging any two of the category domain point sets whose angle between the average normal vectors is less than a first threshold into the same category, to obtain a plurality of fused category domain point sets; Any two of the fused category domain point sets whose angle between the average normal vectors satisfies a preset threshold range are taken as horizontal-vertical category pairs; wherein the preset threshold range is 90°±α, and α is an angle tolerance threshold; For each horizontal-vertical category pair, respectively calculate the angle between the average normal vector of the two fused category domain point sets and the y-axis unit vector, and use the fused category domain point set with a smaller angle as the horizontal point cloud category, and use the fused category domain point set with a larger angle as the vertical point cloud category; Calculating the angle between the average normal vector of each of the category domain point sets and the average normal vector of the horizontal point cloud category, adding the category domain point sets with an angle less than a second threshold to the horizontal point cloud category to obtain a screened horizontal point cloud category; and The angle between the average normal vector of each of the category domain point sets and the average normal vector of the vertical point cloud category is calculated, and the category domain point sets with an angle less than a second threshold are added to the vertical point cloud category to obtain a screened vertical point cloud category.
7. The HHA encoding method of depth image according to claim 1, characterized in that: The step of optimizing the gravity direction according to the horizontal point cloud category and the vertical point cloud category to obtain the point cloud data after cluster screening includes: Obtaining an optimized target matrix of the gravity direction according to a first angle between the average normal vector of the horizontal point cloud category and the gravity direction and a second angle between the average normal vector of the vertical point cloud category and the gravity direction; Performing eigenvalue decomposition on the optimization target matrix of the gravity direction to obtain the current optimal gravity direction; The above steps are repeated several times using the Gauss-Newton method to obtain the final optimal gravity direction as the optimized gravity direction, and the cluster-screened point cloud data is obtained based on the optimized gravity direction.
8. The HHA encoding method of depth image according to claim 2, characterized in that: The disparity map, the height map, and the normal vector and gravity angle map are obtained according to the point cloud data after clustering and screening, including: The disparity map is calculated by using the depth image and fixed parameters of the detection camera; and By using the optimized gravity direction and the y-axis unit vector, a corresponding rotation matrix is calculated, and the height map is obtained according to the point cloud data and the rotation matrix; and A dot product operation is performed on the optimized gravity direction and the point cloud normal vector to obtain a graph of the normal vector and the gravity angle.
9. A HHA encoding device for a depth image, characterized in that: The HHA encoding device of the depth image comprises: An acquisition module, wherein the acquisition module is used to acquire point cloud data of a depth image; A clustering and screening module, wherein the clustering and screening module is used to perform directional clustering and screening on the point cloud data to obtain the point cloud data after clustering and screening, wherein the point cloud data after clustering and screening includes a horizontal point cloud category and a vertical point cloud category, and the clustering and screening module is used to obtain a point cloud normal vector and a point cloud surface curvature based on the point cloud data; perform directional clustering and screening on the point cloud data based on the point cloud normal vector and the point cloud surface curvature to obtain a horizontal point cloud category and a vertical point cloud category; optimize the gravity direction according to the horizontal point cloud category and the vertical point cloud category to obtain the point cloud data after clustering and screening; A calculation module, the calculation module is used to obtain a disparity map, a height map, and a normal vector and gravity angle map according to the point cloud data after clustering and screening; A normalization module is used to normalize the disparity map, the height map, and the normal vector and gravity angle map to obtain a HHA image.
10. An electronic device, characterized in that: The electronic device comprises a processor and a memory connected to each other; The memory is used to store program instructions, and the processor is used to execute the program instructions to implement the HHA encoding method for a depth image according to any one of claims 1 to 8.
11. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the HHA encoding method for a depth image according to any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
Point cloud data processing method, device and equipment and storage medium
CN111986299A
Infrared image and depth image bimodal target segmentation method and device
CN112115864A
DBSCAN clustering-based point cloud semantic segmentation method and system in urban road environment, and storage medium
CN112200248A
Topological map node generation method based on laser point cloud distribution characteristics
CN112348950A
Coal face coal rock recognition method based on laser scanning
CN113486817A
