Metacosm object digitization method based on human-in-loop semantic correction and hyper-quadric surface parameter separation initialization
By using the method based on human-in-loop semantic correction and hyperquadratic surface parameter separation, the problem of insufficient accuracy and applicability of existing SLAM technology in semantic information is solved, and high-precision modeling and interaction of metacosmic objects is realized, which improves the Camera positioning accuracy.
Patent Information
- Application Number
- CN202411429156.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2025-01-14
AI Technical Summary
The existing object-level semantic SLAM technology has insufficient accuracy and applicability in scene semantic information. The success rate of quadratic surface initialization is not high, and it is impossible to effectively distinguish objects in complex scenes, resulting in difficulty in modeling and interaction of metacosmic objects.
Using the method based on human-in-loop semantic correction and ultraquadratic surface parameter separation and initialization, the semantic information is manually corrected, and the local map is constructed based on image depth information to estimate the camera position and hyperquadratic surface parameters. , using local homography matrix to extend semantic correction results, to construct a detailed geometric semantic map.
It effectively reduces the error rate of semantic information of metaverse scenes, improves camera positioning accuracy and object initialization accuracy, realizes meticulous modeling and interaction of complex scenes, and enhances the semantic information input of the metaverse environment.
Smart Images
Figure CN119313839A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of metaverse object modeling, and in particular to a metaverse object digitization method based on human-in-the-loop semantic correction and superquadratic surface parameter separation initialization. Background Art
[0002] Three-dimensional modeling is the cornerstone of building the metaverse. In the space of the metaverse, the process of digitizing physical objects is the process of their three-dimensional reconstruction in the virtual space. Combined with digital twins, physical objects in the real world can be mapped in the virtual space to achieve a 1:1 three-dimensional simulation model. The key technology lies in the accurate modeling and digital representation of the environment and objects.
[0003] Object-level semantic modeling can perceive objects in the environment and facilitate interaction with the environment. SLAM technology is usually used to parameterize objects. Thanks to the development of target detection algorithms, object-level semantic SLAM can obtain semantic information in the environment more quickly. Existing object-level semantic SLAM has developed to use quadratic surfaces instead of cubes and CAD models to construct objects in the environment. The advantage is that it can quickly obtain more accurate object poses as the camera perspective is updated. However, the target detection algorithm will inevitably miss and misdetect any scene, and the shapes of quadratic surfaces and cubes are too simple to distinguish various objects in the scene. There are still challenges in fully modeling the environment and realizing interaction.
[0004] The document "IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 720-728, 2022" proposes to construct a geometric semantic map of objects represented by quadratic surfaces, and for the first time realizes relocation using quadratic surfaces. However, the accuracy of its semantic information depends on the separate training of the target detection algorithm for different scenes, and cannot be directly applied to other scenes, resulting in low applicability of the algorithm. The document "IEEE International Conference on Pattern Recognition (ICPR), 3692-3698, 2022" proposes to recover super-quadratic surfaces of spatial planes and objects from 3D data, which can generate 3D model abstract scenes. However, the centroid of the super-quadratic point cloud needs to meet certain conditions, so most scenes are still represented by planes, the initialization success rate of super-quadratic surfaces is not high, and no complete SLAM algorithm is provided. Summary of the invention
[0005] The purpose of this invention is to propose a method for digitizing metaverse objects based on human-in-the-loop semantic correction and superquadric parameter separation initialization to construct a detailed geometric semantic map of the environment while improving the camera positioning accuracy.
[0006] A method for digitizing metaverse objects based on human-in-the-loop semantic correction and superquadric surface parameter separation initialization, extracting RGB images and depth images to obtain image plane features, point features and semantic features;
[0007] Manual correction of missed detections and false detections in semantic features based on human-in-the-loop semantic correction;
[0008] The feature points are combined with the image depth information to complete the map initialization, and the plane features are projected into the map to generate plane landmarks and build a local map.
[0009] The camera pose is estimated through constraints between map points, plane landmarks, point features of subsequent frames, and plane features, and the pose parameters of the superquadratic surface are obtained at the same time;
[0010] Reliable map points are selected through the depth information corresponding to the feature points in the detection box of the semantic feature, and the minimum convex hull is constructed for the reliable map points to which the object belongs. The shape parameters of the superquadratic surface are obtained by fitting the minimum convex hull. After all the parameters are obtained, the corresponding superquadratic surface landmarks are constructed in the local map to represent the object.
[0011] Use the local homography matrix to associate and fuse the correction results into the subsequent frames, associate the semantic features of the subsequent frames with the superquadric surface in the constructed local map, update the existing superquadric surface pose, and construct a superquadric surface in the map to express the unassociated ones;
[0012] The camera pose is optimized by combining map points, planar landmarks and superquadratic landmark constraints.
[0013] Furthermore, the human-in-the-loop semantic correction method is processed as follows: after the RGB image obtains the initial semantic features of the corresponding object through the YOLOv5 target detection algorithm, when the object is not detected or the label is wrong, the wrong label in the semantic feature is manually modified, and the detection box and label are supplemented for the undetected object to complete the semantic correction of the current frame.
[0014] Furthermore, the manual correction result of the k-th frame image is recorded as The semantic feature of the k-th frame image is recorded as gather and The corresponding detection box in is found by 2D-IoU intersection and union ratio; the vertices of the successfully matched detection box are the corresponding matching point pairs, then the local homography matrix H L The calculation is expressed as:
[0015] x′ i =H L x i (1)
[0016] in, and A pair from and Matching point; convert it into x′ through vector product form i ×H L x i =0, further expressed by matrix:
[0017]
[0018] in, Yes H L The jth row of , solving the local homography matrix becomes an Ah L = 0, and then find h L The least squares solution of
[0019]
[0020] Get H L After that, the position of the current corrected detection frame in the next frame is expressed as follows:
[0021]
[0022] and Together they constitute the final semantic features of the k+1th frame, and are associated with the super-quadratic surface constructed in the local map.
[0023] Furthermore, the shape parameter and pose parameter acquisition process of the super quadratic surface is as follows: according to the detection frame of the semantic feature and the reliable map point to which the object belongs, the tangent plane and the minimum convex hull of the object are respectively constructed to initialize the pose parameter and shape parameter of the super quadratic surface:
[0024] The superquadratic surface has 11 parameters, which means Λ=(θ1, θ2, θ3, t1, t2, t3, a x , a y , a z , ε1, ε2), where ε i Represents the shape parameters that control the superquadric surface, and the remaining 9 parameters are posture parameters; the default shape parameter ε i =1, in this case, the superquadric surface is consistent with the standard quadratic surface form, through the tangent relationship between the superquadric surface and the tangent plane π T Q* π=0 constructs the corresponding linear equation system:
[0025] A(π)Λ Q =0 (5)
[0026] where Λ Q =(θ1, θ2, θ3, t1, t2, t3, a x , a y , a z ) represents the position and axis length of the superquadric surface, A(π) is the coefficient matrix of the tangent plane of the superquadric surface, including the tangent plane generated by the back projection of the detection box of the semantic feature and the tangent plane composed of the map points of the minimum convex hull; Convert equation (5) into a linear least squares problem to obtain Λ Q The least squares solution of :
[0027]
[0028] In getting Λ Q After the least squares solution of i , the distance from the point of the minimum convex hull constructed by the reliable map points to which the object belongs to the superquadratic surface is expressed as follows:
[0029]
[0030] Minimize the sum of the squares of the Euclidean distances between the point and the superquadratic surface to fit the best shape and obtain the shape parameter ε i , indicating that the process is as follows:
[0031]
[0032] where Λ c ={a x , a y , a z , ε1, ε2} are the axis length and shape parameters of the superquadratic surface, c p represents a set of points on the superquadratic surface. The axis length parameter has been obtained when estimating the pose and can be directly substituted into ε i The least squares solution of .
[0033] Furthermore, the optimization of the camera pose by combining map points, plane landmarks and superquadratic surface landmark constraints is specifically to optimize the camera pose and the constructed superquadratic surface by combining geometric information composed of point features and plane features and corrected semantic feature information. The optimization factor is expressed as follows:
[0034]
[0035] Among them, T k，wis the camera pose of the kth frame of the image, Q w is the parameter of the superquadric surface, which is used to determine Q w Still keeping the parameters separate, use and They represent the shape parameters and posture parameters of the superquadric surface, π w represents the plane parameter, X w Represents a map point; The optimization is consistent with the way the superquadric surface is initialized. The constraints between the superquadric surface and its associated detection box are expressed as follows:
[0036]
[0037] in, is with The associated detection box, Represents a superquadric surface Through the transformation matrix T k，w Converted into a 2D detection frame; the superquadratic surface and the tangent plane constraint relationship are expressed as follows:
[0038]
[0039] The constraint relationship between map points and feature points is expressed as follows:
[0040]
[0041] Among them, u k is the feature point of the image. The map point is converted into a two-dimensional point through the camera pose, and the pose is optimized by minimizing the distance between the two-dimensional point and the corresponding feature point. The constraint relationship between the map plane in the map and the plane extracted from the kth frame is expressed as follows:
[0042]
[0043] Among them, π k It is the plane feature extracted from the image.
[0044] Beneficial effects of the present invention:
[0045] (1) This method proposes a human-in-the-loop semantic correction method to facilitate interactive editing and correction of semantic information, thereby providing accurate semantic information input for metaverse environment and object modeling. Unlike other visual SLAM algorithms that use target detection algorithms to obtain semantics, human-in-the-loop semantic correction can effectively reduce the error rate of semantic information in metaverse scenes.
[0046] (2) The present invention associates and fuses the correction results into subsequent frames through a local homography matrix, and proposes a lightweight and intuitive method for extending the semantic correction results, which expands the continuous range of the human-in-the-loop manual correction results in space.
[0047] (3) The present invention proposes to use the semantic information in the image and the minimum convex hull constructed by map points to separately estimate the pose parameters and shape parameters of the superquadric surface, thereby achieving fast initialization of the object represented by the superquadric surface.
[0048] (4) The present invention proposes to jointly optimize the camera by combining points, planes and superquadratic surfaces to improve positioning accuracy, fully combines semantic information and geometric information, successfully constructs a detailed geometric semantic map, and improves the accuracy of camera positioning. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 A flowchart of a method for digitizing a metaverse object based on human-in-the-loop semantic correction and superquadric surface parameter separation initialization provided in this embodiment;
[0050] Figure 2 A schematic diagram of the execution logic provided for this embodiment;
[0051] Figure 3 A schematic diagram of online correction based on human-in-the-loop semantics provided in this embodiment;
[0052] Figure 4 A schematic diagram of superquadratic surface initialization provided in this embodiment. DETAILED DESCRIPTION
[0053] This example provides a method for digitizing metaverse objects based on human-in-the-loop semantic correction and superquadric surface parameter separation initialization. Figure 1 and Figure 2 As shown, the following steps are included:
[0054] Step 1: Input RGB image and depth image;
[0055] Step 2: Extract features from the input image, correct the erroneous semantic features online through the human-in-the-loop semantic correction method, estimate the camera pose and initialize the superquadratic surface;
[0056] In some embodiments, the method specifically includes the following steps:
[0057] Step 2.1: Extract point features and object semantic features from the RGB image, and extract plane features from the depth map;
[0058] Step 2.2: If Figure 3As shown in the figure, the extracted semantic features are observed. When it is found that the superquadratic surface initialization of the corresponding object cannot be completed due to missed detection and false detection, the tracking module is manually paused, the input image is selected with the mouse, and then the corresponding label is entered in the terminal. The initial confidence value is automatically set to 1 to complete the correction of the current frame.
[0059] Step 2.3: Construct constraints based on image feature points and feature planes, calculate the matching relationship between feature points of multiple frames and the relationship between feature points and planes, and calculate the relative pose transformation between frames;
[0060] Step 2.4: After obtaining the correction result of the current frame, the semantic information of a single frame is sometimes not enough to support the initialization of the super-quadratic surface. Multiple frames of observation of consecutive frames are required to extend the semantic correction result to subsequent frames through the local homography matrix, thus extending the continuous range of the manual correction result of the human in the loop in space. The specific calculation method is as follows:
[0061] x′ i =H L x i (14)
[0062] in, and A pair from and The matching point of . By converting it into vector product form, we can get x′ i ×H L x i =0, further expressed by matrix:
[0063]
[0064] in, Yes H L The jth row of , solving the local homography matrix becomes an Ah L = 0, and then find h L The least squares solution of
[0065]
[0066] In actual calculations, there should be at least four point pairs, which means that one pair of detection boxes is sufficient to calculate H L The minimum requirement is to obtain H L After that, the position of the current corrected detection frame in the next frame is expressed as follows:
[0067]
[0068] and Together they form the detection result in the k+1th frame and are passed to the tracking module;
[0069] Step 2.5: Based on the object detection results and the map points to which the object belongs, construct the object's tangent plane and minimum convex hull to initialize the parameters of the superquadratic surface. Superquadratic surface parameters Λ=(θ1,θ2,θ3,t1,t2,t3,a x , a y , a z , ε1, ε2) are 11 in total, and are initialized by separate estimation, such as Figure 4 As shown in Figure 1, we first estimate the pose parameters and axis length parameters of the superquadric surface, and then estimate the shape parameters of the superquadric surface. i =1, then the superquadric surface is consistent with the standard quadratic surface form, through the tangent relationship between the surface and the tangent plane π T Q * π=0 constructs the corresponding linear equation system:
[0070] A(π)Λ Q =0 (18)
[0071] where Λ Q =(θ1, θ2, θ3, t1, t2, t3, a x , a y , a z , ε1, ε2) represent the position and axis length of the superquadric surface, A(π) is the coefficient matrix of the tangent plane of the superquadric surface, including the tangent plane generated by the back projection of the target detection result and the tangent plane composed of the points of the minimum convex hull. Convert it into a linear least squares problem to obtain Λ Q The least squares solution of :
[0072]
[0073] In getting Λ Q After the least squares solution of i , the distance from the point of the minimum convex hull constructed by the map point to which the object belongs to the superquadratic surface is expressed as follows:
[0074]
[0075] The best shape can be fitted by minimizing the sum of the squares of the Euclidean distances between the point and the superquadratic surface, and the shape parameter ε is obtained. i , indicating that the process is as follows:
[0076]
[0077] Among them, Λ c ={a x, a y , a z , ε1, ε2} are the axis length and shape parameters of the superquadratic surface, c P represents a set of points on the superquadratic surface. The axis length parameter has been obtained when estimating the pose and can be directly substituted into ε i The least squares solution of ;
[0078] Step 2.6: Associate the constructed superquadratic surface and the extracted plane features and point features with the corresponding landmarks in the constructed map, add feature information that is not in the map, and remove redundant features;
[0079] Step 3: Combine points, planes and superquadratic surfaces to jointly optimize the camera to improve positioning accuracy. The proposed optimization factors are expressed as follows:
[0080]
[0081] Among them, T k，w is the camera pose of the kth frame of the image, Q w is the parameter of the superquadric surface, which is used to determine Q w Still keeping the parameters separate, use and They represent the shape parameters and posture parameters of the superquadric surface, π w represents the plane parameter, X w Represents a map point. The optimization is consistent with the initialization method. The constraints between the superquadratic surface and the detection box are expressed as follows:
[0082]
[0083] in, is with The associated detection box, Represents a superquadric surface Through the transformation matrix T k，w Converted into a 2D detection box. The super quadratic surface and the tangent plane constraint relationship are expressed as follows:
[0084]
[0085] The constraint relationship between map points and feature points is expressed as follows:
[0086]
[0087] Among them, u k is the feature point of the image, and the pose is optimized by minimizing the distance between the projection of the 3D map point and the feature point. The constraint relationship between the map plane in the map and the plane extracted from the kth frame is expressed as follows:
[0088]
[0089] Among them, π k is the plane feature extracted from the image;
[0090] Step 4: Output a detailed geometric semantic map consisting of planes, superquadratic surfaces, and map points.
[0091] In order to verify the effectiveness of this method in visual positioning progress, the present invention conducted a positioning test on the TUM RGB-D dataset. According to the experimental test results, the average positioning error of the system is 0.0105m. The system proposed in the present invention can obtain high-precision visual positioning pose in real time.
[0092] Table 1 shows the absolute trajectory error (ATE) of the positioning result of the method of the present invention, the unit is meter.
[0093]
Claims
1. A method for digitizing metaverse objects based on human-in-the-loop semantic correction and superquadric surface parameter separation initialization, characterized in that: Extract RGB images and depth images to obtain image plane features, point features, and semantic features; Manual correction of missed detections and false detections in semantic features based on human-in-the-loop semantic correction; The feature points are combined with the image depth information to complete the map initialization, and the plane features are projected into the map to generate plane landmarks and build a local map. The camera pose is estimated through constraints between map points, plane landmarks, point features of subsequent frames, and plane features, and the pose parameters of the superquadratic surface are obtained at the same time; Reliable map points are selected through the depth information corresponding to the feature points in the detection box of the semantic feature, and the minimum convex hull is constructed for the reliable map points to which the object belongs. The shape parameters of the superquadratic surface are obtained by fitting the minimum convex hull. After all the parameters are obtained, the corresponding superquadratic surface landmarks are constructed in the local map to represent the object. Use the local homography matrix to associate and fuse the correction results into the subsequent frames, associate the semantic features of the subsequent frames with the superquadric surface in the constructed local map, update the existing superquadric surface pose, and construct a superquadric surface in the map to express the unassociated ones; The camera pose is optimized by combining map points, planar landmarks and superquadratic landmark constraints.
2. The method for digitizing metaverse objects based on human-in-the-loop semantic correction and superquadric surface parameter separation initialization according to claim 1, characterized in that: The human-in-the-loop semantic correction method is processed as follows: after the RGB image obtains the initial semantic features of the corresponding object through the YOLOv5 target detection algorithm, when the object is not detected or the label is wrong, the wrong label in the semantic feature is manually modified, and the detection frame and label are supplemented for the undetected object to complete the semantic correction of the current frame.
3. The method for digitizing metaverse objects based on human-in-the-loop semantic correction and superquadric surface parameter separation initialization according to claim 1, characterized in that: The manual correction result of the k-th frame image is recorded as The semantic feature of the k-th frame image is recorded as gather and The corresponding detection box in is found by 2D-IoU intersection and union ratio; the vertices of the successfully matched detection box are the corresponding matching point pairs, then the local homography matrix H L The calculation is expressed as: x′ i =H L x i (1) in, and A pair from and Matching point; convert it into x′ through vector product form i ×H L x i =0, further expressed by matrix: in, Yes H L The jth row of , solving the local homography matrix becomes an Ah L = 0, and then find h L The least squares solution of Get H L After that, the position of the current corrected detection frame in the next frame is expressed as follows: and Together they constitute the final semantic features of the k+1th frame, and are associated with the super-quadratic surface constructed in the local map.
4. The method for digitizing metaverse objects based on human-in-the-loop semantic correction and superquadric surface parameter separation initialization according to claim 1, characterized in that: The process of obtaining the shape parameters and pose parameters of the super quadratic surface is as follows: according to the detection frame of the semantic feature and the reliable map points to which the object belongs, the tangent plane and the minimum convex hull of the object are respectively constructed to initialize the pose parameters and shape parameters of the super quadratic surface: The superquadratic surface has 11 parameters, which means Λ=(θ1, θ2, θ3, t1, t2, t3, a x , a y , a z , ε1, ε2), where ε i Represents the shape parameters that control the superquadric surface, and the remaining 9 parameters are posture parameters; the default shape parameter ε i =1, in this case, the superquadric surface is consistent with the standard quadratic surface form, through the tangent relationship between the superquadric surface and the tangent plane π T Q * π=0 constructs the corresponding linear equation system: A(p)A Q =0 (5) where Λ Q =(θ1, θ2, θ3, t1, t2, t3, a x , a y , a z ) represents the position and axis length of the superquadric surface, A(π) is the coefficient matrix of the tangent plane of the superquadric surface, including the tangent plane generated by the back projection of the detection box of the semantic feature and the tangent plane composed of the map points of the minimum convex hull; Convert equation (5) into a linear least squares problem to obtain Λ Q The least squares solution of : In getting Λ Q After the least squares solution of i , the distance from the point of the minimum convex hull constructed by the reliable map points to which the object belongs to the superquadratic surface is expressed as follows: Minimize the sum of the squares of the Euclidean distances between the point and the superquadratic surface to fit the best shape and obtain the shape parameter ε i , indicating that the process is as follows: where Λ c ={a x , a y , a z , ε1, ε2} are the axis length and shape parameters of the superquadratic surface, c p represents a set of points on the superquadratic surface. The axis length parameter has been obtained when estimating the pose and can be directly substituted into ε i The least squares solution of .
5. The method for digitizing metaverse objects based on human-in-the-loop semantic correction and superquadric surface parameter separation initialization according to claim 1, characterized in that: The optimization of the camera pose by combining map points, plane landmarks and superquadratic surface landmark constraints is specifically to optimize the camera pose and the constructed superquadratic surface by combining the geometric information composed of point features and plane features and the corrected semantic feature information. The optimization factor is expressed as follows: Among them, T k，w is the camera pose of the kth frame of the image, Q w is the parameter of the superquadric surface, which is used to determine Q w Still keeping the parameters separate, use and They represent the shape parameters and posture parameters of the superquadric surface, π w represents the plane parameter, X w Represents a map point; The optimization is consistent with the way the superquadric surface is initialized. The constraints between the superquadric surface and its associated detection box are expressed as follows: in, is with The associated detection box, Represents a superquadric surface Through the transformation matrix T k，w Converted into a 2D detection frame; the superquadratic surface and the tangent plane constraint relationship are expressed as follows: The constraint relationship between map points and feature points is expressed as follows: Among them, u k is the feature point of the image. The map point is converted into a two-dimensional point through the camera pose, and the pose is optimized by minimizing the distance between the two-dimensional point and the corresponding feature point. The constraint relationship between the map plane in the map and the plane extracted from the kth frame is expressed as follows: Among them, π k It is the plane feature extracted from the image.