Virtual object matching method and device in complex environment
By acquiring sparse 3D point cloud and depth image information, combining bilateral filters and Gaussian weighting functions, optimizing camera pose, and using the Hough voting algorithm to match virtual objects with real scenes, the matching problem in complex scenes and under occlusion conditions is solved, achieving higher stability and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- THE 20TH RESEARCH INSTITUTE OF CHINA ELECTRONICS TECHNOLOGY GROUP CORP
- Filing Date
- 2022-11-28
- Publication Date
- 2026-05-19
AI Technical Summary
Existing markerless augmented reality technologies cannot guarantee the real-time performance, stability, and accuracy of matching virtual and real objects in complex scenes and under occlusion conditions.
By acquiring sparse 3D point cloud and depth image information, combining bilateral filters and improved Gaussian weighting functions, calculating normal vectors and feature descriptors, optimizing camera pose using Bundle Adjustment, and performing transform estimation using the Hough voting algorithm, the matching of virtual objects and real scenes is achieved.
It improves the stability, accuracy, and real-time performance of matching virtual and real objects in markerless augmented reality, and adapts to matching effects in complex scenes and under occlusion conditions.
Smart Images
Figure CN115775307B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer augmented reality technology, and in particular to a method and device for matching virtual objects in complex environments. Background Technology
[0002] In augmented reality technology, the research on target recognition and matching methods in complex scenes and under occlusion has always been a difficult and hot topic.
[0003] Tejani et al. proposed a method for 3D object detection and pose estimation in complex and occluded scenes (Tejani A, Tang D, Kouskouridas R, et al. Latent-class hough forests for 3D object detection and pose estimation[C] / / European Conference on Computer Vision(ECCV),2014,Springer.). The principle is to introduce a scale-invariant descriptor into the template-matching feature LINEMOD and integrate it into a regression forest using a template-based splitting function. During training, the class distribution at the leaf nodes is used as a latent variable, and the distribution is iteratively updated during inference to obtain estimates of background clutter and foreground occlusion.
[0004] Guo et al. designed a real-time system for handling occlusion (Guo X, Wang C, Qi Y. Real-time augmented reality with occlusion handling based on RGBD images[C] / / International Conference on Virtual Reality and Visualization (ICVRV), 2017, IEEE.). The system works by first tracking the camera in real time, updating and visualizing the reconstruction results during the scanning process, then handling the occlusion between virtual objects and the real scene based on the camera pose, and finally fusing the rendered results with the image.
[0005] Existing label-free augmented reality technologies suffer from a lack of markers for rapid matrix calculations, resulting in limitations in real-time performance and stability when matching virtual and real-world objects. Furthermore, these methods struggle to handle complex scenes and partial object occlusion, significantly reducing matching accuracy. Summary of the Invention
[0006] This application provides a method and device for matching virtual objects in complex environments, which improves the stability, accuracy and real-time performance of matching virtual and real objects in unmarked augmented reality.
[0007] This application provides a method for matching virtual objects in a complex environment, including the following steps:
[0008] The sparse 3D point cloud of the target is determined based on the camera pose, and the depth image (RGB-D) information of the target is obtained.
[0009] By combining RGB-D information with the sparse 3D point cloud, a dense 3D point cloud is obtained;
[0010] Using a specified bilateral filter, outliers in the dense 3D point cloud are removed, and a modified Gaussian weighting function is used to weight neighboring points to calculate the normal vectors of the virtual object corresponding to the target and each point in the real scene. The modified Gaussian weighting function is used to define that the closer the point is to the current point, the greater its influence on the estimation result of the current point's normal vector.
[0011] Based on the dense 3D point cloud and the calculated normal vectors of each point, sampling is performed to obtain key points:
[0012] Based on the key points obtained from point cloud sampling, feature descriptors for virtual models and real scenes are calculated to estimate the correspondence between the two point clouds.
[0013] Transformation estimation is performed based on the correspondence to achieve matching between virtual objects and real scenes.
[0014] Optionally, before determining the sparse 3D point cloud of the target, the virtual object matching method further includes:
[0015] Calculate the degree of adaptation of different camera pose models in the actual scene to determine the camera pose model, which satisfies:
[0016]
[0017]
[0018] Where H represents the use of a homography matrix in a planar scene, F represents the use of a fundamental matrix in a non-planar scene, and M takes the values of H and F respectively, d rc and d cr T represents the measure of symmetry propagation error. M Based on χ 2 Outlier rejection threshold for the distribution, α equals T M x c and x r These are the features of the current frame and the reference frame, respectively;
[0019] The determined camera pose is optimized using Bundle Adjustment to satisfy:
[0020]
[0021] Among them, X i Represents a point in three-dimensional space. Let r represent the covariance matrix associated with the keypoints, r and t be the rotation matrix and translation vector, respectively, and π represent the projection function.
[0022] Optionally, combining the RGB-D information with the sparse 3D point cloud to obtain a dense 3D point cloud includes:
[0023] Only in keyframes, the dense 3D point cloud is calculated by combining the RGB-D information with the sparse 3D point cloud, wherein if the proportion of points in the current frame that are contained in the previous keyframe is lower than a preset proportion threshold, the keyframe is updated to the current frame.
[0024] Optionally, the specified bilateral filter satisfies:
[0025]
[0026] Where p i p represents the original point in the point cloud. j For p i Points in the neighborhood, n i and n j They are p i and p j The normal vector, and These are used to control the smoothness and feature retention, respectively.
[0027] Optionally, a modified Gaussian weighting function can be used to weight the neighboring points to satisfy the following:
[0028]
[0029] Where n is the normal vector of the current point p, p i It is a point in the neighborhood of point p, θ(x) = x -r This represents the Gaussian weight.
[0030] Optionally, the calculated feature descriptor is a binary orientation histogram.
[0031] Optionally, performing the transformation estimation based on the correspondence includes:
[0032] The Hough voting algorithm is used to identify the transformation matrix between the local coordinate system of the 3D object computational model and the scene's global coordinate system.
[0033] The transformation between virtual objects and real-world scenes is calculated using point-to-point error metrics to obtain rotation and translation matrices. An iterative nearest-point algorithm is then introduced to determine the optimal transformation, thus completing the transformation estimation.
[0034] This application also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the aforementioned virtual object matching method in a complex environment.
[0035] This application also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned virtual object matching method in a complex environment.
[0036] The virtual object matching method proposed in this application takes into account the matching accuracy and efficiency between real and virtual objects in complex scenes and under occlusion conditions, and improves the stability, accuracy and real-time performance of matching virtual and real objects in unmarked augmented reality.
[0037] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description
[0038] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0039] Figure 1 An example of an augmented reality framework;
[0040] Figure 2 This is an example of the virtual object matching method flow in an embodiment of this application;
[0041] Figure 3 This is an example of the overall process of the virtual object matching method in this application embodiment. Detailed Implementation
[0042] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0043] A complete augmented reality framework mainly consists of a SLAM module and a registration module, such as Figure 1 As shown, the main function of the SLAM module is to estimate the camera's position and pose by processing each frame of image. First, it initializes the feature matching process using the previous frame and then uses the Bundle Adjustment method to solve for the optimal camera pose. In the registration module, RGB-D data is added, and a dense 3D model is generated by fusing the point cloud based on the camera pose calculated in the SLAM module. Then, the 3D model is used to identify targets and obtain a transformation matrix, which maps the coordinates of the 3D model to each frame of image. Finally, the 3D virtual object is registered in the real scene to achieve augmented reality effects.
[0044] Based on this, embodiments of this application provide a method for matching virtual objects in complex environments, such as... Figure 2 As shown, it includes the following steps:
[0045] In step S201, a sparse 3D point cloud of the target is determined based on the camera pose, and the depth image (RGB-D) information of the target is acquired. Specifically, in this embodiment, the sparse 3D point cloud cannot guarantee matching accuracy. To improve matching accuracy, this application acquires RGB-D information using Kinect and combines it with the sparse point cloud to obtain a dense 3D point cloud. Specifically, in some embodiments, combining the RGB-D information with the sparse 3D point cloud to obtain a dense 3D point cloud includes: calculating the dense 3D point cloud only in keyframes by combining the RGB-D information with the sparse 3D point cloud, wherein if the proportion of points in the current frame that are present in the previous keyframe is lower than a preset proportion threshold, the keyframe is updated to the current frame. The specific preset proportion threshold can be 75% or any specified value.
[0046] In step S202, the RGB-D information is combined with the sparse three-dimensional point cloud to obtain a dense three-dimensional point cloud;
[0047] After obtaining the 3D point cloud of an object, target recognition and matching are required. Existing methods cannot effectively handle complex scenes and occlusion situations. This application proposes a target recognition and matching method for complex scenes and occlusion situations, such as... Figure 3As shown, in step S203, outliers in the dense 3D point cloud are removed using a specified bilateral filter, and a modified Gaussian weighting function is used to weight neighboring points to calculate the normal vectors of the virtual object corresponding to the target and each point in the real scene. The modified Gaussian weighting function defines that points closer to the current point have a greater influence on the estimation result of the current point's normal vector. In this example, the modified Gaussian weighting function ensures the consistency of the local coordinate system, enabling repeated detection of subsequent feature descriptors and ensuring that descriptors can be detected in complex and occluded scenes.
[0048] In step S204, key points are obtained by sampling based on the dense 3D point cloud and the calculated normal vectors of each point. Specifically, an efficient uniform sampling method can be used to select key points in the point cloud and calculate feature descriptors for the key points. In some embodiments, the calculated feature descriptors are binary orientation histograms.
[0049] In step S205, feature descriptors for the virtual model and the real scene are calculated based on the key points obtained from point cloud sampling to estimate the correspondence between the two point clouds. Specifically, erroneous correspondences can be rejected by finding a consistent set to reduce the number of outliers.
[0050] In step S206, transformation estimation is performed based on the correspondence to achieve matching between virtual objects and real scenes.
[0051] The virtual object matching method proposed in this application takes into account the matching accuracy and efficiency between real and virtual objects in complex scenes and under occlusion conditions, and improves the stability, accuracy and real-time performance of matching virtual and real objects in unmarked augmented reality.
[0052] To adapt to different scenarios, embodiments of this application calculate the camera pose using two separate models: a homography matrix H for planar scenes and a fundamental matrix F for non-planar scenes. Specifically, in some embodiments, before determining the sparse 3D point cloud of the target, the virtual object matching method further includes:
[0053] Calculate the degree of adaptation of different camera pose models in the actual scene to determine the camera pose model, which satisfies:
[0054]
[0055]
[0056] Where H represents the use of a homography matrix in a planar scene, F represents the use of a fundamental matrix in a non-planar scene, and M takes the values of H and F respectively, d rc and d cr T represents the measure of symmetry propagation error.M Based on χ 2 Outlier rejection threshold for the distribution, α equals T M x c and x r These are the features of the current frame and the reference frame, respectively. The scores S of the two matrices are calculated separately, and a pose model that is more suitable for the current camera pose is determined.
[0057] By optimizing the determined camera pose using Bundle Adjustment, a more accurate camera pose is obtained, satisfying the following:
[0058]
[0059] Among them, X i Represents a point in three-dimensional space. Let r represent the covariance matrix associated with the keypoints, r and t be the rotation matrix and translation vector, respectively, and π represent the projection function. After obtaining the accurate camera pose, sparse 3D point clouds are obtained through triangulation.
[0060] This application uses an improved bilateral filter to remove outliers from the input point cloud while smoothing neighboring pixels without affecting edges. To enhance feature preservation and improve filtering performance, in some embodiments, the specified bilateral filter satisfies the following:
[0061]
[0062] Where p i p represents the original point in the point cloud. j For p i Points in the neighborhood, n i and n j They are p i and p j The normal vector, and These are used to control the smoothness and feature retention, respectively.
[0063] In some embodiments, a modified Gaussian weighting function is used to weight neighboring points to satisfy the following:
[0064]
[0065] Where n is the normal vector of the current point p, p i It is a point in the neighborhood of point p, θ(x) = x -rThis represents the Gaussian weight. In the specific example, r = 0.55, but it can also be any other specified value. When calculating the normal vector of a point cloud, the influence of distance on the result needs to be considered. The Gaussian weighting function proposed in this embodiment defines that points closer to the current point have a greater influence on the estimation result of the current point's normal vector.
[0066] Specific examples Figure 3 As shown, in step S204, key points are obtained by sampling based on the dense 3D point cloud and the calculated normal vectors of each point. Specifically, an efficient uniform sampling method can be used to select key points in the point cloud and calculate the feature descriptors of the key points. In this example, a binary orientation histogram is selected as the feature descriptor of the key points. In addition, the Gaussian weighting function designed in this application can ensure the consistency of the local coordinate system, followed by correspondence estimation, object recognition, and transformation estimation steps. In some embodiments, completing the transformation estimation based on the correspondence includes:
[0067] The Hough voting algorithm is used to identify the transformation matrix between the local coordinate system of the 3D object calculation model and the global coordinate system of the scene. Specifically, the search radius for Hough voting can be set to 0.015, and the threshold can be set to 2.0.
[0068] The transformation between virtual objects and real-world scenes is calculated by point-to-point error measurement to obtain the rotation matrix R and translation matrix T. An iterative nearest-point algorithm is then introduced to determine the optimal transformation, thereby achieving the matching between virtual objects and real-world scenes.
[0069] Next, virtual objects are registered in the real-world scene, and augmented reality images are generated. This entire process involves converting the world coordinate system to the image coordinate system.
[0070]
[0071] Where u and v represent the coordinates of the center point on the screen, (f x f y (x, y, z) represents the focal length, (u0, v0) represents the point in the camera pixel coordinate system, (X, Y, Z) represents the coordinates of the point in the real scene, and R and T represent the calculated rotation matrix and translation matrix, respectively, thus completing the transformation estimation.
[0072] The method proposed in this application proposes a Gaussian weighting function, which allows for repeated detection of 3D point cloud descriptors, improving computational efficiency and reducing the impact of complex and occluded scenes. This application designs a novel bilateral filtering factor to enhance feature preservation and noise resistance, thereby improving filtering performance. The object recognition and matching algorithm proposed in this application improves the accuracy of model and scene transformation, and establishes virtual-real object fusion through point cloud fusion.
[0073] This application also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the aforementioned virtual object matching method in a complex environment.
[0074] This application also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned virtual object matching method in a complex environment.
[0075] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0076] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0078] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.
Claims
1. A method for matching virtual objects in complex environments, characterized in that, Includes the following steps: The sparse 3D point cloud of the target is determined based on the camera pose, and the depth image RGB-D information of the target is obtained. By combining RGB-D information with the sparse 3D point cloud, a dense 3D point cloud is obtained; Using a specified bilateral filter, outliers in the dense 3D point cloud are removed, and a modified Gaussian weighting function is used to weight neighboring points to calculate the normal vectors of the virtual object corresponding to the target and each point in the real scene. The modified Gaussian weighting function is used to define that the closer the point is to the current point, the greater its influence on the estimation result of the current point's normal vector. Based on the dense 3D point cloud and the calculated normal vectors of each point, sampling is performed to obtain key points; Based on the key points obtained from point cloud sampling, feature descriptors for virtual models and real scenes are calculated to estimate the correspondence between the two point clouds. Transformation estimation is performed based on the correspondence to achieve matching between virtual objects and real scenes; Before determining the sparse 3D point cloud of the target, the virtual object matching method further includes: Calculate the degree of adaptation of different camera pose models in the actual scene to determine the camera pose model, which satisfies: in, This indicates the use of a homography matrix in a planar scene. This indicates the use of the fundamental matrix in a non-planar scene. Take respectively and , and A measure of symmetry propagation error. Based on Outlier rejection threshold of the distribution equal , and These are the features of the current frame and the reference frame, respectively; The determined camera pose is optimized using Bundle Adjustment to satisfy: in, Represents a point in three-dimensional space. This represents the covariance matrix associated with the keypoints, where r and t are the rotation matrix and translation vector, respectively. This represents the projection function.
2. The virtual object matching method in complex environments as described in claim 1, characterized in that, Combining the RGB-D information with the sparse 3D point cloud to obtain a dense 3D point cloud includes: Only in keyframes, the dense 3D point cloud is calculated by combining the RGB-D information with the sparse 3D point cloud, wherein if the proportion of points in the current frame that are contained in the previous keyframe is lower than a preset proportion threshold, the keyframe is updated to the current frame.
3. The virtual object matching method in complex environments as described in claim 1, characterized in that, The specified bilateral filter satisfies: Where p i p represents the original point in the point cloud. j For p i Points in the neighborhood, n i and n j They are p i and p j The normal vector, and These are used to control the smoothness and feature retention, respectively.
4. The virtual object matching method in complex environments as described in claim 3, characterized in that, Using an improved Gaussian weighting function to weight neighboring points satisfies: in, The current point The normal vector, It is a point Points in the neighborhood, This represents the Gaussian weight.
5. The virtual object matching method in complex environments as described in claim 1, characterized in that, The calculated feature descriptor is a binary orientation histogram.
6. The virtual object matching method in complex environments as described in claim 1, characterized in that, The transformation estimation based on the aforementioned correspondence includes: The Hough voting algorithm is used to identify the transformation matrix between the local coordinate system of the 3D object computational model and the scene's global coordinate system. The transformation between virtual objects and real-world scenes is calculated using point-to-point error metrics to obtain rotation and translation matrices. An iterative nearest-point algorithm is then introduced to determine the optimal transformation, thus completing the transformation estimation.
7. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the steps of the virtual object matching method in a complex environment as described in any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the virtual object matching method in a complex environment as described in any one of claims 1 to 6.