Real-time mixed reality visual angle transformation method and system
By generating RGB-D point clouds and using point cloud sculpting network models for perspective transformation, the quality problem of perspective synthesis in mixed reality was solved, achieving efficient and accurate stereoscopic view synthesis and improving the display effect of mixed reality devices.
Patent Information
- Application Number
- CN202511130325.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-13
- Publication Date
- 2025-11-28
AI Technical Summary
Existing technologies in mixed reality suffer from problems such as coarse details in projected point clouds, unclear background and foreground, outliers and noise, resulting in low quality of viewpoint synthesis and failing to meet the requirements of real-time performance and robustness.
By acquiring the depth information of the target image, an RGB-D point cloud is generated. The point cloud sculpting network model is then used for projection and sculpting to supplement the defective point cloud. Finally, a feature aggregation network is used to generate the target image from the user's perspective, achieving high-quality viewpoint synthesis.
It achieves efficient and high-quality real-time stereoscopic view synthesis in mixed reality devices, solves problems such as coarse details in projected point clouds, unclear background and foreground, outliers and noise, and improves the accuracy and quality of view synthesis.
Smart Images
Figure CN121033338A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of image processing and virtual reality technology, in particular to a real-time mixed reality view transformation method and system. BACKGROUND
[0002] Mixed reality (MR) is a deep integration of virtual reality and augmented reality technologies, and its core goal is to achieve real-time and natural interaction between computer-generated virtual objects and real environments.
[0003] As a carrier of MR technology, the head-mounted display device is the key to constructing the virtual-real integration experience, involving multiple technical dimensions such as multi-sensor fusion, spatial positioning, real-time rendering, and interactive feedback.
[0004] Real-time view synthesis is one of the key technologies for wide field of view display images and virtual-real integration. Early researches regarded view synthesis as an interpolation problem, but it required dense input views, which was not suitable for the virtual-real integration effect of mixed reality. Passthrough+ is the only method in this type that solves this problem, but the image quality is limited. With the rapid development of learning-based view synthesis methods, it can be divided into single view, stereo view, multi-view, and multi-view video. Recent important researches are related to neural radiance fields (NeRF).
[0005] NeRF represents a scene as an implicit function encoded by a multilayer perceptron (MLP). The NeRF-inspired approach has led to rapid development of view synthesis. However, these latest researches do not directly address the unique challenges of virtual-real integration in mixed reality. Specifically, many NeRF-derived methods rely on a dense set of views. More importantly, these methods produce high computational costs in scene optimization and volume rendering. Although some recent work has been done to reduce complexity, NeRF-based methods still seem unsuitable for virtual-real integration. Similarly, other works have tried to reduce the number and range of input views to support dynamic scenes and make them more versatile, but these researches have not met the key requirements of robustness and real-time performance. SUMMARY
[0006] To solve the problems existing in the prior art, the purpose of the present application is to provide a real-time mixed reality view transformation method and system, which will effectively solve the problems of rough details of projected point clouds, unclear background and foreground, existence of abnormal points and noise, etc., and realize more accurate and high-quality view synthesis results.
[0007] To achieve the above purpose, the present application provides the following scheme:
[0008] A real-time mixed reality perspective transformation method, comprising:
[0009] Obtaining a target image, estimating depth information corresponding to the target image, preprocessing the depth information, and obtaining an RGB-D point cloud;
[0010] According to the positional change relationship between the input perspective and the user perspective, projecting the RGB-D point cloud to the target image;
[0011] Inputting the projected target image into a point cloud carving network model, obtaining a carved point cloud, supplementing defective point clouds in the perspective change, and obtaining a final point cloud;
[0012] Reprojecting the final point cloud to a two-dimensional image, and obtaining a target image under the user perspective using the two-dimensional image.
[0013] Optionally, obtaining the RGB-D point cloud comprises:
[0014] Obtaining an edge part with discontinuous depth in the depth information, replacing the depth value in the edge part with the depth value of the closest non-edge pixel in the edge part, and smoothing and sharpening the replaced depth information.
[0015] Optionally, obtaining the carved point cloud comprises:
[0016] Inputting the projected target image into a point cloud carving network model, merging the different perspective depth maps into a dense point cloud, and removing outliers using the projection consistency of pixel points in the depth map in different position views.
[0017] In the process of merging into a dense point cloud, forward projection is used to project the pixel points of the current depth map into other depth maps, and a machine learning hyperparameter intensity threshold is set to maximize the preservation of the point cloud.
[0018] Depth sharpening and point cloud carving are performed on the dense point cloud to obtain the carved point cloud.
[0019] Optionally, obtaining the final point cloud comprises:
[0020] Extracting the pixel points in the depth map, if the rendering error of the corresponding position of the pixel point is greater than the machine learning hyperparameter threshold, determining a pixel point set along the pixel ray from the depth map, adding the pixel point set to the carved point cloud, and obtaining the final point cloud.
[0021] Optionally, obtaining the target image under the user perspective comprises:
[0022] Input a two-dimensional image into a feature aggregation network model to obtain a target image under the user's perspective;The feature aggregation network model is trained by using a training set to train a U-Net model with residual connection to obtain;The training set comprises an original two-dimensional image.
[0023] To achieve the above object, the application further provides a real-time mixed reality perspective transformation system, comprising:
[0024] An image estimation module is configured to obtain a target image, estimate depth information corresponding to the target image, pre-process the depth information, and obtain an RGB-D point cloud.
[0025] A point cloud projection module is configured to project the RGB-D point cloud to the target image according to a positional change relationship between an input perspective and a user perspective.
[0026] A point cloud carving module is configured to input the projected target image into a point cloud carving network model, obtain carved point clouds, supplement defective point clouds in perspective change, and obtain final point clouds.
[0027] A point cloud re-projection module is configured to re-project the final point clouds to a two-dimensional image, and obtain a target image under the user's perspective by using the two-dimensional image.
[0028] The application has the following advantages:
[0029] The application calculates target images corresponding to left and right eyes under the perspective of mixed reality user eyes according to the relative position between the head-mounted camera and the eyes of the mixed reality user, and realizes efficient and high-quality real-time stereoscopic view synthesis on a mixed reality display device by using a perspective transformation algorithm. BRIEF DESCRIPTION OF DRAWINGS
[0030] To make the technical solutions of the embodiments of the application or the prior art clearer, the accompanying drawings needed in the embodiments will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor based on these drawings.
[0031] Figure 1 A flow chart of a real-time mixed reality perspective transformation method according to an embodiment of the application. DETAILED DESCRIPTION
[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0033] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0034] like Figure 1 As shown, this embodiment provides an augmented reality-based endoscopic projection method, characterized by the following steps: acquiring a target image using a front-facing camera of a mixed reality head-mounted display device; estimating the depth information corresponding to the target image using a depth estimation algorithm; smoothing and sharpening the discontinuous edge parts of the depth information and fusing it with the RGB information of the image to obtain an RGB-D point cloud; projecting the RGB-D point cloud onto the point cloud seen by the target view according to the positional transformation relationship between the input view and the target view; fusing, sculpting, and generating the point cloud information in a point cloud sculpting network, filtering out redundant erroneous information; using the interactive information of the point cloud to supplement the missing information in the view transformation to obtain the final point cloud; reprojecting the final point cloud into two dimensions and inputting it into a feature aggregation network to obtain the target image.
[0035] The specific implementation scheme of the depth sharpening algorithm used in this embodiment is as follows:
[0036] The depth estimation results contain certain depth discontinuities, which result in numerous outliers in the corresponding point cloud. This severely impacts the image quality of point cloud-based viewpoint transformations. To address this issue, a Sobel edge detector is proposed to detect outliers in the predicted depth map. For edge regions with depth discontinuities, their depth values are replaced with the depth values of the nearest non-edge pixels, smoothing and sharpening the depth estimation results. The optimized depth map improves the quality of RGB-D point cloud projection and effectively reduces outliers in the point cloud.
[0037] The specific implementation plan for the point cloud sculpting network used in this solution is as follows:
[0038] When fusing point clouds, depth maps from different perspectives are merged into a dense point cloud. The step of fusion usually adopts the projection consistency of a certain pixel in different position views to remove outliers, and when a pixel is visible in multiple views, it often needs to be calculated multiple times forward and backward, that is, projecting a pixel into other views and projecting the corresponding pixels of other views back to the original view. Such operation is easy to cause incomplete point cloud with large holes, which is not conducive to the synthesis of a small number of perspectives. Therefore, only forward projection, that is, direct projection to the target view, is considered, and a machine learning-based hyperparameter intensity threshold is used to maximize the completeness of the point cloud.
[0039] The point cloud generation method used in the present scheme has the following specific implementation:
[0040] After depth sharpening and point cloud carving, the generated point cloud often has many vacancies, therefore, the present application proposes a method of adding new points to the pruned point cloud, that is, finding a set of three-dimensional points, as long as the set of three-dimensional points is a superset of the final real point cloud, then the redundant points can be deleted through the gradient optimization of the image loss obtained by final rendering. Specifically, for a pixel μ = (u, v) of the input view, if the rendering error of the position is greater than the machine learning-based hyperparameter threshold φ, then the present application will find a possible pixel point set M (superset of the final real point cloud) along the pixel ray from all input pictures I L ,I R , and add all the points to the existing point cloud for iterative optimization to minimize the rendering error.
[0041] The feature aggregation network method used in the present scheme has the following specific implementation:
[0042] The final iterated point cloud is projected onto a two-dimensional image and input together into a U-Net with residual connection to predict the target image.
[0043] The above-described embodiments are only descriptions of the preferred modes of the present application, and do not limit the scope of the present application, and various modifications and improvements to the technical solutions of the present application made by those of ordinary skill in the art without departing from the design spirit of the present application shall fall within the protection scope determined by the claims of the present application.
Claims
1. A real-time mixed reality perspective transformation method, characterized in that, include: Acquire a target image, estimate the depth information corresponding to the target image, preprocess the depth information, and obtain an RGB-D point cloud; Based on the positional relationship between the input viewpoint and the user viewpoint, the RGB-D point cloud is projected onto the target image; The projected target image is input into the point cloud sculpting network model to obtain the sculpted point cloud, and the defect point cloud is supplemented by the viewpoint change to obtain the final point cloud. The final point cloud is reprojected onto a two-dimensional image, and the target image from the user's perspective is obtained using the two-dimensional image.
2. The real-time mixed reality perspective transformation method according to claim 1, characterized in that, Obtaining the RGB-D point cloud includes: Obtain the edge portion of the depth information where the depth is discontinuous, replace the depth value in the edge portion with the depth value of the closest non-edge pixel in the edge portion, and smooth and sharpen the replaced depth information.
3. The real-time mixed reality perspective transformation method according to claim 1, characterized in that, Obtaining the sculpted point cloud includes: The projected target image is input into the point cloud sculpting network model. The projected target image, i.e., depth maps from different viewpoints, is merged into a dense point cloud. Outliers are removed by using the projection consistency of pixels in the depth map in different position views. During the process of merging into a dense point cloud, forward projection is used to project the pixels of the current depth map onto other depth maps, and the intensity threshold of the hyperparameter of machine learning is set to preserve the point cloud to the maximum extent. The dense point cloud is subjected to depth sharpening and point cloud sculpting to obtain the sculpted point cloud.
4. The real-time mixed reality perspective transformation method according to claim 3, characterized in that, Obtaining the final point cloud includes: Pixels are extracted from the depth map. If the rendering error at the position corresponding to the pixel is greater than the hyperparameter threshold of machine learning, a set of pixels is determined from the depth map along the pixel ray. The set of pixels is added to the sculpted point cloud to obtain the final point cloud.
5. The real-time mixed reality perspective transformation method according to claim 1, characterized in that, Obtaining the target image from the user's perspective includes: A two-dimensional image is input into a feature aggregation network model to obtain a target image from the user's perspective; the feature aggregation network model is trained using a U-Net model with residual connections trained on a training set; the training set includes the original two-dimensional image.
6. A real-time mixed reality perspective transformation system, characterized in that, include: The image estimation module is used to acquire a target image, estimate the depth information corresponding to the target image, preprocess the depth information, and obtain an RGB-D point cloud. The point cloud projection module is used to project the RGB-D point cloud onto the target image based on the positional change relationship between the input viewpoint and the user viewpoint. The point cloud sculpting module is used to input the projected target image into the point cloud sculpting network model, obtain the sculpted point cloud, and supplement the defect point cloud during viewpoint changes to obtain the final point cloud. The point cloud reprojection module is used to reproject the final point cloud onto a two-dimensional image, and use the two-dimensional image to obtain a target image from the user's perspective.