A multi-user holographic communication viewport sharing method, system, terminal and storage medium
By generating depth maps and performing deformation distortion estimation, the problem of inaccurate deformation distortion estimation in holographic communication viewport sharing is solved, thereby improving viewport image quality and enhancing user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PENG CHENG LAB
- Filing Date
- 2026-01-28
- Publication Date
- 2026-05-29
AI Technical Summary
Existing holographic communication viewport sharing methods cannot accurately estimate the deformation and distortion during the viewport sharing process, resulting in low-quality shared viewport images that negatively impact user experience.
By acquiring viewport position information from multiple users, a depth map is generated and deformation distortion is estimated. Users are then grouped based on the estimated values, and finally, a target viewport image is generated.
It improves the quality of generated shared viewport images, enhancing the user experience for multiple users.
Smart Images
Figure CN122120425A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of communications, and more particularly to a method, system, terminal, and storage medium for sharing holographic communication viewports among multiple users. Background Technology
[0002] With the widespread application of holographic communication technology, users have an increasing demand for accessing the virtual world. When multiple users experience the same virtual space, the limited computing resources of the server usually cannot meet the real-time rendering of the viewport images of multiple users. Therefore, the viewport sharing mechanism is often used to reuse the screen content in the view of multiple users in order to reduce the computing load of the server and user terminals.
[0003] Existing viewport sharing methods often combine depth maps to back-project viewport images into 3D space to obtain traditional format point clouds, and then combine them with new viewport positions to project new viewport images. However, this viewport sharing method cannot accurately estimate the deformation and distortion generated during the viewport sharing process, and therefore cannot process the distortion according to the different distortion characteristics of users, resulting in low quality of the generated shared viewport images and affecting the user experience.
[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention
[0005] The main objective of this invention is to provide a multi-user holographic communication viewport sharing method, system, terminal, and storage medium. This invention aims to solve the problem that existing holographic communication viewport sharing methods cannot accurately estimate the deformation and distortion generated during the viewport sharing process, and therefore cannot process the distortion according to the different distortion characteristics of users, resulting in low quality of the generated shared viewport image and affecting the user experience.
[0006] To achieve the above objectives, the present invention provides a multi-user holographic communication viewport sharing method comprising the following steps: Obtain viewport position information of multiple users, and perform depth map generation processing based on the multiple viewport position information to obtain depth maps of multiple users; Based on multiple user depth maps and multiple viewport position information, deformation distortion estimation processing is performed to obtain multiple deformation distortion estimates. The users are grouped according to the multiple deformation distortion estimates to obtain user grouping results; Obtain the target user, and perform viewport image generation processing on the target user according to the user grouping results to obtain the target viewport image.
[0007] Optionally, the multi-user holographic communication viewport sharing method, wherein obtaining viewport position information of multiple users and performing depth map generation processing based on the multiple viewport position information to obtain multiple user depth maps specifically includes: Obtain viewport position information from multiple users, and perform viewport overlay processing based on the multiple viewport position information to obtain multiple local virtual spaces; Obtain an opacity threshold, and perform spatial projection processing on multiple elements in the local virtual space based on the opacity threshold to obtain multiple sets of projected pixels. Depth values are calculated for pixels in the multiple sets of projected pixels to obtain multiple target depth values, and multiple user depth maps are obtained based on the multiple target depth values.
[0008] Optionally, in the multi-user holographic communication viewport sharing method, the step of obtaining an opacity threshold and performing spatial projection processing on elements in multiple local virtual spaces based on the opacity threshold to obtain multiple sets of projected pixels specifically includes: Obtain the opacity threshold and determine whether the elements in the multiple local virtual spaces have a fixed opacity; If yes, then delete the elements whose opacity is less than the opacity threshold from among the multiple elements; if no, then delete the viewpoints of the multiple elements whose opacity is less than the opacity threshold, to obtain multiple target element sets. Projection processing is performed on multiple sets of target elements to obtain multiple sets of projected pixels.
[0009] Optionally, the multi-user holographic communication viewport sharing method further includes, after performing depth value calculation processing on pixels in the multiple sets of projected pixels to obtain multiple target depth values, and obtaining multiple user depth maps based on the multiple target depth values, the method further includes: A first depth map extraction process is performed on multiple local virtual spaces to obtain multiple real depth maps; Multiple update step sizes are obtained, and the opacity threshold is updated according to the multiple update step sizes to obtain multiple candidate opacity thresholds. Then, a second depth map extraction process is performed on the multiple local virtual spaces according to the multiple candidate opacity thresholds to obtain multiple estimated depth maps. A quality evaluation algorithm is determined, and the image evaluation algorithm is used to perform quality evaluation processing on multiple estimated depth maps based on multiple real depth maps to obtain multiple quality scores; The multiple candidate opacity thresholds are filtered based on the multiple quality scores to obtain an adaptive opacity threshold.
[0010] Optionally, the multi-user holographic communication viewport sharing method, wherein the step of performing deformation distortion estimation processing based on multiple user depth maps and multiple viewport position information to obtain multiple deformation distortion estimates specifically includes: Based on multiple user depth maps and multiple viewport position information, parameter calculations are performed to obtain the distance between multiple users, the angle between multiple viewport directions, and the overlap rate of multiple viewports. A neural network is determined, and through the neural network, deformation distortion calculation is performed based on multiple user depth maps, multiple user distances, multiple viewport direction angles, and multiple viewport overlap rates to obtain multiple deformation distortion estimates.
[0011] Optionally, in the multi-user holographic communication viewport sharing method, the step of grouping multiple users based on multiple distortion estimates to obtain user grouping results specifically includes: Multiple clustering points are obtained based on multiple users, and multiple clustering distances are obtained based on multiple deformation distortion estimates; A clustering algorithm is determined, and the clustering algorithm is used to cluster multiple cluster points based on multiple cluster distances to obtain user grouping results.
[0012] Optionally, the multi-user holographic communication viewport sharing method, wherein obtaining the target user and performing viewport image generation processing on the target user according to the user grouping result to obtain the target viewport image specifically includes: Obtain the target user, and perform viewport image transmission processing on the target user according to the user grouping result to obtain the reference viewport image; Obtain the real-time viewport position of the target user, and perform deformation processing on the reference viewport image based on the real-time viewport position to obtain the target viewport image.
[0013] Furthermore, to achieve the above objectives, the present invention also provides a multi-user holographic communication viewport sharing system, wherein the multi-user holographic communication viewport sharing system includes: The depth map generation module is used to acquire viewport position information of multiple users, and perform depth map generation processing based on the multiple viewport position information to obtain depth maps of multiple users. The deformation distortion estimation module is used to perform deformation distortion estimation processing based on multiple user depth maps and multiple viewport position information to obtain multiple deformation distortion estimation values. The user grouping module is used to group multiple users according to multiple deformation distortion estimates to obtain user grouping results. The viewport generation module is used to acquire target users and perform viewport image generation processing on the target users according to the user grouping results to obtain target viewport images.
[0014] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a multi-user holographic communication viewport sharing program stored in the memory and executable on the processor, wherein when the multi-user holographic communication viewport sharing program is executed by the processor, it implements the steps of the multi-user holographic communication viewport sharing method as described above.
[0015] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multi-user holographic communication viewport sharing program, which, when executed by a processor, implements the steps of the multi-user holographic communication viewport sharing method as described above.
[0016] In this invention, viewport position information of multiple users is acquired, and depth map generation processing is performed based on the multiple viewport position information to obtain multiple user depth maps. Deformation distortion estimation processing is performed based on the multiple user depth maps and the multiple viewport position information to obtain multiple deformation distortion estimates. The multiple users are grouped based on the multiple deformation distortion estimates to obtain user grouping results. A target user is acquired, and viewport image generation processing is performed on the target user based on the user grouping results to obtain a target viewport image. This invention performs accurate deformation distortion estimation based on user depth maps, groups users based on deformation distortion estimates between every two users, and generates viewport images based on the user grouping results, thereby improving the generation quality of shared viewport images. Attached Figure Description
[0017] Figure 1 This is a flowchart of a preferred embodiment of the multi-user holographic communication viewport sharing method of the present invention; Figure 2 This is a schematic diagram of the viewport sharing mechanism of the multi-user holographic communication viewport sharing method of the present invention; Figure 3 This is a system overall flowchart of the multi-user holographic communication viewport sharing method of the present invention; Figure 4 This is a flowchart of the depth map generation process for the multi-user holographic communication viewport sharing method of the present invention; Figure 5 This is a comparison of the actual depth map and the estimated depth map of the multi-user holographic communication viewport sharing method of the present invention; Figure 6 This is a flowchart of the deformation distortion estimation method for the multi-user holographic communication viewport sharing method of the present invention; Figure 7 This is a structural diagram of a preferred embodiment of the multi-user holographic communication viewport sharing system of the present invention; Figure 8 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0019] With the widespread application of holographic communication technology, users have an increasing demand for accessing the virtual world. When multiple users experience the same virtual space, the limited computing resources of the server usually cannot meet the real-time rendering of the viewport images of multiple users. Therefore, the viewport sharing mechanism is often used to reuse the screen content in the view of multiple users in order to reduce the computing load of the server and user terminals.
[0020] Existing viewport sharing methods often combine depth maps to back-project viewport images into 3D space to obtain traditional format point clouds, and then combine them with new viewport positions to project new viewport images. However, this viewport sharing method cannot accurately estimate the deformation and distortion generated during the viewport sharing process, and therefore cannot process the distortion according to the different distortion characteristics of users, resulting in low quality of the generated shared viewport images and affecting the user experience.
[0021] To address the aforementioned issues, this invention proposes a multi-user holographic communication viewport sharing method. This method performs accurate deformation distortion estimation based on user depth maps, groups users according to the deformation distortion estimates between every two users, and generates viewport images based on the user grouping results, thereby improving the generation quality of shared viewport images.
[0022] The preferred embodiment of the multi-user holographic communication viewport sharing method of the present invention, such as... Figure 1 As shown, the multi-user holographic communication viewport sharing method includes the following steps: Step S10: Obtain viewport position information of multiple users, and perform depth map generation processing based on the multiple viewport position information to obtain multiple user depth maps.
[0023] Viewport sharing mechanisms face key challenges in practical applications. When a shared viewport is mapped to the receiving end for display, geometric correction of the shared content is required using image distortion techniques to adapt to differences in the position and viewing angle of the receiving end's viewport. This process inevitably introduces distortion problems such as pixel loss and blurring, severely impacting the user experience. Figure 2As shown, when viewport image 1 is shared with viewport 2, there are obvious problems such as pixel loss in some areas (e.g., the lower leg) and blurred edges of clothing (the white or black background is only used to distinguish between real and distorted images).
[0024] Consider a scenario with one server and N users. The server stores all elements in a virtual space, with formats including but not limited to point clouds and textures. Each element has corresponding 3D virtual space coordinates and opacity. ∈[0,1]. Users are randomly distributed across the world, but are connected to the same virtual space via the network, where they can freely explore.
[0025] Since viewport image generation requires computational power, and the computational power of the server and users is insufficient to meet the viewport image generation needs of all users, a viewport sharing mechanism is adopted to satisfy the viewport image needs of all users. For example, by sharing user A's viewport image with user B, user B obtains a new viewport image based on their own viewport position through image deformation (or reprojection), achieving the effect of "one image calculation satisfying multiple viewers". However, incorrect sharing of viewport images between users will introduce significant image distortion (such as pixel loss, blurred object edges, etc.), mainly because the content similarity between user viewports is low. For example, user A focuses on the front of the model, while user B focuses on the side of the model; the large content difference leads to obvious pixel loss.
[0026] To address the image distortion problem (such as missing pixels, blurred object edges, etc.) caused by incorrect sharing of viewport images between users during viewport sharing, this invention proposes a novel multi-user viewport sharing scheme. This scheme includes a depth map extraction method, a deformation distortion estimation method, and a clustering-based user grouping method. By effectively estimating the image distortion between users, it guides the viewport sharing process and improves the multi-user experience quality.
[0027] Whenever the user's viewport image is refreshed, the viewport sharing scheme of this invention will be executed.
[0028] like Figure 3 As shown, the viewport sharing scheme of the present invention includes several steps: collecting viewport locations, outputting a depth map using the proposed depth map extraction method, outputting a deformation distortion estimate using the proposed deformation distortion estimation method, obtaining user groups using a clustering method, and performing multi-user viewport sharing based on the user grouping results. First, the server collects the viewport locations of all users and outputs a multi-user viewport depth map using the depth map extraction method proposed in this invention.
[0029] Specifically, viewport position information of multiple users is obtained, and viewport coverage processing is performed based on the multiple viewport position information to obtain multiple local virtual spaces.
[0030] like Figure 4 As shown, the server collects the viewport positions of all users, including virtual space positions and viewing directions; based on each user's viewport position, the server obtains the viewport cone (referred to as "view cone") corresponding to each user, and obtains the local virtual space based on the coverage area of the view cone.
[0031] Further, an opacity threshold is obtained, and it is determined whether the elements in the multiple local virtual spaces have a fixed opacity. If so, the elements with opacity less than the opacity threshold are deleted. If not, the viewpoints with opacity less than the opacity threshold of the multiple elements are deleted to obtain multiple target element sets. Projection processing is performed on the multiple target element sets to obtain multiple projection pixel sets.
[0032] like Figure 4 As shown, the elements in the virtual space are then preprocessed. The goal of this preprocessing is to remove elements with extremely low opacity in the local virtual space, as these points can easily interfere with the fast calculation of depth values. Although the data structures of elements in the virtual space may differ, such as point clouds or textures, they all have an opacity attribute to represent opaque, semi-transparent, or transparent elements, and opacity is crucial for obtaining depth values.
[0033] Preset opacity threshold (The range of values is) (∈[0,1]). Since the definition of opacity varies slightly depending on the data format, two cases will be explained separately. These two cases involve element preprocessing: 1. For elements with fixed opacity (opacity does not change with viewing angle), delete elements with opacity below the opacity threshold. Element.
[0034] 2. If the opacity changes with the viewing angle, the space occupied by the element can be reduced by eliminating viewing angles with opacity below a threshold. Taking a 3D Gaussian splatter point cloud as an example, the space occupied by a 3D Gaussian splatter point cloud element in virtual space approximates an ellipsoid. Its attributes include opacity, radius r (the radius of the sphere covering the ellipsoid), and eigenvalues of the projection covariance matrix. and These correspond to the semi-major and semi-minor axes of the ellipse, respectively, where the opacity varies with the viewing angle. Combined with an opacity threshold, the semi-major axis of a 3D Gaussian splash point cloud element in virtual space is represented as... The semi-minor axis is represented as ,in, Indicates opacity. This represents the opacity threshold, and min indicates finding the minimum value.
[0035] Furthermore, depth values are calculated for the pixels in the multiple sets of projected pixels to obtain multiple target depth values, and multiple user depth maps are obtained based on the multiple target depth values.
[0036] like Figure 4 As shown, during the preprocessing of each element, for elements that meet the opacity requirement, they need to be projected from virtual space onto the near plane of the viewport. Simultaneously, a projection plane is obtained based on the element's occupied space, and the set of pixels covered by the projection plane is obtained. The depth value of the element is recorded at these pixels. Then, each element covering that pixel compares its own depth with the current "minimum depth." If the element's depth is smaller, the depth value recorded for that pixel is updated, replacing it with its own depth value; otherwise, no replacement is performed. Finally, the depth map of the current user viewport is obtained.
[0037] This invention can quickly obtain depth maps of all users in a short period of time to meet real-time deployment requirements.
[0038] It is worth noting that the element preprocessing step is performed independently for each element. Assuming the number of elements in the local virtual space is K, the computational complexity of the two parts is O(K). Meanwhile, the "determine depth" part processes each pixel independently. Assuming the length and width of the viewport image plane are H and W respectively, the computational complexity of this part is O(n). Therefore, the parallel computation of the depth map extraction method conforms to the parallel computation state of the graphics processing unit (GPU), compressing the runtime latency to the microsecond level, which is significantly lower than the serial implementation of the traditional central processing unit (CPU).
[0039] like Figure 5 As shown, the depth map extraction method proposed in this invention relies on the opacity threshold. Due to factors such as poor virtual space reconstruction quality or lighting interference, the estimated depth map obtained by the depth map extraction method disclosed in this invention may be inaccurate and differ from the real depth map. Therefore, it is necessary to adaptively adjust the opacity threshold to make the depth map extraction result more consistent with the real depth.
[0040] Further, depth value calculation processing is performed on the pixels in the multiple sets of projected pixels to obtain multiple target depth values, and multiple user depth maps are obtained based on the multiple target depth values. Then, the method further includes: performing a first depth map extraction processing on the multiple local virtual spaces to obtain multiple real depth maps.
[0041] The server extracts the real depth map from the virtual space.
[0042] Furthermore, multiple update step sizes are obtained, and the opacity threshold is updated according to the multiple update step sizes to obtain multiple candidate opacity thresholds. Then, a second depth map extraction process is performed on the multiple local virtual spaces according to the multiple candidate opacity thresholds to obtain multiple estimated depth maps.
[0043] Initially, set the opacity threshold. Set to 0, assuming the server has loaded the virtual space, the server collects the historical viewport positions of all users (including virtual space position and viewing direction), and combines this with an opacity threshold. Then, using the depth map extraction method of this invention, it extracts the depth map of all users from the virtual space at each viewport position. To distinguish between the two types of depth maps, the depth map output by the depth map extraction method is here labeled as the estimated depth map.
[0044] Next, within the preset range, several update steps are preset, according to... The rule updates the opacity threshold, where, This indicates the update step size. The estimated depth map for each user is extracted again using the updated opacity threshold until the opacity threshold exceeds the preset range. During the experiment, a large step size is used for a rough traversal to understand the possible range of the optimal opacity threshold. Then, a small step size is used for a detailed traversal to find the optimal opacity threshold.
[0045] Further, an image evaluation algorithm is determined, and the image evaluation algorithm performs quality evaluation processing on multiple estimated depth maps based on multiple real depth maps to obtain multiple quality scores; multiple candidate opacity thresholds are filtered based on the multiple quality scores to obtain an adaptive opacity threshold.
[0046] After collecting estimated and ground truth depth maps from multiple viewport locations, various image evaluation algorithms (including but not limited to Structural Similarity Index Measure (SSIM), Learned Perceptual Image Patch Similarity (LPIPS), and Kullback-Leibler Divergence) are used to calculate the quality score of the estimated depth map. The quality scores at all opacity thresholds are compared, and the threshold with the best quality score is selected as the opacity threshold for the current scene.
[0047] Step S20: Perform deformation distortion estimation processing based on multiple user depth maps and multiple viewport position information to obtain multiple deformation distortion estimates.
[0048] like Figure 3 As shown, the user set is defined as follows: Any combination of two different users ,in, and For any two users. For each combination First and foremost, consider the user. Share viewport image with user In this case, the server uses a distortion estimation method to estimate the user's distortion. Depth map, user and users The spatial distance between the viewports, the angle between the viewport directions, and the viewport overlap rate are taken as inputs, and the deformation distortion estimate is output. Then consider the user. Share viewport image with user In this case, the server uses a distortion estimation method to estimate the user's distortion. Depth map, user and users The spatial distance between the viewports, the angle between the viewport directions, and the viewport overlap rate are taken as inputs, and the deformation distortion estimate is output. After iterating through all combinations, the final estimated distortion value between any two users is obtained.
[0049] It is worth noting that there may be one-way light or complex obstructing objects between two users in virtual space, leading to... ≠ Therefore, deformation distortion estimation methods need to calculate separately. and .
[0050] Specifically, parameter calculations are performed based on multiple user depth maps and multiple viewport position information to obtain the distance between multiple users, the angle between multiple viewport directions, and the overlap rate of multiple viewports.
[0051] Assuming user The virtual space coordinates are The viewing direction is a vector. Its viewport cone covering element set is ;user The virtual space coordinates are The viewing direction is a vector. Its viewport cone covering element set is The spatial distance between users can be obtained as follows: The angle between the viewport directions is Viewport overlap rate .
[0052] Furthermore, a neural network is determined, and through the neural network, deformation distortion calculation processing is performed based on multiple user depth maps, multiple user distances, multiple viewport direction angles, and multiple viewport overlap rates to obtain multiple deformation distortion estimates.
[0053] like Figure 6 As shown, this invention uses a neural network to achieve deformation distortion estimation. It is assumed that this method needs to estimate user... Share viewport image with user Time-varying distortion estimate, where the input is the user. Depth map (obtained by depth extraction method), user and users The spatial distance between the viewports, the angle between the viewport directions, and the viewport overlap rate are taken as inputs, and the deformation distortion estimate is output. The same method is used to obtain the deformation distortion estimate between each pair of users. With the help of parallel computing of neural networks, the estimation can be performed simultaneously in the case of multiple user combinations.
[0054] Step S30: Group the multiple users according to the multiple deformation distortion estimates to obtain user grouping results.
[0055] like Figure 3 As shown, the present invention uses a clustering method to obtain user groups and reference viewports for each group.
[0056] Grouping users based on distortion estimates allows users with similar distortion characteristics to be grouped together. Corresponding correction parameters and graphics generation strategies can be formulated for each group, ensuring the quality of the shared viewport image for each group of users. It also allows users with different devices and different perspectives to obtain the best visual effect that matches their own conditions.
[0057] Specifically, multiple cluster points are obtained based on multiple users, and multiple cluster distances are obtained based on multiple deformation distortion estimates; a clustering algorithm is determined, and the multiple cluster points are clustered using the clustering algorithm based on the multiple cluster distances to obtain user grouping results.
[0058] Assume the server generates at most M viewport images (M less than the number of users) at a two-frame refresh interval (e.g., 33.3 milliseconds at a refresh rate of 30 frames per second). This invention treats the user grouping problem as a K-median clustering problem, where each user is a point to be clustered. To users The distance is defined as the deformation distortion estimate. A greedy algorithm or clustering algorithm (including but not limited to K-median clustering, nearest neighbor propagation, etc.) is used to obtain multiple groups (no more than M groups) and the cluster centers as reference viewports.
[0059] Step S40: Obtain the target user, and perform viewport image generation processing on the target user according to the user grouping result to obtain the target viewport image.
[0060] like Figure 3 As shown, this invention enables multi-user viewport sharing based on user grouping results.
[0061] Specifically, the target user is obtained, and a viewport image transmission process is performed on the target user according to the user grouping result to obtain a reference viewport image; the real-time viewport position of the target user is obtained, and the reference viewport image is deformed according to the real-time viewport position to obtain the target viewport image.
[0062] Based on the user grouping results, each user in the group will receive a reference viewport image. Then, based on their own real-time viewport position, a new viewport image will be obtained through image warping. The sending end of the viewport image depends on the viewport image generation location. For example, if the server generates the viewport image, the server will send the viewport image to the user terminal being shared with. If another device (e.g., another user terminal) generates the viewport image, the corresponding device will send the viewport image to the user terminal being shared with.
[0063] The technical protection points of this invention include: 1. By combining depth map extraction, deformation distortion estimation, and clustering-based user grouping methods, a new viewport sharing method is obtained to optimize the overall experience quality for multiple users.
[0064] 2. This invention obtains accurate depth maps quickly by removing elements with low opacity. Simultaneously, this method supports parallel execution on GPUs, compressing runtime latency to the microsecond level, significantly lower than traditional CPU serial implementations.
[0065] 3. This invention dynamically adjusts the opacity threshold according to the different characteristics of the virtual space (such as static or dynamic scenes) to enhance the robustness of the depth map extraction method to different scenes.
[0066] 4. This invention breaks through the traditional reliance on indicators unrelated to content and innovatively integrates parameters such as depth map, user spatial perspective differences, and overlap rate with viewport content. It uses neural networks to estimate the deformation distortion of all user combinations more accurately in parallel.
[0067] This invention performs accurate deformation distortion estimation based on user depth maps, groups users according to the deformation distortion estimation values between every two users, and generates viewport images based on the user grouping results, thereby improving the generation quality of shared viewport images.
[0068] Furthermore, such as Figure 7As shown, based on the above-described multi-user holographic communication viewport sharing method, the present invention also provides a multi-user holographic communication viewport sharing system, wherein the multi-user holographic communication viewport sharing system includes: The depth map generation module 51 is used to acquire viewport position information of multiple users, perform depth map generation processing based on the multiple viewport position information, and obtain depth maps of multiple users. The deformation distortion estimation module 52 is used to perform deformation distortion estimation processing based on multiple user depth maps and multiple viewport position information to obtain multiple deformation distortion estimation values. User grouping module 53 is used to group multiple users according to multiple deformation distortion estimates to obtain user grouping results; The viewport generation module 54 is used to acquire the target user and perform viewport image generation processing on the target user according to the user grouping result to obtain the target viewport image.
[0069] Furthermore, such as Figure 8 As shown, based on the above-mentioned multi-user holographic communication viewport sharing method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 4 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.
[0070] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a multi-user holographic communication viewport sharing program 40, which can be executed by the processor 10 to implement the multi-user holographic communication viewport sharing method of this application.
[0071] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the multi-user holographic communication viewport sharing method.
[0072] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface.
[0073] In one embodiment, when the processor 10 executes the multi-user holographic communication viewport sharing program 40 in the memory 20, the following steps are performed: Obtain viewport position information of multiple users, and perform depth map generation processing based on the multiple viewport position information to obtain depth maps of multiple users; Based on multiple user depth maps and multiple viewport position information, deformation distortion estimation processing is performed to obtain multiple deformation distortion estimates. The users are grouped according to the multiple deformation distortion estimates to obtain user grouping results; Obtain the target user, and perform viewport image generation processing on the target user according to the user grouping results to obtain the target viewport image.
[0074] Specifically, the step of obtaining viewport position information of multiple users and performing depth map generation processing based on the multiple viewport position information to obtain multiple user depth maps includes: Obtain viewport position information from multiple users, and perform viewport overlay processing based on the multiple viewport position information to obtain multiple local virtual spaces; Obtain an opacity threshold, and perform spatial projection processing on multiple elements in the local virtual space based on the opacity threshold to obtain multiple sets of projected pixels. Depth values are calculated for pixels in the multiple sets of projected pixels to obtain multiple target depth values, and multiple user depth maps are obtained based on the multiple target depth values.
[0075] Specifically, obtaining the opacity threshold and performing spatial projection processing on elements in multiple local virtual spaces based on the opacity threshold to obtain multiple sets of projected pixels includes: Obtain the opacity threshold and determine whether the elements in the multiple local virtual spaces have a fixed opacity; If yes, then delete the elements whose opacity is less than the opacity threshold from among the multiple elements; if no, then delete the viewpoints of the multiple elements whose opacity is less than the opacity threshold, to obtain multiple target element sets. Projection processing is performed on multiple sets of target elements to obtain multiple sets of projected pixels.
[0076] The process of calculating depth values for pixels in multiple sets of projected pixels to obtain multiple target depth values, and obtaining multiple user depth maps based on the multiple target depth values, further includes: A first depth map extraction process is performed on multiple local virtual spaces to obtain multiple real depth maps; Multiple update step sizes are obtained, and the opacity threshold is updated according to the multiple update step sizes to obtain multiple candidate opacity thresholds. Then, a second depth map extraction process is performed on the multiple local virtual spaces according to the multiple candidate opacity thresholds to obtain multiple estimated depth maps. A quality evaluation algorithm is determined, and the image evaluation algorithm is used to perform quality evaluation processing on multiple estimated depth maps based on multiple real depth maps to obtain multiple quality scores; The multiple candidate opacity thresholds are filtered based on the multiple quality scores to obtain an adaptive opacity threshold.
[0077] Specifically, the step of performing deformation distortion estimation processing based on multiple user depth maps and multiple viewport position information to obtain multiple deformation distortion estimates includes: Based on multiple user depth maps and multiple viewport position information, parameter calculations are performed to obtain the distance between multiple users, the angle between multiple viewport directions, and the overlap rate of multiple viewports. A neural network is determined, and through the neural network, deformation distortion calculation is performed based on multiple user depth maps, multiple user distances, multiple viewport direction angles, and multiple viewport overlap rates to obtain multiple deformation distortion estimates.
[0078] Specifically, the step of grouping multiple users based on multiple deformation distortion estimates to obtain user grouping results includes: Multiple clustering points are obtained based on multiple users, and multiple clustering distances are obtained based on multiple deformation distortion estimates; A clustering algorithm is determined, and the clustering algorithm is used to cluster multiple cluster points based on multiple cluster distances to obtain user grouping results.
[0079] The step of obtaining the target user and performing viewport image generation processing on the target user according to the user grouping results to obtain the target viewport image specifically includes: Obtain the target user, and perform viewport image transmission processing on the target user according to the user grouping result to obtain the reference viewport image; Obtain the real-time viewport position of the target user, and perform deformation processing on the reference viewport image based on the real-time viewport position to obtain the target viewport image.
[0080] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multi-user holographic communication viewport sharing program, which, when executed by a processor, implements the steps of the multi-user holographic communication viewport sharing method described above.
[0081] In summary, this invention provides a multi-user holographic communication viewport sharing method, system, and terminal. The method includes: acquiring viewport position information of multiple users; performing depth map generation processing based on the multiple viewport position information to obtain multiple user depth maps; performing deformation distortion estimation processing based on the multiple user depth maps and the multiple viewport position information to obtain multiple deformation distortion estimates; grouping the multiple users based on the multiple deformation distortion estimates to obtain user grouping results; acquiring a target user; and performing viewport image generation processing on the target user based on the user grouping results to obtain a target viewport image. This invention performs accurate deformation distortion estimation based on user depth maps, groups users based on deformation distortion estimates between every two users, and generates viewport images based on user grouping results, thereby improving the generation quality of shared viewport images.
[0082] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.
[0083] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.
[0084] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A method for sharing a holographic communication viewport among multiple users, characterized in that, The multi-user holographic communication viewport sharing method includes: Obtain viewport position information of multiple users, and perform depth map generation processing based on the multiple viewport position information to obtain depth maps of multiple users; Based on multiple user depth maps and multiple viewport position information, deformation distortion estimation processing is performed to obtain multiple deformation distortion estimates. The users are grouped according to the multiple deformation distortion estimates to obtain user grouping results; Obtain the target user, and perform viewport image generation processing on the target user according to the user grouping results to obtain the target viewport image.
2. The multi-user holographic communication viewport sharing method according to claim 1, characterized in that, The step of obtaining viewport position information of multiple users and performing depth map generation processing based on the multiple viewport position information to obtain multiple user depth maps specifically includes: Obtain viewport position information from multiple users, and perform viewport overlay processing based on the multiple viewport position information to obtain multiple local virtual spaces; Obtain an opacity threshold, and perform spatial projection processing on multiple elements in the local virtual space based on the opacity threshold to obtain multiple sets of projected pixels. Depth values are calculated for pixels in the multiple sets of projected pixels to obtain multiple target depth values, and multiple user depth maps are obtained based on the multiple target depth values.
3. The multi-user holographic communication viewport sharing method according to claim 2, characterized in that, The step of obtaining an opacity threshold and performing spatial projection processing on elements in multiple local virtual spaces based on the opacity threshold to obtain multiple sets of projected pixels specifically includes: Obtain the opacity threshold and determine whether the elements in the multiple local virtual spaces have a fixed opacity; If yes, then delete the elements whose opacity is less than the opacity threshold from among the multiple elements; if no, then delete the viewpoints of the multiple elements whose opacity is less than the opacity threshold, to obtain multiple target element sets. Projection processing is performed on multiple sets of target elements to obtain multiple sets of projected pixels.
4. The multi-user holographic communication viewport sharing method according to claim 2, characterized in that, The process of calculating depth values for pixels in multiple sets of projected pixels to obtain multiple target depth values, and obtaining multiple user depth maps based on the multiple target depth values, further includes: A first depth map extraction process is performed on multiple local virtual spaces to obtain multiple real depth maps; Multiple update step sizes are obtained, and the opacity threshold is updated according to the multiple update step sizes to obtain multiple candidate opacity thresholds. Then, a second depth map extraction process is performed on the multiple local virtual spaces according to the multiple candidate opacity thresholds to obtain multiple estimated depth maps. A quality evaluation algorithm is determined, and the image evaluation algorithm is used to perform quality evaluation processing on multiple estimated depth maps based on multiple real depth maps to obtain multiple quality scores; The multiple candidate opacity thresholds are filtered based on the multiple quality scores to obtain an adaptive opacity threshold.
5. The multi-user holographic communication viewport sharing method according to claim 1, characterized in that, The step of performing deformation distortion estimation processing based on multiple user depth maps and multiple viewport position information to obtain multiple deformation distortion estimates specifically includes: Based on multiple user depth maps and multiple viewport position information, parameter calculations are performed to obtain the distance between multiple users, the angle between multiple viewport directions, and the overlap rate of multiple viewports. A neural network is determined, and through the neural network, deformation distortion calculation is performed based on multiple user depth maps, multiple user distances, multiple viewport direction angles, and multiple viewport overlap rates to obtain multiple deformation distortion estimates.
6. The multi-user holographic communication viewport sharing method according to claim 1, characterized in that, The step of grouping multiple users based on multiple deformation distortion estimates to obtain user grouping results specifically includes: Multiple clustering points are obtained based on multiple users, and multiple clustering distances are obtained based on multiple deformation distortion estimates; A clustering algorithm is determined, and the clustering algorithm is used to cluster multiple cluster points based on multiple cluster distances to obtain user grouping results.
7. The multi-user holographic communication viewport sharing method according to claim 1, characterized in that, The step of acquiring the target user and performing viewport image generation processing on the target user based on the user grouping results to obtain the target viewport image specifically includes: Obtain the target user, and perform viewport image transmission processing on the target user according to the user grouping result to obtain the reference viewport image; Obtain the real-time viewport position of the target user, and perform deformation processing on the reference viewport image based on the real-time viewport position to obtain the target viewport image.
8. A multi-user holographic communication viewport sharing system, characterized in that, The multi-user holographic communication viewport sharing system includes: The depth map generation module is used to acquire viewport position information of multiple users, and perform depth map generation processing based on the multiple viewport position information to obtain depth maps of multiple users. The deformation distortion estimation module is used to perform deformation distortion estimation processing based on multiple user depth maps and multiple viewport position information to obtain multiple deformation distortion estimation values. The user grouping module is used to group multiple users according to multiple deformation distortion estimates to obtain user grouping results. The viewport generation module is used to acquire target users and perform viewport image generation processing on the target users according to the user grouping results to obtain target viewport images.
9. A terminal, characterized in that, The terminal includes a memory, a processor, and a multi-user holographic communication viewport sharing program stored in the memory and executable on the processor. When the processor executes the multi-user holographic communication viewport sharing program, it implements the steps of the multi-user holographic communication viewport sharing method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multi-user holographic communication viewport sharing program, which, when executed by a processor, implements the steps of the multi-user holographic communication viewport sharing method as described in any one of claims 1-7.