System and method for augmented reality content delivery in pre-capture environment

Abstract

This disclosure describes systems and methods for augmented reality (AR) content delivery in a pre-capture environment. In one embodiment, an AR client collects information about a physical environment with the aid of a sensor embedded into an AR headset. And the client reconstructs a 3D model of the physical environment according to the collected information and sends the 3D model to the AR content server. Based on the 3D reconstruction information acquired at an AR content server, the server modifies the content. And the AR content server compares the AR content with the 3D reconstruction data and carries out visibility analysis on the AR content. Based on the visibility analysis, AR elements occluded by real objects are removed to reduce the data size of the AR content. Finally, the AR content server streams the modified content to the AR client and the AR client renders the content and sends it to the head-mounted device.

Description

This application is a divisional application of Chinese Patent Application No. 201880009301.7, filed on January 25, 2018, entitled “System and Method for Augmented Reality Content Delivery in a Pre-Capture Environment”, the contents of which are incorporated herein by reference. CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a formal application of U.S. Provisional Patent Application Serial No. 62 / 453,317, filed on February 1, 2017, entitled “System and Method for Augmented Reality Content Delivery in Pre-Captured Environments,” and claims the benefit of 35 U.S.C. §119(e), which is incorporated herein by reference in its entirety. Background Art

[0002] Storing digital media in the cloud enables accessibility from anywhere in the world, as long as the client has an internet connection. Digital media with a greater level of realism is encoded using high-resolution formats that require large file sizes. Transmitting this information (e.g., streaming the digital media to a client device) requires a commensurately large allocation of communication resources. Visually rich virtual reality (VR) content and augmented reality (AR) content both consume large amounts of data, which is problematic for AR and VR content delivery. The limited bandwidth of the data connection between the service delivering the content and the clients consuming the content is a significant bottleneck.

[0003] For example, the spherical video used by movie VR technology greatly increases the video resolution requirements for online video feeds. In movie VR, the transmitted video stream contains not only all the pixels seen on the screen like ordinary 2D video, but also the full 360° field of view pixels from a single viewing point. When movie VR content is consumed, only a small area of ​​the entire video content is visible at any given time. The smaller cropped area of ​​the spherical view displayed on the client device needs to provide full HD (i.e., 1080p) resolution to match the resolution provided by the currently used high-resolution 2D video for the viewing experience. When comparing single-view 360-degree spherical video with stereoscopic 360-degree 3D spherical video (where separate image feeds are required for each eye), the bandwidth required for content delivery becomes the main bottleneck. The next generation of immersive content formats (which not only provide 360-degree stereoscopic views from a single viewpoint, but also allow users to move within a restricted area within the content) will consume exponentially more information and communication resources.

[0004] The ability to move within the VR space is not the only driver behind these next-generation format sizes, as the next generation of AR display technology will preferably exhibit the same level of spatial realism. Current content streaming methods for delivering AR and VR experiences rely on traditional data transmission optimization techniques (e.g., spatial 2D image pixel compression and temporal motion compensation). Traditional data compression techniques do not address the nuances of AR use cases and can be improved. Summary of the invention

[0005] The present application describes systems and methods for selectively providing augmented reality (AR) information to an AR display client device. In some embodiments, the bandwidth for providing AR information to the client device can be reduced by selectively transmitting to the client device only AR content elements whose appearance is not obscured by physical objects in the AR viewing position.

[0006] For example, in one embodiment, a system (e.g., an AR content server) determines the position of an AR display device within an AR viewing environment. For at least a first element of AR content (e.g., a video or an AR object), the system selects a display position within the AR viewing environment. The system determines whether any physical object in the AR viewing environment is located along a line of sight between the AR display device and the display position of the first element. Only in response to determining that no physical object is in the line of sight, the system transmits the first element of the AR content to the AR display device. The system can handle some elements of AR content at different display positions (including elements that may be in motion and thus have a variable display position), and can send to the AR display device only elements whose positions are not blocked by physical objects from the perspective of the AR display device. In some embodiments, the AR element can be a portion of an AR object or video, whereby the unobstructed portion of the object or video is sent to the client and the obstructed portion is not sent.

[0007] The AR head mounted device may perform an exemplary method according to an embodiment. The method includes obtaining a digitally reconstructed three-dimensional (3D) environment of an augmented reality (AR) viewing position, detecting an object in the digitally reconstructed 3D environment, determining the depth and geometry of the object, sending a request for an augmented reality (AR) content stream to a server, the request including information indicating the depth and geometry of the object, and receiving the AR content stream from the server, the AR content stream not including AR content occluded by the object.

[0008] According to an embodiment, the method further comprises determining the AR viewing position using one or more of global positioning system (GPS) data, wireless data, and image recognition. In one embodiment, the digitally reconstructed 3D environment is obtained via one or more of a stored previous AR session, a combination of stored data and real-time collected data, or a depth data point cloud.

[0009] In another embodiment, the digitally reconstructed 3D environment is acquired by scanning the AR viewing position by examining the AR viewing position from multiple viewpoints.

[0010] In one embodiment, the method further comprises tracking the position and orientation of the viewpoint within the digitally reconstructed 3D environment.

[0011] In one embodiment, the information indicative of the depth and the geometry of the object comprises one or more of real-time raw sensor data, a depth map of the object, or an RGB-D data stream.

[0012] In one embodiment, the type of AR content requested is immersive video, and the information indicative of the depth and the geometry of at least one real-world object is formatted as a spherical depth map of a real-world AR viewing position.

[0013] Another embodiment relates to an augmented reality (AR) content server, comprising a processor, a communication interface, and a data storage. In this embodiment, the AR content server is configured to receive a request for AR content via the communication interface, obtain the requested AR content from the data storage, obtain a digitally reconstructed three-dimensional (3D) environment of an AR viewing position, receive a viewpoint position and orientation within the AR viewing position via the communication interface, perform a visibility analysis via a comparison of the requested AR content with the digitally reconstructed 3D environment and the viewpoint position and orientation, and modify the requested AR content by removing data associated with one or more objects that are not visible from the viewpoint position and orientation based on the visibility analysis.

[0014] According to one embodiment, the digitally reconstructed 3D environment is accessible from one or more of the data store, augmented reality (AR client), or a combination of the data store and the AR client.

[0015] In another embodiment, or in the same embodiment, the modification of the requested AR content can be a function of the content type of the requested AR content, where the content type includes one or more of a three-dimensional (3D) scene, a light field, and an immersive video with depth data.

[0016] Furthermore, in one or more embodiments, the modified requested AR content can include inserting a previously removed three-dimensional (3D) object within the requested AR content based on the updated viewpoint position and orientation.

[0017] In one embodiment, the visibility analysis includes identifying where the digitally reconstructed 3D environment has depth values ​​that are smaller than depth data associated with the requested AR content.

[0018] Another embodiment relates to a method, which includes receiving a request for an augmented reality (AR) content video stream, obtaining the requested AR content video stream from a data store, obtaining a digitally reconstructed 3D environment of a real-world AR viewing position, receiving a viewpoint position and orientation of an AR client at the real-world AR viewing position, analyzing the digitally reconstructed 3D environment and the viewpoint position and orientation to determine whether the requested AR content video stream includes one or more objects occluded in the digitally reconstructed 3D environment, and removing the one or more objects occluded in the digitally reconstructed 3D environment.

[0019] In one embodiment, the method includes sending the modified AR content video stream to an AR display device.

[0020] In one embodiment, the analyzing the digitally reconstructed 3D environment and the viewpoint position and orientation includes determining the position of the AR display device, identifying one or more physical objects in the digitally reconstructed 3D environment along a line of sight determined from the viewpoint position and orientation of the AR display device, and determining, based on the line of sight, whether the one or more objects in the AR content stream are obscured by the one or more physical objects.

[0021] In one embodiment, the analyzing the digitally reconstructed 3D environment and the viewpoint position and orientation includes comparing the depth value of the digitally reconstructed 3D environment with the depth value in the AR content video stream, and discarding the portion of the AR content video stream whose depth value is greater than the corresponding depth value in the digitally reconstructed 3D environment to compensate for occlusion caused by one or more dynamic objects and / or static objects lost from the digitally reconstructed 3D environment.

[0022] In one variation of the implementation of the present application, the client knows the type of content it is requesting from the server, and based on this, can provide the environment information in the most compact format possible. For example, for immersive video, the client may not send the entire environment model, but only a spherical depth map of the environment rendered from the viewpoint used at that moment.

[0023] In one variation of an embodiment of the present application, the AR client device does not perform viewpoint and orientation tracking, but instead uses an external outside-looking-in tracking scheme. In a first further variation, the AR client device first receives tracking information from the external tracking scheme, transforms it into the same coordinate system used in the 3D reconstruction, and then sends it to the content server. In a second further variation, the external tracking scheme transforms the viewpoint and orientation into the same coordinate system used in the 3D reconstruction, and then sends it to the content server. In a third further variation, the AR content server receives tracking information from the external tracking scheme, and the server transforms it into the same coordinate system used in the 3D reconstruction.

[0024] In one variation of an embodiment of the present application, the AR content server generates depth information for the light field once the light field data has been uploaded to the server. During runtime, when the AR content server receives a request for light field content, the content can be modified more efficiently by using the depth information that has been previously generated (eliminating the need to perform depth detection during content delivery).

[0025] Any of the embodiments, variations, and permutations described in the following paragraphs and anywhere else in the present disclosure can be implemented with respect to any embodiment (including with respect to any method embodiment and with respect to any system embodiment).

[0026] The disclosed systems and methods can significantly reduce the amount of data transmitted between an AR content service and a viewing client device in many online AR content delivery use cases. The reduction is based on the physical characteristics of the environment (where the client consumes the content) and can be performed on a per-client basis. This approach can be combined with other traditional data compression methods, thereby providing further content delivery optimization without sacrificing the benefits achievable with traditional content delivery compression techniques. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In the accompanying drawings, the same figure numbers in each figure represent the same or functionally similar elements. The accompanying drawings, together with the following detailed description, are integrated into the specification or form part of the specification and are used to further explain the embodiments of the concepts including the claimed invention and to explain the various principles and advantages of these embodiments.

[0028] Figure 1 is a visual overview of a method for AR content delivery in a pre-capture environment according to at least one embodiment.

[0029] Figure 2is a flow chart of a method performed by an AR client for AR content delivery in a pre-capture environment according to at least one embodiment.

[0030] Figure 3 is a flow chart of a method performed by an AR content server for AR content delivery in a pre-capture environment according to at least one embodiment.

[0031] Figure 4 is a sequence diagram of a method for AR content delivery in a pre-capture environment according to at least one embodiment.

[0032] Figure 5 is an example stereogram of a user in an example real-world AR viewing position according to at least one embodiment.

[0033] Figure 6 is a plan view of a digitally reconstructed 3D environment in accordance with at least one embodiment.

[0034] Figure 7 is the requested AR content according to at least one embodiment and Figure 6 A floor plan of a digitally reconstructed 3D environment.

[0035] Figure 8 is an exemplary embodiment of Figure 7 The requested AR content performs a visual analysis of the floor plan.

[0036] Fig. 9 is an exemplary illustration of modified AR content according to at least one embodiment and Figure 6 A floor plan of a digitally reconstructed 3D environment.

[0037] Fig.10 is a diagram showing a method according to at least one embodiment Figure 5 of users see Fig. 9 An exemplary stereogram of modified AR content.

[0038] Fig.11 An exemplary wireless transmit / receive unit (WTRU) is shown that may be used as an AR client viewer in some embodiments.

[0039] Fig.12 Exemplary network entities that may serve as an AR content server or may serve as an AR content storage in some implementations are shown.

[0040] Those skilled in the art will appreciate that for simplicity and clarity, the elements in the figures are illustrated and are not necessarily drawn to scale. For example, the size of some elements in the figures may be magnified relative to other elements to help improve the understanding of the embodiments of the present invention.

[0041] Device and method components are represented in the drawings with conventional symbols where appropriate, showing only certain details relevant to understanding embodiments of the invention so as not to obscure the present disclosure with details that would be apparent to those skilled in the art having understood the description herein. DETAILED DESCRIPTION

[0042] Before proceeding to the detailed description, it is noted that the entities, connections, arrangements, etc. depicted in and described in connection with the various figures are provided by way of example and not by way of limitation. Therefore, any and all statements or other indications (which may be considered absolute in isolation and out of context and therefore limiting) regarding the following may only be properly understood if they are constructively preceded by a clause such as "in at least one embodiment...": what a particular figure "depicts", what a particular element or entity in a particular figure "is" or "has", and any and all similar statements.

[0043] The present application discloses a method and system for delivering augmented reality content in a pre-capture environment. The present disclosure relates to the transmission of content data between an augmented reality (AR) content server and an AR client (e.g., an AR client running on an AR device, such as an optically see-through AR head-mounted device or a video see-through AR head-mounted device). The exemplary method described in the present application utilizes a client device to provide information about the environment in which the augmented content is to be displayed to a content server. The AR content server adjusts the AR content delivery based on the environmental characteristics of each individual client, whereby redundant data that is not visible to a particular client is removed from the content stream transmitted to the client.

[0044] In an exemplary embodiment, at the beginning of an AR session, the AR client device reconstructs a digital model of the current 3D environment. The generation of the model can use previously collected data corresponding to the same location. In other cases, the model is made by scanning the environment using sensors built into the AR client device (e.g., RGB-D or other depth sensors). The combination of previously collected data and currently collected data can be used to improve the accuracy of the model. A copy of the model is then sent to a content server, which provides the server with instant access to a digital version of the current 3D environment. By integrating more sensor data from the sensing client device, the server copy of the model can be updated and improved at any time. The model can be used to perform virtual object visibility analysis. The client device also uses the collected environmental data to assist in position and orientation tracking (pose tracking). The pose (viewpoint position and orientation) of the device is used to synchronize the augmented elements displayed by the AR client with the conversion of the user's head movement. The pose information is sent to the server or calculated by the server using the environmental model or other data to allow the server to estimate the visibility of elements of the AR content from the user's viewpoint. From the user's viewpoint, real-world objects (e.g., tables, buildings, trees, doors, and walls) can block AR content. This occurs when the depth of the AR content is farther from the user than real-world objects but is also on the same line of sight. Real AR content that contains high-resolution depth information is considered to enhance the perception of reality.

[0045] In one embodiment, an AR device for viewing AR content is equipped with a sensor capable of generating depth information from the environment. The sensor or combination of sensors may include one or more sensors, such as an RGB-D camera, a stereo camera, an infrared camera, a lidar, a radar, a sonar, and any other type of sensor known to those skilled in the art of depth sensing. In some embodiments, a combination of sensor types and enhanced processing methods are used for depth detection. When a user initializes a viewing session, the viewing client reconstructs the environment model by collecting depth data from the environment.

[0046] During the reconstruction process performed on the client device, the sensor collects point cloud data from the environment as the user moves the device and the sensor within the environment. Sensor observations with changing viewpoints are combined to form a coherent 3D reconstruction of the complete environment. Once the 3D reconstruction reaches a completion threshold, the AR client sends the reconstructed model and a request for specific AR content to the AR content server. In an exemplary embodiment, for example, the completion of the 3D reconstruction can be measured by the percentage of surrounding area coverage, the number of discrete observations, the sensing duration, etc., as well as any similar quality value that can be used as a threshold. The 3D reconstruction of the environment can be performed using any known reconstruction method, such as in KinectFusion TMor the reconstruction methods described in the Point Cloud Library (PCL).

[0047] In one variation of this process, at the start of an AR viewing session, the client begins to continuously stream RGB-D data to the server. The AR client then sends a content request. In this variation, the AR content server performs a 3D reconstruction process using the received RGB-D data stream and stores the reconstructed environment model. As the server builds a per-client environment model, it also begins to modify the AR content by removing virtual elements that are occluded by the current environment. As the 3D reconstruction becomes more complete, the content removal process becomes more precise.

[0048] A user operating an AR client device can initiate a request for an AR session. The request is sent to the AR content server and indicates the specific content that the user has selected. The AR content to be displayed can be defined as a network link to the AR content server and a reference to the specific AR content held within the AR content server. The operating system can automatically handle initializing the AR client based on detection of the type of link activated by the user. The client can be embedded with an application for receiving the content link, such as a web browser and a mobile messaging application. Once the user starts an AR session, the 3D reconstruction of the environment described previously is initialized.

[0049] After the reconstruction process, the client starts a posture tracking process. The purpose of the posture tracking process is to estimate where the client device is and which direction the client device is facing relative to the previously reconstructed environment. Any known tracking technology can be used for posture tracking, and the reconstructed environment model and client device sensor data can be used to assist in the posture tracking. In some embodiments, once the client device posture has been determined, the client sends the reconstructed environment model and the determined posture to the AR content server along with a request for specific AR content.

[0050] The AR content server receives the AR content request from the AR client and obtains the requested content from the content storage. The AR content server then modifies the requested AR content to remove redundant data. In this case, content delivery optimization involves a process performed by the AR content server. During content delivery optimization, the optimized portion of the requested AR content presented at a location not visible to the viewer will be removed from the content. In AR, the virtual elements of the AR content are preferably perceived as part of the real world seen by the viewer (or merged with it). In order to enhance the perception of reality, the embodiment disclosed in the present application copies the appearance of the physical object of the virtual object that blocks the user's field of view to the virtual object. By removing all elements blocked by real-world elements from the content, the size of the content can be reduced without causing a loss of quality that can be observed by the user. Depending on the type of content requested, the AR content server uses a slightly different content delivery optimization process. In fact, even client devices can take advantage of further improvements based on the type of AR content they request. Different ways to optimize light fields, simulated videos, and synthetic 3D scenes are described in the following sections.

[0051] Light field data can include an array of sub-images acquired simultaneously at a scene from a number of different viewpoints. Based on the array of sub-images, new viewpoints between the original viewpoints can be generated computationally. The array of sub-images can also be processed to achieve refocusing of the assembled final image. Most relevantly, the array of sub-images can be used to extract depth information corresponding to the captured scene, for example via analysis of disparity and inconsistency maps of the sub-images.

[0052] For light field optimization, the AR content server extracts depth information from the light field and aligns the light scene depth map with the environment model received from the AR client. For each sub-image of the light field, the AR server transforms the viewpoint of the combined light scene depth map and the client environment model to match the viewpoint of the light field sub-image. For each pixel of the light field sub-image, the AR content server compares the depth value of the environment model from the client with the corresponding depth value extracted from the light field data. If the depth map of the light field has a depth value greater than the depth value from the client environment model, the corresponding pixels in the light field sub-image are discarded because they are occluded by real-world objects and are therefore not visible to the client.

[0053] As for immersive video (which has associated depth information available), optimization can be performed by comparing the depth values ​​of the immersive video with the depth values ​​of the environment relative to the tracked camera pose. For any content removal, the AR content server operates to re-render the video. During re-rendering, portions of the video data that are occluded by real-world elements in the user's environment are discarded. These large discarded areas can be effectively compressed using traditional video compression methods.

[0054] In some embodiments, a synthetic 3D scene including virtual 3D information can be optimized by performing visibility detection on virtual 3D elements within a geometric view captured from a real-world viewing environment. In the visibility detection, the visibility of the content from the user's viewpoint is determined. Based on the geometric description of the virtual AR elements, visibility detection can be performed on objects or on each vertex of each object. In some embodiments, object-level removal (as opposed to vertex-level removal) is performed, where the removal of portions of the object geometry results in the need for significant object data reorganization. This prevents the 3D rendering algorithm from using buffer objects, which store portions of the content as static entities in GPU memory. Once the relevant content delivery optimization is completed, the modified AR content is sent to the AR client.

[0055] The AR client receives the modified AR content from the AR content server and displays it to the viewer. In order to display the content, the AR client determines the device posture from the client device depth sensor. Based on the orientation information, the display process aligns the received content coordinate system with the user's coordinate system. In some embodiments, after aligning the content, a subprocess compares the depth value received from the client device sensor with the depth value of the modified content received from the AR content server. Areas in the content whose depth values ​​are greater than the corresponding depth values ​​in the client device depth sensor can be discarded and not presented, because otherwise they would be presented in a spatial position that is occluded by real physical elements in the environment. The AR content server has removed AR content elements that were occluded by elements present in the environment model sent by the client to the server, but this runtime depth comparison can handle occlusions caused by dynamic and static elements that are missing from the environment model sent to the content server.

[0056] Figure 1is a visual overview of a method for AR content delivery in a pre-capture environment according to at least one embodiment. In the visual overview, a user 100 is depicted wearing an AR wearable device client device 102. The user 100 is at a real-world AR viewing position 104 including a large table 106. The client device in 102 begins to reconstruct the current environment via the data collection camera and sensors in the AR wearable device client device 102. The client 102 collects data about the shape and position of the table using the camera and sensors in the AR wearable device and sends the data to the AR content server 108.

[0057] The user 100 wishes to consume AR content depicting two avatars. The AR client sends a content request 114 to the AR content server 108. The AR content server 108 performs a visibility analysis on the selected content. The AR content server 108 determines that the top avatar is visible 110 and the bottom avatar 112 is obscured by the table 106 based on the user's position and orientation with respect to the current environment.

[0058] The AR content server 108 then modifies the AR content by removing the virtual character at the bottom. Standard compression techniques can be used to reduce the size of the modified frame. The modified content stream is then sent from the server to the user 100 for consumption.

[0059] One embodiment includes a process performed by an AR wearable device client device 102. The process includes generating a digitally reconstructed 3D environment of a real-world AR viewing location using the AR client device 102. The process includes detecting an object in the digitally reconstructed 3D environment. The process also includes determining a depth and a geometry of the object. The process also includes sending information indicating the depth and geometry of the object to an AR content server 108. The process also includes sending a request for AR content from an AR client to the AR content server 108. The process also includes receiving an AR content stream 116 from the AR content server 108 at the AR client device 102, wherein the received AR content stream 108 has been filtered by the AR content server 108 to exclude AR content that would be occluded by the object 112.

[0060] Figure 2 is a flow chart of a method performed by an AR client for AR content delivery in a pre-capture environment according to at least one embodiment. Figure 2Process 200 is depicted, which includes elements 202-210. In element 202, an AR client (e.g., client 102) determines a real-world AR viewing position. In element 204, the process 200 includes identifying and tracking a viewpoint position and orientation of the AR client (e.g., client 102) within the real-world AR viewing position 104. In element 206, the process 200 includes sending information indicating the viewpoint position and orientation to the AR content server 108. In element 208, the process 200 includes sending a request for AR content to the AR content server 108. In element 201, the process 200 includes receiving an AR content stream, such as stream 116, from an AR content server at the AR client device 102, wherein the received AR content stream (e.g., stream 116) has been filtered by the AR content server (e.g., server 108) to exclude AR content that is not visible in the real-world AR viewing position from the viewpoint position and orientation.

[0061] Figure 3 is a flowchart of a method performed by an AR content server for AR content delivery in a pre-capture environment according to at least one embodiment. Figure 3 Process 300 is depicted, which includes elements 302-310. In element 302, an AR content server (e.g., server 108) receives a request for AR content from an AR client (e.g., client device 102). In element 304, the process 300 includes obtaining a digitally reconstructed 3D environment for a real-world AR viewing position (e.g., position 104). In element 306, the process 300 includes receiving a viewpoint position and orientation of the AR client for the real-world AR viewing position. In element 308, the process 300 includes comparing the requested AR content with the digitally reconstructed 3D environment and the viewpoint position and orientation to perform a visibility analysis of the requested AR content, and modifying the requested AR content by removing portions that are not visible from the viewpoint position and orientation as indicated by the visibility analysis. In element 310, the process 300 includes completing the request for AR content by sending the modified AR content to the AR client.

[0062] In at least one embodiment, obtaining the digitally reconstructed 3D environment of the real-world AR viewing position includes obtaining the reconstructed 3D environment from a data store. In at least one embodiment, obtaining the digitally reconstructed 3D environment of the real-world AR viewing position includes obtaining the reconstructed 3D environment from an AR client. In at least one embodiment, obtaining the digitally reconstructed 3D environment of the real-world AR viewing position includes generating the reconstructed 3D environment using data from the data store and real-time data from an AR client device.

[0063] In at least one embodiment of the AR content server, modifying the requested AR content varies depending on the content type. In one such embodiment, the content type is a 3D scene, and modifying the requested AR content includes: (i) removing occluded 3D objects from the requested AR content, (ii) obtaining updates for the viewpoint position and orientation, and (iii) sending the AR object that was previously removed but is now visible to the AR client based on the updated viewpoint position and orientation. In another such embodiment, the content type is a light field, and modifying the requested AR content includes: (i) processing the light field content frame by frame to remove the occluded portion, and (ii) repackaging the remaining data for streaming. In another such embodiment, the content type is a realistic video with depth data, and modifying the requested AR content includes: (i) removing portions of the video where the depth value of the digitally reconstructed 3D environment is less than the depth value associated with the video, (ii) re-rendering the frame, and (iii) repackaging the video for streaming.

[0064] Figure 4 is a sequence diagram of a method for AR content delivery in a pre-capture environment according to at least one embodiment. For further resource purposes, the following paragraphs contain Figure 4 Example traversal of a sequence graph.

[0065] The client device 102 reconstructs the current AR viewing position 404 in the reconstructed environment. The user 100 launches the AR viewing client 102 at step 406 and scans the current environment 408 using an AR headset that can be directly or wirelessly coupled to the viewing client. The client device 102 can alternatively determine the current AR viewing position and send this information to the AR content server 108, which already has a model of the environment. Scanning and reconstructing the environment is also useful for tracking purposes. Figure 4 In the example embodiment, when the digital reconstruction is completed, in step 410, the user 100 is notified that "the client is ready".

[0066] The user 100 can then initiate an AR content streaming (viewing) session as shown in step 412. The user 100 enters or selects an AR content link 414 and gives it to the AR client 102. Upon receiving the link, the AR client performs pose tracking 414 (but this step can be initiated at an earlier time). The AR client sends a content request 416 for the digitally reconstructed 3D environment and the current AR client pose to the AR content server 108.

[0067] At step 420 , the AR content server 108 retrieves the requested AR content from the AR content storage 402 at get content 418 and step 420 .

[0068] Content streaming and viewing 422 may begin after acquisition. The AR content server 108 optimizes the requested AR content 424 by removing elements that are occluded by the real-world environment. This may involve modifying content frames, re-rendering modified frames, and the like. The AR content server 108 performs visibility analysis (the AR client may also perform this) to determine which AR content is not visible to the user 100 using the digitally reconstructed 3D environment and the client pose. The optimized content 424 is then provided to the AR viewer client 102 via the optimized content stream 426, whereby the user 100 is able to see the displayed content 428.

[0069] At least one embodiment includes a process for digitally reconstructing an augmented reality (AR) viewing position using an AR client. The process also includes sending information describing the digitally reconstructed AR viewing position from the AR client to an AR content server. The process also includes sending a request for AR content from the AR client to the AR content server. The process also includes sending information describing the position and orientation of the AR client within the AR viewing position from the AR client to the AR content server. The process also includes determining the visibility of the requested AR content based on the received information describing the digitally reconstructed AR viewing position and the received information describing the position and orientation of the AR client at the AR content server. The process also includes modifying the requested AR content at the AR content server by removing a portion of the requested AR content determined to be invisible. The process also includes sending the modified AR content from the AR content server to the AR client. The process also includes using the AR client to enhance the AR viewing position with the modified AR content.

[0070] Figure 5 is an illustration of a user 500 and an exemplary real-world AR viewing location 502 according to at least one embodiment. Figure 5 Is used to help Figures 6 to 10 A reference image for the description. Figure 5 Depicted is a user 500 in an exemplary real-world AR viewing location. The viewing location 502 is a room that contains two doorways 504 and 506, a dresser 508, a sofa 510, and a chair 512. The user 500 uses an AR headset 514 to scan the real-world AR viewing location.

[0071] Figure 6 6 is a floor plan of a digitally reconstructed 3D environment according to at least one embodiment. A user 600 uses an AR client to scan an AR viewing position and generate a digitally reconstructed 3D environment. The digital reconstruction includes two doorways 602 and 604, a dressing table 606, a sofa 608, and a chair 610. Figure 2The AR headset worn by the user 600 determines his posture and sends the posture and the digital reconstruction to the AR content server 108 .

[0072] Figure 7 is the requested AR content according to at least one embodiment and Figure 6 6 is an illustration of a digitally reconstructed 3D environment. A user 600 has requested AR content depicting nine stars 700 (e.g., in an AR planetarium application). The coordinate system of the AR content is aligned with the real-world viewing position using the digitally reconstructed 3D environment and the pose information. The alignment can be performed in a variety of ways. For example, the alignment can be selected to minimize the amount of AR content (e.g., the number of stars) that will not be displayed due to occlusion. The alignment can be selected using user input, for example, the user can adjust the alignment until a preferred alignment is selected. The alignment can be selected based on other inputs, for example, one or more accelerometers in the AR display device can be used to ensure that the "up" direction with respect to the AR content is aligned with the "up" direction of the AR viewing environment. In some embodiments, an alignment can be selected so that the direction the user is facing at the start of the AR session can be aligned with the preferred viewing direction of the AR content.

[0073] Based on the selected alignment, the display position of each element of the AR content (eg, the display position of each star) may be determined by transforming the coordinates of these elements from the coordinate system of the AR content to the coordinate system of the 3D viewing environment using the selected alignment.

[0074] Figure 8 According to at least one embodiment Figure 7 6. The AR content server 108 performs visibility analysis on the requested AR content. The AR content server 108 performs visibility analysis using the digitally reconstructed 3D environment and pose information. The AR content server 108 determines that the four stars 802, 804, 806, and 808 included in the original AR content are not visible from the viewpoint of the user 600 and removes them from the content.

[0075] Fig. 9 is a modified AR content according to at least one embodiment and Figure 6 The user has requested AR content depicting nine stars, but the AR content server has modified the AR content and now only five stars appear in the modified data stream.

[0076] Fig.10 According to at least one embodiment Figure 5 of users see Fig. 9 An illustration of the modified AR content. Fig.10The real world environment enhanced by the modified AR content is shown. The five remaining stars 1002, 1004, 1006, 1008 and 1010 are all visible. The four removed stars 802, 804, 806 and 808 are not blocked by walls, chairs, etc., but do not appear in the data stream.

[0077] The example embodiments disclosed herein may be implemented using one or more wired and / or wireless network nodes, such as wireless transmit / receive units (WTRUs) or other network entities.

[0078] Fig.11 1 is a system diagram of an exemplary WTRU 1102 that may be used as an AR client viewer in an embodiment of the present application. Fig.11 As shown in FIG, the WTRU 1102 may include a processor 1118, a communication interface 1119 including a transceiver 1120, a transmit / receive component 1122, a speaker / microphone 1124, a keyboard 1126, a display / touchpad 1128, a non-removable memory 1130, a removable memory 1132, a power source 1134, a global positioning system (GPS) chipset 1136, and a sensor 1138. It should be appreciated that the WTRU 1102 may include any sub-combination of the foregoing components while remaining consistent with an embodiment.

[0079] The processor 1118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like. The processor 1118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 1102 to operate in a wireless environment. The processor 1118 may be coupled to the transceiver 1120, which may be coupled to the transmit / receive element 1122. Although Fig.11 The processor 1118 and the transceiver 1120 are depicted as separate components, however it should be appreciated that the processor 1118 and the transceiver 1120 may also be integrated into one electronic component or chip.

[0080] The transmit / receive component 1122 may be configured to transmit or receive signals to or from a base station via the air interface 1116. For example, in one embodiment, the transmit / receive component 1122 may be an antenna configured to transmit and / or receive RF signals. As an example, in another embodiment, the transmit / receive component 1122 may be a transmitter / detector configured to transmit and / or receive IR, UV or visible light signals. In yet another embodiment, the transmit / receive component 1122 may be configured to transmit and receive RF and light signals. It should be appreciated that the transmit / receive component 1122 may be configured to transmit and / or receive any combination of wireless signals.

[0081] In addition, although Fig.11 1102 as a single component, but the WTRU 1102 may include any number of transmit / receive components 1122. More specifically, the WTRU 1102 may employ MIMO technology. Thus, in one embodiment, the WTRU 1102 may include two or more transmit / receive components 1122 (e.g., multiple antennas) for transmitting and receiving radio signals over the air interface 1116.

[0082] The transceiver 1120 may be configured to modulate signals to be transmitted by the transmit / receive component 1122 and to demodulate signals received by the transmit / receive component 1122. As described above, the WTRU 1102 may have multi-mode capabilities. Thus, the transceiver 1120 may include multiple transceivers that allow the WTRU 1102 to communicate via multiple RATs, such as UTRA and IEEE 802.11.

[0083] The processor 1118 of the WTRU 1102 may be coupled to a speaker / microphone 1124, a keyboard 1126, and / or a display / touchpad 1128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit), and may receive user input data from these components. The processor 1118 may also output user data to the speaker / microphone 1124, the keyboard 1126, and / or the display / touchpad 1128. In addition, the processor 1118 may access information from and store information in any suitable memory such as a non-removable memory 1130 and / or a removable memory 1132. The non-removable memory 1130 may include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 1132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 1118 may access information from, and store data in, memory that is not physically located on the WTRU 1102, such as on a server or a home computer (not shown).

[0084] The processor 1118 may receive power from the power source 1134, and may be configured to distribute and / or control power for use by the other components in the WTRU 1102. The power source 1134 may be any suitable device for powering the WTRU 1102. For example, the power source 1134 may include one or more dry cell batteries (e.g., nickel-cadmium (Ni-Cd), nickel-zinc (Ni-Zn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0085] The processor 1118 may also be coupled to the GPS chipset 1136, which may be configured to provide location information (e.g., longitude and latitude) related to the current location of the WTRU 1102. In addition to or in lieu of the information from the GPS chipset 1136, the WTRU 1102 may receive location information from a base station via the air interface 1116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It should be appreciated that the WTRU 1102 may acquire location information via any suitable positioning method while remaining consistent with an embodiment.

[0086] The processor 1118 may also be coupled to other peripherals 1138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 1138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, modules, FM radio units, digital music players, media players, video game console modules, and Internet browsers, among others.

[0087] Fig.12 An exemplary network entity 1290 that can be used in embodiments of the present disclosure is shown, for example, as an AR content server or as an AR content storage. Fig.12 As shown, the network entity 1290 includes a communication interface 1292 , a processor 1294 , and non-transitory data storage 1296 , all of which are communicatively linked via a bus, network, or other communication path 1928 .

[0088] The communication interface 1292 may include one or more wired communication interfaces and / or one or more wireless communication interfaces. With respect to wired communication, the communication interface 1292 may include one or more interfaces, such as an Ethernet interface. With respect to wireless communication, the communication interface 1292 may include multiple components, such as one or more antennas, one or more transceivers / chipsets designed and configured for one or more types of wireless communication (e.g., LTE), and / or any other components deemed appropriate by those skilled in the art. Further, with respect to wireless communication, the communication interface 1292 may be scaled to be equipped with a configuration (as opposed to the client side) suitable for use as wireless communication (e.g., LTE communication, Wi-Fi communication, etc.) on the network side. Therefore, the communication interface 1292 may include suitable devices and circuits (possibly including multiple transceivers) for serving multiple mobile stations, UEs, or other access terminals in the coverage area.

[0089] Processor 1294 may include one or more processors of any type deemed suitable by one skilled in the art, with some examples including general purpose microprocessors and special purpose DSPs.

[0090] Data storage 1296 may be in the form of any non-transitory computer-readable medium or a combination of these media, some examples of which include flash memory, read-only memory (ROM), and random access memory (RAM), as well as any one or more types of non-transitory data storage that a person skilled in the art would recognize as being usable. Fig.12As shown, data store 1296 contains program instructions 1297 executable by processor 1294 for performing various combinations of the various network entity functions described herein.

[0091] In the above description, specific embodiments have been described. However, it will be appreciated by those skilled in the art that various modifications and changes may be made without departing from the scope of the invention set forth in the claims. Therefore, the description and drawings are to be regarded as illustrative and not restrictive, and all such modifications are included within the scope of the teachings of the present invention.

[0092] Benefits, advantages, solutions to problems that exist or become apparent, and any element that may bring about any benefit, advantage, or solution is not to be construed as a critical, required, or essential feature or element of any or all claims. The present invention is defined solely by the appended claims including any amendments made during the pendency of this application and all equivalents of the claims as issued.

[0093] Additionally, in this application document, relational terms (e.g., first and second, top and bottom, etc.) may be used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual relationship or order between these entities or actions. The terms "comprises," "comprising," "has," "having," "includes," "including," "contains," "containing," or any other variation thereof, are intended to include open inclusion, whereby a process, method, article, or apparatus that includes, has, contains, or contains the listed elements includes not only these elements, but may also include other elements not expressly listed or inherent to these processes, methods, articles, or apparatuses. An element that is "includes," "has," "including," or "containing" is not excluded from the element that is included, has, contains, or contains, without further limitation. The process, method, article or device of the present invention has other identical elements. The terms "a" and "an" are defined as one or more, unless otherwise explicitly stated in this application. The terms "substantially", "essentially", "approximately", "about" or any other version thereof are defined as close to what is understood by a person skilled in the art, and in one non-limiting embodiment the term is defined as within 10%, within 5% in another embodiment, within 1% in another embodiment, and within 0.5% in another embodiment. The term "coupled" used in this application is defined as connected, but not necessarily directly connected and not necessarily mechanically connected. A device or structure that is "configured" in a certain manner is configured in at least that manner, but may also be configured in a manner not listed.

[0094] It is understood that some embodiments include one or more general or special processors (or "processing devices"), such as microprocessors, digital signal processors, custom processors, and field programmable gate arrays (FPGAs) and unique stored program instructions (including software and firmware) that control one or more processors in conjunction with certain non-processor circuits to perform some, most, or all of the functions of the methods and / or devices described herein. Alternatively, some or all of the functions may be implemented by a state machine without stored program instructions, or in one or more application specific integrated circuits (ASICs), where each function or some combination of certain functions is implemented as custom logic. Of course, a combination of both approaches may be used.

[0095] Therefore, some embodiments of the present disclosure or parts thereof can combine one or more processing devices and one or more software components (e.g., program code, firmware, resident software, microcode, etc.) stored in a tangible computer-readable storage device, and these combinations form a specially configured device for performing the functions described in this application. These combinations of specially programmed devices can be collectively referred to as "modules" in this application. The software component part of the module can be written in any computer language and can be part of a monolithic code library, or can be developed in more discrete code parts (e.g., typically in an object-oriented computer language). In addition, the module can be distributed to multiple computer platforms, servers, terminals, etc. A given module can even be implemented as a separate processor device and / or computing hardware platform to perform the described functions.

[0096] In addition, the embodiment can be implemented as a computer-readable storage medium, on which a computer-readable code is stored for programming a computer (e.g., including a processor) to perform the method described and required by the present application. Examples of computer-readable storage media include, but are not limited to, hard disks, CD-ROMs, optical storage devices, magnetic storage devices, ROMs (read-only memories), PROMs (programmable read-only memories), EPROMs (erasable programmable read-only memories), EEPROMs (electrically erasable programmable read-only memories), and flash memory. In addition, it is foreseeable that those skilled in the art, although they may make significant efforts and many design choices, for example, by means of available time, current technology, and economic considerations, can easily generate these software instructions and programs and ICs under the guidance of the concepts and principles disclosed in the present application under minimal experimental conditions.

[0097] The abstract of the present disclosure is provided to allow the reader to quickly ascertain the essence of the technical disclosure. It should be understood that it is not used to interpret or limit the scope or meaning of the claims. In addition, in the aforementioned detailed description section, it can be seen that various features are combined in various embodiments to streamline the present disclosure. This disclosure method is not interpreted as reflecting that the required implementation requires more features than those explicitly stated in each claim. On the contrary, the inventive subject matter reflected in the claims is less than all the features of a single disclosed embodiment. Therefore, the claims are integrated into the detailed description here, and each claim itself is a subject matter claimed for protection alone.

Claims

1. An augmented reality client method, comprising: Using an augmented reality (AR) client to obtain a 3D environment model of a real-world AR viewing location; determining a posture of the client using the 3D environment model; sending a request for AR content to a server, the request including information indicating the pose of the client and also including a compact representation of the 3D environment model; as well as AR content is received from the server, the received AR content excluding occluded portions of the requested AR content based on the pose of the client and the compact representation of the 3D model of the environment.

2. The method according to claim 1, wherein: The 3D environment model includes a plurality of real-world objects, and wherein the compact representation includes fewer than all of the real-world objects. 3 . The method of claim 1 , wherein the compact representation comprises a spherical depth map of at least a portion of the AR viewing location. The method of claim 1 , wherein a type of compact format is selected based on a content type of the requested AR content.

5. The method according to claim 1, wherein: The compact representation of the 3D environment model excludes at least a portion of the 3D environment model. The method of claim 1 , wherein the requested AR content includes a plurality of virtual objects, and wherein the received AR content excludes at least one of the plurality of virtual objects.

7. The method of claim 1, wherein obtaining a 3D environment model of a real-world AR viewing position comprises: The real-world AR viewing location is scanned.

8. An augmented reality client device, comprising one or more processors, wherein the one or more processors are configured to at least execute: Using an augmented reality (AR) client to obtain a 3D environment model of a real-world AR viewing location; determining a posture of the client using the 3D environment model; sending a request for AR content to a server, the request comprising information indicative of the pose of the client and also comprising a compact representation of the 3D environment model; and AR content is received from the server, the received AR content excluding occluded portions of the requested AR content based on the pose of the client and the compact representation of the 3D model of the environment.

9. The device according to claim 8, wherein: The 3D environment model includes a plurality of real-world objects, and wherein the compact representation includes fewer than all of the real-world objects.

10. The apparatus of claim 8, wherein the compact representation comprises a spherical depth map of at least a portion of the AR viewing location. The apparatus of claim 8 , wherein a type of compact format is selected based on a content type of the requested AR content.

12. The device according to claim 8, wherein: The compact representation of the 3D environment model excludes at least a portion of the 3D environment model. 13 . The device of claim 8 , wherein the requested AR content includes a plurality of virtual objects, and wherein the received AR content excludes at least one of the plurality of virtual objects.

14. The apparatus of claim 8, wherein obtaining a 3D environment model of a real-world AR viewing position comprises: The real-world AR viewing location is scanned.

15. An augmented reality server method, comprising: receiving a 3D environment model of a real-world AR viewing location and information indicating a position of a client device in the viewing location; receiving a request for the stored virtual content from the client device; In response to the request, obtaining the requested virtual content from a data storage, the stored virtual content including a plurality of virtual objects; based on the indicated position of the client device, modifying the stored virtual content to remove at least one virtual object having a position obscured in the viewing position; as well as The modified virtual content is sent to the client device.

16. The method of claim 15, wherein the 3D environment model includes a spherical depth map of at least a portion of the AR viewing location.

17. An augmented reality server device, comprising one or more processors, wherein the one or more processors are configured to at least execute: receiving a 3D environment model of a real-world AR viewing location and information indicating a position of a client device in the viewing location; receiving a request for the stored virtual content from the client device; In response to the request, obtaining the requested virtual content from a data storage, the stored virtual content including a plurality of virtual objects; based on the indicated position of the client device, modifying the stored virtual content to remove at least one virtual object having a position obscured in the viewing position; as well as The modified virtual content is sent to the client device.

18. The apparatus of claim 17, wherein the 3D environment model comprises a spherical depth map of at least a portion of the AR viewing location.

19. An immersive video client method, comprising: Scanning a real-world AR viewing position using an augmented reality (AR) client to obtain a spherical depth map of the viewing position; sending the spherical depth map and a request for immersive video to a server; as well as An immersive video is received from the server, the received immersive video excluding a portion of the requested immersive video having a depth value greater than a corresponding depth value in the spherical depth map.

20. An immersive video server method, comprising: At a server, receiving a spherical depth map and a request for an immersive video; In response to the request, obtaining the requested immersive video from a data store; Comparing the acquired depth value of the immersive video with the corresponding depth value of the spherical depth map; as well as In response to the request, a modified immersive video is sent, the modified immersive video excluding portions of the acquired immersive video having depth values ​​greater than the corresponding depth values ​​in the spherical depth map.

Images