2D video with the option for projected viewing in a modeled 3D space

Abstract

Methods in a display system with a head-mounted display (HMD) that includes the following: Receiving video data that represents two-dimensional video content (2D video content); Receiving model data that represents a modeled three-dimensional space (3D space) and represents an area within the modeled 3D space onto which the 2D video content is to be projected; Determine in the display system whether a 2D or a 3D view mode is selected; and in response to the determination that the 3D view mode is selected: Determining the current HMD stance; Generating an initial stereoscopic pair of images that represents a 3D view of the space from the current position; Generating a second stereoscopic pair of images, representing a transformation of the video data into the current posture of the HMD from a flat 2D coordinate reference, to fit the surface in a coordinate reference of the 3D view of the space from the current posture, wherein the second stereoscopic pair of images obscures an area of ​​the first stereoscopic pair of images, the area corresponding to the surface within the modeled 3D space; and Providing a view of the modeled 3D space via the HMD based on the current posture, wherein the view of the modeled 3D space contains a projection of the 2D video content onto the surface based on the current posture.

Description

Background Area of ​​Revelation

[0001] The present disclosure relates generally to multimedia systems and in particular to display systems for virtual reality and enhanced reality. Description of the state of the art

[0002] Multimedia content servers, multimedia disc players, and other multimedia content sources support the viewing of two-dimensional video content (2D video content) on televisions, monitors, computer-enabled mobile phones, computer-enabled watches, and other display devices. With the introduction of user-level virtual reality (VR) and augmented reality (AR) systems, such 2D video presentations often suffer from a lack of context or "presence" compared to the VR and AR display content. However, in conventional display systems, users are limited to either viewing 2D video content in the conventional flat-screen mode or seeing completely context-free VR or AR content. EP 2 966 620 A2 describes a display method for showing an illuminated object in a virtual environment.Patent specification US 8,878,846 B1 describes methods and systems for superimposing views of three-dimensional objects with real-time images. Patent specification US 5,640,171 B1 presents an image display device whose LCD panel is configured to display a combined video signal and a stereoscopic display field-sequentially. The method described in US 2012 / 229,508 A1 involves expanding a photo-representative view based on a selected subject. US 2016 / 286,208 A1 deals with methods and systems for switching between a two-dimensional video view and a view of a three-dimensional environment. Brief description of the drawings

[0003] The present disclosure can be better understood by referring to the accompanying drawings, and its numerous features and advantages will be revealed to those skilled in the art. The use of the same reference numerals in different drawings indicates similar or identical elements. Fig. Figure 1 is a block diagram representing a multimedia distribution system in accordance with some embodiments. Fig. Figure 2 is a diagram that represents a projection of 2D video content into a specified location in a modeled 3D space in accordance with some embodiments. Fig. Figure 3 is a diagram representing a multimedia application that provides 2D / 3D display options for a 2D video stream in accordance with some embodiments. Fig. Figure 4 is a diagram showing a 2D or "flat" display of a 2D video stream in response to the selection of a 2D display option in the multimedia application of Fig. 3 represents in accordance with some embodiments. Fig. Figure 5 is a diagram that displays a 2D video stream projected onto an identified area of ​​a modeled 3D space in response to the selection of a 3D display option in the multimedia application of Fig. 3 represents in accordance with some embodiments. Fig. Figure 6 is a diagram illustrating a processing system of a device for generating a representation of a modeled 3D space in association with 2D video content in accordance with some embodiments. Fig. Figure 7 is a flowchart that presents a method for generating a representation of a modeled 3D space and specifying a projection surface for 2D video content within that modeled space in accordance with some embodiments. Fig. Figure 8 is a diagram illustrating a processing system of a device for providing 2D / 3D options for 2D video content with an associated modeled 3D space in accordance with some embodiments. Fig. Figure 9 is a flowchart that illustrates a method for the optional 2D / 3D display of 2D video content with an associated modeled 3D space in accordance with some embodiments. Detailed description

[0004] Conventional display systems limit the viewing of 2D video content to the traditional 2D or "flat" viewing mode, in which the 2D video content is displayed in a corresponding window or 2D area of ​​an operating system (OS) or a software application display area. In contrast, AV / VR systems allow the viewing of modeled 3D spaces or other 3D content, and many such systems enable a user to effectively "move" through a 3D space by tracking the attitude (location and / or orientation) of the VR / AR equipment using various sensors.As described in more detail here, a 2D / 3D multimedia distribution system allows a user to view 2D video content in the conventional 2D or "flat" viewing mode. Alternatively, if the system is VR / AR-enabled and the user is inclined to do so, a representation of a 3D space can be provided or accessed in association with the 2D video content. The 2D video content can be projected onto a specified surface within the modeled 3D space for the user to view, as if the 2D video were displayed on that surface within the modeled 3D space or projected from it in some other way. Furthermore, in some embodiments, the posture (location and / or orientation) of the VR display device, such as a head-mounted display (HMD), can be monitored, and the projection of the 2D video content can be updated accordingly to align with the current posture of the HMD and the view into the modeled 3D space.The user's posture determines the posture of the HMD. Thus, by providing a display system that allows the user to choose between a 2D and a 3D view mode, and which receives data representing a modeled 3D space, and projects the 2D video content onto a surface within the modeled 3D space when the 3D view mode is selected, the system ensures that the display of the 2D video content is relevant within the modeled 3D space and provides the user with an enhanced user experience. Furthermore, this approach allows for flexibility in the types of systems viewing the 2D video content, as 3D-capable display systems can view the 2D video content in the 3D view mode, while still allowing display systems that are not capable of viewing 3D video content to display the 2D video content in the conventional 2D manner.As such, the techniques disclosed here enable 3D-capable display systems to have an improved user experience while providing backward compatibility for display systems that are not 3D-capable.

[0005] To illustrate this with a non-restrictive example, a YouTube™ user who regularly uploads 2D video clips of the user playing video games can model the user's game room and upload a representation of the resulting 3D model of the game room, corresponding to one or more of the 2D video clips. The 2D video clips can then be streamed as video streaming files to other users via the YouTube™ service.Depending on the VR / AR capabilities of their respective computer systems, these other users can choose to view one of the 2D video clips in a conventional flat 2D display mode in a web browser, or to view one of the 2D video clips via a head-mounted display (HMD) or other VR display device. In this case, the 2D video clip is projected onto a surface corresponding to a computer monitor in the 3D model of the playroom, thus appearing as if the 2D video clip were displayed "on" the computer monitor within the 3D model of the playroom. If a user has 3 degrees of freedom (3DoF) or 6 degrees of freedom (6DoF) capability via the VR display device, the user can change the location and / or orientation of the computer monitor and the 2D video content that appears to be displayed on it.

[0006] Fig. Figure 1 represents a 2D / 3D multimedia distribution system 100 in accordance with at least one embodiment of the present disclosure. In the illustrated example, the system 100 includes a multimedia server 102, which is connected to several user computer systems, such as user computer systems 104 and 106, via one or more wired or wireless networks. The user computer systems 104 and 106 may comprise any of a variety of devices or combinations of devices used to receive and / or present multimedia content, such as desktop computers, notebook computers, tablet computers, computer-enabled mobile phones, personal digital assistants, game consoles, computer-enabled televisions, computer-enabled watches or other wearables, HMDs or other AR / VR display devices, and the like.

[0007] For the following, the user computer system 104 is used as a source of video content and a source of a 3D model of a virtual or real space and is thus referred to here as the content source system 104, while the user computer system 106 is used to present 2D video content to a user in either a 2D viewing mode or a 3D viewing mode and is thus referred to here as the 2D / 3D display system 106. The 2D / 3D display system 106 includes a 2D display device 110 for displaying 2D video content in a 2D / flat display mode, wherein the 2D video content is displayed as a rectangular 2D window on a display panel or monitor of the 2D display device 110. Examples of such devices include desktop / notebook computers, computer-enabled mobile phones, tablet computers, and the like.Furthermore, in some cases, the 2D / 3D display system 106 may also include a VR / AR display device 112 capable of displaying 3D video content. The VR / AR display device 112 may, for example, comprise a separate dedicated HMD or may be implemented by configuring the 2D display device 110 in a special way (such as the Google Cardboard VR view, in which a computer-enabled mobile phone is inserted into a cardboard headgear to form an HMD). In such cases, the 2D / 3D display system 106 is capable of operating in either a 2D view mode, in which 2D video content is displayed in a 2D window, or in a 3D view mode, in which a modeled 3D space is presented via the VR / AR display device 112 and, as described in more detail below, 2D video content is projected onto a specified surface of the modeled 3D space.It should be noted that in some cases devices 110, 112 may comprise the same device.

[0008] During operation, the content source system 104 generates or otherwise obtains 2D video data 114 representing 2D video content and makes the 2D video data 114 available to the multimedia server 102 for storage and subsequent distribution to other users. The 2D video data 114 may, for example, comprise a video clip or other video stream recorded by a user of the content source system 104 (referred to here as the "content provider"). Additionally, the content provider manipulates the content source system 104 to generate or otherwise obtain a 3D model of a virtual or real space that can serve as a background for the 3D display of the 2D video content and provides a representation of this 3D model as 3D model metadata 116 to the multimedia server 102 for distribution to recipients of the 2D video data 114.The 3D model metadata can be integrated as part of the 2D video data. To illustrate this, the 2D video data can be implemented as H.264 video data, and the 3D model metadata can be integrated as metadata into the H.264 data using a variety of metadata insertion techniques. Alternatively, the 3D model metadata can be encoded into the pixels of the video content itself using a variety of steganography tools that embed data within pixel information. Or, the 3D model metadata can be provided separately from, but in association with, the 2D video data. To illustrate this, the 2D video data can contain or reference an identifier (ID) that identifies the 3D model metadata. This ID can, for example, contain the name of the file or other data container that contains the 2D video data 114.As another example, the ID can be provided in the form of an ID of a remote storage location, such as a Hypertext Transfer Protocol (HTTP) link that is mapped to the 3D model metadata location 116.

[0009] The 3D model of the identified virtual space (hereinafter referred to as VR space) or of the real space can be generated in any of a variety of ways. In some embodiments, the Content Source System 104 has the ability to capture images and / or depth information for a real room or other real space and to use this information to generate a VR model of the real space. To illustrate this, the Content Source System 104 can be implemented as a tablet or mobile phone capable of running Google Project Tango, to design one or more rooms with real-world geometry, and to combine this data with captured image data to generate a textured 3D model of the one or more rooms.In other embodiments, the 3D model can be a complete VR space created using a computer-aided design (CAD) tool or another VR modeling tool.

[0010] Using the 3D model thus determined, the content provider further operates the content source system 104 to identify a surface within the modeled 3D space onto which the 2D video content can effectively be "projected" so that it appears as a 2D video "displayed" on this surface within the modeled space. This surface is referred to here as the "projection surface." In some embodiments, the projection surface may, for example, comprise a rectangular planar surface, the four corners of which are identified by four corresponding 3D coordinate points in the reference coordinate system of the 3D model. In other embodiments, the surface may take on other planar shapes with corners and / or edges defined by the corresponding 3D coordinate points. In still other embodiments, the projection surface may have a non-planar shape, such as...a curved surface or a faceted surface in which instances of the projection surface can be identified using a geometric grid or other representation.

[0011] To illustrate this, the surface can be specified as a point in a 3D grid space representing the environment, along with a normal vector indicating the direction the projection surface should point and a marker for a maximum width / height (e.g., in meters) within this grid space. A rectangular 2D video with any degree of curvature and an arbitrary aspect ratio could be played back between these. Alternatively, a content provider could supply a 2D grid and texture coordinates onto which their video should be played back, positioned in the same space as the environment grid. This grid would typically need to be modified to change depending on the aspect ratio of the current video to avoid letterboxing.This would allow content providers to generate pseudo-6DoF video with a static grid by dividing the video texture into distinct areas and then providing a potentially disjoint grid onto which the video would be displayed. This enables a simple setup where a content provider could display two video screens in space with separate content simply by having a grid that places two squares in the environment and provides texture coordinates so that each square receives a different half of the video, effectively allowing for two video surfaces.This could also be implemented with arbitrary grids; for example, an environment could contain a person sitting on a couch, with a static texture as part of the environment, but then the content provider could place part of the video texture over the face (with a grid that matches that face) so that the face can react to the video.

[0012] Using the projection surface thus identified, the content source system 104 integrates a representation of the projection surface into the 3D model metadata 116 as projection location information 118. Depending on the properties of the projection surface, the projection location information 118 may, for example, include representations of the points or vertices of the projection surface, a representation of the geometric grid representing the projection surface, or a 3D point and a vector as mentioned above, and the like.

[0013] With the 2D video data 114 and the 3D model metadata 116 uploaded to the multimedia server 102, other users are then able to access the 2D video data 114 for display on their respective display systems 106. Thus, when a user selects the 2D video content represented by the 2D video data 114 for playback (e.g., by clicking on an associated HTTP link in a hypertext markup language (HTML) page viewed in a web browser displayed on the display system 106), the multimedia server 102 initiates the transmission of the 2D video data 114 to the display system 106. The 2D video data 114 may contain a marker indicating that it offers a 3D display option in addition to the conventional 2D display option. For example, a header or other field in the container for the 2D video data 114 can contain a value representing the availability of this option.In response to identifying the availability of the 3D view mode for the 2D video data 114, the display system 106 determines whether it has the capability to display 2D content, e.g. by determining whether the display system 106 includes a VR / AR display device 112. If 3D display capability is available, the display system 106 can provide the user with the option to view the 2D video content of the 2D video data 114 in a conventional 2D viewing mode, in which case the 2D video content is displayed in a rectangular 2D window or box, or in a 3D viewing mode, in which case the VR / AR display device 112 is used to immerse the user in the modeled 3D space represented by the 3D model metadata 116, and the 2D video content is projected into the projection surface within the modeled 3D space, as specified by the projection location information 118.These display modes are described here in more detail.

[0014] Fig. Figure 3 presents an example of a modeled 3D space with a defined projection surface in accordance with several embodiments. A content provider (that is, a user of the content source system 104) creates a 3D model 202 of a space, either by capturing images and depth information for a real space and "virtualizing" the geometry contained therein, or by constructing a virtual space from scratch. In the illustrated example, the modeled space is the user's living room, which includes a modeled representation of a television set 204. The content provider further defines the display surface of the television set 204 as a projection surface 206 for projecting 2D video content into the modeled 3D space represented by the 3D model 202.The projection surface 206 can be identified within the 3D space 202, for example, by specifying the 3D coordinates within the coordinate frame of the 3D model 202 of the four corners 211, 212, 213, 214 of the rectangular planar surface represented by the projection surface 206.

[0015] The content provider also specifies a 2D video stream 216 for display either in a conventional flat 2D viewing mode or for projection onto the projection surface 206 for a 3D viewing mode. The 2D video stream 216 comprises a sequence of 2D images such as the 2D images 217, 218, and 219 shown. Accordingly, as described in more detail here, when a content consumer (that is, a user of the 2D / 3D display system 106) accesses the 2D video data 114 and 3D model metadata 116 from the multimedia server 102 ( Fig. 1) accesses and selects the 3D view mode, a 3D representation 220 of the 3D model 202 is displayed to the content consumer via the VR / AR display device 112, wherein the perspective of the 3D model 202 may be based on the current posture of the VR / AR display device 112 if the VR / AR display device 112 is capable of tracking posture. To provide the projected 3D view of the 2D video stream within the 3D representation 220, the 2D / 3D display system 106 uses any one of a variety of coordinate system transformation algorithms to transform each frame of the 2D video stream 216 from a 2D coordinate frame into a corresponding texture 222 in the coordinate frame of the 3D model, and the resulting texture 222 is integrated into the projection surface 206 of one or more images 224 displayed in the VR / AR display device 112 to provide the display of the 3D representation 220.

[0016] As explained above, a content consumer (that is, a user of System 106) may not have the equipment or the desire to view the 2D video content projected into a 3D space, and thus the content consumer can be provided with the option to choose between a 2D viewing mode and a 3D viewing mode for the 2D video content. Fig. Figure 3 presents an exemplary approach to providing this option, to select the display mode, to the content consumer. Fig. Figure 4 shows an exemplary approach for the 2D view mode when it is selected, and Fig. Figure 5 illustrates an exemplary approach for the 3D view mode when it is selected. As in Fig. As shown in Figure 3, the content consumer can initially access a video streaming file or other video stream in the 2D / 3D display system 106 by inputting a graphical user interface (GUI) of a software application 300 or by otherwise indicating a desire to view it. In the example shown, the software application 300 comprises a web browser that has accessed a video-sharing website, such as the YouTube™ website (served, for example, by the multimedia server 102). The content consumer selects a 3D-viewable video clip via the web browser. In response, a 2D display window 302 is presented, along with a view selection box 304 (shown in this example as a drop-down menu) that identifies the view modes available for the video clip, namely a 2D view mode and a 3D view mode.In cases where the 2D / 3D display system 106 is not capable of providing 3D display, or in cases where a selected clip is not 3D-displayable, the view selection box 304 may not represent the 3D view mode as an option or may prevent the selection of the 3D view mode. The content consumer can then manipulate the view selection box 304 to choose whether the 2D video clip should be displayed in a flat 2D presentation or by projecting the 2D video content onto a projection surface of a modeled 3D space associated with the video clip.

[0017] As through Fig. As shown in Figure 4, when the content consumer selects “2D” from the drop-down menu representing the view selection box 304 to view the 2D video clip in 2D view mode, the software application 300 (as a web browser) accesses the 2D video data 114 and makes the 2D video content 402 represented therein available for display in the 2D display window 302 of the software application 300 on a display panel 404 of the 2D / 3D display system 106 using conventional 2D video display techniques. In contrast, as shown in Figure 4, Fig. As shown in Figure 5, when the content consumer selects “3D” from the drop-down menu representing the view selection box 304 to view the 2D video clip in 3D view mode, the software application 300 activates the 3D view mode, in which the software application 300 accesses the 3D model metadata 116 associated with the video clip, and the same software application 300 or another software application then provides a 3D display of the modeled 3D space represented by the 3D model metadata 116 in the VR / AR display device 112 (e.g., as parallel streams of stereoscopic images, as in Fig. 5 is shown). As explained above, this 3D display process involves transforming each frame of the 2D video clip into two corresponding surfaces (one for the left eye image and one for the right eye image) for integration into the parallel streams of stereoscopic images, so that the image appears as if it is being displayed on or projected from a projection surface specified for the modeled 3D space.

[0018] Fig. 6 presents an example implementation of the content source system 104 of the multimedia distribution system 100. Fig. 1 in accordance with some embodiments. In the illustrated example, the content source system 104 includes at least one processor 602, one data memory 604, one system memory 606 or another software memory component, and a network interface 608 to the network 108 ( Fig. 1) Furthermore, in some embodiments, the content source system 104 provides optical flow, simultaneous localization and mapping (SLAM) or other image acquisition-based telemetry functionality and can thus include a group of image acquisition cameras 610, 612, a depth sensor 614, an inertial measurement unit (IMU) 616, and a sensor hub 618 for controlling the components 610-616. The IMU 616 can, for example, include one or more of a gyroscope 620, a magnetometer 622, an accelerometer 624, and the like.

[0019] System memory 606 stores a software application 626 (an embodiment of software application 300 of Fig. 3) which contains executable instructions which the at least one processor 602 accesses and executes, and which, when executed, manipulate the processor 602, together with the other components, to provide the various functionalities described herein with respect to the content source system 104. For this purpose, the software application 626 is designed as a group of modules comprising a 3D modeling module 628 and a video projection placement module 630.

[0020] Fig. 7 presents an example procedure 700 of the operation of the implementation of the content source system 104, which is in Fig. Figure 6 illustrates the process for generating and providing the 2D video data 114 and the 3D model metadata 116 for distribution to other systems in accordance with some embodiments. The process 700 starts at block 702 with the identification, generation, or other determination of 2D video content to be made available for either 2D or 3D display. In some embodiments, the determination of 2D video content may include the selection of previously generated 2D video content. For example, a content provider may select a 2D video clip from a database of 2D video clips stored in the data store 604 for 2D / 3D processing as described herein. In other embodiments, the 2D video content is expressly generated to be available for the 2D / 3D display option process described herein.In such cases, the 2D video content can be captured from the image capture cameras 610 and 612, generated using a computer graphics application, or a combination thereof. If the 2D video content is generated at block 702, it can be made available for storage as 2D video data 114 in the data storage device 604. Furthermore, at block 702, the generated or selected 2D video data 114 can be transferred via the network interface 608 to the multimedia server 102 for storage and subsequent distribution.

[0021] In block 704, the 3D modeling module 620 is executed in the content source system 104 to generate a 3D model of a specified space based on user input 632 during the 3D modeling process. As represented by block 703, the content provider may wish to generate a 3D model of a real-world space. To illustrate this, if the 2D video content represents the content provider playing a video game in a living room, the content provider may wish to generate a 3D model of that living room to give a viewer the feeling of being in the living room and watching the game being played.Accordingly, the 3D modeling process of Block 704 may, for example, include the 3D modeling module 628 using the image acquisition cameras 610 and 612 to capture images of real space, using the depth sensor 614 to acquire depth information for real space, and using the IMU 616 to determine posture information during image and depth acquisition. From this information, the module may determine various suitable spatial features of real space within the context of a reference coordinate frame associated with real space. The 3D modeling module 628 can then use these spatial features and this reference coordinate frame to generate a 3D model 634. Fig. 6) to generate the spatial geometry of real space. Alternatively, as represented by Block 705, the content provider may wish to generate a 3D model of a virtual space, in which case the 3D modeling module 628 can use or coordinate with a CAD tool or other VR tool to create a 3D model ( Fig. 6) to construct this virtual space.

[0022] With the specified modeled 3D space, the video projection placement module in the content source system 104 is executed in block 706 to perform the following based on user input 636 ( Fig. 6) To identify a projection surface within the modeled 3D space onto which the 2D video content can be presented or otherwise projected. To illustrate this, the content provider can specify the coordinates of the corners of a planar area within the modeled 3D space to serve as the projection surface.

[0023] In block 708, the video projection placement module 630 combines the data representing the 3D model 634 and the data representing the selected projection surface (i.e., the projection location information 118, Fig. 1) to generate the 3D model metadata 116, and provides the 3D model metadata 116 to the network interface 608 for transmission to the multimedia server 102 for subsequent distribution in association with the corresponding 2D video data 114. In some embodiments, the video projection placement module 630 provides the 3D model metadata 116 as part of the 2D video data 114, as in the following example: For example, by embedding the 3D video data 114 in metadata fields of the container for the 2D video data 114, or by directly encoding the 3D video data 114 into pixels of images of the 2D video data 114. Alternatively, the 3D model metadata 116 can be provided as a separate file or another data container, and the 2D video data 114 can contain an ID or other reference to the 3D model metadata 116, so that the multimedia server 102 can identify the 3D model metadata 116 as being associated with the 2D video data 114, and vice versa.In yet other embodiments, the 3D model metadata 116 can be associated with the content provider and thus apply to all 3D-displayable video clips provided by the content provider. To illustrate this, again using the YouTube™ example above, the content provider can model the living room in which the content provider plays video games, and any video game clip uploaded by the content provider can be available by default for projected 3D display within this modeled living room on the basis that the video clip originates from this particular content provider.

[0024] Fig. Figure 8 presents an example implementation of the 2D / 3D display system 106 of the multimedia distribution system 100. Fig. 1 in accordance with at least one embodiment. In the illustrated example, the system 106 includes at least one application processor 802, at least one graphics processing unit (GPU) 804, the system memory 806 or another memory component, and a network interface 808 for coupling to the network 108 ( Fig. 1) A display control unit 810, a display panel 812, and the VR / AR display device 112, which in the illustrated embodiment includes an HMD 814 and an HMD control unit 816. The system memory 806 stores one or more software applications that support the functionality of the 2D / 3D display system 106 as described herein, such as a main display software 818, which represents the one or more software applications that operate to manipulate the 2D / 3D display system 106 to display one or more GUIs to enable the display of a list, matrix, or other enumeration of 2D video streams available for selection by the user (e.g.,(via a web page displayed in a web browser) to enable access to the 2D video data 114 corresponding to a video stream selected by the user from the multimedia server 102, and the 2D display of the video content represented by the 2D video data 114, provided that the 2D viewing mode is selected for the chosen 2D video stream. For this purpose, the main display software 818 may, for example, include a playback module 820 to decode the 2D video data 114 and reproduce the decoded data as a series of images representing the video content of the 2D video data 114, as detailed below.

[0025] The software applications stored in the system memory 806 further include one or more VR software applications 822 that operate to manipulate the 2D / 3D display system 106 to reproduce parallel stereoscopic image streams for displaying VR or AR content in the HMD 814 via the HMD control unit 816. Furthermore, if the user chooses to view the video content in 3D view mode, the one or more VR software applications operate to support the display of the video content in 3D view mode. For this purpose, the one or more software applications 822 include a 3D model extraction / display module 824 and a 2D video projection module 826. The operation of the modules 824 and 826 is described in more detail below.

[0026] Fig. 9 presents an example procedure 900 of the operation for implementing the 2D / 3D display system 106 of Fig. 8 in accordance with at least one embodiment of the present disclosure. Method 900 starts at block 902 with the user selecting a 3D-displayable 2D video stream for presentation in the 2D / 3D display system 106 and receiving the 2D video data 114 for the selected 2D video stream in response to this selection. To illustrate this, the user can browse a video sharing webpage and select the video stream from several video streams presented as options. In response to this selection, the main display software application 818 sends a request for the corresponding 2D video data 114 to the multimedia server 102 via the network interface 808. The multimedia server 102 then sends the 2D video data 114 back to the 2D / 3D display system 106, where it is buffered for processing.

[0027] At block 904, the main display software application 818 determines whether the selected video stream is capable of 3D display and, if so, presents the user with the option to view the selected video stream in a flat-screen mode (i.e., the conventional 2D viewing mode) or a VR mode (i.e., the 3D viewing mode described here). The main display software application 818 can make this determination, for example, based on a field or value stored in the 2D video data 114 itself, based on the presence or absence of 3D model metadata 116 associated with the 2D video data 114, based on the source or content provider of the 2D video data 114, and the like.As described above, the option to view the video content in either 2D view mode or 3D view mode can be presented, for example, as a user input field or other GUI widget within the GUI of a video sharing web page or other software application that serves as the source of the selected 2D video stream.

[0028] In the case where the user selects flat screen mode, at block 906 the application processor 802 and / or the GPU 804 decodes the 2D video data 114 and displays a sequence of images represented by the resulting decoded video data, and each image is sent for display in a corresponding display window (e.g., display window 302 of Fig. 3) presented in the display panel 812 via the display control unit 810 in accordance with the conventional 2D display process.

[0029] Returning to Block 904, if the user selects the 3D view mode for the selected video stream, Block 908 accesses the 3D model extraction / display module 824 to the 3D model metadata 116 associated with the selected video stream. As explained above, in some embodiments the 3D model metadata 116 is embedded as metadata in the 2D video data 114 or encoded within the pixel data itself, in which case the 3D model extraction / display module 824 can access the 3D model metadata 116 by extracting it from the 2D video data 114.In other embodiments, the 3D model metadata 116 is made available separately from the 2D video data 114, taking into account bandwidth, compatibility, or flexibility. This enables backward compatibility by allowing the 2D video data 114 to be viewed on a conventional 2D-based display system (which can disregard the 3D model metadata 116), or by allowing the 2D video data to be viewed in 3D mode by a 3D-capable display system using the separately available 3D model metadata 116. To illustrate this, the 3D model metadata 116 can be stored as a separate file, which is referenced by the 2D video data 114 via an HTML link or another ID of a remote storage location.In such cases, the 3D model extraction / display module 824 can access the 3D model metadata 116 using the ID of the remote storage location to request the 3D model metadata 116 from the multimedia server 102. In other embodiments, the 3D model metadata 116 may have been pre-stored in the 2D / 3D display system 106. To illustrate this, the content provider can use the same modeled 3D space for all 2D video streams provided by the content provider, and thus, when a user views a 2D video stream, the 3D model metadata 116 representing the modeled 3D space can be downloaded and stored in the 2D / 3D display system 106 for use with both the current 2D video stream and any subsequent 2D video stream from the same content provider.In such cases, accessing the 3D model metadata 116 may involve identifying that a copy of the 3D model metadata 116 is already stored in a local location (e.g., a local hard drive), and accessing the 3D model metadata 116 from the identified local location.

[0030] It is noted that the VR / AR display device 112 has a refresh rate that represents the rate at which the HMD control unit 816 controls the left / right eye images to the HMD 814 for display. In one embodiment, the process described below for blocks 910, 912, 914, 916, and 918 is performed for each pair of stereoscopic display images to be provided to the HMD 814 for display, and thus iterations of this process are performed at a refresh rate of the VR / AR display device 112. Furthermore, it is assumed below that the refresh rate of the selected video stream is equal to the refresh rate of the VR / AR display device 112. However, if the refresh rate of the selected video stream is lower, the same image from the selected video stream can be used for two or more display frames for the VR / AR display device 112.Alternatively, if the frame rate for the selected video stream is higher, selective omission of frames from the video stream can be used to achieve an effective match between the frame rates.

[0031] At block 910, the playback module 820 decodes a section of the 2D video data 114, reproduces an image 828 from the resulting decoded data, and provides the image 828 to the 2D video projection module 826. In parallel, at block 912, the processor 802 determines a current 6DoF or 3DoF attitude 830 ( Fig. 8) of the HMD 814 based on sensor information from an IMU (not shown) of the HMD 814, based on spatial features detected in images captured by image acquisition cameras (not shown) of the HMD 814, and the like, using a variety of known HMD posture determination techniques.

[0032] In block 914, the 3D model extraction / display module 824 uses the 3D model of a room, represented by the 3D model metadata 116, and the current attitude 830 to reproduce a stereoscopic pair of images 831, 832 (one image each for the user's left and right eyes) that represents a 3D view of the room from the current attitude 830, and writes the pair of images 831, 832 to a frame buffer 834 for temporary storage.At the same time, in block 916, the video projection module 826 uses the current attitude 830 and the projection surface, represented by the projection location information 118 extracted from the 3D model metadata, to transform the rendered image 828 from the flat 2D coordinate reference to fit the projection surface in the coordinate reference of the modeled 3D space with respect to the current attitude 830, resulting in a pair of transformed image textures 835, 836, one each for the left and right eye. The transformed image textures 835, 836 are then written to and stored in the frame buffer 834, such that the transformed image texture 835 covers an area of ​​the image 831 corresponding to the position of the projection surface in the image 831, and the transformed image texture 836 similarly covers an area of ​​the image 832 corresponding to the position of the projection surface in the image 832.At block 918, the HMD control unit 816 accesses the frame buffer 834 and transmits the resulting final display images 837, 838 to the HMD 814 for simultaneous display, so that they are perceived by the user as 3D image content. This makes the 2D image 828 from the 2D video data 114 appear as if it is projected from a defined area within the space represented in the 3D image content. The process of blocks 910–918 can then be repeated for the next image from the 2D video data 114 and an updated attitude 830.

[0033] In some embodiments, specific aspects of the techniques described above can be implemented by one or more processors of a software-executing processing system. The software comprises one or more sets of executable instructions that are stored on a non-volatile, computer-readable storage medium or otherwise materially integrated. The software can contain the instructions and specific data that, when executed by the one or more processors, manipulate the one or more processors to perform one or more aspects of the techniques described above. The non-volatile, computer-readable storage medium can, for example, be a storage device with a magnetic or optical disk, solid-state storage devices such as...Flash memory, a cache, random access memory (RAM), or other non-volatile storage device(s), and the like. The executable instructions stored on the non-volatile, machine-readable storage medium can be source code, assembly language code, object code, or any other instruction format that is interpreted by one or more processors or otherwise executable.

[0034] A computer-readable storage medium can contain any storage medium or combination of storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media can include, but are not limited to, optical media (e.g., compact disc (CD), digital versatile disc (DVD), Blu-ray disc), magnetic media (e.g., floppy disk, magnetic tape, or magnetic hard disk), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or storage media based on microelectromechanical systems (MEMS). The computer-readable storage medium can be embedded in the computer system (e.g., system RAM or ROM), permanently attached to the computer system (e.g., magnetic hard disk), or removable from the computer system (e.g., a hard drive).an optical disk or a flash memory based on the Universal Serial Bus (USB)) or connected to the computer system via a wired or wireless network (e.g. network-attached storage (NAS)).

[0035] It should be noted that not all activities or elements described above in the general description are necessary, that a section of a specific activity or device may be unnecessary, and that one or more additional activities or elements may be included beyond those described. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Moreover, the concepts have been described with reference to specific embodiments. However, a person skilled in the art will recognize that various modifications and changes can be made without departing from the scope of protection of this disclosure as set forth in the claims below.Accordingly, the specification and the figures are to be understood in an explanatory rather than a restrictive sense, and all such modifications are to be included within the scope of protection of the present disclosure.

[0036] Benefits, other advantages, and solutions to problems have been described above with reference to specific embodiments. The benefits, advantages, and solutions to problems, and any feature(s) that may cause any benefit, advantage, or solution to occur or become more apparent, are not to be interpreted as critical, necessary, or essential features of any or all of the claims. Furthermore, the specific embodiments disclosed above are merely illustrative, since the disclosed subject matter can be modified and practiced in different but equivalent ways, which are obvious to those skilled in the art who derive benefit from the teachings presented herein. No limitations are intended on the details of the construction or design shown herein, other than those described in the claims below.It is therefore obvious that the specific embodiments disclosed above can be altered or modified, and all such variations are considered to be within the scope of protection of the disclosed subject matter. Accordingly, the protection sought here is set out in the following claims.

Claims

[1] Method in a display system with a head-mounted display (HMD) comprising the following: Receiving video data that represents two-dimensional video content (2D video content); Receiving model data that represents a modeled three-dimensional space (3D space) and represents an area within the modeled 3D space onto which the 2D video content is to be projected; Determine in the display system whether a 2D or a 3D view mode is selected; and in response to the determination that the 3D view mode is selected: Determining the current HMD stance; Generating an initial stereoscopic pair of images that represents a 3D view of the space from the current position; Generating a second stereoscopic pair of images, representing a transformation of the video data into the current posture of the HMD from a flat 2D coordinate reference, to fit the surface in a coordinate reference of the 3D view of the space from the current posture, wherein the second stereoscopic pair of images obscures an area of ​​the first stereoscopic pair of images, the area corresponding to the surface within the modeled 3D space; and Providing a view of the modeled 3D space via the HMD based on the current posture, wherein the view of the modeled 3D space contains a projection of the 2D video content onto the surface based on the current posture. [2] The method of claim 1, further comprising: In response to the determination that the 2D view mode is selected, provide a 2D display of the 2D video content via a display panel of the display system. [3] Method according to claim 1 or 2, wherein: Receiving the model data includes receiving the model data as metadata in the video data. [4] Method according to claim 1 or 2, wherein: Receiving the model data includes: Received along with the video data of an identifier of a remote storage location of the model data; and Accessing the model data from the remote storage location using the identifier. [5] Method according to claim 1 or 2, wherein: Receiving the model data includes: Received along with the video data of an identifier that is associated with the model data; and Accessing model data from a local data storage location of the display system based on the identifier. [6] A method according to any of the preceding claims, further comprising: Presenting to a user, via the display system, an option to view the 2D video content either in 2D view mode or in 3D view mode; and where determining whether the 2D view mode or the 3D view mode is selected includes determining whether the 2D view mode or the 3D view mode is selected based on user input in response to the presentation of the option. [7] Display system comprising the following: a head-mounted device (HMD); at least one processor coupled with the HMD; and a memory component coupled to the at least one processor, wherein the memory component serves to store a group of executable instructions which the at least one processor can access, wherein the group of executable instructions is configured to manipulate the at least one processor to: Receiving video data that represents two-dimensional video content (2D video content); Receiving model data that represents a modeled three-dimensional space (3D space) and a surface within the modeled 3D space onto which the 2D video content is to be projected; Determine whether a 2D or 3D view mode is selected; and in response to the determination that the 3D view mode is selected: Determining the current HMD stance; Generating an initial stereoscopic pair of images that represents a 3D view of the space from the current position; Generating a second stereoscopic pair of images, representing a transformation of the video data into the current posture of the HMD from a flat 2D coordinate reference, to fit the surface in a coordinate reference of the 3D view of the space from the current posture, wherein the second stereoscopic pair of images obscures an area of ​​the first stereoscopic pair of images, the area corresponding to the surface within the modeled 3D space; and Providing for the HMD the display of a view of the modeled 3D space based on the current posture, wherein the view of the modeled 3D space contains a projection of the 2D video content onto the surface based on the current posture. [8] Display system according to claim 7, wherein the group of executable instructions is further configured to manipulate the at least one processor so that it: in response to the determination that the 2D view mode is selected, the 2D video content is provided via a display panel of the display system of a 2D display. [9] Display system according to claim 7 or claim 8, wherein: The display system is configured to receive the model data as metadata in the video data. [10] Display system according to claim 7 or claim 8, wherein: the display system is configured to receive, along with the video data, an identifier of a remote storage location of the model data; and The instructions include manipulating at least one processor to receive the model data, and instructions to manipulate at least one processor to access the model data from the remote memory location using the identifier. [11] Display system according to claim 7 or claim 8, wherein: The instructions include manipulating at least one processor to receive the model data, and instructions to manipulate at least one processor to: Received along with the video data of an identifier that is associated with the model data; and Accessing model data from a local data storage location of the display system based on the identifier. [12] Display system according to any one of claims 7 to 11, wherein the group of executable instructions further includes instructions to manipulate the at least one processor to: Presenting to a user, via the display system, an option to view the 2D video content either in 2D view mode or in 3D view mode; and Determine whether the 2D view mode or the 3D view mode is selected, based on user input in response to the presentation of the option. [13] Method in a processing device comprising: Determining a three-dimensional model (3D model) of a space; Identifying a surface in the 3D model of the space onto which two-dimensional video content (2D video content) is to be projected; and Providing video data representing the 2D video content and model data representing the 3D model of the space for storage so that they are available for subsequent access by a display system, wherein the video data includes a stereoscopic pair of images representing a transformation of the video data from a flat 2D coordinate reference to fit the surface in a coordinate reference of the 3D view of the space, the stereoscopic pair of images obscuring the surface. [14] Method according to claim 13, wherein: the space comprises a real space; and Determining the 3D model of the space includes: Capturing images and spatial features of real space using a group of one or more sensors of the processing device, wherein the group of one or more sensors comprises an image capture camera and / or a depth sensor; and Generating a 3D model of the room based on the captured images and room features. [15] Method according to claim 13, wherein: This includes determining the 3D model of the space and constructing the 3D model using a virtual reality modeling tool in the processing device. [16] Method according to any one of claims 13 to 15, wherein: This includes providing the video data and model data for storage in a multimedia server. [17] Method according to claim 16, wherein: This includes providing the video data and model data for storage in the multimedia server, providing the video data as a video stream, and providing the model data as metadata in the video stream. [18] Method according to claim 16, wherein: Providing the video data and model data for storage in the multimedia server includes providing the video data as a video stream and providing an identifier associated with the model data in the video stream, wherein the identifier includes an identifier of a storage location of the model data in the multimedia server and / or an identifier of the model data for use when accessing a local copy of the model data stored in the display system. [19] Processing device comprising the following: at least one processor; and a memory component coupled to the at least one processor, wherein the memory component serves to store a group of executable instructions, wherein the group of executable instructions serves to manipulate the at least one processor to: Determining a three-dimensional model (3D model) of a space; Identifying a surface in the 3D model of the space onto which two-dimensional video content (2D video content) is to be projected; and Providing video data representing the 2D video content and model data representing the 3D model of the space for storage so that they are available for subsequent access by a display system, wherein the video data includes a stereoscopic pair of images representing a transformation of the video data from a flat 2D coordinate reference to fit the surface in a coordinate reference of the 3D view of the space, the stereoscopic pair of images obscuring the surface. [20] Processing device according to claim 19, further comprising: a group of one or more sensors, wherein the group of one or more sensors includes an image capture camera and / or a depth sensor; where the space comprises a real space; and where the instructions to manipulate at least one processor to determine the 3D model of the space include instructions to manipulate at least one processor to: Capturing images and identifying spatial features of real space using a group of one or more sensors; and Generating a 3D model of the room based on the captured images and room features. [21] Processing device according to claim 19, wherein: the space is a virtual space; and The instructions for manipulating the at least one processor to determine the 3D model of the space include instructions for manipulating the at least one processor to construct the 3D model via virtual reality modeling. [22] Processing device according to any one of claims 19 to 21, further comprising: a network interface for connecting to a network; and the processing device serves to provide the video data and model data for storage in a multimedia server via the network. [23] Processing device according to claim 22, wherein: The processing device serves to provide video data for storage in the multimedia server as a video streaming file and to provide the model data as metadata in the video streaming file. [24] Processing device according to claim 22, wherein: The processing device serves to provide the video data as a video streaming file for storage in the multimedia server and to provide an identifier associated with the model data in the video streaming file, wherein the identifier includes a marker of a storage location of the model data in the multimedia server and / or an identifier of the model data for use when accessing a local copy of the model data stored in the display system.

Images