System and method for projecting content in an environment

Abstract

The present disclosure describes systems and methods for placing users. More specifically, content is received and depth information corresponding to an external environment of a computing device is obtained. An indication is received from a user to associate the content with a location based on the depth image. Information of at least one plane associated with the depth information at the location is obtained; and a portion of the selected content is warped to match the at least one plane.

Description

Background Technology

[0001] Augmented reality (AR) offers users, especially those using smartphones, numerous opportunities to develop interactive content and effects. In examples, translation services, design visualization services, and other information services can provide users with information that would otherwise be difficult to obtain or would require significant effort from them. For instance, a user wanting to view a menu in a foreign language can easily download and use a language translation app to translate the menu in real time. Furthermore, users can place, for example, a 2D image of a chair within an image of a room to visualize how the chair would look in their room without actually possessing it. Additionally, entertainment applications have similarly incorporated AR as a means to enhance the user experience and / or the overall usability of the application.

[0002] The embodiments have been described with these and other general considerations in mind. Although relatively specific problems have been discussed, it should be understood that the examples described herein are not limited to solving the specific problems identified in the background section above. Summary of the Invention

[0003] Based on examples of this disclosure, systems and methods are described for placing user-provided content into a user's world imaged by an imaging device of a computing device. More specifically, content accessible by the computing device is received and depth information corresponding to the external environment of the computing device is obtained. An instruction is received from the user to associate the selected content with a location based on the depth information. Information on at least one plane associated with the depth image at that location is obtained; and a portion of the content is distorted to match the at least one plane.

[0004] According to examples of this disclosure, a method for placing content in an imaging environment outside a computing device is described. The method may include receiving content, obtaining depth information corresponding to the imaging environment outside the computing device, receiving an indication that associates the content with a position in the imaging environment based on the depth information, and distorting at least a portion of the content based on the depth information corresponding to the imaging environment outside the computing device.

[0005] According to examples of this disclosure, a system configured to place content in an imaging environment outside a computing device is described. The system may include a processor and a memory including instructions that, when executed by the processor, cause the processor to: receive content; obtain depth information corresponding to the imaging environment outside the computing device; receive an instruction to associate the content with a position in the imaging environment based on the depth information; and distort at least a portion of the content based on the depth information corresponding to the imaging environment outside the computing device.

[0006] According to examples of this disclosure, a computer-readable storage medium is described. The computer-readable storage medium may include instructions that, when executed by a processor, cause the processor to: receive content; acquire depth information corresponding to an imaging environment outside a computing device; receive an instruction to associate selected content with a position in the imaging environment based on the depth information; and distort at least a portion of the content based on the depth information corresponding to the imaging environment outside the computing device.

[0007] This summary is provided to introduce the selected concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0008] Non-restrictive and non-exhaustive examples are described with reference to the following figures.

[0009] Figure 1 An example system according to the present disclosure is described, which provides a user with the ability to place content (such as content) into a user's environment or world (such as that viewed through an augmented reality user interface).

[0010] Figure 2A-2G An example user interface for placing content is depicted according to the examples in this disclosure.

[0011] Figures 3A-3B Details of the acquisition of depth information according to examples of this disclosure are described.

[0012] Figure 4 Details of content placement according to examples of this disclosure are depicted.

[0013] Figure 5A and 5B Details of the data structures exemplified by this disclosure are described.

[0014] Figure 6 Additional details regarding the placement of content are depicted according to examples of this disclosure.

[0015] Figure 7 An example system based on the examples in this disclosure is described.

[0016] Figure 8 Details of a method for placing content into the user's world, based on examples of this disclosure, are described.

[0017] Figure 9 Details of a method for placing content into the user's world and acquiring video, based on an example of this disclosure, are described.

[0018] Figure 10 It is a block diagram illustrating the physical components (e.g., hardware) of a computing device that can implement various aspects of the present disclosure.

[0019] Figure 11A-11B A mobile computing device is illustrated that can be used to practice embodiments of the present disclosure.

[0020] Figure 12 The diagram illustrates one aspect of the architecture of a system used for data processing. Detailed Implementation

[0021] In the following detailed description, reference is made to the accompanying drawings, which form part of this specification, illustrating specific embodiments or examples by way of illustration. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from this disclosure. Embodiments may be practiced as methods, systems, or devices. Therefore, embodiments may take the form of hardware implementations, entirely software implementations, or implementations combining software and hardware aspects. Consequently, the following detailed description should not be construed as limiting, and the scope of this disclosure is defined by the appended claims and their equivalents.

[0022] Figure 1 An example system 100 according to an example of this disclosure is depicted, which provides a user 102 with the ability to place content 104 (such as content 112) into the user's environment or world 106 (such as as viewed through an augmented reality user interface). Content 104 may include, but is not limited to, text, audio, one or more images, one or more animations, and one or more videos. The augmented reality user interface or display may be generated by an application 110 residing on or otherwise executing at least partially on computing device 108. In the example, the user 102 may desire to place content 104 into their world captured by computing device 108. In another example, content 104 may be content provided by application 110, and accordingly, the user 102 may select the desired content 104 to place such content into their world as a preset of, for example, an application 110 installed on computing device 108. In another embodiment, content 104 may also be content captured in real time by computing device 108. For example, user 102 may want to upload, select, or capture content 112 into their environment, such as by capturing and making it visible by computing device 108. User 102 may upload, select, or capture content 112 and place content 112 as depicted in example user interface 120.

[0023] The first content 112A is placed in a position in a way that makes it appear as part of the environment or world 106. Therefore, when the user moves the computing device 108, causing the first content 112A to become invisible in the user interface of the computing device 108, the first content 112A becomes visible again when the user returns to the position where it was placed. Furthermore, the first content 112A, which appears to be placed within the environment or world 106, can be seen from different angles or positions within the environment or world 106. For example, when the user moves the computing device 108, an image or video acquired by the application 110 may be shown from an angle different from the angle at which the first content 112A was placed within the environment or world 106. As another example, when the user moves the computing device 108, an image or video acquired by the application 110 may be shown from a position different from the position of the computing device 108 when the first content 112A was placed within the environment or world 106.

[0024] Furthermore, content 104 may appear to be projected onto the surface of the environment or world 106. For example, first content 112A appears to be projected onto the surface of a railing depicted in the user interface 120. In some cases, content 104 may be placed on two or more surfaces or planes. Thus, as Figure 1 As shown, content 104 can be placed on two surfaces or planes. More specifically, as... Figure 1 As shown, the second content 112B corresponding to content 104 can be positioned so that it appears to be projected into a corner. The second content 112B can be a copy of the first content 112A. The second content 112B can be an image or video different from the first content 112A. In other words, when an image or video acquired by the user interface provided by application 110 is viewed, at least a portion of the second content 112B appears to be placed on a surface different from the surface on which another different portion of the second content 112B is placed. In the example, once content 104, such as content 112, is placed, the user can record their interactions with the placed content.

[0025] User 102 can obtain depth information associated with environment or world 106, and then place or otherwise attach content 112 to a location within the environment or world, where the location is based on the obtained depth information. For example, the depth information may be obtained based on multiple sensors, such as LiDAR sensors, structured light sensors, or other sensors. In one embodiment, the depth information may be obtained based on multiple images of the environment or world 106 acquired by the image sensor of computing device 108. The depth information may exist as a depth map and provides distance information for objects, surfaces, etc., within user 102's environment or world 106, for which the image sensor of computing device 108 has already acquired such images. Once the user has determined the location for placing content 112, the user can place the additional content 112 and / or begin capturing video or images of the user environment or world 106 including the placed content 112. The captured video of the user environment or world 106, together with the placed content 112, can then be provided as video frames 122 to one or more video hosting platforms 114 via networking environment 105. In the example, another computing device 124 can access and view video frame 122. The networking environment 105 may be, for example, a wireless access network such as LTE or 5G, a local area network (LAN), a wide area network (WAN) such as the Internet, or a wireless LAN (WLAN). It should be understood that this is not intended to be limiting, and the scope of this disclosure includes implementations in which computing devices 108 and / or 124 and (one or more) video hosting platforms 114 can be operatively linked via some other communication coupling. Computing devices 108 and / or 124 may be configured to communicate with the networking environment 105 via wireless or wired connections. Furthermore, in one example, computing devices 108 and / or 124 may be configured to communicate directly with each other via wireless or wired connections.

[0026] Figure 2A-2G An example user interface 202 for placing content, according to an example of this disclosure, is depicted. Figure 2A As shown, it is displayed on a device such as computing device 108 ( Figure 1 The user interface 202 at a computing device such as 108 can depict a first image 204 captured by the image sensor of the computing device 108. A first user control 206 can prompt the user to select content, such as a video or image. In the example, an icon associated with content accessible by the computing device can be displayed as icon 208. In the example, the icon can depict one or more scenes of a video or image being displayed. The user can select an icon 210 associated with a desired video or image to be included in the user's environment or world when viewed from the image sensor of the computing device.

[0027] Figure 2AThis is just an illustrative example. In other examples, it is shown in devices such as computing device 108 ( Figure 1 The user interface 202 at the computing device can depict a first image 204 captured by the image sensor of the computing device 108. Another icon ( Figure 2A (Not shown) can prompt the user to capture content, such as video or images. Therefore, the captured video or images can be included in the user's environment or world when viewed from the image sensor of the computing device.

[0028] like Figure 2B The depicted display is shown on a computing device 108 ( Figure 1 The user interface 202 at the computing device can depict a second image 212 imaged by the image sensor of the computing device. The second image 212 may have the same scene as the first image 204, but from a different viewpoint. The user interface 202 can instruct the user to move the computing device to obtain depth information, i.e., via displaying a prompt 214 on the user interface. For example, the prompt 214 may be displayed on the display of the computing device. The prompt 214 may include an image 216 depicting a hand holding a phone, a ring 218 indicating the state of depth information collection, and text 220 stating the user's actions. The prompt 214 is for illustrative purposes, and other types of prompts may also be used; this disclosure is not limited to any particular prompt. Figure 2C As shown, it is displayed on a device such as computing device 108 ( Figure 1 The user interface 202 at the computing device can depict a third image 222 imaged by the image sensor of the computing device. The third image 222 may have the same scene as the first image 204 and the second image 212, but from a different viewpoint. The user interface 202 can depict a ring 218 indicating the state of depth information collection as complete. Therefore, it can display... Figure 2D The user interface 202 depicted in the document.

[0029] like Figure 2D As shown, it is displayed on a device such as computing device 108 ( Figure 1 The user interface 202 at the computing device can depict a fourth image 226 imaged by the image sensor of the computing device. The fourth image 226 may have the same scene as the first image 204, the second image 212, and the third image 222, but from a different viewpoint. The user interface 202 can be displayed... Figure 2A Content selected or captured in real time by a computing device. For example, content 231 may be displayed, where content 231 may indicate a video or image. The user interface 202 may also include a prompt 230 that suggests the user to place content 231. For example, the user may expect to place content 231 at the corner of the first side of railing 229 and the second side of railing 233. Therefore, content 231 may be placed on two or more surfaces or planes depicted in the fourth image 226. Figure 2D As depicted in user interface 202, content 231 appears to be projected onto two or more surfaces in the environment or world depicted in the fourth image 226. Once the user is satisfied with how content 231 appears to be projected onto the two or more surfaces, the user can select control 232 to associate content 231 with coordinate positions corresponding to the depicted location of content 231. For example, the coordinate positions may be identified as X and Y coordinates based on accelerometers or other position sensors and estimated depths provided by depth information (e.g., a depth map). The depth map may also provide a Z coordinate. In addition to associating content 231 with coordinate positions corresponding to the depicted location of content 231, depth information (e.g., a depth map used by the depicted content 231) may be saved and associated with the coordinate positions and content 231.

[0030] like Figure 2E As shown, it is displayed on a device such as computing device 108 ( Figure 1 The user interface 202 at the computing device can depict a fifth image 234 imaged by the image sensor of the computing device. The fifth image 234 may have the same scene as the first image 204, second image 212, third image 222, and fourth image 226, but from a different viewpoint. In this example, the different viewpoints are due to the computing device being moved or otherwise adjusted to image different views of the scene and / or scenes adjacent to or otherwise adjacent to the preceding scene. The user interface 202 can be displayed... Figure 2A 236. This refers to content selected or captured in real-time by a computing device. In other words, the user might expect... Figure 2A Another copy of the selected or captured content in real time by a computing device is placed in the environment or world. Alternatively or additionally, content 236 may be different from the previously placed content 231. For example, content 231 may display an image or video of a clock as the first content, while content 236 may display a different image or video as the second different content. Content 236 may be displayed in a fifth image 234, where content 236 may indicate a video or image. The user interface 202 may further include a prompt 230 that suggests the user to place content 236. For example, the user may expect to place content 236 on a railing 238. Thus, content 236 may be placed on a single surface or plane. Alternatively or additionally, railing 238 may be associated with varying depths, such that content 236 is placed on two or more surfaces or planes depicted in the fifth image 234. Figure 2EAs shown in user interface 202, content 236 appears to be projected onto a railing in the environment or world depicted in the fifth image 234. Once the user is satisfied with how content 236 appears to be projected into the environment or world, the user can select control 232 to associate content 236 with coordinate positions corresponding to the depicted location of content 236. For example, the coordinate positions can be identified as X and Y coordinates based on accelerometers or other position sensors and estimated depths provided by depth information (e.g., a depth map). Depth information, such as a depth map, can also provide Z coordinates. In addition to associating content 236 with coordinate positions corresponding to the depicted location of content 236, the depth information used by the depicted content 236 (e.g., a depth map) can be saved and associated with the coordinate positions and content 236.

[0031] Once content 236 has been placed in the user's world, user interface 202 can display a sixth image 240, such as Figure 2F As shown, the sixth image 240 may have the same scene as the first image 204, second image 212, third image 222, fourth image 226, and fifth image 234, but from a different viewpoint, and the sixth image 240 may depict content 236 based on associated coordinate positions. For example, when a user moves the computing device and an image acquired from the computing device is displayed in the user interface 202, content 236 may be displayed in the acquired image, such as the sixth image 240, when the image acquired by the image sensor includes coordinate positions associated with content 236. The sixth image 240 may display one or more of the placed content 231 / 236 and prompt the user with a prompt 242 to begin acquiring video. Therefore, when acquiring video of the environment or world, the display of the computing device may output an augmented reality display depicting content 231 / 236 as placed within the environment or world. For example, as Figure 2G As shown, image 244 can depict content 231 and content 236 from an angle or position different from the original angle or position used to place content 231 and content 236, respectively. Figure 2G As depicted in image 244, the user can change their position, prompting the computing device to image from different locations—in this case, when viewed from different positions, content 236 and 231 may appear to be projected onto one or more surfaces of the environment. The user interface 202 can also use status message 246 to indicate to the user that the computing device is acquiring video; this status message may be an icon, text, graphic, or other indication.

[0032] Figures 3A-3B Details of the acquisition of depth information according to examples of this disclosure are described. More specifically, in Figure 3A In this case, user 302 can move computing device 304 slightly around, thus... Figure 3B User 302 in the middle gets fewer images. Therefore, by Figure 3A The depth information obtained by the computing device 304 in the middle is less than that obtained by the computing device 304 in the middle. Figure 3B The computing device 304 in the environment acquires depth information. Therefore, the more the user 302 moves the computing device, the more images are processed, thus acquiring additional depth information for the environment 306. In the example, the images acquired by the computing device can be processed by a depth information generator, which can be accessed, for example, by an application programming interface. The depth information generator can utilize multiple device images from different angles and compare them to generate an estimate of the distance to each pixel in the image as the user moves their computing device (e.g., a phone). If the computing device has a supported hardware depth sensor, such as, but not limited to, a time-of-flight sensor, such data obtained from the supported hardware sensor can be included in the processed depth, in addition to being used directly as depth information, to enhance the accuracy of the existing depth information. In some examples, the additional depth information for the environment 306 can be obtained from a grid representation of the depth image, from previously acquired existing depth information, or based on multiple sensors, such as a LiDAR sensor, a structured light sensor, or other sensors.

[0033] Figure 4 Details of content placement according to examples of this disclosure are described. More specifically, coordinate positions within the environment or world are obtained based on acquired depth information, such as depth maps. As an example, axis 404, with an origin established when the user selects or captures content or when the user begins placing content, is used to track X and Y movement provided by the accelerometer or other position sensors of the computing device. Furthermore, axis 404 may track depth information, such as distance Z provided by one or more depth maps. In the example, as the user places, moves, scales, or rotates content within the environment or world, the coordinate positions, an object identifier uniquely identifying the content, and depth information (e.g., depth maps) used for placing, moving, scaling, or rotating the content can be recorded in data structures such as data structures 408 and 412. As shown in the data structures, content identified by an object ID is placed at different locations within the environment.

[0034] like Figure 5A As shown, according to an example of this disclosure, the location corresponding to the placed content can be stored in data structure 504. Data structure 504 can uniquely identify the content placed using the object_ID field, the location of the content placed using X, Y, and Z coordinates, and the specific depth map used. For example... Figure 5BFurther described, each object_ID can uniquely identify content or a version of content. In the example, although a user may place the same content multiple times in a video, the start and stop times associated with the content (e.g., video content or image content) can vary. Therefore, different versions of the content can be uniquely identified. For example, data structure 508 may include object_IDs (e.g., C_1) corresponding to a first version of video Video_1, and object_IDs (e.g., C_2) corresponding to a second version of video Video_1. Furthermore, data structure 508 may include object_IDs (e.g., C_3) corresponding to a second version of video Video_1. The second version of the video may be the same as the first version of the video; however, the second version of the video may differ in some respects. For example, the second version of the video may start at a different start point, be displayed at a different frame rate, and / or end at a different stop point.

[0035] Figure 6 Further details regarding the placement of content according to examples of this disclosure are described. More specifically, the placement of content 608 is achieved by combining depth information and position tracking information. Instead of maintaining the form of the content (possibly a 3D representation), the content is distorted as if it were a flexible piece of cloth or a projection. This projection effect is achieved by centering the content within the camera frustum and moving the Z component of the content vertices along the positive vector of the camera, where the camera may correspond to an image sensor associated with a computing device. The amount of movement for each vertex depends on the current depth estimate of the nearest pixel to the content's X and Y coordinates. Furthermore, in some examples, a confidence score associated with the depth information is utilized, and determining whether to project the content may depend on the confidence level of the depth information. In one embodiment, if the confidence level of the depth information is too low, for example, less than a threshold, the projection of the content may be stopped, and the content may be positioned in front of a camera that is floating in the air. If the confidence level of the depth information is greater than the threshold, the content 608 is placed and projected accordingly based on a position selected by the user.

[0036] like Figure 6As shown, according to an example of this disclosure, computing device 604 can display an image of the environment or world. A user may wish to place content 608 at the corner of surface 616 of imaging surface 620 and surface 624 of imaging surface 628. Therefore, user interface 636 can depict an example representation of content 608 placed at the corner formed by surfaces 620 and 628. When the user is satisfied with the appearance of the current projection, they can place the content in world space. At this time, a copy of the projection is made by saving the position of the content in world space and the depth information at the time of placement. Once the projection is saved, textures are added to the projection using content 608. Since the plane is used as the initial content, video or image content can be stretched to match the aspect ratio of the plane. When content 608 is applied to the plane, a method of stretching to height can be applied to content 608.

[0037] In the example, one or more of three options can be selected to process content 608: no processing, video matting, and chroma keying. Video and / or image matting separates a person or other object in a video or image from the background, allowing the video or image of the person or other object to be projected without a background. Chroma keying uses the color value of each pixel to determine whether they match a common halftone color. For the chroma keying process, the boundaries of the video or image are first viewed to determine if a common background color exists throughout the video or image. The video or image can then be modified so that the background color determined by the selected common background color is transparent. Example colors include red, green, blue, black, or white, which can be selected as transparent common background colors. For example, a user can select green in a video or image with a green background, making all content in the video or image with green appear transparent.

[0038] In some examples, camera input can be captured and projected back into the user's world. This can be used to blend user-uploaded or captured content (e.g., videos or images) with real-world textures. Thus, the input image can be stretched directly onto the real world, creating a frozen time view of the real world from a perspective the user might not currently have.

[0039] Figure 7 An example system 700 according to an example of this disclosure is depicted. System 700 may correspond to a system for placing content in a user environment or world that can be imaged from an imaging device of a camera. In the example, device 702 may include an image acquisition module 704 configured to acquire a plurality of images 706 of the user environment. The plurality of images 706 may be acquired from an image sensor 708. Image sensor 708 may be incorporated into or otherwise included in a computing device such as computing device 108 ( Figure 1In a computing device, multiple images 706 may be provided to a depth information generator 710. The depth information generator 710 may utilize and compare multiple device images from different angles to generate an estimate of the distance to each pixel in the image as the user moves their computing device. If the computing device has a supported hardware depth sensor, such as, but not limited to, a time-of-flight sensor, this type of data may be included in the processed depth to enhance the accuracy of existing depth maps. Depth information 712 may be provided to a position tracking and depth estimation module 714. In an example, depth information 712 may include a depth map comprising a depth image containing a distance representation at each pixel in the depth image.

[0040] The position tracking and depth estimation module 714 can receive content 716 and distort it as if it were elastic fabric or a projection. In one example, if the user selects content 716, the content can be provided from content storage device 718. This projection effect is achieved by centering content 716 within the camera frustum and moving the Z component of the vertices of content 716 along the positive vector of the camera or image sensor. The amount of movement for each vertex depends on the current depth estimate of the pixel whose X and Y coordinates are closest to the content, where the X and Y coordinates can be based on position information provided from position sensor 707. In one example, the position tracking and depth estimation module 714 can receive user input 720 indicating that the displayed projection is acceptable, which sends an instruction to place content 716. Thus, a copy of the projection is made by saving the position of content 716 in world space and the depth information at the time of placement as 722. In one example, the determination of whether to project content can depend on the confidence level of the depth information. In one example, if the confidence level of the depth information is too low, for example, less than a threshold, the projection of the content can be stopped, and the content can be positioned in front of the camera floating in the air. If the confidence level of the depth information is greater than the threshold, the content 716 is placed and projected accordingly based on the user-selected location.

[0041] Video processing module 728 can texture a stored user projection using content 716. For example, the projection may include depth information, such that the depth information is textured using content 716. Content 716 is stretched to match the aspect ratio of one or more planes, where the one or more planes may correspond to planes derived from distance information in depth information 712. Video processing module 728 can then display an augmented reality image, including the stretched content placed in the user's environment, at user interface 726. In an example, video acquisition module 730 may acquire multiple images (e.g., videos) or a single image along with the placed content 716. Video acquisition module 730 may then provide multiple images (e.g., videos) or a single image along with the placed content 716 to content storage device 718 for storage, wherein the content storage device may reside at and / or outside device 702.

[0042] Figure 8 Details of a method 800 for placing content in the user's world, according to an example of this disclosure, are depicted. Figure 8 The general sequence of steps in method 800 is shown. Generally, method 800 begins at 802 and ends at 830. Method 800 may include more or fewer steps, or may follow a sequence consistent with... Figure 8 The order of the steps shown is not necessarily the same as the order of the steps in the example. Method 800 can be executed as a set of computer-executable instructions that are executed by a computer system and encoded or stored on a computer-readable medium. In the example, aspects of method 800 are executed by one or more processing devices, such as a computer. Furthermore, method 800 can be executed by gates or circuits associated with a processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), system-on-a-chip (SoC), neural processing unit, or other hardware device. Hereinafter, method 800 will be referred to in conjunction with... Figure 1-7 Explain using the described system, components, modules, software, data structures, user interface, etc.

[0043] The method begins at 802, and the process can proceed to 804. At 804, content is received. For example, a user can select video or images from their computing device's camera roll. In another example, the user can capture video or images in real time. In yet another example, the content can be selected from options provided by an application installed on the computing device running the application. Method 800 can proceed to 808, where depth information associated with the user's world is obtained. In one example, as previously described, the depth information can be obtained from multiple images. That is, multiple device images from different angles can be acquired and compared to generate an estimate of the distance to each pixel in the image as the user moves their computing device. In another example, the depth information can be obtained from a supported hardware depth sensor, such as, but not limited to, a time-of-flight sensor. Furthermore, this type of data can be included in processed depth data to enhance the accuracy of existing depth information, such as depth maps acquired from multiple images. At 810, a confidence threshold for the depth map information is obtained. In the example, if the confidence level of the depth map information is not greater than a threshold (e.g., the computing device is not certain that the depth information is correct), the method proceeds to 822. Otherwise, the method proceeds to 812, where the content is projected onto the user's world.

[0044] For example, at 812, the content is distorted as if it were a flexible piece of fabric or a projection. This projection effect is achieved by centering the content within the frustum of the camera and moving the Z component of the content vertices along the positive vector of the camera. The amount of movement for each vertex depends on the current depth estimate of the nearest pixel to the content's X and Y coordinates, where the X and Y coordinates may be based on position information provided by a position sensor. At 816, an indication that the user is satisfied with the projection position can be received. Therefore, at 816, a copy of the depth information and a copy of the content are saved. Method 800 can proceed to 818, where the content is applied as a texture to one or more surfaces. For example, one or more surfaces (or planes) can be determined from the depth information; the content portion corresponding to the respective plane or surface can be distorted based on the depth information. In the example, the user may optionally select one or more of video keying and / or chroma keying to separate a person or another object in a video or image from the background, or otherwise make the background transparent. Thus, only people or objects can be projected into the user's world.

[0045] If the depth information does not meet the confidence interval, at 822, the content may be located in front of the camera of the computing device. When an instruction from the user to accept the position is received at 824, the position of the content may be saved at 826, so that the content is displayed at 828 according to the position information saved at 826, just as if the content were projected onto a single plane or surface. Method 800 may end at 830.

[0046] Figure 9 Details of a method 900 for placing content in the user's world and acquiring video, according to an example of this disclosure, are described. Figure 9 The general sequence of steps in method 900 is shown. Generally, method 900 begins at 902 and ends at 922. Method 900 may include more or fewer steps, or may follow a sequence consistent with... Figure 9 The steps shown are arranged in a different order. Method 900 can be executed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer-readable medium. In the example, aspects of method 900 are executed by one or more processing devices, such as a computer. Furthermore, method 900 can be executed by gates or circuits associated with a processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), system-on-a-chip (SoC), neural processing unit, or other hardware device. Hereinafter, method 900 will be referred to in conjunction with... Figure 1-8 Explain using the described system, components, modules, software, data structures, user interface, etc.

[0047] The method begins at 902, and the process can proceed to 904. At 904, content is received. For example, a user can select video or images from their computing device's camera roll. In another example, the user can capture video or images in real time. In yet another example, the content can be selected from options provided by an application installed on the computing device running the application. The method can proceed to 906, where depth information associated with the user's world is obtained. In this example, as previously described, the depth information can be obtained from multiple images. That is, as the user moves their computing device, multiple images from different angles are acquired and compared to generate an estimate of the distance to each pixel in the images. In another example, the depth information can be obtained from a supported hardware depth sensor (such as, but not limited to, a time-of-flight sensor). Furthermore, this type of data can be included in processed depth data to enhance the accuracy of existing depth information. At 908, the content is projected onto the user's world.

[0048] For example, in 908, the content is distorted as if it were a flexible piece of fabric or a projection. This projection effect is achieved by centering the content within the camera's frustum and moving the Z component of the content's vertices along the camera's positive vector. The amount of movement for each vertex depends on the current depth estimate of the pixel whose X and Y coordinates are closest to the content, where the X and Y coordinates can be based on positional information provided from a position sensor. In the example, the user can optionally select one or more of video keying and / or chroma keying to separate a person or another object in a video or image from the background, or otherwise make the background transparent. Thus, only people or objects can be projected into the user's world.

[0049] At 910, an indication that the user is satisfied with the projection position can be received. Therefore, at 912, a copy of the depth information (e.g., a depth map) and a copy of the content are saved. Method 900 can proceed to 914, where the content is applied as a texture to one or more surfaces. For example, one or more surfaces (or planes) can be determined from the depth information; the content portion corresponding to the respective plane or surface can be distorted based on the depth information.

[0050] The method may proceed to 916, where a video including projections or content placed in the user's world is acquired. For example, the computing device may acquire one or more images / videos describing the content from different angles and / or positions. In the example, the acquired video is then provided to a video hosting platform for sharing at 920. Method 900 may end at 922.

[0051] Figure 10 This is a block diagram illustrating the physical components (e.g., hardware) of a computing device 1000 that can implement various aspects of the present disclosure. The computing device components described below are applicable to the computing and / or processing devices described above. In a basic configuration, the computing device 1000 may include at least one processing unit 1002 and system memory 1004. Depending on the configuration and type of the computing device, the system memory 1004 may include, but is not limited to, volatile storage devices (e.g., random access memory (RAM)), non-volatile storage devices (e.g., read-only memory (ROM)), flash memory, or any combination of such memories.

[0052] System memory 1004 may include operating system 1005 and one or more program modules 1006 suitable for running software application 1020, such as one or more components supported by the system described herein. As an example, system memory 1004 may include image acquisition module 1021, depth information generator 1022, position tracking and depth estimation module 1023, video processing module 1024, and video acquisition module 1025. Image acquisition module 1021 may be the same as or similar to image acquisition module 704; depth information generator 1022 may be the same as or similar to depth information generator 710; position tracking and depth estimation module 1023 may be the same as or similar to position tracking and depth estimation module 714; video processing module 1024 may be the same as or similar to video processing module 728; and video acquisition module 1025 may be the same as or similar to video acquisition module 730. For example, operating system 1005 may be suitable for controlling the operation of computing device 1000.

[0053] Furthermore, embodiments of this disclosure can be practiced in conjunction with graphics libraries, other operating systems, or any other applications, and are not limited to any particular application or system. This basic configuration is... Figure 10 The components within the dashed line 1008 are illustrated. The computing device 1000 may have additional features or functionalities. For example, the computing device 1000 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. Such additional storage devices... Figure 10 The diagram shows a removable storage device 1009 and a non-removable storage device 1010.

[0054] As described above, multiple program modules and data files may be stored in system memory 1004. When executed on processing unit 1002, program module 1006 (e.g., software application 1020) may perform processes including, but not limited to, those described herein. Other program modules that may be used according to aspects of this disclosure may include email and contact applications, word processing applications, spreadsheet applications, database applications, PowerPoint presentation applications, drawing or computer-aided programs, etc.

[0055] Furthermore, embodiments of this disclosure can be practiced in circuits, discrete electronic components, packages containing logic gates or integrated electronic chips, circuits utilizing microprocessors, or single chips containing electronic components or microprocessors. For example, embodiments of this disclosure can be practiced via a system-on-a-chip (SOC), wherein... Figure 10Each or more of the components illustrated may be integrated onto a single integrated circuit. Such a SoC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functionalities, all integrated (or “programmed”) onto a chip substrate as a single integrated circuit. When operating via the SoC, the functionality described herein regarding the ability to switch client protocols may be operated via dedicated logic integrated with other components of the computing device 1000 on the single integrated circuit (chip). Embodiments of this disclosure may also be practiced using other techniques capable of performing logical operations, such as AND, OR, and NOT, including but not limited to mechanical, optical, fluid, and quantum technologies. Furthermore, embodiments of this disclosure may be practiced in general-purpose computers or any other circuit or system.

[0056] The computing device 1000 may also have one or more input devices 1012, such as a keyboard, mouse, pen, voice or speech input device, touch or swipe input device, etc. One or more input devices 1012 may include an image sensor. It may also include one or more output devices 1014, such as a display, speaker, printer, etc. The above devices are examples, and other devices may be used. The computing device 1000 may include one or more communication connections 1016 that allow communication with other computing devices / systems 1050. Examples of suitable communication connections 1016 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; universal serial buses (USB), parallel and / or serial ports.

[0057] As used herein, the term computer-readable medium may include computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, or program modules. System memory 1004, removable storage device 1009, and non-removable storage device 1010 are examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other article of manufacture that can be used to store information and is accessible by computing device 1000. Any such computer storage media may be part of computing device 1000. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0058] Communication media may be embodied in computer-readable instructions, data structures, program modules, or other data in modulated data signals, such as carrier waves or other transmission mechanisms, and include any information transmission medium. The term "modulated data signal" can describe a signal having one or more characteristics set or altered in a manner that encodes information in the signal. By way of example, and not limitation, communication media may include wired media, such as wired networks or direct wired connections, and wireless media, such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0059] Figure 11A-11B The illustration depicts a mobile computing device 1100, such as a mobile phone, smartphone, wearable computer (e.g., smartwatch), tablet computer, laptop computer, smart home device, etc., embodiments of this disclosure can be practiced therein. In some examples, mobile computing device 1100 may be the same as or similar to computing device 108. In some aspects, the client may be a mobile computing device. Reference Figure 11A The illustration depicts one aspect of a mobile computing device 1100 used to implement these aspects. In a basic configuration, the mobile computing device 1100 is a handheld computer with input and output elements. The mobile computing device 1100 typically includes a display 1105 and one or more input buttons 1110 that allow a user to input information into the mobile computing device 1100. The display 1105 of the mobile computing device 1100 can also be used as an input device (e.g., a touchscreen display).

[0060] If included, optional side input element 1115 allows for further user input. Side input element 1115 can be a rotary switch, button, or any other type of manual input element. Alternatively, mobile computing device 1100 may incorporate more or fewer input elements. For example, in some embodiments, display 1105 may not be a touchscreen.

[0061] In another alternative embodiment, the mobile computing device 1100 is a portable telephone system, such as a cellular phone. The mobile computing device 1100 may also include an optional keypad 1135. The optional keypad 1135 may be a physical keypad or a “soft” keypad generated on a touchscreen display.

[0062] In various embodiments, output elements include a display 1105 for displaying a graphical user interface (GUI), a visual indicator 1120 (e.g., a light-emitting diode), and / or an audio transducer 1125 (e.g., a speaker). In some aspects, the mobile computing device 1100 includes a vibration transducer for providing haptic feedback to a user. In another aspect, the mobile computing device 1100 incorporates input and / or output ports for sending signals to or receiving signals from external devices, such as audio inputs (e.g., a microphone jack), audio outputs (e.g., a headphone jack), and video outputs (e.g., an HDMI port).

[0063] Figure 11B This is a block diagram illustrating one aspect of the architecture of a mobile computing device. That is, the mobile computing device 1100 may incorporate aspects of a system (e.g., architecture) 1102. In one embodiment, the system 1102 is implemented as a "smartphone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, the system 1102 is integrated as a computing device, such as integrating a personal digital assistant (PDA), cordless phone, wearable device, smart home device, etc.

[0064] One or more applications 1166 may be loaded into memory 1162 and run on or associated with operating system 1164. Examples of applications include telephone dialers, email programs, personal information management (PIM) programs, word processing programs, spreadsheet programs, internet browser programs, messaging programs, mapping programs, and so on. System 1102 also includes a non-volatile storage area 1168 within memory 1162. The non-volatile storage area 1168 can be used to store permanent information that should not be lost in the event of a power outage of system 1102. Applications 1166 may use and store information in the non-volatile storage area 1168, such as other messages used by email or email applications. A synchronization application (not shown) also resides on system 1102 and is programmed to interact with a corresponding synchronization application residing on the host to keep the information stored in the non-volatile storage area 1168 synchronized with the corresponding information stored on the host. It should be understood that other applications may be loaded into memory 1162 and run on the mobile computing device 1100 described herein.

[0065] System 1102 has a power supply 1170, which may be implemented as one or more batteries. The power supply 1170 may also include an external power source, such as an AC adapter or an electric docking station for replenishing or recharging the batteries.

[0066] System 1102 may also include a radio interface layer 1172, which performs the functions of transmitting and receiving radio frequency communications. Radio interface layer 1172 facilitates a wireless connection between system 1102 and the "outside world" via a communications operator or service provider. Transmissions to and from radio interface layer 1172 are carried out under the control of operating system 1164. In other words, communications received by radio interface layer 1172 can be propagated to application 1166 via operating system 1164, and vice versa.

[0067] A visual indicator 1120 may be used to provide visual notifications, and / or an audio interface 1174 may be used to generate auditory notifications via an audio transducer 1125. In the illustrated embodiment, the visual indicator 1120 is a light-emitting diode (LED), and the audio transducer 1125 is a speaker. These devices may be directly coupled to a power supply 1170 such that when activated, they remain on for a duration specified by the notification mechanism, even if the processor 1160 and other components may be turned off to conserve battery power. The LED may be programmed to remain on indefinitely until the user takes action to indicate the device's power-on status. The audio interface 1174 is used to provide and receive auditory signals from the user. For example, in addition to being coupled to the audio transducer 1125, the audio interface 1174 may also be coupled to a microphone to receive auditory input, such as for facilitating telephone conversations. According to embodiments of this disclosure, the microphone may also be used as an audio sensor to facilitate control of notifications, as described below. System 1102 may also include a video interface 1176, which enables the operation of the airborne camera 1130 to record still images, video streams, etc.

[0068] The mobile computing device 1100 implementing system 1102 may have additional features or functionalities. For example, the mobile computing device 1100 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. Such additional storage devices... Figure 11B The diagram shows a non-volatile storage region 1168.

[0069] Data / information generated or captured by mobile computing device 1100 and stored via system 1102 may be locally stored on mobile computing device 1100, as described above, or the data may be stored on any number of storage media accessible by the device via radio interface layer 1172 or via a wired connection between mobile computing device 1100 and an independent computing device associated with mobile computing device 1100, such as a server computer in a distributed computing network (such as the Internet). It should be understood that such data / information may be accessed via mobile computing device 1100, via radio interface layer 1172, or via a distributed computing network. Similarly, such data / information may be easily transferred between computing devices for storage and use according to known data / information transmission and storage methods, including email and collaborative data / information sharing systems.

[0070] Figure 12 The illustration depicts one aspect of a system architecture for processing data received at a computing system from a remote source, such as, as described above, a personal computer 1204, a tablet computing device 1106, or a mobile computing device 1208. The personal computer 1204, tablet computing device 1206, or mobile computing device 1208 may include one or more applications; such applications may include, but are not limited to, an image acquisition module 1221, a depth information generator 1222, a position tracking and depth estimation module 1223, a video processing module 1224, and a video acquisition module 1225. The image acquisition module 1221 may be the same as or similar to the image acquisition module 704; the depth information generator 1222 may be the same as or similar to the depth information generator 710; the position tracking and depth estimation module 1223 may be the same as or similar to the position tracking and depth estimation module 714; the video processing module 1224 may be the same as or similar to the video processing module 728; and the video acquisition module 1225 may be the same as or similar to the video acquisition module 730.

[0071] As described above, server device 1202 and / or personal computer 1204, tablet computing device 1206 or mobile computing device 1208 may employ one or more of the previously described program modules 1006 or software applications 1020.

[0072] Server device 1202 can provide data to client computing devices (e.g., personal computer 1204, tablet computing device 1206, and / or mobile computing device 1208 (e.g., smartphone) via network 1215. By way of example, the above-described computer system may be embodied in personal computer 1204, tablet computing device 1206, and / or mobile computing device 1208 (e.g., smartphone).

[0073] Furthermore, the aspects and functionalities described herein can run on distributed systems (e.g., cloud-based computing systems), where application functionalities, memory, data storage and retrieval, and various processing functions can operate remotely to each other via distributed computing networks (e.g., the Internet or an intranet). Various types of user interfaces and information can be displayed via onboard computing device displays or via remote display units associated with one or more computing devices. For example, various types of user interfaces and information can be displayed and interacted with on a wall on which various types of user interfaces and information are projected. Interaction with multiple computing systems on which embodiments of this disclosure can be practiced includes keystroke input, touchscreen input, voice or other audio input, gesture input, wherein the associated computing device is equipped with detection (e.g., camera) functionality for capturing and interpreting user gestures to control the functionality of the computing device, and so on.

[0074] For example, aspects of this disclosure have been described above with reference to block diagrams and / or operating instructions of methods, systems, and computer program products according to various aspects of this disclosure. The functions / actions marked in the boxes may not occur in the order shown in any flowchart. For example, depending on the functionality / action involved, two boxes shown consecutively may actually be executed substantially simultaneously, or sometimes in reverse order.

[0075] This disclosure relates to systems and methods for placing content in an imaging environment outside a computing device, based at least on the examples provided in the following sections:

[0076] (A1) In one aspect, some examples include a method for placing content in an imaging environment outside a computing device. The method includes receiving content; obtaining depth information corresponding to the imaging environment outside the computing device; receiving an indication that associates the content with a position in the imaging environment based on the depth information; and distorting at least a portion of the content based on the depth information corresponding to the imaging environment outside the computing device.

[0077] (A2) In some examples of A1, the method includes acquiring at least one video or image content of an imaging environment outside the computing device, wherein at least one video or image content includes a distorted portion of the content.

[0078] (A3) In some examples of A1-A2, the method includes receiving second content; obtaining second depth information corresponding to an imaging environment outside the computing device; receiving a second indication that associates the second content with a second location in the imaging environment based on the second depth information; and distorting at least a portion of the second content based on the second depth information corresponding to the imaging environment outside the computing device, wherein at least one of the video or image content of the imaging environment outside the computing device includes a distorted portion of the content and a distorted portion of the second content.

[0079] (A4) In some examples of A1-A3, the content includes video content comprising multiple video frames, wherein the multiple video frames are displayed in at least one of video or image content in an imaging environment outside the computing device.

[0080] (A5) In some examples of A1-A4, depth information is obtained from multiple images acquired from the image sensor of the computing device.

[0081] (A6) In some examples of A1-A5, the method includes determining that depth information corresponding to the imaging environment is associated with a confidence score greater than a threshold; and distorting at least a portion of the content to match at least one plane in the imaging environment.

[0082] (A7) In some examples of A1-A7, the method includes determining that depth information corresponding to the imaging environment is associated with a confidence score less than a threshold; and displaying content at that location in the imaging environment.

[0083] (A8) In some examples of A1-A7, the method includes receiving second content in response to input information; obtaining second depth information corresponding to an imaging environment outside the computing device; determining that the second depth information corresponding to the imaging environment is associated with a confidence score less than a threshold; receiving a second indication that associates the second content with a second location in the imaging environment based on the second depth information; and displaying the second content at the second location in the imaging environment.

[0084] (A9) In some examples of A1-A8, the method includes determining that second depth information corresponding to the imaging environment is associated with a confidence score greater than a threshold; receiving a second indication that associates second content with a second location in the imaging environment based on the second depth information; and distorting at least a portion of the second content to match at least one plane in the imaging environment.

[0085] (A10) In some examples from A1-A9, the background color of a portion of the content is modified before the distortion.

[0086] (A11) In some examples of A1-A10, the method includes modifying the content by separating entities in the content from the background of the content, or modifying the background color of a portion of the content before distorting the content.

[0087] (A12) In some examples of A1-A11, the depth information includes a depth map image, and the method further includes saving the depth map image from the depth information when an indication is received to associate the content with a location in the imaging environment.

[0088] On the other hand, some examples include computing systems that include one or more processors and memory coupled to the one or more processors, the memory storing one or more instructions that, when executed by one processor, cause one or more processors to perform any of the methods described herein (e.g., A1-A12 described above).

[0089] On another front, some examples include a non-transitory computer-readable storage medium that stores one or more programs for execution by one or more processors of a storage device, the one or more programs including instructions for performing any of the methods described herein (e.g., A1-A12 described above).

[0090] The descriptions and illustrations of one or more aspects provided in this application are not intended to limit or restrict the scope of the claimed disclosure in any way. The aspects, examples, and details provided in this application are considered sufficient to convey ownership and enable others to make and use the best mode of the claimed disclosure. The claimed disclosure should not be construed as limited to any aspect, example, or detail provided in this application. Various features (structures and methods) are intended to be selectively included or omitted, whether shown and described in combination or separately, to produce embodiments with a particular set of features. Having provided the descriptions and illustrations of this application, those skilled in the art will envision variations, modifications, and alternatives falling within the spirit of the broader aspects of the overall inventive concept embodied in this application, without departing from the broader scope of the claimed disclosure.

Claims

1. A method for placing content in an imaging environment outside a computing device, the method comprising: Received content; Obtain at least one of depth information corresponding to the imaging environment outside the computing device and video or image content for the imaging environment; Receive an instruction to associate the content with its location in the imaging environment based on the depth information; as well as At least a portion of the content is distorted based on the depth information corresponding to the imaging environment outside the computing device, such that at least one of the video or image content includes a distorted portion of the content, wherein the distortion of the at least a portion of the content is achieved by centering the content in the camera frustum and moving the Z component of the content vertices along the positive vector of the camera, the amount of movement of each content vertex depending on the current depth estimate of the nearest pixel to the X and Y coordinates of the content.

2. The method according to claim 1, further comprising: Receive the second content; Obtain second depth information corresponding to the imaging environment outside the computing device; Receive a second instruction that associates the second content with a second location in the imaging environment based on the second depth information; as well as Distorting at least a portion of the second content based on the second depth information corresponding to the imaging environment outside the computing device. At least one of the video or image content of the imaging environment outside the computing device includes the distorted portion of the content and the distorted portion of the second content.

3. The method of claim 1, wherein the content includes video content, the video content includes a plurality of video frames, and wherein the plurality of video frames are displayed in at least one of the video or image content of the imaging environment outside the computing device.

4. The method of claim 1, wherein the depth information is obtained from a plurality of images acquired by an image sensor of the computing device.

5. The method according to claim 1, further comprising: The depth information corresponding to the imaging environment is determined to be associated with a confidence score greater than a threshold. as well as Distort at least a portion of the content to match at least one plane in the imaging environment.

6. The method according to claim 1, further comprising: The depth information corresponding to the imaging environment is determined to be associated with a confidence score less than a threshold. as well as The content is displayed at the location in the imaging environment.

7. The method according to claim 1, further comprising: Responding to input information to receive the second content; Obtain second depth information corresponding to the imaging environment outside the computing device; The second depth information corresponding to the imaging environment is determined to be associated with a confidence score less than a threshold. Receive a second instruction that associates the second content with a second location in the imaging environment based on the second depth information; as well as The second content is displayed at the second location in the imaging environment.

8. The method according to claim 7, further comprising: The second depth information corresponding to the imaging environment is determined to be associated with a confidence score greater than a threshold; Receive a second instruction that associates the second content with a second location in the imaging environment based on the second depth information; as well as Distort at least a portion of the second content to match at least one plane in the imaging environment.

9. The method according to claim 1, further comprising: The content is modified by separating entities within the content from its background, or by modifying the background color of a portion of the content before it is distorted.

10. The method of claim 1, wherein the depth information includes a depth map image, and the method further comprises: When the indication that associates the content with the location in the imaging environment is received, the depth map image from the depth information is saved.

11. A system configured to place content in an imaging environment outside a computing device, the system comprising: processor; as well as The memory includes instructions that, when executed by the processor, cause the processor to: Received content; Obtain at least one of depth information corresponding to the imaging environment outside the computing device and video or image content for the imaging environment; Receive an instruction to associate the content with its location in the imaging environment based on the depth information; as well as At least a portion of the content is distorted based on the depth information corresponding to the imaging environment outside the computing device, such that at least one of the video or image content includes a distorted portion of the content, wherein the distortion of the at least a portion of the content is achieved by centering the content in the camera frustum and moving the Z component of the content vertices along the positive vector of the camera, the amount of movement of each content vertex depending on the current depth estimate of the nearest pixel to the X and Y coordinates of the content.

12. The system of claim 11, wherein the instructions, when executed by the processor, cause the processor to: Receive the second content; Obtain second depth information corresponding to the imaging environment outside the computing device; Receive a second instruction that associates the second content with a second location in the imaging environment based on the second depth information; as well as Distorting at least a portion of the second content based on the second depth information corresponding to the imaging environment outside the computing device. The at least one of the video or image content of the imaging environment outside the computing device includes the distorted portion of the content and the distorted portion of the second content.

13. The system of claim 11, wherein the content includes video content, the video content includes a plurality of video frames, and wherein the plurality of video frames are displayed in at least one of the video or image content of the imaging environment outside the computing device.

14. The system of claim 11, wherein the instructions, when executed by the processor, cause the processor to store the depth information corresponding to the receipt of the indication that associates the content with the location in the imaging environment.

15. A computer-readable storage medium including instructions, said instructions, when executed by a processor, causing the processor to: Received content; Obtain at least one of depth information corresponding to the imaging environment outside the computing device and video or image content for the imaging environment; Receive an instruction to associate the content with its location in the imaging environment based on the depth information; as well as At least a portion of the content is distorted based on the depth information corresponding to the imaging environment outside the computing device, such that at least one of the video or image content includes a distorted portion of the content; wherein the distortion of the at least a portion of the content is achieved by centering the content in the camera frustum and moving the Z component of the content vertices along the positive vector of the camera, the amount of movement of each content vertex depending on the current depth estimate of the nearest pixel to the X and Y coordinates of the content.

16. The computer-readable storage medium of claim 15, wherein the instructions, when executed by a processor, cause the processor to: Receive the second content; Obtain second depth information corresponding to the imaging environment outside the computing device; Receive a second instruction that associates the second content with a second location in the imaging environment based on the second depth information; and Distorting at least a portion of the second content based on the second depth information corresponding to the imaging environment outside the computing device. The at least one of the video or image content of the imaging environment outside the computing device includes the distorted portion of the content and the distorted portion of the second content.

17. The computer-readable storage medium of claim 15, wherein the content includes video content comprising a plurality of video frames, and wherein the plurality of video frames are displayed in at least one of the video or image content of the imaging environment outside the computing device.

Images