Structured display shutdown of video pass-through electronic device

Abstract

The subject technology provides structured shutdown mode versions of video views of a physical setting of an electronic device. The electronic device can provide a video view of a portion of the physical setting at a first power consumption rate, the portion being physically obstructed from direct view by a user of the device by the electronic device itself. When the electronic device determines that it is in a low power or failure state, a structured shutdown mode version of the video view can be provided to a display of the device at a second power consumption rate that is lower than the first power consumption rate. In this way, the electronic device can reduce functionality and power consumption of the display in a structured manner to continue to provide a version of the video view to the user in the low power or failure mode state.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 083,806, filed September 25, 2020, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This instruction manual pertains to extended reality environments. Background Technology

[0004] Augmented reality (AR) technology aims to bridge the gap between virtual and physical environments by providing an enhanced physical environment that utilizes computer-generated content that is not part of the physical environment. Therefore, the computer-generated content, which is not part of the physical environment, appears to be part of the physical environment as perceived by the user. AR is sometimes provided by devices that offer the user a pass-through video view of the physical environment and add computer-generated content to that view. Attached Figure Description

[0005] Some features of this subject matter are set forth in the appended claims. However, for illustrative purposes, several embodiments of this subject matter are illustrated in the following figures.

[0006] Figures 1A to 1B An exemplary system for various extended reality technologies, including virtual reality and mixed reality, is described according to one or more specific implementations.

[0007] Figure 2 An example of a physical scene is shown, which is an electronic device displaying a video view of a portion of a physical scene according to one or more specific implementations.

[0008] Figure 3 It is shown according to one or more specific implementations Figure 2 An example of a structured shutdown mode version of the video view.

[0009] Figure 4 It is shown according to one or more specific implementations Figure 2 Another example of a structured shutdown mode version of the video view.

[0010] Figure 5 A flowchart illustrating an exemplary process for providing structured display shutdown for a video pass-through electronic device, according to a specific implementation of the subject matter, is shown. Detailed Implementation

[0011] The specific embodiments shown below are intended to describe various configurations of the subject matter and are not intended to represent the only configuration in which the subject matter can be practiced. The accompanying drawings are incorporated herein and form part of the specific embodiments. The specific embodiments include particular details intended to provide a thorough understanding of the subject matter. However, the subject matter is not limited to the specific details set forth herein, but may be practiced in one or more other specific embodiments. In one or more specific embodiments, structures and components are shown in block diagram form to avoid obscuring the concepts of the subject matter.

[0012] Humans can interact with and / or perceive the physical environment or physical world without the aid of electronic devices. The physical environment can include physical features, such as physical objects or surfaces. An example of a physical environment is a physical forest that includes physical plants and animals. Humans can directly perceive and / or interact with the physical environment through various means, such as hearing, vision, taste, touch, and smell. In contrast, humans can use electronic devices to interact with and / or perceive a fully or partially simulated extended reality (XR) environment. This XR environment can include mixed reality (MR) content, augmented reality (AR) content, virtual reality (VR) content, and so on. Using an XR system, a person's physical movements, or some of their representations, can be tracked, and in response, the characteristics of virtual objects simulated in the XR environment can be adjusted in a manner consistent with at least one physical law. For example, the XR system can detect movement of the user's head and adjust the graphical and auditory content presented to the user (similar to how such views and sounds change in a physical environment). For example, the XR system can detect movement of electronic devices (e.g., mobile phones, tablets, laptops, etc.) presenting the XR environment and adjust the graphical and auditory content presented to the user (similar to how such views and sounds change in a physical environment). In some cases, the XR system can adjust the characteristics of the graphical content in response to other inputs such as representations of physical motion (e.g., voice commands).

[0013] Many different types of electronic systems enable users to interact with and / or perceive XR environments. A non-exclusive list of examples includes head-up displays (HUDs), head-mounted systems, projection-based systems, windows or vehicle windshields with integrated display capabilities, displays formed as lenses placed over the user's eyes (e.g., contact lenses), head-mounted receivers / headsets, input systems with or without haptic feedback (e.g., wearable or handheld controllers), speaker arrays, smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have opaque displays and one or more speakers. Other head-mounted systems may be configured to accept opaque external displays (e.g., smartphones). Head-mounted systems may include one or more image sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light is directed to the user's eyes. Displays can utilize various display technologies, such as uLED, OLED, LED, liquid crystal on silicon, laser scanning light sources, digital light projection, or combinations thereof. Optical waveguides, optical reflectors, holographic media, optical combiners, or combinations thereof, or other similar technologies can be used as the medium. In some implementations, transparent or translucent displays can be selectively controlled to become opaque. Projection-based systems can utilize retinal projection technology, which projects graphic images onto a user's retina. Projection systems can also project virtual objects into the physical environment (e.g., as holograms or onto physical surfaces).

[0014] In XR systems with opaque displays or transparent or semi-transparent displays controlled to become opaque (e.g., XR systems including those implementing video pass-through electronics), users may not have a direct view of part or all of the physical set of these XR systems when the system / device is in use. In VR systems, the user's entire view is a virtual set. In pass-through AR systems, the user relies on the system's camera to provide a video view of the set. In scenarios where the system has low battery power, or in cases where a part of the system providing information for processing video feeds (e.g., eye-tracking sensors and / or depth sensors) and / or video feeds processed from the camera (e.g., a video graphics processor) to provide a video view fails, aspects of the subject matter can help maintain a view of the physical set.

[0015] The specific implementations of the subject matter techniques described herein provide various features for ensuring that users of XR electronic devices and / or video pass-through electronic devices receive a view of the scene in the event of low battery power or system or subsystem failure. As described in further detail below, one or more structured shutdown mode versions of the video view can be provided in the event of low power or failure of the device.

[0016] Figure 1A and Figure 1B An exemplary system 100 for various extended reality technologies is depicted.

[0017] In some examples, such as Figure 1A As shown, system 100 includes device 100a. Device 100a includes various components such as processor 102, RF circuitry 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120, and touch-sensitive surface 122. These components optionally communicate via communication bus 150 of device 100a.

[0018] In some examples, components of system 100 are implemented in a base station device (e.g., a computing device, such as a remote server, mobile device, or laptop computer), and other components of system 100 are implemented in a second device (e.g., a head-mounted device). In some examples, device 100a is implemented in either the base station device or the second device.

[0019] like Figure 1B As shown, in some examples, system 100 includes two (or more) communicating devices, such as via wired or wireless connections. A first device 100b (e.g., a base station device) includes a processor 102, RF circuitry 104, and a memory 106. These components optionally communicate via a communication bus 150 of device 100b. A second device 100c (e.g., a head-mounted device) includes various components such as processor 102, RF circuitry 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120, and touch-sensitive surface 122. These components optionally communicate via a communication bus 150 of device 100c.

[0020] System 100 includes a processor 102 and a memory 106. Processor 102 includes one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 106 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) storing computer-readable instructions configured to be executed by processor 102 to perform the techniques described below.

[0021] System 100 includes RF circuitry 104. RF circuitry 104 optionally includes circuitry for communicating with electronic devices, networks (such as the Internet, intranets), and / or wireless networks (such as cellular networks and wireless local area networks (LANs)). RF circuitry 104 optionally includes circuitry for using near-field communication and / or short-range communication (such as... The circuit used for communication.

[0022] System 100 includes a display 120. One or more displays 120 may have an opaque display. One or more displays 120 may have a transparent or translucent display, which may be combined with a substrate through which light representing an image is directed to an individual's eye. One or more displays 120 may combine LEDs, OLEDs, digital light projectors, laser scanning light sources, liquid crystal on silicon, or any combination of these technologies. The light-transmitting substrate may be an optical waveguide, an optical combiner, a light reflector, a holographic substrate, or any combination of these substrates. In one example, the transparent or translucent display may selectively switch between an opaque state and a transparent or translucent state. Other examples of one or more displays 120 include head-up displays, automotive windshields capable of displaying graphics, windows capable of displaying graphics, lenses capable of displaying graphics, tablet computers, smartphones, and desktop or laptop computers. Alternatively, system 100 may be designed to receive an external display (e.g., a smartphone). In some examples, system 100 is a projection-based system that uses retinal projection to project images onto an individual's retina or to project virtual objects into a physical setting (e.g., onto a physical surface or as a hologram).

[0023] In some examples, system 100 includes a touch-sensitive surface 122 for receiving user input, such as tap input and swipe input. In some examples, display 120 and touch-sensitive surface 122 form a touch-sensitive display.

[0024] System 100 includes image sensor 108. One or more image sensors 108 optionally include one or more visible light image sensors, such as charge-coupled device (CCD) sensors and / or complementary metal-oxide-semiconductor (CMOS) sensors, operable to acquire images of physical elements from a physical scene. One or more image sensors also optionally include one or more infrared (IR) sensors, such as passive or active IR sensors, for detecting infrared light from the physical scene. For example, an active IR sensor includes an IR emitter, such as an IR point emitter, for emitting infrared light into the physical scene. One or more image sensors 108 also optionally include one or more event cameras configured to capture movement of physical elements within the physical scene. One or more image sensors 108 also optionally include one or more depth sensors configured to detect the distance between physical elements and system 100. In some examples, system 100 uses a combination of CCD sensors, event cameras, and depth sensors to detect the physical scene surrounding system 100. In some examples, image sensor 108 includes a first image sensor and a second image sensor. The first and second image sensors are optionally configured to capture images of physical elements in a physical setting from two different perspectives. In some examples, system 100 uses image sensor 108 to receive user input, such as gestures. In some examples, system 100 uses one or more image sensors 108 to detect the position and orientation of system 100 and / or one or more displays 120 in the physical setting. For example, system 100 uses one or more image sensors 108 to track the position and orientation of one or more displays 120 relative to one or more fixed elements in the physical setting.

[0025] In some examples, system 100 includes microphone 112. System 100 uses one or more microphones 112 to detect sound from a user and / or the user's physical set. In some examples, microphone 112 includes a microphone array (including multiple microphones) that optionally cooperate to identify ambient noise or locate sound sources in the space of the physical set.

[0026] System 100 includes an orientation sensor 110 for detecting the orientation and / or movement of system 100 and / or display 120. For example, system 100 uses one or more orientation sensors 110 to track changes in the position and / or orientation of system 100 and / or one or more displays 120, such as relative to physical elements in a physical setting. The orientation sensor 110 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0027] Figure 2An exemplary physical setting of an electronic device implemented as a video pass-through electronic device is shown. Figure 2 In the example, the physical setting 200 of an electronic device, such as electronic device 100a, includes physical objects 202 and 210. As shown, the user's eye 201 may have a line of sight 203, which will fall on physical object 202 when electronic device 100a is not present. In this example, physical object 210 will be visible to the user even if the user's gaze is directed at physical object 202.

[0028] As shown in the figure, electronic device 100a (e.g., including display 205 of electronic device, such as...) Figure 1A and Figure 1B The specific implementation of the display 120 may obstruct the user's direct view of physical objects 202 and 210. To provide the user with a view of the physical objects 202 and 210 (and / or other parts of the physical set) obstructed by the presence of the electronic device 100a and the display 205, a video view (e.g., a video pass-through view) of the physical set 200 may be displayed by the display 205. For example, one or more cameras of the electronic device 100a may capture images of the physical set that can be displayed by the display 205.

[0029] exist Figure 2 In the example, a video view of at least a portion of the physical set 200 that is blocked from direct view by the electronic device 100a is displayed to the user by the display 205, including a video version 204 of physical object 202 and a video version 208 of physical object 210. As shown, the electronic device 100a can also generate and display virtual content 212, which can be overlaid or otherwise added to and / or merged with the video view of the physical set to generate an extended reality set for the user of the electronic device.

[0030] exist Figure 2 In the example, video version 204 of physical object 202 and video version 208 of physical object 210 are displayed by display 205 to appear to the user as if they were at their actual physical locations (e.g., the correct angular position and depth), as if they were directly viewed by the user (e.g., in the absence of electronic device 100a). For example, electronic device 100a may use depth sensor 129 (e.g., to implement...). Figure 1A and Figure 1BThe image sensor 108, orientation sensor 110, and / or position sensor 116 (or one or more of these) obtain a depth map of a portion of the physical set 200, and may include one or more additional cameras and / or sensors (such as eye sensor 222) that track the gaze position of each of the user's eyes and / or the location where the gaze directions of the user's eyes converge (e.g., at the gaze plane). The depth map of the physical set 200 and the current eye data of the user's eye 201 can be used to perform pixel-level transformations (distortions or reprojections) on image frames from camera 119 to generate display frames from the correct viewpoint and position of the user's eyes at any given time, and to reproject each pixel at the correct depth according to the depth map.

[0031] For example, to provide the most realistic experience for the user, the view of the physical scene 200 displayed by the monitor 205 will include all light from the physical scene that enters the user's eyes when the electronic device 100a is not present (e.g., if the monitor 205 and / or other parts of the electronic device 100a do not obstruct the user's direct view of the physical scene). However, as Figure 2 As shown, one or more of the cameras in the cameras 119 that collect light from the physical set 200 may be offset relative to the position and / or gaze direction of the user's eye 201. That is, at any given time (e.g., based on the gaze direction of each eye at that time), the line of sight 203 from the user's eye 201 to the physical object 202 may be different from the line of sight 206 of the camera.

[0032] Therefore, the camera view of the physical scene can be distorted by the electronic device 100a (e.g., transformed using reprojection of each pixel based on current eye data from eye sensor 222 and depth map generated using sensor 129) to display the physical scene and physical objects therein to the user from the user's perspective.

[0033] The field of view of any of the cameras in camera 119 may also be limited or different compared to the field of view of a human user. For example, parts of the physical scene in the user's peripheral vision may not be within the field of view of the color camera of the electronic device. As another example, distorting image data to transform it to the user's perspective (e.g., to the current line of sight 203 as determined by eye sensor 222) may cause parts of objects that would normally be visible to the user (e.g., parts along the edges of objects) to be displayed incorrectly because these parts are not visible from the camera's perspective. To provide the user with a more complete field of view of the physical scene and / or correct for differences in perspective with display 205, images from multiple cameras (e.g., camera feeds or video feeds) may be combined to form each display frame displayed to the user by display 205.

[0034] In one or more embodiments, the camera 119 of the electronic device 100a may include, for example, one or more main cameras and one or more auxiliary cameras. The auxiliary cameras may consume less power than the main cameras. For example, the main cameras may include one or more color cameras, and the auxiliary cameras may include one or more monochrome (e.g., black and white) cameras. The main cameras may have a higher resolution than the auxiliary cameras. In one or more embodiments, the color image obtained by the main camera can be used (e.g., distorted, processed, and / or rendered) to generate a video pass-through view of the physical scene displayed to the user, while the (e.g., monochrome) image from the auxiliary camera can be used for other tasks such as object detection and tracking. In one or more embodiments, portions of the physical scene 200 that are not included in the main camera (e.g., color) image (e.g., due to the different physical position and / or field of view of the auxiliary camera relative to the main camera) can also be used to fill in missing portions of the main camera's field of view. For example, Figure 2 Parts 221 (e.g., at or near the edge of the user's field of view) and 223 (e.g., along the edge of display version 208 of physical object 210) of the display frame are shown, which have been filled with monochrome camera images.

[0035] To provide a user with a visually seamless video view of the physical set, portions of a monochrome image used to fill the missing parts of the color camera's field of view can be colored. Coloring these portions of the monochrome image can include (e.g., by providing color information from the color image to a machine learning model at an electronic device) determining the color of each pixel in the monochrome image used to fill the missing parts of the color camera's field of view based on the colors of pixels in the color image. The determined colors can then be used to display portions 221 and 223.

[0036] Obtaining eye-tracking data and depth maps, processing image data from camera 119 to distort, fill, and / or colorize the image data, and / or generating virtual content and adding it to the processed image data to generate each display frame can be processor- and / or power-intensive operations (e.g., these operations combined to consume power at a first power consumption rate during normal operation of the display). Performing these processor- and / or power-intensive operations depends on the power of the device (e.g., from a power source, such as one or more batteries at the electronic device) and / or the availability of various processing resources (e.g., general-purpose and / or dedicated processing resources such as those for sensor 129 and eye sensor 222, and video processing components and / or processes, rendering components and / or processes, virtual content generation components and / or processes). For example, an electronic device such as electronic device 100a may include a first processing circuitry (e.g., an image signal processor (ISP) for streaming camera data from camera 119 to display 205) and a second processing circuitry (e.g., an additional video processing chip or processing core) for receiving camera data and sensor data and generating distorted, filled, and / or colored video views for display to the user.

[0037] In some cases, processing resources may fail (e.g., due to mechanical, electrical, or software problems) and / or power may become limited and / or unavailable (e.g., due to system failure and / or low battery power), resulting in system shutdown or limitation of some processing resources. During the use of general electronic devices displaying content unrelated to the physical scene, low-power states (e.g., states with limited, unavailable, and / or insufficient power to process and display camera data and / or virtual content) or failure states (e.g., states where one or more of the general processing resources and / or dedicated processing resources (such as video processing components and / or processes, rendering components and / or processes, virtual content generation components and / or processes) used to generate display frames for display become damaged or unavailable) can lead to a loss of display functionality, which may be inconvenient or undesirable for the user.

[0038] In electronic devices such as electronic device 100a, users use the camera and display of the electronic device to view and / or navigate part or all of their own physical set (e.g., via a camera, display, or other means). Figure 2 A low-power or faulty state that causes a loss of display functionality (as shown in the video view of the physical scene) can be particularly undesirable. This is especially true when a portion of the scene viewed by a user on the display of an electronic device is obstructed by the display and / or the electronic device itself (e.g., by the user holding the device to their eyes or wearing a head-mounted system that partially or completely obstructs their view of part or all of the physical scene).

[0039] The electronic devices disclosed herein (such as electronic device 100a) implement various structured display shutdown measures to help maintain at least one version of the video view of the physical scene for the user (e.g., at a second power consumption rate lower than the first power consumption rate), even in the case of a low power state or a fault state of the electronic device.

[0040] For example, in Figure 2 In the illustrated configuration, an electronic device (e.g., electronic device 100a, also referred to herein as a device) is provided with a display 205 configured to display virtual content 212 to a user located within a physical set 200. As shown, the device also includes one or more cameras 119 configured to obtain video feeds of at least a portion of the physical set 200. The device can process the video feeds to generate a video view of that portion of the physical set (e.g., a portion of the physical set including one or more physical objects such as physical objects 202 and 210 at various locations and / or depths) and provide the display with a video view corresponding to the processed video feed for display alongside the virtual content.

[0041] As mentioned above Figure 2 As described, the device can process video feeds to generate a video view (e.g., in the device's normal operating mode) by distorting the video feed to conform the video feed from the view of camera 119 to the user's view and combining virtual content 212 with the distorted video feed for display. In one or more embodiments, the device can obtain the user's current eye data (e.g., line of sight 203 corresponding to the gaze direction of one or both of the user's eyes, and / or gaze plane corresponding to the convergence position of the user's gaze direction) from at least one eye sensor, such as eye sensor 222 (e.g., including light sources for each of the user's eyes and / or eye-tracking sensors for the camera), and distort the video feed to the user's view based on the current eye data. Distorting the video feed based on the current eye data may include performing pixel-by-pixel distortion or reprojection of the video feed based on the current eye data and a depth map of the physical scene.

[0042] As mentioned above Figure 2 As described, the device can also process video feeds to generate a video view by: obtaining a monochrome video feed from at least one monochrome camera (e.g., a monochrome or black-and-white camera, such as a monochrome visible light or infrared camera); performing a color estimate of a portion of the monochrome video feed based on the video feed from the color camera; and filling a portion of the video feed (e.g., portions 221 and 223 corresponding to the displayed video frames) using the monochrome video feed and the color estimate.

[0043] In one or more embodiments, the device may identify a display modification state (e.g., a degraded state of the device, such as a low-power state or a fault state). In response to identifying a display modification, the device may modify the processing of the video feed to provide a structured shutdown mode version of the video view of a portion of the physical scene to the display 205 while the device is in a display modification state (e.g., at a second power consumption rate lower than the first power consumption rate). The structured shutdown mode version of the video view can be generated using reduced power and / or processing resources relative to the generation of the video view. Figure 3 and Figure 4 An example of a structured shutdown mode version of a video view that can be generated when the device is in a modified display state (e.g., a low power state or a fault state) is shown. The device may provide one possible structured shutdown mode version of the video view (e.g., one example in the examples described herein), or multiple possible structured shutdown mode versions of the video view may be provided (e.g., each version may be available depending on the severity of the low power state or fault state).

[0044] Figure 3 It shows Figure 2 Examples of structured shutdown mode versions of the video view. For instance, a structured shutdown mode version of the video view can be generated in a display modification state (e.g., low power state or fault state) where the eye sensor 222, depth sensor, and / or video graphics processor are unavailable. Figure 3 In the example, electronic device 100a has modified the video feed processing to generate a structured shutdown mode version of the video view by: interrupting the distortion of the video feed based on current eye data; obtaining the user's nominal eye data; and performing a transformation of the video feed to a predetermined surface (e.g., an optical surface such as surface 300) based on the nominal eye data. The transformation of the video feed to the predefined surface can be a homography transformation.

[0045] Nominal eye data may include interpupillary distance (IPD) and an assumption of the user's neutral gaze position (e.g., the user's gaze position under the assumption that the user is looking directly at the horizon). Interpupillary distance (IPD) may be obtained from device memory, such as from the last known IPD measurement from eye sensor 222, or from hardware components of a head-mounted system (e.g., a linear encoder) that physically locate the display module of the device.

[0046] In one example, during normal operation, an IPD measurement can be determined based on eye-tracking data from eye sensor 222 (e.g., periodically or continuously) and stored in memory (e.g., at the display or in the ISP's memory) for later use in low-power or fault states of the electronic device that triggers a display modification state. In another example, the IPD can be determined (e.g., by the user or by the device) based on the mechanical and / or electronic setup of the physical distance between display modules for each of the user's eyes, which can be obtained from a linear encoder to serve as the user's interpupillary distance. The IPD can be determined based on the IPD and the assumed eye position under neutral gaze (e.g., when measurements of the user's actual gaze direction are interrupted to save power and / or processed in low-power or fault modes, or when unavailable due to a fault mode). Figure 3 The structured shutdown mode version of the projected video view is 300 meters away from the surface.

[0047] exist Figure 3 In the example, surface 300 is depicted as a planar optical surface. In other specific implementations, other optical surfaces may be used for homography transformation, such as spherical optical surfaces on which video pixels can be projected.

[0048] In one or more specific implementations, Figure 3 The transformation of the video feed shown can be a pixel-independent homography transformation. For example, each pixel of a video frame can be projected onto the same predetermined surface 300, regardless of the depth of the physical object represented in that pixel, which is consistent with... Figure 2 The distortion shown in the figure creates a contrast. For the distortion shown in the figure, the individual pixels of the video frame are reprojected for the user based on the user's current gaze position and the depth of the physical object in that pixel, so that the content of that pixel is projected to the correct optical depth.

[0049] like Figure 3 As shown, when video versions 204 and 208 of physical objects 202 and 210 are projected onto the common surface 300, video versions 204 and 208 may appear to the user as planar objects 302 and 310 at a common optical depth, rather than appearing at the correct corresponding depth and having a three-dimensional shape in the physical scene (e.g., as shown). Figure 2 (As shown). Therefore, as Figure 3 The transformation and projection shown onto the predetermined surface 300 are therefore more complex than the combination. Figure 2 The description is a distorted, lower-quality projection.

[0050] However, since IPD determination is performed only once (e.g., compared to current eye tracking used for warping), and because the transformation to the predetermined surface is calculated pixel-independently, projecting the video view onto such a surface... Figure 3Reduced power and / or processing resources can be used to perform this on the predetermined surface 300 shown. Therefore, the video processing is modified to... Figure 3 The homography transformation can provide a specific implementation of a structured shutdown mode version of the video feed that can be provided when the device is in a display modification state (e.g., a low power state or a fault state).

[0051] Figure 3 It also illustrates other modifications that can be performed when the device is in a display modification state (e.g., along with and / or in lieu of transformations and projections onto a predetermined surface 300) to generate a structured shutdown mode version of the video view. For example, such as Figure 3 As shown, in a structured shutdown mode version of the video view, the generation and display of virtual content 212 can be interrupted. In one or more structured shutdown mode versions of the video view, transformations and projections onto a predetermined surface 300 can be performed without combining the virtual content with the transformed video feed, such as... Figure 3 As shown. In one or more structured shutdown mode versions of the video view, the device can modify the processing of the video feed to continue performing distortions of the video feed based on the current eye data (e.g., as...). Figure 2 (as shown) Instead of combining virtual content with distorted video feeds to generate a structured shutdown mode version of the video view.

[0052] Figure 3 It also shows how the device can modify the processing of the video feed to generate a structured shutdown mode version of the video view by interrupting the filling of portions 221 and / or 223 and / or interrupting the color estimation of the filled pixels using data from other sensors and / or cameras (such as monochrome cameras). Figure 3 In the example, color estimation is interrupted, and grayscale fill is provided in sections 221 and 223. In one or more implementations, the fill itself may also be interrupted.

[0053] In one or more embodiments, the displayed modified state may be a low-power state, which triggers a switching of a structured shutdown mode version of the device's video view and may be identified based on the device's own power level (e.g., battery charging level) and / or determined based on additional information. For example, the device may use one or more of its sensors to determine the user's position (e.g., seated or standing). The device may identify the low-power state of the device at least in part based on the user's position detected by the device. For example, the power threshold generated for identifying the low-power state and thus triggering a power-saving structured shutdown mode version of the video view may be higher when the user is standing than when the user is sitting. In this way, if the user is in a seated or other stationary position, the device can continue to operate in normal operating mode for a longer period of time.

[0054] Figure 4 This illustrates another modification to a structured shutdown mode version that can be executed to generate a video view when the device is in a modified display state (e.g., a low-power state or a fault state). Figure 4 In the example, the device has interrupted all transformations of the video feed from camera 119 and displays the video feed from the camera without modification. For example... Figure 4 As shown, displaying the video feed without modification can cause video versions 204 and 208 of physical objects 202 and 210 to appear to be in positions that are shifted relative to their actual physical locations (e.g., because the line of sight 206 of camera 119 is different from the line of sight 203 of the user's eye). Figure 4 The structured shutdown mode version of the video view can be a distorted view that includes the missing portion 421 of the view (e.g., the area corresponding to the area outside the camera's field of view), and is therefore provided as a last resort for the structured shutdown mode version of the video view before the device and / or display are completely powered off.

[0055] In various operational scenarios, the above text combines Figure 2 , Figure 3 and Figure 4 Any or all of the structured shutdown mode versions of the video view can be used to structurally reduce the display's functionality and / or power consumption rate while continuing to provide the user with a view of the user's physical surroundings in any of the device's various low-power modes and / or fault modes. In scenarios where the user does not take action to resolve the device's display modification state, the device can detect critical faults in the structured shutdown mode versions of the video view (e.g., due to further failure of the processing system and / or components, and / or due to the power level dropping below the minimum level required to operate the camera and / or display). The device can provide warnings to the user (e.g., audio warnings, haptic warnings, visual warnings, and / or any other output of the device to alert the user) to move the display to remove obstructions to the direct view due to critical faults (e.g., to remove the head-mounted system from the user's head before the display and / or device power completely fails).

[0056] Figure 5 A flowchart illustrating an exemplary process 500 for providing a structured shutdown mode video view of the physical setting of an electronic device, according to a specific implementation of the subject matter, is shown. For illustrative purposes, this document primarily refers to... Figure 1A and Figure 1B The process 500 is described using electronic device 100a. However, process 500 is not limited to... Figure 1A and Figure 1BThe electronic device 100a, and one or more blocks (or operations) of process 500 may be performed by one or more other components of other suitable devices (including electronic devices 100b and / or electronic devices 100c). Further for illustrative purposes, some blocks of process 500 are described herein as occurring sequentially or linearly. However, multiple blocks of process 500 may occur in parallel. Furthermore, the blocks of process 500 need not be performed in the order shown, and / or one or more blocks of process 500 need not be performed and / or may be replaced by other operations.

[0057] like Figure 5 As shown, at box 502, a virtual content (e.g., virtual content 212) located within a physical setting is displayed to a user using a display (e.g., display 205) of an electronic device such as electronic device 100a. This virtual content may be generated by the electronic device to be displayed as part of an extended reality setting that includes a video view of part or all of the physical setting.

[0058] At frame 504, a video feed of at least a portion of the physical set is obtained using a camera of an electronic device. The camera may be a color camera or a monochrome camera. The electronic device may include one or more cameras (such as camera 119 as described herein) and / or one or more sensors (such as sensor 129 and / or eye sensor 222 as described herein).

[0059] At box 506, the video feed can be processed to generate a video view of a portion of the physical set (e.g., at a first power consumption rate). In one or more embodiments, processing the video feed to generate the video view may include: obtaining current eye data of a user from at least one eye sensor; warping the video feed based on the current eye data so that the video feed from the camera's view conforms to the user's view; and combining virtual content with the warped video feed for display (e.g., as combined above). Figure 2 (As described).

[0060] In one or more embodiments, the camera may be a color camera, and processing the video feed to generate a video view may include: obtaining a monochrome video feed from at least one monochrome camera of the electronic device; performing a color estimate of a portion of the monochrome video feed based on the video feed from the color camera; and filling a portion of the video feed using the monochrome video feed and the color estimate (e.g., as combined above). Figure 2(As described). Various device operations for obtaining a user's current eye data from at least one eye sensor, distorting a video feed based on the current eye data to make a video feed from a view from a camera conform to the user's view, combining virtual content with the distorted video feed for display, obtaining a monochrome video feed from at least one monochrome camera of an electronic device, performing color estimation on a portion of the monochrome video feed based on a video feed from a color camera, and / or using the monochrome video feed and color estimation to fill a portion of the video feed may each consume device power, and the combination of these device powers results in power consumption at a first power consumption rate.

[0061] At box 508, a video view corresponding to the processed video feed can be provided to the display for display along with virtual content (e.g., as described above). Figure 2 (As described).

[0062] At box 510, a modified state (e.g., low power state or fault state) may be identified for the device. Identifying the modified state may include identifying a low power state, which may include determining that the device battery's charge level is below a threshold (e.g., a fixed threshold or a threshold determined based on the user's location and / or movement). Identifying the modified state may include identifying a fault state, which may include identifying a malfunction of a sensor, camera, or processing resource such as a video graphics processor.

[0063] At box 512, in response to recognizing a display modification state, a structured shutdown mode version of the video feed processing can be modified to provide a video view of a portion of the physical scene to the display when the device is in a display modification state (e.g., at a second power consumption rate lower than the first power consumption rate). In one or more embodiments, when the structured shutdown mode view is provided (e.g., in a display modification state), a notification may be displayed or otherwise provided to the user indicating that the device is in an adjusted operating state, such as a display modification state. The structured shutdown mode version of modifying the video feed processing to provide a video view of a portion of the physical scene to the display may include interrupting video feed distortion, interrupting the generation and display of virtual content, interrupting fill and / or color estimation operations, and / or providing the video feed to the display unmodified, as described above in conjunction with, for example... Figure 2 , Figure 3 and Figure 4 As described. It should be understood that interrupting the distortion of the video feed, interrupting the generation and display of virtual content, interrupting fill and / or color estimation operations, and / or providing the video feed to the display without modification can reduce the power consumption of the device and result in providing a version of the video feed for display at a second power consumption rate lower than the first power consumption rate.

[0064] In various embodiments, modifying the video feed process may include providing the display with an unmodified video feed, such as that captured by a camera, or interrupting processing with distortion, fill, and / or color estimation operations. In one or more embodiments, a structured shutdown mode version of modifying the video feed process to provide a portion of a video view of a physical scene to the display may include: interrupting the distortion of the video feed based on current eye data; obtaining the user's nominal eye data; and performing a transformation of the video feed to a predefined surface based on the nominal eye data. In one or more embodiments, distorting the video feed based on the current eye data may include performing pixel-by-pixel distortion based on the current eye data. The transformation of the video feed may include pixel-independent transformations, such as transformations to a predetermined surface. In one or more embodiments, a structured shutdown mode version of modifying the video feed process to generate a video view includes interrupting at least one of color estimation or fill.

[0065] The various processes defined in this document consider options for obtaining and using users' personal information. For example, such personal information may be used to provide video views of physical sets. However, the extent to which such personal information is collected should be based on the user's informed consent. As described in this document, users should understand and control the use of their personal information.

[0066] Personal information will be used by the appropriate parties only for lawful and reasonable purposes. Parties using such information will comply with privacy policies and practices that are at least in accordance with applicable laws and regulations. Furthermore, such policies should be comprehensive, user-accessible, and considered to meet or exceed government / industry standards. In addition, parties may not distribute, sell, or otherwise share such information except for any reasonable and lawful purpose.

[0067] However, users can limit the extent to which parties can access or otherwise obtain their personal information. For example, settings or other preferences can be adjusted so that users can decide whether their personal information can be accessed by various entities. Furthermore, while some of the characteristics defined herein are described in the context of the use of personal information, aspects of these characteristics can be implemented without the need for such information. For example, if user preferences, account names, and / or location history are collected, this information can be obfuscated or otherwise generalized so that it does not identify the corresponding user.

[0068] These functions can be implemented in computer software, firmware, or hardware. The technology can be implemented using one or more computer program products. Programmable processors and computers can be included in or packaged as mobile devices. The process and logic flow can be executed by one or more programmable processors and one or more programmable logic circuits. General-purpose and special-purpose computing devices, as well as storage devices, can be interconnected via communication networks.

[0069] Some specific implementations include electronic components such as microprocessors, storage devices, and memories that store computer program instructions in machine-readable or computer-readable media (also known as computer-readable storage media, machine-readable media, or machine-readable storage media). Examples of such computer-readable media include RAM, ROM, read-only optical discs (CD-ROM), recordable optical discs (CD-R), rewritable optical discs (CD-RW), read-only digital versatile optical discs (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and / or solid-state hard disk drives, read-only and recordable Blu-ray discs... Disks, high-density optical discs, any other optical or magnetic media, and floppy disks. Computer-readable media may store computer programs that can be executed by at least one processing unit and include a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as machine code generated by a compiler, and files that include higher-level code that can be executed by a computer, electronic components, or microprocessor using an interpreter.

[0070] While the above discussion primarily concerns microprocessors or multi-core processors that execute software, some implementations are performed by one or more integrated circuits such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some implementations, such integrated circuits execute instructions stored on the circuit itself.

[0071] As used in this specification and any claim of this patent application, the terms "computer," "server," "processor," and "memory" refer to electronic or other technical devices. These terms exclude persons or groups of persons. For the purposes of this specification, the terms "displayed" or "being displayed" mean displayed on an electronic device. As used in this specification and any claim of this patent application, the terms "computer-readable medium" and "computer-readable media" are entirely limited to tangible, touchable objects that store information in a form readable by a computer. These terms do not include any wireless signals, wired download signals, or any other transient signals.

[0072] To provide interaction with the user, specific embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user and a keyboard and pointing device that the user can use to provide input to the computer. The display device is, for example, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and the pointing device is, for example, a mouse or trackball. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser in response to a request received from a web browser on the user's client device.

[0073] Implementations of the subject matter described in this specification can be implemented in a computing system that includes back-end components, such as a data server, or middleware components, such as an application server, or front-end components, such as a client computer with a graphical user interface or web browser through which a user can interact with a specific implementation of the subject matter described in this specification, or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), interconnected networks (e.g., the Internet) and peer-to-peer networks (e.g., self-organizing peer-to-peer networks).

[0074] A computing system may include clients and servers. Clients and servers are generally geographically separated but can interact via a communication network. The client-server relationship is established by computer programs running on respective computers and having a client-server relationship with each other. In some specific implementations, the server transmits data (e.g., HTML pages) to the client device (e.g., to display data to a user interacting with that client device and to receive user input from that user). Data generated at the client device (e.g., the result of user interaction) can be received from the client device at the server.

[0075] According to an aspect of this disclosure, a method is provided, comprising: acquiring a video feed of at least a portion of a physical scene using a camera of an electronic device; processing the video feed at a first power consumption rate to generate a video view of the portion of the physical scene; providing the video view corresponding to the processed video feed for display; identifying a display modification state of the device; and, in response to identifying the display modification state, modifying the processing of the video feed to provide a structured shutdown mode version of the video view of the portion of the physical scene for display at a second power consumption rate lower than the first power consumption rate when the device is in the display modification state.

[0076] According to an aspect of this disclosure, an apparatus is provided, the apparatus including a display; a camera configured to acquire a video feed of at least a portion of a physical scene; and one or more processors configured to: process the video feed at a first power consumption rate to generate a video view of the portion of the physical scene; provide the video view corresponding to the processed video feed to the display for display; identify a display modification state of the apparatus; and, in response to identifying the display modification state, modify the processing of the video feed to provide a structured shutdown mode version of the video view of the portion of the physical scene to the display at a second power consumption rate lower than the first power consumption rate when the apparatus is in the display modification state.

[0077] According to an aspect of this disclosure, a non-transitory computer-readable medium is provided that stores instructions, when executed by one or more processors of a device, to cause the one or more processors to: acquire a video feed of at least a portion of a physical scene using a camera of an electronic device; process the video feed at a first power consumption rate to generate a video view of the portion of the physical scene; provide the video view corresponding to the processed video feed for display; identify a display modification state of the device; and, in response to identifying the display modification state, modify the processing of the video feed to provide a structured shutdown mode version of the video view of the portion of the physical scene for display at a second power consumption rate lower than the first power consumption rate when the device is in the display modification state.

[0078] Those skilled in the art will recognize that the various exemplary blocks, modules, elements, components, methods, and algorithms described herein can be implemented as electronic hardware, computer software, or a combination of both. To illustrate this interchangeability between hardware and software, the various exemplary blocks, modules, elements, components, methods, and algorithms have been generally described above in terms of functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The functionality may be implemented differently for each specific application. Various components and blocks may be arranged differently (e.g., in different orders or divided in different ways) without departing from the scope of the subject matter.

[0079] It should be understood that the specific order or hierarchical structure of the steps in the process disclosed in this invention is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchical structure of the steps in the process can be rearranged. Some steps in the process can be performed simultaneously. The appended method claims present elements of various steps in the sample order and are not intended to limit one to the specific order or hierarchical structure presented.

[0080] The preceding description is provided to enable those skilled in the art to practice the various aspects described herein. The preceding description provides various examples of the subject matter, and the subject matter is not limited to these examples. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, this claim is not intended to be limited to the aspects shown herein, but rather to be consistent with the language of the claim, wherein reference to elements in the singular form is not intended to mean “only one,” but rather “one or more,” unless specifically indicated. Unless otherwise specifically stated, the term “some” means one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are for convenience only and do not limit the invention described herein.

[0081] As used herein, the term "website" can include any aspect of a website, including one or more web pages, one or more servers used to host or store web-related content, etc. Therefore, the term "website" is used interchangeably with the terms "web page" and "server." The predicates "configured to," "capable of operating," and "programmed to" do not imply any specific tangible or intangible modification to a particular subject but are intended to be used interchangeably. For example, a component or a processor configured to monitor and control operations may also mean that the processor is programmed to monitor and control operations or that the processor is capable of operating to monitor and control operations. Similarly, a processor configured to execute code can be interpreted as a processor programmed to execute code or capable of operating to execute code.

[0082] As used herein, the term "automatic" can include execution by a computer or machine without user intervention; for example, by instructions in response to a predicate action of a computer or machine or other initiating mechanism. The word "example" is used herein to mean "serving as an example or illustration." No aspect or design described herein as an "example" is necessarily to be construed as preferred or advantageous over other aspects or designs.

[0083] Phrases such as "aspect" do not imply that this aspect is essential to the subject matter or that this aspect applies to all configurations of the subject matter. Disclosures relating to an aspect may apply to all configurations, or one or more configurations. An aspect may provide one or more examples. Phrases such as "aspect" may refer to one or more aspects, or vice versa. Phrases such as "implementation" do not imply that this implementation is essential to the subject matter or that this implementation applies to all configurations of the subject matter. Disclosures relating to an implementation may apply to all implementations, or one or more implementations. An implementation may provide one or more examples. Phrases such as "implementation" may refer to one or more implementations, or vice versa. Phrases such as "configuration" do not imply that this configuration is essential to the subject matter or that this configuration applies to all configurations of the subject matter. Disclosures relating to a configuration may apply to all configurations or one or more configurations. A configuration may provide one or more examples. Phrases such as "configuration" may refer to one or more configurations, or vice versa.

Claims

1. An apparatus comprising: monitor; and One or more processors, said one or more processors being configured to: Obtain at least a portion of the input video feed of the physical set; The input video feed is processed to generate the processed video feed by the following operations: The input video feed is warped from the camera's view to the user's view to generate the warped video feed, and The virtual content is combined with the distorted video feed for display; A video view of the portion of the physical set is generated using the processed video feed at a first power consumption rate. Provide the display with the video view corresponding to the processed video feed for display; Identify the display modification status of the device; as well as In response to recognizing the display modification state, the processing of the input video feed is modified by omitting the virtual content from the distorted video feed to provide a structured shutdown mode version of the video view of the portion of the physical scene to the display at a second power consumption rate lower than the first power consumption rate when the device is in the display modification state.

2. The device according to claim 1, wherein the display modification state includes a low power state or a fault state.

3. The device of claim 2, further comprising at least one eye sensor, wherein the one or more processors are further configured to: Obtain the user's current eye data from the at least one eye sensor; and The input video is fed and distorted into the user's view based on the current eye data.

4. The device of claim 3, wherein the one or more processors are further configured to modify the processing of the input video feed to generate the structured shutdown mode version of the video view by: The distortion of the input video feed is interrupted based on the current eye data; Obtain the user's nominal eye data; as well as The input video is fed to a predefined surface and transformed based on the nominal eye data.

5. The device of claim 4, wherein the current eye data includes the gaze direction of each eye of the user, and wherein the nominal eye data includes interpupillary distance and neutral gaze.

6. The device of claim 4, wherein the one or more processors are configured to warp the input video feed based on the current eye data by performing pixel-by-pixel warping based on the current eye data and a depth map, and wherein the transformation of the input video feed comprises pixel-independent transformation.

7. The device of claim 3, wherein the one or more processors are configured to omit the virtual content from the distorted video feed by continuing to perform the distortion of the input video feed based on the current eye data without combining the virtual content with the distorted video feed.

8. The apparatus of claim 1, further comprising a color camera and at least one monochrome camera configured to generate the input video feed, and wherein the one or more processors are configured to process the input video feed to generate the video view by: Obtain monochrome video feed from the at least one monochrome camera; Color estimation is performed on a portion of the monochrome video feed based on the input video feed from the color camera; as well as The monochrome video feed and the color estimation are used to fill in a portion of the input video feed.

9. The device of claim 8, wherein the one or more processors are configured to modify the processing of the input video feed by interrupting at least one of the color estimation or the fill to generate the structured shutdown mode version of the video view.

10. The device of claim 1, wherein the one or more processors are configured to identify the display modification state at least in part by identifying a low-power state of the device based at least in part on the location of the user detected by the device.

11. An apparatus comprising: monitor; and One or more processors, said one or more processors being configured to: Obtain at least a portion of the input video feed of the physical set; The input video feed is processed at a first power consumption rate to generate a video view of the portion of the physical set; Provide the display with the video view corresponding to the processed input video feed for display; Identify the display modification status of the device; as well as In response to recognizing the display modification state, the processing of the input video feed is modified to provide a structured shutdown mode version of the video view of the portion of the physical scene to the display at a second power consumption rate lower than the first power consumption rate when the device is in the display modification state, wherein the display is configured to block a direct view of at least a portion of the physical scene from the user when the user is wearing the device, and wherein the one or more processors are further configured to: Detect critical faults in the structured shutdown mode version of the video view; as well as A warning is provided to the user to move the display to remove the obstruction to the direct view due to the critical malfunction.

12. The device of claim 11, wherein the one or more processors are configured to modify the processing of the input video feed by interrupting at least one of a distortion operation, a fill operation, or a color estimation operation of the processing.

13. A method comprising: Using electronic devices to obtain at least a portion of the input video feed of the physical set; The input video feed is processed at least in part by the following operations to generate the processed video feed: Obtain the user's current eye data from at least one eye sensor, and Based on the current eye data, the input video feed is distorted from the camera's view to the user's view; A video view of the portion of the physical set is generated using the processed video feed at a first power consumption rate; Provide the video view corresponding to the processed video feed for display; Identify the display modification status of the electronic device; as well as In response to recognizing the display modification state, the processing of the input video feed is modified by interrupting the distortion of the input video feed from the camera's view to the user's view based on the current eye data, so as to provide a structured shutdown mode version of the video view of the portion of the physical scene for display at a second power consumption rate lower than the first power consumption rate when the electronic device is in the display modification state.

14. The method of claim 13, wherein modifying the processing of the input video feed to generate the structured shutdown mode version of the video view further comprises: Obtain the user's nominal eye data; as well as The input video is fed to a predefined surface and transformed based on the nominal eye data.

15. The method of claim 14, wherein warping the input video feed based on the current eye data comprises performing pixel-by-pixel warping based on the current eye data and a depth map, and wherein the transformation of the input video feed comprises pixel-independent transformation.

16. The method of claim 13, wherein obtaining the input video feed comprises obtaining the input video feed from a color camera of the electronic device, and wherein processing the input video feed to generate a processed video feed comprises: Monochrome video feed is obtained from at least one monochrome camera of the electronic device; Color estimation is performed on a portion of the monochrome video feed based on the input video feed from the color camera; as well as The monochrome video feed and the color estimation are used to fill in the missing field of view of the input video feed.

17. The method of claim 16, wherein modifying the processing of the input video feed to generate the structured shutdown mode version of the video view includes interrupting at least one of the color estimation or the fill.

18. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of an electronic device, cause the one or more processors to: The input video feed of at least a portion of the physical scene is obtained using the color camera of the electronic device; The input video feed is processed at least in part by the following operations to generate the processed video feed: Obtain monochrome video feed from at least one monochrome camera; Color estimation is performed on a portion of the monochrome video feed based on the input video feed from the color camera; as well as The monochrome video feed and the color estimation are used to fill in a portion of the input video feed; A video view of the portion of the physical set is generated using the processed video feed at a first power consumption rate. Provide the video view corresponding to the processed video feed for display; Identify the display modification status of the electronic device; as well as In response to recognizing the display modification state, the processing of the input video feed is modified by interrupting at least one of the color estimation or the fill, so as to provide a structured shutdown mode version of the video view of the portion of the physical scene for display at a second power consumption rate lower than the first power consumption rate when the electronic device is in the display modification state.

19. The non-transitory computer-readable medium of claim 18, wherein the instructions, when executed by the one or more processors of the electronic device, cause the one or more processors to further modify the processing of the input video feed by providing the input video feed captured by the camera of the electronic device to a display without modification.

20. The non-transitory computer-readable medium of claim 18, wherein the instructions, when executed by the one or more processors of the electronic device, cause the one or more processors to further modify the processing of the input video feed by an interrupt warp operation.

Images