Device with dynamic transcode throttling

Abstract

Dynamic transcode throttling methods and devices for processing resource management and thermal mitigation in electronic devices, such as eyewear devices, are provided. The electronic device monitors its temperature and configures the transcoding service to operate at different rates in response to the temperature. A frame delay module is configured to add a delay between read frames prior to the transcoding service. This allows the electronic device to consume less power when temperatures are too high to provide thermal mitigation, and can be performed without powering down the electronic device.

Description

Technology Field

[0001] This application claims priority to U.S. Application Serial No. 17 / 373,559 filed on July 12, 2021 and U.S. Provisional Application Serial No. 63 / 208,937 filed on June 9, 2021, the contents of both of these applications are fully incorporated herein by reference.

[0002] The examples presented in this disclosure relate to the field of extended reality (XR; e.g., augmented, virtual, and mixed reality) and electronic devices, such as mobile devices and eyewear devices. More specifically, but not limited thereto, this disclosure describes dynamic throttling when transcoding extended reality images to process power management and thermal mitigation.

[0003] Augmented reality devices generate images (e.g., overlays) for the user to view and present them on displays. Devices can record and store images for later viewing. Devices typically store images in memory in a video file format (e.g., MP4). Devices can further transcode the original video file format (e.g., MP4) into one or more other video file formats. The one or more other video file formats may be different video file formats (e.g., MOV, WMV, FLV, AVI, etc.) or the same video file format with different attributes (e.g., MP4 with different bit rates, frame rates, resolutions, or combinations thereof). Transcoding video files consumes processing power and generates heat. Brief explanation of the drawing

[0004] The features of the various disclosed embodiments will be easily understood from the following detailed description, which refers to the accompanying drawings. Reference numbers are used for each element throughout the detailed description and the various drawings. Where multiple similar elements exist, a single reference number may be assigned to the similar elements by adding a lowercase letter to designate specific elements.

[0005] The various elements depicted in the drawings are not drawn to actual scale unless otherwise indicated. The dimensions of the various elements may be enlarged or reduced for clarity. The drawings illustrate one or more implementations and are presented merely as examples and should not be interpreted as limitations. The drawings include the following:

[0006] FIG. 1a is a side view (right) of an exemplary hardware configuration of an eyewear device suitable for use in a power mode configuration system.

[0007] FIG. 1b is a cross-sectional view of the right corner of the eyewear device of FIG. 1a, depicting the right visible light camera and the circuit board.

[0008] FIG. 1c is a side view (left) of an exemplary hardware configuration of the eyewear device of FIG. 1a, illustrating a left visible light camera.

[0009] FIG. 1d is a cross-sectional view of the left corner of the eyewear device of FIG. 1c, depicting the left visible light camera and the circuit board.

[0010] FIGS. 2a and 2b are rear views of exemplary hardware configurations of an eyewear device used in an extended reality generation system.

[0011] Figure 3 is a schematic depiction of a three-dimensional scene, a left raw image captured by a left visible light camera, and a right raw image captured by a right visible light camera.

[0012] FIG. 4a is a functional block diagram of an exemplary power mode configuration generation system including a server system connected to an electronic device through various networks and an electronic device (e.g., a wearable device) implementing an extended reality generation system.

[0013] Figure 4b is a functional block diagram illustrating the transcoding service process flow.

[0014] Figure 5 is a schematic diagram of an exemplary hardware configuration for a mobile device of the extended reality generation system of Figure 4a.

[0015] Figure 6 is a schematic example of a user in an exemplary environment to be used to explain simultaneous localization and mapping in an augmented reality generation system.

[0016] FIG. 7 is a flowchart listing the steps of an exemplary method for determining the position of an eyewear device in an environment using multiple sensors.

[0017] FIGS. 8a and 8b are flowcharts listing the steps of exemplary methods for dynamic power configuration of wearable electronic devices for thermal management.

[0018] Figures 9a and 9b are graphs illustrating the system for net chip power and the system for net chip energy for transcoding, respectively, for the transcoding operation rate in a wearable electronic device. Specific details for implementing the invention

[0019] Various implementations and details are described with reference to examples including dynamic transcode throttling methods for processing resource management and thermal mitigation in electronic devices such as eyewear devices. The electronic device monitors its temperature and configures the transcoding service to operate at different rates in response to the temperature (e.g., to prevent overheating). A frame delay module is configured to add a delay between read frames prior to the transcoding service. This allows the electronic device to consume less power when temperatures are too high to provide thermal mitigation, and can be performed without powering down the electronic device.

[0020] The following detailed description includes systems, methods, techniques, instruction sequences, and computing machine program products that exemplify the examples presented in this disclosure. Numerous details and examples are included to provide a complete understanding of the disclosed claims and their related teachings. However, those skilled in the art can understand how to apply the related teachings without these details. The aspects of the disclosed claims are not limited to the specific devices, systems, and methods described, as the related teachings may be applied or practiced in various ways. The terms and nomenclature used herein are for the purpose of describing specific aspects only and are not intended to be limiting. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily illustrated in detail.

[0021] As used herein, the terms “coupled” or “connected” may refer to any logical, optical, physical, or electrical connection, including links that allow electrical or magnetic signals generated or supplied by one system element to be transmitted to another coupled or connected element. Unless otherwise described, coupled or connected elements or devices are not necessarily directly connected to one another and may be separated by intermediate components, elements, or communication media, one or more of which may modify, manipulate, or transmit electrical signals. The term “on” means directly supported by an element or indirectly supported by an element through another element that is integrated into or supported by the element.

[0022] The orientations of the device incorporating the camera and / or inertial measurement unit as depicted in any of the drawings, the associated components, and any other finished devices are given merely as examples for the purposes of illustration and discussion. In operation, the eyewear device may be oriented in any other direction suitable for the specific application of the eyewear device, e.g., upward, downward, sideways, or any other orientation. Additionally, to the extent used herein, any directional terms such as forward, backward, inward, outward, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, lateral, horizontal, vertical, and diagonal are used merely by example and are not limited to the orientation or orientation of any motion tracker or component of the motion tracker configured as otherwise described herein.

[0023] In one example, an eyewear device provides extended reality services. The eyewear device comprises a frame configured to be worn on a user's head, a camera system supported by the frame—the camera system includes a first camera and a second camera—a display system supported by the frame, a temperature sensor supported by the frame—the temperature sensor is configured to detect the temperature of the eyewear device—and an electronic device supported by the frame, wherein the electronic device is coupled to the camera system, the display system and the temperature sensor, and the electronic device has at least two power configuration modes and includes a processing system. The processing system executes application services of the eyewear device—at least one of the application services is configured to use at least one of the camera system or the display system—; monitors the temperature of the temperature sensor; compares the monitored temperature with a threshold temperature; and notifies at least one application service of the next change from the first power configuration mode among at least two power configuration modes to the second power configuration mode among at least two power configuration modes in response to the monitored temperature reaching the threshold temperature; And after notifying at least one application service, the electronic device is configured to change from a first power configuration mode to a second power configuration mode.

[0024] In another example, a method for use with an eyewear device that provides extended reality services comprises: executing application services on an electronic device of the eyewear device — at least one application service is configured to use at least one of a camera system or a display system —; monitoring the temperature of the eyewear device; comparing the monitored temperature with a threshold temperature; notifying at least one application service of a subsequent change from a first power configuration mode among at least two power configuration modes for the electronic device to a second power configuration mode among at least two power configuration modes for the electronic device in response to the monitored temperature reaching the threshold temperature; and changing the electronic device from the first power configuration mode to the second power configuration mode after notifying the application services.

[0025] In another example, a non-transient computer-readable medium includes instructions for configuring an eyewear device that provides extended reality services. The instructions are configured to perform the following functions when executed by the processing system of the eyewear device: a function to execute application services on the electronic device of the eyewear device—at least one application service is configured to use at least one of a camera system or a display system—; a function to monitor the temperature of the eyewear device; a function to compare the monitored temperature with a threshold temperature; a function to notify at least one application service of the next change from a first power configuration mode of at least two power configuration modes of the electronic device to a second power configuration mode of at least two power configuration modes of the electronic device in response to the monitored temperature reaching the threshold temperature; and a function to change the electronic device from the first power configuration mode to the second power configuration mode after notifying the application services.

[0026] Additional purposes, advantages, and novel features of the examples will be presented in part in the following description and will become apparent to those skilled in the art upon review of the following and the attached drawings, or can be learned through the production or operation of the examples. The purposes and advantages of the claimed invention may be realized and achieved, in particular, by the methodologies, apparatuses, and combinations set forth in the appended claims.

[0027] Now, the examples illustrated in the attached drawings and discussed below are referenced in detail.

[0028] FIG. 1a is a side view (right) of an exemplary hardware configuration of an eyewear device (100) comprising a touch-sensing input device or a touchpad (181). As illustrated, the touchpad (181) may have boundaries that are difficult to perceive and are not easily visible; alternatively, the boundaries may be clearly visible or may include raised or otherwise tactile edges that provide the user with feedback regarding the location and boundaries of the touchpad (181). In other embodiments, the eyewear device (100) may include a touchpad on the left side.

[0029] The surface of the touchpad (181) is configured to detect finger touches, taps, and gestures (e.g., move touches) for use with the GUI displayed on the eyewear device on the image display, so that the user can navigate menu options and select menu options in an intuitive manner, which enhances and simplifies the user experience.

[0030] Detection of finger inputs on the touchpad (181) can enable various functions. For example, when any location on the touchpad (181) is touched, the GUI can display or highlight an item on an image display, which can be projected onto at least one of the optical assemblies (180A, 180B). Double-tapping on the touchpad (181) can select an item or icon. Sliding or swiping the finger in a specific direction (e.g., from front to back, from back to front, from top to bottom, or from bottom to top) causes items or icons to slide or scroll in a specific direction; for example, to move to the next item, icon, video, image, page, or slide. Sliding the finger in a different direction causes sliding or scrolling in the opposite direction; for example, to move to the previous item, icon, video, image, page, or slide. The touchpad (181) can be virtually any location on the eyewear device (100).

[0031] In one example, an identified finger gesture of a single tap on the touchpad (181) initiates the selection or pressing of a graphic user interface element in an image presented on the image display of the optical assembly (180A, 180B). Adjustment of the image presented on the image display of the optical assembly (180A, 180B) based on the identified finger gesture may be a primary action of selecting or submitting a graphic user interface element on the image display of the optical assembly (180A, 180B) for additional display or execution.

[0032] As described herein, the eyewear device (100) includes a right visible light camera (114B). As further described herein, the two cameras (114A, 114B) capture image information of the screen from two separate viewpoints. The two captured images can be used to project a three-dimensional display onto an image display for viewing with 3D glasses.

[0033] The eyewear device (100) includes a right optical assembly (180B) having an image display for presenting images such as depth images. As illustrated in FIGS. 1a and 1b, the eyewear device (100) includes a right visible light camera (114B). The eyewear device (100) may include a plurality of visible light cameras (114A, 114B) forming a passive type three-dimensional camera such as a stereo camera, among which the right visible light camera (114B) is located on the right corner (110B). As illustrated in FIGS. 1c and 1d, the eyewear device (100) also includes a left visible light camera (114A).

[0034] The left and right visible light cameras (114A, 114B) are sensitive to wavelengths in the visible light range. Each of the visible light cameras (114A, 114B) has a different forward-facing field of view that overlaps to enable the generation of three-dimensional depth images; for example, the right visible light camera (114B) illustrates the right field of view (111B). Generally, a "field of view" is a part of a scene visible through a camera at a specific location and orientation in space. The fields of view (111A and 111B) have an overlapping field of view (304) (Fig. 3). When the visible light cameras capture an image, objects or object features outside the field of view (111A, 111B) are not recorded in the raw image (e.g., a photograph or picture). A field of view describes the angular range or degree to which the image sensors of the visible light cameras (114A, 114B) pick up electromagnetic radiation of a given scene within the captured image of that scene. The field of view can be expressed as the angular size of the view cone, that is, the field of view. The field of view can be measured horizontally, vertically, or diagonally.

[0035] In one example, visible light cameras (114A, 114B) have a field of view with a field of view of 40° to 110°, e.g., approximately 100°, and have a resolution of 480 x 480 pixels or more. “Coverage angle” describes the range of angles that the lens of the visible light cameras (114A, 114B) or the infrared camera (410) (see FIG. 2a) can effectively image. Typically, the camera lens creates an image circle large enough to completely cover the film or the camera sensor, possibly including some vignetting (e.g., darkening of the image toward the edges relative to the center). If the coverage angle of the camera lens does not fill the sensor, the image circle will be visible and typically have strong vignetting toward the edges, and the effective angle of view will be limited by the coverage angle.

[0036] Examples of these visible light cameras (114A, 114B) include high-resolution CMOS (complementary metal-oxide-semiconductor) image sensors and digital VGA cameras (video graphics arrays) that enable resolutions of 640p (e.g., 640 x 480 pixels in the case of a total of 0.3 megapixels), 720p, or 1080p. The cameras (114A, 114B) may be rolling shutter cameras in which lines of the sensor array are exposed sequentially, or global shutter cameras in which all lines of the sensor array are exposed simultaneously. Other examples of visible light cameras (114A, 114B) may be used, which, for example, can capture HD (high-definition) still images and store them at a resolution of 1642 x 1642 pixels (or more); Alternatively, high-definition (HD) video can be recorded at a high frame rate (e.g., 30 to 60 frames per second or more) and the recording can be saved at a resolution of 1216 x 1216 pixels (or more).

[0037] The eyewear device (100) can capture image sensor data from visible light cameras (114A, 114B) along with geolocation data, which is digitized by an image processor for storage in memory. The visible light cameras (114A, 114B) capture individual left and right raw images in a two-dimensional spatial domain comprising a matrix of pixels on a two-dimensional coordinate system including an X-axis for horizontal position and a Y-axis for vertical position. Each pixel includes a color attribute value (e.g., red pixel light value, green pixel light value, or blue pixel light value); and a position attribute (e.g., X-axis coordinate and Y-axis coordinate).

[0038] To capture stereo images for subsequent display as a three-dimensional projection, an image processor (412) (shown in FIG. 4a) may be coupled to visible light cameras (114A, 114B) to receive and store visual image information. The image processor (412) or another processor may control the operation of the visible light cameras (114A, 114B) to act as stereo cameras simulating human binocular vision and add timestamps to each image. Timestamps on each pair of images allow for the simultaneous display of images as part of a three-dimensional projection. Three-dimensional projections create desirable immersive and realistic experiences in various extended reality contexts.

[0039] FIG. 1b is a perspective cross-sectional view of the right corner (110B) of the eyewear device (100) of FIG. 1a, illustrating the right visible light camera (114B) and circuit board (140B) of the camera system. FIG. 1c is a side view (left) of an exemplary hardware configuration of the eyewear device (100) of FIG. 1a, illustrating the left visible light camera (114A) of the camera system. FIG. 1d is a perspective cross-sectional view of the left corner (110A) of the eyewear device of FIG. 1c, illustrating the left visible light camera (114A) and circuit board (140A) of the three-dimensional camera. The configuration and arrangement of the left visible light camera (114A) are substantially similar to the right visible light camera (114B), except that the connections and couplings are located on the left lateral side (170A).

[0040] As illustrated in the example of FIG. 1b, the eyewear device (100) includes a right visible light camera (114B) and a circuit board (140B), which may be a flexible printed circuit board (PCB). A right hinge (126B) connects the right corner (110B) to the right temple (125B) of the eyewear device (100). In some examples, components of the right visible light camera (114B), the flexible PCB (140B), or other electrical connectors or contacts may be located on the right temple (125B) or the right hinge (126B).

[0041] The right corner (110B) includes a corner body (190) and a corner cap, the corner cap may be omitted in the cross-section of FIG. 1b. Inside the right corner (110B), various interconnect circuit boards, such as PCBs or flexible PCBs, are arranged, including controller circuits for the right visible light camera (114B), microphone(s), low-power wireless circuits (e.g., for wireless short-range network communication via Bluetooth™), and high-speed wireless circuits (e.g., for wireless local area network communication via Wi-Fi).

[0042] The right visible light camera (114B) is coupled to or placed on a flexible PCB (140B) and is covered by a visible light camera cover lens aimed through an opening(s) formed in the frame (105). For example, the right rim (107B) of the frame (105) shown in FIG. 2a is connected to the right corner (110B) and includes an opening(s) for the visible light camera cover lens. The frame (105) includes a front side configured to face outward away from the user's eyes. An opening for the visible light camera cover lens is formed on and through the front or outward side of the frame (105). In the example, the right visible light camera (114B) has an outward field of view (111B) (shown in FIG. 3) having a line of sight or vision corresponding to the right eye of the user of the eyewear device (100). The visible light camera cover lens may also be attached to the front side or outward surface of the right corner (110B), where an opening is formed that has an outward angle of coverage but faces a different outward direction. The coupling may also be indirect through interposed components.

[0043] As illustrated in FIG. 1b, the flexible PCB (140B) is placed inside the right corner (110B) and coupled to one or more other components housed in the right corner (110B). Although shown as being formed on the circuit boards of the right corner (110B), the right visible light camera (114B) may be formed on the circuit boards of the left corner (110A), the temples (125A, 125B), or the frame (105).

[0044] As illustrated in the example of FIG. 1d, the eyewear device (100) includes a left visible light camera (114A) and a circuit board (140A), which may be a flexible printed circuit board (PCB). A left hinge (126A) connects the left corner (110A) to the left temple (125A) of the eyewear device (100). In some examples, components of the left visible light camera (114A), the flexible PCB (140A), or other electrical connectors or contacts may be located on the left temple (125A) or the left hinge (126A).

[0045] The left corner (110A) includes a corner body (190) and a corner cap, the corner cap being omitted in the cross-section of FIG. 1d. Inside the left corner (110A), various interconnect circuit boards, such as PCBs or flexible PCBs, are arranged, including controller circuits for the left visible light camera (114A), microphone(s), low-power wireless circuits (e.g., for wireless short-range network communication via Bluetooth™), and high-speed wireless circuits (e.g., for wireless local area network communication via Wi-Fi).

[0046] The left visible light camera (114A) is coupled to or placed on a flexible PCB (140A) and is covered by a visible light camera cover lens aimed through an opening(s) formed in the frame (105). For example, the left rim (107A) of the frame (105) shown in FIG. 2a is connected to the left corner (110A) and includes an opening(s) for the visible light camera cover lens. The frame (105) includes a front side configured to face outward away from the user's eyes. An opening for the visible light camera cover lens is formed on and through the front or outward side of the frame (105). In the example, the left visible light camera (114A) has an outward field of view (111A) (shown in FIG. 3) having a line of sight or vision corresponding to the user's left eye of the eyewear device (100). The visible light camera cover lens may also be attached to the front side or outward surface of the left corner (110A), where the opening is formed in a different outward direction but has an outward angle of coverage. The coupling may also be indirect through interposed components.

[0047] As illustrated in FIG. 1d, the flexible PCB (140A) is placed inside the left corner (110A) and coupled to one or more other components housed in the left corner (110A). Although shown as being formed on the circuit boards of the left corner (110A), the left visible light camera (114A) may be formed on the circuit boards of the right corner (110B), the temples (125A, 125B), or the frame (105).

[0048] FIGS. 2a and 2b are rear perspective views of exemplary hardware configurations of an eyewear device (100) including two different types of image displays. The eyewear device (100) is shaped in such a form and has a size configured to be worn by a user; the form of glasses is illustrated in this example. The eyewear device (100) may take other forms and may include other types of frameworks, such as headgear, a headset, or a helmet.

[0049] In the example of eyeglasses, the eyewear device (100) comprises a frame (105) including a left rim (107A) connected to a right rim (107B) via a bridge (106) adapted to be supported by the user's nose. The left and right rims (107A, 107B) include individual apertures (175A, 175B) that hold individual optical elements (180A, 180B), such as lenses and display devices. As used herein, the term lens means comprising transparent or translucent pieces of glass or plastic having curved or flat surfaces that cause light to converge / diverge or to have little or no convergence or divergence.

[0050] Although illustrated as having two optical elements (180A, 180B), the eyewear device (100) may include other arrangements, such as a single optical element, depending on the application or intended user of the eyewear device (100) (or may not include any optical elements (180A, 180B)). As additionally illustrated, the eyewear device (100) includes a left corner (110A) adjacent to the left lateral side (170A) of the frame (105) and a right corner (110B) adjacent to the right lateral side (170B) of the frame (105). The corners (110A, 110B) may be implemented as separate components attached to the frame (105) on individual sides (170A, 170B) (as illustrated). Alternatively, the corners (110A, 110B) can be incorporated into temples (not shown) attached to the frame (105).

[0051] In one example, the image display of the optical assembly (180A, 180B) includes an integrated image display. As illustrated in FIG. 2a, each optical assembly (180A, 180B) includes a suitable display matrix (177), such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or any other such display. Each optical assembly (180A, 180B) also includes an optical layer or layers (176) that may include any combination of lenses, optical coatings, prisms, mirrors, waveguides, optical strips, and other optical components. The optical layers (176A, 176B, … 176N) (illustrated in FIG. 2a and herein as 176A-N) may include a prism having a suitable size and configuration and comprising a first surface for receiving light from the display matrix and a second surface for emitting light to the user's eye. The prisms of the optical layers (176A-N) extend across all or at least a portion of the individual apertures (175A, 175B) formed on the left and right rims (107A, 107B), allowing the user to see a second surface of the prism when the user's eyes are looking at the corresponding left and right rims (107A, 107B). The first surface of the prisms of the optical layers (176A-N) faces upward from the frame (105), and the display matrix (177) is placed over the prism so that photons and light emitted by the display matrix (177) collide with the first surface. The prisms are shaped with a size such that light is refracted within the prisms and directed toward the user's eyes by the second surface of the prisms of the optical layers (176A-N). In this regard, the second surface of the prisms of the optical layers (176A-N) may be convex to direct light toward the center of the eye.The prism can be optionally sized and shaped to enlarge an image projected by the display matrix (177), and light travels through the prism so that the image seen from the second surface is larger in one or more dimensions than the image emitted from the display matrix (177).

[0052] In one example, the optical layers (176A-N) may include an LCD layer that is transparent (keeping the lens open) unless a voltage is applied that makes the layer opaque (closing or blocking the lens) and until a voltage is applied. An image processor (412) on the eyewear device (100) may execute programming to apply a voltage to the LCD layer to create an active shutter system, which makes the eyewear device (100) suitable for viewing visual content when displayed as a three-dimensional projection. LCDs and other technologies may be used during the active shutter mode, including other types of response layers that respond to voltage or other types of input.

[0053] In another example, the image display device of the optical assembly (180A, 180B) includes a projected image display as shown in FIG. 2b. Each optical assembly (180A, 180B) includes a laser projector (150), which is a three-color laser projector using a scanning mirror or a galvanometer. During operation, a light source such as the laser projector (150) is placed on or above one of the temples (125A, 125B) of the eyewear device (100). In this example, the optical assembly (180B) includes one or more optical strips (155A, 155B, … 155N) (shown as 155A-N in FIG. 2b) spaced apart from the width of the lens of each optical assembly (180A, 180B) and spaced across it or across the depth of the lens between the front surface and the rear surface of the lens.

[0054] As photons projected by the laser projector (150) travel across the lenses of each optical assembly (180A, 180B), the photons encounter optical strips (155A-N). When a specific photon encounters a specific optical strip, that photon is redirected toward the user's eye or passed to the next optical strip. A combination of modulation of the laser projector (150) and modulation of the optical strips can control specific photons or light beams. In one example, a processor controls the optical strips (155A-N) by initiating mechanical, acoustic, or electromagnetic signals. Although illustrated as having two optical assemblies (180A, 180B), the eyewear device (100) may include other arrangements such as a single or three optical assemblies, or each optical assembly (180A, 180B) may have a different arrangement depending on the intended user or application of the eyewear device (100).

[0055] As additionally illustrated in FIGS. 2a and 2b, the eyewear device (100) includes a left corner (110A) adjacent to the left lateral side (170A) of the frame (105) and a right corner (110B) adjacent to the right lateral side (170B) of the frame (105). The corners (110A, 110B) may be incorporated into the frame (105) on individual sides (170A, 170B) (as illustrated) or implemented as separate components attached to the frame (105) on individual sides (170A, 170B). Alternatively, the corners (110A, 110B) may be incorporated into temples (125A, 125B) attached to the frame (105).

[0056] In another example, the eyewear device (100) illustrated in FIG. 2b may include two projectors, namely a left projector (not shown) and a right projector (150). The left optical assembly (180A) may include a left display matrix (not shown) or a left set of optical strips (not shown) configured to interact with light from the left projector. Similarly, the right optical assembly (180B) may include a right display matrix (not shown) or a right set of optical strips (155A, 155B, … 155N) configured to interact with light from the right projector (150). In this example, the eyewear device (100) includes a left display and a right display.

[0057] FIG. 3 is a schematic diagram of a three-dimensional scene (306), a left raw image (302A) captured by a left visible light camera (114A), and a right raw image (302B) captured by a right visible light camera (114B). The left field of view (111A) may overlap with the right field of view (111B) as illustrated. The overlapping field of view (304) represents the corresponding portion of the image captured by both cameras (114A, 114B). When referring to the field of view, the term “overlapping” means that the matrix of pixels of the generated raw images overlaps by 30 percent (30%) or more. “Significantly overlapping” means that the matrix of pixels of the generated raw images (or infrared images of the scene) overlaps by 50 percent (50%) or more. As described herein, the two raw images (302A, 302B) can be processed to include timestamps, which allows the images to be displayed together as part of a three-dimensional projection.

[0058] For the capture of stereo images, as illustrated in FIG. 3, a pair of raw red, green, and blue (RGB) images, namely a left raw image (302A) captured by the left camera (114A) and a right raw image (302B) captured by the right camera (114B), is captured from the actual scene (306) at a given moment. When the pair of raw images (302A, 302B) is processed by an image processor (e.g., 412; FIG. 4a), depth images are generated. The generated depth images may be displayed on an optical assembly (180A, 180B) of an eyewear device, on an image display (580) on another display (e.g., a mobile device (401); FIG. 5), or on a screen.

[0059] The generated depth images are in a three-dimensional spatial domain and may include a matrix of vertices on a three-dimensional location coordinate system that includes an X-axis for horizontal position (e.g., length), a Y-axis for vertical position (e.g., height), and a Z-axis for depth (e.g., distance). Each vertex may include color attributes (e.g., red pixel light value, green pixel light value, and / or blue pixel light value); position attributes (e.g., X location coordinate and Y location coordinate, and Z location coordinate); texture attributes; reflection attributes; or a combination thereof. Texture attributes quantify the perceived texture of the depth image, such as spatial arrangements or intensities of color in regions of the vertices of the depth image.

[0060] In one example, an interactive augmented reality system (400) (Fig. 4a) comprises an eyewear device (100), the eyewear device (100) comprises a frame (105), a left temple (110A) extending from the left lateral side (170A) of the frame (105), and a right temple (125B) extending from the right lateral side (170B) of the frame (105). The eyewear device (100) may further comprise at least two visible light cameras (114A, 114B) having an overlapping field of view. In one example, the eyewear device (100) comprises a left visible light camera (114A) having a left field of view (111A) as illustrated in Fig. 3. The left camera (114A) is connected to the frame (105) or the left temple (110A) to capture a left raw image (302A) from the left side of the scene (306). The eyewear device (100) further includes a right visible light camera (114B) having a right field of view (111B). The right camera (114B) is connected to a frame (105) or a right temple (125B) to capture a right raw image (302B) from the right side of the scene (306).

[0061] FIG. 4a is a functional block diagram of an exemplary interactive augmented reality system (400) having a dynamic power configuration including a wearable device (e.g., eyewear device (100)), a mobile device (401), and a server system (498) connected via various networks (495) such as the Internet. The interactive augmented reality system (400) includes a low-power wireless connection (425) and a high-speed wireless connection (437) between the eyewear device (100) and the mobile device (401).

[0062] As illustrated in FIG. 4a, the eyewear device (100) includes a camera system (402) having one or more visible light cameras (114A, 114B) and an image processor (412). As described herein, one or more visible light cameras (114A, 114B) capture still images, video images, or both still images and video images. The cameras (114A, 114B) may have direct memory access (DMA) to a high-speed circuit (430) and may function as stereo cameras. The cameras (114A, 114B) may be used to capture initial-depth images that can be rendered into three-dimensional (3D) models, which are texture mapping images of red, green, and blue (RGB) image scenes. The device (100) may also include a depth sensor (213) that uses infrared signals to estimate the positions of objects relative to the device (100). In some examples, the depth sensor (213) includes one or more infrared emitters (215) and infrared cameras (410).

[0063] The eyewear device (100) further comprises a display system (404) having two image displays of each optical assembly (180A, 180B) (one associated with the left side (170A) and one associated with the right side (170B)) and an image display driver (442). The eyewear device (100) also comprises electronic devices (406), such as a low-power circuit (420) and a high-speed circuit (430). The image displays of each optical assembly (180A, 180B) are intended to present images including still images, video images, or still and video images. The image display driver (442) is coupled to the image displays of each optical assembly (180A, 180B) to control the display of images.

[0064] The eyewear device (100) further comprises one or more speakers (440) (e.g., one associated with the left side of the eyewear device and the other with the right side of the eyewear device) and a temperature sensor (408). In one example, the temperature sensor (408) comprises one or more thermistors positioned within the eyewear device (100) adjacent to an electronic device (406). The temperature sensor may obtain a single temperature from a single thermistor or total temperatures from multiple thermistors. The speakers (440) may be integrated into the frame (105), temples (125), or corners (110) of the eyewear device (100). One or more speakers (440) are driven by an audio processor (443) under the control of a low-power circuit (420), a high-speed circuit (430), or both. The speakers (440) are intended to provide audio signals, such as bit tracks. An audio processor (443) is coupled to speakers (440) to control the presentation of sound.

[0065] For the eyewear device (100), the components illustrated in FIG. 4a are located on one or more circuit boards, such as a printed circuit board (PCB) or a flexible printed circuit (PCB), which are located on the rims or temples. Alternatively or additionally, the illustrated components may be located on the corners, frames, hinges, or bridges of the eyewear device (100). The left and right visible light cameras (114A, 114B) may include any other individual visible or light capture elements that can be used to capture data, including digital camera elements, such as a CMOS (complementary metal-oxide-semiconductor) image sensor, a charge-coupled device, a lens, or still images or videos of scenes having unknown objects.

[0066] As illustrated in FIG. 4a, the high-speed circuit (430) includes a high-speed processor (432), memory (434), and a high-speed wireless circuit (436). In the example, an image display driver (442) is coupled to the high-speed circuit (430) and is operated by the high-speed processor (432) to drive the left and right image displays of each optical assembly (180A, 180B). The high-speed processor (432) may be any processor capable of managing the operation of any general computing system and high-speed communications required for the eyewear device (100). The high-speed processor (432) includes processing resources necessary to manage high-speed data transmissions over a high-speed wireless connection (437) to a wireless local area network (WLAN) using the high-speed wireless circuit (436).

[0067] In some examples, the high-speed processor (432) executes an operating system, such as a LINUX operating system, or other such operating system of the eyewear device (100), and the operating system is stored in memory (434) for execution. In addition to any other responsibilities, the high-speed processor (432) executes a software architecture for the eyewear device (100) used to manage data transfers to the high-speed wireless circuit (436). In some examples, the high-speed wireless circuit (436) is configured to implement the IEEE (Institute of Electrical and Electronic Engineers) 802.11 communication standards, also referred to herein as Wi-Fi. In other examples, other high-speed communication standards may be implemented by the high-speed wireless circuit (436).

[0068] The low-power circuit (420) includes a low-power processor (422) and a low-power wireless circuit (424). The low-power wireless circuit (424) and the high-speed wireless circuit (436) of the eyewear device (100) may include short-range transceivers (Bluetooth™ or BLE (Bluetooth Low-Energy)) and wireless wide, local, or wide-area network transceivers (e.g., cellular or Wi-Fi). A mobile device (401) including transceivers communicating via a low-power wireless connection (425) and a high-speed wireless connection (437) may be implemented using the details of the architecture of the eyewear device (100), as well as other elements of the network (495).

[0069] The memory (434) comprises any storage device capable of storing various data and applications, including, among other things, camera data generated by the left and right visible light cameras (114A, 114B), infrared camera(s) (410), and image processor (412), and images generated for display by the image display driver (442) on the image displays of each optical assembly (180A, 180B). Although the memory (434) is depicted as being integrated with the high-speed circuit (430), in other examples, the memory (434) may be an independent standalone element of the eyewear device (100). In these specific examples, electrical routing lines may provide a connection from the image processor (412) or the low-power processor (422) to the memory (434) through a chip including the high-speed processor (432). In other examples, the high-speed processor (432) can manage the addressing of the memory (434) so ​​that the low-power processor (422) boots the high-speed processor (432) whenever a read or write operation involving the memory (434) is required.

[0070] As illustrated in FIG. 4a, the high-speed processor (432) of the eyewear device (100) can be coupled to a camera system (visible light cameras (114A, 114B)), an image display driver (442), a user input device (491), and a memory (434). As illustrated in FIG. 5, the CPU (540) of the mobile device (401) can be coupled to a camera system (570), a mobile display driver (582), a user input layer (591), and a memory (540A).

[0071] The server system (498) may be one or more computing devices as part of a service or network computing system, for example, including a processor, memory, and a network communication interface for communicating with the eyewear device (100) and the mobile device (401) through the network (495).

[0072] The output components of the eyewear device (100) include visual elements such as left and right image displays (e.g., displays such as a liquid crystal display, a plasma display panel, a light emitting diode display, a projector, or a waveguide) associated with each lens or optical assembly (180A, 180B) as described in FIGS. 2A and 2B. The eyewear device (100) may include a user-facing indicator (e.g., an LED, a loudspeaker, or a vibration actuator) or an outward signal (e.g., an LED, a loudspeaker). The image displays of each optical assembly (180A, 180B) are driven by an image display driver (442). In some exemplary configurations, the output components of the eyewear device (100) further include additional indicators, such as audible elements (e.g., loudspeakers), tactile components (e.g., actuators such as vibration motors for generating haptic feedback), and other signal generators. For example, the device (100) may include indicators of a user-facing set and signals of an outward set. The indicators of the user-facing set are configured to be seen by or otherwise detected by the user of the device (100). For example, the device (100) may include an LED display positioned for the user to see, one or more speakers positioned to generate sound that the user can hear, or actuators for providing haptic feedback that the user can feel. The signals of the outward set are configured to be seen by or otherwise detected by an observer near the device (100). Similarly, the device (100) may include an LED, a loudspeaker, or an actuator configured and positioned to be detected by an observer.

[0073] The input components of the eyewear device (100) may include alphanumeric input components (e.g., a touch screen or touchpad configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric-configurable elements), point-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing mechanism), haptic input components (e.g., a button switch detecting the location, force, or location and force of touches or touch gestures, a touch screen or touchpad, or other haptic-configurable elements), and audio input components (e.g., a microphone). The mobile device (401) and the server system (498) may include alphanumeric, pointer-based, haptic, audio, and other input components.

[0074] In some examples, the eyewear device (100) includes a set of motion-sensing components referred to as an inertial measurement unit (472). The motion-sensing components may often be micro-electro-mechanical systems (MEMS) having microscopic moving parts small enough to be part of a microchip. In some exemplary configurations, the inertial measurement unit (IMU) (472) includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer detects the linear acceleration of the device (100) about three orthogonal axes (x, y, z) (including acceleration due to gravity). The gyroscope detects the angular velocity of the device (100) about three rotational axes (pitch, roll, yaw). At the same time, the accelerometer and gyroscope may provide position, orientation, and motion data for the device about six axes (x, y, z, pitch, roll, yaw). If present, a magnetometer detects the heading of the device (100) relative to magnetic north. The position of the device (100) may be determined by location sensors, such as a GPS unit (473), one or more transceivers for generating relative position coordinates, altitude sensors or barometers, and other orientation sensors. These positioning system coordinates may also be received from the mobile device (401) via wireless connections (425, 437) via a low-power wireless circuit (424) or a high-speed wireless circuit (436).

[0075] The IMU (472) may include or cooperate with a digital motion processor or programming that collects raw data from components and can calculate a number of useful values ​​for the position, orientation, and motion of the device (100). For example, acceleration data collected from an accelerometer may be integrated to obtain velocity for each axis (x, y, z); and may be integrated again to obtain the position of the device (100) (in linear coordinates (x, y, and z)). Angular velocity data from a gyroscope may be integrated to obtain the position of the device (100) (in spherical coordinates). Programming for calculating these useful values ​​may be stored in memory (434) and executed by a high-speed processor (432) of the eyewear device (100).

[0076] The eyewear device (100) may optionally include additional peripheral sensors, such as biosensors, special sensors, or display elements integrated with the eyewear device (100). For example, peripheral device elements may include output components, motion components, position components, or any I / O components including any other such elements described herein. For example, biosensors may include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), or identifying a person (e.g., voice, retina, facial features, fingerprints, or electrical biosignals, such as identification based on electroencephalogram data).

[0077] The mobile device (401) may be a smartphone, tablet, laptop computer, access point, or any other such device capable of connecting to the eyewear device (100) using both a low-power wireless connection (425) and a high-speed wireless connection (437). The mobile device (401) is connected to a server system (498) and a network (495). The network (495) may include any combination of wired and wireless connections.

[0078] An interactive augmented reality system (400) as illustrated in FIG. 4a includes a computing device, such as a mobile device (401), coupled to an eyewear device (100) via a network. The interactive augmented reality system (400) includes memory for storing commands and a processor for executing commands. Execution of commands of the interactive augmented reality system (400) by the processor (432) configures the eyewear device (100) to cooperate with the mobile device (401). The interactive augmented reality system (400) may utilize memory (434) of the eyewear device (100) or memory elements (540A, 540B, 540C) of the mobile device (401) (Fig. 5). Additionally, the interactive augmented reality system (400) may utilize the processor elements (432, 422) of the eyewear device (100) or the CPU (central processing unit) (540) of the mobile device (401) (Fig. 5). Furthermore, the interactive augmented reality system (400) may additionally utilize the memory and processor elements of the server system (498). In this aspect, the memory and processing functions of the interactive augmented reality system (400) may be shared or distributed across the eyewear device (100), the mobile device (401), and the server system (498).

[0079] Memory (434) includes song files (482) and virtual objects (484). Song files (482) include a tempo (e.g., a beat track) and optionally a series of notes and note values. A note is a symbol representing a specific pitch or other musical sound. Note values ​​include the duration of note playback for the tempo and may include other characteristics, such as the intensity, emphasis, articulation, and phrasing of the sound for other notes. In some implementations, the tempo includes a default value along with a user interface that allows the user to select a specific tempo to use while playing a song. Virtual objects (484) include image data for identifying objects or features in images captured by cameras (114). Objects may be physical features, such as known pictures or physical markers, for use in localizing the eyewear device (100) within an environment.

[0080] The memory (434) further includes a power management service (450), a position tracking service (451), a plane detection service (452), a rendering service (453), an over-render service (454), a display service (455), background service(s) (456), a capture service (457), and a transcoding service (458) for execution by the processor (432). The power management service (450) manages the thermal operation mode of the eyewear device (100) by switching between at least two power configuration modes (e.g., normal operation mode and low power mode). In normal operation mode, all processing resources (e.g., cameras, graphics processing units, displays, processes such as noise reduction, etc.) are available and operate at a relatively high operation rate (e.g., 20 or 30 frames per second). In low-power mode, one or more processing resources (e.g., one camera is enabled and another is disabled) are constrained, or operate at a relatively low operating rate (e.g., 0 to 5 frames per second), or processes such as noise reduction are turned off, or a combination thereof is performed.

[0081] For example, in normal operation mode, when the temperature of the electronic device (100) is below a critical temperature (e.g., 60 degrees Celsius), all services operate in a first operating state (e.g., normal operation state) that provides the best performance. According to this example, in low-power mode, when the temperature of the electronic device reaches the critical temperature, the power management service (450) notifies one or more services (e.g., all services) that they must switch to operating in a second operating state (e.g., adaptive state) where resources are constrained and processing resources are limited while providing acceptable performance. Thermal relaxation is achieved with fewer processing resources, which allows the eyewear device (100) to operate for a longer and cooler period.

[0082] Although the examples described herein relate to services having two operating states, services may have more than two operating states, for example, different operating states are associated with different threshold temperatures and resource constraints (e.g., stepping from both cameras available at 45 or 60 frames per second, to both cameras available at 20 or 30 frames per second, to one camera available at 20 or 30 frames per second). Additionally, different services may implement different operating states at different threshold temperatures under the control of a power management service (451). According to this example, a processor (432) executing the power management service (451) may utilize a look-up table (not shown) maintained in memory (434) to command which service will switch to which state and when to switch.

[0083] In one example, the critical temperature is the internal temperature of the eyewear device (100) near the electronic device (406) corresponding to a known hot spot on the surface of the eyewear device (100). In one example, it is desirable to maintain the surface temperature of the eyewear device (100) at less than 43 degrees Celsius, which corresponds to an internal temperature of, for example, 64 degrees Celsius.

[0084] The position tracking service (451) determines the position of the eyewear device within the environment, and the plane detection service (452) identifies plane surfaces within the environment. The position tracking service (451) and the plane detection service (452) may include one or more trackers. In one example, the trackers may be classified into the following three categories. Use a single camera, such as a natural feature tracker or a face tracker. While using two cameras (stereoscopic) is preferred, it can operate with a single camera (potentially degrading performance, for example, using VIO (visual odometry) for 6-degrees-of-freedom tracking). Two cameras (stereoscopic) are required, for example, for real-time depth extraction and plane detection.

[0085] Multiple position tracking and plane detection services may be activated simultaneously. Power consumption (and consequent heat generation) may be reduced by commanding services to accommodate a power management service (450) that restricts resource availability and switches from a normal operating state to an adaptive state when the temperature reaches a critical temperature (e.g., 64 degrees Celsius), for example, by disabling one of the cameras when two cameras are not required (e.g., preceding categories 1 and 2), reducing the frame rate of the camera(s) (e.g., preceding categories 1, 2 and 3), or performing a combination thereof. When the temperature drops below another critical temperature (e.g., 61 degrees Celsius), the power management service (450) may restore resource availability by enabling both cameras, increasing the frame rate, or performing both. Additionally, the power management service (450) can notify the services that they can be switched to a normal operating state because the resources are no longer restricted.

[0086] The rendering service (453) manages how often new frames are displayed on the GPU (graphical processing unit). In the normal state, a high render frame rate (e.g., 45 or 60 frames per second) ensures animations and transitions appear smooth to the user. In the adaptive state, the render frame rate is reduced (e.g., to 30 frames per second) while thermal pressure is applied to extend the runtime.

[0087] The over-render service (454) implements late-warping to reduce motion-to-photon latency in AR / VR / XR headsets. The over-render edge indicates how many pixels out of the display field of view are rendered to avoid displaying black pixels after late-warping is performed (i.e., to leverage the entire display FOV despite late-warping due to, for example, fast head movements). The more pixels rendered, the more power is consumed. In the normal state, a relatively large area out of the display field of view is rendered. In the adaptive state, the area out of the display field of view is reduced or removed.

[0088] The display service (455) presents images on the displays for the user to view. In the normal state, the amount of current consumed by the display systems is set to a desired level to achieve a desired level of brightness. In the adaptive state, the current is limited (e.g., up to 60% of the normal state current consumption), which is to trade brightness for heat reduction. Additionally, processes such as noise reduction are available in the normal state. However, in the adaptive state, one or more of these processes may be turned off to lower power consumption for heat relief. For example, by turning off the noise reduction processes (e.g., in addition to or separately from reducing the FPS), the quality of the displayed images may be gradually reduced to keep them below the thermal limit.

[0089] Background service(s) (456) run in the background (e.g., uploading or transcoding content), and are therefore typically less critical / time-sensitive and may be deferred until the eyewear (100) returns to normal operation mode. In normal state, background services (456) run when called without restriction. In adaptive state, when the eyewear is under thermal pressure, background services (456) are deferred and run only when necessary.

[0090] The capture service (457) captures (i.e., records) AR experiences. In a normal state, the capture service (457) records AR experiences. In an adaptive state, the capture service (457) does not record AR experiences and first informs the user that the eyewear device (100) must be allowed to cool down.

[0091] The transcoding service (458) transcodes video files from one video format to another video format. The transcoding service transcodes the original video file format (e.g., MP4) into one or more other video file formats. The video file "container" format may or may not be changed. In one example, the one or more other video file formats may be different video file "container" formats (e.g., MP4-MOV, WMV, FLV, AVI, etc.) or the same video file container format having different attributes (e.g., MP4 H.265 standard to MP4 H.264 standard having different bit rates, frame rates, resolutions, or combinations thereof). Under normal conditions, the transcoding service (458) converts the video files at a first frame rate (e.g., 20 or 30 frames per second). In the adaptive state, when the eyewear is subjected to thermal pressure, the transcoding service (458) operates at a second frame rate (e.g., 0 to 5 frames per second).

[0092] Memory (434) further includes a position detection utility (460), a marker registration utility (462), a localization utility (464), a virtual object rendering utility (466), a physics engine (468), and a prediction engine (470) for execution by the processor (432). The position detection utility (460) configures the processor (432) to determine a position (location and orientation) within an environment, for example, using the localization utility (464). The marker registration utility (462) configures the processor (432) to register markers within an environment. The markers may be predefined physical markers that are assigned to a specific location by the processor (432) for the environment in which the eyewear device (100) is operating or for the eyewear itself, or have known locations within the environment. The localization utility (464) configures the processor (432) to acquire localization data to be used to determine the position of the eyewear device (100), virtual objects presented by the eyewear device, or a combination thereof. The location data may be derived from a series of images, an IMU unit (472), a GPS unit (473), or a combination thereof. The virtual object rendering utility (466) configures the processor (432) to render virtual images for display by the image display (180) under the control of the image display driver (442) and the image processor (412). The physics engine (468) configures the processor (432) to apply physical laws, such as gravity and friction, between virtual words, e.g., virtual game pieces. The prediction engine (470) configures the processor (432) to predict the expected movement of an object, such as an eyewear device (100), based on the current heading of the eyewear device (100), input from sensors such as an IMU (472), environmental images, or a combination thereof.

[0093] FIG. 4b depicts an exemplary transcode process flow (444). A frame reader (445B) receives an input file (445A) containing video data from memory (e.g., an MP4 H.264 file from memory (434)) and parses the boundaries of the video data to read frames of the video data (e.g., by identifying known header information). The frame reader (445B) may be implemented as a software module executed by a processor (432).

[0094] The frame delay module (446) optionally introduces a delay to the frames of video data before passing the frames to the video processing engine (447). In one example, the frame delay module (446) passes the frames of video data at a first rate (e.g., 20 or 30 frames per second) while in a normal state and at a reduced rate (e.g., 0 to 5 frames per second) while in an adaptive state. The frame delay module (446) may be implemented by a processor (432) under the control of a power management service (450) that provides the operating rate in this example. By using the frame delay module (446) separately from the video processing engine (447), changes in transcoding and processing can be performed without reconfiguring the codec performing decoding and encoding, which may require a restart (and associated unacceptable delays).

[0095] The video processing engine (447) transcodes video data stored in a first video file format (e.g., MP4 H.264 or MP4 H.265) into video data stored in a second video file format (e.g., MP4 H.265 or MP4 H.264). The video processing engine (447) includes a decoder (447A) that converts each frame of video data from the first video file format to the raw video format, and an encoder (447B) that converts the raw video format to the second video file format, for example, in an encoded bit stream (448A). In one example, a processor (432) or an image processor (412) implements the video processing engine (447) using a working memory (e.g., memory (434)) for transcoding operations. The video processing engine (447) generates an output file (448B) containing video data stored in the second video file format. The output file (448B) can be stored in memory (434). In one example, the decoder (447A) and encoder (447B) are implemented using a conventional coder / decoder (codec).

[0096] FIG. 5 is a high-level functional block diagram of an exemplary mobile device (401). The mobile device (401) includes a flash memory (540A) that stores a program to be executed by a CPU (540) to perform all or a subset of the functions described herein.

[0097] The mobile device (401) may include at least two visible light cameras (first and second visible light cameras having overlapping fields of view) or at least one visible light camera having substantially overlapping fields of view and a depth sensor, a camera (570) and an IMU (572). The flash memory (540A) may further include a plurality of images or videos generated through the camera (570).

[0098] As illustrated, the mobile device (401) includes an image display (580), a mobile display driver (582) for controlling the image display (580), and a display controller (584). In the example of FIG. 5, the image display (580) includes a user input layer (591) (e.g., a touchscreen) that is layered on top of the screen used by the image display (580) or otherwise integrated into the screen.

[0099] Examples of touchscreen-type mobile devices that may be used include smartphones, PDAs (personal digital assistants), tablet computers, laptop computers, or other portable devices (but are not limited thereto). However, the structure and operation of touchscreen-type devices are provided as examples; and the claimed technology described herein is not intended to be limited thereto. Accordingly, for the purposes of this discussion, FIG. 5 provides a block diagram example of an exemplary mobile device (401) having a user interface comprising a touchscreen input layer (591) for receiving input (by touch, multi-touch or gesture, etc., by hand, by stylus or other tool) and an image display (580) for displaying content.

[0100] As illustrated in FIG. 5, the mobile device (401) includes at least one digital transceiver (XCVR) (510) illustrated as WWAN XCVRs for digital wireless communications over a wide area wireless mobile communication network. The mobile device (401) also includes additional digital or analog transceivers, such as short-range transceivers (XCVR) (520) for short-range network communication over NFC, VLC, DECT, ZigBee, Bluetooth™, or Wi-Fi. For example, the short-range XCVRs (520) may take the form of any available bidirectional WLAN (wireless local area network) transceiver of a type compatible with one or more standard communication protocols implemented in wireless local area networks, such as one of the Wi-Fi standards under IEEE 802.11.

[0101] To generate location coordinates for positioning the mobile device (401), the mobile device (401) may include a global positioning system (GPS) receiver. Alternatively or additionally, the mobile device (401) may utilize one or both of the short-range XCVRs (520) and WWAN XCVRs (510) to generate location coordinates for positioning. For example, cellular network, Wi-Fi, or Bluetooth™-based positioning systems can generate very accurate location coordinates, especially when used in combination. These location coordinates may be transmitted to the eyewear device via one or more network connections through the XCVRs (510, 520).

[0102] Transceivers (510, 520) (i.e., network communication interfaces) comply with one or more of the various digital wireless communication standards utilized by modern mobile networks. Examples of WWAN transceivers (510) include, for instance, transceivers configured to operate according to 3GPP (3rd Generation Partnership Project) Type 2 (or 3GPP2) and 3GPP network technologies including LTE, sometimes referred to as “4G”, and CDMA (Code Division Multiple Access) without limitation (but not limited thereto). For instance, transceivers (510, 520) provide bidirectional wireless communication of information including digitized audio signals, still images and video signals, web page information for display as well as web-related inputs, and various types of mobile message communications to and from a mobile device (401).

[0103] The mobile device (401) further includes a microprocessor that functions as a central processing unit (CPU); illustrated as the CPU (540) in FIG. 4a. A processor is a circuit having elements structured and arranged to perform one or more processing functions, namely, typically various data processing functions. Individual logic components may be used, but examples utilize components that form a programmable CPU. For example, the microprocessor includes one or more integrated circuit (IC) chips that integrate electronic elements to perform the functions of the CPU. For example, the CPU (540) may be based on any known or available microprocessor architecture, such as RISC (Reduced Instruction Set Computing), using the ARM architecture as is commonly used in mobile devices and other portable electronic devices today. Of course, other arrangements of processor circuits may be used to form the CPU (540) or processor hardware in smartphones, laptop computers, and tablets.

[0104] The CPU (540) acts as a programmable host controller for the mobile device (401) by configuring the mobile device (401) to perform various operations according to, for example, programming or instructions executable by the CPU (540). For example, these operations may include various general operations of the mobile device as well as operations related to programming applications on the mobile device. The processor may be configured using hardwired logic, but the typical processors of mobile devices are general processing circuits configured by the execution of programming.

[0105] A mobile device (401) includes a memory or storage system for storing programming and data. In an example, the memory system may include flash memory (540A), random access memory (RAM) (540B), and other memory components (540C) as needed. The RAM (540B) serves as a short-term storage unit for instructions and data processed by the CPU (540), for example, as an operational data processing memory. The flash memory (540A) typically provides long-term storage.

[0106] Accordingly, in the example of a mobile device (401), flash memory (540A) is used to store programs or instructions for execution by the CPU (540). Depending on the type of device, the mobile device (401) stores and executes a mobile operating system that enables specific applications to run. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, etc.

[0107] A processor (432) within the eyewear device (100) constructs a map of the environment surrounding the eyewear device (100), determines the location of the eyewear device within the mapped environment, and determines the relative position of the eyewear device with respect to one or more objects in the mapped environment. In one example, the processor (432) constructs the map and determines location and position information using a simultaneous localization and mapping (SLAM) algorithm applied to data received from one or more sensors. In the context of augmented reality, the SLAM algorithm is used to construct and update the map of the environment while simultaneously tracking and updating the location of the device (or user) within the mapped environment. The mathematical solution can be approximated using various statistical methods such as particle filters, Kalman filters, extended Kalman filters, and covariance intersection.

[0108] Sensor data includes images received from one or both of the cameras (114A, 114B), distance(s) received from a laser rangefinder, position information received from a GPS unit (473) or from other sensors that provide data useful for determining position information, or a combination of two or more of these sensor data.

[0109] FIG. 6 illustrates an exemplary environment (600) with elements useful for natural feature tracking (NFT; e.g., tracking applications using SLAM algorithms). A user (602) of an eyewear device (100) is present in an exemplary physical environment (600) (an inner room in FIG. 6). A processor (432) of the eyewear device (100) determines its position relative to one or more objects (604) within the environment (600) using captured images, constructs a map of the environment (600) using a coordinate system (x, y, z) for the environment (600), and determines its position within the coordinate system. Additionally, the processor (432) determines the head pose (roll, pitch, and yaw) of the eyewear device (100) within the environment by using two or more location points associated with a single object (604a) (e.g., three location points (606a, 606b, and 606c)) or by using one or more location points (606) associated with two or more objects (604a, 604b, 604c). In one example, the processor (432) of the eyewear device (100) may position a virtual object (484) (such as the key shown in FIG. 6) within the environment (600) for an extended reality view through image displays (180).

[0110] FIG. 7 is a flowchart (700) illustrating a method for visual-inertial tracking for a wearable device (e.g., an eyewear device). Although the steps are described with reference to the eyewear device (100) as described herein, other implementations of the steps described for other types of devices will be understood by those skilled in the art from the description herein. Additionally, it is considered that one or more of the steps illustrated in FIG. 7 and other figures and described herein may be omitted, performed simultaneously or in series, performed in a different order than the example and description, or performed with additional steps.

[0111] In block (702), the eyewear device (100) captures one or more input images of the physical environment (600) near the eyewear device (100). The processor (432) may continuously receive input images from the visible light camera(s) (114) and store these images in memory (434) for processing. Additionally, the eyewear device (100) may capture information from other sensors (e.g., location information from the GPS unit (473), orientation information from the IMU (472), or distance information from the laser distance sensor).

[0112] In block (704), the eyewear device (100) compares the objects of the captured image with objects stored in a library of images to identify a match. In some implementations, the processor (432) stores the captured images in memory (434). A library of images of known objects is stored in a virtual object database (484).

[0113] In one example, the processor (432) is programmed to identify specific predefined objects (e.g., specific photos (604a) hanging at known locations on a wall, windows (604b) on other walls, or safes (604c) positioned on floors). Other sensor data, such as GPS data (e.g., images associated with rooms identified solely by GPS coordinates), may be used to narrow down the number of known objects for comparison. In another example, the processor (432) is programmed to identify general predefined objects (e.g., one or more trees in a park).

[0114] In block (706), the eyewear device (100) determines its position relative to the object(s). The processor (432) may determine its position relative to the objects by processing the distances between two or more points of the captured images (e.g., between two or more location points on one object (604) or between location points (606) on each of the two objects (604)) by comparing them with the known distances between corresponding points of the identified objects. If the distances between the points of the captured images are greater than the distances between the points of the identified objects, it indicates that the eyewear device (100) is closer to the identified object than the imager that captured the image containing the identified object. On the other hand, if the distances between the points of the captured images are smaller than the distances between the points of the identified objects, it indicates that the eyewear device (100) is farther from the identified object than the imager that captured the image containing the identified object. By processing relative distances, the processor (432) can determine the position of the object(s). Alternatively or additionally, other sensor information, such as laser distance sensor information, may be used to determine the position of the object(s).

[0115] In block (708), the eyewear device (100) constructs a map of the environment (600) surrounding the eyewear device (100) and determines its position within the environment. In one example, if the identified object (block (704)) has a predefined coordinate system (x, y, z), the processor (432) of the eyewear device (100) constructs a map using the predefined coordinate system and determines the position of the eyewear device (100) within that coordinate system based on the positions (block (706)) determined for the identified objects. In another example, the eyewear device constructs a map using images of permanent or semi-permanent objects (604) (e.g., trees or park benches in a park) within the environment. According to this example, the eyewear device (100) can define a coordinate system (x', y', z') used in the environment.

[0116] In block (710), the eyewear device (100) determines the head pose (roll, pitch, and yaw) of the eyewear device (100) within the environment. The processor (432) determines the head pose by using two or more location points (e.g., three location points (606a, 606b, and 606c)) for one or more objects (604) or by using one or more location points (606) for two or more objects (604). Using conventional image processing algorithms, the processor (432) determines the roll, pitch, and yaw by comparing the angles and lengths of lines extending between location points of the captured images and known images.

[0117] In block (712), the eyewear device (100) presents visual images to the user. The processor (432) uses the image processor (412) and the image display driver (442) to present images to the user on image displays (180). The processor develops and presents visual images through the image displays in response to the position of the eyewear device (100) within the environment (600). The processor (432) may store the visual images presented to the user in a first file format, for example, in memory (434), for viewing and further processing (e.g., transcoding into a different file format).

[0118] In block (714), the steps described above with reference to blocks (706-712) are repeated to update the position of the eyewear device (100) and what the user (602) sees as the user moves through the environment (600).

[0119] Referring again to FIG. 6, an exemplary method for implementing the augmented reality applications described herein includes virtual markers (e.g., virtual marker (610a)) associated with physical objects (e.g., FIG. (604a)) and virtual markers associated with virtual objects (e.g., keys (608)). In one example, an eyewear device (100) uses markers associated with physical objects to determine the position of the eyewear device (100) within an environment, and uses markers associated with virtual objects to generate overlay images presenting the associated virtual object(s) (608) of the environment (600) at the virtual marker positions on the display of the eyewear device (100). For example, markers are registered at locations in the environment to be used to track and update the locations of users, devices, and objects (virtual and physical objects) in the mapped environment. Markers are sometimes registered on high-contrast physical objects, such as relatively dark objects (604a) mounted on bright-colored walls, to assist cameras and other sensors in the task of detecting markers. Markers can be pre-assigned or assigned by the eyewear device (100) when entering the environment. Markers are also registered on locations in the environment to be used to present virtual images at the corresponding locations in the mapped environment.

[0120] Markers may be encoded as information or otherwise linked to information. A marker may include position information, a physical code (e.g., a barcode or QR code; visible to or hidden from the user), or a combination thereof. A set of data associated with a marker is stored in the memory (434) of the eyewear device (100). The set of data includes information about the marker (610a), the position (location and orientation) of the marker, and one or more virtual objects or a combination thereof. The marker position may include three-dimensional coordinates for one or more marker landmarks (616a), such as the corner of the generally rectangular marker (610a) shown in FIG. 6. The marker location may be expressed in terms of real-world geographical coordinates, a system of marker coordinates, the position of the eyewear device (100), or other coordinate systems. One or more virtual objects associated with the marker (610a) may include any of the various materials, including still images, video, audio, haptic feedback, executable applications, interactive user interfaces and experiences, and combinations or sequences of these materials. Any type of content that can be stored in memory and retrieved when the marker (610a) encounters or associates with the assigned marker may be classified as a virtual object in this context. The key (608) illustrated in FIG. 6 is a virtual object displayed, for example, as a still image that is 2D or 3D at the marker location.

[0121] In one example, the marker (610a) may be located near a physical object (604a) (e.g., a framed artwork shown in FIG. 6) and registered in memory as associated with it. In another example, the marker may be registered in memory as a specific position for the eyewear device (100).

[0122] FIGS. 8a and 8b are individual flowcharts (800 and 820) listing the steps of exemplary dynamic power reduction methods for thermal mitigation. The steps of flowchart (800) describe the operation of a power management control system (e.g., a power management service (450) implemented by a processor (432)). The steps of flowchart (820) describe the operation of a transcoding service (458) controlled by a power management system. Although the steps are described with reference to an eyewear device (100) as described herein, other implementations of the steps described for other types of mobile devices will be understood by those skilled in the art from the description herein. Additionally, it is considered that one or more of the steps illustrated in FIGS. 8a and 8b and described herein may be omitted, performed simultaneously or in series, performed in a different order than the example and description, or performed with additional steps.

[0123] FIG. 8a is a flowchart (800) illustrating a power management (and thermal mitigation) method. In block (802), the electronic device (406) of the eyewear device (100) executes application services (e.g., services (450-458)). In one example, a processor (432) executes the application services (450-458), and the power management service (450) provides instructions to other services (451-458) to adapt to the operating modes in which the eyewear device (100) is operating based on the temperature of the eyewear device (100) (e.g., the temperature of the electronic device (406)).

[0124] In block (804), the power management service (450) monitors the temperature of the eyewear device (100). In one example, a processor (432) running the power management service (450) receives the temperature read from the temperature sensor (408) periodically (e.g., every 100ms).

[0125] In block (806), the power management service (450) compares the monitored temperature with a threshold. In one example, the processor (432) executing the power management service (450) compares the monitored temperature with a threshold temperature stored in memory (434). Depending on the mode in which the eyewear device (100) is operating, there may be multiple threshold temperatures. For example, when the eyewear device is in normal operating mode, a first threshold (e.g., 64 degrees Celsius) may be used to determine when to switch to low-power mode (e.g., due to overheating). Similarly, when the eyewear device is in low-power operating mode, a second threshold (e.g., 60 degrees Celsius) may be used to determine when to switch back from low-power mode to normal operating mode (e.g., once the temperature is reduced to an acceptable level). According to this example, the eyewear device (100) can switch between a normal operation mode and a low-power operation mode to maximize the amount of time the eyewear device can operate in normal mode without overheating (e.g., to provide the best performance).

[0126] In the decision block (808), the power management service (450) determines whether a threshold has been reached. In one example, the processor (432) executing the power management service (450) determines whether a threshold has been reached (e.g., using a first threshold when the eyewear device (100) is in normal operation mode and using a second threshold when the eyewear device is in low-power operation mode). If the threshold has been reached, processing proceeds in block (810). If the threshold has not been reached, processing proceeds in block (804).

[0127] In block (810), the power management service (450) notifies other application service(s) (451-458) that the power configuration mode of the eyewear device (100) is about to change. In one example, the processor (432) running the power management service (450) transmits a communication to the other application service(s) (451-458) (e.g., via inter-process communication).

[0128] In block (812), the power management service (450) monitors the timer. In one example, the processor (432) running the power management service (450) monitors the timer (e.g., maintained by the processor (432) using a runtime application).

[0129] In the decision block (814), the power management service (450) determines whether the monitored time has reached a predefined waiting time. In one example, the processor (432) executing the power management service (450) compares the monitored time with a predefined waiting time (e.g., 5 seconds). The predefined waiting time is set to provide other application service(s) (451-458): (1) when in a normal state—time to transition from the normal state to the adaptive state because all resources used during the normal state may no longer be available, and (2) when in an adaptive state—time to transition from the adaptive state to the normal state because additional resources will be switched to become available. If the waiting time is reached, processing proceeds in block (816). If the waiting time is not reached, processing proceeds in block (812).

[0130] In block (816), the power management service (450) changes the power configuration mode of the eyewear device. In one example, a processor (432) executing the power management service (450) changes the power configuration mode of the eyewear device (100). When the eyewear device (100) is in a normal operation mode, the processor (432) changes the operation mode to a low-power operation mode (in which available resources are reduced). When the eyewear device (100) is in a low-power operation mode, the processor (432) changes the operation mode to a normal operation mode (i.e., in which available resources are increased).

[0131] FIG. 8b is a flowchart (820) illustrating an exemplary method for switching between operating states of a transcoding service (458) having a first state (normal state) for transcoding frames of video data at a first rate (e.g., 45 or 60 frames per second) and a second state (adapting state) for transcoding frames of video data at a second rate (e.g., 20 or 30 frames per second). Modifications of this exemplary method for use with other application services will be understood by those skilled in the art from the description herein.

[0132] In block (822), the electronic device (406) sets the frame pass rate of the transcoding service (458). In one example, the processor (432) executing the transcoding service (458) sets the frame pass rate to a first operating rate (e.g., 20 or 30 frames per second; normal state) when the electronic device is in a first operating mode, and sets the frame pass rate to a second operating rate (e.g., 0 to 5 frames per second; adaptive state) when the electronic device is in a second operating mode. In the normal state, the transcoding service (458) is set up for relatively high performance designed to utilize the resources of the eyewear device when operating in the normal operating mode. In the adaptive state, the transcoding service (458) is set up for relatively low but acceptable performance designed to reduce the use of the resources of the eyewear device when operating in a low-power operating mode.

[0133] In block (824), the electronic device (406) reads frames of video data in a first video file format. In one example, a processor (432) executing a transcoding service (458) implements a frame reader (445) to read frames of video data.

[0134] In block (826), the electronic device (406) transmits frames of video data to the frame delay module (446). In one example, a processor (432) executing the transcoding service (458) transmits frames of video data to the frame delay module (446).

[0135] In block (828), the electronic device (406) passes frames of video data frame by frame at a set frame operation rate. In one example, a processor (432) executing the transcoding service (458) implements a frame delay module (446) and passes frames of video data frame by frame at a set frame operation rate.

[0136] In block (830), the electronic device (406) receives frames of video data in frames at a set frame pass rate. In one example, a processor (432) executing a transcoding service (458) implements a video processing engine (447) to transcode frames of video data. In block (832), the video processing engine (447) uses a decoder (447A) to decode each frame of video data of a first video file format into a raw video format, and in block (834), uses an encoder (447B) to encode each frame of video data into a second video encoding format.

[0137] In block (836), the electronic device (406) stores frames of encoded video data in different file formats, for example, having different compression settings. In one example, a processor (432) executing a transcoding service (458) stores frames of video data in memory (434) in different file formats.

[0138] Aspects of the examples described herein address thermal mitigation of AR eyewear devices. Thermal mitigation of such devices has conflicting objectives, including (1) ensuring that the devices do not exceed specific thermal limits, (2) maximizing the thermal runtime of AR experiences, (3) ensuring that the devices continue to operate for as long as possible without overheating, and maximizing responsiveness (e.g., the device must not reach "shut-off" thermal limits, and instead, certain high-power features are disabled to avoid excess heat and appear responsive to the user).

[0139] FIG. 9a illustrates power consumption for exemplary frame rates for transcoding in three different scenarios (video data without lens / overlay, video data with alien theme lens / overlay, and video data with basketball theme overlay). As illustrated, as the operating rate decreases, power consumption decreases.

[0140] Figure 9b illustrates energy consumption for exemplary frame rates for transcoding in the same three scenarios as in Figure 9a (video data without lens / overlay, video data with alien theme lens / overlay, and video data with basketball theme overlay). As illustrated, as the frame rate decreases, energy consumption increases.

[0141] As illustrated by FIGS. 9a and 9b, power consumption (Fig. 9a) and energy consumption (Fig. 9b) have an inverse relationship. When processing resources are readily available and the device operates within an acceptable temperature range, a suitable operating rate (e.g., 20 or 30 frames per second) is selected for transcoding to provide the fastest results. On the other hand, when processing resources are limited or the device operates beyond an acceptable temperature range, a relatively low operating rate (e.g., 0 to 5 frames per second) is selected for transcoding to provide acceptable results in terms of processing and thermal constraints. Specific frame rate values ​​may be selected based on the use case and trade-offs between processing resources / thermal mitigation and acceptable performance.

[0142] Any function described herein may be implemented as one or more computer software applications or sets of programming instructions as described herein. According to some examples, “function,” “functions,” “application,” “applications,” “instruction,” “instructions,” or “programming” is a program(s) that execute functions defined in programs. Various programming languages ​​may be used to create one or more of applications configured in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In certain examples, a third-party application (e.g., an application developed by an entity other than the vendor of a particular platform using the ANDROID™ or IOS™ SDK (software development kit)) may include mobile software running on a mobile operating system, such as IOS™, ANDROID™, WINDOWS® Phone, or other mobile operating systems. In such an example, a third-party application may invoke API calls provided by the operating system to facilitate the functions described herein.

[0143] Accordingly, machine-readable media can take many forms of tangible storage media. Non-volatile storage media include optical or magnetic disks, such as any of the storage devices in any computer devices, such as those that can be used to implement client devices, media gateways, transcoders, etc., as illustrated in the drawings. Volatile storage media include dynamic memory, such as the main memory of such computer platforms. Tangible transmission media include coaxial cables, copper wires, and optical fibers, including wires containing buses within a computer system. Carrier wave transmission media can take the form of electrical or electromagnetic signals, such as those generated during RF (radio frequency) and infrared (IR) data communications, or sound waves or light waves. Accordingly, general forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punch cards, paper tapes, any other physical storage media having patterns of holes, RAM, PROM and EPROM, FLASH-EPROM, any other memory chips or cartridges, carriers carrying data or instructions, cables or links carrying such carriers, or any other media in which a computer can read programming code and / or data. Most of the various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0144] Except as mentioned immediately above, nothing mentioned or illustrated, whether or not cited in the claims, is intended or should not be interpreted to provide the general public with any component, step, feature, purpose, benefit, advantage, or equivalent.

[0145] It will be understood that the terms and expressions used herein have the general meanings given to them in relation to their corresponding individual areas of inquiry and study, except where specific meanings are otherwise presented herein. Relational terms, such as first, second, etc., may be used merely to distinguish one entity or action from another and do not necessarily require or imply any actual such relationship or order between these entities or actions. The terms “include,” “included,” “comprising,” “comprising,” or any other variation thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising or comprising a list of elements or steps may not only comprise such elements or steps but may also comprise other elements or steps that are not explicitly listed or are unique to such process, method, article, or device. A singular element does not exclude, without additional restrictions, the presence of additional identical elements in a process, method, article, or device comprising the element.

[0146] Unless otherwise stated, any and all measurements, values, grades, positions, sizes, dimensions, and other specifications described herein, including the following claims, are not exact but approximate. These quantities are intended to have a reasonable range consistent with what is customary in the functions to which they relate and in the relevant field. For example, unless otherwise explicitly stated, parameter values, etc., may vary by ± 10% from the stated amount.

[0147] Furthermore, in the foregoing detailed description, various features are grouped together in various examples for the purpose of simplifying the disclosure. The method of the disclosure should not be interpreted as reflecting an intention for the claimed examples to require more features than explicitly stated in each claim. Rather, as reflected in the following claims, the subject matter to be protected is less than all the features of any single disclosed example. Accordingly, the following claims are incorporated into the detailed description, and each claim is described independently as a separately claimed subject matter.

[0148] While the foregoing describes what are considered to be the best mode and other examples, it is understood that various modifications may be made to the subject matter of the claims disclosed herein and that the subject matter disclosed herein may be implemented in various forms and examples and may be applied to numerous applications, and that only a part thereof has been described herein. The following claims are intended to claim any and all modifications and alterations within the true scope of the concepts.

Claims

Claim 1 An eyewear device providing augmented reality services, comprising: a frame configured to be worn on a user's head; a display system supported by the frame; a temperature sensor supported by the frame and configured to detect the temperature of the eyewear device; and an electronic device supported by the frame — the electronic device being coupled to the display system and the temperature sensor, and the electronic device having at least two power configuration modes and including a memory system and a processing system — wherein the processing system executes application services on the electronic device of the eyewear device — at least one of the application services including a transcoding service configured to transcode frames of video data stored in the memory into a first video file format presented by the display system —; monitors the temperature of the temperature sensor; compares the monitored temperature with a threshold temperature; and changes the electronic device from a first power configuration mode to a second power configuration mode in response to the monitored temperature reaching the threshold temperature; In a frame delay module, when the electronic device is in the first power configuration mode, the frame pass rate is set to a first operating rate, and when the electronic device is in the second power configuration mode, the frame pass rate is set to a second operating rate; frames of the video data of the first video file format are read from the memory; the read frames of the video data are transmitted to the frame delay module; frames of the video data are received from the frame delay module at the frame pass rate; and frames of the video data of the first video file format received from the frame delay module at the frame pass rate are transcoded into a second video file format;An eyewear device providing extended reality services configured to save transcoded frames of the above video data in the above second video file format.; Claim 2 An eyewear device providing extended reality services, wherein the processing system comprises: a frame reader configured to read frames of the video data from the memory; a video processing engine configured to transcode frames of the video data; and a frame delay module positioned between the frame reader and the video processing engine, wherein the frame delay module is configured to pass frames of the video data from the frame reader to the video processing engine at the frame pass rate. Claim 3 An eyewear device providing extended reality services, wherein the video processing engine comprises: a decoder configured to decode frames of video data of the first video file format into raw video frames; and an encoder configured to encode the raw video frames into frames of video data of the second video file format. Claim 4 An eyewear device providing extended reality services, wherein at least one of the bit rate, frame rate, or resolution of the first video file format is different from the second video file format. Claim 5 An eyewear device providing extended reality services, wherein the processing system is further configured to notify the transcoding service of the next change from the first power configuration mode to the second power configuration mode in response to the monitored temperature reaching the threshold temperature. Claim 6 An eyewear device providing extended reality services, wherein, in claim 5, the processing system is further configured to wait for a predetermined time period after notifying the transcoding service before changing from the first power configuration mode to the second power configuration mode. Claim 7 An eyewear device providing extended reality services, wherein the transcoding service includes a first operating state (normal state) corresponding to the first operating rate and a second operating state (adapting state) corresponding to the second operating rate, and the transcoding service changes from the first operating state to the second operating state in response to a notification of the next change. Claim 8 An eyewear device that provides extended reality services, wherein the first operation rate is higher than the second operation rate in claim 1. Claim 9 A method for use with an eyewear device that provides augmented reality services, wherein the eyewear device comprises a display system and an electronic device, and the method comprises: executing application services on the electronic device of the eyewear device—at least one of the application services comprises a transcoding service configured to transcode frames of video data stored in memory in a first video file format presented by the display system—; monitoring the temperature of the eyewear device; comparing the monitored temperature with a threshold temperature; changing the electronic device from a first power configuration mode to a second power configuration mode in response to the monitored temperature reaching the threshold temperature; in a frame delay module, setting the frame pass rate to a first operating rate when the electronic device is in the first power configuration mode and setting the frame pass rate to a second operating rate when the electronic device is in the second power configuration mode; reading frames of the video data in the first video file format from the memory; and transmitting the read frames of the video data to the frame delay module. A method comprising: receiving frames of video data from a frame delay module at the frame pass rate; transcoding frames of video data of the first video file format received from the frame delay module at the frame pass rate into a second video file format; and storing the transcoded frames of the video data in the second video file format. Claim 10 In claim 9, the step of reading frames of the video data from the memory is performed by a frame reader, and the step of transcoding the frames of the video data is performed by a video processing engine, and the method further comprises the step of passing the frames of the video data from the frame reader to the video processing engine at the frame pass rate by the frame delay module. Claim 11 A method according to claim 10, wherein the transcoding step comprises: a step of decoding frames of video data of the first video file format into raw video frames; and a step of encoding the raw video frames into frames of video data of the second video file format. Claim 12 In claim 11, a method wherein at least one of the bit rate, frame rate, or resolution of the first video file format is different from that of the second video file format. Claim 13 A method according to claim 9, further comprising the step of notifying the transcoding service of the next change from the first power configuration mode to the second power configuration mode in response to the monitored temperature reaching the critical temperature. Claim 14 A method according to claim 13, further comprising the step of waiting for a predetermined time period after notifying the transcoding service before changing from the first power configuration mode to the second power configuration mode. Claim 15 In claim 14, the transcoding service includes a first operating state (normal state) corresponding to the first operating rate and a second operating state (adapted state) corresponding to the second operating rate, and the transcoding service changes from the first operating state to the second operating state in response to a notification of the next change. Claim 16 In claim 9, the method wherein the first operation rate is higher than the second operation rate. Claim 17 A non-transient computer-readable medium comprising instructions for configuring an eyewear device that provides extended reality services, wherein the eyewear device comprises a display system and an electronic device, the instructions configure the processing system to perform functions when executed by the processing system of the eyewear device, the functions comprising: a function to execute application services on the electronic device of the eyewear device—at least one of the application services comprising a transcoding service configured to transcode frames of video data stored in memory in a first video file format presented by the display system—; a function to monitor the temperature of the eyewear device; a function to compare the monitored temperature with a threshold temperature; a function to change the electronic device from a first power configuration mode to a second power configuration mode in response to the monitored temperature reaching the threshold temperature; and, in a frame delay module, a function to set the frame pass rate to a first operating rate when the electronic device is in the first power configuration mode and to set the frame pass rate to a second operating rate when the electronic device is in the second power configuration mode. A non-transient computer-readable medium comprising: a function for reading frames of video data of the first video file format from the memory; a function for transmitting the read frames of the video data to the frame delay module; a function for receiving frames of the video data from the frame delay module at the frame pass rate; a function for transcoding the frames of the video data of the first video file format received from the frame delay module at the frame pass rate into a second video file format; and a function for storing the transcoded frames of the video data in the second video file format. Claim 18 A non-transient computer-readable medium according to claim 17, wherein the function of reading frames of the video data from the memory is performed by a frame reader, the function of transcoding the frames of the video data is performed by a video processing engine, and the instructions further configure the processing system to perform functions including the function of passing the frames of the video data from the frame reader to the video processing engine at the frame pass rate by the frame delay module. Claim 19 A non-transient computer-readable medium according to claim 18, wherein the transcoding function comprises: a function of decoding frames of video data of the first video file format into raw video frames; and a function of encoding the raw video frames into frames of video data of the second video file format. Claim 20 In claim 19, a non-transient computer-readable medium in which at least one of the bit rate, frame rate, or resolution of the first video file format is different from the second video file format.

Images