Multimodal input for computer-generated reality

Abstract

Specific implementations of the subject technology provide for determining an operating mode of an electronic device based at least in part on whether the electronic device is communicatively coupled to an associated base device. Based on the determined operating mode, the subject technology identifies a set of input modalities for initiating recording of content within the field of view of the electronic device. The subject technology monitors sensor information generated by at least one sensor included in the electronic device or communicatively coupled to the electronic device. In addition, when the monitored sensor information indicates that at least one input modality in the identified set of input modalities has been triggered, the subject technology initiates recording of content within the field of view of the electronic device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 897,909, filed on September 9, 2019, entitled “Multimodal Inputs for Computer-Generated Reality,” the disclosure of which is incorporated herein in its entirety. Technical Field

[0003] The present specification generally relates to computer-generated reality environments, including utilizing multimodal input in computer-generated reality environments. Background Art

[0004] Augmented reality technology aims to bridge the gap between virtual and physical environments by providing an augmented physical environment augmented with electronic information. Therefore, the electronic information appears to be part of the physical environment perceived by the user. In one example, augmented reality technology further provides a user interface to interact with the electronic information overlaid in the augmented physical environment. Summary of the invention

[0005] According to one aspect of the present disclosure, a method is provided, comprising: determining an operating mode of an electronic device based at least in part on whether the electronic device is communicatively connected to an associated base device; based on the determined operating mode, identifying a set of input modalities for initiating recording of content within a field of view of the electronic device; monitoring sensor information generated by at least one sensor included in the electronic device or communicatively connected to the electronic device; and initiating recording of content within the field of view of the electronic device when the monitored sensor information indicates that at least one input modality in the identified set of input modalities has been triggered.

[0006] According to another aspect of the present disclosure, a system is provided, comprising: a processor; a memory device, the memory device containing instructions, which, when executed by the processor, causes the processor to perform operations including the following: determining an operating mode of the electronic device based at least in part on whether the electronic device is communicatively connected to an associated base device; based on the determined operating mode, identifying a set of input modalities for initiating recording of content within a field of view of the electronic device; monitoring sensor information generated by at least one sensor included in the electronic device or communicatively connected to the electronic device; and initiating recording of content within the field of view of the electronic device when the monitored sensor information indicates that at least one input modality in the identified set of input modalities has been triggered.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable medium is provided, comprising instructions, which, when executed by a computing device, cause the computing device to perform operations comprising: determining an operating mode of the electronic device based at least in part on whether the electronic device is communicatively connected to an associated base device; identifying a set of input modalities for initiating recording of content within a field of view of the electronic device based on the determined operating mode; monitoring sensor information generated by at least one sensor included in the electronic device or communicatively connected to the electronic device; initiating recording of content within the field of view of the electronic device when the monitored sensor information indicates that at least one input modality in the identified set of input modalities has been triggered; detecting that the operating mode of the electronic device has changed; and in response to detecting the change, updating the set of input modalities for initiating recording of content based on the changed operating mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Certain features of the subject technology are set forth in the appended claims. However, for purposes of explanation, several embodiments of the subject technology are set forth in the following drawings.

[0009] Figure 1 An exemplary system architecture including various electronic devices that may implement the subject system is shown according to one or more implementations.

[0010] Figure 2 An exemplary software architecture that may be implemented on an electronic device in accordance with one or more implementations of the subject technology is shown.

[0011] Figure 3A An example of facial expression tracking to initiate a computer-generated reality recording in accordance with an implementation of the subject technology is shown.

[0012] Figure 3B and Figure 3C An example of tracking gaze direction to initiate computer-generated reality recording in accordance with an implementation of the subject technology is shown.

[0013] Figure 4A , Figure 4B and Figure 4C Examples of determining an area of ​​interest within a computer-generated reality environment and initiating recording based on the area of ​​interest are shown in accordance with some implementations of the subject technology.

[0014] Figure 5A , Figure 5B and Figure 5C Examples of providing annotations to various objects or entities within a computer-generated reality environment in accordance with some implementations of the subject technology are shown.

[0015] Figure 6 A flow chart illustrating an example process for initiating recording of content within a field of view of an electronic device is shown.

[0016] Figure 7 A flow chart of an exemplary process for updating a record of a set of input modalities for initiating content on an electronic device is shown in accordance with one or more implementations.

[0017] Figure 8 A flow chart of an exemplary process for determining a quality of service metric associated with an operating mode of an electronic device in accordance with one or more implementations is shown.

[0018] Fig. 9 An electronic system is shown that may implement one or more implementations of the subject technology. DETAILED DESCRIPTION

[0019] The specific embodiments shown below are intended to be descriptions of various configurations of the subject technology and are not intended to represent the only configuration that the subject technology can be practiced. The accompanying drawings are incorporated herein and constitute a part of the specific embodiments. The specific embodiments include specific details intended to provide a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein, but can be practiced using one or more other specific implementations. In one or more specific implementations, structures and components are shown in block diagram form to avoid blurring the concept of the subject technology.

[0020] Computer generated reality (CGR) systems enable physical environments and virtual environments to be combined to varying degrees to facilitate interaction with a user in a real-time manner. Thus, as described herein, such CGR systems may include various possible combinations of physical environments and virtual environments, including augmented reality, which primarily includes physical elements and is closer to a physical environment than a virtual environment (e.g., without physical elements). In this way, a physical environment may be connected to a virtual environment through a CGR system. A user immersed in a CGR environment may navigate in such an environment, and the CGR system may track the user's viewpoint to provide visualization based on how the user is located in the environment.

[0021] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0022] In contrast, a computer generated reality (CGR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic systems. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one law of physics. For example, a CGR system may detect body and / or head turns of a person, and in response, adjust graphical content and sound fields presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0023] A person may sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides a perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0024] Examples of CGR include virtual reality and mixed reality.

[0025] A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0026] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or its representation, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is any environment between, but not including, a fully physical environment at one end and a virtual reality environment at the other.

[0027] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause movement so that virtual trees appear stationary relative to the physical ground.

[0028] An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a portion of the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment, and perceive virtual objects superimposed on a portion of the physical environment and / or behind a portion of the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video", meaning that the system uses one or more image sensors to capture images of the physical environment, and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0029] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) portions thereof so that the modified portions may be representative but not true versions of the original captured images. For another example, a representation of a physical environment may be transformed by graphically eliminating portions thereof or blurring portions thereof.

[0030] An augmented virtual (AV) environment is a simulated environment in which a virtual or computer-generated environment combines one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0031] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include mobile devices, tablet devices, projection-based systems, head-up displays (HUDs), head-mounted systems, vehicle windshields integrated with display capabilities, windows integrated with display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets or tablet devices, and desktop / laptop computers. For example, a head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). The head-mounted system can be combined with one or more imaging sensors for capturing images or videos of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, silicon-based liquid crystal, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0032] Specific implementations of the subject technology described herein provide a CGR system that can use different input modalities to achieve multimodality for recording content within a CGR environment. Examples of different input modalities include facial expressions, gestures, voice, and / or explicit hardware input, each of which can work alone and / or in combination with one or more of the other input modalities. Therefore, the input modalities described herein can work in a complementary manner. In addition, the subject technology enables selection of regions of interest in a CGR environment and provides annotations of objects and / or events detected in the CGR environment.

[0033] Figure 1 An exemplary system architecture 100 including various electronic devices that may implement the subject system according to one or more implementations is shown. However, not all of the depicted components may be used in all implementations, and one or more implementations may include additional or different components than those shown in the figures. Changes in the arrangement and types of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0034] The system architecture 100 includes an electronic device 105, a handheld electronic device 104, an electronic device 110, an electronic device 115, and a server 120. For the purpose of explanation, the system architecture 100 is shown in FIG. Figure 1 1. System architecture 100 is shown as including electronic device 105, handheld electronic device 104, electronic device 110, electronic device 115, and server 120; however, system architecture 100 may include any number of electronic devices and any number of servers or a data center including multiple servers.

[0035] The electronic device 105 can be implemented, for example, as a tablet device, a handheld and / or mobile device, or as a head-mounted portable system (e.g., worn by the user 101). The electronic device 105 includes a display system capable of presenting a visualization of a computer-generated reality environment to the user. The electronic device 105 can be powered by a battery and / or another power source. In one example, the display system of the electronic device 105 provides a stereoscopic presentation of the computer-generated reality environment to the user, enabling a three-dimensional visual display of a particular scene rendering to be provided. In one or more specific implementations, instead of or in addition to using the electronic device 105 to access the computer-generated reality environment, the user can use a handheld electronic device 104, such as a tablet computer, a watch, a mobile device, etc.

[0036] The electronic device 105 may include one or more cameras, such as a camera 150 (e.g., a visible light camera, an infrared camera, etc.). In addition, the electronic device 105 may include various sensors 152, including but not limited to cameras, image sensors, touch sensors, microphones, inertial measurement units (IMUs), heart rate sensors, temperature sensors, depth sensors (e.g., lidar sensors, radar sensors, sonar sensors, time-of-flight sensors, etc.), GPS sensors, Wi-Fi sensors, near-field communication sensors, wireless radio frequency sensors, etc. In addition, the electronic device 105 may include hardware elements that may receive user input, such as hardware buttons or switches. The user input detected by such sensors and / or hardware elements corresponds to various input modalities, such as for initiating a co-occurrence session within an application. For example, such input modalities may include but are not limited to facial tracking, eye tracking (e.g., gaze direction), hand tracking, gesture tracking, biometric readings (e.g., heart rate, pulse, pupil dilation, breathing, temperature, electroencephalogram, smell), recognition of voice or audio (e.g., specific hot words), and activation of buttons or switches, etc.

[0037] In one or more specific implementations, the electronic device 105 may be communicatively coupled to a base device, such as the electronic device 110 and / or the electronic device 115. Generally speaking, such base devices may include more computing resources and / or available power than the electronic device 105. In one example, the electronic device 105 may operate in various modes. For example, the electronic device 105 may operate in a standalone mode independent of any base device. When the electronic device 105 operates in a standalone mode, the number of input modalities may be constrained by the power and / or processing limitations of the electronic device 105 (such as the available battery power of the device). In response to power limitations, the electronic device 105 may deactivate certain sensors within the device itself to preserve battery power and / or release processing limitations.

[0038] The electronic device 105 may also operate in a wireless connection mode (e.g., connected to a base device via a wireless connection) to work in conjunction with a given base device. The electronic device 105 may also operate in a connection mode in which the electronic device 105 is physically connected to the base device (e.g., via a cable or some other physical connector) and may utilize power resources provided by the base device (e.g., where the base device charges the electronic device 105 when physically connected).

[0039] When the electronic device 105 is operated in a wireless connection mode or a connection mode, at least a portion of processing user input and / or rendering a computer-generated reality environment can be offloaded to the base device, thereby reducing the processing burden on the electronic device 105. For example, in one specific implementation, the electronic device 105 works in conjunction with the electronic device 110 or the electronic device 115 to generate a computer-generated reality environment, which includes physical objects and / or virtual objects that implement different forms of interaction (e.g., visual, auditory and / or physical or tactile interaction) between the user and the computer-generated reality environment in real time. In one example, the electronic device 105 provides a rendering of a scene corresponding to a computer-generated reality environment that can be perceived by the user and interacted with in real time, such as a host environment for a co-existence session with another user. In addition, as part of presenting the rendered scene, the electronic device 105 may provide sound and / or tactile or haptic feedback to the user. The content of a given rendered scene may depend on available processing power, network availability and capacity, available battery power, and current system workload.

[0040] The electronic device 105 may also detect an event that has occurred within a scene of the computer-generated reality environment. Examples of such events include detecting the presence of a particular person, entity, or object in the scene. In response to the detected event, the electronic device 105 may provide an annotation (e.g., in the form of metadata) corresponding to the detected event in the computer-generated reality environment.

[0041] The network 106 may communicatively couple (directly or indirectly) electronic device 104, electronic device 105, electronic device 110, and / or electronic device 115, for example, with each other device and / or server 120. In one or more specific implementations, the network 106 may be an interconnected network that may include the Internet or devices communicatively coupled to the Internet.

[0042] The electronic device 110 may include a touch screen and may be, for example, a smart phone including a touch screen, a portable computing device such as a laptop computer including a touch screen, a companion device including a touch screen (e.g., a digital camera, headphones), a tablet device including a touch screen, a wearable device including a touch screen (such as a watch, wristband, etc.), any other suitable device including, for example, a touch screen, or any electronic device with a touchpad. In one or more specific implementations, the electronic device 110 may not include a touch screen, but may support touch screen-like gestures, such as in a computer-generated reality environment. In one or more specific implementations, the electronic device 110 may include a touchpad. In Figure 1In the embodiment, by way of example, the electronic device 110 is depicted as a mobile smart phone device having a touch screen. In one or more specific implementations, the electronic device 110, the handheld electronic device 104 and / or the electronic device 105 may be and / or may include the following relative to the following relative to Fig. 9 The electronic system discussed herein refers to all or part of the electronic device discussed herein. In one or more implementations, the electronic device 110 may be another device, such as an Internet Protocol (IP) camera, a tablet computer, or a companion device such as an electronic stylus.

[0043] The electronic device 115 may be, for example, a desktop computer, a portable computing device such as a laptop computer, a smart phone, a companion device (e.g., a digital camera, a headset), a tablet device, a wearable device such as a watch, a wristband, etc. Figure 1 In the embodiment, the electronic device 115 is depicted as a desktop computer by way of example. The electronic device 115 may be and / or may include the following with respect to Fig. 9 All or part of the electronic system in question.

[0044] The server 120 may form all or part of a computer network or server group 130, such as in a cloud computing or data center implementation. For example, the server 120 stores data and software and includes specific hardware (e.g., processors, graphics processors and other dedicated or custom processors) for rendering and generating content of a computer-generated reality environment, such as graphics, images, video, audio, and multimedia files. In one specific implementation, the server 120 may be used as a cloud storage server that stores any of the aforementioned computer-generated reality content generated by the above-mentioned devices and / or server 120.

[0045] Figure 2 An exemplary software architecture 200 that may be implemented on an electronic device 105 according to one or more specific implementations is shown. For purposes of illustration, the software architecture 200 is described as being composed of Figure 1 The software architecture 200 may be implemented by any other electronic device, including the electronic device 115 and / or the electronic device 120. However, not all of the components depicted may be used in all implementations, and one or more implementations may include additional or different components than those shown in the figures. Changes in the arrangement and type of these components may be made without departing from the spirit or scope of the claims set forth herein. Additional components, different components, or fewer components may be provided.

[0046] The software architecture 200 as implemented on the electronic device 105 includes a framework. As used herein, a framework may refer to a software environment that provides specific functionality to facilitate the development of software applications as part of a larger software platform, and may provide one or more application programming interfaces (APIs) that developers may utilize to programmatically design computer-generated reality environments and process operations for such computer-generated reality environments.

[0047] As shown, a recording framework 230 is provided. The recording framework 230 can provide functionality for recording a computer-generated reality environment provided by the input modalities as discussed above. An event detector 220 is provided that receives information corresponding to inputs from various input modalities. A system manager 210 is provided to monitor resources from the electronic device 105 and determine a quality of service metric based on the available resources. The system manager 210 can make decisions to select specific hardware components corresponding to the corresponding input modalities to activate and / or deactivate based on the quality of service metric, such as to free up processing resources, save power resources, etc. For example, a camera used to track facial expressions can be turned off, or another camera used to track gestures can be turned off.

[0048] In one or more specific implementations, when specific hardware is deactivated, the electronic device 105 can provide a notification to warn the user that the specific input modality is unavailable. Similarly, the electronic device 105 can provide a notification to warn the user that the specific input modality is available when the specific hardware is activated.

[0049] Figure 3A An example of facial expression tracking to initiate a computer-generated reality recording according to a specific implementation of the subject technology is shown. The following discussion relates to components of the electronic device 105, which includes various cameras or image sensors to enable facial tracking of a user's face.

[0050] In a specific implementation, the electronic device 105 can utilize various sensors to track the facial expressions of the user 301 using the electronic device 105. As shown, different areas within the user's face can be tracked by the sensors of the electronic device 105. For example, a camera can track the movement of the right eyebrow 310 and the left eyebrow 312 of the user 301. Another camera can track the movement of the area 302 including the right eye and the area 304 including the left eye. Different cameras can track the movement of the first area 308 (e.g., including the vertex and nostrils of the nose) and / or the second area 306 (e.g., including the dorsum and / or bridge of the nose). Yet another camera can track the mouth 316 of the user 301. In addition, a specific camera can track the jaw 314 including the chin of the user 301.

[0051] Although various cameras are discussed above, it should be understood that the same camera can track more than one portion of a user's face and still be within the scope of the subject technology. For example, the same camera can be used to track the user's jaw 314 and the mouth 316 of the user 301.

[0052] The information from the various cameras may be analyzed independently or used in combination by event detector 220. Event detector 220 may use the information to detect a facial expression on the user's face. In response to the detected facial expression, event detector 220 may send a request to recording framework 230 to initiate a recording within the computer-generated reality environment, e.g., a recording of a particular region of interest or field of view.

[0053] Different types of facial expressions may correspond to various emotions of the user 301. In an example, the electronic device 105 determines that the detected facial expression corresponds to a particular emotion (e.g., surprise, anger, happiness) and initiates recording of the computer-generated reality environment in response to the emotion based on the detected facial expression.

[0054] Figure 3B and Figure 3C An example of tracking gaze direction to initiate computer-generated reality recording according to a specific implementation of the subject technology is shown. The following discussion relates to components of an electronic device 105 that includes various cameras to enable tracking of the gaze direction of the eyes of a user's face.

[0055] like Figure 3B As shown, the image captured by at least one camera of the electronic device 105 is analyzed to determine the relative position of the user's eyes within the field of view. In a specific implementation, the electronic device 105 can distinguish the user's pupils, and can use the relative position of the pupils relative to the eye position to determine the gaze direction. For example, in Figure 3C In the embodiment of the present invention, the electronic device 105 can use the detected position of the user's pupil relative to the user's eyes and determine the area on the display of the electronic device 105 that the user is looking at within the field of view 320. In addition, in a specific implementation, the electronic device 105 can also detect movement such as the user closing his or her eyes for a specific period of time, which can be used to initiate recording within the computer-generated reality environment.

[0056] Event detector 220 may analyze the above information to determine the gaze direction. Event detector 220 may use this information to determine the gaze direction of the user's eyes. In response to the determined gaze direction, event detector 220 may send a request to recording framework 230 to initiate recording within the computer-generated reality environment. For example, in response to determining that the user's gaze direction is in a specific direction or towards a specific object or person in the current scene of the computer-generated reality environment, event detector 220 may send such a request to recording framework 230 to initiate recording.

[0057] Figure 4A , Figure 4B and Figure 4C An example of determining an area of ​​interest within a computer-generated reality environment and initiating recording based on the area of ​​interest is shown in accordance with some implementations of the subject technology. The following discussion relates to components of the electronic device 105 .

[0058] In a specific implementation, one or more input modalities may be used to identify an area of ​​interest that is the focus of a recording within a computer-generated reality environment. For example, a user may perform a gesture or some other interaction (e.g., pressing a button or switch on the electronic device 105, providing a hot word or voice) to identify an area of ​​interest. It should also be understood that a user may utilize one or more input modes in combination to identify an area of ​​interest and / or initiate a recording. In addition, as described above, a recording may be initiated when an event occurring within a scene (such as the presence of a person in the scene) is detected.

[0059] like Figure 4A As shown, scene 410 shows a sporting event (e.g., ice hockey) taking place in a computer-generated reality environment. In this example, by using a particular input modality, a user has selected an area of ​​interest 404 corresponding to a puck in the current scene of the computer-generated reality environment. In scene 410, the puck moves toward a person 402 corresponding to a first hockey player. Event detector 220 has detected the presence of a particular person 406 (e.g., a star hockey player), and in response, initiates recording of the computer-generated reality environment by sending a request to recording framework 230.

[0060] exist Figure 4B4, scene 420 shows that particular person 406 has moved to a different location than the initial location in scene 410. The recording frame 230 continues to record the computer-generated reality environment and focuses on the area of ​​interest 404 as the puck moves closer to the stick of person 402 in scene 420. Thus, the area of ​​interest 404 may be moving or in motion, and the recording moves or tracks the area of ​​interest. In a specific implementation, the recording frame 230 records the entire scene 420, notwithstanding the area of ​​interest 404. During future playback of the recording, the presentation of the recording may focus on the area of ​​interest 404 corresponding to the puck.

[0061] exist Figure 4C 4, scene 430 shows a particular person 406 moving to a different location than in scene 420 and completely within the view frame of scene 430. The recording frame 230 continues to record the computer generated reality environment and focuses on the area of ​​interest 404 as the puck moves across the ice rink in scene 430.

[0062] Figure 5A , Figure 5B and Figure 5C Examples of providing annotations to various objects or entities within a computer-generated reality environment according to some implementations of the subject technology are shown. The following discussion relates to components of the electronic device 105 .

[0063] exist Figure 5A In , scene 502 is rendered to the user and includes various objects or entities within a computer generated reality environment. Figure 5B, event detector 220 detects the presence of person 504, animal 508, and vehicle 506. In the example, detection of the object occurs when the recorded video stream is passed to event detector 220. Event detector 220 forwards information corresponding to detected person 504, animal 508, and vehicle 506 to recording framework 230. Based on the received information, recording framework 230 generates annotation 512 corresponding to person 504. Recording framework 230 also generates annotation 514 corresponding to vehicle 506. In addition, recording framework 230 generates annotation 516 corresponding to animal 508. Alternatively, event detector 220 can generate the above annotations, and recording framework 230 can store the annotations as metadata with coordinates (and / or other information) corresponding to the annotations. The above annotations can be stored as metadata associated with the recording of the scene content, such as, in the example, adding metadata to be included as part of the recording of the content (e.g., a modified version of the content recording that now includes metadata). In the example, the object is identified and recognized as a person, animal, etc., and the metadata is stored in association with the identified object. In a specific implementation, such metadata corresponding to the annotations may be stored in a memory of the electronic device 105, and / or included in the computer-generated reality recording, and / or stored separately in a different electronic device (e.g., a server or base device). It should also be understood that different sets of annotations may be applied to a given computer-generated reality recording, thereby enabling various uses of different annotations in conjunction with playback of the recording.

[0064] exist Figure 5C , the electronic device 105 renders an update to the scene 502, which now displays annotations 512 corresponding to a person 504, annotations 514 corresponding to a vehicle 506, and annotations 516 corresponding to an animal 508. As shown, the annotations are rendered as part of the scene 502, which may include elements corresponding to a physical environment mixed with digitally generated content (e.g., annotations). In some implementations, such annotations may be provided in different formats or not displayed in the scene. For example, the annotations may be provided to the user in audio form, serving as a narration of the computer-generated reality environment that the user is currently experiencing.

[0065] Figure 6 1 shows a flow chart of an example process 600 for initiating recording of content within a field of view of an electronic device 105 according to one or more implementations. Figure 1 and Figure 2 The process 600 is described with reference to the electronic device 105. However, the process 600 is not limited to Figure 1 and Figure 2The electronic device 105 in the embodiment of the present invention, and one or more blocks (or operations) of process 600 may be performed by one or more other components of other suitable devices. Further for the purpose of explanation, the blocks of process 600 are described herein as occurring sequentially or linearly. However, multiple blocks of process 600 may occur in parallel. In addition, the blocks of process 600 do not have to be performed in the order shown, and / or one or more blocks of process 600 do not have to be performed and / or may be replaced by other operations.

[0066] like Figure 6 As shown, the electronic device 105 determines an operating mode based at least in part on whether the electronic device 105 is communicatively coupled to an associated base device (610). Based on the determined operating mode, the electronic device 105 identifies a set of input modalities for initiating recording of content within the field of view of the electronic device 105 (612). The electronic device 105 monitors sensor information generated by at least one sensor included in the electronic device 105 or communicatively coupled to the electronic device 105 (614). When the monitored sensor information indicates that at least one input modality in the identified set of input modalities has been triggered, the electronic device 105 initiates recording of content within the field of view of the electronic device 105 (616).

[0067] Figure 7 A flow chart of an exemplary process 700 for updating a set of input modalities for initiating content on an electronic device 105 according to one or more implementations is shown. Figure 1 and Figure 2 The process 700 is described with reference to the electronic device 105. However, the process 700 is not limited to Figure 1 and Figure 2 The electronic device 105 in the embodiment of the present invention, and one or more blocks (or operations) of process 700 may be performed by one or more other components of other suitable devices. Further for the purpose of explanation, the blocks of process 700 are described herein as occurring sequentially or linearly. However, multiple blocks of process 700 may occur in parallel. In addition, the blocks of process 700 do not have to be performed in the order shown, and / or one or more blocks of process 700 do not have to be performed and / or may be replaced by other operations.

[0068] like Figure 7 As shown, the electronic device 105 detects that the operating mode of the electronic device 105 has changed (710). In response to detecting the change, the electronic device 105 updates a set of input modalities for initiating recording of content based on the changed operating mode (712).

[0069] Figure 8A flowchart 800 is shown of an exemplary process for determining a quality of service metric associated with an operating mode of an electronic device 105 in accordance with one or more implementations.

[0070] like Figure 8 As shown, the electronic device 105 determines a quality of service metric associated with an operating mode of the electronic device 105 (810). Based at least in part on the quality of service metric, the electronic device 105 selects at least one input modality (812). The electronic device 105 provides the at least one input modality as the set of input modalities for initiating recording of content within the field of view of the electronic device 105 (814).

[0071] As described above, one aspect of the present technology is to collect and use data from various sources. The present disclosure contemplates that, in some instances, these collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, social network identifiers, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0072] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit the user. The present disclosure also contemplates uses of personal information data that benefit the user. For example, health and fitness data can be used to provide insights into the user's overall health, or can be used as positive feedback to individuals using technology to pursue health goals.

[0073] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate uses. In addition, such collection / sharing should be performed after receiving informed consent from the user. In addition, such entities should consider taking any necessary steps to defend and secure access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including specific considerations of jurisdiction. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained in each country for different types of personal data.

[0074] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates providing hardware elements and / or software elements to prevent or block access to such personal information data. For example, the present technology may be configured to allow users to “opt in” or “opt out” of participating in the collection of personal information data at any time during or after registration for a service. In addition to providing “opt in” and “opt out” options, the present disclosure contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then be reminded again just before the personal information data is accessed by the application.

[0075] In addition, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how data is stored (e.g., aggregating data between users), and / or other methods when appropriate.

[0076] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user based on non-personal information data or a small amount of personal information, such as content requested by a device associated with the user, other non-personal information, or publicly available information.

[0077] Fig. 9 An electronic system 900 is shown that can be utilized to implement one or more implementations of the subject technology. The electronic system 900 may be Figure 1 The electronic device 105, the electronic device 104, the electronic device 110, the electronic device 115 and / or the server 120 shown may be and / or may be a part thereof. The electronic system 900 may include various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 900 includes a bus 908, one or more processing units 912, a system memory 904 (and / or cache), a ROM 910, a permanent storage device 902, an input device interface 914, an output device interface 906, and one or more network interfaces 916, or a subset and variant thereof.

[0078] The bus 908 generally represents all system buses, peripheral busses, and chipset buses that communicatively connect the many internal devices of the electronic system 900. In one or more specific implementations, the bus 908 communicatively connects one or more processing units 912 with the ROM 910, the system memory 904, and the permanent storage device 902. The one or more processing units 912 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes disclosed in the present subject matter. In different specific implementations, the one or more processing units 912 can be a single processor or a multi-core processor.

[0079] ROM 910 stores static data and instructions required by one or more processing units 912 and other modules of the electronic system 900. On the other hand, permanent storage device 902 can be a read-write memory device. Permanent storage device 902 can be a non-volatile memory unit that stores instructions and data even when the electronic system 900 is turned off. In one or more specific implementations, a mass storage device (such as a magnetic disk or optical disk and its corresponding disk drive) can be used as permanent storage device 902.

[0080] In one or more specific implementations, a removable storage device (such as a floppy disk, a flash drive and its corresponding disk drive) can be used as the permanent storage device 902. Like the permanent storage device 902, the system memory 904 can be a read-write memory device. However, unlike the permanent storage device 902, the system memory 904 can be a volatile read-write memory, such as a random access memory. The system memory 904 can store any of the instructions and data that one or more processing units 912 may need at runtime. In one or more specific implementations, the processes disclosed in the subject matter are stored in the system memory 904, the permanent storage device 902, and / or the ROM 910. The one or more processing units 912 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes of one or more specific implementations.

[0081] The bus 908 is also connected to an input device interface 914 and an output device interface 906. The input device interface 914 enables a user to transmit information and select commands to the electronic system 900. Input devices that can be used with the input device interface 914 may include, for example, an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). The output device interface 906 may, for example, enable the display of images generated by the electronic system 900. Output devices that can be used with the output device interface 906 may include, for example, a printer and a display device, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid-state display, a projector, or any other device for outputting information. One or more specific implementations may include a device that acts as both an input device and an output device, such as a touch screen. In these specific implementations, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input.

[0082] Finally, if Fig. 9 As shown, bus 908 also couples electronic system 900 to one or more networks and / or to one or more network nodes via one or more network interfaces 916, such as Figure 1 . In this manner, electronic system 900 may be part of a computer network, such as a LAN, a wide area network ("WAN"), or an intranet, or may be part of a network of networks, such as the Internet. Any or all components of electronic system 900 may be used with the subject disclosure.

[0083] The above functions can be implemented in computer software, firmware or hardware. The technology can be implemented using one or more computer program products. The programmable processor and computer can be included in a mobile device or packaged as a mobile device. The process and logic flow can be performed by one or more programmable processors and one or more programmable logic circuits. General and special computing devices and storage devices can be interconnected through a communication network.

[0084] Some specific implementations include electronic components such as microprocessors, storage devices, and memories that store computer program instructions in machine-readable or computer-readable media (also referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, compact disk-read only (CD-ROM), compact disk-recordable (CD-R), compact disk-rewritable (CD-RW), digital versatile disk-read only (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini-SD card, micro-SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable A computer readable medium may store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as that produced by a compiler, and files including higher-level code that can be executed by a computer, electronic component, or microprocessor using an interpreter.

[0085] Although the above discussion mainly involves microprocessors or multi-core processors that execute software, some implementations are performed by one or more integrated circuits such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some implementations, such integrated circuits execute instructions stored on the circuits themselves.

[0086] As used in this specification and any claims of this patent application, the terms "computer", "server", "processor" and "memory" refer to electronic or other technical devices. These terms exclude people or groups of people. For the purposes of this specification, the terms display or displaying means displaying on an electronic device. As used in this specification and any claims of this patent application, the terms "computer-readable medium" and "computer-readable media" are entirely limited to tangible, tangible objects that store information in a form that can be read by a computer. These terms do not include any wireless signals, wired download signals, and any other transient signals.

[0087] To provide interaction with a user, implementations of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser in response to a request received from the web browser on the user's client device.

[0088] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface or a web browser, through which a user can interact with a specific implementation of the subject matter described in this specification, or includes any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), interconnected networks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0089] A computing system may include a client and a server. The client and the server are generally remote from each other and may interact via a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some specific implementations, the server transmits data (e.g., an HTML page) to a client device (e.g., in order to display data to a user interacting with the client device and to receive user input from a user interacting with the client device). Data generated at the client device (e.g., a result of a user interaction) may be received at the server from the client device.

[0090] Those skilled in the art will recognize that various illustrative frames, modules, elements, parts, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative frames, modules, elements, parts, methods and algorithms have been generally described above according to functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The functions can be implemented in different ways for each specific application. Various parts and frames can be arranged differently (e.g., arranged in different orders, or divided in different ways), without departing from the scope of the subject technology.

[0091] It should be understood that the specific order or hierarchy of steps in the process disclosed by the present invention is an illustration of an exemplary method. Based on design preference requirements, it should be understood that the specific order or hierarchy of steps in the process can be rearranged. Some of the steps can be performed simultaneously. The attached method claims present elements of various steps in a sample order and are not meant to be limited to the specific order or hierarchy presented.

[0092] The previous description is provided to enable those skilled in the art to practice the various aspects described herein. The previous description provides various examples of the subject technology, and the subject technology is not limited to these examples. Various modifications of these aspects are obvious to those skilled in the art, and the general principles defined herein can be applied to other aspects. Therefore, the claims are not intended to be limited to the various aspects shown herein, but are intended to make the full scope consistent with the language claims, wherein the reference to the singular element is not intended to mean "only one", but to refer to "one or more", unless specifically indicated. Unless otherwise specifically stated, the term "some" refers to one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are only for convenience and do not limit the present invention described herein.

[0093] As used herein, the term website may include any aspect of a website, including one or more web pages, one or more servers for hosting or storing web-related content, and the like. Thus, the term website may be used interchangeably with the terms web page and server. The predicate words "configured to," "capable of operating to," and "programmed to" do not imply any specific tangible or intangible modification of a subject matter but are intended to be used interchangeably. For example, a component or a processor configured to monitor and control operations may also mean that the processor is programmed to monitor and control operations or that the processor is capable of operating to monitor and control operations. Similarly, a processor configured to execute code may be interpreted as a processor that is programmed to execute code or is capable of operating to execute code.

[0094] As used herein, the term automatically may include execution by a computer or machine without user intervention; for example, by instructions in response to a predicate action of a computer or machine or other initiating mechanism. The word "exemplary" is used herein to mean "serving as an example or illustration." Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs.

[0095] Phrases such as "aspects" do not mean that this aspect is essential to the subject technology or that this aspect applies to all configurations of the subject technology. The disclosure related to an aspect may apply to all configurations, or one or more configurations. An aspect may provide one or more examples. Phrases such as aspects may refer to one or more aspects, and vice versa. Phrases such as "implementation" do not mean that this implementation is essential to the subject technology or that this implementation applies to all configurations of the subject technology. The disclosure related to an implementation may apply to all implementations, or one or more implementations. An implementation may provide one or more examples. Phrases such as "implementation" may refer to one or more implementations, and vice versa. Phrases such as "configuration" do not mean that this configuration is essential to the subject technology or that this configuration applies to all configurations of the subject technology. The disclosure related to a configuration may apply to all configurations or one or more configurations. A configuration may provide one or more examples. Phrases such as "configuration" may refer to one or more configurations, and vice versa.

Claims

1. A method for initiating recording of content, comprising: determining an operating mode of the electronic device based at least in part on whether the electronic device is communicatively coupled to an associated base device; Based on the determined mode of operation, identifying a set of input modalities for initiating recording of content within a field of view of the electronic device; activating or deactivating at least one hardware component corresponding to the identified set of input modalities based on available processing resources and / or power resources corresponding to the determined operating mode; monitoring sensor information generated by at least one sensor included in or communicatively coupled to the electronic device; as well as When the monitored sensor information indicates that at least one input modality of the identified set of input modalities has been triggered, recording of content within the field of view of the electronic device is initiated.

2. The method according to claim 1, further comprising: detecting that the operating mode of the electronic device has changed; as well as In response to detecting the change, the set of input modalities used to initiate the recording of content is updated based on the changed operating mode. 3 . The method of claim 1 , wherein the associated base device corresponds to a mobile computing device, a laptop computer, or a desktop computer.

4. The method of claim 1 , wherein identifying the set of input modalities for initiating the recording of content further comprises: determining a quality of service metric associated with the operating mode of the electronic device; selecting at least one input modality based at least in part on the quality of service metric; as well as The at least one input modality is provided as the set of input modalities for initiating the recording of the content within the field of view of the electronic device. 5 . The method of claim 4 , wherein the quality of service metric is based at least in part on available computing resources or available power in the electronic device, the available power comprising an amount of battery power.

6. The method according to claim 4, further comprising: Deactivate a specific input modality corresponding to at least one sensor in the electronic device, wherein the specific input modality includes facial expression, gaze direction, eye position, gesture, hardware input, voice, or recognition of an object or person in a scene.

7. The method according to claim 4, further comprising: When the quality of service metric is lower than a threshold, a specific sensor in the electronic device is deactivated, wherein the specific sensor comprises a camera, an inertial measurement unit, a microphone, or a touch sensor.

8. The method according to claim 1, further comprising: A region of interest in the field of view of the electronic device is determined. The method of claim 8 , wherein the region of interest is determined based on a gesture or an indicator corresponding to the region of interest.

10. The method according to claim 1, further comprising: generating an annotation corresponding to the record of content; as well as The annotation is added as metadata to the record of the content.

11. A system for initiating recording of content, comprising: processor; a memory device containing instructions that, when executed by the processor, cause the processor to perform operations including: determining an operating mode of the electronic device based at least in part on whether the electronic device is communicatively coupled to an associated base device; Based on the determined mode of operation, identifying a set of input modalities for initiating recording of content within a field of view of the electronic device; activating or deactivating at least one hardware component corresponding to the identified set of input modalities based on available processing resources and / or power resources corresponding to the determined operating mode; monitoring sensor information generated by at least one sensor included in or communicatively coupled to the electronic device; as well as When the monitored sensor information indicates that at least one input modality of the identified set of input modalities has been triggered, recording of content within the field of view of the electronic device is initiated.

12. The system of claim 11, wherein the memory device contains additional instructions that, when executed by the processor, further cause the processor to perform additional operations, the additional operations further comprising: detecting that the operating mode of the electronic device has changed; as well as In response to detecting the change, the set of input modalities used to initiate the recording of content is updated based on the changed operating mode.

13. The system of claim 11, wherein the associated base device corresponds to a mobile computing device, a laptop computer, or a desktop computer.

14. The system of claim 11, wherein identifying the set of input modalities for initiating the recording of content further causes the processor to perform additional operations, the additional operations further comprising: determining a quality of service metric associated with the operating mode of the electronic device; selecting at least one input modality based at least in part on the quality of service metric; as well as The at least one input modality is provided as the set of input modalities for initiating the recording of the content within the field of view of the electronic device.

15. The system of claim 14, wherein the quality of service metric is based at least in part on available computing resources or available power in the electronic device, the available power including an amount of battery power.

16. The system of claim 14, wherein the memory device contains additional instructions that, when executed by the processor, further cause the processor to perform additional operations, the additional operations further comprising: Deactivate a specific input modality corresponding to at least one sensor in the electronic device, wherein the specific input modality includes facial expression, gaze direction, eye position, gesture, hardware input, voice, or recognition of an object or person in a scene.

17. The system of claim 14, wherein the memory device contains additional instructions that, when executed by the processor, further cause the processor to perform additional operations, the additional operations further comprising: When the quality of service metric is lower than a threshold, a specific sensor in the electronic device is deactivated, wherein the specific sensor comprises a camera, an inertial measurement unit, a microphone, or a touch sensor.

18. The system of claim 11, wherein the memory device contains additional instructions that, when executed by the processor, further cause the processor to perform additional operations, the additional operations further comprising: An area of ​​interest in the field of view of the electronic device is determined, wherein the area of ​​interest is determined based on a gesture or an indicator corresponding to the area of ​​interest.

19. The system of claim 11, wherein the memory device contains additional instructions that, when executed by the processor, further cause the processor to perform additional operations, the additional operations further comprising: generating an annotation corresponding to the record of content; as well as The annotation is added as metadata to the record of the content.

20. A non-transitory computer-readable medium comprising instructions that, when executed by a computing device, cause the computing device to perform operations comprising: determining an operating mode of the electronic device based at least in part on whether the electronic device is communicatively coupled to an associated base device; Based on the determined mode of operation, identifying a set of input modalities for initiating recording of content within a field of view of the electronic device; activating or deactivating at least one hardware component corresponding to the identified set of input modalities based on available processing resources and / or power resources corresponding to the determined operating mode; monitoring sensor information generated by at least one sensor included in or communicatively coupled to the electronic device; initiating recording of content within a field of view of the electronic device when the monitored sensor information indicates that at least one input modality from the identified set of input modalities has been triggered; detecting that the operating mode of the electronic device has changed; as well as In response to detecting the change, the set of input modalities used to initiate the recording of content is updated based on the changed operating mode.

Images