Capturing image data using a user-specific biometric motion model in an augmented reality device

Abstract

Various implementations disclosed herein include devices, systems, and methods that facilitate an image-based process using captured image data that satisfies the process' image quality requirement. For example, a process may include obtaining sensor data from one or more sensors in a physical environment in which a user is wearing the wearable device tracking a device. The process may further identify a motion pattern of the user wearing the wearable device based on the sensor data, the motion pattern corresponding to an activity type of the user and select at least one camera for image capture based on the identified motion pattern and an image quality requirement associated with a process. The process may further obtain an image from the selected at least one camera and initiate the process using the obtained image.

Description

Attorney Docket No. 097425-01469(P66694WOl)CAPTURING IMAGE DATA USING A USER- SPECIFIC BIOMETRIC MOTION MODEL IN AN AUGMENTED REALITY DEVICETECHNICAL FIELD

[0001] The present disclosure generally relates to systems, methods, and devices that that perform an image-based process using captured image data that satisfies an image quality requirement.BACKGROUND

[0002] Existing image processing systems may be improved with respect to reducing image artifacts to improve image quality.SUMMARY

[0003] Various implementations disclosed herein include systems, methods, and devices that facilitate an image-based process that uses captured image data from a selected camera that satisfies an image quality requirement of the image-based process. For example, an image quality requirement may be related to a requirement that images have limited blur or other motion-induced artifacts. The image-based process may be implemented via an algorithm operating via a first wearable device being worn by a user. For example, a first wearable device may be, inter alia, an optical see through (OST) device, an augmented reality (AR) device, a head mounted device (HMD), artificial intelligence (A / I) glasses (e.g., integrated with generative Al assistance for real time translation, contextual answers and summarization), etc.

[0004] In some implementations, a camera may be selected based on an identified motion type that best satisfies the image quality requirement. For example, an identified motion type may include walking gait, a standing pattern, a sitting pattern, a running pattern, etc.

[0005] In some implementations, a specified motion type(s) may enable selection of a camera not being worn by the user of the wearable device enabling the image-based process. For example, a camera within a physical environment of the user.Attorney Docket No. 097425-01469(P66694WOl)

[0006] In some implementations, a specified motion type(s) may enable selection of a camera of a wearable device being worn on a specific body part such as, for example, a wrist, a head, etc.

[0007] In some implementations, a specified motion type(s) may enable selection of more than one camera from a same wearable device (e.g., the first wearable device) or a different wearable device such as, for example, a watch, an arm band, a ring, etc.

[0008] In some implementations, images from multiple cameras of a single or multiple wearable devices may be fused to increase a signal to noise ratio (SNR) and / or reduce motion blur within a combined image.

[0009] In some implementations, an additional camera may be used to selectively capture only a region of interest.

[0010] In some implementations, a wearable device has a processor (e g., one or more processors) that executes instructions stored in a non-transitory computer-readable medium to perform a method. The method performs one or more steps or processes. In some implementations, the wearable electronic device obtains sensor data from one or more sensors in a physical environment in which a user is wearing the wearable device. In some implementations, a motion pattern of the user wearing the wearable device is identified based on the sensor data. The motion pattern may correspond to an activity type of the user. In some implementations, at least one camera may be selected for image capture based on the identified motion pattern and an image quality requirement associated with a process. In some implementations, an image may be obtained from the selected camera and the process is initiated using the obtained image.

[0011] In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors and the one or more programs include instructions for performing or causing performance of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions, which, when executed by one or more processors of a device, cause the device to perform or cause performance of any of the methods described herein. In accordance with some implementations, a device includes: one or more processors, aAttorney Docket No. 097425-01469(P66694WOl) non-transitory memory, and means for performing or causing performance of any of the methods described herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be had by reference to aspects of some illustrative implementations, some of which are shown in the accompanying drawings.

[0013] Figure 1 illustrates exemplary electronic devices operating in a physical environment, in accordance with some implementations.

[0014] Figure 2 illustrates a view of a user leveraging data from multiple wearable devices to predict and compensate for motion occurring during image capture, in accordance with some implementations.

[0015] Figure 3 illustrates a view representing differing motions of a user and associated gait-based sensor / motion data, in accordance with some implementations.

[0016] Figure 4 illustrates an adaptive system that leverages sensor data from wearable device motion sensors to recognize user motion and optimize image capture, in accordance with some implementations.

[0017] Figure 5 is a flowchart representation of an exemplary method that implements an image-based process using captured image data that satisfies an image quality requirement, in accordance with some implementations.

[0018] Figure 6 is an example electronic device in accordance with some implementations.

[0019] In accordance with common practice the various features illustrated in the drawings may not be drawn to scale. Accordingly, the dimensions of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method or device. Finally, like reference numerals may be used to denote like features throughout the specification and figures.Attorney Docket No. 097425-01469(P66694WOl)DESCRIPTION

[0020] Numerous details are described in order to provide a thorough understanding of the example implementations shown in the drawings. However, the drawings merely show some example aspects of the present disclosure and are therefore not to be considered limiting. Those of ordinary skill in the art will appreciate that other effective aspects and / or variants do not include all of the specific details described herein. Moreover, well-known systems, methods, components, devices and circuits have not been described in exhaustive detail so as not to obscure more pertinent aspects of the example implementations described herein.

[0021] Figure 1 illustrates exemplary electronic devices 105, 110, and 125 operating in a physical environment 100. In the example of Figure 1, the physical environment 100 is a room that includes a desk 120. The electronic devices 105, 110, and 125 may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about and evaluate the physical environment 100 and the objects within it, as well as information about the user 102 of electronic devices 105, 1 10, and 125. The information about the physical environment 100 and / or user 102 may be used to provide visual and audio content and / or to identify the current location of the physical environment 100 and / or the location of the user within the physical environment 100.

[0022] In some implementations, views of an extended reality (XR) environment may be provided to one or more participants (e.g., user 102 and / or other participants not shown) via electronic device 105 (e.g., a wearable device such as a head mounted / worn device (HMD), an optical see through (OST) device, artificial intelligence (A / I) glasses integrated with generative Al assistance for real time translation, contextual answers and summarization), electronic device 110 (e.g., a wearable device such as a watch, a band, a headset, a belt, etc.), and / or electronic device 125 that is not a wearable device such as, for example, a camera within the physical environment 100. Such an XR environment may include views of a 3D environment that is generated based on camera images and / or depth camera images of the physical environment 100 as well as a representation of user 102 based on camera images and / or depth camera images of the user 102. Such an XR environment may include virtual content that is positioned at 3D locations relative to a 3DAttorney Docket No. 097425-01469(P66694WOl) coordinate system associated with the XR environment, which may correspond to a 3D coordinate system of the physical environment 100.

[0023] In some implementations, the electronic device 105 does not include a display and the user views the physical environment 100 directly, e.g., using optical see-through components and / or through transparent lenses on the electronic device 105. In some implementations, such lenses are not configured to display content. In some implementations, such lenses are configured to display content (e.g., presenting an extended reality (XR) environment by displaying augmentations or other virtual content (that augments the user’s view of the physical environment 100) using optical waveguides that transmit light to display content on the lenses). In some implementations, the electronic device 105 includes a display that presents views of the physical environment 100 that are based on images captured by outward- facing cameras on the electronic device 105, e.g., by providing passthrough video, with or without added virtual content. In some implementations, the display presents an XR environment by displaying augmentations or other virtual content (that augments displayed depictions of the physical environment 100).

[0024] In some implementations, sensor data may be obtained from a sensor(s) (e.g., of electrical device 105, 110, and / or 125) in physical environment 100 in which a user is wearing the wearable device. For example, the sensor(s) may be located on any or all of electrical devices 105, 110, and / or 125.

[0025] In some implementations, a motion pattern of user 102 wearing a wearable device (e.g., electronic device 105) may be identified based on the sensor data. The motion pattern may correspond to an activity type of user 102. For example, the motion pattern may represent a six degrees of freedom (6DoF) motion of one or more body parts (of user 102) upon which a sensor of a wearable device (e.g., electrical device 105 or 110) is being worn. In some implementations, the motion pattern may include, inter alia, a gait pattern corresponding to user 102 walking, a standing pattern corresponding to user 102 standing, a sitting pattern corresponding to user 102 sitting, a running pattern corresponding to user 102 running, etc. In some implementations, specific motion patterns of different wearable devices may be identified based on movement of different body parts occurring when user 102 performs different motion types (e.g., walking,Attorney Docket No. 097425-01469(P66694WOl) sitting, standing, about to stand, about to sit, running, working out, playing / sports activities, etc.) and different types of interactions with the environment (e.g., cooking, washing, cleaning, vacuuming, etc.). In some implementations, the motion pattern may be identified based on sensor data obtained from on-board sensors (e.g., of electronic device 105), sensors of other wearable devices (of electronic device 110), sensors of devices that are not wearable devices (e.g., electronic device 125). In some implementations, the motion pattern may be identified based on past motion patterns, health condition data, height and weight of user 102, etc.

[0026] In some implementations, a camera(s) may be selected for image capture based on the identified motion pattern (e.g., and associated activity type) and an image quality requirement associated with a process. For example, a camera may be selected to minimize motion blur, improve signal to noise ratio, etc. for a specified algorithm or machine learning (ML) model requesting the image. In some implementations, a camera on another wearable device may be selected based on a prediction indicating that the other wearable device may minimize motion blur in a captured image. Likewise, a camara may be selected based on: predicting (a) a best time to capture images; (b) a region of interest; and / or (c) additional capture parameters such as binning, ISP processing, and motion compensation.

[0027] In some implementations, an image may be obtained from the selected camera and the process may be initiated using the obtained image.

[0028] Figure 2 illustrates a view 200 of a device such as an HMD (e.g., device 205a), an OST device, Al glasses, etc. being used by a user 207 to leverage data from multiple wearable devices 205a...205n to predict and compensate for motion occurring during image capture, in accordance with some implementations. In some implementations, motion of a user (e.g., user 207) may cause artifacts such as, inter alia, motion blur, etc. when using wearable devices (e.g., wearable devices 205a...205n) for image capture. Likewise, user differences such as age, physical condition, a type of activity being performed, and external conditions such as lighting or speed of motion may further increase motion induced artifacts during image capture. In some implementations, computer vision (CV) or ML algorithms may be sensitive to motion blur in an image. However, a level of sensitivity to motion blur may vary between algorithms of different type. For example,Attomey Docket No. 097425-01469(P66694WOl) some algorithms may be sensitive to motion blur and other algorithms may be more sensitive to noise in the image rather than motion blur. In some implementations, an amount of motion blur in an image may depend on the which wearable device was used to capture an image, which camera on the wearable device was used, capture parameters used, etc.

[0029] Accordingly, motion induced artifacts in images may be reduced or eliminated by creating personalized motion prediction models (for each user) that analyze a user's motion patterns obtained from data from their wearable devices (e.g., wearable devices 205a. . . 205n) over time to enable precise motion predictions. In some implementations, sensors (e.g., accelerometers, gyroscopes, etc.) of wearable devices 205a.. . 205n may be configured to obtain real-time data such as, inter alia, head movement data, wrist movement data, body motion data, etc. to enable specific motion pattern predictions based on user activity. In some implementations, the real time data may be used with a specific algorithm and / or to train a machine learning (ML) model to predict how a user's body moves during capture of images or video. For example, motion of a wrist 209e (e.g., detected via wearable device 205e such as a watch) and motion of a head 209n (e.g., detected via wearable device 205n such as a headset) of user 207 may be used to predict best possible moments in time for capturing an image with minimal blur. In some implementations, a type of wearable device and associated camera type (e.g., front-facing, rear facing, wide-angle, etc.) being used for image capture may influence an amount of motion blur due to, for example, a field of view, a sensor quality, image processing capabilities, etc. Accordingly, a camera and wearable device configuration may be selected dynamically for image capture depending on an identified motion type and activity. Likewise capture parameters (e.g., exposure time, frame rate, stabilization attributes, etc.) of the selected camera may be adjusted. For example, for some algorithms during periods of high motion, a frame rate may be increased and exposure time may be reduced to minimize blur, while other algorithms that prioritize noise reduction may be configured to adjust these settings differently.

[0030] In some implementations, in order to meet image quality requirements of an algorithm running on a wearable device such as an HMD or OST device, a system for intelligently selecting wearable devices and associated cameras may be implemented basedAttorney Docket No. 097425-01469(P66694WOl) on detected and predicted user motion patterns and associated activities and the specific needs of a specific algorithm running on the wearable device as follows:

[0031] The system may initially collect motion / sensor data (e.g., from gyroscopes, accelerometers, etc.) from wearable devices 205a. . . 205n (on user 207) to enable a motion model (e.g., a rule-based model / algorithm, an ML model, etc.) to learn the user's motion profile, accounting for individual differences in movement during activities such as, for example, walking, etc. Based on the motion model, the system may identify which wearable device and associated camera will be selected (based on an algorithm running on the wearable device) to capture an image. For example, if a head 209a (of user 207) is more stable (with respect to movement) than a wrist 209e while walking, a camera of a head mounted device such as wearable device 205a may be selected to capture the image. Conversely, during certain moments of walking, if wrist 209e exhibits less motion, a camera on wearable device 205e may be selected.

[0032] Subsequent to selecting the wearable device and associated camera for image capture, the camera's capture parameters may be adjusted to meet the algorithm’s requirements. For example, capture parameters such as resolution and field of view (FoV), color vs. monochrome, stereo vs. mono capture, etc. may be adjusted as follows:

[0033] Depending on a task (e.g., object detection vs. scene recognition), camera resolution and FoV may be adjusted. For example, for tasks requiring finer detail, higher resolution and narrower FoV may selected. Likewise, the algorithm may benefit from color information (e.g., for identifying specific objects or environments) so color capture may selected. Conversely, if contrast or motion detection is prioritized, monochrome may be used to reduce noise. In some implementations, for depth perception or 3D reconstruction tasks, stereo image capture may be utilized for 2D tasks or lower computation load and mono capture may be utilized.

[0034] In some implementations, an integration time (e.g., a start and end time during which the image is captured) may be adjusted based on user motion and a type of image required. For example, when wrist 209e is more stable between footsteps, an integration time may be optimized to capture the image during the stable periods.Attorney Docket No. 097425-01469(P66694WOl)

[0035] In some implementations, the system may fine-tune an exact moment to capture an image when motion artifacts such as blur are minimized by predicting the user's movement such as, for example, a footstep pattern or hand movement while walking.

[0036] In some implementations, an ML model or neural network may be trained using sensor data to understand a user's motion pattern in real time. This model may be configured to predict which part of the body is most stable for image capture at any given moment.

[0037] In some implementations, a deterministic, rule-based approach may be used to analyze a user's motion pattern and select an optimal wearable camera for image capture by defining conditions that account for motion stability, user activities, and environmental factors. For example, based on sensor data such as IMU data, a set of rules may be created to determine which camera to activate for image capture based on detected motion patterns.

[0038] In some implementations, as user motion changes (e.g., walking vs. running), the system may dynamically update the motion model to select the best wearable device and camera combination for capturing images that meet the algorithm’s requirements. For example, if an algorithm requests an image while a user is walking, the motion model may predict that wrist 209e is stable during certain moments between footsteps. Therefore, if the algorithm's quality requirements are tolerant to slight wrist motion, a camera on wearable device 205 e may be selected. However, if higher stability is needed, wearable device 205a may be selected assuming head 209n of user 207 is more stable during that activity. Accordingly, the system may ensure that captured images meet specific quality requirements of the wearable device algorithm while minimizing motion artifacts.

[0039] Figure 3 illustrates a view representing differing motions 302a. . . 302n of a user 300 and associated gait-based sensor / motion data 307, in accordance with some implementations. In some implementations, an image capture process may be dynamically adjusted (e.g., wearable device / camera selection, image capture parameters of a selected wearable device / camera, etc.) based on a specific type of motion (e.g., of motions 302a... 302n) of user 300, such as, for example, walking with respect to a stability of different body parts during that motion. For example, if an algorithm of a wearable device such as an HMD or OST device requests an image while user 300 is walking and the user'sAttorney Docket No. 097425-01469(P66694WOl) hand is relatively stable during a specific phase of the walking gait (e.g., during motion 302c), the image capture process may prioritize selecting a camera mounted on a wrist (e.g., electronic device 205e of figure 2) of user 300 to capture that image. The following example further describes a system for detecting user movement and selecting a camera for image capture as follows:

[0040] During user movement, the system detects that user 300 is currently walking based on gait patterns from sensor / motion data 307. Subsequently, an optical character recognition (OCR) algorithm of an HMD or OST device on user 300 requests an image. Since the OCR algorithm requires a sharp image with minimal motion blur for text recognition, the system searches for stable portion of the body of user 300 and identifies a moment in the walking cycle where a hand (wrist) is more stable than a head. In response, a camera on a device on a wrist of user 300 is selected to capture the image thereby ensuring that the image quality is optimized for the OCR algorithm's requirements by considering both the user's current motion and the relative stability of body parts.

[0041] Figure 4 illustrates adaptive system 400 that leverages sensor data 410 from wearable device motion sensors 405 to recognize user motion and optimize image capture, in accordance with some implementations. The adaptive system 400 includes motion sensors 405 (e.g., of electronic devices 205a... 205n of figure 2) such as, inter alia, inertial measurement unit (IMU) sensors, accelerometers, gyroscopes, etc. Likewise, the adaptive system 400 includes cameras 406, sensor data 410, tools / software 408, and a control system 420 that, in some implementations, communicates over a data communication network 402, e.g., a local area network (LAN), a wide area network (WAN), the Internet, a mobile network, or a combination thereof.

[0042] Tools / software 408 comprise image quality requirement tools 416 and camera / device selection tools 412.

[0043] Adaptive system 400 may be configured to dynamically adjust camera selection, capture timing, and additional imaging parameters to reduce motion blur and enhance an overall quality of captured images.

[0044] In some implementations, adaptive system 400 is configured to utilize motion sensors 405 from wearable devices (e.g., electronic devices 205a. . . 205n of figure 2) andAttorney Docket No. 097425-01469(P66694WOl) ecosystem devices (e.g., laptop computer cameras, mobile device cameras, etc.) to track and recognize motion and activities of a user. For example, types of motion and activities may include, inter alia, gait (walking, running, etc.), standing or sitting patterns, working out or sports activities, interactions with the environment (e.g., cooking, cleaning, etc.), etc.

[0045] In some implementations, a tracking / recognition process may rely on analyzing six degrees of freedom (6D0F) motion for different body parts and associated wearable devices in combination with individual factors such as past motion patterns, health condition data, height, weight, etc.

[0046] In some implementations, adaptive system 400 is configured to predict a best time to capture an image by analyzing a user's current activity and motion. For example, during walking, stable phases (e.g., a moment when the wrist is stable) may be identified to set a start and end of image frame integration to minimize motion blur.

[0047] In some implementations, adaptive system 400 is configured to select an optimal wearable device (e.g., via camera / device selection tools 414) for image capture based on motion recognition and an image quality requirement of an algorithm provided by image quality requirement tools 416. Subsequent to the optimal wearable device being selected, an associated camera of the device is selected to minimize motion blur.

[0048] In some implementations, adaptive system 400 is configured to predict and select a region of interest (ROI) to be captured based on a specific task (e.g., OCR, object detection, etc.) and associated motion. In some implementations, the ROI may be adjusted dynamically based on an activity such as, for example, focusing on an area where the user is interacting with the environment.

[0049] In some implementations, adaptive system 400 is configured to optimize parameters such as: binning (i.e., combining adjacent pixels to reduce noise and improve low-light performance), image signal processing (i.e., adjusting post-processing steps such as color correction and sharpness), motion compensation (i.e., using motion sensors to stabilize the image), etc. In some implementations, the parameters may be adjusted to match motion, camera characteristics, and an algorithm’s needs.Attorney Docket No. 097425-01469(P66694WOl)

[0050] In some implementations, adaptive system 400 is configured to enable more than one camera from same or different wearable devices to capture multiple frames simultaneously. For example, a first camera may capture a frame with a longer integration time to increase SNR and a second camera may capture multiple frames with shorter integration times during a same period to reduce motion blur. Subsequently, the captured images may be fused together thereby balancing the advantages of both long exposure (better SNR) and short exposure (minimal motion blur).

[0051] In some implementations, the second camera may only capture the ROI overlapping with the first camera thereby allowing for efficient processing and resource use.

[0052] In some implementations, the second camera may capture images that match a pixel resolution of the first camera thereby ensuring that both sets of images may be fused without distortion or loss of detail.

[0053] Figure 5 is a flowchart representation of an exemplary method 500 that implements an image-based process using captured image data that satisfies an image quality requirement, in accordance with some implementations. In some implementations, the method 500 is performed by a wearable device such as wearable devices 105 and 110 of figure 1. In some implementations, the wearable device has a screen for displaying images and / or a screen for viewing stereoscopic images such as a head-mounted display (an HMD or OST device such as e.g., device 105 of Figure 1), an OST-AR device, etc. In some implementations, the method 500 is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method 500 is performed by a processor executing code stored in a non-transitory computer- readable medium (e.g., a memory). Each of the blocks in the method 500 may be enabled and executed in any order.

[0054] At block 502, the method 500 obtains sensor data from one or more sensors in a physical environment in which a user is wearing the wearable device. For example, sensors such as accelerometers, gyroscopes, etc. of wearable devices 205a...205n obtaining real-time data such as head movement data, wrist movement data, body motion data, etc. as described with respect to figure 2.Attorney Docket No. 097425-01469(P66694WOl)

[0055] At block 504, the method 500 identifies a motion pattern of the user wearing the wearable device based on the sensor data. The motion pattern may correspond to an activity type of the user. For example, motion / sensor data from gyroscopes, accelerometers, etc. may enable a learned motion model (e.g., a rule-based model / algorithm, an ML model, etc. to learn user's motion profile that accounts for individual differences in movement during activities such as, for example, walking, etc. as described with respect to figure 2.

[0056] In some implementations, the motion pattern represents six degrees of freedom (6DoF) motion of a body part of the user upon which a sensor of the one or more sensors is being worn. For example, 6DoF motion for different body parts of a user 300 as described with respect to figures 3 and 4.

[0057] In some implementations, the sensor is located on the wearable device.

[0058] In some implementations, the sensor is located on another wearable device differing from the wearable device.

[0059] In some implementations, the sensor is located within the physical environment.

[0060] In some implementations, the motion pattern is a gait pattern corresponding to walking motion of the user as described with respect to figure 3.

[0061] In some implementations, the motion pattern is a standing pattern corresponding the user standing.

[0062] In some implementations, the motion pattern is a sitting pattern corresponding to the user sitting.

[0063] In some implementations, the motion pattern is a running pattern corresponding to the user running.

[0064] In some implementations, the motion pattern further corresponds to a specified interaction with the physical environment.

[0065] In some implementations, the motion pattern corresponds to a single body part of the user.Attorney Docket No. 097425-01469(P66694WOl)

[0066] In some implementations, the motion pattern corresponds to multiple body parts of the user.

[0067] In some implementations, identifying the motion pattern of the user is further based on data differing from the sensor data. For example, data differing from the sensor data may include past motion patterns, health condition data, height and weight of the user, etc. as described with respect to figure 4.

[0068] At block 506, the method 500 selects at least one camera for image capture based on the identified motion pattern and an image quality requirement associated with a process. For example, a system may identify which wearable device and associated camera will be selected (based on an algorithm running on the wearable device) to capture an image based on a motion model as described with respect to figure 2.

[0069] In some implementations, the image quality requirement is associated with reducing motion related artifacts such as, for example, minimizing motion blur, improving signal to noise ratio, etc.

[0070] In some implementations, the at least one camera may be selected from a second wearable device based (e.g., a smart watch) on a prediction that the second wearable device will minimize motion blur in a captured image.

[0071] In some implementations, the at least one camera may be selected based on camera capture parameters of the at least one camera.

[0072] In some implementations, the camera capture parameters may be used to predict a best time to capture images.

[0073] In some implementations, the camera capture parameters may be used to predict a region of interest of the user.

[0074] At block 508, the method 500 obtains an image from the selected at least one camera as described with respect to figures 2 and 4.

[0075] At block 510, the method 500 initiates the process (e.g., an (OCR) algorithm of an HMD or OST device as described with respect to figure 3) using the obtained image.

[0076] Figure 6 is a block diagram of an example device 600. Device 600 illustrates an exemplary device configuration for electronic devices 105 and 110 of Figure 1. WhileAttorney Docket No. 097425-01469(P66694WOl) certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations the device 600 includes one or more processing units 602 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, and / or the like), one or more input / output (I / O) devices and sensors 606, one or more communication interfaces 608 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.1 lx, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or the like type interface), one or more programming (e.g., I / O) interfaces 610, one or more displays 612, one or more interior and / or exterior facing image sensor systems 614, a memory 620, and one or more communication buses 604 for interconnecting these and various other components.

[0077] In some implementations, the one or more communication buses 604 include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I / O devices and sensors 606 include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptics engine, one or more depth sensors (e.g., a structured light, a time-of-flight, or the like), and / or the like.

[0078] In some implementations, the one or more displays 612 are configured to present a view of a physical environment or a graphical environment to the user. In some implementations, the one or more displays 612 are configured to present content (determined based on a determined user / object location of the user within the physical environment) to the user. In some implementations, the one or more displays 612 correspond to holographic, digital light processing (DLP), liquid-crystal display (LCD), liquid-crystal on silicon (LCoS), organic light-emitting field-effect transitory (OLET), organic light-emitting diode (OLED), surface- conduction electron-emitter display (SED), field-emission display (FED), quantum-dot light-emitting diode (QD-LED), micro-electromechanical system (MEMS), and / or the like display types. In some implementations, the one or more displays 612 correspond to diffractive, reflective,Attorney Docket No. 097425-01469(P66694WOl) polarized, holographic, etc. waveguide displays. In one example, the device 600 includes a single display. In another example, the device 600 includes a display for each eye of the user.

[0079] In some implementations, the one or more image sensor systems 614 are configured to obtain image data that corresponds to at least a portion of the physical environment 105. For example, the one or more image sensor systems 614 include one or more RGB cameras (e.g., with a complimentary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, depth cameras, event-based cameras, and / or the like. In various implementations, the one or more image sensor systems 614 further include illumination sources that emit light, such as a flash. In various implementations, the one or more image sensor systems 614 further include an on-camera image signal processor (ISP) configured to execute a plurality of processing operations on the image data.

[0080] In some implementations, sensor data may be obtained by device(s) (e.g., devices 105 and 110 of Figure 1) during a scan of a room of a physical environment. The sensor data may include a 3D point cloud and a sequence of 2D images corresponding to captured views of the room during the scan of the room. In some implementations, the sensor data includes image data (e.g., from an RGB camera), depth data (e.g., a depth image from a depth camera), ambient light sensor data (e.g., from an ambient light sensor), and / or motion data from one or more motion sensors (e.g., accelerometers, gyroscopes, IMU, etc.). In some implementations, the sensor data includes visual inertial odometry (VIO) data determined based on image data. The 3D point cloud may provide semantic information about one or more elements of the room. The 3D point cloud may provide information about the positions and appearance of surface portions within the physical environment. In some implementations, the 3D point cloud is obtained over time, e.g., during a scan of the room, and the 3D point cloud may be updated, and updated versions of the 3D point cloud obtained over time. For example, a 3D representation may be obtained (and analyzed / processed) as it is updated / adjusted over time (e.g., as the user scans a room).

[0081] In some implementations, sensor data may be positioning information, some implementations include a VIO to determine equivalent odometry information usingAttorney Docket No. 097425-01469(P66694WOl) sequential camera images (e.g., light intensity image data) and motion data (e.g., acquired from the IMU / motion sensor) to estimate the distance traveled. Alternatively, some implementations of the present disclosure may include a simultaneous localization and mapping (SLAM) system (e.g., position sensors). The SLAM system may include a multidimensional (e.g., 3D) laser scanning and range-measuring system that is GPS independent and that provides real-time simultaneous location and mapping. The SLAM system may generate and manage data for a very accurate point cloud that results from reflections of laser scanning from objects in an environment. Movements of any of the points in the point cloud are accurately tracked over time, so that the SLAM system can maintain precise understanding of its location and orientation as it travels through an environment, using the points in the point cloud as reference points for the location.

[0082] In some implementations, the device 600 includes an eye tracking system for detecting eye position and eye movements (e g., eye gaze detection). For example, an eye tracking system may include one or more infrared (IR) light-emitting diodes (LEDs), an eye tracking camera (e.g., near-IR (NIR) camera), and an illumination source (e.g., an NIR light source) that emits light (e.g., NIR light) towards the eyes of the user. Moreover, the illumination source of the device 600 may emit NIR light to illuminate the eyes of the user and the NIR camera may capture images of the eyes of the user. In some implementations, images captured by the eye tracking system may be analyzed to detect position and movements of the eyes of the user, or to detect other information about the eyes such as pupil dilation or pupil diameter. Moreover, the point of gaze estimated from the eye tracking images may enable gaze-based interaction with content shown on the near-eye display of the device 600.

[0083] The memory 620 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, the memory 620 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 620 optionally includes one or more storage devices remotely located from the one or more processing units 602. The memory 620 includes a non-transitory computer readable storage medium.Attorney Docket No. 097425-01469(P66694WOl)

[0084] In some implementations, the memory 620 or the non-transitory computer readable storage medium of the memory 620 stores an optional operating system 630 and one or more instruction set(s) 640. The operating system 630 includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the instruction set(s) 640 include executable software defined by binary information stored in the form of electrical charge. In some implementations, the instruction set(s) 640 are software that is executable by the one or more processing units 602 to carry out one or more of the techniques described herein.

[0085] The instruction set(s) 640 includes a motion pattern instruction set 642 and a camera selection instruction set 644. The instruction set(s) 640 may be embodied as a single software executable or multiple software executables.

[0086] The motion pattern instruction set 642 is configured with instructions executable by a processor to identify a motion pattern of a user wearing a wearable device based on sensor data.

[0087] The camera selection instruction set 644 is configured with instructions executable by a processor to select a camera(s) for image capture based on an identified motion pattern and associated activity type of a user and an image quality requirement associated with a process such as an algorithm operating via a wearable device.

[0088] Although the instruction set(s) 640 are shown as residing on a single device, it should be understood that in other implementations, any combination of the elements may be located in separate computing devices. Moreover, Figure 6 is intended more as functional description of the various features which are present in a particular implementation as opposed to a structural schematic of the implementations described herein. As recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. The actual number of instructions sets and how features are allocated among them may vary from one implementation to another and may depend in part on the particular combination of hardware, software, and / or firmware chosen for a particular implementation.

[0089] Those of ordinary skill in the art will appreciate that well-known systems, methods, components, devices, and circuits have not been described in exhaustive detailAttorney Docket No. 097425-01469(P66694WOl) so as not to obscure more pertinent aspects of the example implementations described herein. Moreover, other effective aspects and / or variants do not include all of the specific details described herein. Thus, several details are described in order to provide a thorough understanding of the example aspects as shown in the drawings. Moreover, the drawings merely show some example embodiments of the present disclosure and are therefore not to be considered limiting.

[0090] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0091] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0092] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarilyAttorney Docket No. 097425-01469(P66694WOl) require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0093] Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively, or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0094] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing. The apparatus can include special purpose logic circuitry, e.g., anFPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a crossplatform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computingAttorney Docket No. 097425-01469(P66694WOl) model infrastructures, such as web services, distributed computing and grid computing infrastructures. Unless specifically stated otherwise, it is appreciated that throughout this specification discussions utilizing the terms such as “processing,” “computing,” “calculating,” “determining,” and “identifying” or the like refer to actions or processes of a computing device, such as one or more computers or a similar electronic computing device or devices, that manipulate or transform data represented as physical electronic or magnetic quantities within memories, registers, or other information storage devices, transmission devices, or display devices of the computing platform.

[0095] The system or systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device can include any suitable arrangement of components that provides a result conditioned on one or more inputs. Suitable computing devices include multipurpose microprocessor-based computer systems accessing stored software that programs or configures the computing system from a general purpose computing apparatus to a specialized computing apparatus implementing one or more implementations of the present subject matter. Any suitable programming, scripting, or other type of language or combinations of languages may be used to implement the teachings contained herein in software to be used in programming or configuring a computing device.

[0096] Implementations of the methods disclosed herein may be performed in the operation of such computing devices. The order of the blocks presented in the examples above can be varied for example, blocks can be re-ordered, combined, and / or broken into sub-blocks. Certain blocks or processes can be performed in parallel. The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0097] The use of “adapted to” or “configured to” herein is meant as open and inclusive language that does not foreclose devices adapted to or configured to perform additional tasks or steps. Additionally, the use of “based on” is meant to be open and inclusive, in that a process, step, calculation, or other action “based on” one or more recited conditions or values may, in practice, be based on additional conditions or valueAttorney Docket No. 097425-01469(P66694WOl) beyond those recited. Headings, lists, and numbering included herein are for ease of explanation only and are not meant to be limiting.

[0098] It will also be understood that, although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first node could be termed a second node, and, similarly, a second node could be termed a first node, which changing the meaning of the description, so long as all occurrences of the “first node” are renamed consistently and all occurrences of the “second node” are renamed consistently. The first node and the second node are both nodes, but they are not the same node.

[0099] The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the claims. As used in the description of the implementations and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0100] As used herein, the term “if’ may be construed to mean “when” or “upon” or “in response to determining” or “in accordance with a determination” or “in response to detecting,” that a stated condition precedent is true, depending on the context. Similarly, the phrase “if it is determined [that a stated condition precedent is true]” or “if [a stated condition precedent is true]” or “when [a stated condition precedent is true]” may be construed to mean “upon determining” or “in response to determining” or “in accordance with a determination” or “upon detecting” or “in response to detecting” that the stated condition precedent is true, depending on the context.

Claims

1. Attorney Docket No. 097425-01469(P66694WOl)What is claimed is:

1. A method comprising: at a wearable device having a processor: obtaining sensor data from one or more sensors in a physical environment in which a user is wearing the wearable device; identifying a motion pattern of the user wearing the wearable device based on the sensor data, the motion pattern corresponding to an activity type of the user; selecting at least one camera for image capture based on the identified motion pattern and an image quality requirement associated with a process; obtaining an image from the selected at least one camera; and initiating the process using the obtained image.

2. The method of claim 1, wherein the motion pattern represents six degrees of freedom (6DoF) motion of a body part of the user upon which a sensor of the one or more sensors is being worn.

3. The method of claim 2, wherein the sensor is located on the wearable device.

4. The method of claim 2, wherein the sensor is located on another wearable device differing from the wearable device.

5. The method of claim 2, wherein the sensor is located within the physical environment.

6. The method of any of claims 1-5, wherein the motion pattern comprises a gait pattern corresponding to walking motion of the user.

7. The method of any of claims 1-5, wherein the motion pattern comprises a standing pattern corresponding the user standing.Attorney Docket No. 097425-01469(P66694WOl)8. The method of any of claims 1-5, wherein the motion pattern comprises a sitting pattern corresponding to the user sitting.

9. The method of any of claims 1-5, wherein the motion pattern comprises a running pattern corresponding to the user running.

10. The method of any of claims 1-5, wherein the motion pattern further corresponds to a specified interaction with the physical environment.

11. The method of any of claims 1-10, wherein the motion pattern corresponds to a single body part of the user.

12. The method of any of claims 1-10, wherein the motion pattern corresponds to multiple body parts of the user.

13. The method of any of claims 1-12, wherein said identifying the motion pattern of the user is further based on data differing from the sensor data.

14. The method of any of claims 1-13, wherein the image quality requirement is associated with reducing motion related artifacts.

15. The method of any of claims 1-14, wherein the at least one camera is selected from a second wearable device based on a prediction that the second wearable device will minimize motion blur in a captured image.

16. The method of any of claims 1-15, wherein the at least one camera is selected based on camera capture parameters of the at least one camera.

17. The method of claim 16, wherein the camera capture parameters are used to predict a best time to capture images.Attorney Docket No. 097425-01469(P66694WOl)18. The method of any of claims 15-16, wherein the camera capture parameters are used to predict a region of interest of the user.

19. A wearable device comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the electronic device to perform operations comprising: obtaining sensor data from one or more sensors in a physical environment in which a user is wearing the wearable device; identifying a motion pattern of the user wearing the wearable device based on the sensor data, the motion pattern corresponding to an activity type of the user; selecting at least one camera for image capture based on the identified motion pattern and an image quality requirement associated with a process; obtaining an image from the selected at least one camera; and initiating the process using the obtained image.

20. A non-transitory computer-readable storage medium, storing program instructions executable by one or more processors to perform operations comprising: at a wearable electronic device having a processor: obtaining sensor data from one or more sensors in a physical environment in which a user is wearing the wearable device; identifying a motion pattern of the user wearing the wearable device based on the sensor data, the motion pattern corresponding to an activity type of the user; selecting at least one camera for image capture based on the identified motion pattern and an image quality requirement associated with a process; obtaining an image from the selected at least one camera; and initiating the process using the obtained image.

Images