Motion blur reduction for controller tracking

Abstract

Provides motion blur reduction for controller tracking. [Solution] A handheld controller is instructed to illuminate according to camera data. The camera's exposure characteristics are predicted for additional frames. The exposure characteristics are predicted based on the current frame captured by the camera and one or more previous frames captured by the camera of the first device. Based on the predicted one or more exposure characteristics, an LED emission timing is determined for a controller device separate from the first device. The emission command is sent to the controller device in accordance with the LED emission timing. Future frames captured by the camera are used to determine the controller's position information.

Description

Technical Field

[0001] [Background Art] Some devices can generate and present an extended reality (XR) environment. The XR environment can include a fully or partially simulated environment in which people perceive and / or interact via an electronic system. In XR, a subset of a person's body movements or their representation is tracked, and in response, one or more characteristics of one or more virtual objects simulated within the XR environment are adjusted to behave with realistic characteristics.

[0002] Handheld controllers can be used in an XR environment to enhance user input. The handheld controller can be used as an input system for interacting with a virtual environment. This can provide a more intuitive and natural way to interact with virtual content and improve the immersive experience. These controllers can be tracked by the system to provide input, for example, based on illuminators on the controller. For example, the image data of the controller can be captured to determine the characteristics corresponding to the input. However, what is needed is an improvement of the track controller.

Brief Description of the Drawings

[0003] [Figure 1A] An exemplary diagram of a user interacting with a controller in an extended reality environment according to some embodiments is shown. [Figure 1B] An exemplary diagram of a user interacting with a controller in an extended reality environment according to some embodiments is shown.

[0004] [Figure 2A] Exemplary image data of a hand based on emitter settings on a controller according to one or more embodiments is shown. [Figure 2B] Exemplary image data of a hand based on emitter settings on a controller according to one or more embodiments is shown.

[0005] [Figure 3] A flowchart shows a technique for determining the attitude of a controller by tracking the emitter, according to one or more embodiments.

[0006] [Figure 4] A flowchart of the technique for generating emission commands for a controller, according to several embodiments, is shown.

[0007] [Figure 5] A flowchart of the technique for determining the ideal release time, according to several embodiments, is shown.

[0008] [Figure 6] A system diagram of an electronic device that can be used for controller tracking, according to one or more embodiments, is shown.

[0009] [Figure 7] This document presents an exemplary system for use in various augmented reality technologies. [Modes for carrying out the invention]

[0010] This disclosure relates to a system, method, and computer-readable medium that enables controller tracking in an augmented reality environment. In particular, the techniques described herein relate to adjusting lighting settings on a controller for improved image capture for controller tracking.

[0011] In some augmented reality contexts, handheld controllers can be used to generate user input. These handheld controllers may be tracked to determine the characteristics of their movement or posture, which can then be converted into user input. As an example, a handheld controller may include one or more illuminators, such as light-emitting diodes (LEDs), that emit light that can be detected in image data by a user device to track the controller. Similarly, other features of the controller can be tracked in image data to determine the characteristics of the controller's movement. However, if the illuminators cause blurring in the image data, the accuracy of the detected motion characteristics may be impaired. With current technology, the camera used to track the controller may also be used for other image-based processing such as scene understanding, hand or body tracking, and object detection. Thus, the camera setup may be tied to non-controller processing. Furthermore, the camera may use rolling shutter techniques to capture images, which can produce motion blur artifacts when the controller or camera moves rapidly. Motion blur can degrade the tracking accuracy and performance of the controller tracking system.

[0012] In some embodiments, motion blur reduction techniques may be implemented to reduce or eliminate motion blur artifacts in a controller tracking system. The techniques described herein involve predicting future exposure characteristics of a camera used for controller tracking in order to adjust the lighting on the controller to improve controller tracking. In some embodiments, the device may use a camera to capture images of a controller having one or more light-emitting diodes (LEDs) or other illuminators. This technique may include predicting the exposure characteristics of one or more future frames based on the exposure characteristics of the camera used to capture the current frame and one or more previous frames. In some embodiments, additional considerations, such as information on ongoing or planned camera usage, may be used to predict the exposure characteristics. This may be received, for example, from an image signal processor (ISP), a motion sensor group (MSG), etc. The LED transmission timing of the controller may be determined based on the predicted exposure characteristics. The LED transmission timing may specify when and for how long the LEDs should emit light in order to optimize or improve the controller's image capture and tracking by reducing blur in the resulting image data. It may also include sending an emission command to the controller based on the determined LED transmission timing. In some embodiments, LED-based tracking may be used within the controller's multimodal tracking and may be weighted against other sensor data collected by the device and / or controller.

[0013] The techniques described herein offer a technical improvement over illuminator-based controller tracking by adapting controller-based lighting to expected camera settings in order to reduce blur. Other camera-based processes can then continue to use image data captured by the camera based on those other settings. Thus, controller tracking can be improved without damaging other systems that rely on camera data.

[0014] In the following disclosure, "physical environment" refers to the physical world that people can perceive and / or interact with without the aid of electronic devices. A physical environment may include physical features such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park, including physical trees, physical buildings, and physical people. People can directly perceive and / or interact with the physical environment through senses such as sight, touch, hearing, taste, and smell. In contrast, an XR environment refers to a fully or partially simulated environment that people perceive and / or interact with through electronic devices. For example, an XR environment may include Augmented Reality (AR) content, Mixed Reality (MR) content, Virtual Reality (VR) content, etc. In an XR system, a subset or representation of a person's bodily movements is tracked, and accordingly, one or more properties of one or more virtual objects simulated within the XR environment are adjusted to behave according to at least one law of physics. As an example, an XR system can detect a person's head movements and, in response, adjust the graphic content and sound field presented to that person in a manner similar to how such views and sounds would change in the physical environment. As another example, an XR system can detect the movement of an electronic device presenting an XR environment (e.g., a mobile phone, tablet, laptop) and, in response, adjust the graphic content and sound field presented to that person in a manner similar to how such views and sounds would change in the physical environment. In some situations (e.g., for accessibility reasons), an XR system can adjust the characteristics of the graphic content within the XR environment in response to a representation of bodily movement (e.g., a voice command).

[0015] The existence of a wide variety of electronic systems enables people to perceive and / or interact with various XR environments. Examples include head-mountable systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be positioned over a person's eyes (similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mountable system may have one or more speakers and an integrated opaque display. Alternatively, a head-mountable system may be configured to accept an external opaque display (e.g., a smartphone). A head-mountable system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. A head-mountable system may have a transparent or translucent display instead of an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a holographic medium, an optical coupler, an optical reflector, or any combination thereof. In some implementations, the transparent or translucent display may be configured to be selectively opaque. A projection-based system may employ retinal projection technology to project a graphical image onto a person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as a hologram or onto a physical surface.

[0016] The following description provides numerous specific details for illustrative purposes to enhance understanding of the disclosed concepts. As part of this description, some of the drawings in this disclosure represent structures and devices in block diagram form to avoid obscuring novel aspects of the disclosed concepts. Also, for clarity, not all features of actual implementations are described herein. Furthermore, as part of this description, some of the drawings in this disclosure may be provided in flowchart form. Any particular boxes in a flowchart may be presented in a particular order. However, it should be understood that any particular sequence in any flowchart is used only to illustrate one embodiment. In other embodiments, any of the various components shown in the flowchart may be omitted, or the illustrated sequence of operations may be performed in a different order or simultaneously. In addition, other embodiments may include additional steps not shown as part of the flowchart. Furthermore, the language used in this disclosure has been chosen primarily for readability and explanatory purposes, and not to limit or restrict the subject matter of the invention, and it is necessary to rely on the claims to determine such subject matter of the invention. In this disclosure, any reference to “one embodiment” or “one embodiment” means that a particular feature, structure, or characteristic described in relation to the embodiment is included in at least one embodiment of the disclosed subject matter, and any multiple references to “one embodiment” or “one embodiment” should not be understood as all referring to the same embodiment.

[0017] It should be understood that in the development of actual implementations (such as software and / or hardware development projects), numerous decisions must be made to achieve the developer's specific objectives (e.g., compliance with system and business-related constraints), and these objectives may vary depending on the implementation. It should also be understood that while such development efforts can be complex and time-consuming, they are nevertheless routine work for those skilled in the art who are engaged in the design and implementation of graphic modeling systems that are of interest to this disclosure.

[0018] Figure 1 is an illustrative diagram showing an example of a controller tracking system according to several embodiments. Figure 100 includes a user 105 using a controller 115 for user input by using the controller tracking function of an electronic device 120. The electronic device 120 is shown as a head-mounted device (HMD), but other illustrative devices include smartphones, tablets, laptops, desktops, game consoles, or any other suitable devices. The electronic device 120 may include one or more cameras that can be used to capture image data that can track the controller 115. Illustrative cameras include depth cameras, infrared cameras, visible light cameras, or any other suitable cameras. The controller 115 may be a handheld device such as a wand, glove, ring, baton, or any other suitable device used to interact with applications running on the electronic device 120 based on the detected movement of the controller 115. Illustrative related applications include game applications, virtual reality applications, augmented reality applications, or any other suitable applications that utilize the controller 115 as an input device. In the example shown above, the movement of the controller 115 is used, for example, to drag the icon 145 on the virtual display 125 in an extended reality environment using augmented reality, virtual reality, or enhanced reality.

[0019] The controller 115 may include one or more illuminators 135, such as LEDs or any other suitable light sources, that emit light in an adjustable pattern or sequence. The illuminators 135 may be positioned in a default arrangement or other configuration, for example, within the field of view 125 of the device's camera, enabling the device 120 to identify and track the controller 115 based on the emitted light detected in the captured image.

[0020] In some embodiments, the illuminator-based tracking technology can be used as part of a multimodal tracking process. To that end, the controller 115 can also include a motion sensor such as an accelerometer, gyroscope, magnetometer, or any other suitable sensor that detects the movement and orientation of the controller 115. The motion sensor can provide motion data that can be used in conjunction with illuminator-based tracking to determine the position and / or orientation of the controller 115. As another example, the device 120 can additionally use hand tracking, object tracking, or other techniques to contribute to the determination of the position and / or orientation of the controller 115.

[0021] According to some embodiments, one or more cameras of the device 120 can be used by multiple functions or pipelines for different functions such as scene understanding, tracking, image or video capture. To that end, the camera settings can change dynamically based on the requirements of the process that utilizes the image data captured by one or more cameras. Thus, in some embodiments, the device 120 can predict future exposure parameters for the next frame based on previous exposure data and / or expected or ongoing image processing tasks in which the image data will or is to be captured by one or more cameras. The predicted exposure parameters can then be used to preemptively instruct the controller to adjust the lighting pattern based on the expected camera settings. The emission instructions can include, for example, start time, duration, frequency, luminance, color, or any other suitable parameters indicating how the illuminator 135 should emit light for improved image capture based on the expected camera settings such as exposure parameters. The emission instructions may be transmitted from the device 120 to the controller 115 via a wired or wireless connection such as Bluetooth, Wi-Fi, or any other suitable protocol.

[0022] Figures 2A-2B show exemplary image data of a hand based on emitter settings on a controller according to one or more embodiments. In particular, FIG. 2A shows exemplary diagrams of image data and exposure when the illuminator is not adjusted, and FIG. 2B shows exemplary diagrams of image data and exposure when the illuminator is adjusted for a predicted camera setting and / or exposure parameter.

[0023] In FIG. 2A, a set of image frames 200 is captured without correction of the illuminator for the predicted exposure parameters. For example purposes, the image frames 200 correspond to the movement of the user and controller described above with respect to FIG. 1. In this example, the emitter on the controller is active longer than the global shutter period, resulting in the emitter being captured at multiple positions during the camera shutter time, thereby causing blur in frames 205A and 205B.

[0024] The blur shown in the frames is explained when considering exposure diagram 240. For example purposes of this example, rolling shutter exposure is used. However, it should be understood that one or more embodiments described herein can be applied to cameras having a global shutter or other configurations. A camera with a rolling shutter integrates rows of pixels at different times. As shown in exposure diagram 240, exposure time 215 indicates the length of time any given row is integrating, and frame time 220 indicates the length of time any given row is not integrating. The rows are integrated in a rolling process, which means that, at times, not all rows are integrated simultaneously. In some examples, all rows are integrated simultaneously. This is considered the global shutter period 230, during which a rolling shutter camera functions as a global shutter camera. However, in some embodiments, the combination of exposure time 215 and frame time 220 can result in a scenario where there is no global shutter.

[0025] In some embodiments, when the uncorrected emission 210 causes the illuminator to emit light for a longer period than the global shutter duration 230, blurring may occur within the image frame 200. That is, the controller's movement is captured differently by some lines than by others. This is shown as the uncorrected emission time 225 exceeding the global shutter duration 230. Since controller tracking can rely on determining the change in the illuminator's position across frames, the blurring can make it difficult to identify the change in position and / or orientation from frame 205A to 205B.

[0026] In contrast, referring to Figure 2B, image frame 250 is presented. Image frame 250 includes image frames 255A and 255B that reflect the user and controller movements in Figure 1. In this scenario, the illuminator is clearly visible. According to one or more embodiments, the result is achieved not by adjusting the camera settings, but by adjusting the illuminator settings on the controller based on the expected exposure parameters. Here, exposure figure 290 shows the corrected emission 260. The ideal emission time 275 is determined based on the global shutter period 230. In some embodiments, the ideal emission time 275 may be shorter than the global shutter period 230 to account for a time synchronization buffer 280 to account for potential synchronization problems between the device and the controller. Since the ideal emission time 275 is shorter than the global shutter period, all rows capture the illuminator for the same amount of time, resulting in a sharper and less blurred image of the illuminator. Thus, controller tracking can rely more accurately on the determined change in the position and / or orientation of the illuminator from frame 255A to frame 255B.

[0027] Figure 3 shows a flowchart of a technique for determining the attitude of a controller by tracking the emitter, according to one or more embodiments. In particular, Figure 3 shows a technique for providing emission commands to the controller to facilitate controller tracking, according to one or more embodiments. While the flowchart describes various components as performing a particular process, it should be understood that the flow in the figure may differ in some embodiments, and the function of the components may also differ in some embodiments. Furthermore, the various processes may be performed in different orders.

[0028] Flowchart 300 begins at block 305, where controller data for the current frame is acquired. In some embodiments, as shown in block 310, the controller data may include image data of the controller (e.g., controller 115 shown in Figure 1). The image data may include, for example, pixel values, color values, depth values, or any other suitable data that captures an image of the controller in the current frame. The image data may be captured by a camera using a rolling shutter technique in which different rows of the camera sensor are exposed at different times.

[0029] Optionally, in block 315, obtaining controller data for the current frame may include obtaining controller motion data. Controller motion data may include, for example, acceleration, angular velocity, or any other appropriate data indicating the position and / or orientation of the controller. Controller motion data may be obtained from one or more sensors in the controller or from another appropriate source. The current frame may be a frame being captured or processed by camera 130 at a given time.

[0030] In block 320, flowchart 300 includes predicting the exposure characteristics of one or more future frames from the current frame and one or more previous frames. The exposure characteristics may include, for example, intermediate exposure time, integration time, frame time, or any other appropriate parameters indicating the camera's exposure settings. The current frame exposure characteristics may be the camera settings used to capture the image data in block 310. One or more sets of exposure characteristics may be called from one or more previous frames. Future frames may be frames to be captured or processed by the camera after the frames captured in block 310. The prediction may be performed using the current exposure parameters and historical exposure parameters, as well as other data indicating ongoing or upcoming changes in the exposure parameters. For example, the prediction process may consider ISP data from the camera 130's ISP, MSG data from various sensors in the device, etc. The prediction process may use various models, algorithms, or techniques such as machine learning, or any other appropriate method, to predict the exposure characteristics of future frames.

[0031] In block 325, flowchart 300 includes determining the LED transmission timing of the controller based on predicted exposure characteristics. The LED transmission timing may specify when and for how long the illuminator on the controller should emit light in order to optimize the controller's image capture and tracking. The LED transmission timing may be determined based on various data about the device and / or controller, such as predicted exposure characteristics, target integral ratio, minimum emission duration, and maximum emission duration, obtained in block 320. The determination may also use various models, algorithms, or techniques, such as optimization, heuristics, or any other preferred method, as described below with respect to Figure 4.

[0032] Flowchart 300 proceeds to block 330. In block 330, an emission command is sent to the controller based on the LED transmission timing. The emission command may include, for example, start time, duration, frequency, brightness, color, or any other appropriate parameters indicating how the illuminator 135 should operate. The emission command may be transmitted by the device via a wired or wireless connection such as Bluetooth, Wi-Fi, or any other suitable protocol.

[0033] In block 335, flowchart 300 includes a decision on whether any additional frames are to be captured. If no additional frames are to be captured, the controller is no longer being tracked, and the flowchart terminates. That is, since the emit command is based on the expected camera configuration, the final emit command may be sent for future frames that will never be captured.

[0034] Returning to block 335, if additional frames have been captured, flowchart 300 proceeds to block 340. In block 340, method 300 may capture one or more additional frames of the controller while the controller is using LED emission timing. The additional frames may be frames captured or processed by the camera after the current frame and before the next current frame. In some embodiments, the emission command may be sent with a delay so that the image captured using LED emission timing is a future frame after the emission command has been sent, rather than the immediately following frame that is captured.

[0035] Flowchart 300 proceeds to block 345, where the controller's attitude is determined from additional frames, such as frames captured in block 340. The attitude may include, for example, the controller's position and / or orientation. Optionally, the controller can be tracked by performing multimodal tracking, as shown in block 350. Multimodal tracking may use captured images from block 340 and additional sensor data. In some embodiments, the multimodal tracking technique may involve combining or fusing data from different sources, such as a camera, motion sensors within the controller, or other data, to determine the controller's attitude. Multimodal tracking may also involve weighting or adjusting data from different sources based on their reliability, accuracy, or confidence level. For example, the controller attitude determined from the additional frames in block 345 may be weighted according to a confidence value determined for position information, such as position and / or orientation. As another example, controller attitude data from additional frames may be weighted based on the characteristics of the emission command sent to the controller. For example, if the emission command instructs the controller to emit light during an ideal time period, the attitude information can be given more weight and reliability than if the emission command instructs the controller to perform a flood light operation.

[0036] Figure 4 is a flowchart of a technique for generating an emitted command for a controller, according to several embodiments. In particular, the flowchart presented in Figure 4 illustrates an exemplary technique for determining the characteristics of an emitted command. For illustrative purposes, the following steps are described as being performed by specific components. However, it should be understood that various actions may be performed by alternative components. Furthermore, various actions may be performed in different orders. Moreover, some actions may be performed simultaneously, some may not be necessary, or other actions may be added.

[0037] This flowchart begins with predicting exposure characteristics in block 320 of Figure 3. In block 405, flowchart 400 includes obtaining the integration time and mid-exposure time of the previous frame. The integration time and mid-exposure time may be obtained from metadata associated with the previous frame, or they may be received from another suitable source, such as an image processing pipeline. The previous frame may be one or more frames captured or processed by the camera prior to the current frame. In some embodiments, the integration time and mid-exposure time may be obtained for each camera from which image data is used to track the controller. For example, if the device uses a stereoscopic camera system, the integration time and mid-exposure time may be determined for each camera.

[0038] Flowchart 400 proceeds to block 410, where ISP data is acquired. The ISP data may include data from the camera's image signal processor, such as auto exposure data, or any other relevant data indicating the camera's exposure parameters. Furthermore, in block 415, MSG data may be acquired. The MSG data may include data from a group of motion sensors in the device, such as accelerometer data, gyroscope data, magnetometer data, or any other appropriate data indicating the camera's movement and orientation. In some embodiments, the MSG data may indicate characteristics that affect the camera's operation, such as head movement or device movement, which may affect the frame rate or otherwise affect the exposure parameters.

[0039] In block 420, flowchart 400 includes applying current and previous values ​​of integral time and mid-exposure time, ISP data, and MSG data to a predictive model to obtain predicted mid-exposure time and integral time for each camera. The predictive model may be a linear regression model, a machine learning model, or any other suitable model that predicts the exposure characteristics of one or more future frames based on the current and previous frames. In some embodiments, the model may be further configured to predict confident values ​​of the predicted mid-exposure time and / or integral time.

[0040] Flowchart 400 then proceeds to block 325 in Figure 3, which generally concerns determining the LED transmission timing for the controller. Specifically, in block 425, flowchart 400 includes determining whether the predicted intermediate exposure time and integration time provide a global shutter period for each camera. The global shutter period may be a period of time within the camera exposure in which all rows of the camera sensor are integrating light simultaneously. The global shutter period may be determined by subtracting the frame time from the integration time. In block 430, a decision is made as to whether a global shutter period is available. If a global shutter period is available for each camera, flowchart 400 may proceed to block 435.

[0041] In block 435, flowchart 400 includes determining the ideal emission time. The ideal emission time may be less than or equal to the global shutter duration and satisfy the target integral ratio. The ideal emission time represents the ideal illumination time and duration based on the predicted exposure parameters. An example of a technique for determining the ideal emission time is described in more detail below with respect to Figure 5.

[0042] The flowchart proceeds to block 440, where a determination is made as to whether the global shutter duration fits the ideal emission time. In some embodiments, determining whether the global shutter duration fits the ideal emission time may include determining whether the global shutter duration satisfies a combination of the ideal emission time and an additional time synchronization buffer to account for issues related to delays between the two devices. If it is determined in block 440 that the global shutter duration fits the ideal emission time, the flowchart 400 ends in block 450, and the system may generate an emission command for an LED emission setting adjusted so that the emission occurs within the global shutter duration. For example, the emission timing may include an emission for the ideal emission time near a predicted intermediate exposure value.

[0043] Returning to block 440, if it is determined that the global shutter period does not fit the ideal emission time, the flowchart terminates in block 445, and the device may generate an emission command for flood emission. The device may also generate an emission command for flood emission if it is determined in block 430 that the global shutter period is unavailable. Flood emission can be made to ensure that the LED emission period in the controller covers the entire integral time or a substantial portion thereof. Flood emission can ensure that the LED emission is captured by all rows of the camera sensor, but it can result in overexposure or underexposure of the LED emission depending on the ambient brightness and the movement of the controller. Flood emission can also consume more power and generate more heat than the ideal emission scenario.

[0044] Figure 5 shows flowcharts of techniques for determining the ideal release time according to several embodiments. For illustrative purposes, the following steps are described as being performed by specific components. However, it should be understood that various actions may be performed by alternative components. Furthermore, various actions may be performed in different orders. In addition, some actions may be performed simultaneously, some may not be necessary, or other actions may be added.

[0045] In block 505, flowchart 500 may obtain predicted intermediate exposure time and integral time for each camera. The predicted intermediate exposure time and integral time may be obtained, for example, from the exposure prediction technique described above with respect to block 420 in Figure 4. The predicted intermediate exposure time and integral time may indicate the predicted exposure parameters of the camera for one or more future frames. In some embodiments, the prediction may be made for each camera in, for example, a stereo camera system.

[0046] In block 510, flowchart 500 includes obtaining a target integral ratio. The target integral ratio may be a default value or a value adjusted by, for example, a user, an application using the controller. The target integral ratio may represent a desired ratio between the LED emission time and the integration time. Flowchart 500 proceeds to block 515, where the system can determine the integral value from the target integral ratio and the integration time. The integral value may be determined by multiplying the target integral ratio by the integration time.

[0047] In block 520, flowchart 500 includes determining the maximum value from the integral values ​​determined in block 515 and determining the minimum emission duration. The minimum emission duration may be a default or configurable value that indicates the lower limit of the controller's acceptable emission time. The minimum emission duration may be selected by the user or commanded by the application using user input from the tracked controller. The maximum value may be determined by selecting a larger value between the integral value and the minimum emission duration.

[0048] Flowchart 500 proceeds to block 525, where the minimum value is determined from the previously determined maximum value and maximum emission duration from block 520. The maximum emission duration may be a default or configurable value indicating the upper limit of the acceptable emission time. The maximum emission duration may be selected by the user or commanded by the application using user input from a tracked controller. The minimum value may be determined by selecting a smaller value between the previously determined maximum value and maximum emission duration. The minimum value identified in block 525 may be the determined ideal duration.

[0049] Referring to Figure 6, a simplified system diagram is shown. In particular, the system includes an electronic device 600 and a physical controller 670. The electronic device 600 may be part of a multifunction device such as a mobile phone, tablet computer, personal digital assistant, portable music / video player, wearable device, head-mounted system, projection system, base station, laptop computer, desktop computer, network device, or any other electronic system described herein. The electronic device 600 may include one or more additional devices such as a server device, base station, or accessory device, in which various functions may be included or distributed throughout. Exemplary networks include, but are not limited to, local networks such as a Universal Serial Bus (USB) network, an organization's local area network, and a wide area network such as the Internet. According to one or more embodiments, the electronic device 600 is used to interact with the user interface of an application. It should be understood that the various components and functions within the electronic device 600 may be distributed differently across modules or components, or even across additional devices.

[0050] The electronic device 600 may include one or more processors 620, such as a central processing unit (CPU) or a graphics processing unit (GPU). The electronic device 600 may also include memory 630. Memory 630 may include one or more different types of memory that can be used in conjunction with the processor(s) 620 to perform device functions. For example, memory 630 may include cache, ROM, RAM, or any type of temporary or non-temporary computer-readable storage medium capable of storing computer-readable code. Memory 630 may store various programming modules for execution by the processor 620, including an exposure prediction module 655, a controller tracking module 635, and one or more applications 645.

[0051] The exposure prediction module 655 may be used to predict the exposure parameters of one or more cameras of the electronic device 600 in future frames, such as camera 605. The exposure prediction module 655 may use data from camera 605 and / or other sensors 610 of the electronic device 600 to predict future exposure parameters. The controller tracking module 635 can instruct the physical controller to activate the illuminator 650 using a determined timing. In addition, the controller tracking module 635 may determine the position and / or orientation information of the physical controller 670 by, for example, analyzing image data captured by camera 605. The position and / or orientation information from the physical controller may be combined in a multimodal tracking process with additional data, for example, from the motion sensor 660 of the physical controller 670, or other data. The position and / or orientation information may then be used for user input, for example, for application 645.

[0052] The electronic device 600 may also include a storage device 640. The storage device 640 may include one or more non-temporary computer-readable media, including, for example, magnetic disks and tapes (fixed, floppy, and removable), optical media such as CD-ROMs and digital video discs (DVDs), and semiconductor memory devices such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM). The storage device 640 may be used to store various data and structures that can be used to store data related to controller tracking. In addition, the storage device 640 may be configured to store user registration data 625 that may include user-specific characteristics used for controller tracking.

[0053] In one or more embodiments, each of the one or more cameras 605 may be a conventional RGB camera or a depth camera. Furthermore, the cameras 605 may include a stereo camera or other multi-camera system. In addition, the electronic device 600 may include other sensors that can collect sensor data for tracking user movement, such as a depth camera, an infrared sensor, or one or more gyroscopes, accelerometers, and other orientation sensors.

[0054] The electronic device 600 may also include a display 680 that can present a user interface (UI) for user interaction. The display 680 may be opaque, semi-transparent, or transparent. The display 680 can incorporate LEDs, OLEDs, digital light projectors, liquid crystal displays on silicon, and the like.

[0055] Although the electronic device 600 is shown as including the numerous components described above, in one or more embodiments, various components may be distributed across multiple devices. Therefore, while specific calls and transmissions are described in this specification with respect to a particular illustrated system, in one or more embodiments, various calls and transmissions may be directed in different directions based on differently distributed functions. Furthermore, additional components may be used, and combinations of some of the functionalities of any of the components may be linked.

[0056] Referring next to Figure 7, a simplified functional block diagram of an exemplary multifunctional electronic device 700 according to one embodiment is shown. Each of the electronic devices may be a multifunctional electronic device, or may have some or all of the described components of the multifunctional electronic device described herein. The multifunctional electronic device 700 may include a processor 705, a display 710, a user interface 715, graphics hardware 720, device sensors 725 (e.g., proximity sensor / ambient light sensor, accelerometer, and / or gyroscope), a microphone 730, one or more audio codecs 735, one or more speakers 740, a communication circuit 745, a digital image capture circuit 750 (e.g., including a camera system), one or more video codecs 755 (e.g., supporting a digital image capture unit), a memory 760, a storage device 765, and a communication bus 770. The multifunctional electronic device 700 may be, for example, a digital camera or a personal electronic device such as a personal digital assistant (PDA), a personal music player, a mobile phone, or a tablet computer.

[0057] The processor 705 can execute instructions necessary to perform or control the operation of numerous functions performed by the device 700 (e.g., image generation and / or processing as disclosed herein). The processor 705 can, for example, drive the display 710 and receive user input from the user interface 715. The user interface 715 may enable a user to interact with the device 700. For example, the user interface 715 can take various forms such as buttons, keypads, dials, click wheels, keyboards, display screens, and / or touchscreens, gaze input, and / or gestures. The processor 705 may also be a system-on-a-chip, such as those found in mobile devices, and may include a dedicated GPU. The processor 705 may be based on a reduced instruction set computer (RISC) or composite instruction set computer (CISC) architecture or any other suitable architecture, and may include one or more processing cores. The graphics hardware 720 may be dedicated computing hardware for processing graphics and / or assisting the processor 705 in processing graphics information. In one embodiment, the graphics hardware 720 may include a programmable GPU.

[0058] The image capture circuit 750 may include two (or more) lens assemblies 780A and 780B, each having a distinct focal length. For example, lens assembly 780A may have a shorter focal length than lens assembly 780B. Each lens assembly may have separate associated sensor elements 790A and 790B. Alternatively, two or more lens assemblies may share a common sensor element. The image capture circuit 750 can capture still images and / or video images. The output from the image capture circuit 750 may be processed by a video codec(single or multiple) 755 and / or a processor 705 and / or graphics hardware 720, and / or a dedicated image processing unit or pipeline incorporated within the circuit 750. The images thus captured may be stored in memory 760 and / or storage device 765.

[0059] The image capture circuit 750 can capture still images and video images, which, in accordance with this disclosure, may be processed, at least in part, by a video codec(s) 755 and / or a processor 705 and / or graphics hardware 720, and / or a dedicated image processing unit incorporated within the circuit 750. Images thus captured may be stored in memory 760 and / or storage device 765. Memory 760 may include one or more different forms of media used by the processor 705 and graphics hardware 720 to perform the functions of the device. For example, memory 760 may include a memory cache, read-only memory (ROM), and / or random access memory (RAM). Storage device 765 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. The storage device 765 may include one or more non-temporary computer-readable storage media, such as magnetic disks and tapes (fixed, floppy, and removable), optical media such as CD-ROMs and DVDs, and semiconductor memory devices such as EPROMs and EEPROMs. The memory 760 and storage device 765 can be organized into one or more modules and used to tangibly hold computer program instructions or code written in any desired computer programming language. For example, when executed by the processor 705, such computer program code can perform one or more of the methods described herein.

[0060] The various processes defined herein consider options for obtaining and using user identification information. For example, such personal information may be used to track user movements. However, insofar as such personal information is collected, such information should be obtained with the user's informed consent, and the user should have knowledge and control over the use of that personal information.

[0061] Personal information will be used only for legitimate and reasonable purposes by the appropriate parties. Those who use such information will adhere to privacy policies and practices that comply with at least the applicable laws and regulations. Furthermore, such policies should be well-established and meet or exceed government / industry standards. In addition, these parties will not distribute, sell, or share such information for any purpose other than a reasonable and legitimate purpose.

[0062] Furthermore, the intent of this disclosure is that personal data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data when it is no longer needed. In addition, where applicable, including in certain health-related applications, data anonymization can be used to protect user privacy. Anonymization can be facilitated, where appropriate, by removing certain identifiers (e.g., date of birth), controlling the amount or specificity of data stored (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data across users), and / or by other means.

[0063] It should be understood that the above description is illustrative and not limiting. The material is presented in the content of specific embodiments so that a person skilled in the art can manufacture and use the disclosed subject matter as claimed, and variations of those embodiments will be readily apparent to a person skilled in the art (for example, some of the disclosed embodiments may be used in combination with one another). Accordingly, the specific configurations of steps or actions shown in Figures 2-5, or the configurations of elements shown in Figures 1 and 6-7, should not be interpreted as limiting the scope of the disclosed subject matter. Accordingly, the scope of the present invention should be determined by referring to the appended claims and the entire scope of equivalents given to such claims. In the appended claims, the words “including” and “in which” are used as plain English equivalents of the terms “comprising” and “wherein,” respectively.

Claims

1. It is a method, Based on the current and previous frames captured by the camera of the first device, predict the exposure characteristics of the camera for additional frames in order to obtain predicted exposure characteristics. Based on the predicted exposure characteristics, the adjusted LED emission settings of the second device are determined. The first device transmits an emission command to the second device in accordance with the adjusted LED emission settings, The position information of the second device is determined by the camera based on LED emission captured within the additional frame, in a method.

2. The method according to claim 1, wherein the adjusted LED emission setting is different from the current LED emission setting of the second device.

3. The method according to claim 1, wherein the position information of the second device is determined using multimodal tracking, and the position information of the second device is weighted with respect to additional position information using additional tracking modes.

4. The method according to claim 3, wherein the position information of the second device is weighted with respect to the additional position information based on the predicted exposure characteristics.

5. The method according to claim 3, wherein the additional tracking mode includes one or more of the following: visual tracking performed by the first device and motion sensor data received from the second device.

6. The method according to any one of claims 1 to 5, wherein predicting the exposure characteristics includes determining a target camera exposure based on an active process on the first device using the camera.

7. The method according to claim 6, wherein the camera is a rolling shutter camera, and determining the LED emission timing of the second device includes determining a target exposure period in which all rows of camera sensors for the rolling shutter camera are integrating simultaneously.

8. Determining the LED emission timing of the second device is: In response to the determination that the target camera exposure satisfies the flood emission parameters, The method according to claim 6, comprising generating an emission command to cause the second device to emit light throughout the exposure period.

9. The method according to claim 6, wherein the camera is a global shutter camera.

10. A non-temporary computer-readable medium containing computer-readable code, wherein the computer-readable code is Based on the current and previous frames captured by the camera of the first device, predict the exposure characteristics of the camera for additional frames in order to obtain predicted exposure characteristics. Based on the predicted exposure characteristics, the adjusted LED emission settings for the second device are determined. The first device is executable by one or more processors to send emission commands to the second device according to the adjusted LED emission settings. The position information of the second device is determined by the camera based on LED emission captured within the additional frame, in a non-temporary computer-readable medium.

11. The non-temporary computer-readable medium according to claim 10, wherein the adjusted LED emission setting is different from the current LED emission setting of the second device.

12. The non-temporary computer-readable medium according to claim 10, wherein the position information of the second device is determined using multimodal tracking, and the position information of the second device is weighted against additional position information using additional tracking modes.

13. The non-temporary computer-readable medium according to claim 12, wherein the position information of the second device is weighted with respect to the additional position information based on the predicted exposure characteristics.

14. The non-temporary computer-readable medium according to claim 12, wherein the additional tracking mode includes one or more of visual tracking performed by the first device and motion sensor data received from the second device.

15. The computer-readable code for predicting the exposure characteristics is: A non-temporary computer-readable medium according to any one of claims 10 to 14, comprising a computer-readable code for determining a target camera exposure based on an active process on the first device using the camera.

16. The camera is a rolling shutter camera, and the computer-readable code for determining the LED emission timing of the second device is, The non-temporary computer-readable medium according to claim 15, comprising a computer-readable code for determining a target exposure period in which all rows of the camera sensor for the rolling shutter camera are simultaneously integrating.

17. The computer-readable code for determining the LED emission timing of the second device is, in response to the determination that the target camera exposure satisfies the flood emission parameters, A non-temporary computer-readable medium according to claim 15, comprising a computer-readable code for generating emission commands to cause the second device to emit light throughout the exposure period.

18. The non-temporary computer-readable medium according to claim 15, wherein the camera is a global shutter camera.

19. It is a system, One or more processors, A computer-readable medium comprising one or more computer-readable media containing computer-readable code, wherein the computer-readable code is Based on the current and previous frames captured by the camera of the first device, predict the exposure characteristics of the camera for additional frames in order to obtain predicted exposure characteristics. Based on the predicted exposure characteristics, the adjusted LED emission settings for the second device are determined. The first device is executable by one or more processors to send emission commands to the second device according to the adjusted LED emission settings. The position information of the second device is determined by the camera based on the LED emission captured within the additional frame in the system.

20. The system according to claim 19, wherein the adjusted LED emission setting is different from the current LED emission setting of the second device.

21. The system according to claim 19, wherein the position information of the second device is determined using multimodal tracking, and the position information of the second device is weighted with respect to additional position information using additional tracking modes.

22. The system according to claim 21, wherein the position information of the second device is weighted with respect to the additional position information based on the predicted exposure characteristics.

23. The system according to claim 21, wherein the additional tracking mode includes one or more of the following: visual tracking performed by the first device and motion sensor data received from the second device.

24. The computer-readable code for predicting the exposure characteristics is: The system according to any one of claims 19 to 23, comprising a computer-readable code for determining a target camera exposure based on an active process on the first device using the rolling shutter camera.

25. The camera is a rolling shutter camera, and the computer-readable code for determining the LED emission timing of the second device is, The system according to claim 24, comprising a computer-readable code for determining a target exposure period in which all rows of camera sensors for the rolling shutter camera are simultaneously integrating.

26. The computer-readable code for determining the LED emission timing of the second device is, in response to the determination that the target camera exposure satisfies the flood emission parameters, The system according to claim 24, comprising a computer-readable code for generating emission commands to cause the second device to emit light throughout the exposure period.

27. The system according to claim 24, wherein the camera is a global shutter camera.

Images