Camera settings adjustment based on event mapping

Abstract

Systems, methods, and non-transitory media are provided for adjusting camera settings based on event data. An example method may include acquiring, via an image capture device of a mobile device, an image depicting at least a portion of an environment, determining a match between one or more visual features extracted from the image and one or more visual features associated with a key frame, and adjusting one or more settings of the image capture device based on the match.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates generally to camera setting adjustment. For example, aspects of the present disclosure relate to camera setting adjustment based on event mapping. [Background technology]

[0002] Electronic devices are increasingly equipped with camera hardware for capturing and using images and / or videos. For example, a computing device may include a camera (e.g., a mobile device such as a cell phone or smartphone may include one or more cameras) that allows the computing device to capture video or images of a scene, person, object, etc. The image or video may be captured, processed, and stored or output for use (e.g., displayed on the device and / or another device) by the computing device (e.g., a mobile device, an IP camera, an extended reality (XR) device, a connected device, etc.). In some cases, the image or video may be further processed for effects (e.g., compression, image enhancement, image restoration, scaling, frame rate conversion, etc.) and / or specific applications such as computer vision (CV), extended reality (e.g., augmented reality (AR), virtual reality (VR), etc.), object detection, image recognition (e.g., face recognition, object recognition, scene recognition, etc.), feature extraction, authentication, automation, etc., among others.

[0003] In some cases, the electronic device may process the image to detect objects, faces, and / or any other items captured by the image. Object detection may be useful for various applications, such as, for example, authentication, automation, gesture recognition, surveillance, extended reality, computer vision, among others. In some examples, the electronic device may implement a low-power or "always on (AON)" camera that operates persistently or periodically to automatically detect certain objects in the environment. The low-power camera may be implemented for various use cases, such as, for example, persistent gesture detection, persistent object (e.g., face / person, animal, vehicle, device, airplane, etc.) detection, persistent object scanning (e.g., quick response (QR) code scanning, barcode scanning, etc.), persistent face recognition for authentication, etc. However, persistent and / or more frequent operation of the low-power camera and other camera settings may result in higher overall power consumption. Furthermore, mobile devices implementing such low-power cameras may experience reduced battery life, and stationary devices may require more complex heat dissipation designs and / or exhibit unacceptably low power efficiency during long-term use. Thus, significantly higher power consumption may adversely affect electronic device usage, device performance, and user experience. Summary of the Invention

[0004] Systems and techniques for camera setting adjustment based on event mapping are described herein. According to at least one example, a method is provided for adjusting camera settings based on event data. The method may include acquiring, via an image capture device of an electronic device, an image depicting at least a portion of an environment, determining a match between one or more visual features extracted from the image and one or more visual features associated with a key frame associated with one or more detected events, and adjusting one or more settings of the image capture device based on the match.

[0005] According to at least one example, a non-transitory computer-readable medium for adjusting camera settings based on event data is provided. The non-transitory computer-readable medium can include instructions stored thereon that, when executed by one or more processors, cause the one or more processors to capture, via an image capture device of an electronic device, an image illustrating at least a portion of an environment, determine a match between one or more visual features extracted from the image and one or more visual features associated with a key frame associated with one or more detection events, and adjust one or more settings of the image capture device based on the match.

[0006] According to at least one example, an apparatus for adjusting camera settings based on event data is provided that can include a memory and one or more processors configured to acquire, via an image capture device of the apparatus, images depicting at least a portion of an environment, determine a match between one or more visual features extracted from the images and one or more visual features associated with key frames associated with one or more detected events, and adjust one or more settings of the image capture device based on the match.

[0007] According to at least one example, another apparatus is provided for adjusting camera settings based on event data, which may include means for acquiring, via an image capture device of the apparatus, images depicting at least a portion of an environment, determining a match between one or more visual features extracted from the images and one or more visual features associated with key frames associated with one or more detected events, and adjusting one or more settings of the image capture device based on the match.

[0008] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above can periodically reduce a count of each of the detection events associated with a keyframe entry in the event data in the electronic device, where the count of each of the detection events is reduced proportionally across all keyframe entries in the event data.

[0009] In some examples, adjusting one or more settings of the image capture device may include changing a power mode of the image capture device. In some cases, changing the power mode of the image capture device may include changing at least one of a frame rate of the image capture device, a resolution of the image capture device, a binning mode of the image capture device, an imaging mode of the image capture device, and a number of image sensors activated by at least one of the image capture device and the electronic device. In some examples, changing the power mode of the image capture device may include decreasing at least one of the frame rate, the resolution, the binning mode, the imaging mode, and the number of image sensors activated based on a determination that a likelihood of an event of interest occurring in the environment is below a threshold. In some examples, changing the power mode of the image capture device may include increasing at least one of the frame rate, the resolution, the binning mode, the imaging mode, and the number of image sensors activated based on a determination that a likelihood of an event of interest occurring in the environment is above a threshold.

[0010] In some examples, the image capture device may include a first image capture device, and the methods, non-transitory computer-readable media, and apparatus described above may further include increasing a power mode of a second image capture device of the electronic device based on a determination that a likelihood of an event of interest occurring in the environment is above a threshold, the second image capture device employing at least one of a higher power mode than the first image capture device, a higher frame rate than the first image capture device, a higher resolution than the first image capture device, a greater number of image sensors than the first image capture device, and a higher power processing pipeline than a processing pipeline associated with the first image capture device. In some examples, increasing the power mode of the second image capture device may include initializing the second image capture device.

[0011] In some cases, the key frames are included in event data at the electronic device, and the event data may include multiple key frames from detection events associated with the image capture device, and in some cases, the event data further includes a separate count of detection events associated with each key frame of the multiple key frames.

[0012] In some aspects, the methods, non-transitory computer-readable media, and devices described above may include determining a likelihood that an event of interest will occur in the environment based on the match and one or more keyframes from the multiple keyframes.

[0013] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include determining a likelihood that an event of interest will occur in the environment based on the matches and the distinct count of detected events associated with the key frames.

[0014] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include incrementing a distinct count of detected events associated with the key frame in the event data in response to determining a match between one or more visual features extracted from the image and one or more visual features associated with the key frame.

[0015] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include acquiring, via an image capture device, a different image depicting at least a portion of a different environment; determining that one or more visual features extracted from the different image do not match visual features associated with any keyframes in the event data at the electronic device; and creating a new entry in the event data corresponding to the different image, wherein the new entry is created in response to determining that the one or more visual features extracted from the different image do not match visual features associated with any keyframes in the event data and at least one of a determination that an event of interest was detected in the different image, a time elapsed since an individual keyframe in the event data was last created, and a time elapsed since an individual match was last identified between a particular keyframe in the event data and a particular image captured by the image capture device.

[0016] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include determining that an event of interest is detected in the different image; determining a second likelihood that an additional event of interest will occur in the different environment based on the determination that the event of interest is detected in the different image; and adjusting at least one setting of the image capture device based on the second likelihood that the additional event of interest will occur in the different environment.

[0017] In some cases, one or more settings of the image capture device may include a frame rate, and the methods, non-transitory computer-readable media, and apparatus described above may include determining the frame rate based on a predetermined frame rate for the detected event and a likelihood of an event of interest occurring in the environment, and maintaining the frame rate at least until expiration of a set period of time or until a subsequent determination of a match between a different image captured by the image capture device and at least one key frame in the event data at the electronic device.

[0018] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include adjusting the frame rate of the image capture device to a different frame rate after expiration of a set period of time or after determining a subsequent match. In some examples, the different frame rate may include a default frame rate or a particular frame rate determined based on a predetermined frame rate and a second likelihood of detecting an event of particular interest associated with the subsequent match.

[0019] In some examples, the predetermined frame rate may include a highest frame rate supported by the image capture device, and determining the frame rate may include multiplying the predetermined frame rate by a value corresponding to a likelihood that a detection event will occur within the environment.

[0020] In some examples, the predetermined frame rate is higher than a default frame rate, and the methods, non-transitory computer-readable media, and apparatus described above may include reducing the default frame rate to a lower frame rate in response to adding one or more key frames to the event data at the electronic device.

[0021] In some cases, determining a match between the one or more visual features extracted from the image and the one or more visual features associated with the key frame may further include comparing at least one of the one or more visual features extracted from the image to the one or more visual features associated with the key frame, and comparing a first descriptor of the one or more visual features extracted from the image to a second descriptor of the one or more visual features associated with the key frame.

[0022] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include adjusting one or more different settings of at least one of an active depth transmitter of the electronic device, an audio algorithm, and a location service associated with at least one of a Global Navigation Satellite (GNSS) system, a wireless location area network connection, and a Global Positioning System (GPS) based on the likelihood that an event of interest will occur within the environment.

[0023] In some examples, adjusting the one or more different settings may include turning off or running at least one of an active depth transmitter, an audio algorithm, and a location service.

[0024] In some examples, the one or more detection events may include detection of at least one of a face depicted by the image, a hand gesture depicted by the image, an emotion depicted by the image, a scene depicted by the image, one or more people depicted by the image, an animal depicted by the image, a machine readable code depicted by the image, an infrared light depicted by the image, a two-dimensional surface depicted by the image, and text depicted by the image. In some cases, the keyframes are part of the event data at the electronic device, and the event data includes multiple keyframes.

[0025] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include determining a likelihood that the event of interest will occur within the environment based on the match, and adjusting one or more settings of the image capture device further based on the likelihood that the event of interest will occur within the environment.

[0026] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include determining the likelihood of an event further based on data from a non-image capture device of the electronic device, or adjusting one or more settings of the image capture device further based on data from a non-image capture device of the electronic device.

[0027] In some aspects, each of the devices described above may be part of or include a mobile device, a smart or connected device, a camera system, and / or an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device). In some examples, the device may include or be part of a vehicle, a mobile device (e.g., a mobile phone or a so-called "smartphone" or other mobile device), a wearable device, a personal computer, a laptop computer, a tablet computer, a server computer, a robotics device or system, an aviation system, or other device. In some aspects, the device includes an image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, the device includes one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device includes one or more speakers, one or more light emitting devices, and / or one or more microphones. In some aspects, the devices described above may include one or more sensors. In some cases, one or more sensors may be used to determine the location of the device, the state of the device (e.g., tracking state, operating state, temperature, humidity level, and / or other state), and / or for other purposes.

[0028] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0029] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0030] Illustrative examples of the present application are described in detail below with reference to the following figures: [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 illustrates an example of an electronic device that may be used to determine event data and control one or more components and / or operations of the electronic device based on the event data, in accordance with some examples of the present disclosure. [Figure 2A] FIG. 1 illustrates an example system process for mapping events associated with an environment and controlling device settings based on the mapped events, according to some examples of the disclosure. [Figure 2B] FIG. 1 illustrates an example system process for mapping events associated with an environment and controlling device settings based on the mapped events, according to some examples of the disclosure. [Figure 3A] FIG. 1 illustrates an example process for updating an event map, according to some examples of the present disclosure. [Figure 3B] FIG. 1 illustrates an example process for updating an event map, according to some examples of the present disclosure. [Figure 4] FIG. 11 illustrates example settings adjusted at different times based on matches between features in captured frames and features in key frames of event data, in accordance with some examples of the present disclosure. [Diagram 5] 1A-1C are diagrams illustrating an example of different power states that an electronic device may be in when in different scenes, according to some examples of the present disclosure. [Figure 6] 1 is a flowchart illustrating an example process for adjusting camera settings based on event data, according to some examples of the present disclosure. [Figure 7] FIG. 2 illustrates an example of a computing device architecture, according to some examples of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Specific aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting.

[0033] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0034] An electronic device (e.g., a mobile phone, a wearable device (e.g., a smart watch, a smart bracelet, smart glasses, etc.), a tablet computer, an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, etc.), a connected device, a laptop computer, etc.) can implement a camera for detecting and / or recognizing events of interest. For example, an electronic device can implement a camera, which can operate in a low power mode and / or can operate as a lower power camera (e.g., with lower power than the capacity of that camera and / or the capacity of any other camera on the electronic device) and can detect and / or recognize events of interest on-demand, on-going, or periodically. In some examples, a low power camera can include a camera that operates in a reduced or lower power mode / consumption (e.g., compared to another camera having a power mode / consumption capability of that camera and / or a higher power mode / consumption capability). In some cases, a low power camera may employ low power settings (e.g., low power mode, low power operation, low power hardware, low power camera pipeline, etc.) to enable sustained imaging with limited or reduced power consumption compared to other cameras and / or camera pipelines, such as a main camera and / or main camera pipeline.Low power settings employed by a low power camera may include, for example, but are not limited to, lower resolution, a smaller quantity of image sensors (and / or image sensor(s) having lower power consumption than other image sensors on the electronic device), lower frame rate, on-chip SRAM rather than dynamic random-access memory (DRAM) which may generally consume more power than static random-access memory (SRAM), island voltage rails, oscillators for lock sourcing (e.g., rather than phase lock loops (PLLs) which may have higher power consumption), and / or other hardware / software components / settings that result in lower power consumption.

[0035] As described above, cameras can be used to detect events of interest. Examples of events of interest can include gestures (e.g., hand gestures, etc.), actions (e.g., actions by a device, person, and / or animal, etc.), the presence or occurrence of one or more objects, etc. Objects associated with events of interest can include and / or refer to, for example, but not limited to, a face, a hand, one or more fingers, a part of a human body, a code (e.g., a Quick Response (QR) code, a barcode, etc.), a document, a scene or environment, a link, a machine-readable code, etc. Low-power cameras can implement low-power hardware and / or energy-efficient image processing software used to detect events of interest. Low-power cameras can remain on or "wake up" to monitor motion and / or objects in a scene and detect events in the scene while using less battery power than other devices, such as high-power / high-resolution cameras.

[0036] For example, a camera can monitor motion and / or activity in a scene to discover objects. In some examples, the camera can employ a low power setting for lower or limited power consumption as compared to a camera employing a high power setting or another camera, as described above. By way of example, an XR device can implement a camera that periodically discovers an XR controller and / or other tracked objects, a mobile phone can implement a camera that periodically checks for codes (e.g., QR codes) or documents to be scanned, a smart home assistant can implement a camera that periodically checks for the presence of a user, etc. Upon discovering any object, the camera can trigger one or more actions, such as, for example, object detection, object recognition, authentication (e.g., facial recognition, etc.), and / or image processing tasks, among other actions. In some cases, the camera can "wake up" other devices and / or components, such as other cameras, sensors, processing hardware, etc.

[0037] The power consumption of a camera employing a low power setting may be relatively low compared to a camera or another camera employing a higher power setting (e.g., higher resolution for the low power setting, a larger amount of image sensor for the low power setting (and / or image sensor(s) having higher power consumption than other image sensors on the device), a higher frame rate for the low power setting, DRAM instead of on-chip SRAM, PLL instead of ring oscillator, etc.) compared to the low power setting, and the power consumption may also increase as the occurrence of events of interest increases. Furthermore, in some use cases, there are latency goals and / or requirements that dictate the use of higher frame rates, higher resolution, a greater number of camera sensors, etc., which may further increase the power consumption of the camera. For example, in some cases, when detecting a QR code, the camera may use a higher frame rate to register the QR code faster and reduce the latency in registration. Thus, over time, the amount of power consumed by the camera may increase, which may affect the overall power consumption in the device. In some cases, as the number of false positive events increases, the low power camera may consume an even more unnecessary amount of power.

[0038] A camera may be more likely to encounter an event of particular interest in a particular location / environment. For example, a user of an XR device may typically play an augmented reality game in a particular room of a house. The XR device may implement a camera that discovers a controller used with the XR device to play a virtual reality game. Thus, when the XR device is in that room, the camera of the XR device may consume more power because it will encounter frequent detection events in that environment. As another example, a user may typically scan a QR code on a restaurant menu when ordering delivery from the kitchen. Thus, a camera implemented by the user's mobile device to discover the QR code may cause higher power consumption when the mobile device is in the kitchen (e.g., unlike when the mobile device is in one or more other environments) because the camera will encounter more frequent detection events in that environment.

[0039] As explained above, different environments may trigger higher or lower power consumption of the camera than other environments, and certain use cases may have higher or lower latency requirements that may be more or less frequent in certain environments. This may lead to unnecessary and / or excessive power consumption in some cases and environments, as well as excessive or insufficient performance / device settings (e.g., frame rate, resolution, number of sensors and frequency of use, etc.) in some cases and environments.

[0040] Systems, apparatus, methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and techniques") for camera setting adjustment based on event data are described herein. In some examples, a camera may implement a process that detects an event in an environment and associates the detected event with the environment and / or a location associated with the environment. The process may use the association of the environment / location with the detected event to determine the likelihood of an event of interest in a particular environment / location. The process may use the likelihood information to adjust the camera settings to reduce unnecessary power consumption and / or adjust performance settings according to more likely / more expected latency requirements. For example, when the camera is in an environment where there is a higher likelihood of a detection event of interest (e.g., detection of an event of interest) occurring, the camera may increase one or more settings (e.g., frame rate, resolution, number of image sensors activated, power mode, etc.) to reduce the latency of the camera. As another example, when the camera is in an environment where a target detection event is less likely to occur, the camera may lower one or more settings (e.g., frame rate, resolution, number of activated image sensors, power mode, etc.) to reduce the camera's power consumption.

[0041] In some aspects, an electronic device having a camera (e.g., an XR device, a mobile device, a connected device, a tablet computer, a laptop computer, a smart wearable device, etc.) can conserve power and improve camera performance by adjusting camera settings (e.g., frame rate, resolution, number of image sensors activated, power mode, etc.) based on event data identifying occurrences of events of interest in one or more environments (e.g., an event map relating one or more environments (and / or relevant regions / portions) to previously detected events of interest, a classification map, classification data, one or more key frames, features extracted from the frames, event statistics, historical data, etc.). In some examples, to generate the event data (e.g., the event map, extracted features, one or more key frames, classification data, event statistics, etc.) used to adjust the camera settings, the electronic device can implement a feature extractor that advantageously extracts visual features from incoming frames captured by the camera. The feature extractor may implement detectors and / or algorithms, examples of which include scale-invariant feature transform (SIFT), speeded up robust feature (SURF), Oriented FAST and rotated BRIEF (ORB), and / or any other detector / algorithm. To generate event data (e.g., event maps, extracted features, classification data, etc.) used to adjust camera settings, the electronic device may also implement a keyframe matcher that compares features of incoming camera frames with features of keyframes in the event data.

[0042] In some examples, the key frames in the event data may include key frames created based on camera frames associated with a detected event of interest. The electronic device may implement a mapper that determines whether to create a new key frame (or replace an existing key frame) associated with the event and records (or updates) an event count associated with that key frame in the event data. The event data may include a number of key frames corresponding to locations / environments where camera events were observed, along with event counts associated with those key frames. The controller may use the map to adjust one or more settings of the camera (e.g., frame rate, resolution, power mode, binning mode, imaging mode, number of image sensors activated, etc.) based on a match between an incoming camera frame and the key frames in the map.

[0043] In some cases, the event data may include a dictionary or data structure having entries including visual features of the keyframes and the number of occurrences of the camera events of interest that match each of the keyframes (and / or the number of matches with the keyframes). For example, the event data may include a dictionary having entries indicating that a number n of face detection events (e.g., the number n of previous detections of a face) are associated with one or more visual features corresponding to keyframe x. A total count for each event (e.g., for the recorded keyframes) may be calculated by summing the counts of the events of interest across keyframes that relate to a particular environment and / or region / portion of the environment associated with the event of interest. The total count for an event may be used to determine the prior probability of that event for a given keyframe associated with the event. For example, the total count (or number of matches with keyframes x and y) of detection events (e.g., previous detections of the event of interest) for keyframes x and y may be used to determine that a certain percentage of detection events are associated with keyframe x and a certain percentage of detection events are associated with keyframe y. When the camera is within an environment associated with a keyframe in the mapper, this probability can be used to adjust the camera settings (e.g., based on a match between a camera frame captured in that environment and a keyframe in the mapper associated with that environment).

[0044] The mapper may determine whether to create a new entry in the map based on one or more factors. For example, in some cases, the mapper may determine whether to create a new entry in the map depending on whether the current frame matches an existing key frame in the map, whether a camera event of interest is detected in the current frame, the time since the last key frame was created, the time since the last key frame was matched, etc. In some cases, the mapper may employ a periodic culling process to remove map entries with low event likelihoods (e.g., map entries with event likelihoods below a threshold). In some examples, the periodic culling process may be adaptive based on one or more factors, such as resource availability (e.g., available memory, etc.). For example, if the amount of available memory on the device is above a threshold, the mapper may not perform (or may skip or delay) the culling process, even if there are one or more map entries with low event likelihoods (e.g., likelihoods below a threshold). As another example, if the amount of available memory on a device falls below a threshold, the mapper may perform a culling process to remove one or more map entries, even if such entries have an event likelihood above a certain threshold that would cause the mapper to retain such entries when the amount of available memory does not fall below the threshold.

[0045] The controller can adjust one or more settings of the camera (e.g., frame rate, resolution, power mode, number of image sensors activated, binning mode, imaging mode, etc.) based on the camera event prior probability for the current (or most recently within a threshold period) matched keyframe. In some examples, the controller can incorporate a configurable frame rate for each camera event of interest. In some examples, whenever the mapper indicates a matched keyframe, the controller can set the camera's frame rate to a frame rate equal to the configurable frame rate multiplied by the prior probability of the matched keyframe entry in the map. In some examples, the camera can maintain this frame rate for a configurable period or until the next matched keyframe. In some cases, when / if there is no current (or most recently within a threshold period) matched keyframe, the camera can run at a default frame rate, such as a lower frame rate that can result in lower power consumption during periods of unlikely detection events.

[0046] An electronic device (and / or a camera on an electronic device) can monitor (and implement the systems and techniques described herein for) various types of events. Non-limiting examples of detected events may include face detection, scene detection (e.g., sunset, document scanning, etc.), human group detection, animal / pet detection, code (e.g., QR code, barcode, etc.) detection, infrared LED detection (e.g., similar to a 6 degrees of freedom (DOF) motion tracker), 2D plane detection, text detection, device detection (e.g., detection of controllers, screens, gadgets, computing devices, etc.), gesture detection (e.g., smile detection, emotion detection, hand waving, pointing, etc.), and the like.

[0047] In some cases, the mapper can employ a periodic decay normalization process where event counts are proportionally reduced across map entries by a configurable amount. This can keep the count values ​​numerically bounded, allowing for more rapid adjustment of the prior distribution as event / keyframe correlations change.

[0048] In some cases, the camera may implement a default setting, such as a frame rate, that decays. For example, a reduced frame rate may be associated with increased event detection latency (e.g., a higher frame rate may result in less latency). For a default frame rate (e.g., a frame rate when no key frames are matched), the controller may default the camera's frame rate to a lower frame rate (e.g., a frame rate that may result in longer detection latency) when the map contains fewer recorded events (e.g., below a threshold). In such a case, the controller may implement a default frame rate that starts at a high frame rate but decays as more events are added to the map (e.g., as the device learns which locations / environments are most associated with events of interest).

[0049] In some cases, the electronic device can use non-camera events for mapper decision making. For example, the electronic device can use non-camera events to create new keyframes, cull / delete old keyframes, etc. In some cases, the controller can adjust non-camera workloads / resources based on camera keyframes and / or non-camera events. For example, the controller can implement audio algorithms (e.g., beamforming, etc.) that are adjusted by camera keyframes and / or audio-based presence detection. As another example, the controller can implement location services (e.g., Global Navigation Satellite System (GNSS), Wi-Fi, etc.), application data (e.g., data from one or more applications on the electronic device, such as an XR application), one or more user inputs, and / or other data adjusted by use of camera keyframes and / or location services.

[0050] Various aspects of the application are described with respect to the figures.

[0051] 1 illustrates an example of an electronic device 100 that may be used to map events and control one or more components and / or operations of the electronic device 100 based on the mapped events, in accordance with some examples of the present disclosure. In some examples, the electronic device 100 may include an electronic device configured to provide one or more functions, such as, for example, imaging functions, extended reality (XR) functions (e.g., locating / tracking, detection, classification, mapping, content rendering, etc.), image processing functions, device management and / or control functions, gaming functions, autonomous driving or navigation functions, computer vision functions, robotics functions, automation, computer vision, etc.

[0052] For example, in some cases, electronic device 100 can be an XR device (e.g., a head-mounted display, a head-up display device, smart glasses, etc.) configured to detect, locate, and map the location of the XR device, provide XR functionality, and map events as described herein to control one or more operations / states of the XR device. In some cases, electronic device 100 can implement one or more applications, such as, for example, but not limited to, an XR application, an application for managing and / or controlling components and / or operations of electronic device 100, a smart home application, a video game application, a device control application, an autonomous driving application, a navigation application, a productivity application, a social media application, a communication application, a modeling application, a media application, an e-commerce application, a browser application, a design application, a map application, and / or any other application.

[0053] In one illustrative example shown in FIG. 1, electronic device 100 may include one or more image sensors, such as image sensor 102 and image sensor 104, audio sensor 106 (e.g., ultrasonic sensor, microphone, etc.), inertial measurement unit (IMU) 108, and one or more computational components 110. In some cases, electronic device 100 may optionally include one or more other / additional sensors, such as, for example, but not limited to, radar, light detection and ranging (LIDAR) sensors, touch sensors, pressure sensors (e.g., air pressure sensors and / or any other pressure sensors), gyroscopes, accelerometers, magnetometers, and / or any other sensors. In some examples, electronic device 100 may include additional components, such as, for example, light-emitting diode (LED) devices, storage devices, caches, communication interfaces, displays, memory devices, etc. An exemplary architecture and exemplary hardware components that may be implemented by electronic device 100 are further described below with respect to FIG. 7.

[0054] Electronic device 100 may be part of or implemented by a single computing device or multiple computing devices. In some examples, electronic device 100 may be part of electronic device(s), such as a camera system (e.g., digital camera, IP camera, video camera, security camera, etc.), a telephone system (e.g., smartphone, cellular phone, conferencing system, etc.), a laptop or notebook computer, a tablet computer, a set-top box, a smart television, a display device, a game console, an XR device such as an HMD, a drone, an in-vehicle computer, an IoT (Internet of Things) device, a smart wearable device, or any other suitable electronic device(s).

[0055] In some implementations, the image sensor 102, the image sensor 104, the audio sensor 106, the IMU 108, and / or one or more of the computational components 110 may be part of the same computing device. For example, in some cases, the image sensor 102, the image sensor 104, the audio sensor 106, the IMU 108, and / or one or more of the computational components 110 may be integrated with or into a camera system, a smartphone, a laptop, a tablet computer, a smart wearable device, an XR device such as an HMD, an IoT device, a gaming system, and / or any other computing device. In other implementations, the image sensor 102, the image sensor 104, the audio sensor 106, the IMU 108, and / or one or more of the computational components 110 may be part of or implemented by two or more separate computing devices.

[0056] The one or more computational components 110 of the electronic device 100 may include, for example, without limitation, a central processing unit (CPU) 112, a graphics processing unit (GPU) 114, a digital signal processor (DSP) 116, and / or an image signal processor (ISP) 118. In some examples, the electronic device 100 may include other processors, such as, for example, a computer vision (CV) processor, a neural network processor (NNP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or the like. The electronic device 100 can use one or more computational components 110 to perform various computing operations, such as, for example, extended reality operations (e.g., tracking, localization, object detection, classification, pose estimation, mapping, content anchoring, content rendering, etc.), device control operations, image / video processing, graphics rendering, event mapping, machine learning, data processing, modeling, calculations, computer vision, and / or any other operations.

[0057] In some cases, one or more of the computing components 110 may include other electronic circuitry or hardware, computer software, firmware, or any combination thereof for performing any of the various operations described herein. In some examples, one or more of the computing components 110 may include more or fewer computing components than those shown in Figure 1. Additionally, CPU 112, GPU 114, DSP 116, and ISP 118 are merely illustrative examples of computing components provided for purposes of explanation.

[0058] Image sensor 102 and / or image sensor 104 may include any image and / or video sensor or capture device, such as, for example, a digital camera sensor, a video camera sensor, a smartphone camera sensor, an image / video capture device on an electronic device such as a television or computer, a camera, etc. In some cases, image sensor 102 and / or image sensor 104 may be part of a camera or computing device, such as, for example, a digital camera, a video camera, an IP camera, a smartphone, a smart television, a gaming system, etc. Additionally, in some cases, image sensor 102 and image sensor 104 may include multiple image sensors, such as rear and front sensor devices, and may be part of a dual camera or other multi-camera assembly (e.g., including two cameras, three cameras, four cameras, or other number of cameras).

[0059] In some examples, image sensor 102 may be part of a camera, such as a camera that implements or is capable of implementing a low power camera setting as described above, and image sensor 104 may be part of a camera, such as a camera that implements or is capable of implementing a high power camera setting (e.g., compared to a camera associated with image sensor 102). In some examples, the camera associated with image sensor 102 may implement lower power hardware (e.g., compared to a camera associated with image sensor 104) and / or more energy efficient image processing software (e.g., compared to a camera associated with image sensor 104) used to detect events and / or process captured image data. In some cases, the camera may implement lower power settings and / or power modes than the camera associated with image sensor 104, such as, for example, a lower frame rate, a lower resolution, a fewer number of image sensors, a low power mode, a low power imaging mode, etc. In some examples, the camera may implement fewer and / or lower power image sensors than higher power cameras, may use low power memory such as on-chip static random access memory (SRAM) rather than dynamic random access memory (DRAM), may use island voltage rails to reduce leakage, may use ring oscillators rather than phase locked loops (PLLs) as clock sources, and / or may use other low power processing hardware / components.

[0060] In some cases, the camera associated with image sensor 102 and / or image sensor 104 can remain on or "wake up" to monitor and / or detect motion and / or events in a scene while using less battery power than other devices, such as higher power / resolution cameras. For example, the camera associated with image sensor 102 can monitor or wake up continuously (e.g., by a proximity sensor or by waking up periodically) to monitor motion and / or activity in a scene to discover objects in the scene. In some cases, upon discovering an event, the camera can trigger one or more actions, such as, for example, object detection, object recognition, face recognition, image processing tasks, among other actions. In some cases, the camera associated with image sensor 102 and / or image sensor 104 can also "wake up" other devices, such as other sensors, processing hardware, etc.

[0061] In some examples, each of the image sensors 102 and 104 can capture image data, generate frames based on the image data, and / or provide image data or frames to one or more computing components 110 for processing. A frame can include a video frame of a video sequence or a still image. A frame can include an array of pixels representing a scene. For example, a frame can be a Red-Green-Blue (RGB) frame with red, green, and blue color components per pixel, a Luminance, Red Difference, Blue Difference (YCbCr) frame with one Luminance component and two Chrominance (Color) components (Red Difference and Blue Difference) per pixel, or any other suitable type of color or monochrome image.

[0062] In some examples, one or more of the computational components 110 can use data from the image sensor 102, the image sensor 104, the audio sensor 106, the IMU 108, and / or any other sensors and / or components to perform image / video processing, event mapping, XR processing, device management / control, and / or other operations described herein. For example, in some cases, the one or more computational components 110 can perform event mapping, device control / management, tracking, localization, object detection, object classification, pose estimation, shape estimation, scene mapping, content anchoring, content rendering, image processing, modeling, content generation, gesture detection, gesture recognition, and / or other operations based on data from the image sensor 102, the image sensor 104, the audio sensor 106, the IMU 108, and / or any other components. In some examples, the one or more computational components 110 can use data from the image sensor 102, the image sensor 104, the audio sensor 106, the IMU 108, and / or any other components to generate event data (e.g., an event map correlating detected events to particular environments and / or regions / portions of an environment) and adjust the state (e.g., power modes, settings, etc.) and / or operation of one or more components, such as, for example, the image sensor 102, the image sensor 104, the audio sensor 106, the IMU 108, the one or more computational components 110, and / or any other components of the electronic device 100. In some examples, the one or more computational components 110 can detect and map events in a scene and / or control the operation / state of the electronic device 100 (and / or one or more components thereof) based on the data from the image sensor 102, the image sensor 104, the audio sensor 106, the IMU 108, and / or any other components.

[0063] In some examples, one or more of the computational components 110 may implement one or more software engines and / or algorithms, such as, for example, feature extractor 120, keyframe matcher 122, mapper 124, and controller 126, as described herein. In some cases, one or more of the computational components 110 may implement one or more additional components and / or algorithms, such as machine learning model(s), computer vision algorithm(s), neural network(s), and / or any other algorithms and / or components.

[0064] In some examples, feature extractor 120 may extract visual features from one or more frames captured by a camera device, such as a camera device associated with image sensor 102. Feature extractor 120 may implement detectors and / or algorithms for extracting visual features, such as, for example, but not limited to, Scale Invariant Feature Transform (SIFT), Speed ​​Up Robust Features (SURF), Oriented FAST and Rotational BRIEF (ORB), and / or any other detector / algorithm.

[0065] In some examples, the keyframe matcher 122 can compare features of one or more frames captured by the camera device (e.g., visual features extracted by the feature extractor 120) to features of keyframes in the event data (e.g., an event map) generated by the mapper 124. In some cases, the event data can include an event map that correlates detected events of interest (and / or associated data such as keyframes, extracted image features, event counts, etc.) with one or more particular environments (and / or portions / areas of particular environments) in which such detected events occurred (and / or were detected). In some examples, the keyframes in the event data can include keyframes created based on frames associated with the detected events of interest. The mapper 124 can determine whether to create a new keyframe (or replace an existing keyframe) associated with the event and record (or update) the event count associated with that keyframe in the event data. The event data can include a number of keyframes corresponding to one or more locations / environments where the events of interest were observed (e.g., detected from one or more frames captured by the camera device) and the event counts associated with those keyframes. The controller 126 can use the event data to adjust one or more settings of the camera device (e.g., frame rate, resolution, power mode, number of image sensors activated, binning mode, imaging mode, etc.) (e.g., one or more settings of the image sensor 102 and / or one or more other hardware and / or software components) based on a match between an incoming frame from the camera device and a key frame in the event data.

[0066] In some cases, the event data may include an event map, classification data, keyframe data, features extracted from the frames, classification map data, event statistics, and / or a dictionary having entries including visual features of the keyframes and the number of occurrences of the detected event of interest that match each of the keyframes (and / or the number of matches with the keyframes). For example, the event data may include a dictionary having an entry indicating that a number n of face detection events are associated with one or more visual features corresponding to keyframe x. In some examples, the mapper 124 may calculate a count for each event (e.g., for the recorded keyframes). In some cases, the count may include a count and / or an average of occurrences of the event of interest within a particular time period. In some cases, the count may include a total count of occurrences of the event of interest. In some examples, the mapper 124 may determine a total count based on a sum of counts of the event of interest across keyframes associated with the environment(s) and / or region / portion of the environment(s) corresponding to the event. The count for an event may be used to determine a prior probability of the event for a given keyframe associated with the event. For example, a count of detection events for keyframes x and y (or the number of matches with keyframes x and y) can be used to determine that a first degree of detection events is associated with keyframe x and a second degree of detection events is associated with keyframe y.

[0067] Controller 126 may use this probability to adjust one or more settings of a camera device (e.g., a camera device associated with image sensor 102) when the camera device is within an environment associated with a key frame in the event data (e.g., based on a match between a frame captured within that environment and a key frame associated with that environment in the event data). In some cases, controller 126 may alternatively or additionally use the probability to adjust one or more settings of another camera device (e.g., a camera device associated with image sensor 104, a camera device that employs and / or has the ability to employ higher power camera settings than the camera device associated with image sensor 102, etc.), processing components, and / or other components of electronic device 100, such as a pipeline, when electronic device 100 is within or is not within an environment associated with a key frame in the event data.

[0068] The mapper 124 may determine whether to create a new entry in the event data based on one or more factors. For example, in some cases, the mapper 124 may determine whether to create a new entry in the event data depending on whether a current frame captured by a camera device (e.g., a camera device associated with the image sensor 102) matches an existing key frame in the event data, whether a camera event of interest is detected in the current frame, the time elapsed since the last key frame was created, the time elapsed since the last key frame was matched, etc. In some cases, the mapper 124 may employ a periodic culling process to remove map entries with lower event likelihoods (e.g., having a likelihood with a probability value below a threshold).

[0069] In some examples, the controller 126 may adjust one or more settings of a camera device (e.g., a camera device associated with the image sensor 102), such as, but not limited to, increasing and decreasing a frame rate, a resolution, a power mode, an imaging mode, a number of image sensors activated to capture one or more images associated with an event of interest, processing actions and / or processing pipelines for processing the captured images and / or detecting an event within the captured images, etc. For example, in some cases, the electronic device 100 may be statistically more likely to encounter a particular event of interest in a particular environment. In some cases, to avoid wasting unnecessary power when the electronic device 100 is located in an environment where the likelihood of the electronic device 100 encountering an event of interest is lower, the controller 126 may turn off the camera device or adjust one or more settings of the camera device to reduce power usage by the camera device when the electronic device 100 is in an environment where the likelihood of the electronic device 100 encountering an event of interest is lower. When the electronic device 100 is located in an environment in which the electronic device 100 is more likely to encounter an event of interest, the controller 126 may turn on the camera device or adjust one or more settings of the camera device to increase power usage by the camera device and / or performance of the camera device when the electronic device 100 is in an environment in which the electronic device 100 is more likely to encounter an event of interest.

[0070] In some cases, the controller 126 can adjust one or more settings of the camera device(s) (e.g., image sensor 102, image sensor 104) based on the camera event prior probability for a currently (or recently within a threshold period) matched keyframe. In some examples, the controller 126 can incorporate a configurable frame rate for each camera event of interest. In some examples, when the mapper 124 indicates a matched keyframe, the controller 126 can set the frame rate of the camera device to a frame rate equal to the configurable frame rate multiplied by the prior probability of a matched keyframe entry in the event data. In some examples, the camera device can maintain this frame rate for a configurable period or until the next matched keyframe. In some cases, when / if there is no currently (or recently within a threshold period) matched keyframe, the camera device can run a default frame rate, such as a lower frame rate resulting in lower power consumption during periods of less likely detection events.

[0071] In some examples, electronic device 100 (and / or controller 126 thereon) may monitor (and implement techniques described herein to) various types of events. Non-limiting examples of detected events may include face detection, scene detection (e.g., sunset, room, etc.), human group detection, animal / pet detection, code (e.g., QR code, etc.) detection, document detection, infrared LED detection (e.g., similar to a six degree of freedom (6DOF) motion tracker), plane detection, text detection, device detection (e.g., controller, screen, gadget, etc.), gesture detection (e.g., smile detection, emotion detection, hand gestures, etc.).

[0072] In some cases, the mapper 124 can employ a periodic decay normalization process in which event counts are reduced by a configurable amount across map entries, which can keep the count values ​​numerically bounded and allow for more rapid adjustment of the prior distribution as event / keyframe correlations change.

[0073] In some cases, the camera device (e.g., image sensor 102) may implement a default setting, such as a frame rate, that decays. For example, a reduced frame rate may be associated with increased event detection latency (e.g., a higher frame rate may result in less latency). For a default frame rate (e.g., a frame rate when no key frames are matched), the controller 126 may default the frame rate of the camera device to a lower frame rate (e.g., a frame rate that may result in a longer detection latency) when the event data includes a smaller number of recorded events (e.g., below a threshold). In such a case, the controller 126 may implement a default frame rate that starts at a threshold level but decays as more events are added to the event data (e.g., as the electronic device learns which locations / environments are most associated with events of interest).

[0074] In some cases, the mapper 124 can use non-camera events for mapper decision making. For example, the mapper 124 can use non-camera events to create new keyframes, cull / delete old keyframes, etc. In some cases, the controller 126 can adjust non-camera workloads / resources based on keyframes and / or non-camera events. For example, the controller 126 can implement one or more audio algorithms (e.g., beamforming, etc.) that are adjusted by the camera device keyframes and / or audio-based presence detection. As another example, the controller 126 can implement location services (e.g., Global Navigation Satellite System (GNSS), Wi-Fi, etc.), application data, user input, and / or other data adjusted by the use of the camera device keyframes and / or location services.

[0075] In some cases, the IMU 108 can detect acceleration, angular velocity, and / or orientation of the electronic device 100 and generate measurements based on the detected acceleration. In some cases, the IMU 108 can detect and measure the orientation, linear velocity, and / or angular velocity of the electronic device 100. For example, the IMU 108 can measure the movement and / or pitch, roll, yaw of the electronic device 100. In some examples, the electronic device 100 can calculate the orientation of the electronic device 100 in three-dimensional space using measurements obtained by the IMU 108 and / or data from one or more of the image sensor 102, the image sensor 104, the audio sensor 106, etc. In some cases, the electronic device 100 can additionally or alternatively use sensor data from the image sensor 102, the image sensor 104, the audio sensor 106, and / or any other sensor to perform tracking, pose estimation, mapping, generating event data entries, and / or other operations described herein.

[0076] The components shown in FIG. 1 for electronic device 100 are illustrative examples provided for purposes of explanation. In other examples, electronic device 100 may include more or fewer components than those shown in FIG. 1. Although electronic device 100 is shown as including certain components, one of ordinary skill in the art will understand that electronic device 100 may include more or fewer components than those shown in FIG. 1. For example, electronic device 100 may include, in some examples, one or more memory devices (e.g., RAM, read only memory (ROM), cache, etc.), one or more networking interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices, cache, storage devices, and / or other hardware or processing devices not shown in FIG. 1. Illustrative examples of computing devices and hardware components that may be implemented by electronic device 100 are further described below with respect to FIG. 7.

[0077] 2A illustrates an example system process 200 for mapping events associated with an environment and controlling settings (e.g., power state, operation, parameters, etc.) of a camera device based on the mapped events. In this example, the image sensor 102 can capture frames 210 of scenes and / or events in an environment in which the image sensor 102 is located. In some examples, the image sensor 102 can monitor the environment and / or look for events of interest in the environment and capture frames of any events of interest in the environment. Events of interest can include, for example, but are not limited to, gestures (e.g., hand gestures, etc.), emotions (e.g., smiles, etc.), activities or actions (e.g., by a person, device, animal, etc.), occurrence or presence of an object, etc. An object associated with an event of interest may include, represent, and / or point to, for example, without limitation, a face, a scene (e.g., a sunset, a park, a room, etc.), a person, a group of people, an animal, a document, a code (e.g., a QR code, a barcode, etc.) on an object (e.g., a device, a document, a display, a structure such as a door or wall, a sign, etc.), a light, a pattern, a link on an object, a plane in physical space (e.g., a plane on a surface, etc.), text, an infrared (IR) light emitting diode (LED) detection, and / or any other object.

[0078] The electronic device 100 can perform one or more image processing operations on the frames 210 using the image processing module 212. In some examples, the one or more image processing operations can include object detection to detect one or more camera event detection triggers 214 based on the frames 210. For example, the image processing module 212 can extract features from the frames and use the extracted features for object detection. The image processing module 212 can detect one or more camera event detection triggers 214 based on the extracted features. The one or more camera event detection triggers 214 can include one or more events of interest, as described above. In some examples, the one or more image processing operations can include a camera processing pipeline associated with the image sensor 102. In some cases, the one or more image processing operations can detect and / or recognize one or more camera event detection triggers 214. The image processing module 212 can provide the one or more camera event detection triggers 214 to the feature extractor 120, the mapper 124, the controller 126, and / or the application 204 on the electronic device 100. The feature extractor 120, the mapper 124, the controller 126, and / or the application 204 can use the one or more camera event detection triggers 214 to perform one or more actions described herein, such as, for example, an application action, a camera setting adjustment, object detection, object recognition, etc. For example, the one or more camera event detection triggers 214 can be configured to trigger one or more actions by the feature extractor 120, the mapper 124, the controller 126, and / or the application 204, as described herein.

[0079] In some examples, the application 204 may include any application on the electronic device 100 that may use information about events detected in the environment. For example, the application 204 may include an authentication application (e.g., a face recognition application, etc.), an XR application, a navigation application, an application for ordering or purchasing an item, a video game application, a photo application, a device management application, a web application, a communication application (e.g., a messaging application, a video and / or voice application, etc.), a media playback application, a social media network application, a browser application, a scanning application, etc. The application 204 may use one or more camera event detection triggers 214 to perform one or more actions. For example, if the application 204 is an XR application, the application 204 may use one or more camera event detection triggers 214 to discover events, such as devices (e.g., controllers or other input devices, etc.), hands, boundaries, people, etc., in an environment used by the application 204. As another example, the application 204 may use one or more camera event detection triggers 214 to scan a document, link, or code and perform an action based on the document, link, or code. As yet another example, if the application 204 is a smart home assistant application, the application 204 can use one or more camera event detection triggers 214 to discover the presence of people and / or objects and trigger operations / actions in smart home devices.

[0080] Additionally, the feature extractor 120 may analyze the frames 210 to extract visual features in the frames 210. In some examples, the feature extractor 120 may perform object recognition to extract visual features in the frames 210 and classify events associated with the extracted features. In some cases, the feature extractor 120 may execute algorithms to extract visual features from the frames 210 and determine a descriptor(s) for the extracted features. Non-limiting examples of feature extractor / detector algorithms may include SIFT, SURF, ORB, etc.

[0081] The feature extractor 120 may provide the features extracted from the frames 210 and the associated descriptor(s) to the keyframe matcher 122 and the mapper 124. The associated descriptor(s) may identify and / or describe the features extracted from the frames 210 and / or the events detected from the extracted features. In some examples, the associated descriptor(s) may include a tag, a label, an identifier, and / or any other descriptor.

[0082] The keyframe matcher 122 can use the features and / or descriptor(s) from the feature extractor 120 to determine whether (or determine the likelihood that) the electronic device 100 is located in an environment (or an area / location in the environment) where an event of interest was previously detected. In some examples, the keyframe matcher 122 can use the event data 202 including the keyframes to determine whether (or determine the likelihood that) the electronic device 100 is located in an environment (or an area / location in the environment) where an event of interest was previously detected. In some examples, the event data 202 can include keyframes corresponding to frames capturing the detected event of interest. In some cases, the event data 202 can include a number of keyframes corresponding to locations / environments where the event of interest was observed (e.g., detected from one or more frames acquired by the image sensor 102 or the image sensor 104) and event counts associated with those keyframes.

[0083] In some cases, the event data 202 can include an event map, an event entry, classification data, one or more key frames, a classification map, event statistics, extracted features, feature descriptors, and / or any other data. In some examples, the event data 202 can include a dictionary having entries including visual features of key frames and the number of occurrences (e.g., event counts) of detected events of interest that match each of the key frames (and / or the number of matches with key frames). For example, the event data 202 can include a dictionary having entries indicating that a number n of QR code detection events (e.g., a number n of previous detections of a QR code) is associated with one or more visual features corresponding to key frame x. In some cases, the key frame matcher 122 can use a periodic decay normalization process in which the event counts are reduced by a configurable amount across one or more event data entries. For example, the key frame matcher 122 can use a periodic decay normalization process in which the event counts are reduced by a configurable amount across all event data entries. This allows the count values ​​to remain numerically bounded, allowing for faster adjustment of the prior distribution as event / keyframe correlations change.

[0084] In some examples, the key frame matcher 122 can compare the features extracted from the frame 210 to features of key frames in the event data 202. In some examples, the key frame matcher 122 can compare a descriptor(s) of the features extracted from the frame 210 to the descriptors of the features of the key frames in the event data 202. Because the key frames in the event data 202 correspond to an environment or a location within the environment where an event of interest was previously detected, a match between the features (and / or associated descriptors) extracted from the frame 210 and the features (and / or associated descriptors) of the key frames in the event data 202 can indicate that the electronic device 100 is located (or the likelihood that the electronic device 100 is located) in the environment or a location within the environment where an event of interest was previously detected. In some examples, the key frame matcher 122 can correlate the frame 210 and / or the features extracted from the frame 210 to a particular environment (and / or a location / area within the particular environment) based on a match between the features extracted from the frame 210 and the feature key frames in the event data 202. Such a correlation may indicate that the electronic device 100 is located in a particular environment (and / or a location / area within a particular environment). Information regarding the location of the electronic device 100 in an environment or a location within an environment where an event of interest was previously detected can be used to determine the likelihood that the event of interest will be detected again when the electronic device 100 is located in the environment or location within the environment.

[0085] For example, in some cases, the likelihood that the event of interest is observed / detected in an environment may increase or decrease depending on whether the event of interest has been previously observed / detected in the environment and / or depending on the number of times the event of interest has been previously observed / detected in the environment. For example, a determination that the event of interest is frequently observed / detected in a particular room may suggest a higher likelihood that the event of interest will be observed / detected again when the electronic device 100 is in the particular room than a determination that the event of interest has not been previously observed / detected in the particular room. Thus, a determination that the electronic device 100 is located in an environment or location within an environment where the event of interest was previously detected may be used to determine the likelihood that the event of interest will be detected again when the electronic device 100 is located in the environment or location within the environment. As previously mentioned, in some examples, a determination that the electronic device 100 is located in an environment or location where the event of interest was previously detected may be based on a match between features extracted from the frame 210 and features of one or more key frames in the event data that correlate to a particular environment or location.

[0086] As described further herein, the likelihood that an event of interest will be detected again when electronic device 100 is located in an environment or a location within the environment can be used to control device and / or processing settings (e.g., power modes, operation, device configuration, processing configuration, processing pipeline, etc.) to reduce power consumption and improve power conservation when electronic device 100 is located in an environment (or location thereof) associated with a lower likelihood of event detection. Similarly, the likelihood that an event of interest will be detected again when electronic device 100 is located in an environment or a location within the environment can be used to control device and / or processing settings to improve performance and / or operating conditions of electronic device 100 when electronic device 100 is located in an environment (or location thereof) associated with a higher likelihood of event detection, such as improving event detection performance, imaging and / or image processing performance (e.g., image / imaging quality, resolution, scaling, frame rate, etc.).

[0087] The key frame matcher 122 may provide the mapper 124 with the results of the comparison of the features extracted from the frame 210 with the features of the key frames in the event data 202. For example, the key frame matcher 122 may provide an indication to the mapper 124 that the features extracted from the frame 210 match the features of the key frames in the event data 202 or do not match the features of any key frames in the event data 202. The mapper 124 may use the information from the key frame matcher 122 to determine whether to create a new key frame (or replace an existing key frame) associated with the event and record (or update) an event count associated with that key frame. For example, if the features extracted from the frame 210 match the features of a key frame in the event data 202, the mapper 124 may increment the count of detected events associated with that key frame in the event data 202. In some examples, the mapper 124 may record or update an entry with the count of detected events associated with that key frame.

[0088] In some cases, if features extracted from frame 210 do not match features of any keyframe in the event data 202, the mapper 124 may create a new keyframe in the event data 202, which (e.g., the new keyframe) may correspond to frame 210. For example, as described above, the electronic device 100 may previously detect (e.g., via the image processing module 212) one or more camera event detection triggers 214 based on frame 210, which may indicate that an event of interest was detected in frame 210. Thus, if features extracted from frame 210 do not match features of any keyframe in the event data 202, the mapper 124 may add a new keyframe corresponding to frame 210 to the event data 202. The new keyframe may associate features from frame 210 with the detected event of interest and / or the associated environment (and / or its location). The mapper 124 can include an event detection count associated with the new keyframe in the event data 202 and can increment the count each time the new frame (or its features) matches the features associated with the new keyframe.

[0089] Mapper 124 may use a count of detected events associated with a keyframe (e.g., a new keyframe or an existing keyframe) corresponding to frame 210 and features associated with frame 210 to determine or update an event prior probability 216 associated with that keyframe. For example, in some cases mapper 124 may use a total and / or average count of events associated with a keyframe corresponding to frame 210, an associated environment, and / or features associated with frame 210 to determine or update an event prior probability 216 associated with that keyframe. Event prior probability 216 may include a value representing an estimated likelihood / probability of detecting an event of interest in an environment or a location within an environment associated with frame 210 and / or features of frame 210. For example, if the frame 210 is captured from a particular room and the visual features in the frame 210 correspond to a particular room or an area / object within the particular room, the event prior probability 216 may indicate an estimated likelihood of detecting an event of interest when the electronic device 100 is in the particular room (and / or area of ​​the particular room) and / or when the visual features in a frame captured in the particular room (or area of ​​the particular room) match visual features in a key frame in the event data 202 associated with the particular room and / or area / object of the particular room.

[0090] In some examples, the event prior probability 216 can be based at least in part on a count of detected events of interest recorded for a keyframe associated with the event prior probability 216. In some cases, the likelihood / probability value in the event prior probability 216 associated with a keyframe may increase as the count of detected events of interest associated with that keyframe increases. In some cases, the likelihood / probability values ​​in the event prior probability 216 may be further based on one or more other factors, such as, for example, the amount of time between detected target events associated with the key frame, the amount of time since the last detected target event associated with the key frame and / or the last number n of detected target events associated with the key frame, the type and / or characteristics of the detected target event(s) associated with the key frame, the number of frames captured in the environment associated with the key frame that resulted in a positive detection result versus the number of frames captured in that environment associated with the key frame that resulted in a negative detection result, one or more characteristics of the environment associated with the key frame (e.g., the size of the environment, the number or density of potential target events in the environment, common activities taking place in the environment, etc.), the frequency (and / or amount of time of use) of the electronic device 100 in the environment associated with the key frame (e.g., a higher rate of use with fewer positive detection results may be used to reduce the likelihood / probability value, and vice versa), and / or any other factor.

[0091] For example, an increase or decrease in the time between events of interest detected in an environment associated with a key frame can be used to increase or decrease the likelihood / probability value in the event prior probability 216. As another example, an increase or decrease in the number of frames captured in an environment that resulted in a positive detection versus the number of frames captured in the environment that resulted in a negative detection can be used to increase or decrease the likelihood / probability value in the event prior probability 216. As yet another example, the number of events of interest detected in an environment versus the amount of usage of the electronic device 100 in the environment can be used to decrease or increase the likelihood / probability value in the event prior probability 216 (e.g., more usage with fewer events of interest detected can result in a lower likelihood / probability value than more events of interest detected or less usage with the same amount of events of interest detected).

[0092] The mapper 124 can provide the event prior probability 216 associated with the matched keyframe to the controller 126. The controller 126 can use the event prior probability 216 to control / adjust one or more settings associated with the image sensor 102 and / or image processing associated with the image sensor 102. For example, the controller 126 can use the event prior probability 216 to adjust one or more settings to reduce power consumption of the electronic device 100 when the event prior probability 216 indicates a lower likelihood / probability of the event of interest in the current environment of the electronic device 100, or adjust one or more settings to improve performance and / or processing capabilities of the electronic device 100 (e.g., performance of the image sensor 102) when the event prior probability 216 indicates a higher likelihood / probability of the event of interest in the current environment of the electronic device 100.

[0093] By way of example, when the event prior probability 216 indicates a higher likelihood / probability of an event of interest in the current environment of the electronic device 100, the controller 126 may increase the frame rate, resolution, scale factor, image stabilization, power mode, and / or other settings associated with the image sensor 102, may activate additional image sensors, may execute image processing pipelines and / or operations associated with higher performance, complexity, functionality, and / or processing power, may activate or initialize a higher power camera device or main camera device (e.g., the image sensor 104), may turn on an active depth transmitter system, such as a structured light system or flood illuminator, a dual camera system for depth and stereo, or a time-of-flight camera component, etc.

[0094] On the other hand, when the event prior probability 216 indicates a lower likelihood / probability of the event of interest in the current environment of the electronic device 100, the controller 126 may turn off the image sensor 102, may reduce the frame rate, resolution, scale factor, image stabilization, and / or other settings associated with the image sensor 102, may activate fewer image sensors, may execute image processing pipelines and / or operations associated with lower power consumption, performance, complexity, functionality, and / or processing capabilities, may turn off active depth transmitter systems such as structured light systems or flood illuminators, dual camera systems, time-of-flight camera components, etc. In this manner, the controller 126 may enhance power conservation, performance, and / or capabilities of the electronic device 100 and associated components based on the likelihood / probability of the presence / occurrence of the event of interest in the current environment of the electronic device 100.

[0095] In some cases, the controller 126 can also adjust the non-camera settings 220 based on the event prior probability 216 (e.g., based on the likelihood / probability of the presence / occurrence of an event of interest in the current environment of the electronic device 100). For example, the controller 126 can switch on / off, increase / decrease power mode, increase / decrease processing power and / or complexity of one or more components, algorithms, services, etc., such as audio algorithm(s) (e.g., beamforming, etc.), location services (e.g., GNSS or GPS, WIFI, etc.), tracking algorithms, audio devices (e.g., audio sensors 106), non-camera workload, additional processors, etc. (e.g., based on the likelihood / probability of the presence / occurrence of an event of interest in the current environment of the electronic device 100).

[0096] In some cases, electronic device 100 can also leverage non-camera events to map events associated with the environment and control device settings (e.g., power states, operations, parameters, etc.) based on the mapped events. For example, with reference to FIG. 2B , electronic device 100 can use data 234 from non-camera sensors 232 to update event data 202 (e.g., add new keyframes and associated detection counts, update existing keyframes and / or detection counts, remove existing keyframes and / or detection counts) and / or calculate event prior probabilities 236 for matched keyframes.

[0097] The non-camera sensors 232 may include, for example, without limitation, audio sensors (e.g., audio sensor 106), IMUs (e.g., IMU 108), radar, GNSS or GPS sensors / receivers, wireless receivers (e.g., Wi-Fi, cellular, etc.), etc. The data 234 from the non-camera sensors 232 may include, for example, without limitation, information regarding the location / position of the electronic device 100, the distance between the electronic device 100 and one or more objects, the location of one or more objects in the environment, the movement of the electronic device 100, sounds captured in the environment, the time of one or more events, etc.

[0098] In some examples, data 234 from non-camera sensors 232 can be used to supplement data (e.g., key frames and / or associated data) associated with updates to event data 202. For example, data 234 from non-camera sensors 232 can be used to add a timestamp of an event associated with a key frame that was added, updated, or removed in event data 202, can indicate a location / position of a detected event associated with a key frame, can indicate a location / position of electronic device 100 before, during, and / or after a detected event, can indicate movement of electronic device 100 during a detected event, can indicate a proximity of electronic device 100 to a detected event, can indicate audio features associated with a detected event and / or environment, can indicate one or more characteristics of the environment (e.g., location, geometry, configuration, activity, objects, etc.), etc. The mapper 124 can use the data 234 along with features / descriptors and / or counts associated with keyframes in the event data 202 to help determine a likelihood / probability value in the event prior probability 236 of the matching keyframe, can provide more granular information (e.g., location, activity, movement, time, etc.) about the environment, the electronic device 100, and / or the detected event associated with the matching keyframe, can cull / prune old keyframes, can determine whether to add, update, or remove a keyframe (and / or related information) from the event data 202, can validate the detected event, etc.

[0099] In some cases, controller 126 can also use data 234 from non-camera sensors 232 to determine how or which settings to adjust / adjust, as described above. For example, controller 126 can use data 234 along with event prior probability 236 to determine which settings to adjust (and / or how to adjust) image sensor 102, which settings to adjust (and / or how to adjust) one or more other devices on electronic device 100, which of non-camera settings 220 to adjust (and / or how to adjust), etc. For example, as described above, in some cases, if the event prior probability indicates a higher likelihood that an event of interest will occur within the current environment of electronic device 100, controller 126 can increase the frame rate of image sensor 102 and / or activate a higher power camera device or a main camera device with higher frame rate capabilities (e.g., as compared to the camera device associated with image sensor 102). In this example, if the data 234 indicates a threshold amount of movement associated with a matching key frame and / or detected event of the electronic device 100, the controller 126 may increase the frame rate of the image sensor 102 and / or the high power camera device or main camera device more than if the data 234 indicates that the amount of movement is below the threshold.

[0100] As another example, if the data 234 indicates that the electronic device 100 is approaching a location in the environment in the vicinity of the location of the previous event of interest, the controller 126 may activate a higher power camera device or a main camera device (e.g., a camera device having a higher power capability / setting compared to the camera device associated with the image sensor 102) and / or increase the settings of the image sensor 102 before the electronic device 100 reaches the location of the previous event of interest. Similarly, if the data 234 indicates that the electronic device 100 is moving away from the environment, the controller 126 may adjust one or more settings (e.g., turn off the image sensor 102 or another device, reduce the power mode of the image sensor 102 or another device, reduce processing complexity and / or power consumption, etc.) to reduce power consumption by the electronic device 100 even if the event of interest is determined to have a higher likelihood / probability of occurring in the environment (e.g., based on the event prior probability). As yet another example, if the event prior probability indicates a higher likelihood of an event of interest occurring within the environment of the electronic device 100 and the data 234 indicates a threshold amount of movement by the electronic device 100 and / or one or more objects within the environment, the controller 126 may activate image stabilization settings / operations and / or increase the complexity / performance of the image stabilization settings / operations to ensure better image stabilization of any frames capturing the event of interest within the environment.

[0101] 3A illustrates an example process 300 for updating event data (e.g., event data 202). In this example, at block 302, electronic device 100 may extract features from frames captured by camera devices of electronic device 100 (e.g., a camera device associated with image sensor 102, a camera device associated with image sensor 104).

[0102] At block 304, the electronic device 100 may determine whether an event of interest is detected in the frame based on the extracted features. If an event of interest is not detected in the frame, at block 306, the electronic device 100 does not add a new key frame to the event data. If an event of interest is detected in the frame, then, optionally, at block 308, the electronic device 100 may determine whether a timer has expired since the key frame was created (e.g., added to the event data) by the electronic device 100, and / or at block 312, the electronic device 100 may determine whether a match has been identified between the key frame in the event data and the visual features extracted from the frame.

[0103] The timer may be programmable. In some cases, the timer (e.g., the amount of time set to trigger expiration of the timer) may be determined based on one or more factors, such as, for example, a usage history and / or pattern associated with electronic device 100, a pattern of detection events (e.g., a pattern associated with previous detection of one or more events of interest), a type of event of interest configured to trigger a detection event (e.g., trigger detection of an event of interest), one or more characteristics of one or more environments, etc. In some examples, the timer may be set to prevent a high number of keyframes from being created and / or to prevent keyframes from being created too frequently.

[0104] For example, assume that electronic device 100 detects a QR code within a frame capturing the QR code from a restaurant menu on a refrigerator in the kitchen and creates a keyframe associated with the detected QR code within the restaurant menu on the refrigerator. The detection of the QR code and the creation of the keyframe may indicate that electronic device 100 is in the same room (e.g., the kitchen) as the QR code and is likely to remain in that same room for at least a period of time. In this example, a timer may prevent electronic device 100 from creating additional keyframes for events associated with that environment while electronic device 100 is likely to remain in that environment. Thus, the timer may reduce the amount of keyframes created within a period of time, the power consumption from creating additional keyframes within that period of time, and the resource usage in creating additional keyframes within that period of time.

[0105] If the timer has not expired, process 300 may return to block 306, where electronic device 100 determines not to add a new key frame to the event data. If the timer has expired, at block 310, electronic device 100 may restart or reset the timer. At block 312, electronic device 100 may determine whether features extracted from the frame at block 302 match features of a key frame in the event data. For example, electronic device 100 may compare features and / or associated descriptors extracted from the frame at block 302 with features and / or associated descriptors in a key frame in the event data.

[0106] At block 314, if the electronic device 100 finds a match between the features extracted from the frame at block 302 and the features of a key frame in the event data, the electronic device 100 may increment a count of detected events (e.g., a count of previous detections of one or more events of interest) associated with the matching key frame in the event data. At block 316, if the electronic device 100 does not find a match between the features extracted from the frame at block 302 and the features of any key frame in the event data, the electronic device 100 may create a new entry in the event data for the detected event (e.g., detection of an event of interest) associated with the features extracted from the frame at block 302. In some examples, the new entry may include a key frame that includes the features extracted from the frame at block 302 and an event count indicating the number of occurrences of the event of interest associated with the detected event. In some examples, the new entry may also include a descriptor of the features associated with the key frame.

[0107] In some cases, process 300 may not run a timer and / or may not check whether the timer has expired, as described with respect to blocks 308 and 310. For example, with reference to FIGURE 3B, in some cases, after determining that an event of interest has been detected in block 304, electronic device 100 may proceed to block 312 to determine whether features extracted from the frame in block 302 match features of a key frame in the event data.

[0108] In other cases, process 300 may run a timer, but checking whether the timer has expired may be performed at a different point in the process. For example, in some cases, electronic device 100 may check whether the timer has expired (e.g., as described with respect to block 308) before determining whether the event of interest has been detected (e.g., as described with respect to block 304). In some examples, if the timer has expired, electronic device 100 may restart the timer (e.g., as described with respect to block 310) before determining whether the event of interest has been detected (e.g., as described with respect to block 304) or after determining that the event of interest has been detected.

[0109] 4 illustrates example settings adjusted (e.g., via controller 126) at different times based on matches between features in captured frames and features in keyframes on event data (e.g., event data 202). In this example, the setting adjusted is the frame rate of a camera device (e.g., image sensor 102) of electronic device 100. However, as previously discussed, in other examples, electronic device 100 may (additionally or alternatively) adjust other settings (e.g., via controller 126) of the camera device, another device of electronic device 100, operation of electronic device 100, processing pipeline, etc.

[0110] In some examples, the camera device may execute a default frame rate 402 at time t1. The default frame rate 402 may be any frame rate supported by the camera device. For example, the default frame rate 402 may be the lowest frame rate of the camera device, the highest frame rate of the camera device, or any other frame rate supported by the camera device. In FIG. 4, the default frame rate 402 is lower than the highest frame rate supported by the camera device.

[0111] At time t2, the electronic device 100 finds a matching key frame 410 in the event data (e.g., the event data 202) based on features in the frames captured at the default frame rate 402. The matched key frame 410 may include an event prior probability calculated based at least in part on a count of detected events associated with the key frame 410 (e.g., a count of previous detections of one or more events of interest associated with the key frame 410). In response to finding the matching key frame 410, the electronic device 100 may determine a frame rate 404 (e.g., via the controller 126) and change the frame rate of the camera device from the default frame rate 402 to the frame rate 404. In some examples, the electronic device 100 may determine the frame rate 404 based on an event prior probability associated with the matching key frame 410. For example, the electronic device 100 may multiply a configurable frame rate or a peak frame rate associated with the camera device by a value of the event prior probability. For example, if the value of the event prior probability is 0.75, the electronic device 100 may multiply the configurable frame rate or the peak frame rate by 0.75 to determine the frame rate 404. In this example, the frame rate 404 may be the result of multiplying the configurable frame rate or the peak frame rate by 0.75.

[0112] The peak frame rate may be a configurable frame rate used to determine the frame rate as described above. For example, the peak frame rate may be a frame rate selected from the frame rates supported by the camera device. In some cases, the highest frame rate supported by the camera device may be selected as the peak frame rate.

[0113] The camera device may maintain the frame rate 404 for a configurable period of time or until another keyframe match is found. In Figure 4, at time t3, the electronic device 100 determines that a timer 412 for maintaining the frame rate 404 has expired before another keyframe match. Based on the expiration of the timer 412, the electronic device 100 may change the frame rate of the camera device from the frame rate 404 back to the default frame rate 402.

[0114] At time t4, the electronic device 100 finds a matching key frame 414 in the event data (e.g., the event data 202) based on features in the frames captured at the default frame rate 402. The matching key frame 414 may include an event prior probability, as described above. In response to finding the matching key frame 414, the electronic device 100 may determine a frame rate 406 (e.g., via the controller 126) and change the frame rate of the camera device from the default frame rate 402 to the frame rate 406. The electronic device 100 may determine the frame rate 406 based on the event prior probability associated with the matching key frame 414. In some cases, the frame rate 406 may be the same as the frame rate 404. In other cases, the frame rate 406 may be a higher or lower frame rate than the frame rate 404. For example, in some cases, frame rate 406 may be higher or lower than frame rate 404 depending on the value of the event prior probability associated with the matched keyframe 414 and / or the configurable frame rate or peak frame rate used to calculate frame rate 406.

[0115] At time t5, before the timer expires, the electronic device 100 finds a matching key frame 416 in the event data (e.g., the event data 202) based on features in the frames captured at the frame rate 406. The matching key frame 416 may include an event prior probability, as described above. In response to finding the matching key frame 416, the electronic device 100 may determine a frame rate 408 (e.g., via the controller 126) and change the frame rate of the camera device from the frame rate 406 to the frame rate 408. The electronic device 100 may determine the frame rate 408 based on the event prior probability associated with the matching key frame 416. In some examples, the frame rate 408 may be a frame rate higher than the frame rate 406 based on the likelihood / probability value in the event prior probability associated with the matching key frame 416 being higher than the likelihood / probability value in the event prior probability associated with the matching key frame 414. In other examples, frame rate 408 may be a lower frame rate than frame rate 406 based on a likelihood / probability value in the event prior probability associated with matched keyframe 416 being lower than the likelihood / probability value in the event prior probability associated with matched keyframe 414. In yet other examples, frame rate 408 may be a higher or lower frame rate than frame rate 406 based on a different configurable frame rate or peak frame rate used to determine frame rate 408 (e.g., as compared to the configurable frame rate or peak frame rate used to determine frame rate 406).

[0116] At time t6, the electronic device 100 determines that the timer 418 for maintaining the frame rate 408 has expired before another key frame match. Based on the expiration of the timer 418, the electronic device 100 can change the frame rate of the camera device from the frame rate 408 back to the default frame rate 402. In some examples, the timer 418 can be the same as the timer 412. In other examples, the timer 418 can include a different expiration period than the timer 412.

[0117] In some cases, the electronic device 100 can implement a default frame rate that decays. For example, a reduced frame rate can be associated with increased event detection latency (e.g., a higher frame rate can result in lower latency). For the default frame rate 402, the electronic device 100 can default the frame rate of the camera device to a lower frame rate (e.g., can result in higher detection latency) when the event data includes a smaller number of recorded events (e.g., below a threshold). In such a case, the electronic device 100 can implement a default frame rate that starts at a high frame rate but decays as more events are added to the event data (e.g., as the electronic device 100 learns which locations / environments are most associated with events of interest).

[0118] For example, when the event data includes a smaller number of events (e.g., below a threshold number of events), the default frame rate can start higher than when the event data includes a larger number of events (e.g., above a threshold number of events) to reduce detection latency while the electronic device 100 gathers more information about the event of interest and / or associated environment to better learn which location / environment is most associated with the event of interest and / or to increase the accuracy of the prior probability associated with the location / environment and / or event of interest. As more entries are added to the event data and the electronic device 100 has more robust data / statistics for the event of interest, the electronic device 100 can begin to reduce the default frame rate. The reduced default frame rate can allow the electronic device 100 to reduce power consumption while still running the default frame rate when the electronic device 100 has more data (e.g., more map entries) about the environment and / or event that can be used to increase the confidence and / or accuracy of the determined prior probability reflecting the likelihood of detecting the event of interest within a particular environment. Thus, the electronic device 100 may run a lower default frame rate to avoid longer latency when an event of interest occurs, while running a higher default frame rate when the electronic device 100 has less information to assess the likelihood of detecting an event of interest, and may run a lower default frame rate when the amount of information (e.g., the confidence associated with a lower likelihood of detecting an event) increases to reduce power consumption by the camera device when an event of interest is not detected and / or the likelihood of detecting such an event is determined to be lower (e.g., below a threshold) based on a greater amount of associated data.

[0119] FIG. 5 is a diagram illustrating an example of different power states that the electronic device 100 executes when it is in different scenes. In this example, when the electronic device 100 is located in a scene 500 where the likelihood / probability of detecting / encountering an event of interest is lower, the electronic device 100 is operating in a low power state 502. The low power state 502 can be based on the likelihood / probability of detecting / encountering an event of interest in the scene 500. The low power state 502 can include, for example, a low power setting or mode of a camera device of the electronic device 100 (e.g., compared to a high power setting or mode supported by the camera device or another camera device). In some examples, the low power state 502 can also include low power operations and / or processing pipelines executed to process frames captured by the camera device of the electronic device 100. For example, the low power state 502 can include a setting for executing an image processing operation(s) or image processing pipeline associated with reduced processing complexity and / or power consumption. In some examples, the low power state 502 may also include a low power setting or mode of another device of the electronic device 100, such as a sensor, a processor, another camera device, and / or any other device.

[0120] The scene 510 also has a lower likelihood / probability of detecting / encountering an event of interest. When the electronic device 100 is located in the scene 510, the electronic device 100 can execute a default power state 512. The default power state 512 can be based on the likelihood / probability of detecting / encountering an event of interest in the scene 510. The default power state 512 in the scene 510 can be the same or different than the low power state 502 in the scene 500. In some examples, the default power state 512 can be associated with a lower power consumption than the low power state 502. In some examples, the default power state 512 can be the lowest power setting or mode supported by a camera device of the electronic device 100. In other examples, the default power state 512 can be associated with the same or higher power consumption as the low power state 502. In some cases, the default power state 512 can include a default frame rate, such as the default frame rate 402 shown in FIG. 4.

[0121] Meanwhile, the scene 520 has a higher likelihood / probability of detecting / encountering an event of interest. Here, the electronic device 100 can execute the high power state 522 based on the higher likelihood / probability of detecting / encountering an event of interest in the scene 520. In some examples, the high power state 522 can include a high power setting of a camera device or an image processing pipeline (e.g., higher frame rate, higher up-conversion, higher resolution, more image sensors activated, higher fidelity image processing pipeline, etc.). In some examples, the high power state 522 can include a state in which the electronic device 100 activates and / or uses a high power camera device based on a higher likelihood / probability. For example, in the high power state 522, the electronic device 100 can activate and / or execute a high power camera device or main camera device with higher capabilities, a processing pipeline with higher complexity / fidelity, etc., than the low power camera device implemented by the electronic device 100 in the scene 500 and the scene 510.

[0122] When the electronic device 100 encounters an object 530 in the scene 520, the electronic device 100 may capture an image of the object 530 (e.g., via the image sensor 102 and / or the image sensor 104). The electronic device 100 may perform object detection to capture an image of the object 530 and detect the object 530 in the image while in the high power state 522. In some examples, the high power state 522 may enable the electronic device 100 to capture higher quality images (e.g., higher resolution, higher frame rate, etc.) than when in the low power state 502 or the default power state 512. In some examples, the high power state 522 may enable the electronic device 100 to perform more robust / complex and / or higher fidelity image processing than when in the low power state 502 or the default power state 512.

[0123] 6 is a flow chart illustrating an example process 600 for adjusting camera settings based on event data. At block 602, the process 600 may include acquiring, via an image capture device (e.g., image sensor 102) of an electronic device (e.g., electronic device 100), an image (e.g., frame) depicting at least a portion of an environment. The environment may include, for example, but not limited to, a room (e.g., kitchen, bedroom, office, living room, garage, basement, etc.), a space or area (e.g., garden, patio, staircase, field, park, etc.), and / or any other environment.

[0124] At block 604, the process 600 may include determining a match between one or more visual features extracted from the image and one or more visual features associated with the key frame. In some cases, the key frame may include one of multiple key frames in event data (e.g., event data 202) on the electronic device. In some examples, the key frame may be associated with one or more detection events. For example, the key frame may include visual features of an event previously captured by an image and detected in the image by the electronic device. In some examples, the one or more detection events may include detection of a face captured by the image, a hand gesture captured by the image, an emotion captured by the image, a scene captured by the image, one or more people captured by the image, an animal captured by the image, a machine-readable code captured by the image (e.g., a QR code, a bar code, a link, etc.), an infrared light captured by the image, a two-dimensional surface or plane captured by the image, and / or text captured by the image.

[0125] In some examples, the event data may include a number of keyframes from a detection event associated with an image capture device, and / or a separate count of the detection events associated with each keyframe of the number of keyframes in the event data.

[0126] In some examples, determining a match between the one or more visual features extracted from the image and the one or more visual features associated with the keyframes in the event data may include comparing the one or more visual features extracted from the image to the one or more visual features associated with the keyframes, and / or comparing a first descriptor of the one or more visual features extracted from the image to a second descriptor of the one or more visual features associated with the keyframes.

[0127] In some cases, process 600 may include estimating a likelihood that the event of interest will occur in the environment based on the match. In some examples, the electronic device may calculate an event prior probability that includes a likelihood or probability value that represents an estimated likelihood that the event of interest will occur in the environment.

[0128] At block 606, process 600 may include adjusting (e.g., changing / adjusting) one or more settings of the image capture device based on the match. In some cases, process 600 may include adjusting one or more settings of the image capture device further based on the likelihood that the event of interest will occur in the environment.

[0129] In some examples, adjusting one or more settings of the image capture device may include changing a power mode of the image capture device. In some cases, changing a power mode of the image capture device may include changing a frame rate of the image capture device, a resolution of the image capture device, a binning mode of the image capture device, an imaging mode of the image capture device, and / or a number of image sensors activated by the image capture device and / or the electronic device.

[0130] In some examples, changing the power mode of the image capture device may include lowering the frame rate, resolution, binning mode, scaling factor, imaging mode, and / or number of activated image sensors based on a determination that a likelihood of an event of interest occurring in the environment is below a threshold. In some examples, changing the power mode of the image capture device may include increasing the frame rate, resolution, binning mode, scaling factor, imaging mode, and / or number of activated image sensors based on a determination that a likelihood of an event of interest occurring in the environment is above a threshold.

[0131] In some examples, the image capture device may include a first image capture device. In some aspects, the process 600 may include increasing a power mode of a second image capture device of the electronic device based on a determination that a likelihood of an event of interest occurring in the environment is above a threshold. In some cases, the second image capture device may employ a higher power mode than the first image capture device, a higher frame rate than the first image capture device, a higher resolution than the first image capture device, more image sensors than the first image capture device, and / or a higher power processing pipeline than a processing pipeline associated with the first image capture device. In some examples, increasing the power mode of the second image capture device may include turning on the second image capture device and / or initializing the second image capture device.

[0132] In some cases, the key frames may be contained or included in event data (e.g., event data 202) at the electronic device. In some examples, the event data 202 may include an event map that correlates features of one or more key frames associated with one or more environments (and / or one or more portions of one or more environments) with the one or more environments. In some cases, the event data may include a key frame. In some examples, the event data may include multiple key frames from a previous detection event (e.g., previous detection of one or more events of interest) by the image capture device. In some cases, the event data may also include a separate count of detection events associated with each key frame of the multiple key frames.

[0133] In some examples, process 600 can include estimating a likelihood that an event of interest will occur in the environment. In some cases, estimating a likelihood that an event of interest will occur in the environment can include estimating a likelihood that a detected event will occur in the environment based on a match and distinct count of detected events (e.g., previous detections of the event(s) of interest) associated with a keyframe. In some examples, process 600 can include incrementing a distinct count of detected events associated with a keyframe in response to determining a match between one or more visual features extracted from the image and one or more visual features associated with the keyframe.

[0134] In some examples, process 600 may include acquiring, via an image capture device, a different image depicting at least a portion of a different environment, determining that one or more visual features extracted from the different image do not match visual features associated with any keyframes in the event data at the electronic device, and creating a new entry in the event data corresponding to the different image. In some cases, the new entry is created in response to determining that one or more visual features extracted from the different image do not match visual features associated with any keyframes in the event data and at least one of a determination that an event of interest was detected in the different image, a time that has elapsed since a distinct keyframe in the event data was last created, and a time that has elapsed since a distinct match was last identified between a particular keyframe in the event data and a particular image captured by the image capture device.

[0135] In some examples, process 600 may include determining that an event of interest is detected in the different image, determining a second likelihood that an additional event of interest will occur in the different environment based on the determination that an event of interest is detected in the different image, and adjusting at least one setting of the image capture device based on the second likelihood that the additional event of interest will occur in the different environment.

[0136] In some examples, one or more settings of the image capture device can include a frame rate, and the process 600 can include determining the frame rate based on a predetermined frame rate for a detection event and a likelihood of an event of interest occurring in the environment, and maintaining the frame rate at least until expiration of a set period (e.g., expiration of a timer) or until a subsequent determination of a match between a different image captured by the image capture device and at least one key frame in the event data. In some examples, the process 600 can include adjusting the frame rate of the image capture device to a different frame rate after expiration of the set period or after a subsequent determination of a match. In some cases, the different frame rate can include a default frame rate or a specific frame rate calculated based on a predetermined frame rate and a different likelihood of a detection event associated with a subsequent match. In some cases, the different frame rate can include a default frame rate or a specific frame rate determined based on a predetermined frame rate and a second likelihood of detecting a specific event of interest associated with a subsequent match.

[0137] In some cases, the predetermined frame rate may include a highest frame rate supported by the image capture device. In some examples, determining the frame rate may include multiplying the predetermined frame rate by a value corresponding to a likelihood that the event of interest will occur in the environment. In some cases, the predetermined frame rate is higher than a default frame rate. In some examples, the process 600 may include reducing the default frame rate to a lower frame rate in response to adding one or more key frames to the event data.

[0138] In some examples, the process 600 can include adjusting one or more different settings of an active depth transmitter system, such as a flood illuminator of an electronic device, a depth sensor device of an electronic device, a dual image capture device system of an electronic device, a structured light system of an electronic device, a time-of-flight system of an electronic device, etc., based on the likelihood of an event of interest occurring within an environment. In some cases, the process 600 can include adjusting one or more settings of an audio algorithm, a location service associated with at least one of a global navigation satellite system (GNSS), a wireless location area network connection and / or data (e.g., WIFI), and / or a global positioning system (GPS), based on the likelihood of an event of interest occurring within the environment. In some examples, adjusting the one or more different settings can include turning off or running (e.g., turning on, activating, initializing, powering on, etc.) a flood illuminator, a depth sensor device, a dual image capture device system, a structured light system, a time-of-flight system, an audio algorithm, and / or a location service. In some examples, adjusting the one or more different settings can include turning off or running an active depth transmitter.

[0139] In some aspects, process 600 may include determining a likelihood that the event of interest will occur within the environment based on the match, and adjusting one or more settings of the image capture device further based on the likelihood that the event of interest will occur within the environment.

[0140] In some aspects, process 600 may include determining the likelihood of the event further based on data from a non-image capture device of the electronic device, or adjusting one or more settings of the image capture device further based on data from a non-image capture device of the electronic device.

[0141] In some examples, the process 600 can include periodically reducing the count of each of the detection events associated with a keyframe entry in the event data. In some cases, the count of each of the detection events is reduced proportionally across all keyframe entries in the event data.

[0142] In some examples, the process 600 may be performed by one or more computing devices or apparatuses. In one illustrative example, the process 600 may be performed by the electronic device 100 shown in FIG. 1. In some examples, the process 600 may be performed by one or more computing devices having the computing device architecture 700 shown in FIG. 7. In some cases, such computing devices or apparatuses may include a processor, microprocessor, microcomputer, or other components of a device configured to perform the steps of the process 600. In some examples, such computing devices or apparatuses may include one or more sensors configured to capture image data and / or other sensor measurements. For example, the computing device may include a smartphone, a head mounted display, a mobile device, or other suitable device. In some examples, such computing devices or apparatuses may include a camera configured to capture one or more images or videos. In some cases, such computing devices may include a display for displaying the images. In some examples, the one or more sensors and / or cameras are separate from the computing device, in which case the computing device receives the sensed data. Such computing devices may further include a network interface configured to communicate data.

[0143] The components of a computing device may be implemented in a circuit. For example, the components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform various operations described herein. The computing device may further include a display (as an example of an output device or in addition to an output device), a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0144] Process 600 is illustrated as a logical flow diagram, whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement a process.

[0145] Additionally, process 600 may be executed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a number of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0146] 7 illustrates an exemplary computing device architecture 700 of an exemplary computing device that may implement various techniques described herein. For example, the computing device architecture 700 may implement at least some portions of the electronic device 100 illustrated in FIG. 1. The components of the computing device architecture 700 are shown in electrical communication with each other using a connection 705, such as a bus. The exemplary computing device architecture 700 includes a processing unit (CPU or processor) 710 and a computing device connection 705 that couples various computing device components to the processor 710, including a computing device memory 715, such as a read only memory (ROM) 720 and a random access memory (RAM) 725.

[0147] The computing device architecture 700 may include a cache of high-speed memory directly connected to the processor 710, near the processor 710, or integrated as part of the processor 710. The computing device architecture 700 may copy data from the memory 715 and / or the storage device 730 to the cache 712 for faster access by the processor 710. In this manner, the cache may provide a performance boost that avoids delays to the processor 710 while waiting for data. These and other modules may control or may be configured to control the processor 710 to perform various actions. Other computing device memories 715 may also be available. The memory 715 may include multiple different types of memory with different performance characteristics. The processor 710 may include any general-purpose processor and hardware or software services stored in the storage device 730 that are configured to control the processor 710 and special-purpose processors with software instructions built into the processor design. The processor 710 may be a self-contained system that includes multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0148] To enable user interaction with the computing device architecture 700, the input device 745 can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. The output device 735 can also be one or more of several output mechanisms known to those skilled in the art, such as a display, projector, television, speaker device, etc. In some cases, a multimodal computing device may enable a user to provide multiple types of input to communicate with the computing device architecture 700. The communication interface 740 can generally govern and manage user input and computing device output. There is no constraint to operate on any particular hardware configuration, and thus the basic functions herein may be easily replaced with improved hardware or firmware configurations as they are developed.

[0149] The storage device 730 is a non-volatile memory and may be a hard disk or other type of computer readable medium capable of storing data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a random access memory (RAM) 725, a read only memory (ROM) 720, and hybrids thereof. The storage device 730 may include software, code, firmware, etc. for controlling the processor 710. Other hardware or software modules are contemplated. The storage device 730 may be connected to the computing device connection 705. In one aspect, a hardware module that performs a particular function may include software components stored in a computer readable medium that connects with the necessary hardware components, such as the processor 710, the connection 705, the output device 735, etc., to perform the function.

[0150] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, store, or convey instruction(s) and / or data. Computer-readable media may include non-transitory media on which data is stored and does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory, or memory devices. Computer-readable media may have code and / or machine-executable instructions stored on the computer-readable medium, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0151] In some embodiments, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when mentioned, non-transitory computer readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0152] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For clarity of explanation, in some cases, the present technology may be presented as including individual functional blocks comprising devices, device components, and steps or routines in a method embodied in software or a combination of hardware and software. Additional components may be used other than the components shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0153] Individual embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowcharts may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of steps may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagrams. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0154] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored on or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general purpose computer, a special purpose computer, or a processing device to perform a particular function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to the described examples include magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, network-attached storage devices, etc.

[0155] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) to perform the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor(s) may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small-footprint personal computers, personal digital assistants, rack-mounted devices, standalone devices, and the like. The functionality described herein may also be embodied in a peripheral device or an add-in card. Such functionality may also be implemented on a circuit board among different chips, or on different processes executing in a single device, as further examples.

[0156] The instructions, media for carrying such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0157] In the above description, aspects of the present application are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, while exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in various other ways, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. The various features and aspects of the present application described above may be used individually or jointly. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative and not restrictive. For purposes of illustration, the methods have been described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in an order different from that described.

[0158] Those skilled in the art will understand that the less than ("<") and greater than (">") symbols or terminology used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of the present specification.

[0159] When a component is described as being "configured to" perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0160] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component and / or that is in communication, either directly or indirectly, with another component (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0161] Claim language or other language in this disclosure reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can additionally include unrecited items within the set of A and B.

[0162] The various exemplary logic blocks, modules, circuits, and algorithm steps described with respect to the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, and such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0163] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium including program code including instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (non-volatile RAM, NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage medium, etc. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, which may carry or communicate program code in the form of instructions or data structures and which may be accessed, read, and / or executed by a computer.

[0164] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0165] Illustrative examples of the present disclosure include the following: Aspect 1. An apparatus for acquiring images, comprising: a memory; and one or more processors coupled to the memory, wherein the one or more processors are configured to acquire images depicting at least a portion of an environment via an image capture device of the apparatus; determine a match between one or more visual features extracted from the images and one or more visual features associated with key frames associated with one or more detection events; and adjust one or more settings of the image capture device based on the match.

[0166] Aspect 2. The apparatus of aspect 1, wherein to adjust one or more settings of the image capture device, the one or more processors are configured to change a power mode of the image capture device.

[0167] Aspect 3. The apparatus of aspect 2, wherein to change the power mode of the image capture device, the one or more processors are configured to change at least one of: a frame rate of the image capture device, a resolution of the image capture device, a binning mode of the image capture device, an imaging mode of the image capture device, and a number of image sensors activated by at least one of the image capture device and the apparatus.

[0168] Aspect 4. The apparatus of aspect 3, wherein to change the power mode of the image capture device, the one or more processors are configured to reduce at least one of the frame rate, resolution, binning mode, imaging mode, and number of image sensors activated based on a determination that a likelihood of an event of interest occurring in the environment is below a threshold.

[0169] Aspect 5. The apparatus of aspect 3, wherein to change the power mode of the image capture device, the one or more processors are configured to increase at least one of the frame rate, resolution, binning mode, imaging mode, and number of image sensors activated based on a determination that a likelihood of an event of interest occurring in the environment exceeds a threshold.

[0170] Aspect 6. The apparatus of any of aspects 1-5, wherein the image capture device includes a first image capture device, and the one or more processors are configured to increase a power mode of a second image capture device of the apparatus based on a determination that a likelihood of an event of interest occurring in the environment exceeds a threshold, the second image capture device employing at least one of a higher power mode than the first image capture device, a higher frame rate than the first image capture device, a higher resolution than the first image capture device, a greater number of image sensors than the first image capture device, and a higher power processing pipeline than a processing pipeline associated with the first image capture device.

[0171] Aspect 7. The apparatus of aspect 6, wherein to increase a power mode of the second image capture device, the one or more processors are configured to initialize the second image capture device.

[0172] Aspect 8. An apparatus as described in any of aspects 1-7, wherein the keyframes are included in event data in the apparatus, the event data including a number of keyframes from a detection event associated with the image capture device.

[0173] Aspect 9. The apparatus of aspect 8, wherein the event data further includes a separate count of detected events associated with each keyframe of the multiple keyframes.

[0174] Aspect 10. The apparatus of aspect 8 or 9, wherein the one or more processors are configured to determine the likelihood of an event of interest occurring in the environment, and to determine the likelihood of an event occurring in the environment, the one or more processors are configured to determine the likelihood of an event occurring in the environment based on matches and a distinct count of detected events associated with key frames.

[0175] Aspect 11. The apparatus of aspect 9 or 10, wherein the one or more processors are configured to increase a distinct count of detected events associated with a key frame in the event data in response to determining a match between one or more visual features extracted from the image and one or more visual features associated with the key frame.

[0176] Aspect 12. An apparatus as described in any of aspects 9 to 11, wherein the one or more processors are configured to determine the likelihood of an event of interest occurring within the environment, and to determine the likelihood of an event occurring within the environment, the one or more processors are configured to determine the likelihood of an event occurring within the environment based on the match and one or more keyframes from the multiple keyframes.

[0177] Aspect 13. The apparatus of any of aspects 1-12, wherein the one or more processors are configured to acquire, via an image capture device, different images depicting at least a portion of a different environment, determine that one or more visual features extracted from the different images do not match visual features associated with any keyframes in the event data in the apparatus, and create a new entry in the event data corresponding to the different images, wherein the new entry is created in response to determining that one or more visual features extracted from the different images do not match visual features associated with any keyframes in the event data, and at least one of: a determination that an event of interest was detected in the different images, a time elapsed since an individual keyframe in the event data was last created, and a time elapsed since an individual match was last identified between a particular keyframe in the event data and a particular image captured by the image capture device.

[0178] Aspect 14. The apparatus of aspect 13, wherein the one or more processors are configured to determine that an event of interest has been detected in a different image, determine a second likelihood that an additional event of interest will occur in the different environment based on the determination that the event of interest has been detected in the different image, and adjust at least one setting of the image capture device based on the second likelihood that an additional event of interest will occur in the different environment.

[0179] Aspect 15. An apparatus as described in any of aspects 1-14, wherein one or more settings of the image capture device include a frame rate, and wherein the one or more processors are configured to determine the frame rate based on a predetermined frame rate for a detected event and a likelihood of an event of interest occurring in the environment, and to maintain the frame rate at least until expiration of a set period of time or until a subsequent determination of a match between a different image captured by the image capture device and at least one key frame in the event data at the apparatus.

[0180] Aspect 16. The apparatus of aspect 15, wherein the one or more processors are configured to adjust the frame rate of the image capture device to a different frame rate after expiration of a set period of time or after determining a subsequent match.

[0181] Aspect 17. The apparatus of aspect 16, wherein the different frame rate includes a default frame rate or a specific frame rate determined based on a predetermined frame rate and a second likelihood of detecting a particular target event associated with a subsequent match.

[0182] Aspect 18. The device of any of aspects 15-17, wherein the predetermined frame rate is higher than a default frame rate, and the one or more processors are configured to reduce the default frame rate to a lower frame rate in response to adding one or more key frames to event data in the device.

[0183] Aspect 19. The apparatus of aspect 16, wherein the predetermined frame rate includes the highest frame rate supported by the image capture device, and to determine the frame rate, the one or more processors are configured to multiply the predetermined frame rate by a value corresponding to a likelihood that a detection event will occur within the environment.

[0184] Aspect 20. The apparatus of any of aspects 1-19, wherein to determine a match between one or more visual features extracted from the image and one or more visual features associated with the key frame, the one or more processors are configured to compare at least one of the one or more visual features extracted from the image with one or more visual features associated with the key frame, and to compare a first descriptor of the one or more visual features extracted from the image with a second descriptor of the one or more visual features associated with the key frame.

[0185] Aspect 21. A device as described in any of aspects 1 to 20, wherein the one or more processors are configured to adjust one or more different settings of at least one of the device's active depth transmitter, audio algorithms, and location services associated with at least one of a Global Navigation Satellite System (GNSS) system, a wireless location area network connection, and a Global Positioning System (GPS) based on a likelihood of an event of interest occurring within the environment.

[0186] Aspect 22. The device of aspect 21, wherein one or more processors are configured to turn off or execute at least one of an active depth transmitter, an audio algorithm, and a location service to adjust one or more different settings.

[0187] Aspect 23. An apparatus as described in any of aspects 1 to 22, wherein the one or more processors are configured to periodically reduce a count of each of the detection events associated with a key frame entry in the event data in the apparatus, and the count of each of the detection events is reduced proportionally across all key frame entries in the event data.

[0188] Aspect 24. The apparatus of any of aspects 1-23, wherein the one or more detection events include detection of at least one of a face depicted by the image, a hand gesture depicted by the image, an emotion depicted by the image, a scene depicted by the image, one or more people depicted by the image, an animal depicted by the image, a machine-readable code depicted by the image, infrared light depicted by the image, a two-dimensional surface depicted by the image, and text depicted by the image.

[0189] Aspect 25. A device according to any one of aspects 1 to 24, wherein the key frame is part of event data in the device, and the event data includes a plurality of key frames.

[0190] Aspect 26. An apparatus as described in any of aspects 1 to 25, wherein the one or more processors are configured to determine a likelihood of a target event occurring within the environment based on the match, and adjust one or more settings of the image capture device further based on the likelihood of a target event occurring within the environment.

[0191] Aspect 27. The device described in Aspect 26, wherein the one or more processors are further configured to determine the likelihood of an event based further on data from a non-image capture device of the device, or to adjust one or more settings of the image capture device based further on data from a non-image capture device of the device.

[0192] Embodiment 28. The apparatus of any of embodiments 1 to 27, wherein the apparatus comprises a mobile device.

[0193] Example 29. The apparatus of any of Examples 1 to 28, wherein the apparatus includes an augmented reality device.

[0194] Aspect 30. A method for acquiring an image, the method including: acquiring an image depicting at least a portion of an environment via an image capture device of an electronic device; determining a match between one or more visual features extracted from the image and one or more visual features associated with key frames associated with one or more detection events; and adjusting one or more settings of the image capture device based on the match.

[0195] Aspect 31. The method of aspect 30, wherein adjusting one or more settings of the image capture device includes changing a power mode of the image capture device.

[0196] Aspect 32. The method of aspect 31, wherein changing the power mode of the image capture device includes changing at least one of the frame rate of the image capture device, the resolution of the image capture device, the binning mode of the image capture device, the imaging mode of the image capture device, and the number of image sensors activated by at least one of the image capture device and the electronic device.

[0197] Aspect 33. The method of aspect 31 or 32, wherein changing the power mode of the image capture device includes reducing at least one of the frame rate, resolution, binning mode, imaging mode, and number of image sensors activated based on a determination that the likelihood of an event of interest occurring in the environment is below a threshold.

[0198] Aspect 34. The method of aspect 31 or 32, wherein changing the power mode of the image capture device includes increasing at least one of the frame rate, resolution, binning mode, imaging mode, and number of activated image sensors based on a determination that the likelihood of an event of interest occurring in the environment exceeds a threshold.

[0199] Aspect 35. A method according to any of aspects 30 to 34, wherein the image capture device includes a first image capture device, and the method further includes increasing a power mode of a second image capture device of the electronic device based on a determination that a likelihood of an event of interest occurring in the environment exceeds a threshold, wherein the second image capture device employs at least one of a higher power mode than the first image capture device, a higher frame rate than the first image capture device, a higher resolution than the first image capture device, a greater number of image sensors than the first image capture device, and a higher power processing pipeline than a processing pipeline associated with the first image capture device.

[0200] Aspect 36. The method of aspect 35, wherein increasing the power mode of the second image capture device includes initializing the second image capture device.

[0201] Aspect 37. The method of any of aspects 30-36, wherein the key frames are included in event data at the electronic device, the event data including a number of key frames from a detected event associated with the image capture device.

[0202] Aspect 38. The method of aspect 37, wherein the event data further includes a separate count of detected events associated with each keyframe of the multiple keyframes.

[0203] Aspect 39. The method of aspect 37 or 38, further comprising determining a likelihood that an event of interest will occur in the environment based on the matches and the distinct count of detected events associated with the key frames.

[0204] Aspect 40. The method of aspect 38 or 39, further comprising, in response to determining a match between one or more visual features extracted from the image and one or more visual features associated with the key frame, increasing a distinct count of detected events associated with the key frame in the event data.

[0205] Embodiment 41. The method of any of embodiments 38-40, further comprising determining a likelihood that an event of interest will occur in the environment based on the match and one or more keyframes from the multiple keyframes.

[0206] Aspect 42. The method of any of aspects 30-41, further comprising: capturing a different image depicting at least a portion of a different environment via an image capture device; determining that one or more visual features extracted from the different image do not match visual features associated with any keyframe in the event data at the electronic device; and creating a new entry in the event data corresponding to the different image, wherein the new entry is created in response to determining that one or more visual features extracted from the different image do not match visual features associated with any keyframe in the event data and at least one of: a determination that an event of interest has been detected in the different image; a time elapsed since an individual keyframe in the event data was last created; and a time elapsed since an individual match was last identified between a particular keyframe in the event data and a particular image captured by the image capture device.

[0207] Aspect 43. The method of aspect 42, further comprising: determining that an event of interest is detected in the different image; determining a second likelihood that an additional event of interest will occur in the different environment based on the determination that an event of interest is detected in the different image; and adjusting at least one setting of the image capture device based on the second likelihood that an additional event of interest will occur in the different environment.

[0208] Aspect 44. The method of any of aspects 30 to 43, wherein one or more settings of the image capture device include a frame rate, the method further including: determining the frame rate based on a predetermined frame rate for the detected event and a likelihood of an event of interest occurring in the environment; and maintaining the frame rate at least until expiration of a set period of time or until determination of a subsequent match between a different image captured by the image capture device and at least one key frame in the event data at the electronic device.

[0209] Aspect 45. The method of aspect 44, further comprising adjusting the frame rate of the image capture device to a different frame rate after expiration of the set period or after a subsequent determination of a match.

[0210] Aspect 46. The method of aspect 45, wherein the different frame rate includes a default frame rate or a specific frame rate determined based on a predetermined frame rate and a second likelihood of detecting a particular target event associated with a subsequent match.

[0211] Aspect 47. The method of any of aspects 44 to 46, wherein the predetermined frame rate is higher than a default frame rate, and the method further includes reducing the default frame rate to a lower frame rate in response to adding one or more key frames to the event data at the electronic device.

[0212] Aspect 48. The method of aspect 47, wherein the predetermined frame rate includes the highest frame rate supported by the image capture device, and determining the frame rate may include multiplying the predetermined frame rate by a value corresponding to a likelihood that a detection event will occur within the environment.

[0213] Aspect 49. The method of any of aspects 30 to 48, wherein determining a match between one or more visual features extracted from the image and one or more visual features associated with the key frame further includes comparing at least one of the one or more visual features extracted from the image with one or more visual features associated with the key frame, and comparing a first descriptor of the one or more visual features extracted from the image with a second descriptor of the one or more visual features associated with the key frame.

[0214] Aspect 50. The method of any of aspects 30 to 49, further comprising adjusting one or more different settings of at least one of an active depth transmitter of the electronic device, an audio algorithm, and a location service associated with at least one of a Global Navigation Satellite System (GNSS) system, a wireless location area network connection, and a Global Positioning System (GPS) based on a likelihood that an event of interest will occur within the environment.

[0215] Aspect 51. The method of aspect 50, wherein adjusting one or more different settings includes turning off or running at least one of an active depth transmitter, an audio algorithm, and a location service.

[0216] Aspect 52. The method of any of aspects 30 to 51, further comprising periodically reducing a count of each of the detection events associated with a key frame entry in the event data in the electronic device, wherein the count of each of the detection events is reduced proportionally across all key frame entries in the event data.

[0217] Aspect 53. The method of any of aspects 30 to 52, wherein the one or more detection events include detection of at least one of a face depicted by the image, a hand gesture depicted by the image, an emotion depicted by the image, a scene depicted by the image, one or more people depicted by the image, an animal depicted by the image, a machine-readable code depicted by the image, an infrared light depicted by the image, a two-dimensional surface depicted by the image, and text depicted by the image.

[0218] Aspect 54. The method of any of aspects 30 to 53, wherein the key frame is part of event data in the electronic device, and the event data includes a plurality of key frames.

[0219] Aspect 55. A method as described in any of aspects 30 to 54, comprising determining a likelihood that the target event will occur within the environment based on the match, and adjusting one or more settings of the image capture device further based on the likelihood that the target event will occur within the environment.

[0220] Aspect 56. The method of aspect 55, further comprising determining the likelihood of an event based further on data from a non-image capture device of the electronic device, or adjusting one or more settings of the image capture device based further on data from a non-image capture device of the electronic device.

[0221] Embodiment 57. The method of any one of embodiments 30 to 56, wherein the electronic device comprises a mobile device.

[0222] Embodiment 58. The method of any of embodiments 30 to 57, wherein the electronic device includes an augmented reality device.

[0223] Embodiment 59. An apparatus comprising means for carrying out the method according to any of embodiments 30-58.

[0224] Aspect 60. The apparatus of aspect 59, wherein the apparatus comprises a mobile device.

[0225] Aspect 61. The method of aspect 59 or 60, wherein the apparatus includes an augmented reality device.

[0226] Aspect 62. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform a method according to any of aspects 30-58.

[0227] Aspect 63. The non-transitory computer-readable medium of aspect 62, wherein the electronic device comprises a mobile device.

[0228] Aspect 64. The non-transitory computer-readable medium of aspect 62 or 63, wherein the electronic device includes an augmented reality device.

Claims

1. An apparatus for acquiring an image, comprising: a memory storing event data including a plurality of key frames from detection events associated with an image capture device of the apparatus; one or more processors coupled to the memory, the one or more processors being configured to: acquire an image depicting at least a portion of an environment via the image capture device; determine a match between one or more visual features extracted from the image and one or more visual features associated with key frames in the event data; and adjust one or more settings of the image capture device based on the match. one or more processors; and an apparatus comprising the same.

2. To adjust the one or more settings of the image capture device, the one or more processors are configured to change a power mode of the image capture device, and To change the power mode of the image capture device, the one or more processors are configured to change at least one of a frame rate of the image capture device, a resolution of the image capture device, a binning mode of the image capture device, an imaging mode of the image capture device, and a number of image sensors activated by at least one of the image capture device and the apparatus, and To change the power mode of the image capture device, the one or more processors are configured to: reduce at least one of the frame rate, the resolution, the binning mode, the imaging mode, and the number of activated image sensors based on a determination that a likelihood of a target event occurring in the environment is below a threshold; or increase at least one of the frame rate, the resolution, the binning mode, the imaging mode, and the number of activated image sensors based on a determination that a likelihood of a target event occurring in the environment is above a threshold. The apparatus according to claim 1.

3. The image capture device includes a first image capture device, and the one or more processors are configured to: Based on a determination that the likelihood of a target event occurring within the environment exceeds a threshold, the power mode of the second image capture device of the apparatus is configured to be increased, and the second image capture device employs at least one of a higher power mode than the first image capture device, a higher frame rate than the first image capture device, a higher resolution than the first image capture device, a larger number of image sensors than the first image capture device, and a higher power processing pipeline than the processing pipeline associated with the first image capture device. The apparatus according to claim 1, wherein, in order to increase the power mode of the second image capture device, the one or more processors are configured to initialize the second image capture device.

4. The apparatus according to claim 1, wherein the event data further includes an individual count of detection events associated with each of a number of key frames.

5. The one or more processors are configured to determine the likelihood of a target event occurring within the environment, and in order to determine the likelihood of the event occurring within the environment, the one or more processors are configured to determine the likelihood of the event occurring within the environment based on the match and one or more key frames from a number of key frames. The apparatus according to claim 1.

6. The apparatus according to claim 5, wherein the one or more processors are configured to determine the likelihood of a target event occurring within the environment, and in order to determine the likelihood of the event occurring within the environment, the one or more processors are configured to determine the likelihood of the event occurring within the environment based on the match and the individual count of detection events associated with the key frame.

7. The one or more processors are configured to increase the individual count of detection events associated with the key frame in the event data in response to determining a match between one or more visual features extracted from the image and one or more visual features associated with the key frame. The apparatus according to claim 5.

8. The one or more processors Obtain different images depicting at least a portion of different environments via the image capture device, Determine that one or more visual features extracted from the different images do not match any visual features associated with key frames in the event data in the device, Create a new entry in the event data corresponding to the different images, configured such that the new entry determines that the one or more visual features extracted from the different images do not match any visual features associated with key frames in the event data, a determination that a target event has been detected in the different images, the time elapsed since the last individual key frame in the event data was created, and the time elapsed since the last individual match was identified between a specific key frame in the event data and a specific image captured by the image capture device, and is created in response to at least one of them, The one or more processors, Determine that the target event has been detected in the different images, Based on the determination that the target event has been detected in the different images, determine a second likelihood that an additional target event will occur in the different environments, Adjust at least one setting of the image capture device based on the second likelihood that the additional target event will occur in the different environments, The apparatus according to claim 1, configured as such.

9. The one or more settings of the image capture device include a frame rate, and the one or more processors, Determine the frame rate based on a predetermined frame rate for detecting events and the likelihood that a target event will occur in the environment, Maintain the frame rate until at least the expiration of a set period or the determination of a subsequent match between different images captured by the image capture device and at least one key frame in the event data in the device, configured as such, The one or more processors, After the expiration of the set period or after the determination of the subsequent match, are configured to adjust the frame rate of the image capture device to a different frame rate. The different frame rates include a default frame rate, or a specific frame rate determined based on the predetermined frame rate and a second likelihood of detecting a specific target event associated with the subsequent match. The apparatus according to claim 1.

10. The one or more processors are configured to adjust one or more different settings of at least one of the active depth transmitter of the apparatus, an audio algorithm, and a location service associated with at least one of a Global Navigation Satellite System (GNSS), a wireless location area network connection, and a Global Positioning System (GPS) based on a likelihood of a target event occurring in the environment. The apparatus according to claim 1, wherein the one or more processors are configured to turn off or execute at least one of the active depth transmitter, the audio algorithm, and the location service to adjust the one or more different settings.

11. The one or more processors are configured to periodically reduce the respective count of detected events associated with key frame entries in the event data of the apparatus, and the respective count of the detected events is proportionally reduced across all key frame entries in the event data. The apparatus according to claim 1.

12. The one or more detected events include detection of at least one of a face depicted by the image, a hand gesture depicted by the image, an emotion depicted by the image, a scene depicted by the image, one or more persons depicted by the image, an animal depicted by the image, a machine-readable code depicted by the image, infrared light depicted by the image, a two-dimensional surface depicted by the image, and text depicted by the image. The apparatus according to claim 1.

13. The one or more processors determine a likelihood of a target event occurring in the environment based on the match, and further adjust the one or more settings of the image capture device based on the likelihood of the target event occurring in the environment. configured to The one or more processors further determining the likelihood of the event based on data from a non-image capture device of the apparatus, or further adjusting the one or more settings of the image capture device based on the data from the non-image capture device of the apparatus The apparatus according to claim 1, further configured as such. **Claim 14** wherein the apparatus includes a mobile device, The apparatus according to claim 1, wherein the apparatus includes an augmented reality device. **Claim 15** A method for acquiring an image, comprising: acquiring, via an image capture device of an electronic device, an image depicting at least a portion of an environment, wherein the electronic device stores event data including a plurality of key frames from a detection event associated with the image capture device in a memory of the electronic device; determining a match between one or more visual features extracted from the image and one or more visual features associated with a key frame in the event data; adjusting one or more settings of the image capture device based on the match and a method.

Images