Method for determining a region of interest for camera autofocus - Patents.com

Abstract

The method includes receiving an image frame captured by an image capture device. The method also includes determining a saliency heatmap representing the saliency of pixels in the image frame. The method further includes determining a primary region of interest (ROI) and a secondary ROI for the image frame based on the saliency heatmap. The method further includes determining a filtered ROI for the image frame, where the filtered ROI is updated from the previous filtered ROI to the primary ROI or the secondary ROI based on a saliency difference between the previous filtered ROI and the primary ROI or the secondary ROI exceeding a first threshold. The method also includes applying one or more autofocus processes based on the filtered ROI, the primary ROI, or the secondary ROI.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 378,648, filed October 6, 2022, the disclosure of which is incorporated herein by reference in its entirety. [Background technology]

[0002] Many modern computing devices, including mobile phones, personal computers, and tablets, are equipped with image capture devices, some of which are configured with telephoto capabilities. Summary of the Invention

[0003] In an embodiment, a method includes receiving an image frame captured by an image capture device. The method also includes determining a saliency heatmap representing the saliency of pixels in the image frame. The method further includes determining a primary region of interest (ROI) and a secondary ROI for the image frame based on the saliency heatmap. The method further includes determining a filtered ROI for the image frame, where the filtered ROI is updated from a previous filtered ROI to the primary ROI based on a saliency difference between the previous filtered ROI and the primary ROI exceeding a first threshold. The method also includes applying one or more autofocus processes based on at least one of the filtered ROI, the primary ROI, or the secondary ROI.

[0004] In another embodiment, a system includes a processor and a non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by the processor, causing the processor to perform operations. The operations include receiving an image frame captured by an image capture device. The operations also include determining a saliency heatmap representing saliency of pixels in the image frame. The operations further include determining a primary region of interest (ROI) and a secondary ROI for the image frame based on the saliency heatmap. The operations further include determining a filtered ROI for the image frame, the filtered ROI being updated from a previous filtered ROI to the primary ROI based on a saliency difference between the previous filtered ROI and the primary ROI being greater than a first threshold. The operations also include applying one or more autofocus processes based on at least one of the filtered ROI, the primary ROI, or the secondary ROI.

[0005] In an embodiment, the image capture device includes a camera and a control system. The control system is configured to receive image frames captured by the image capture device. The control system is also configured to determine a saliency heat map representing the saliency of pixels in the image frames. The control system is further configured to determine a primary region of interest (ROI) and a secondary ROI for the image frames based on the saliency heat map. The control system is further configured to determine a filtered ROI for the image frames, where the filtered ROI is updated from a previous filtered ROI to the primary ROI based on a saliency difference between the previous filtered ROI and the primary ROI exceeding a first threshold. The control system is also configured to apply one or more autofocus processes based on at least one of the filtered ROI, the primary ROI, or the secondary ROI.

[0006] In another embodiment, a system is provided that includes means for receiving an image frame captured by an image capture device. The system also includes means for determining a saliency heatmap representing the saliency of pixels in the image frame. The system further includes means for determining a primary region of interest (ROI) and a secondary ROI for the image frame based on the saliency heatmap. The system further includes means for determining a filtered ROI for the image frame, where the filtered ROI is updated from a previous filtered ROI to the primary ROI based on a saliency difference between the previous filtered ROI and the primary ROI exceeding a first threshold. The system also includes means for applying one or more autofocus processes based on at least one of the filtered ROI, the primary ROI, or the secondary ROI.

[0007] The above summary is illustrative only and is not intended to be in any way limiting. In addition to the exemplary aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the figures and the following detailed description, and accompanying drawings. [Brief explanation of the drawings]

[0008] [Figure 1] 1 illustrates an exemplary computing device, according to an exemplary embodiment. [Figure 2] FIG. 1 is a simplified block diagram illustrating some of the components of an exemplary computing system. [Figure 3] FIG. 1 illustrates the training and inference phases of one or more trained machine learning models, according to an example embodiment. [Figure 4a] 1 is an image according to an example embodiment. [Figure 4b] 1 is a heat map according to an example embodiment. [Figure 5] 1 illustrates a heatmap with bounding boxes according to an example embodiment. [Figure 6A]1 illustrates an anchor bounding box in accordance with an example embodiment; [Figure 6B] 1 illustrates anchor bounding box locations according to an example embodiment. [Figure 7] 1 illustrates a salient region of interest (ROI), according to an exemplary embodiment. [Figure 8] 1 illustrates an image with an ROI, according to an exemplary embodiment. [Figure 9] 1 illustrates a finite state machine in accordance with an illustrative embodiment. [Figure 10] 1 illustrates a finite state machine manager in accordance with an exemplary embodiment. [Figure 11] 1 is a flowchart of a method according to an example embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Exemplary methods, devices, and systems are described herein. It should be understood that the words "exemplary" and "exemplary" are used herein to mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as "exemplary" or "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or features, unless otherwise indicated. Other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein.

[0010] Thus, the exemplary embodiments described herein are not meant to be limiting, as it will be readily understood that the aspects of the present disclosure, as generally described herein and illustrated in the Figures, may be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

[0011] Throughout this description, the articles "a" or "an" are used to introduce elements of exemplary embodiments. Any reference to "a" or "an" refers to "at least one," and any reference to "the" refers to "the at least one," unless otherwise specified or clearly dictated by context. Use of the conjunction "or" in a list of at least two terms is intended to refer to either of the listed terms or any combination of the listed terms.

[0012] The use of ordinal numbers such as "first," "second," and "third" is to distinguish between elements and not to indicate a particular order of these elements. For purposes of this description, the terms "multiple" and "a plurality of" refer to "two or more" or "more than one."

[0013] Furthermore, unless the context suggests otherwise, the features illustrated in each of the figures may be used in combination with one another. Thus, the figures should be viewed generally as aspects of components of one or more overall embodiments, with the understanding that not all of the illustrated features are required for each embodiment. In the figures, like symbols typically identify like components unless the context dictates otherwise. Furthermore, unless otherwise noted, the figures are not drawn to scale and are used for illustrative purposes only. Furthermore, the figures are merely representative, and not all components are shown. For example, additional structural or limiting components may not be shown.

[0014] Furthermore, any recitation of elements, blocks, or steps in the specification or claims is for purposes of clarity, and thus, such recitation should not be construed as requiring or implying that these elements, blocks, or steps adhere to a particular arrangement or be performed in a particular order.

[0015] I. Overview To capture an image using an image capture device, including digital cameras, smartphones, laptops, and the like, a user may power on the device and initiate a startup sequence for the image sensor (e.g., camera). The user may initiate the startup sequence by selecting an application or by simply powering on the device. The startup sequence typically includes a repetitive optical and software setting adjustment process (e.g., autofocus, autoexposure, autowhite balance). After the startup sequence is complete, the image capture device may capture an image. Ideally, the image capture device may have the ability to precisely focus the image and / or apply various autofocus processes to the image so that the captured image and / or a preview of the captured image includes a focused view of any object of interest in the image.

[0016] However, determining which area of ​​the environment to focus on (e.g., which area of ​​the environment contains the object of interest) can be complicated by the variety of objects present in the environment, many of which may be focusable areas. Furthermore, the object on which the image capture device is focused may move or be moved to a different location and / or outside the frame of the image capture device. Therefore, it may be important for the image capture device to be able to quickly switch to other focus areas. Furthermore, due to the presence of various objects in the environment, the image capture device may associate similar levels of interest with different areas, and the image capture device may fluctuate between focusing on one area and focusing on other areas having similar levels of interest.

[0017] Described herein are techniques for an image capture device to autofocus on areas of an image frame associated with high saliency while reducing instability and false positives of salient areas. In some examples, by utilizing machine learning techniques, the image capture device may detect visually salient regions within an image frame, generate one or more bounding boxes surrounding one or more visually salient regions, determine a visually salient region to focus on, and apply one or more autofill processes to the visually salient region within the image frame.

[0018] In some embodiments, the image capture device may determine a primary region of interest (ROI) and a more stable filtered ROI based on the primary ROI. For each image frame, the image capture device may first set the filtered ROI to be the same as the previous filtered ROI and update the confidence value of the filtered ROI to the average saliency value of the pixels in the updated heat map. Based on the amount of overlap and / or relative confidence of the primary ROI and the previous filtered ROI, the image capture device may determine whether to match the filtered ROI with the primary ROI or leave the filtered ROI the same as the previous filtered ROI. In some embodiments, the image capture device may determine whether the amount of overlap between the primary ROI and the previous filtered ROI does not exceed a threshold, and based on that determination, the image capture device may update the filtered ROI to the primary ROI. If the image capture device determines that the amount of overlap exceeds the threshold, the image capture device may maintain the filtered ROI. In that case, the image capture device may set the filtered ROI to be the same as the previous filtered ROI and update the confidence value of the filtered ROI to the average of the saliency values ​​at the pixels of the updated heatmap. Additionally and / or alternatively, the computing system may identify a saliency difference between the previous filtered ROI and the primary ROI, and based on the saliency difference exceeding a threshold, the computing system may update the previous filtered ROI to the primary ROI. If the computing system determines that the saliency difference does not exceed a threshold, the computing system may maintain the filtered ROI in the state of the previous ROI.

[0019] When updating the filtered ROI based on the primary ROI and the previous filtered ROI, the filtered ROI may remain the previous filtered ROI with an updated confidence value based on the updated heatmap until the region becomes less salient, at which point the filtered ROI may be updated to the primary ROI. Allowing the filtered ROI to remain the previous filtered ROI may promote stability when image frames change slightly from frame to frame, as well as when a frame has multiple objects of similar salience.

[0020] In a further embodiment, the image capture device may determine both a primary region of interest (ROI) and a secondary ROI. The secondary ROI may be determined such that the secondary ROI does not overlap the primary ROI or the prohibited area around the primary ROI. Because the computing device uses at least two ROIs of the current image frame to determine where to focus, the computing device may consider two salient objects and / or regions simultaneously, potentially making switching focus from one region to the other faster and more seamless. Furthermore, the computing device may apply a low-pass filter to the primary ROI, perhaps by applying a threshold to the saliency difference and / or the amount of overlap, before switching from the filtered ROI to the primary ROI, potentially making the filtered ROI more stable and power-efficient.

[0021] In a further embodiment, the computing device may select between the filtered ROI, the primary ROI, and the secondary ROI to determine an area within the image frame on which to focus in one or more autofocus processes. To address the potential problem that saliency detection may exhibit instability over time, a finite state machine (FSM) may be used. The FSM may require multiple consecutive frames with consistent saliency detection for autofocus to commit to a salient ROI and / or multiple consecutive frames with inconsistent saliency detection for autofocus to abandon a salient ROI. In this context, consistent detection refers to overlap of detected bounding boxes between consecutive frames and / or high reliability of these detections.

[0022] Another potential challenge is that saliency detection often reports salient regions that are not salient. This is known as a false positive, and this type of error can negatively impact the user experience, especially with cameras. For example, imagine a camera trying to focus on a shiny object in the background of a scene. While an FSM can help address this challenge because it handles momentary false positives, it is less likely to detect consecutive false positives than a single-frame false positive. In addition, to prevent false positives, the application of the saliency autofocus process can be limited based on global on-device signals. For example, certain requirements (e.g., a minimum scene luminance value, a zoom ratio within a certain range, and / or no device motion) can be imposed before autofocus is enabled. Furthermore, certain salient regions can be discarded when the estimated depth is out of range, when the estimated depth differs too much from the estimated distance of the current ROI, and / or when the bounding box location is too far from the center.

[0023] II. Exemplary Systems and Methods FIG. 1 illustrates an exemplary computing device 100. Computing device 100 is shown in a mobile phone form factor. However, computing device 100 may alternatively be implemented in a laptop computer, a tablet computer, and / or a wearable computing device, among other possibilities. Computing device 100 may include various elements, such as a body 102, a display 106, and buttons 108 and 110. Computing device 100 may further include one or more cameras, such as a front-facing camera 104 and one or more rear-facing cameras 112. Each of the rear-facing cameras may have a different field of view. For example, the rear-facing cameras may include a wide-angle camera, a main camera, and a telephoto camera. The wide-angle camera may capture a wider portion of the environment compared to the main camera and the telephoto camera, and the telephoto camera may capture a more detailed image of a smaller portion of the environment compared to the main camera and the wide-angle camera.

[0024] The front camera 104 may be located on the side of the body 102 that normally faces the user during operation (e.g., the same side as the display 106). The rear camera 112 may be located on the side of the body 102 opposite the front camera 104. The designation of the cameras as front and rear is arbitrary, and the computing device 100 may include multiple cameras located on various sides of the body 102.

[0025] Display 106 may represent a cathode ray tube (CRT) display, a light emitting diode (LED) display, a liquid crystal (LCD) display, a plasma display, an organic light emitting diode (OLED) display, or any other type of display known in the art. In some embodiments, display 106 may display digital representations of a current image being captured by front camera 104 and / or rear camera 112, an image to be captured by one or more of these cameras, an image recently captured by one or more of these cameras, and / or a modified version of one or more of these images. Thus, display 106 may function as a camera viewfinder. Display 106 may also support touchscreen functionality, which may allow settings and / or configurations of one or more aspects of computing device 100 to be adjusted.

[0026] The front-facing camera 104 may include an image sensor and associated optical elements, such as a lens. The front-facing camera 104 may provide zoom capabilities or have a fixed focal length. In other embodiments, interchangeable lenses may be used with the front-facing camera 104. The front-facing camera 104 may have a variable mechanical aperture and a mechanical and / or electronic shutter. The front-facing camera 104 may also be configured to capture still images, video images, or both. Furthermore, the front-facing camera 104 may represent, for example, a monocular camera, a stereoscopic camera, or a multi-lens camera. The rear-facing camera 112 may be similarly or differently positioned. Furthermore, one or more of the front-facing camera 104 and / or the rear-facing camera 112 may be an array of one or more cameras.

[0027] One or more of the front-facing camera 104 and / or rear-facing camera 112 may include or be associated with an illumination component that provides a light field that illuminates the target object. For example, the illumination component may provide flash or constant illumination of the target object. The illumination component may also be configured to provide a light field that includes one or more of structured light, polarized light, and light with specific spectral content. In the context of the examples herein, it is known to reconstruct three-dimensional (3D) models from objects, and other types of light fields that may be used in this manner are possible.

[0028] Computing device 100 may also include an ambient light sensor that may continuously or occasionally determine the ambient lighting of a scene that may be captured by cameras 104 and / or 112. In some implementations, the ambient light sensor may be used to adjust the display brightness of display 106. Additionally, the ambient light sensor may be used to determine or facilitate the determination of an exposure length for one or more of cameras 104 or 112.

[0029] Computing device 100 may be configured to capture images of an object of interest using display 106 and front camera 104 and / or rear camera 112. The captured images may be multiple still images or a video stream. Image capture may be triggered by activating button 108, by pressing a soft key on display 106, or by some other mechanism. Depending on the implementation, images may be captured automatically at specific time intervals, for example, by pressing button 108, by moving computing device 100 a predetermined distance in suitable lighting conditions for the object of interest, or according to a predetermined capture schedule.

[0030] 2 is a simplified block diagram illustrating some of the components of an exemplary computing system 200. By way of example and not limitation, computing system 200 may include a cellular mobile phone (e.g., a smartphone), a computer (such as a desktop, notebook, tablet, server, or handheld computer), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, a game console, a robotic device, a vehicle, or some other type of device. Computing system 200 may represent, for example, an aspect of computing device 100.

[0031] 2, computing system 200 may include a communications interface 202, a user interface 204, a processor 206, data storage 208, and a camera component 224, all of which may be communicatively linked together by a system bus, network, or other connection mechanism 210. Computing system 200 may be equipped with at least some image capture and / or image processing functionality. It should be understood that computing system 200 may represent a physical image processing system, or a specific physical hardware platform on which image sensing and / or processing applications run in software, or other combinations of hardware and software configured to perform image capture and / or processing functions.

[0032] Communications interface 202 may enable computing system 200 to communicate with other devices, access networks, and / or transport networks using analog or digital modulation. Thus, communications interface 202 may facilitate circuit-switched and / or packet-switched communications, such as plain old telephone service (POTS) communications and / or Internet Protocol (IP), or other packetized communications. For example, communications interface 202 may include a chipset and antenna arranged for wireless communication with a wireless access network or access point. Communications interface 202 may also take the form of or include a wired interface, such as an Ethernet, Universal Serial Bus (USB), or High-Definition Multimedia Interface (HDMI) port, among other possibilities. Communications interface 202 may also take the form of or include a wireless interface, such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or 3GPP Long Term Evolution (LTE)), among other possibilities. However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used via communication interface 202. Additionally, communication interface 202 may include multiple physical communication interfaces (e.g., a Wi-Fi interface, a BLUETOOTH interface, and a wide area wireless interface).

[0033] The user interface 204 may function to enable the computing system 200 to interact with a human or non-human user, such as by receiving input from the user and providing output to the user. Thus, the user interface 204 may include input components such as a keypad, keyboard, touch-sensitive panel, computer mouse, trackball, joystick, and microphone. The user interface 204 may also include one or more output components, such as a display screen, which may be combined with a touch-sensitive panel. The display screen may be based on CRT, LCD, LED, and / or OLED technology, or other technologies now known or later developed. The user interface 204 may also be configured to generate audible output(s) via speakers, speaker jacks, audio output ports, audio output devices, earphones, and / or other similar devices. The user interface 204 may also be configured to receive and / or capture audible speech(es), noise(s), and / or signal(s) via a microphone and / or other similar devices.

[0034] In some embodiments, user interface 204 may include a display that functions as a viewfinder for still camera and / or video camera functions supported by computing system 200. Additionally, user interface 204 may include one or more buttons, switches, knobs, and / or dials that facilitate configuration and focus of camera functions and image capture. Some or all of these buttons, switches, knobs, and / or dials may be capable of being implemented with a touch-sensitive panel.

[0035] The processor 206 may comprise one or more general-purpose processors, such as microprocessors, and / or one or more special-purpose processors, such as digital signal processors (DSPs), graphics processing units (GPUs), floating-point units (FPUs), network processors, or application-specific integrated circuits (ASICs). In some cases, the special-purpose processors may be capable of image processing, image alignment, and image merging, among other possibilities. The data storage 208 may include one or more volatile and / or non-volatile storage components, such as magnetic storage, optical storage, flash storage, or organic storage, and may be wholly or partially integrated with the processor 206. The data storage 208 may include removable and / or non-removable components.

[0036] Processor 206 may be capable of executing program instructions 218 (e.g., compiled or uncompiled program logic and / or machine code) stored on data storage 208 to perform various functions described herein. Thus, data storage 208 may include a non-transitory computer-readable medium having program instructions stored thereon that, when executed by computing system 200, cause computing system 200 to perform any of the methods, processes, or operations disclosed herein and / or in the accompanying drawings. Execution of program instructions 218 by processor 206 may cause processor 206 to use data 212.

[0037] By way of example, program instructions 218 may include an operating system 222 (e.g., an operating system kernel, device driver(s), and / or other modules) and one or more application programs 220 (e.g., camera functionality, address book, email, web browsing, social networking, audio-to-text functionality, text translation functionality, and / or gaming applications) installed on computing system 200. Similarly, data 212 may include operating system data 216 and application data 214. Operating system data 216 may be primarily accessible by operating system 222, and application data 214 may be primarily accessible by one or more of application programs 220. Application data 214 may be located in a file system visible to a user of computing system 200 or in a file system hidden from the user.

[0038] Application programs 220 may communicate with operating system 222 via one or more application programming interfaces (APIs). These APIs may facilitate, for example, application programs 220 to read and / or write application data 214, send or receive information via communications interface 202, receive and / or display information on user interface 204, etc.

[0039] In some instances, application program 220 may be referred to as an "app" for short. Additionally, application program 220 may be downloadable to computing system 200 via one or more online application stores or application markets. However, application programs may also be installed on computing system 200 in other ways, such as via a web browser or through a physical interface (e.g., a USB port) of computing system 200.

[0040] The camera component 224 may include, but is not limited to, an aperture, a shutter, a recording surface (e.g., photographic film and / or an image sensor), a lens, a shutter button, an infrared projector, and / or a visible light projector. The camera component 224 may include, among other possibilities, components configured to capture images in the visible light spectrum (e.g., electromagnetic radiation having wavelengths between 380 and 700 nanometers) and / or components configured to capture images in the infrared light spectrum (e.g., electromagnetic radiation having wavelengths between 701 nanometers and 1 millimeter). The camera component 224 may be controlled, at least in part, by software executed by the processor 206.

[0041] FIG. 3 shows a diagram 300 illustrating the training phase 302 and inference phase 304 of trained machine learning model(s) 332, according to an example embodiment. Some machine learning techniques involve training one or more machine learning algorithms with an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions regarding (the patterns in) the training data. The resulting trained machine learning algorithms may be referred to as trained machine learning models. For example, FIG. 3 shows a training phase 302 in which one or more machine learning algorithms 320 are trained with training data 310 to become trained machine learning models 332. Generating the trained machine learning model(s) 332 during the training phase 302 may include determining one or more hyperparameters, such as one or more stride values ​​for one or more layers of the machine learning model, as described herein. Next, during the inference phase 304, the trained machine learning model 332 may receive input data 330 and one or more inference / prediction requests 340 (perhaps as part of the input data 330) and, in response, provide as output one or more inferences and / or predictions 350. The one or more inferences and / or predictions 350 may be based in part on one or more learned hyperparameters, such as one or more learned stride values ​​for one or more layers of the machine learning model, as described herein.

[0042] Thus, the trained machine learning model(s) 332 may include one or more models of one or more machine learning algorithms 320. The machine learning algorithm(s) 320 may include, but are not limited to, artificial neural networks (e.g., convolutional neural networks described herein, recurrent neural networks, Bayesian networks, hidden Markov models, Markov decision processes, logistic regression functions, support vector machines, suitable statistical machine learning algorithms, and / or heuristic machine learning systems). The machine learning algorithm(s) 320 may be supervised or unsupervised and may perform any suitable combination of online and offline learning.

[0043] In some examples, the machine learning algorithm(s) 320 and / or the trained machine learning model(s) 332 may be accelerated using on-device coprocessors, such as graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and / or application-specific integrated circuits (ASICs). Using such on-device coprocessors, the machine learning algorithm(s) 320 and / or the trained machine learning model(s) 332 may be accelerated. In some examples, the trained machine learning model(s) 332 may be trained, reside, execute to provide inference on a particular computing device, and / or may otherwise perform inference for a particular computing device.

[0044] During the training phase 302, the machine learning algorithm(s) 320 may be trained by providing at least the training data 310 as training inputs using unsupervised, supervised, semi-supervised, and / or reinforcement learning techniques. Unsupervised learning involves providing some (or all) of the training data 310 to the machine learning algorithm(s) 320, and the machine learning algorithm(s) 320 determining one or more output inferences based on the provided portion (or all) of the training data 310. Supervised learning involves providing some (or all) of the training data 310 to the machine learning algorithm(s) 320, and the machine learning algorithm(s) 320 determining one or more output inferences based on the provided portion (or all) of the training data 310, where the output inference(s) are either accepted or corrected based on the correct results associated with the training data 310. In some embodiments, the supervised learning of the machine learning algorithm(s) 320 may be governed by a set of rules and / or a set of labels for the training inputs, and the set of rules and / or the set of labels may be used to correct the inferences of the machine learning algorithm(s) 320.

[0045] Semi-supervised learning involves having correct results for some, but not all, of the training data 310. During semi-supervised learning, supervised learning is used for the portions of the training data 310 that have correct results, and unsupervised learning is used for the portions of the training data 310 that do not have correct results.

[0046] Reinforcement learning involves machine learning algorithm(s) 320 receiving a reward signal for a previous inference, where the reward signal may be a numerical value. During reinforcement learning, machine learning algorithm(s) 320 may output an inference and receive a reward signal in response thereto, where the machine learning algorithm(s) 320 are configured to attempt to maximize the numerical value of the reward signal. In some embodiments, reinforcement learning also utilizes a value function that provides a numerical value representing the expected sum of the numerical values ​​provided by the reward signal over time. In some embodiments, machine learning algorithm(s) 320 and / or trained machine learning model(s) 332 may be trained using other machine learning techniques, including, but not limited to, incremental learning and curriculum learning.

[0047] In some embodiments, the machine learning algorithm(s) 320 and / or the trained machine learning model(s) 332 may use transfer learning techniques. For example, transfer learning techniques may include the trained machine learning model(s) 332 being pre-trained on a dataset and further trained using the training data 310. More specifically, the machine learning algorithm(s) 320 may be pre-trained on data from one or more computing devices, and the resulting trained machine learning model(s) may be provided to the computing device CD1, which is intended to execute the trained machine learning model during the inference phase 304. Then, during the training phase 302, the pre-trained machine learning model(s) may be further trained using the training data 310. This further training of the machine learning algorithm(s) 320 and / or the pre-trained machine learning model(s) using the training data 310 of the data from CD1 may be performed using either supervised learning or unsupervised learning. Once the machine learning algorithm(s) 320 and / or the pre-trained machine learning model(s) have been trained on at least the training data 310, the training phase 302 may be complete. The resulting machine learning model after training may be used as at least one of the trained machine learning model(s) 332.

[0048] Specifically, once the training phase 302 is complete, the trained machine learning model(s) 332 may be provided to the computing device if not already present on the computing device. After the trained machine learning model(s) 332 are provided to the computing device CD1, the inference phase 304 may begin.

[0049] During the inference phase 304, the trained machine learning model(s) 332 may receive the input data 330 and generate and output one or more corresponding inferences and / or prediction(s) 350 regarding the input data 330. Thus, the input data 330 may be used as input to the trained machine learning model(s) 332 to provide the corresponding inference(s) and / or prediction(s) 350. For example, the trained machine learning model(s) 332 may generate the inference(s) and / or prediction(s) 350 in response to one or more inference / prediction requests 340. In some embodiments, the trained machine learning model(s) 332 may be executed by another piece of software. For example, the trained machine learning model(s) 332 may be executed by an inference or prediction daemon so that they are readily available to provide inferences and / or predictions upon request. The input data 330 may include data from the computing device CD1 running the trained machine learning model(s) 332 and / or input data from one or more computing devices other than CD1.

[0050] The exemplary image capture devices described herein may include, among other components, one or more cameras and sensors. The image capture device may be a smartphone, tablet, laptop, or digital camera, among other types of computing devices capable of performing the operations described herein.

[0051] As an example, a computing device may include one or more processors having logic for executing instructions, at least one integrated or peripheral image sensor (e.g., a camera), and an input / output device (e.g., a display panel) for displaying a user interface. The computing device may further include a computer-readable medium (CRM). The CRM may include any suitable memory or storage, such as random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), non-volatile RAM (NVRAM), read-only memory (ROM), or flash. The computing device stores device data (e.g., user data, multimedia data, applications, and / or the device's operating system) in the CRM. The device data may include executable instructions for an auto-zoom process. The auto-zoom process may be part of a process that the operating system executes for the image capture device, or may be a process that a separate component executes within an application environment (e.g., a camera application) or within a "framework" provided by the operating system.

[0052] The computing device may implement machine-learned techniques (“visual saliency models”). The visual saliency models may be implemented with one or more of a support vector machine (SVM), a recurrent neural network (RNN), a convolutional neural network (CNN), a dense neural network (DNN), one or more heuristics, other machine learning techniques, combinations thereof, and the like. The visual saliency models may be iteratively trained off-device, given training scenes, training sequences, and / or training events. For example, training may include providing the visual saliency model with images (e.g., digital photographs) that include user-drawn bounding boxes surrounding visually salient regions (e.g., regions where one or more objects of particular interest to the user may reside). In some embodiments, these images may include bounding boxes or heat maps, which are generated by tracking the annotator's eyes while viewing the images to determine which areas of the image are most salient. Additionally, in some embodiments, the visual saliency models may be trained with heat maps that may be generated by tracking the location of the image where the annotator is looking. For example, if five annotators view a first area and ten annotators view a second area, the first area in the image may be determined to be half as salient as the second area in the image. Providing an image containing user-drawn bounding boxes may facilitate training the visual saliency model to identify visually salient regions in the image. As a result of training, the visual saliency model may generate a visual saliency heat map for the given image. The computing device may then generate various bounding boxes based on the visual saliency heat map, including a bounding box enclosing the region with the highest visual saliency score. In this manner, the visual saliency model can predict visually salient regions in the image. After sufficient training, the visual saliency model may undergo model compression using distillation, which allows for selection of an optimal model architecture based on model latency and power consumption.The visual saliency model may then be deployed as a separate module in the CRM of the computing device or may be implemented in the automatic zoom process.

[0053] The computing device may perform the auto-zoom process, perhaps automatically or in response to a received trigger signal, such as a user-performed gesture (e.g., tap, press) made on the input / output device, etc. The computing device may receive one or more captured images from the image sensor.

[0054] The computing device may utilize a visual saliency model to generate a visual saliency heatmap using one or more captured images.

[0055] For example, FIG. 4a illustrates an image 400 according to an exemplary embodiment. FIG. 4b illustrates a heatmap 450 according to an exemplary embodiment. As illustrated in FIGS. 4a and 4b, a computing device may utilize a visual saliency model to generate a visual saliency heatmap for a captured image. One or more processors may calculate the visual saliency heatmap during background operation of the device. In some examples, the image capture device may not display the visual saliency heatmap to a user. Illustratively, the visual saliency heatmap represents the magnitude of visual saliency probability on a scale from black to white, with white indicating a high saliency probability and black indicating a low saliency probability. Each pixel in the visual saliency heatmap may be assigned a saliency metric, which represents the degree of saliency of the region represented by the metric, based on the computing device applying a pre-trained machine learning model to the image frame, where the pre-trained machine learning model outputs a saliency metric for each pixel in the visual saliency heatmap.

[0056] The visual saliency model may generate a bounding box that surrounds the region where the visual saliency probability is greatest. Figure 5 shows a heatmap 500 with a bounding box 502, according to an example embodiment.

[0057] 5, a visual saliency heatmap includes a bounding box 502 that surrounds regions in an image that have a maximum visual saliency probability. A visual saliency model may be trained to output the heatmap, and a computing device may use the heatmap to determine one or more objects of interest in the captured image and to generate one or more bounding boxes around those objects of interest. These generated bounding boxes may include one or more regions predicted by the computing device to be salient.

[0058] In some examples, a computing device may experiment with anchor boxes of various sizes and various positions, as described in Faster R-CNN by Ren et al., 2016. Specifically, as described herein, anchor boxes may be considered to determine regions of interest, e.g., areas with high or highest average saliency values. For example, FIG. 6A shows anchor bounding box sizes 602, 604, and 606 according to an example embodiment. As FIG. 6A shows, a computing device may evaluate anchor bounding boxes of different aspect ratios (e.g., 1:2, 1:1, and 2:1), and for each of the different aspect ratios, the computing device may evaluate anchor bounding boxes of various sizes. For the various anchor bounding box positions, the computing device may determine an average saliency value for the anchor bounding boxes of each aspect ratio and size. Specifically, each pixel in the heatmap may be associated with a saliency value, and the computing device may determine the average of the saliency values ​​of the pixels within the anchor bounding boxes to determine the average saliency value.

[0059] FIG. 6B illustrates anchor bounding box positions according to an example embodiment. As FIG. 6B shows, the computing device may evaluate average saliency values ​​of anchor bounding boxes of various sizes and aspect ratios every few pixels. The centers of the anchor bounding boxes may be equally spaced based on a stride value, and at each position, the computing device may evaluate various sizes and aspect ratios of anchor bounding boxes. For example, the computing device may determine average saliency values ​​of anchor bounding boxes for nine anchor bounding box sizes 602, 604, and 606 with aspect ratios of 1:2, 1:1, and 2:1 at positions 650, 652, and 654, among other positions, separated by a stride of three pixels. Based on the average saliency values ​​of the anchor bounding boxes, the computing device may determine one or more regions of interest (ROIs).

[0060] 7 illustrates salient ROIs according to an exemplary embodiment. In an exemplary process, a computing device may determine a saliency heatmap 700 according to the process described above, perhaps using a machine learning model or other algorithm that predicts a saliency metric at each pixel of an image frame. Based on the saliency heatmap 700, the computing device may determine a primary ROI with the highest average saliency, perhaps by calculating the average saliency value of pixels within various anchor bounding boxes as previously described in connection with FIGS. 6A-6B.

[0061] Next, the computing device may determine a secondary ROI based on the primary ROI, where the secondary ROI may be an ROI having pixels with a lower average saliency value compared to the primary ROI. The secondary ROI may be required to be at least a threshold distance away from the primary ROI. As shown in image 704, the computing device may determine the secondary ROI based on a keep-out zone 714. The keep-out zone 714 may be an area around the primary ROI 712 within which the secondary ROI cannot be located. The computing device may determine the secondary ROI 716 such that the secondary ROI 716 does not overlap with the primary ROI 712 or the keep-out zone 714. In some embodiments, the computing device may apply the aforementioned anchor bounding box method to pixels outside the primary ROI 712 and the keep-out zone 714 to determine the secondary ROI 716. Specifically, the primary ROI 712 may be the region with the highest average saliency, and the secondary ROI 716 may be the region with the second-highest average saliency, subject to the aforementioned constraints.

[0062] The computing system may then determine a more stable filtered ROI based on the primary ROI 712 and the secondary ROI 716, which may then potentially be used for one or more autofocus processes. For example, Figure 8 shows an image 800 having ROIs 802, 804, and 806, according to an example embodiment. Image 800 may include primary ROI 806, secondary ROI 802, and previous filtered ROI 804.

[0063] The primary ROI 806 may include an area in the image frame that is most salient, which the computing device may determine using the anchor bounding box method described above. The secondary ROI 802 may include an area in the image frame that is less salient than the primary ROI 806. The previous filtered ROI 804 may consist of an area in the previous image frame that the computing device previously determined based on the previous primary ROI.

[0064] Based on the primary ROI 806 and the previous filtered ROI 804, the computing device may determine a new filtered ROI for the area within the image frame. The filtered ROI may then be used to apply one or more autofocus processes. For example, the computing device may determine a confidence value for the previous filtered ROI 804 and a confidence value for the primary ROI 806. Specifically, the computing device may determine the confidence value for the previous filtered ROI 804 based on an average saliency value for the previous filtered ROI calculated from a saliency heat map of the image frame, even though the previous filtered ROI 804 is associated with the previous image frame. The confidence value for the primary ROI may be based on the average saliency values ​​for each ROI calculated from the saliency heat map generated from the image frame. The computing device may determine that the saliency difference between the previous filtered ROI 804 and the primary ROI 806 exceeds a threshold (e.g., the confidence value for the primary ROI 806 exceeds the confidence value of the previous filtered ROI 804 by a threshold). Based on this determination, the computing device may update the filtered ROI from the previous filtered ROI 804 to the primary ROI 806. Additionally and / or alternatively, if the computing device determines that the saliency difference between the previous filtered ROI 804 and the primary ROI 806 does not exceed a threshold, the computing device may maintain the previous filtered ROI.

[0065] In some alternative embodiments, the computing device may also consider the secondary ROI 802 and determine that the saliency difference between the previous filtered ROI 804 and the secondary ROI 802 exceeds a threshold (e.g., the confidence value of the secondary ROI 802 exceeds the confidence value of the previous filtered ROI 804 by a threshold), and the computing device may update the filtered ROI from the previous filtered ROI 804 to the secondary ROI 802. Other considerations, such as the stability of the primary ROI, the secondary ROI, and / or the previous filtered ROI, may also be taken into account.

[0066] In some embodiments, the computing device may determine the filtered ROI based on the amount that either the primary ROI 806 and / or the secondary ROI 802 overlaps with the previous filtered ROI 804. Specifically, the computing system may determine a first amount of overlap of the previous filtered ROI 804 with the primary ROI 806 and a second amount of overlap of the secondary ROI 802 with the previous filtered ROI 804. If both the first amount of overlap and the second amount of overlap do not exceed a threshold, the computing device may update the filtered ROI to either the primary ROI 806 or the secondary ROI 802 based on which of the primary ROI 806 and the secondary ROI 802 is associated with a larger average saliency value. If the overlap is small (e.g., the overlap does not exceed a threshold), it may indicate that the impact of updating the filtered ROI may be large, while if the overlap is large (e.g., the overlap is above a threshold), it may indicate that the impact of updating the filtered ROI may be small. When the filtered ROI has a large overlap with another position, updating the filtered ROI to the other position may result in a poor user experience because the area of ​​focus may change rapidly over time. In a further embodiment, both the saliency difference and the amount of overlap may be considered when determining whether to update the filtered ROI.

[0067] In some embodiments, the computing system may determine whether to update the filtered ROI from the primary ROI to the secondary ROI based on whether the previous filtered ROI overlaps with the primary ROI or the secondary ROI. For example, the computing device may update the filtered ROI to the secondary ROI when the saliency difference between the primary ROI and the secondary ROI does not exceed a threshold and when the secondary ROI overlaps with the previous ROI or overlaps by at least a threshold amount. Alternatively, when the saliency difference between the primary ROI and the secondary ROI exceeds a threshold, the computing device may determine to update the filtered ROI to the primary ROI.

[0068] After determining the filtered ROI, the computing device may apply one or more autofocus processes to the image frame. In some examples, applying the one or more autofocus processes may include adjusting a camera lens so that the lens focuses on the filtered ROI. In further examples, the computing device may apply blur to areas of the image frame outside the filtered ROI, perhaps by artificially blurring the background of the image frame, thereby causing the focus of the image frame to be on areas within the filtered ROI.

[0069] As noted above, in some implementations, saliency detection may become unstable over time and may report salient regions that are not actually salient. For example, a computing system may alternate between two regions of approximately equal saliency, which may result in periodic and / or random out-of-focus image frames. Out-of-focus image frames may make it difficult to run further algorithms (e.g., classifiers that detect objects in images, perhaps to make the images more easily searchable) and may result in a poor user experience.

[0070] To facilitate determining which image frame to focus, the computing device may determine whether to apply one or more autofocus processes based on the filtered ROI being associated with a particular state of a finite state machine. Figure 9 illustrates a finite state machine 900, according to an exemplary embodiment. The finite state machine 900 may help facilitate avoiding momentary false positives.

[0071] 9 , the finite state machine 900 includes a committed state 902, a pending state 904, a standby state 906, and a probation state 908. The committed state 902 may indicate that the filtered ROI is available, the pending state 904 may indicate that the filtered ROI is awaiting stability verification, the probation state 908 may indicate that the filtered ROI is available but awaiting unsuccessful stability verification, and the standby state 906 indicates that the filtered ROI is unavailable. In the standby state 906, the computing device may choose not to proceed with the application of one or more autofocus processes based on the ROI, while in the committed state 902 or the probation state 908, the computing device may choose to proceed with the application of one or more autofocus processes. In the pending state 904, the computing device may choose not to proceed with the application of one or more autofocus processes, perhaps until the ROI is verified to be stable or not stable.

[0072] For an image frame that includes a primary ROI, a secondary ROI, and a filtered ROI, the computing device may assign a state to each of the ROIs, which may be updated each time a new primary ROI, secondary ROI, and / or filtered ROI is determined. For example, if the primary ROI, secondary ROI, or filtered ROI is each associated with a confidence measure (e.g., an average saliency value) that does not exceed a threshold, the computing device may update the respective ROI state from a pending state 904 to a standby state 906, from a committed state 902 to a probation state 908, and / or from the probation state 908 to the standby state 906.

[0073] Additionally, if the primary ROI, secondary ROI, or filtered ROI is inconsistent and / or inconsistent with any of the previous primary ROI, previous secondary ROI, and / or previous filtered ROI, the computing device may update the state of the respective ROI from a pending state 904 to a standby state 906, from a committed state 902 to a probation state 908, and / or from a probation state 908 to a standby state 906. In some embodiments, the consistency of a primary ROI (or secondary ROI, or filtered ROI) may be defined as having a threshold amount of overlap with the previous primary ROI and not having too abrupt changes in depth.

[0074] Furthermore, if the confidence value (e.g., the determined average saliency value of the pixels within the ROI) is above a threshold and the consistency is adequate (e.g., the amount of overlap is above a threshold, perhaps among other factors), the computing device may update the state of the respective ROI from the pending state 904 to the committed state 902, or from the probation state 908 to the committed state 902.

[0075] In some embodiments, the filtered ROI, primary ROI, and secondary ROI may function as object detectors, with each ROI indicating one or more objects. However, the filtered ROI, primary ROI, and / or secondary ROI may not necessarily detect objects, but rather may detect locations that are most salient within an image frame. For example, the filtered ROI, primary ROI, and / or secondary ROI may also track an off-center object next to a textured wall or a group of people in the background, as long as these objects are most salient. Thus, a computing system performing the methods described herein may output an off-center focus area over the textured wall or the group of people in the background, such that the focus area is moving but does not necessarily track the object.

[0076] Furthermore, a computing system performing the methods described herein may be able to quickly switch focus from one area represented by one ROI to another area represented by another ROI because the computing system identifies multiple ROIs. Thus, if an object that was located in the filtered ROI is no longer present and the filtered ROI is no longer very salient, the computing system may quickly switch to focus on the primary ROI or the secondary ROI. Furthermore, if an image frame contains two objects with similar salience in the primary ROI and the secondary ROI, a computing device performing the methods described herein may maintain stability on the primary ROI for a certain amount of image frames, perhaps before switching to the secondary ROI, rather than continuously switching between the primary ROI and the secondary ROI.

[0077] FIG. 10 illustrates a finite state machine manager 1000 according to an exemplary embodiment. The finite state manager 1000 includes a salient ROI state machine 1002 and a salient ROI state machine 1004. The finite state manager 1000 may prepare a candidate ROI (e.g., a primary ROI, a secondary ROI, a previous filtered ROI, or a filtered ROI) for input to either the salient ROI state machine 1002 or the salient ROI state machine 1004 by verifying several factors. For example, if a candidate ROI is already associated with a salient ROI state machine, the candidate ROI cannot be input to another salient ROI state machine. If an object inside the candidate ROI is not within a valid distance range, the candidate ROI may be discarded. And, if the candidate ROI is not within a valid window near the center of the image, the candidate ROI may also be discarded.

[0078] Having two salient ROI state machines can help facilitate switching between applying one or more autofocus algorithms to one area of ​​an image frame and another area of ​​the image frame. For example, if an object being focused in an image frame disappears within a short time frame and the object was associated with salient ROI state machine 1002, the computing device can check whether the state of the candidate ROI region associated with salient ROI state machine 1004 is an acceptable state. If the state is acceptable, the computing device can quickly switch to focusing on the candidate ROI.

[0079] 11 is a flowchart of a method 1100 according to an example embodiment. Method 1100 may be performed by one or more computing systems (e.g., computing system 200 of FIG. 2) and / or one or more processors (e.g., processor 206 of FIG. 2). Method 1100 may be performed on a computing device, such as computing device 100 of FIG. 1.

[0080] The method 1100 includes, at block 1102, receiving an image frame captured by an image capture device.

[0081] The method 1100 includes, at block 1104, determining a saliency heatmap representing the saliency of pixels in the image frame.

[0082] The method 1100 includes, at block 1106, determining a primary ROI and a secondary ROI for the image frame based on the saliency heatmap.

[0083] The method 1100 includes determining a filtered ROI for the image frame at block 1108. The filtered ROI is updated from the previous filtered ROI to the primary ROI based on a saliency difference between the previous filtered ROI and the primary ROI exceeding a first threshold.

[0084] The method 1100 includes, at block 1110, applying one or more autofocus processes based on at least one of the filtered ROI, the primary ROI, or the secondary ROI.

[0085] In some embodiments, determining the primary and secondary ROIs is based on the primary ROI having a greater average saliency than the secondary ROI.

[0086] In some embodiments, the filtered ROI is updated from the previous filtered ROI to the primary ROI further based on both the first overlap amount of the previous filtered ROI with the primary ROI and the second overlap amount of the previous filtered ROI not exceeding a second threshold.

[0087] In some embodiments, once the filtered ROI is set to the previous filtered ROI, the filtered ROI is associated with an updated average saliency value based on the saliency heatmap.

[0088] In some embodiments, determining the primary ROI and the secondary ROI based on the saliency heatmap includes determining a plurality of candidate anchor boxes distributed on the saliency heatmap, each of the candidate anchor boxes being associated with a saliency measure, and determining the primary ROI and the secondary ROI based on the saliency measures of each of the candidate anchor boxes.

[0089] In some implementations, the plurality of candidate anchor boxes includes a plurality of anchor boxes having a plurality of different aspect ratios at a given location within the image frame.

[0090] In some embodiments, the candidate anchor boxes are evenly distributed on the saliency heatmap.

[0091] In some implementations, the plurality of candidate anchor boxes includes a plurality of anchor boxes having a plurality of sizes at a given location within the image frame.

[0092] In some embodiments, based on the saliency heatmap, the primary ROI is associated with a primary confidence measure, the previous filtered ROI is associated with a filtered confidence measure, and the saliency difference is based on the primary confidence measure and the filtered confidence measure.

[0093] In some embodiments, the primary reliability measure is based on an average of one or more saliency values ​​at one or more pixels in the primary ROI, and the filtered reliability measure is based on an average of one or more saliency values ​​at one or more pixels in the previous filtered ROI.

[0094] In some embodiments, determining the primary and secondary ROIs is based on the primary ROI being at least a threshold distance away from the secondary ROI.

[0095] In some embodiments, determining the primary ROI and the secondary ROI includes determining the primary ROI based on a saliency heat map, determining a keep-out region around the primary ROI, and determining the secondary ROI based on the keep-out region around the primary ROI and the saliency heat map such that the secondary ROI is not within the primary ROI or within the keep-out region around the primary ROI.

[0096] In some embodiments, the previous filtered ROI is based on a previous image frame that was captured before the image frame.

[0097] In some examples, determining a saliency heatmap representing the saliency of each pixel in the image frame includes applying a pre-trained machine learning model to the image frame to determine the saliency heatmap.

[0098] In some embodiments, applying one or more autofocus processes includes causing a camera lens to adjust focus on the filtered ROI.

[0099] In some embodiments, applying one or more autofocus processes includes applying a blur to a region of the image frame outside the filtered ROI.

[0100] In some embodiments, the method 1100 further includes applying a finite state machine to the filtered ROI, wherein applying the one or more autofocus processes is based on the filtered ROI being associated with a particular state of the finite state machine.

[0101] In some embodiments, the finite state machine includes a committed state indicating that the filtered ROI is available, a pending state indicating that the filtered ROI is awaiting stability verification, a probation state indicating that the filtered ROI is available but awaiting unsuccessful stability verification, and a standby state indicating that the filtered ROI is unavailable.

[0102] In some embodiments, the method 1100 further includes updating a state associated with the filtered ROI, where updating the state associated with the filtered ROI includes updating the state from a pending state to a standby state, from a committed state to a probation state, or from a probation state to a standby state based on a determination that the reliability measure associated with the filtered ROI does not exceed a second threshold.

[0103] In some embodiments, the method 1100 further includes updating a state associated with the filtered ROI, where updating the state associated with the filtered ROI includes updating the state from a pending state to a standby state, from a committed state to a probation state, or from a probation state to a standby state based on a determination that the filtered ROI does not overlap with a previous filtered ROI.

[0104] In some embodiments, the particular state of the finite state machine is a committed state.

[0105] In some embodiments, the method 1100 further includes applying a finite state machine to each of the filtered ROI, the primary ROI, and the secondary ROI, and applying the one or more autofocus processes is based on the respective states of the finite state machines associated with each of the filtered ROI, the primary ROI, and the secondary ROI.

[0106] In some embodiments, the method 1100 is performed by an image capture device including a camera and a control system configured to perform the steps of the method 1100 .

[0107] In such an embodiment, the image capture device is a mobile device and the image frames are captured by a camera.

[0108] In some embodiments, a method may include receiving an image frame captured by an image capture device. The method may also include determining a saliency heatmap representing the saliency of pixels in the image frame. The method may further include determining a primary ROI based on the saliency heatmap. The method may further include determining a keep-out region around the primary ROI. The method may also include determining a secondary ROI based on the saliency heatmap such that the secondary ROI does not overlap with the primary ROI and does not overlap with the keep-out region. The method may further include controlling one or more autofocus processes based on the primary ROI and the secondary ROI.

[0109] III. Conclusion The present disclosure is not limited with respect to the specific embodiments described herein, which are intended to be illustrative of various aspects. As will be apparent to those skilled in the art, many modifications and variations may be made without departing from its scope. In addition to the methods and apparatus described herein, functionally equivalent methods and apparatuses within the scope of the present disclosure will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to be included within the scope of the appended claims.

[0110] The above detailed description describes various features and operations of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the drawings, like numerals generally identify like components unless otherwise dictated by context. The exemplary embodiments described in the specification and drawings are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein and illustrated in the drawings, may be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

[0111] With respect to any or all of the message flow diagrams, scenarios, and flowcharts shown in the figures and discussed herein, each step, block, and / or communication may represent the processing of information and / or the transmission of information according to the example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages may be executed in an order different from that shown or discussed, including substantially simultaneously or in reverse order, depending on the functionality involved. Furthermore, more or fewer blocks and / or operations may be used in any of the message flow diagrams, scenarios, and flowcharts discussed herein, and these message flow diagrams, scenarios, and flowcharts may be combined with each other, either partially or in whole.

[0112] Steps or blocks representing the processing of information may correspond to circuitry that can be configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, blocks representing the processing of information may correspond to modules, segments, or portions of program code (including associated data). The program code may include one or more instructions executable by a processor to perform specific logical operations or actions in the method or technique. The program code and / or associated data may be stored in any type of computer-readable medium, such as a storage device including a random access memory (RAM), a disk drive, a solid-state drive, or other storage medium.

[0113] Computer-readable media may also include non-transitory computer-readable media, such as register memory, processor cache, and computer-readable media for short-term data storage, such as RAM. Computer-readable media may also include non-transitory computer-readable media for long-term storage of program code and / or data. Thus, computer-readable media may include, for example, secondary storage or persistent long-term storage, such as a read-only memory (ROM), an optical or magnetic disk, a solid-state drive, or a compact disk read-only memory (CD-ROM). Computer-readable media may also be any other volatile or non-volatile storage system. Computer-readable media may be considered, for example, a computer-readable storage medium or a tangible storage device.

[0114] Additionally, steps or blocks representing one or more information transmissions may correspond to information transmissions between software and / or hardware modules within the same physical device, although other information transmissions may be between software and / or hardware modules of different physical devices.

[0115] The particular arrangement shown in the drawings should not be considered limiting. It should be understood that other embodiments may include more or fewer of each element shown in a given drawing. Furthermore, some of the illustrated elements may be combined or omitted. Furthermore, example embodiments may include elements not shown in the drawings.

[0116] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, the true scope of which is indicated by the following claims.

Claims

1. 1. A method comprising: receiving an image frame captured by an image capture device; determining a saliency heatmap representing the saliency of pixels within the image frame; determining a primary region of interest (ROI) and a secondary ROI for the image frame based on the saliency heatmap; determining a filtered ROI for the image frame, wherein the filtered ROI is updated from the previous filtered ROI to the primary ROI based on a saliency difference between the previous filtered ROI and the primary ROI exceeding a first threshold, and the method further comprises: applying one or more autofocus processes based on at least one of the filtered ROI, the primary ROI, or the secondary ROI; A method comprising:

2. The method of claim 1 , wherein determining the primary ROI and the secondary ROI is based on the primary ROI having a greater average saliency than the secondary ROI.

3. 2. The method of claim 1, wherein the filtered ROI is updated from the previous filtered ROI to the primary ROI further based on an amount of overlap of the previous filtered ROI with the primary ROI not exceeding a second threshold.

4. The method of claim 1 , wherein once the filtered ROI is set to the previous filtered ROI, the filtered ROI is associated with an updated average saliency value based on the saliency heatmap.

5. determining the primary ROI and the secondary ROI based on the saliency heatmap includes determining a plurality of candidate anchor boxes distributed on the saliency heatmap; Each of the candidate anchor boxes is associated with a saliency measure; The method of claim 1 , wherein determining the primary and secondary ROIs is based on the saliency measure of each of the candidate anchor boxes.

6. The method of claim 5 , wherein the plurality of candidate anchor boxes comprises a plurality of anchor boxes having a plurality of different aspect ratios at a given location within the image frame.

7. The method of claim 5 , wherein the candidate anchor boxes are evenly distributed on the saliency heatmap.

8. The method of claim 5 , wherein the plurality of candidate anchor boxes comprises a plurality of anchor boxes having a plurality of sizes at a given location within the image frame.

9. Based on the saliency heatmap, the primary ROI is associated with a primary confidence measure and the previously filtered ROI is associated with a filtered confidence measure; The method of claim 1 , wherein the saliency difference is based on the primary confidence measure and the filtered confidence measure.

10. 10. The method of claim 9, wherein the primary confidence measure is based on an average of one or more saliency values ​​at one or more pixels in the primary ROI, and the filtered confidence measure is based on an average of one or more saliency values ​​at one or more pixels in the previously filtered ROI.

11. The method of claim 1 , wherein determining the primary ROI and the secondary ROI is based on the primary ROI being at least a threshold distance from the secondary ROI.

12. Determining the primary ROI and the secondary ROI includes: determining the primary ROI based on the saliency heatmap; and determining a keep-out region around the primary ROI; determining a secondary ROI based on the forbidden region around the primary ROI and the saliency heatmap, such that the secondary ROI is not within the primary ROI or within the forbidden region around the primary ROI; The method of claim 1 , comprising:

13. The method of claim 1 , wherein the previous filtered ROI is based on a previous image frame captured before the image frame.

14. 2. The method of claim 1 , wherein determining a saliency heatmap representing the saliency of each pixel in the image frame comprises applying a pre-trained machine learning model to the image frame to determine the saliency heatmap.

15. The method of claim 1 , wherein applying the one or more autofocus processes comprises causing a camera lens to adjust focus on the filtered ROI.

16. The method of claim 1 , wherein applying the one or more autofocus processes comprises applying a blur to an area of ​​the image frame outside the filtered ROI.

17. The method further includes applying a finite state machine to the filtered ROI; The method of claim 1 , wherein applying the one or more autofocus processes is based on the filtered ROI being associated with a particular state of the finite state machine.

18. 18. The method of claim 17, wherein the finite state machine includes a committed state indicating that the filtered ROI is available for use, a pending state indicating that the filtered ROI is awaiting stability verification, a probation state indicating that the filtered ROI is available but awaiting unsuccessful stability verification, and a standby state indicating that the filtered ROI is unavailable.

19. The method further includes updating a state associated with the filtered ROI; 20. The method of claim 18, wherein updating the state associated with the filtered ROI comprises updating the state from the pending state to the standby state, from the committed state to the probation state, or from the probation state to the standby state based on a determination that a reliability measure associated with the filtered ROI does not exceed a second threshold.

20. The method further includes updating a state associated with the filtered ROI; 20. The method of claim 18, wherein updating the state associated with the filtered ROI comprises updating the state from the pending state to the standby state, from the committed state to the probation state, or from the probation state to the standby state based on a determination that the filtered ROI does not overlap with the previous filtered ROI.

21. 18. The method of claim 17, wherein the particular state of the finite state machine is a committed state.

22. The method further includes applying a finite state machine to each of the filtered ROI, the primary ROI, and the secondary ROI; 2. The method of claim 1, wherein applying the one or more autofocus processes is based on respective states of the finite state machines associated with each of the filtered ROI, the primary ROI, and the secondary ROI.

23. 1. An image capture device, comprising: A camera and a control system, receiving image frames captured by the image capture device; determining a saliency heatmap representing the saliency of pixels within the image frame; determining a primary region of interest (ROI) and a secondary ROI for the image frame based on the saliency heatmap; determining a filtered ROI for the image frame, wherein the filtered ROI is updated from the previous filtered ROI to the primary ROI based on a saliency difference between the previous filtered ROI and the primary ROI exceeding a first threshold; and the control system is further configured to: applying one or more autofocus processes based on at least one of the filtered ROI, the primary ROI, or the secondary ROI; an image capture device configured to perform

24. 24. The image capture device of claim 23, wherein the image capture device is a mobile device and the image frames are captured by the camera.

25. 1. A non-transitory computer-readable medium storing program instructions executable by one or more processors, the instructions causing the one or more processors to perform operations, the operations including: receiving an image frame captured by an image capture device; determining a saliency heatmap representing the saliency of pixels within the image frame; determining a primary region of interest (ROI) and a secondary ROI for the image frame based on the saliency heatmap; determining a filtered ROI of the image frame, wherein the filtered ROI is updated from the previous filtered ROI to the primary ROI based on a saliency difference between the previous filtered ROI and the primary ROI exceeding a first threshold, and the operations further include: applying one or more autofocus processes based on at least one of the filtered ROI, the primary ROI, or the secondary ROI; 1. A non-transitory computer-readable medium comprising:

Images