Stabilized object tracking at high magnification ratios

Abstract

An example method includes displaying a preview of an image through a display screen of an image capture device, the preview representing a field of view of the image capture device. The method includes determining a region of interest in the preview. The method includes transitioning the image capture device from a normal operating mode to a zoomed operating mode. The zoomed operating mode includes determining a motion trajectory for the region of interest based on sensor data acquired by a sensor associated with the image capture device; and generating an adjusted preview representing a zoomed portion of the field of view based on the determined motion trajectory. The adjusted preview displays the region of interest at or near a center of the zoomed portion.The method includes providing the customized preview of the portion of the field of view.

Description

GENERAL STATE OF THE ART

[0001] Many modern computing devices, including mobile phones, PCs, and tablets, incorporate image capture devices. Some image capture devices are configured with telephoto capabilities. SUMMARY

[0002] The present disclosure generally relates to stabilizing an image in a viewfinder of an image capture device at high magnification ratios. In one aspect, an image capture device may be configured to frame and track an object of interest within a narrow field of view resulting from a high magnification ratio. The image capture device is powered by a system of machine-learned components and may be configured to stabilize and maintain the frame.

[0003] In a first aspect, a computer-implemented method is provided. The method includes displaying a preview of an image through a display screen of an image capture device, representing a field of view of the image capture device. The method also includes determining a region of interest in the preview of the image. The method further includes transitioning the image capture device from a normal operating mode to a zoomed operating mode, the zoomed operating mode comprising: determining a motion trajectory for the region of interest based on sensor data collected by a sensor associated with the image capture device; and generating an adjusted preview representing a zoomed portion of the field of view based on the determined motion trajectory, the adjusted preview displaying the region of interest at or near a center of the zoomed portion.The method additionally includes providing the customized preview of the portion of the field of view through the display screen.

[0004] In a second aspect, an apparatus is provided. The apparatus includes one or more processors operable to perform operations. The operations include displaying a preview of an image through a display screen of an image capture device that represents a field of view of the image capture device. The operations also include determining a region of interest in the preview of the image.The operations further include transitioning the image capture device from a normal operating mode to a zoomed operating mode, the zoomed operating mode comprising: determining a motion trajectory for the region of interest based on sensor data collected by a sensor associated with the image capture device; and generating an adjusted preview representing a zoomed portion of the field of view based on the determined motion trajectory, the adjusted preview displaying the region of interest at or near a center of the zoomed portion. The operations additionally include providing the adjusted preview of the portion of the field of view through the display screen.

[0005] In a third aspect, an article of manufacture is provided. The article of manufacture may include a non-transitory, computer-readable medium having stored thereon program instructions that, when executed by one or more processors of a computing device, cause the computing device to perform operations. The operations include displaying, by a display screen of an image capture device, a preview of an image representing a field of view of the image capture device. The operations further include determining a region of interest in the preview of the image.The operations further include transitioning the image capture device from a normal operating mode to a zoomed operating mode, the zoomed operating mode comprising: determining a motion trajectory for the region of interest based on sensor data collected by a sensor associated with the image capture device; and generating an adjusted preview representing a zoomed portion of the field of view based on the determined motion trajectory, the adjusted preview displaying the region of interest at or near a center of the zoomed portion. The operations additionally include providing the adjusted preview of the portion of the field of view through the display screen.

[0006] In a fourth aspect, a system is provided. The system includes means for displaying a preview of an image representing a field of view of the image capture device through a display screen of an image capture device; means for determining a region of interest in the preview of the image; means for transitioning the image capture device from a normal operating mode to a zoomed operating mode, the zoomed operating mode including: means for determining a motion trajectory for the region of interest based on sensor data collected by a sensor associated with the image capture device; and means for generating an adjusted preview representing a zoomed portion of the field of view based on the determined motion trajectory, the adjusted preview displaying the region of interest at or near a center of the zoomed portion;and means for providing the customized preview of the portion of the field of view through the screen.;

[0007] Other aspects, embodiments and implementations will become apparent to those skilled in the art upon reading the following detailed description, with reference, where appropriate, to the accompanying drawings. SHORT DESCRIPTION OF THE CHARACTERS Fig. 1 is a diagram illustrating an adjusted preview of a portion of a field of view, according to example embodiments. Fig. 2 is a diagram illustrating a stabilized object tracking alert notification, according to example embodiments. Fig. 3A is an example workflow for stabilized object tracking according to example embodiments. Fig. 3B is an example workflow for applying zoom stabilization according to example embodiments. Fig. 4 is an example workflow for processing consecutive frames in a hybrid tracker according to example embodiments. Fig. 5 depicts an exemplary tracking optimization process according to exemplary embodiments. Fig. 6 depicts an exemplary method for tracking optimization according to exemplary embodiments Fig. 7 depicts another exemplary tracking optimization process according to exemplary embodiments. Fig. 8 depicts an exemplary tracking optimization process for two regions of interest according to exemplary embodiments. Fig. 9 illustrates an example image with stabilized object tracking according to example embodiments. Fig. 10 illustrates another example image with stabilized object tracking according to example embodiments. Fig. 11 illustrates another example image with stabilized object tracking according to example embodiments. Fig. 12 is a diagram illustrating training and inference phases of a machine learning model, according to example embodiments. Fig. 13 depicts a distributed computing architecture according to example embodiments. Fig. 14 is a block diagram of a computing device according to example embodiments. Fig. 15 depicts a network of computing clusters arranged as a cloud-based server system, according to example embodiments. Fig. 16 is a flowchart of a method according to example embodiments. DETAILED DESCRIPTION

[0008] Example methods, apparatus, and systems are described herein. It should be understood that the terms "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration." An embodiment or feature described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Other embodiments may be utilized and other changes may be made without departing from the scope of the subject matter presented herein.

[0009] Therefore, the exemplary embodiments described herein are not to be considered limiting. Aspects of the present disclosure, as generally described herein and illustrated in the figures, may be implemented, substituted, combined, separated, and configured in a wide variety of different configurations, all of which are contemplated herein.

[0010] Furthermore, unless the context suggests otherwise, the features illustrated in the individual figures may be used in combination with one another. Therefore, the figures should generally be considered component aspects of one or more overall embodiments, with the understanding that not all illustrated features are required for every embodiment. Overview

[0011] This application relates to image stabilization using machine learning techniques, such as, but not limited to, neural network techniques. When a user of an image acquisition device previews an image at a high magnification ratio, the resulting image may be unstable, and challenges may arise in framing an object of interest within the image. Furthermore, for example, after framing the object of interest, the small field of view (FOV) resulting from the high magnification ratio may result in additional challenges in maintaining a moving frame in a smooth, continuous, and / or stable manner. This presents an image processing-related technical problem involving stabilizing the object of interest in the preview and maintaining smooth motion for the object of interest within a moving frame.For example, an image processing-related technical problem also arises, which involves smoothly transitioning an operation of the image capturing device between different modes (e.g., corresponding to different magnification ratios).

[0012] Telephoto cameras are becoming increasingly popular in flagship devices. Higher and higher optical zoom lenses combined with higher-resolution image sensors have been used to increase the maximum magnification ratio with each successive device release. Image quality at high magnification ratios has continuously improved. However, the extremely narrow FOV at a high magnification ratio (e.g., FOV of approximately 1° at 80× magnification ratio) makes it challenging to frame an object of interest within the FOV, and it is especially challenging to maintain such framing while simultaneously pressing the shutter button to capture an image. A game of hide-and-seek can result, with a user attempting to locate the object of interest within the narrow field of view.Although existing optical image stabilization (OIS) and / or electronic image stabilization (EIS) algorithms attempt to improve this situation to a limited extent, residual motion remains, which is magnified at higher magnification ratios.

[0013] For example, with an OIS limited to approximately ±0.9° and without EIS, "what you see is what you get" (WYSIWYG) is likely. However, maintaining a region of interest (ROI) within the frame can be challenging, and finding the ROI in a zoomed frame can be challenging. Even when EIS is applied (e.g., limited by noise due to a gyro and / or sensor), there may be a loss of the WYSIWYG feature and a trailing effect may appear in the image. Therefore, some smart devices configured with cameras attempt to solve this problem by reducing a maximum magnification ratio for a video mode compared to a photo mode.

[0014] In general, basic EIS can compensate for hand shake without tracking capability. However, maintaining focus on moving objects within the viewfinder at high magnification rates can be challenging. Also, for example, the trajectory of moving objects may not be smooth.

[0015] Gyro- and / or OIS-based EIS can be used as a stabilization technique in some situations. This stabilization technique is sensor-based, such as gyro sensing and / or OIS sensing. While this can compensate for camera pose changes without depending on image content, it results in a limitation of the FOV of the output stabilized image, as the edge is used to generate the stable virtual pose. At high magnification ratios, hardware limitations such as gyro noise, OIS sensing noise, OIS calibration errors, signal latency, and so on can also introduce visible residual motion.

[0016] Some techniques, in addition to the EIS techniques described above, may also involve detecting a face's location. While this approach can result in a stabilized frame along facial motion, it is limited to faces and not to general objects of interest. Like the EIS approach, it may also result in a limitation of the FOV of the output stabilized frame due to post-stabilization cropping.

[0017] The techniques described herein address these challenges by enabling a smooth photo and / or video capture and easy framing experience at high magnification ratios. This is achieved by reliably tracking the ROI and locking it at or near the center of a viewfinder, or by maintaining smooth motion within the frame. Gyro and / or OIS sensor information is used to determine a motion trajectory, but image-based ROI tracking effectively avoids hardware limitations (such as gyro noise, OIS sensor noise, OIS calibration errors, signal latency, and so on).

[0018] The techniques described herein may include aspects of image-based techniques in combination with techniques based on motion and optical image stabilization (OIS) data. A neural network, such as a convolutional neural network, may be trained and applied to perform one or more aspects as described herein. In some examples, the neural network may be arranged as an encoder / decoder neural network.

[0019] This method targets high magnification ratios and utilizes the digital zoom's cropping margin for stabilization without any additional cropping. The zoom stabilization pipeline can be configured to enable determination of a saliency map and an object tracker with respect to the entire image sensor region or the ROI-centered cropped sensor region, so that stabilization can be achieved along the ROI as long as it is within the image sensor. As described herein, saliency detection, object tracking, and optical flow are used to jointly suggest an ROI to be stabilized. Furthermore, potential frame delay is handled by a camera pipeline depth (e.g., 5 frames).Additionally, a new user interface (UI) design and / or a new user experience (UX) design are described that enable a frame-in-frame viewfinder when the zoom stabilization mode is active and the bounding box moves relative to the frame to indicate a stabilized and magnified FOV relative to the sensor area or the entire FOV. The described techniques also maintain the user-defined FOV without sacrificing an additional edge to stabilize the frame.

[0020] In one example, (a copy of) the trained neural network to detect salient objects may be located on a mobile computing device. The mobile computing device may include a camera capable of capturing an input photo or video. A user of the mobile computing device may view the input photo or video and determine that an object in the input photo or video should be tracked. The input photo or video and motion data may be provided to the trained neural network located on the mobile computing device. In response, the trained neural network may generate a predicted output showing an ROI with a bounding box.In other examples, the trained neural network is not located on the mobile computing device; rather, the mobile computing device provides the input photo or video and the motion data to a remotely located trained neural network (e.g., over the Internet or other data network). The remotely located neural network can process the input photo or video and the motion data as noted above and generate an output. Additionally, in other examples, non-mobile computing devices can use the trained neural network to stabilize object tracking in images and videos at high magnification ratios, including photos or videos not captured by a camera of the computing device.

[0021] Thus, the techniques described herein can enhance image capture devices by stabilizing images and providing a zoomed-in view, thereby improving their actual and / or perceived quality. Improving the actual and / or perceived quality of photos or videos can provide benefits to the user experience. These techniques are flexible and can be applied to a wide range of videos, both indoors and outdoors. Image stabilization techniques using neural networks

[0022] One of the key features of image stabilization is maintaining an accurate and reliable tracker for a region of interest (ROI). Minimizing tracking noise and reducing noise caused by gyroscopes and / or OIS can significantly contribute to image stabilization. Other challenges include pose changes, occlusions, and objects entering and / or leaving the sensor region. Latency issues can also arise due to delays between image processing at the hardware level and subsequent changes at the software or application level. For example, pipeline depth (e.g., five frames) can introduce delays. For a mobile camera application, additional image cropping in photo mode may not be allowed. In addition, stabilization must enable seamless transitions between multiple camera modes.

[0023] Accordingly, as described herein, a zoom stabilization mode of a mobile device can capture a steady photo and / or video at high magnification ratios by reliably tracking and locking a user's region of interest (ROI) (or an object of interest) in the center of the field of view (FOV) or smoothly moving the ROI within the image frame to assist the user in framing.

[0024] Noise from a gyroscope and / or OIS or from calibration errors can negatively impact stable tracking. Furthermore, challenges can arise from the complicated integration of a camera application with the underlying hardware layer, the salience node, Rectiface and / or the EIS node, and other components. Furthermore, for example, image quality can be poor at high magnification ratios, particularly in remosaic mode. Accordingly, the zoom stabilization mode can also be configured to support a binning transition. In some embodiments, remosaic mode can result in higher image quality under bright light conditions (e.g., outdoors) due to higher resolution compared to binning mode. In low light conditions, binning mode can provide higher image quality due to the poor noise performance of remosaic mode.

[0025] As described herein, zoom stabilization can offer several technical advantages, such as low power consumption and a low latency budget for real-time photo preview. Zoom stabilization can also be configured to handle multiple objects (e.g., animal herds). Furthermore, zoom stabilization is compatible with existing features (e.g., HDR+, longshot video, etc.).

[0026] Fig. 1 is a diagram illustrating a customized preview of a portion of a field of view, according to example embodiments. In some embodiments, a display screen of an image capture device may display a preview representing the field of view of the image capture device. For example, display screen 100A may display a field of view 105 that may include an object of interest, such as an image of a crescent moon 110. When operating at a high magnification ratio for the image capture device, the field of view 105 may be narrow, and small hand movements may cause the crescent moon 110 to fall out of the field of view 105.

[0027] Some embodiments involve determining a region of interest in the preview of the image. For example, there may be no ROI in the field of view, or the ROI may have moved out of the field of view. In such embodiments, movement of the background within the field of view may be tracked. In some embodiments, a new ROI may be detected within the field of view. For example, an ROI tracker may identify a new object. An important feature of the ROI tracker is to reliably predict what a camera user is attempting to capture. This may be achieved alone or as a combination of user input and a machine learning-based algorithm. For example, a tap-on ROI tracker in the camera application may allow a user to tap the display screen and indicate an object and / or region of interest.Additionally, for example, a saliency map may be generated using a machine learning model, where the saliency map indicates a region of interest to the user. Although existing saliency maps output a fixed-size bounding box for an ROI, the zoom stabilization described herein is configured to estimate a size of the ROI and output a corresponding bounding box for the ROI. In some embodiments, confidence in an ROI may be low. In such embodiments, background motion, a centered ROI, or a combination of both may be used to maintain smooth framing.

[0028] Some embodiments involve transitioning the image capture device from a normal operating mode to a zoomed operating mode. For example, the image may be captured at different zoom levels. At high magnification ratios, the field of view may be significantly narrower, and small movements of the camera may cause sudden changes in the image captured by the field of view. In some embodiments, a threshold magnification ratio may be used to determine whether image stabilization algorithms, such as the zoom stabilization algorithms described herein, may need to be turned on or off. For example, some cameras may be configured to switch modes at a magnification ratio of 15×. For example, the zoom stabilization mode may be turned off for magnification ratios greater than 15× and off for magnification ratios less than 15×.Additional and / or alternative magnification ratios can be used.

[0029] In some embodiments, the zoomed mode of operation may include determining an adjusted preview representing a zoomed portion of the field of view based on sensor data collected by a sensor associated with the image capture device, wherein the adjusted preview displays the region of interest at or near a center of the zoomed portion. As illustrated on display screen 100B, field of view 105 may be displayed within a view bounded by an outer frame 140. An inner frame 145 may be displayed within outer frame 140. Inner frame 145 may include the object of interest, crescent moon 110. Accordingly, display screen 100B may display an adjusted preview 115 (e.g., a magnified view) representing a zoomed portion (e.g., within inner frame 145) of the field of view (e.g.,the field of view 105 as displayed within the outer frame 140). As illustrated, the display screen 100B displays a magnified view 115 of a portion of the field of view, including a magnified view of the crescent moon 110A. Insofar as the image of the crescent moon 110 within an original field of view 105 may indicate non-smooth motion, the inner frame 145 is a stabilized image and the magnified view 115 is a stabilized zoomed view of the crescent moon 110A.

[0030] Display screen 100B may include additional features related to a camera application. For example, multiple modes may be available to a user, including a motion mode 120, a portrait mode 122, a video mode 126, and a video bokeh mode 128. As illustrated, the camera application may be in camera mode 124. Camera mode 124 may provide additional features, such as a rear icon 130 for activating a rear camera view, a shutter button 132 for capturing a preview image, and a photo stream icon 134 for accessing a database of captured images. Additionally, for example, a magnification ratio slider 138 may be displayed, and a user may move a virtual object along the magnification ratio slider 138 to select a magnification ratio.In some embodiments, a user may use the display screen to adjust the magnification ratio (e.g., by moving two fingers on the display screen 100B in an outward motion away from each other), and the magnification ratio slider 138 may automatically display the magnification ratio.

[0031] As indicated, the magnification ratio slider 138 may be set to a specific value, 30×. For a camera configured to switch operating modes with a magnification ratio of 15×, if the magnification ratio slider 138 moves beyond 15×, the camera may switch from a normal mode to a zoom stabilization mode, and image stabilization may be automatically activated. In such cases, an object of interest may be identified, and the magnified view 115, the outer frame 140, the inner frame 145, etc., may be displayed.

[0032] The camera application may also provide various user-adjustable features for adjusting one or more image properties (e.g., brightness, hue, contrast, shadows, highlights, global brightness adjustment for an entire image, local brightness adjustments for a region of interest, and so on). For example, in some embodiments, slider 136A may be provided to adjust property A, slider 136B may be provided to adjust property B, and in some embodiments, slider 136C may be provided to adjust property C.

[0033] In some embodiments, the zoomed mode of operation may include determining a motion trajectory for the region of interest based on sensor data collected by a sensor associated with the image capture device, and generating an adjusted preview representing a zoomed portion of the field of view based on the determined motion trajectory, wherein the adjusted preview displays the region of interest at or near a center of the zoomed portion.

[0034] Fig. 2 is a diagram illustrating a stabilized object tracking alert notification, according to example embodiments. Display screen 200A (or display screen 200B) may include additional features related to a camera application. For example, multiple modes may be available to a user, including a motion mode 120, a portrait mode 122, a video mode 126, and a video bokeh mode 128. As illustrated, the camera application may be in camera mode 124. Camera mode 124 may provide additional features, such as a reverse icon 130 for activating a rear camera view, a shutter button 132 for capturing a preview image, and a photo stream icon 134 for accessing a database of captured images.Additionally, for example, a magnification ratio slider 138 may be displayed, and a user may slide a virtual object along the magnification ratio slider 138 to select a magnification ratio. In some embodiments, a user may use the display screen 200A (or display screen 200B) to adjust the magnification ratio (e.g., by moving two fingers on the display screen 200A (or display screen 200B) in an outward motion away from each other), and the magnification ratio slider 138 may automatically display the magnification ratio.

[0035] As indicated, the magnification ratio slider 138 may be set to a specific value. 15× For a camera configured to change operating modes at a magnification ratio of 15×, if the magnification ratio slider 138 is moved to 15×, the camera may switch from normal mode to zoom stabilization mode, and image stabilization may be automatically activated. In such cases, an object of interest may be identified, and the magnified view 115, outer frame 140, inner frame 145, etc., may be displayed. The zoom ratios are for illustrative purposes only and may vary depending on device and / or system configurations.

[0036] At high magnification ratios, operating a camera to track a moving object can cause pause-move-pause motions, which with traditional EIS result in large residual motions. This can be caused, for example, by a lead term, E Vorsprungas described below in Eq. 1. Some embodiments include the display screen providing an image overlay that displays a representation of the zoomed portion relative to the field of view. For example, a frame-in-frame feature stabilizes the image and guides the user. Some embodiments include determining a bounding frame for the region of interest, where providing the image overlay comprises framing the region of interest within that bounding frame. As illustrated, the display screen 200A displays a field of view within the outer frame 140, and the inner frame 145 within the outer frame 140 frames the ROI, an image of the moon 150. A customized preview, such as a zoomed-in view 115 corresponding to the inner frame 145, is displayed with a zoomed-in view of the moon 150. As illustrated, the image of the moon 150 is centered within the inner frame 145.

[0037] Some embodiments include detecting that the region of interest is approaching a boundary of the image overlay. For example, the image of the moon 150 may be moving due to camera movement and / or movement of the object within the field of view. In such embodiments, the zoom stabilization mode is capable of maintaining a stabilized magnified view 115 with the image of the moon centered within the magnified view 115. In some embodiments, movement of the camera and / or the object of interest may cause the object of interest to approach the boundary of the inner frame 145, as illustrated in the display screen 200B. Although the image of the moon is centered within the magnified view 115, the image may be closer to the boundary of the inner frame 145.

[0038] Such embodiments also include providing a notification to the user indicating that the region of interest is approaching the boundary of the image overlay. For example, the boundary of the inner frame 145 may turn red, begin flashing, the device may vibrate, an audio notification may be provided, a voice prompt may be generated, an arrow may be displayed indicating a direction of movement for the camera to keep the image of the moon away from the boundary of the image overlay. Accordingly, the frame-in-frame may be configured to guide the user to locate their object of interest, and the zoom stabilization mode may be configured to provide a notification to the user in the event that the object of interest is closer to the boundary.

[0039] In general, the inner frame 145 may be configured to automatically slide with the field of view as the object moves to detect and track the salient object without requiring a user to center the object. In some embodiments, the zoom stabilization algorithm may stabilize and track the object while maintaining it at or near the center of the magnified view 115. Although the outer frame 140 represents the field of view 105, the region defined by the inner frame 145 may generally be cropped and displayed as the magnified view 115.As described herein, an object of interest may be identified and tracked, and an inner frame 145 may be determined and cropped to generate the magnified view 115, wherein an enlarged representation of the object of interest is displayed at or near the center of the magnified view 115 while maintaining a stabilized image with smooth motion. Accordingly, the object of interest may be fixed at or near the center or may be smoothly displayed within the frame. Generally, the object of interest will be fixed at the center if the ROI is static and / or moving at a constant speed. In some embodiments, the motion trajectory for the region of interest is indicative of a variable motion speed between successive frames.In such embodiments, adjusting the preview involves maintaining smooth motion of the region of interest at or near the center of the zoomed portion between successive preview frames. For example, the object of interest is rendered smooth within the frame when the ROI is moving at a variable speed. The smooth motion tracks the ROI as it moves. Such an approach enables high-zoom capture without the need for a tripod.

[0040] Generally, the relative position of the image of the moon 150 remains stable within the enlarged preview 115 while frames 140 and 145 track the moon, so that the preview displayed to the user is well-centered and / or the moon moves smoothly and remains relatively stable within frames 140 and 145. This represents a significant improvement in the display functionality of the image capture device, particularly in that the stabilized image of the moon may move out of the enlarged preview 115 and the moon may disappear from view during manual tracking by the user. However, zoom stabilization algorithms are capable of tracking the moon and alerting the user when the moon is at or near the boundary of the inner frame 145. This is particularly useful at high magnification ratios, as the preview FOV can be very narrow.For example, at 30×, an object of interest may move slightly out of the preview, and the zoom stabilization tracking is able to detect the object of interest, frame it, track it smoothly, and notify the user.

[0041] Fig. 3A illustrates an exemplary workflow 300A for stabilized object tracking according to exemplary embodiments. Some image capture devices include a hardware abstraction layer (HAL) that connects the high-level camera framework application programming interfaces (APIs) in a camera application (APP) layer to the underlying camera drivers and hardware.

[0042] At 302, a telephoto preview (or zoom stabilization mode) may be activated when the magnification ratio exceeds a threshold magnification ratio (e.g., 15×). Zoom ratios are for illustrative purposes only and may vary depending on device and / or system configurations.

[0043] At 304, an input tracker may be initialized. At step 1, the system may proceed to block 306 to determine whether a tap ROI is being tracked. In some embodiments, determining the region of interest includes receiving a user indication of the region of interest. At step 2, the system may determine that the tap ROI is being tracked. At block 308, user input of an ROI may be detected and an initial ROI center may be derived from that input.

[0044] At step 3, the system may determine that the tap ROI is not being tracked. Some embodiments include generating the saliency map through a neural network. For example, at block 310, a saliency detection algorithm and / or a face recognition algorithm may be activated to identify an ROI. For example, at block 310, a machine learning (ML)-based saliency map may be determined if no user tap is detected. Salience may be applied directly to the sensor region without further cropping. This allows for the detection of a new saliency ROI that may be outside the user's final zoomed field of view. The algorithm may then extract the initial ROI center from the ML-based saliency map. In some embodiments, the system may estimate the ROI size using motion vectors from the neighboring frames.

[0045] In some embodiments, the algorithm tracks the ROI using: (i) a combination of a motion vector and optical flow, or (ii) using an ML hybrid tracker. The motion vector process can provide higher accuracy when the ROI transformation is rigid and without occlusion. The hybrid tracker, on the other hand, is more reliable when the transformation is non-rigid or occlusion occurs.

[0046] In some embodiments, the hybrid tracker uses the cropped centered ROI frame to make the ROI trackable after zooming out. In some embodiments, the crop ratio can be set to a digital target zoom ratio or to a slightly optimized ratio that maintains the ROI within the cropped frame. The process then proceeds to the ROI region to be tracked at step 4.

[0047] At block 312, the hybrid tracker may jointly stabilize the ROI using combined salience detection, an object tracker, and optical flow to obtain a reliable and accurate ROI. The algorithm uses a non-hybrid tracker if a delta difference for the ROI center between the hybrid and non-hybrid trackers is small, and may use additional weights to smoothly weight the hybrid tracker if the delta difference between two trackers is large. In general, the non-hybrid tracker, also called an ILK tracker, uses a motion vector map (similar to that used in optical flow, but with a patch size of 64×64) to determine a shift of the ROI between adjacent frames. The term "II,K" as used herein generally refers to an inverse search variant of the Lucas-Kanade algorithm for optical flow estimation.The term "II,K tracker" refers to an optical flow-based tracker. Joint stabilization compensates for any frame delay due to the depth of the camera pipeline (e.g., 5 frames). For example, information from the hybrid tracker can be combined with the motion vectors from ILK to predict the ROI. This resolves the possible frame delay due to the depth of the camera pipeline. The process then proceeds to step 5 to determine the ROI center and ROI confidence.

[0048] At block 314, EIS inputs, such as gyro and / or OIS data, and frame metadata are provided to the zoom stabilization algorithm. For example, real-time filtering and resource-efficient optimization are performed to stabilize the frame using gyro and / or OIS data and ROI inputs. The process then proceeds to the zoom stabilization algorithm 316 at step 6.

[0049] At block 316, the algorithm may generate a stabilized frame with EIS inputs (e.g., gyro sensor, OIS sensor, etc.) based on real-time filtering and resource-efficient optimization, determining the motion trajectory while compensating for hardware limitations such as gyro noise, OIS sensor noise, OIS calibration error, or signal latency. In some embodiments, the low-resolution full-sensor frame, the stabilized frame center coordinates, and the crop ratio may be provided to the user interface to generate the frame-in-frame viewfinder, as described above.

[0050] Fig. 3B is an example workflow 300B for applying zoom stabilization according to example embodiments. In particular, the features of the zoom stabilization algorithm 316 of Fig. 3A. The zoom stabilization algorithm is based on the following relations: argminwi,oj w0E Rotational smoothness + w1E Translational smoothness + w2 EROI−center + w3 E Lead + w4 E Following the real pose + other terms

[0051] Gyro / OIS noise 340 is provided to camera motion analysis 342. Camera motion analysis 342 may determine a motion trajectory for the region of interest. For example, spatial information about the position of one or more objects of interest (e.g., a face, a bounding frame) may be extracted from each captured image frame. Some embodiments include determining a motion vector associated with a previous frame and a current frame. For example, a motion vector may be generated from the spatial information by averaging two adjacent frames. For example, a motion vector may be extracted between consecutive frames at each 64×64 patch. This results in improved tracking capability.

[0052] For magnification ratios above a threshold (e.g., 15×), a user-tap to define an ROI may be preferred over automated tracking. In the absence of a user tap, the system may use a face detection algorithm to detect a face or a salience model to detect an object of interest. In the absence of a face or object of interest, the motion vector may be based on the center of the frame. The motion vector between a previous frame and a current frame is determined to obtain an approximate model for frame-by-frame motion.

[0053] The motion vector information can be combined with an output of the salience model or the face detection model to determine tracking for the object of interest. For example, a real camera pose 344 is provided to video stabilization 346. This process controls virtual rotation stabilization (e.g., camera roll). For example, the termargmin decreases wi , oj w0E Rotationsglätte the pitch / yaw weights to reduce sensitivity to gyro noise. In addition, for example, the term w1E Translationsglätte Weightings to make them less sensitive to OIS noise.

[0054] For example, the ROI center and confidence 348 provide the virtual translation stabilization for the video stabilization 346. The term w2E ROI-Zentrum is introduced to stabilize the ROI in the center. The term w3E Vorsprungcorresponds to a pause-move-pause motion pattern, which results in large residual motion in conventional EIS. Based on the real camera pose 344, an actual ROI position can be determined, and optimization can be performed based on the combined image information. Based on the input from the camera motion analysis 342 and the ROI center and confidence 348, the video stabilization 346 provides the virtual camera pose 350 to the distortion block 354. The image 352 is also provided to the distortion block 354.

[0055] With further reference to Fig. 3A, at step 7, a frame distortion from the zoom stabilization algorithm 316 is used to generate a stabilized frame 318. Further, at step 8, a bounding box of the stabilized region is provided to the frame-in-frame UI feedback 320. A centered crop may be performed on the frame if no reliable ROI is available from the previous frame. Otherwise, an ROI-centered crop is performed.

[0056] At step 9, the algorithm provides the low-resolution full-sensor frame (before stabilization) to block 320. At block 320, the algorithm provides the low-resolution full-sensor frame, the coordinates of the stabilized frame center, and the user interface crop ratio to generate the frame-in-frame viewfinder and enable dynamic visualization of the preview bounding box.

[0057] In general, a non-hybrid tracker may result in better accuracy. A hybrid tracker may provide a better compromise between occlusion handling and accuracy. In some embodiments, the non-hybrid and hybrid trackers may be combined to achieve an optimally reliable and accurate ROI.

[0058] To maintain high stabilization quality, gyro and / or OIS noise that increases with magnification ratio can be suppressed. Rotation effects can also be corrected, a seamless transition can be achieved, and other optimizations can be achieved.

[0059] When a preview contains multiple objects, the most salient object can be identified. For example, when there are multiple objects in the preview, attention can be focused on one object instead of switching between the multiple objects. For example, face recognition matching can be performed to identify a relevant face among multiple faces in an image. For example, a salience score can also be generated by a machine learning model (e.g., a visual salience model) for multiple candidate salient objects detected in an image. The zoom stabilization algorithm can select an object with a high salience score as the salient object. The salience model can be trained with training data reflecting the user's interest and / or preference, and the trained salience model can predict an object of interest to the user.

[0060] For example, a visual salience model may be trained with a training dataset comprising training scenes, sequences, and / or events. For example, the training dataset may include images (e.g., digital photographs) that include user-drawn bounding boxes enclosing a visual salience region (e.g., a region in which one or more objects of particular interest to the user may be located). Based on the training dataset, the visual salience model may predict visual salience regions within images. For example, as a result of training, the visual salience model may generate a visual salience heatmap for a given image and create a bounding box enclosing the region with the greatest probability of visual salience (e.g., highest salience score). One or more processors may compute the visual salience heatmap in the background operations of the device.In some embodiments, the visual salience heatmap may represent the probability value of the visual salience on a scale from black to white, where white indicates high probability and black indicates low probability.

[0061] In some embodiments, the visual salience heatmap includes a bounding box enclosing the region within the image containing the greatest likelihood of visual salience. In the event that there are multiple objects of interest in a photographic scene, whereby the visual salience model identifies multiple salience regions within a captured image, the model can be trained to generate a bounding box enclosing the salience region closest to the center of the captured image. This trained technique assumes that a user is interested in the most central object in the image. Alternatively, the visual salience model can be trained to create a bounding box enclosing all objects of interest in a captured image.

[0062] The image capture device may perform operations under the control of an automatic zoom manager that implements various aspects of the zoom stabilization mode. In some embodiments, either automatically or in response to a received trigger signal, including, for example, a user-performed gesture (e.g., tap or press) on the input / output device, the automatic zoom manager may implement several steps that calibrate the image capture device. For example, the automatic zoom manager may receive one or more captured images from an image sensor of the image capture device and utilize the visual salience model to generate a visual salience heatmap using the one or more captured images. The visual salience model may also output a bounding box enclosing the region with the highest probability of visual salience.

[0063] In the event that there are multiple objects of interest in the preview, causing the visual salience model to identify multiple salience regions within the preview, the visual salience model can be trained to create a bounding box that encloses the salience region closest to the center of the preview. This trained technique assumes that a user is interested in the most central object in the image. Alternatively, the visual salience model can be trained to create a bounding box that encloses all objects of interest in the preview or the object of interest with the highest salience score.

[0064] Some embodiments include determining that the adjusted preview has a magnification ratio below a threshold magnification ratio. For example, the magnification ratio may be below 15×. Such embodiments include a transition from the zoomed mode of operation to the normal mode of operation. Accordingly, the zoom stabilization mode may be disabled and the normal mode of operation enabled. In some embodiments, object tracking may no longer be performed in the normal mode of operation. In some embodiments, object tracking may still be performed. However, a frame-in-frame view and / or a magnified view of a portion of the entire field of view—e.g., a cropped region of the entire FOV corresponding to the border of the ROI—may no longer be generated and / or displayed.

[0065] Fig. 4 is an example workflow for processing consecutive frames in a hybrid tracker according to example embodiments. A user tap-based ROI, touch ROI 405, may be detected in frame t - N. The hybrid tracker 410 may track the ROI to select an ROI in frame t - N as the ROI. t-N (IH(tN)) as indicated in block 415.

[0066] The hybrid tracker path can be determined as follows: IH(t)=f(f(f(f)(H,t−N),t−N1),...t−1) where f(H, t) = H(t) + ILK(t + 1) , f(f(x) denotes the composition of the function f with itself and IH(t) represents the coordinates of the ROI, which are based on the hybrid tracker with ILK motion vectors to predict the position of the ROI in frame t.

[0067] The non-hybrid tracker 430 (e.g., an optical flow-based ILK) can generate full-screen motion vectors (MV) for a frame t as well as motion vectors for the ROI in frame t - 1 as ROI t-1 as indicated in block 435. In some embodiments, tuning may be applied in step 1 to determine ROI t (IO(t)) at block 440. A non-hybrid tracker path can be determined as follows: IO(t)=O(t−1)+ILK(t) where IO(t) represents the coordinates of the ROI based on the non-hybrid or ILK tracker. To determine a final ROI, a threshold condition 420 can be checked: O(t)=|IO(t−N)−IH(t−N)|>threshold? where O(t) represents the finalized ROI coordinates. In some embodiments, a selection may be made between IH(t) and I0(t). For example, if it is determined that threshold condition 420 is not met, the system may select IH(t), as defined in Eq. 2, as the selected ROI. As indicated in block 425, this ROI may be based on a combination of IH(t - N) and the ILKs from the non-hybrid tracker. For example, if the results of the two tracking methods—the hybrid tracker combined with ILK motion vectors and the non-hybrid tracker (or ILK tracker)—differ significantly, this indicates occlusion and / or a non-rigid transformation. Accordingly, IH(t) is chosen because the hybrid tracker is more robust in handling occlusion / non-rigid transformation. In such situations, the choice O(t) = IH(t) is used.

[0068] In addition, for example, upon determining that threshold condition 420 is met, the system may select the ROI as IO(t), as determined by Equation 3. For example, if the results of the two tracking methods—the hybrid tracker in combination with ILK motion vectors and the non-hybrid (or ILK) tracker—are close to each other, the ILK tracker provides higher accuracy, and the selection O(t) = IO(t) is used.

[0069] The selected ROI can be set as the new ROI for the iterative process.

[0070] Blocks 450 to 460 illustrate the process with an object of interest represented by the letter "A." At block 450, the letter "A" is shown in frame t - 2. At block 455, the letter "A" is shown at a new position in the next frame t - 1. Accordingly, the hybrid tracker calculates the following: H(t−1)=H(t−2)+ILK(t−1)

[0071] At block 460, the letter "A" is shown as having moved further to the right in the next frame t. Accordingly, the hybrid tracker calculates the following: H(t)=H(t−1)+ILK(t)

[0072] Although the example illustrates the computation using three consecutive frames, a similar iterative approach applies to N consecutive frames. In some embodiments, the hybrid tracker may downscale the frame to 320 × 240, and a centered, cropped ROI frame may be initialized as input to the hybrid tracker so that the object of interest can be tracked even after downscaling.

[0073] As described herein, the inner frame (e.g., inner frame 145) may be cropped from the entire FOV to create a magnified, stabilized view. The crop ratio for such a crop may be set to a digital target magnification ratio or a more intelligent desired ratio to ensure that the ROI is within the cropped frame.

[0074] With further reference to Fig. 3B, an actual ROI position can be determined based on the real camera pose 344, and based on the combined image information, optimization can be performed as follows: argminwi,ojw0ERotational smoothness+w1ETranslational smoothness+ +w2ELead+w3EROITracking where x p a tracking point in the real domain and t v is the target point in the virtual domain. The term argmin wi,oj w0E Rotationsglättereduces the pitch / yaw weights to make stabilization less sensitive to gyro noise. Furthermore, the term w1E Translationsglätte For example, certain weightings are used to make the stabilization less sensitive to OIS noise. The term w2E Vorsprung corresponds to a pause-move-pause movement pattern, which leads to large residual movements in conventional EIS. The virtual pose can be described by the term E ROIVerfolgung be determined as follows: EROIVattainment=|KvRvRp−1Kp−1xp−tv| where A -1 is an inverse of a matrix A. Here K denotes v the intrinsic matrix of the camera, which corresponds to a virtual camera pose, and R v a predicted rotation for the virtual camera pose; K p denotes the intrinsic matrix of the camera, which corresponds to a real camera pose, and R pis a predicted rotation for the real camera pose. The weighting term can be smaller than a conventional EIS in the pitch / yaw axis, but the same weighting can be maintained for roll. Accordingly, the tracking term can then dominate the pitch / yaw compensation to reduce residual motion caused by gyro / OIS noise.

[0075] In some embodiments, a two-stage optimization may be performed.

[0076] Step 1: Target point t v where the ROI will be in a stabilized frame.

[0077] Step 2: Find a virtual camera pose.

[0078] Fig. 5 depicts an exemplary tracking optimization process according to exemplary embodiments. Three consecutive input frames are shown with an image of a cat. Input frame 1 505 shows the cat on the left of the display with a bounding box around its face. Input frame 2 510 shows the cat in the center of the display, also with a bounding box around its face. An initial motion vector is generated based on the position of the consecutive bounding boxes in input frame 1 505 and input frame 2 510, as indicated by the dashed line.

[0079] Input frame 3 515 shows the cat on the right side of the display with a bounding box around its face. A motion vector is generated based on the motion vector in input frame 2 510 and the position of the consecutive bounding boxes in input frame 2 510 and input frame 3 515, as indicated by the two dashed lines. The point x p is a tracking point in the real domain. A stabilized frame 520 can be generated based on the input frames 1-3, and t v is displayed as a target point in the virtual domain. The point t v can be solved from a closed form equation as follows: tv=c1.tv,front+c2.xp+c3.center where the operation “.” represents multiplication, c1, c2 and c3 are positive weight coefficients that satisfy the condition c1 + c2 + c3 = 1.

[0080] Fig. 5 illustrates how the first term c1. t v,voris determined in Eq. 9, where t v,vor represent the coordinates of the virtual target center in a previous frame. The term c1. t v,vor restricts the coordinates of t v to be close to the coordinates of t v,vor to lie, and this in turn ensures that the center of the virtual ROI is stabilized.

[0081] Fig. 6 depicts an exemplary method for tracking optimization according to exemplary embodiments. Input frames 605, 615, and 625 show an image of a cat moving from left to right across the center. Point t v is determined in successive frames to generate the stabilized frames 610, 620 and 630, which correspond to the input frames 605, 615 and 625, respectively. Fig. 6 illustrates the effect of the middle term c2.x p in Eq. 9. Here x prepresents the position of the ROI in a real pose (e.g. in an unstabilized frame), and therefore the term c2.x p be adjusted to control how closely the virtual target follows the real position. For example, the input frame 615 corresponds to a certain state c2 = 0, and the target point t v,1 can be determined. For example, this holds if c2 = 0 if diff(t v,vor , x p ) is smaller than a second threshold, thresh2, where "diff" represents a difference or distance. The input frame 625 corresponds to the case c2 > 0 and the target point t v,2 can be determined. As indicated by stabilized frames 610, 620, and 630, although the position of the cat shifts in input frames 605, 615, and 620, the position of the cat, as indicated by stabilized frames 610, 620, and 630, is maintained at or near the center of the frame. The point t vcan be solved from the closed form equation, Eq.9.

[0082] Fig. Fig. 7 depicts another exemplary tracking optimization process according to exemplary embodiments. Input frames 705, 715, and 725 show an image of a cat on the left, and stabilized frames 710, 720, and 730 correspond to input frames 705, 715, and 725, respectively. Again, point t v can be solved from the closed form equation, Eq. 9. This example illustrates the third term c3. center in Eq. 9. In this example, this holds, c3 = 0 if diff(t v,vor ,center) * digital_zoom_factor is smaller than a third threshold, thresh3. The term diff (t v ,center) represents a distance between t vand an absolute center of the stabilized frame. The condition that this distance is smaller than a threshold ensures that the virtual ROI is not located at or near the border of the stabilized frame. Stabilized frame 2 720 illustrates the case c3 > 0, while stabilized frame 3 730 illustrates the case c3 = 0.

[0083] Fig. Figure 8 depicts an exemplary tracking optimization process for two regions of interest according to exemplary embodiments. For example, input frame 1 805 includes an image of a cat, and in input frame 2 815, a new object of interest, represented by a dog, is detected. The new ROI may be detected by a touch-tap event (e.g., when a user taps the display to indicate the ROI). In the case of the new ROI, the hybrid tracker described herein overrides the non-hybrid tracker (also referred to herein as an ILK tracker). The point t v can be solved from the equation in closed form: tv=KvRvRp−1Kp−1xp where x p represents the position of the ROI in a real pose, t v represents the position of the ROI in a virtual pose, A -1 is an inverse of a matrix A. Here K denotes vthe intrinsic matrix of the camera, which corresponds to a virtual camera pose, and R v a predicted rotation for the virtual camera pose; K p denotes the intrinsic matrix of the camera, which corresponds to a real camera pose, and R p is a predicted rotation for the real camera pose.

[0084] Fig. 9 illustrates an example image with stabilized object tracking according to example embodiments. An initial FOV 905 is shown with a bounding box 915 indicating an object of interest (e.g., an aircraft). A distortion mesh 910 is shown. For example, after determining a virtual camera pose, a stabilization mesh, such as distortion mesh 910, may be generated from a physical camera to a virtual camera by determining a source quadrangle and a destination quadrangle for each horizontal swath. In some embodiments, distortion mesh 910 may be cropped from the entire initial FOV 905 to create an enlarged FOV 920. The object of interest, in this example, an aircraft 925, is shown in a zoomed-in view within the enlarged FOV 920.As illustrated, the aircraft 925 is clearly visible and can be smoothly tracked based on motion vectors in consecutive frames.

[0085] Fig. 10 illustrates another example image with stabilized object tracking according to example embodiments. An initial FOV 1005 shows a bounding box 1015 indicating an object of interest (e.g., a mailbox displaying a house number). A distortion mesh 1010 is shown. For example, after determining a virtual camera pose, a stabilization mesh, such as distortion mesh 1010, may be generated from a physical camera to a virtual camera by determining a source quadrangle and a destination quadrangle for each horizontal strip. In some embodiments, distortion mesh 1010 may be cropped from the entire initial FOV 1005 to create a zoomed-in FOV 1020. The object of interest, in this example, mailbox 1025 displaying the house number, is shown in a zoomed-in view within zoomed-in FOV 1020.As illustrated, mailbox 1025 is in clear view and the house number “705” is visible.

[0086] Fig. 11 illustrates another example image with stabilized object tracking according to example embodiments. An initial FOV 1105 is shown with a bounding box 1115 indicating an object of interest (e.g., the moon). A distortion mesh 1110 is shown. For example, after determining a virtual camera pose, a stabilization mesh, such as distortion mesh 1110, may be generated from a physical camera to a virtual camera by determining a source quadrangle and a destination quadrangle for each horizontal swath. In some embodiments, distortion mesh 1110 may be cropped from the entire initial FOV 1105 to create a zoomed-in FOV 1120. The object of interest, in this example, the moon 1125, is shown in a zoomed-in view within the zoomed-in FOV 1120.As illustrated, the moon 1125 is clearly visible and can be smoothly tracked based on motion vectors in consecutive frames.

[0087] As described herein, a magnification ratio for telephoto, Tele-RM, may be 9.4x, and a magnification ratio for zoom stabilization may be 15x. The zoom ratios are for illustrative purposes only and may vary depending on device and / or system configurations. The image capture device may transition between two modes: a normal mode without zoom stabilization and a zoom stabilization mode. In some embodiments, the zoom stabilization mode may be controlled by a combination of the magnification ratio and grid interpolation. In some embodiments, the transition may be between a baseline EIS mode and a centered ROI-based zoom stabilization mode. Generally, transitions are seamless, with and / or without an ROI tracking term.For example, the transition between a centered ROI and an ROI source can be seamless by adjusting the virtual ROI target and retracking the transition between different ROI sources. Furthermore, a transition between a binning mode and a remosaic mode, for example, can be seamless by performing a frame-in-frame YUV crop in the camera application and adjusting the EIS border accordingly. Training machine learning models to generate conclusions / predictions

[0088] Fig. 12 shows a diagram 1200 illustrating a training phase 1202 and an inference phase 1204 of one or more trained machine learning models 1232 according to example embodiments. Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions about (patterns in) the training data. The resulting trained machine learning algorithm may be referred to as a trained machine learning model. For example, Fig. 12 illustrates the training phase 1202, in which one or more machine learning algorithms 1220 are trained on training data 1210 to become a trained machine learning model 1232. During the inference phase 1204, the trained machine learning model 1232 may receive input data 1230 and one or more requests for inferences and / or predictions 1240 (optionally as part of the input data 1230) and may then provide one or more inferences and / or predictions 1250 as output.

[0089] As such, the trained machine learning model 1232 may include one or more models of a machine learning algorithm 1220. The machine learning algorithms 1220 may include, but are not limited to, an artificial neural network (e.g., a convolutional neural network described herein, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system). The machine learning algorithms 1220 may be supervised or unsupervised and may implement any suitable combination of online and offline learning.

[0090] In some examples, the machine learning algorithms 1220 and / or the trained machine learning models 1232 may be accelerated using on-device coprocessors, such as graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and / or application-specific integrated circuits (ASICs). Such on-device coprocessors may be used to accelerate the machine learning algorithms 1220 and / or the trained machine learning models 1232. In some examples, the trained machine learning model 1232 may be trained, executed, and deployed on a particular computing device to provide inference and / or otherwise provide inference to that computing device.

[0091] During the training phase 1202, the machine learning algorithm 1220 may be trained by providing at least the training data 1210 as training input using unsupervised, supervised, semi-supervised, and / or reinforcement learning techniques. Unsupervised learning includes providing a portion (or all) of the training data 1210 to the machine learning algorithm 1220, and the machine learning algorithm 1220 determining one or more inferences based on the provided portion (or all) of the training data 1210. Supervised learning includes providing a portion of training data 1210 to the machine learning algorithm 1220, wherein the machine learning algorithm 1220 determines one or more inferences based on the provided portion, and wherein the output is either accepted or corrected based on correct results associated with the training data 1210.In some examples, the supervised learning of the machine learning algorithm 1220 may be guided by a set of rules and / or a set of labels for the training input, where the set of rules and / or the set of labels may be used to correct the inferences of the machine learning algorithm 1220.

[0092] Semi-supervised learning means that correct results are obtained for some, but not all, of the training data 1210. During semi-supervised learning, supervised learning is used for a portion of the training data 1210 that has correct results, and unsupervised learning is used for a portion of the training data 1210 that does not have correct results. Reinforcement learning includes the machine learning algorithm 1220 receiving a reward signal related to a previous inference, where the reward signal may be a numerical value. During reinforcement learning, the machine learning algorithm 1220 may output an inference and subsequently receive a reward signal, where the machine learning algorithm 1220 is configured to attempt to maximize the numerical value of the reward signal.In some examples, reinforcement learning also uses a value function that provides a numerical value representing an expected total of the numerical values provided by the reward signal over time. In some examples, the machine learning algorithm 1220 and / or the trained machine learning model 1232 may be trained using other machine learning techniques, including, but not limited to, incremental learning and curriculum learning.

[0093] In some examples, the machine learning algorithm 1220 and / or the trained machine learning model 1232 may use transfer learning techniques. For example, transfer learning techniques may include pre-training the trained machine learning model 1232 on a dataset and additionally training it using the training data 1210. In particular, the machine learning algorithm 1220 may be pre-trained on data from one or more computing devices, and a resulting trained machine learning model may be provided to the computing device CD1, wherein CD1 is configured to execute the trained machine learning model during the inference phase 1204. Subsequently, during the training phase 1202, the pre-trained machine learning model may be additionally trained using the training data 1210, wherein the training data 1210 may be derived from kernel and non-kernel data of the computing device CD1.This further training of the machine learning algorithm 1220 and / or the pre-trained machine learning model with training data 1210 from the data of CD1 can be performed within the framework of supervised or unsupervised learning. Once the machine learning algorithm 1220 and / or the pre-trained machine learning model has been trained at least with the training data 1210, the training phase 1202 can be completed. The resulting trained machine learning model can be used as one of the trained machine learning models 1232.

[0094] In particular, the trained machine learning model 1232 may be provided to a computing device after completion of the training phase 1202, if it is not already located on the computing device. The inference phase 1204 may begin after the trained machine learning model 1232 has been provided to the computing device CD1.

[0095] During the inference phase 1204, the trained machine learning model 1232 may receive input data 1230 and generate and output one or more corresponding inferences and / or predictions 1250 to the input data 1230. Thus, input data 1230 may be used as input to the trained machine learning model 1232 to provide corresponding inferences and / or predictions 1250 for kernel components and non-kernel components. For example, the trained machine learning model 1232 may generate one or more inferences and / or predictions 1250 in response to one or more requests for inferences or predictions 1240. In some examples, the trained machine learning model 1232 may be executed by a piece of other software. For example, the trained machine learning model 1232 may be executed by an inference or prediction daemon to be immediately available on demand to provide inference and / or prediction.Input data 1230 may include data from the computing device CD1 executing the trained machine learning model 1232 and / or input data from one or more computing devices other than CD1.

[0096] The training data set 1210 may include images (e.g., digital photographs) containing user-drawn bounding boxes that encompass a visual salience region (e.g., a region in which one or more objects of particular interest to a user may be located).

[0097] Input data 1230 may include one or more captured images or a preview of an image. Other types of input data are also possible.

[0098] Inferences and / or predictions 1250 may include output images, a bounding box enclosing a region of interest with the highest probability of visual salience, and / or other output data generated by the trained machine learning model 1232 based on input data 1230 (and training data 1210). In some examples, the trained machine learning model 1232 may use output inferences and / or predictions 1250 as input feedback 1260. The trained machine learning model 1232 may also rely on past inferences as inputs to generate new inferences.

[0099] Convolutional neural networks, such as a visual salience model, may be examples of a machine learning algorithm 1220. After training, the trained version of a convolutional neural network may be an example of a trained machine learning model 1232. In this approach, an example inference or prediction request 1240 may be a request to predict a region of interest in an image preview, and a corresponding example inference and / or prediction 1250 may be an output image with bounding boxes containing a visual salience region.

[0100] In some examples, a computing device CD_SOLO may include the trained version of the convolutional neural network 100, possibly after the convolutional neural network has been trained. The computing device CD_SOLO may then receive requests to predict a region of interest in an image preview and use the trained version of the convolutional neural network to generate an output image with bounding boxes containing a visual salience region.

[0101] In some examples, two or more computing devices, CD_CLI and CD_SRV, may be used to provide output images; e.g., a first computing device, CD_CLI, may generate and send a request to predict a region of interest in an image preview to a second computing device, CD_SRV. CD_SRV may then use the trained version of the convolutional neural network, possibly after training the convolutional neural network, to generate an output image with bounding boxes containing a visual salience region, and respond to CD_CLI's request to predict a region of interest in an image preview. CD_CLI may then, upon receiving responses to the requests, provide the requested region of interest using a user interface and / or a display. Example data network

[0102] Fig. 13 depicts a distributed computing architecture 1300 according to example embodiments. Distributed computing architecture 1300 includes server devices 1308 and 1310 configured to communicate with programmable devices 1304a, 1304b, 1304c, 1304d, and 1304e over network 1306. Network 1306 may be a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, a corporate intranet, the public internet, or any other type of network configured to provide a communication path between networked computing devices. Network 1306 may also correspond to a combination of one or more LANs, WANs, corporate intranets, and / or the public internet.

[0103] Although Fig. 13 illustrates only five programmable devices, distributed application architectures may support dozens, hundreds, or thousands of programmable devices. Furthermore, programmable devices 1304a, 1304b, 1304c, 1304d, 1304e (or additional programmable devices) may be any type of computing device, such as a mobile computing device, a desktop computer, a wearable computing device, a head-mountable device (HMD), a network terminal, and so on. In some examples, as illustrated by programmable devices 1304a, 1304b, 1304c, and 1304e, programmable devices may be directly connected to network 1306.In other examples, such as illustrated by programmable device 1304d, programmable devices may be indirectly connected to network 1306 via an associated computing device, such as programmable device 1304c. In this example, programmable device 1304c may act as an associated computing device to relay electronic communications between programmable device 1304d and network 1306. In other examples, such as illustrated by programmable device 1304e, a computing device may be part of or located within a vehicle, such as a car, truck, bus, boat or ship, aircraft, etc. In other examples, as illustrated in . Fig. 13, a programmable device may be connected to the network 1306 both directly and indirectly.

[0104] Server devices 1308 and 1310 may be configured to perform one or more services requested by programmable devices 1304a, 1304e. For example, server devices 1308 and / or 1310 may provide content to programmable devices 1304a-1304e. The content may include, but is not limited to, web pages, hypertext, scripts, binary data such as compiled software, images, audio, and / or video. The content may include compressed and / or uncompressed content. The content may be encrypted and / or unencrypted. Other types of content are also possible.

[0105] As another example, server devices 1308 and / or 1310 may provide programmable devices 1304a-1304e with access to software for database, search, computing, graphics, audio, video, web / Internet usage, and / or other functions. Many other examples of server devices are also possible. Computing device architecture

[0106] Fig. 14 is a block diagram of a computing device 1400 according to exemplary embodiments. In particular, the Fig. 14 may be configured to perform at least one function of method 1600 and / or a function related thereto.

[0107] Computing device 1400 may include a user interface module 1401, a network communication module 1402, one or more processors 1403, a data store 1404, one or more cameras 1418, one or more sensors 1420, and a power system 1422, all of which may be coupled together via a system bus, network, or other interconnection mechanism 1405.

[0108] The user interface module 1401 may be configured to send and / or receive data to and / or from external user input / output devices. For example, the user interface module 1401 may be configured to send and / or receive data to and / or from user input devices such as a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, and / or other similar devices. The user interface module 1401 may be configured to provide output to user display devices such as one or more cathode ray tubes (CRTs), liquid crystal displays, light-emitting diodes (LEDs), digital light processing (DLP) displays, printers, light bulbs, and / or other similar devices either now known or developed in the future.The user interface module 1401 may be configured to generate audible outputs, for example, via devices such as a speaker, a speaker jack, an audio output port, an audio output device, headphones, and / or other similar devices. The user interface module 1401 may be further configured with one or more haptic devices that can generate haptic outputs, such as vibrations and / or other signals detectable by touch and / or physical contact with the computing device 1400. In some examples, the user interface module 1401 may be used to provide a graphical user interface (GUI) for using the computing device 1400.

[0109] The network communication module 1402 may include one or more devices that provide one or more wireless interfaces 1407 and / or one or more wired interfaces 1408 configured to communicate over a network.The wireless interface(s) 1407 may include one or more wireless transmitters, receivers, and / or transceivers, such as a Bluetooth™ transceiver, a Zigbee® transceiver, a Wi-Fi™ transceiver, a WiMAX™ transceiver, an LTE™ transceiver, and / or other types of wireless transceivers configured to communicate over a wireless network. The wired interface(s) 1408 may include one or more wired transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or a similar transceiver configured to communicate with a wired network over a twisted pair cable, a coaxial cable, a fiber optic connection, or a similar physical connection.

[0110] In some examples, the network communication module 1402 may be configured to provide reliable, secure, and / or authenticated communication. For any communication described herein, information to facilitate reliable communication (e.g., guaranteed message delivery) may be provided, possibly as part of a message header and / or footer (e.g., packet or message sequencing information, encapsulation headers and / or footers, size or timing information, and transmission verification information such as cyclic redundancy check (CRC) and / or parity check values). Communication may be secured (e.g., encoded or encrypted) and / or decrypted or encrypted using one or more cryptographic protocols and / or algorithms.decoded, for example, using the Data Encryption Standard (DES), the Advanced Encryption Standard (AES), a Rivest-Shamir-Adleman (RSA) algorithm, a Diffie-Hellman algorithm, a secure sockets protocol such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and / or the Digital Signature Algorithm (DSA). To secure (and subsequently decrypt or decode) the communication, another cryptographic protocol and / or algorithm may be used in addition to, or instead of, the cryptographic protocols and / or algorithms listed herein.

[0111] One or more processors 1403 may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application-specific integrated circuits, etc.). One or more processors 1403 may be configured to execute computer-readable instructions 1406 contained in data storage 1404 and / or other instructions as described herein.

[0112] The data storage 1404 may include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of the one or more processors 1403. The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage, which may be integrated in whole or in part with at least one of the one or more processors 1403. In some examples, the data storage 1404 may be implemented using a single physical device (e.g., an optical, magnetic, organic, or other memory or disk storage device), while in other examples, the data storage 1404 may be implemented using two or more physical devices.

[0113] The data storage 1404 may include computer-readable instructions 1406 and possibly additional data. In some examples, the data storage 1404 may include storage required to perform at least a portion of the methods, scenarios, and techniques described herein and / or at least a portion of the functionality of the devices and networks described herein. In some examples, the data storage 1404 may include storage for a trained neural network model 1412 (e.g., a model of trained convolutional neural networks). Particularly in these examples, the computer-readable instructions 1406 may include instructions that, when executed by the one or more processors 1403, enable the computing device 1400 to provide some or all of the functionality of the trained neural network model 1412.

[0114] In some examples, computing device 1400 may include one or more cameras 1418. Camera 1418 may include one or more image capture devices, such as still and / or video cameras, configured to capture light and record the captured light in one or more images; that is, camera 1418 may generate one or more images of the captured light. The one or more images may be one or more still images and / or one or more images used in video images. Camera(s) 1418 may capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or one or more other frequencies.

[0115] In some examples, computing device 1400 may include one or more sensors 1420. Sensors 1420 may be configured to measure conditions within computing device 1400 and / or conditions in the environment of computing device 1400 and to provide data about those conditions. For example, sensors 1420 may include one or more of the following: (i) sensors for obtaining data about computing device 1400, such as, but not limited to, a thermometer for measuring a temperature of computing device 1400, a battery sensor for measuring the performance of one or more batteries of power system 1422, and / or other sensors that measure conditions of computing device 1400;(ii) an identification sensor for identifying other objects and / or devices, such as, but not limited to, a radio frequency identification (RFID) reader, a proximity sensor, a one-dimensional barcode reader, a two-dimensional barcode (e.g., quick response (QR) code) reader, and a laser tracker, wherein the identification sensors may be configured to read identifiers, such as RFID tags, barcodes, QR codes, and / or other devices and / or objects configured to read them, and to provide at least identifying information; (iii) sensors for measuring locations and / or movements of the computing device 1400, such as, but not limited to, a tilt sensor, a gyroscope, an accelerometer, a Doppler sensor, a GPS device, a sonar sensor, a radar device, a laser displacement sensor, and a compass;(iv) an environmental sensor for obtaining data indicative of an environment of the computing device 1400, such as, but not limited to, an infrared sensor, an optical sensor, a light sensor, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a motion sensor, a microphone, a sound sensor, an ultrasonic sensor, and / or a smoke sensor; and / or (v) a force sensor for measuring one or more forces (e.g., inertial forces and / or G-forces) acting on the computing device 1400, such as, but not limited to, one or more sensors that measure: ; Forces in one or more dimensions, torque, ground force, friction, and / or a zero moment point (ZMP) sensor that identifies ZMPs and / or ZMP locations. Many other examples of sensors 1420 are also possible.

[0116] The power system 1422 may include one or more batteries 1424 and / or one or more external interfaces 1426 for providing electrical power to the computing device 1400. Each of the one or more batteries 1424, when electrically coupled to the computing device 1400, may function as a source of stored electrical power for the computing device 1400. One or more batteries 1424 of the power system 1422 may be configured to be portable. Some or all of the one or more batteries 1424 may be readily removable from the computing device 1400. In other examples, some or all of the one or more batteries 1424 may be located within the computing device 1400 and thus may not be readily removable from the computing device 1400. Some or all of the batteries 1424 may be rechargeable.For example, a rechargeable battery may be recharged via a wired connection between the battery and another power supply, such as one or more power supplies located external to the computing device 1400 and connected to the computing device 1400 via the one or more external power interfaces. In other examples, some or all of one or more of the batteries 1424 may be non-rechargeable batteries.

[0117] One or more external power interfaces 1426 of the power system 1422 may include one or more wired power interfaces, such as a USB cable and / or a power cable, enabling wired electrical power connections to one or more power supplies located external to the computing device 1400. One or more external power interfaces 1426 may include one or more wireless power interfaces, such as a Qi wireless charger, enabling wireless electrical power connections, such as via a Qi wireless charger, to one or more external power supplies. Once an electrical power connection to an external power source is established via one or more external power interfaces 1426, the computing device 1400 may receive electrical power from the external power source via the established electrical power connection.In some examples, the power system 1422 may include related sensors, such as battery sensors associated with one or more batteries or other types of electrical power sensors. Cloud-based servers

[0118] Fig. 15 depicts a cloud-based server system according to exemplary embodiments. In Fig. 15, the functionality of convolutional neural networks and / or a computing device may be distributed among the computing clusters 1509a, 1509b, 1509c. The computing cluster 1509a may include one or more computing devices 1500a, cluster storage arrays 1510a, and cluster routers 1511a connected by a cluster local network 1512a. Similarly, the computing cluster 1509b may include one or more computing devices 1500b, cluster storage arrays 1510b, and cluster routers 1511b connected by a cluster local network 1512b. Likewise, the computing cluster 1509c may include one or more computing devices 1500c, cluster storage arrays 1510c, and cluster routers 1511c connected by a cluster local network 1512c.

[0119] In some embodiments, each of compute clusters 1509a, 1509b, and 1509c may include an equal number of compute devices, an equal number of cluster storage arrays, and an equal number of cluster routers. However, in other embodiments, each compute cluster may include a different number of compute devices, a different number of cluster storage arrays, and a different number of cluster routers. The number of compute devices, cluster storage arrays, and cluster routers in each compute cluster may depend on the compute task(s) assigned to each compute cluster.

[0120] For example, in the compute cluster 1509a, computing devices 1500a may be configured to perform various computational tasks of a convolutional neural network, confidence learning, and / or computing device. In one embodiment, the various functionalities of a convolutional neural network, confidence learning, and / or computing device may be distributed among one or more of the computing devices 1500a, 1500b, and 1500c. The computing devices 1500b and 1500c in the respective compute clusters 1509b and 1509c may be configured similarly to the computing devices 1500a in the compute cluster 1509a. On the other hand, in some embodiments, the computing devices 1500a, 1500b, and 1500c may be configured to perform different functions.

[0121] In some embodiments, computational tasks and stored data associated with a convolutional neural network and / or a computing device may be distributed among computing devices 1500a, 1500b, and 1500c based at least in part on the processing requirements of the convolutional neural network and / or the computing device, the processing capabilities of computing devices 1500a, 1500b, and 1500c, the latency of network connections between computing devices within a cluster as well as between clusters themselves, and / or other factors that may contribute to cost, speed, fault tolerance, resiliency, efficiency, and / or other design goals of the overall system architecture.

[0122] The cluster storage arrays 1510a, 1510b, 1510c of the compute clusters 1509a, 1509b, 1509c may be data storage arrays configured to manage read and write access to groups of disk drives. The disk array controllers, alone or in conjunction with their respective computing devices, may also be configured to manage backup or redundant copies of the data stored in the cluster storage arrays to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.

[0123] Similar to how the functions of convolutional neural networks and / or a computing device may be distributed among the computing devices 1500a, 1500b, 1500c of the computing clusters 1509a, 1509b, 1509c, various active portions and / or backup portions of these components may also be distributed among the cluster memory arrays 1510a, 1510b, 1510c. For example, some cluster memory arrays may be configured to store a portion of the data of one convolutional neural network and / or a computing device, while other cluster memory arrays may store other portions of data from the same convolutional neural network and / or the same computing device. Furthermore, for example, some cluster memory arrays may be configured to store the data of a first convolutional neural network, while other cluster memory arrays store the data of a second and / or third convolutional neural network.In addition, some cluster storage arrays may be configured to store backup versions of data stored in other cluster storage arrays.

[0124] The cluster routers 1511a, 1511b, 1511c in computing clusters 1509a, 1509b, 1509c may include network equipment configured to provide internal and external communications for the computing clusters. For example, the cluster routers 1511a in computing cluster 1509a may include one or more internet switching and routing devices configured to provide (i) local area network communications between computing devices 1500a and cluster storage arrays 1510a via cluster local area network 1512a, and (ii) wide area (WAN) communications between computing cluster 1509a and computing clusters 1509b and 1509c via WAN connection 1513a to network 1306.Cluster routers 1511b and 1511c may include network equipment similar to cluster routers 1511a, and cluster routers 1511b and 1511c may perform similar network functions for compute clusters 1509b and 1509b that cluster routers 1511a perform for compute cluster 1509a.

[0125] In some embodiments, the configuration of cluster routers 1511a, 1511b, 1511c may be based at least in part on the data communication requirements of the computing devices and the cluster storage arrays, the data communication capabilities of the network equipment in cluster routers 1511a, 1511b, 1511c, the latency and throughput of cluster local networks 1512a, 1512b, 1512c, the latency, throughput, and cost of wide area network connections 1513a, 1513b, 1513c, and / or other factors that may contribute to the cost, speed, fault tolerance, resiliency, efficiency, and / or other design criteria of the moderation system architecture. Exemplary operating procedures

[0126] Fig. 16 illustrates a method 1600 according to example embodiments. The method 1600 may include various blocks or steps. The blocks or steps may be performed individually or in combination. The blocks or steps may be performed in any order and / or serially or in parallel. Furthermore, blocks or steps may be omitted or added to the method 1600.

[0127] The blocks of method 1600 may be performed by various elements of computing device 1400, as described with respect to Fig. 8 illustrated and described.

[0128] Block 1610 includes displaying a preview of an image through a display screen of an image capture device, the image representing a field of view of the image capture device.

[0129] Block 1620 includes determining a region of interest in the preview of the image.

[0130] Block 1630 includes transitioning the image capture device from a normal operating mode to a zoomed operating mode, wherein the zoomed operating mode comprises determining a motion trajectory for the region of interest based on sensor data acquired by a sensor associated with the image capture device, and generating an adjusted preview based on the determined motion trajectory, wherein the adjusted preview represents a zoomed portion of the field of view and displays the region of interest at or near the center of the zoomed portion.

[0131] Block 1640 includes providing the adjusted preview of the portion of the field of view through the display screen.

[0132] Some embodiments include the display screen providing an image overlay that displays a representation of the zoomed portion relative to the field of view.

[0133] Some embodiments include determining a bounding frame for the region of interest, wherein providing the image overlay comprises framing the region of interest within that bounding frame.

[0134] Some embodiments include determining one or more of the following: (i) a lower-resolution version of the displayed image, (ii) the coordinates of the adjusted region of interest within the adjusted preview, or (iii) a crop ratio. Such embodiments also include generating an image overlay to enable dynamic visualization of the bounding box.

[0135] Some embodiments include detecting that the region of interest is approaching a boundary of the image overlay. Such embodiments also include providing a notification to the user indicating that the region of interest is approaching the boundary of the image overlay.

[0136] Some embodiments include determining a motion vector associated with a previous frame and a current frame. Such embodiments also include determining the size of the region of interest based on the determined motion vector.

[0137] Some embodiments include determining an optical flow corresponding to the region of interest. Such embodiments further include tracking the region of interest within the portion of the field of view based on the determined optical flow, wherein adjusting the image preview is based on tracking the region of interest.

[0138] Some embodiments include tracking the region of interest within the field of view based on a combination of a motion vector process and optical flow. For example, if a transformation associated with the region of interest is determined to be rigid and without occlusion, the combination of the motion vector process and optical flow may be used to track the region of interest.

[0139] Some embodiments include tracking the region of interest in the field of view using a hybrid tracker. For example, the hybrid tracker can be used to track the region of interest when it is determined that its associated transform is non-rigid or an occlusion is present. In some embodiments, the hybrid tracker relies on a cropped centered frame to track the region of interest after a zoom operation. In some embodiments, the hybrid tracker includes: (a) one or more motion vectors associated with a current image frame, and (b) a saliency map indicative of the region of interest.

[0140] Some embodiments include generating the saliency map by a neural network.

[0141] In some embodiments, the preview includes a plurality of objects, and the method includes using a saliency map to select one object from the plurality of objects, wherein determining the region of interest is based on the selected object.

[0142] In some embodiments, the sensor is either a gyroscope or an optical image stabilization (OIS) sensor.

[0143] In some embodiments, the motion trajectory for the region of interest is an indicator of a variable motion speed between successive frames. In such embodiments, adjusting the preview includes maintaining smooth motion of the region of interest at or near the center of the zoomed portion between successive preview frames.

[0144] In some embodiments, the motion trajectory for the region of interest is indicative of a nearly constant motion speed between consecutive frames. In such embodiments, adjusting the preview includes locking a position for the region of interest at or near the center of the zoomed portion between consecutive frames of the preview.

[0145] In some embodiments, determining the region of interest comprises receiving a user indication of the region of interest.

[0146] In some embodiments, determining the region of interest comprises determining a saliency map indicative of the region of interest based on a neural network.

[0147] Some embodiments include determining that the adjusted preview has a magnification ratio below a threshold magnification ratio. Such embodiments include transitioning from the zoomed mode of operation to the normal mode of operation.

[0148] The specific arrangements shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or fewer of the elements shown in a given figure. Furthermore, some of the illustrated elements may be combined or omitted. Furthermore, an exemplary embodiment may include elements not illustrated in the figures.

[0149] A step or block representing information processing may correspond to circuitry that may be configured to perform the specific logical functions of a method or technique described herein. Alternatively or additionally, a step or block representing processing information may correspond to a module, segment, or portion of program code (including associated data). The program code may include one or more processor-executable instructions for implementing specific logical functions or acts in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including a floppy disk, a hard disk, or other storage medium.

[0150] The computer-readable medium may also include non-transitory computer-readable media, such as computer-readable media that stores data for short periods of time, such as register memory, processor cache, and random access memory (RAM). The computer-readable media may also include non-transitory computer-readable media that stores program code and / or data for longer periods of time. The computer-readable media may therefore include secondary or persistent long-term storage, such as read-only memory (ROM), optical or magnetic disks, or compact disc read-only memory (CD-ROM). The computer-readable media may also be any other volatile or non-volatile storage systems. For example, a computer-readable medium may be considered a computer-readable storage medium or a tangible storage device.

[0151] Although various examples and embodiments have been disclosed, other examples and embodiments will be apparent to those skilled in the art. The various disclosed examples and embodiments are for illustrative purposes and are not intended to be limiting, with a true scope being indicated by the following claims.

Claims

[1] Computer-implemented method comprising: Displaying a preview of an image through a display screen of an image capture device representing a field of view of the image capture device; Identifying a region of interest in the image preview; Transferring the image capture device from a normal operating mode to a zoomed operating mode, wherein the zoomed operating mode comprises: Determining a movement trajectory for the region of interest based on sensor data acquired by a sensor associated with the image capture device, and Generating an adjusted preview representing a zoomed portion of the field of view based on the determined motion trajectory, the adjusted preview displaying the region of interest at or near a center of the zoomed portion; and Providing the customized preview of the section of the field of view through the display screen. [2] The method of claim 1, further comprising: Providing an image overlay by the display screen that displays a representation of the zoomed portion relative to the field of view. [3] The method of claim 2, further comprising: Determine a bounding box for the region of interest and wherein providing the image overlay comprises providing the region of interest framed within the bounding box. [4] The method of claim 3, further comprising: Determining one or more of (i) a lower resolution version of the displayed image, (ii) coordinates of the adjusted region of interest within the adjusted preview, or (iii) a crop ratio; and Generate the image overlay to enable dynamic visualization of the bounding box. [5] The method of claim 2, further comprising: Detecting that the region of interest is approaching a boundary of the image overlay; and Provide a user notification indicating that the region of interest is approaching the boundary of the image overlay. [6] The method of claim 1, further comprising: Determining a motion vector associated with a previous image frame and a current image frame; and Determining a size of the region of interest based on the determined motion vector. [7] The method of claim 1, further comprising: Determining an optical flow corresponding to the region of interest; and Tracking the region of interest within the portion of the field of view based on the determined optical flow and where adjusting the preview of the image is based on tracking the region of interest. [8] The method of claim 1, further comprising: Tracking the region of interest within the field of view based on a combination of a motion vector process and optical flow. [9] The method of claim 1, further comprising: Tracking the region of interest within the field of view based on a hybrid tracker. [10] The method of claim 9, wherein the hybrid tracker relies on a cropped centered frame to track the region of interest after a downsizing operation. [11] The method of claim 9, wherein the hybrid tracker comprises: (a) one or more motion vectors associated with a current image frame, and (b) a saliency map indicating the region of interest. [12] The method of claim 11, further comprising: Generating the salience map using a neural network. [13] The method of claim 1, wherein the preview comprises a plurality of objects, and further comprising: Using a salience map to select an object from the plurality of objects, and where determining the region of interest is based on the selected object. [14] The method of claim 1, wherein the sensor is one of a gyroscope or an optical image stabilization (OIS) sensor. [15] The method of claim 1, wherein the motion trajectory for the region of interest indicates a variable motion speed between successive frames, and wherein adjusting the preview further comprises: Maintain smooth motion for the region of interest at or near the center of the zoomed section between successive frames of the preview. [16] The method of claim 1, wherein the motion trajectory for the region of interest indicates a nearly constant speed of motion between successive frames, and wherein adjusting the preview of the image further comprises: Lock a position for the region of interest at or near the center of the zoomed section between successive frames of the preview. [17] The method of claim 1, wherein determining the region of interest further comprises: Receiving a user indication of the region of interest. [18] The method of claim 1, wherein determining the region of interest further comprises: Determining a salience map indicating the region of interest based on a neural network. [19] The method of claim 1, further comprising: Determining that the adjusted preview is at a magnification ratio that is below a threshold magnification ratio; and Transition from zoomed operating mode to normal operating mode. [20] The method of claim 1, wherein the region of interest comprises a human face and wherein determining the region of interest is based on a face detection algorithm. [21] Mobile device comprising: an image capture device comprising a display screen; one or more processors; and a data store, wherein the data store has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the mobile device to perform functions comprising the computer-implemented method of any of claims 1-20. [22] A non-transitory computer-readable medium comprising program instructions executable by one or more processors to cause the one or more processors to perform operations comprising the computer-implemented method of any one of claims 1-20.

Images