Scaling image resolution for motion estimation

Abstract

Systems and techniques are provided for conserving resources when performing motion estimation. An example process may include determining, based on an input image and a reference image, a motion vector identifying motion between the input image and the reference image; determining whether the motion indicated by the motion vectors is below a first threshold; based on a determination that the motion indicated by the motion vectors is below the first threshold, avoiding determining local motion between the input image and the reference image; determining a transformation matrix based on the motion vectors and without using local motion between the input image and the reference image; and adjusting the input image based on the transformation matrix.

Description

Technical Field

[0001] The present disclosure generally relates to image processing. For example, aspects of the present disclosure relate to techniques and systems for dynamically scaling image resolution for motion estimation to save resources. Background Art

[0002] Electronic devices are increasingly equipped with camera hardware that can be used to capture image frames (e.g., still images and / or video frames) for consumption. For example, an electronic device (e.g., a mobile device, an Internet Protocol (IP) camera, an extended reality device, a connected device, a laptop, a smart phone, a smart wearable device, a game console, etc.) may include one or more cameras integrated with the electronic device. The electronic device may use a camera to capture an image or video of a scene, a person, an object, or anything else of interest to a user of the electronic device. The electronic device may capture an image or video (e.g., via a camera) and process, output, and / or store the image or video for consumption (e.g., display on an electronic device, save on a storage device, transmit or stream to another device, etc.).

[0003] In some cases, the electronic device may further process the image or video to obtain certain effects, such as depth of field or portrait effects, extended reality (e.g., augmented reality, virtual reality, etc.) effects, image stylization effects, image enhancement effects, etc., and / or for certain applications, such as computer vision, extended reality, object detection, recognition (e.g., facial recognition, object recognition, scene recognition, etc.), compression, feature extraction, authentication, segmentation, and automation, etc. However, further processing of the image or video may increase the amount of resources utilized and / or consumed by the electronic device, such as (for example, but not limited to) power and computing resources, etc. Therefore, there is a need in the art to reduce the amount of resources utilized and / or consumed by the electronic device when generating, processing, and / or manipulating image frames (e.g., still images or video frames). Summary of the invention

[0004] Systems and techniques for saving resources when performing motion estimation are described herein. According to at least one example, a method for saving resources when performing motion estimation is provided. The method may include: determining, by a computing device, motion vectors indicating motion between an input image and a reference image based on an input image and a reference image; determining, by the computing device, whether the motion indicated by the motion vectors is below a first threshold; avoiding determining local motion between the input image and the reference image based on a determination that the motion indicated by the motion vectors is below the first threshold; determining, by the computing device, a transformation matrix based on the motion vectors and without using the local motion between the input image and the reference image; and adjusting, by the computing device, the input image based on the transformation matrix.

[0005] In another example, an apparatus for processing image data is provided, the apparatus comprising at least one memory and at least one processor (e.g., configured in a circuit), the at least one processor coupled to the at least one memory. The at least one processor is configured to: determine motion vectors indicating motion between the input image and the reference image based on an input image and a reference image; determine whether the motion indicated by the motion vectors is below a first threshold; avoid determining local motion between the input image and the reference image based on a determination that the motion indicated by the motion vectors is below the first threshold; determine a transformation matrix based on the motion vectors and without using the local motion between the input image and the reference image; and adjust the input image based on the transformation matrix.

[0006] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which, when executed by one or more processors, causes the one or more processors to: determine, based on an input image and a reference image, motion vectors indicating motion between the input image and the reference image; determine whether the motion indicated by the motion vectors is below a first threshold; based on a determination that the motion indicated by the motion vectors is below the first threshold, avoid determining local motion between the input image and the reference image; determine a transformation matrix based on the motion vectors and without using the local motion between the input image and the reference image; and adjust the input image based on the transformation matrix.

[0007] In another example, an apparatus for processing image data is provided. The apparatus includes: means for determining motion vectors indicating motion between the input image and the reference image based on an input image and a reference image; means for determining whether the motion indicated by the motion vectors is below a first threshold; means for avoiding determining local motion between the input image and the reference image based on a determination that the motion indicated by the motion vectors is below the first threshold; means for determining a transformation matrix based on the motion vectors and without using the local motion between the input image and the reference image; and means for adjusting the input image based on the transformation matrix.

[0008] In some aspects, each of the above-mentioned devices is a mobile device, a smart or connected device, a camera system, a computer system, an advanced driver assistance computer, a robotic system, and / or an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), or may be part of or may include these devices or systems. In some examples, the device may include a vehicle, a mobile device (e.g., a mobile phone or so-called "smart phone" or other mobile device), a wearable device, a personal computer, a laptop computer, a tablet computer, a server computer, a robotic device or system, an aviation system, or other device, or may be part of these devices. In some aspects, the device includes an image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, the device includes one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device includes one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, the device described above may include one or more sensors. In some cases, the one or more sensors may be used to determine the location of the device, the state of the device (eg, tracking state, operating state, temperature, humidity level, and / or another state), and / or for other purposes.

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0010] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Illustrative examples of the present application are described in detail below with reference to the following drawings:

[0012] Figure 1 is a simplified block diagram illustrating an example electronic device according to some examples of the present disclosure;

[0013] Figure 2 is a diagram illustrating an example flow for a specific implementation of motion estimation according to some examples of the present disclosure;

[0014] Figure 3A and Figure 3B is a diagram illustrating an example flow for a specific implementation of motion estimation with a feedback mechanism according to some examples of the present disclosure;

[0015] Figure 4 is a diagram illustrating an example state machine according to some examples of the present disclosure, showing various example processing states and thresholds for estimating motion;

[0016] Figure 5 is a diagram illustrating example state transitions and corresponding thresholds according to some examples of the present disclosure;

[0017] Figure 6 is a flowchart illustrating an example process for saving resources when performing motion estimation according to some examples of the present disclosure; and

[0018] Figure 7 is a diagram illustrating an example system architecture for implementing certain aspects described herein. DETAILED DESCRIPTION

[0019] Provided below are some aspects and examples of the present disclosure. As will be apparent to those skilled in the art, some of these aspects and examples may be applied independently, and some of them may be applied in combination. In the following description, specific details are set forth for explanation purposes in order to provide a thorough understanding of each example and aspect of the application. However, it is apparent that various aspects and examples may be practiced without these specific details. Each drawing and description are not intended to be restrictive.

[0020] The following description provides examples only and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the subsequent description of each example and aspect of the present disclosure will provide an enabling description for implementing the examples of the present disclosure to those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the scope of the present application as set forth in the appended claims.

[0021] As previously explained, more and more electronic devices are equipped with camera hardware for capturing images and / or videos for use. For example, an electronic device (e.g., a mobile device, an IP camera, an extended reality device, a laptop, a tablet computer, a smart TV, a head mounted display, smart glasses, a game console, a camera system, a connected device, a smart phone, etc.) may include a camera to allow the electronic device to capture video or images of a scene, a person, an object, etc. The image or video may be captured and processed by the electronic device and stored or output for use (e.g., displayed on the electronic device and / or another device).

[0022] In some cases, the camera hardware and the images and / or video frames captured by the camera hardware may be used for a variety of applications, such as, for example, but not limited to, computer vision, extended reality (e.g., augmented reality, virtual reality, etc.), object detection, image recognition (e.g., facial recognition, object recognition, scene recognition, etc.), feature extraction, positioning, authentication, photography, automation, compression, motion estimation, image stabilization, temporal noise reduction, etc. In addition, further processing of the images or videos for certain effects may increase the amount of resources utilized and / or consumed by the electronic device. For example, further processing of a frame (e.g., a still image or a video frame) may increase power consumption and resource utilization (e.g., computing resources, storage resources, power resources, etc.) of the electronic device, etc.

[0023] In some examples, certain features and functionality implemented by an electronic device, such as, for example, but not limited to, image stabilization and temporal noise reduction, rely on global and local motion information. Global motion can be estimated based on input from a gyroscope of the electronic device. The input from the gyroscope, together with texture processing, can accurately represent the motion of the electronic device. The motion of the electronic device and texture processing within a frame is referred to herein as global motion. In some cases, motion vectors can be estimated using one or more motion estimation techniques, such as, for example, but not limited to, Harris corner detection and related techniques (hereinafter referred to as descriptor matching estimation (DME)). In some examples, a DME algorithm can determine a global motion matrix that indicates or reflects the global motion associated with the electronic device.

[0024] In some cases, the electronic device may generate a local motion estimate by comparing consecutive frames to determine the relative motion of the consecutive frames. The electronic device may estimate a local motion map using one or more local motion estimation techniques (such as, for example, but not limited to, descriptor matching estimation (DME), semi-global matching (SGM), etc., referred to herein as DMM or local motion estimation). The local motion map may provide an estimate of the motion between two different frames (e.g., how much the scene depicted in the frame and / or features in the scene depicted in the frame have moved between the frames). In some examples, the local motion or local motion map (e.g., estimated motion between frames) may indicate how much a frame should be warped to account for motion between the frame and a previous frame from a sequence of frames.

[0025] In some cases, a local motion estimation implementation (e.g., a DMM implementation) may use an SGM algorithm to generate a motion map. In some cases, because the entire image associated with the SGM operation is scanned to generate the motion map, the SGM operation may be time-constrained and may be computationally expensive. In some examples, in order to achieve a balance between image quality and power consumption, the SGM operation may be performed on a reduced version of the target image. In some cases, an engine for video analysis (EVA) may generate a motion map and integrate global motion estimation to create a transformation matrix. An image processing engine (IPE) of an electronic device may use the transformation matrix to perform final image stabilization on the target image.

[0026] In some cases, even processing a smaller resolution image to generate a motion map can result in significant power consumption by the electronic device. Furthermore, in implementations where there is little or minimal local or global motion, or when local motion is confined to a small area, the electronic device can still consume significant power to estimate the motion. In some cases, image quality enhancement may not justify such high power consumption.

[0027] Systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively, "systems and techniques") for saving resources when performing motion estimation are described herein. For example, the systems and techniques described herein can reduce the amount of resources utilized and / or consumed by an electronic device when generating, processing, and / or manipulating image frames (e.g., still images or video frames). In some examples, the systems and techniques described herein can provide algorithmic techniques for reducing power consumption of an electronic device when estimating motion in the presence of little to no motion (e.g., global and / or local motion) and / or motion is locally confined to a smaller area (e.g., an area less than a threshold size). In some examples, the systems and techniques described herein provide algorithms configured to significantly reduce power consumption of an electronic device in such situations (e.g., where there is little to no motion and / or where motion is locally confined to a smaller area). In some examples, the systems and techniques described herein can provide a mechanism for returning to full-resolution local motion estimation (e.g., DMM) when motion is detected locally.

[0028] In some cases, the systems and techniques described herein may implement one or more components and / or algorithms. For example, in some cases, the systems and techniques described herein may implement a combination of motion maps, motion vectors, and temporal filter indications (TFIs) generated by an image processing engine (IPE) to dynamically switch between global motion estimation (e.g., DME), smaller resolution local motion estimation (e.g., smaller resolution DMM), and full resolution local motion estimation (e.g., full resolution DMM). The systems and techniques described herein may be implemented in various contexts, applications, situations, and / or use cases. For example, the systems and techniques described herein may be implemented in video recording and / or image processing situations with one or more conditions.

[0029] To illustrate, in some cases, the systems and techniques described herein may be implemented in video recording and / or image processing scenarios, such as a mobile device set on a tripod recording a scene (e.g., a natural landscape) with no motion (or motion below a threshold); a mobile device on a tripod recording a representation of a subject's motion below a threshold (e.g., small or minimal); a mobile device recording frames where the mobile device has a relatively low amount of motion (e.g., below a threshold) in its field of view (FOV), and / or the mobile device records at a higher frames per second (FPS) rate (e.g., a recording rate above a threshold) such that frames are captured at a faster rate and the relative motion between frames is less than when the same scene is recorded at a lower FPS; in cases involving multi-frame high dynamic range (MFHDR) merging; in cases where motion compensated temporal filtering (MCTF) processing is affected; in cases involving thermal mitigation to reduce power consumption of electronic devices and prevent thermal runaway conditions in high power encoding implementations, etc. In some cases, applicable scenarios and / or use cases may be further expanded depending on how far the main subject or object on the screen is from the recording point and how much motion is perceived.

[0030] Examples of systems and techniques for processing data described herein are Figures 1 to 7 exemplified in and described below.

[0031] Figure 1 is a diagram of an example electronic device 100 illustrating some examples according to the present disclosure. The electronic device 100 can implement the systems and techniques disclosed herein. For example, in some cases, the electronic device 100 can dynamically scale image resolution for motion estimation in order to save power, as further described herein.

[0032] The electronic device 100 may also perform various tasks and operations, such as (for example, but not limited to) extended reality (e.g., augmented reality, virtual reality, mixed reality, virtual reality with pass-through video, etc.) tasks and operations (e.g., tracking, mapping, positioning, content presentation, pose estimation, object detection / recognition, etc.), image / video processing and / or post-processing, data processing and / or post-processing, computer graphics, machine vision, object modeling and registration, multimedia rendering and / or synthesis, object detection, object recognition, positioning, scene recognition and / or any other data processing tasks, effects and / or calculations.

[0033] exist Figure 1 In the example shown, the electronic device 100 includes one or more image sensors 102, one or more inertial sensors 104 (e.g., one or more inertial measurement units, etc.), one or more other sensors 106 (e.g., one or more radio detection and ranging (radar) sensors, light detection and ranging (LIDAR) sensors, acoustic / sound sensors, infrared (IR) sensors, magnetometers, touch sensors, laser rangefinders, light sensors, proximity sensors, motion sensors, active pixel sensors, machine vision sensors, ultrasonic sensors, etc.), storage 108, computing components 110, and processing engines 120. In some cases, the processing engine 120 may include one or more engines, such as (for example, but not limited to) one or more motion estimation engines, one or more image processing engines, one or more image front ends (e.g., one or more image pre-processing engines), one or more video analysis engines, one or more machine learning engines, one or more image post-processing engines, one or more rendering engines, etc. In some examples, the electronic device 100 may include additional software and / or software engines, such as, for example, extended reality (XR) applications, camera applications, video game applications, video conferencing applications, etc.

[0034] Figure 1 The components 102 to 120 shown are non-limiting examples provided for purposes of illustration and explanation. In other examples, the electronic device 100 may include more Figure 1 For example, in some cases, electronic device 100 may include more, fewer, and / or different components than those shown in FIG. Figure 1 One or more display devices, one or more other processing engines, one or more receivers (e.g., global positioning system, global navigation satellite system, etc.), one or more communication devices (e.g., radio frequency (RF) interface and / or any other wireless / wired communication receiver / transmitter), one or more other hardware components, and / or one or more other software and / or hardware components not shown. Figure 7 An example architecture and example hardware components that may be implemented by electronic device 100 are further described.

[0035] The one or more image sensors 102 may include any number of image sensors. For example, the one or more image sensors 102 may include a single image sensor, two image sensors in a dual camera implementation, or more than two image sensors in other multi-camera implementations. The electronic device 100 may be part of or implemented by a single computing device or multiple computing devices. In some examples, the electronic device 100 may be part of an electronic device (or multiple electronic devices), such as a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc.), a phone system (e.g., a smart phone, a cellular phone, a conferencing system, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a smart TV, a display device, a game console, a video streaming device, an IoT (Internet of Things) device, a smart wearable device (e.g., a head-mounted display (HMD), smart glasses, etc.), or any other suitable electronic device.

[0036] In some implementations, one or more image sensors 102, one or more inertial sensors 104, other sensors 106, storage 108, computing components 110, and processing engine 120 may be part of the same computing device. For example, in some cases, one or more image sensors 102, one or more inertial sensors 104, one or more other sensors 106, storage 108, computing components 110, and processing engine 120 may be integrated into a smartphone, a laptop, a tablet, a smart wearable device, a gaming system, and / or any other computing device. In other implementations, one or more image sensors 102, one or more inertial sensors 104, other sensors 106, storage 108, computing components 110, and processing engine 120 may be part of two or more separate computing devices. For example, in some cases, some of components 102 to 120 may be part of or implemented by one computing device, and the remaining components may be part of or implemented by one or more other computing devices.

[0037] Image sensor(s) 102 may include one or more image sensors. In some examples, image sensor(s) 102 may include any image and / or video sensor or capture device, such as a digital camera sensor, a video camera sensor, a smartphone camera sensor, an image / video capture device on an electronic device (such as a television or computer), a camera, etc. In some cases, image sensor(s) 102 may be part of a multi-camera system or computing device (such as an extended reality (XR) device (e.g., HMD, smart glasses, etc.), a digital camera system, a smartphone, a smart television, a gaming system, etc.). Image sensor(s) 102 may capture image and / or video content (e.g., raw image and / or video data) that may be processed by computing component 110.

[0038] In some examples, one or more image sensors 102 may capture image data and generate frames based on the image data, and / or provide the image data or frames to computing component 110 for processing. A frame may include a video frame or a static image in a video sequence. A frame may include an array of pixels representing a scene. For example, a frame may be: a red-green-blue (RGB) frame having red, green, and blue color components per pixel; a luminance, redness, blueness (YCbCr) frame having a luminance component and two chrominance (color) components (redness and blueness) per pixel; or any other suitable type of color or monochrome picture.

[0039] The electronic device 100 may include one or more inertial sensors 104. The one or more inertial sensors 104 may include, for example, but not limited to, a gyroscope, an accelerometer, an inertial measurement unit (IMU), and / or any other inertial sensor. The one or more inertial sensors 104 may detect the movement (e.g., translation and / or rotation) of the electronic device 100. For example, the one or more inertial sensors 104 may detect a specific force and / or angular velocity of the electronic device 100. In some cases, the one or more inertial sensors 104 may detect the orientation of the electronic device 100. The one or more inertial sensors 104 may generate linear acceleration measurements, rotation rate measurements, and / or heading measurements. In some examples, the one or more inertial sensors 104 may be used to measure the pitch, roll, and yaw of the electronic device 100.

[0040] The electronic device 100 may optionally include one or more other sensors 106. In some examples, the one or more other sensors 106 may detect and generate other measurements used by the electronic device 100. In some cases, the computing component 110 may use data and / or measurements from one or more image sensors 102, one or more inertial sensors 104, and / or one or more other sensors 106 to track the posture of the electronic device 100. As previously described, in other examples, the electronic device 100 may also include other sensors, such as magnetometers, acoustic / sound sensors, IR sensors, machine vision sensors, smart scene sensors, radio detection and ranging (RADAR) sensors, light detection and ranging (LIDAR) sensors, depth sensors, light sensors, etc.

[0041] The storage device 108 may be any storage device for storing data. In addition, the storage device 108 may store data from any of the components of the electronic device 100. For example, the storage device 108 may store data from one or more image sensors 102 (e.g., image or video data), data from one or more inertial sensors 104 (e.g., measurements), data from one or more other sensors 106 (e.g., measurements), data from the computing component 110 (e.g., processing parameters, timestamps, preferences, virtual content, rendered content, scene mapping, tracking and positioning data, object detection data, configurations, motion vectors, XR application data, recognition data, synchronization data, outputs, etc.), and / or data from the processing engine 120. In some examples, the storage device 108 may include a buffer for storing frames and / or other camera data for processing by the computing component 110.

[0042] The one or more computing components 110 may include a central processing unit (CPU) 112, a graphics processing unit (GPU) 114, a digital signal processor (DSP) 116, and / or an image signal processor (ISP) 118. The computing component 110 may perform various operations such as camera synchronization, image enhancement, computer vision, graphics rendering, extended reality (e.g., tracking, localization, pose estimation, mapping, content anchoring, content rendering, etc.), image / video processing, sensor processing, recognition (e.g., text recognition, facial recognition, object recognition, feature recognition, tracking or pattern recognition, scene recognition, occlusion detection, etc.), machine learning, filtering, object detection, and any of the various operations described herein. Figure 1In the example shown, computing component 110 may implement processing engine 120. For example, the operations of processing engine 120 may be implemented by any computing component in computing component 110. Processing engine 120 may include one or more neural network models, such as the unsupervised learning models described herein. In some examples, computing component 110 may also implement one or more other processing engines.

[0043] Although electronic device 100 is shown as including certain components, those skilled in the art will appreciate that electronic device 100 may include more than Figure 1 For example, in some instances, the electronic device 100 may further include one or more memory devices (e.g., RAM, ROM, cache, etc.), one or more network interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices, and / or Figure 1 Other hardware or processing devices not shown. Figure 7 Illustrative examples of computing devices and hardware components that may be implemented using electronic device 100 are described.

[0044] In some examples, the electronic device 100 may implement one or more algorithms for estimating global motion associated with the electronic device 100 and / or local motion associated with frames captured by one or more image sensors 102 of the electronic device 100. In addition, the electronic device 100 may implement the systems and techniques described herein to reduce power consumption of the electronic device 100 when estimating global and / or local motion. In some cases, when the amount of motion detected, estimated, and / or predicted by the video analysis engine is below a threshold, the electronic device 100 may shut down or disable the motion estimation processing pipeline implemented by the video analysis engine. In such examples, the electronic device 100 may rely on global motion vectors (e.g., global motion vectors estimated using the Harris corner detection algorithm and / or similar algorithms) to calculate image transformation matrices.

[0045] In other examples, such as in the case of intermediate motion, when the estimated motion is above a first threshold (referred to as a lower threshold) and below a second threshold (referred to as an upper threshold) that is greater than the first threshold, the electronic device 100 may switch to using an input image with a reduced resolution based on the calculation of the temporal filtering indication (TFI). For example, the electronic device 100 may reduce the input image to a lower resolution (e.g., by 4, 8, 16, or any other factor) before running a semi-global matching operation on the reduced input image, thereby saving power of the device. When the motion map processing senses the need, the algorithm implemented by the electronic device 100 may revert to full-resolution motion estimation. For example, when the estimated motion is above a threshold (e.g., above a second threshold or an upper threshold), the algorithm may revert to full-resolution motion estimation. In some cases, the algorithm may be smooth and may switch to processing a reduced image, such as an image reduced by a factor of 16, rather than reverting to global motion estimation (e.g., motion vector estimation using Harris corner detection) depending on an assessment of image quality (IQ).

[0046] Figure 2 is a diagram illustrating an example process 200 for a specific implementation of motion estimation. The example process 200 shows a pipeline for motion estimation including global motion estimation, local motion estimation between image frames, and semi-global matching (SGM).

[0047] In this example, the front-end engine 202 downscales the input image from the video stream 204 to generate a downscaled image 206. The front-end engine 202 provides the downscaled image 206 to the video analysis engine 230 for processing. The video analysis engine 230 uses the target image 208 and the reference image 210 to perform motion vector estimation 214. In some examples, the target image 208 may be the same as the downscaled image 206 or may be generated based on the downscaled image 206. In some cases, the motion vector estimation 214 may estimate the motion vector using the Harris corner detection algorithm, etc. In some examples, the motion vector estimation 214 may estimate the global motion associated with the target image 208, the reference image 210, and / or the electronic device 100. In some cases, before processing the target image 208 and the reference image 210, the electronic device 100 may process the target image 208 and the reference image 210 to remove noise from the image.

[0048] The motion vector estimation 214 may generate a motion vector for the target image 208. In some examples, the motion vector may indicate a global motion associated with the target image 208 and / or the electronic device 100. The motion vector generated by the motion vector estimation 214 may then be processed by the alignment block 216 to account for the global motion. In some examples, the alignment block 216 may use the sensor data 212 to align the motion vector generated by the motion vector estimation 214 to account for the global motion associated with the electronic device 100. The sensor data 212 may include one or more measurements obtained by one or more inertial sensors 104 of the electronic device 100. For example, in some cases, the sensor data 212 may include gyroscope measurements obtained by a gyroscope from one or more inertial sensors 104. The gyroscope measurements may include an orientation and / or angular velocity of the electronic device 100 measured by the gyroscope. The alignment block 216 may use the orientation and / or angular velocity of the electronic device 100 to align the motion vector generated by the motion vector estimation 214 to account for the global motion of the electronic device 100 (e.g., to account for the orientation and / or angular velocity of the electronic device 100).

[0049] In some examples, the alignment block 216 may warp the motion vectors from the motion vector estimation 214 based on the sensor data 212 (e.g., based on gyroscope measurements, such as orientation and angular velocity measurements). The alignment block 216 may input the warped motion vectors into the SGM block 218 configured to perform semi-global matching. The SGM block 218 may process the warped motion vectors, the target image 208, and the reference image 210 to generate a dense motion map 220. In some cases, the SGM block 218 may determine the local motion associated with the motion vectors from the motion vector estimation 214. In some examples, the SGM block 218 may compare the target image 208 with the reference image 210 to determine the motion between the target image 208 and the reference image 210. For example, the SGM block 218 may compare the target image 208 with the reference image 210 to determine how the local motion between the target image 208 and the reference image 210 is.

[0050] In some cases, the dense motion map 220 may reflect local motion between the target image 208 and the reference image 210. In some cases, the dense motion map 220 may reflect local motion between the target image 208 and the reference image 210 as well as global motion estimated for the target image 208 and / or the reference image 210. In some examples, the dense motion map 220 may include motion estimates for blocks or regions of image data in the target image 208 (e.g., for each block or region). The blocks or regions of image data may include pixel blocks or regions of the target image 208. For example, the blocks or regions of image data may include NxN pixel blocks (e.g., 4x4, 8x8, etc.). In this example, the dense motion map may include motion estimates for each NxN pixel block in the target image 208.

[0051] The domain change block 222 may use the global stabilization matrix and the dense motion map 220 to generate a transformation matrix 224. For example, the domain change block 222 may use the global stabilization matrix to warp the dense motion map 220 to obtain the transformation matrix 224. The domain change block 222 may provide the transformation matrix 224 to the image processing engine 226, which may use the transformation matrix 224 to generate an output 228. For example, the image processing engine 226 may use the transformation matrix 224 to perform an image stabilization operation on one or more image frames, such as one or more image frames of the video stream 204. To illustrate, the image processing engine 226 may use the transformation matrix 224 to stabilize one or more image frames from the video stream 204.

[0052] Figure 3A is a diagram illustrating an example flow 300 for a specific implementation of motion estimation with a feedback mechanism. In this example, flow 300 includes a higher power consumption pipeline including blocks 214, 216, 218, and 222, and a lower power consumption pipeline including blocks 214 and 222. The electronic device 100 can switch between the higher power consumption pipeline and the lower power consumption pipeline based on local motion estimation, as further described herein.

[0053] In addition, the flow 300 provides a pipeline for motion estimation in which feedback is incorporated between the processor 306 and the video analysis engine 230. The feedback allows the video analysis engine 230 to switch between using motion vectors for certain operations (such as image stabilization) and using dense motion maps for those operations. Switching between using motion vectors and dense motion maps can allow the electronic device 100 to reduce its power consumption without reducing (or with minimal reduction) the accuracy and / or quality of the output generated by the electronic device 100.

[0054] In some examples, switching between the use of motion vectors and the use of dense motion maps may include switching between using a lower resolution image (e.g., a scaled-down image) and using a higher resolution image (e.g., a full resolution image) for a particular operation. In some cases, switching between the use of motion vectors and the use of dense motion maps may be based on the amount of motion of the electronic device 100 and / or the amount of motion between images in the sequence of image frames. For example, when the electronic device 100 determines that the amount of global and / or local motion is below a threshold (e.g., meaning less motion), the electronic device 100 may switch to a motion estimation operation associated with lower power consumption, and when the amount of global and / or local motion is above a threshold (e.g., meaning more motion), the electronic device 100 may switch to a motion estimation operation associated with higher power consumption.

[0055] In the example process 300, the front-end engine 202 downscales the input image from the video stream 204 to generate the downscaled image 206, as previously described with respect to FIG. Figure 2 The front end engine 202 provides the downscaled image 206 to the video analysis engine 230, which performs motion vector estimation 214 to generate the video analysis engine 230 as previously described. Figure 2 Motion vector estimation 214 may generate a motion vector for target image 208. In some examples, the motion vector may indicate a global motion associated with target image 208 and / or electronic device 100. The motion vector generated by motion vector estimation 214 may then be processed by alignment block 216 to account for the global motion. In some examples, alignment block 216 may use sensor data 212 to align the motion vector generated by motion vector estimation 214 to account for the global motion associated with electronic device 100.

[0056] In some examples, the alignment block 216 may warp the motion vectors from the motion vector estimation 214 based on the sensor data 212 (e.g., based on gyroscope measurements, such as orientation and angular velocity measurements). The alignment block 216 may input the warped motion vectors into the SGM block 218, which is configured to perform semi-global matching. The SGM block 218 may process the warped motion vectors and the target image 208 to generate a dense motion map 302.

[0057] The domain change block 222 may use the global stabilization matrix and the dense motion map 302 to generate a transformation matrix 224. For example, the domain change block 222 may use the global stabilization matrix to warp the dense motion map 302 to obtain the transformation matrix 224. The domain change block 222 may provide the transformation matrix 224 to the image processing engine 226, which may use the transformation matrix 224 to generate the output 310. For example, the image processing engine 226 may use the transformation matrix 224 to perform an image stabilization operation on one or more image frames, such as one or more image frames of the video stream 204. To illustrate, the image processing engine 226 may use the transformation matrix 224 to stabilize one or more image frames from the video stream 204.

[0058] In some examples, the image processing engine 226 may generate a temporal filter indicator 304. In some cases, the temporal filter indicator 304 may indicate an amount of motion associated with an input image (e.g., the target image 208), the electronic device 100, and / or the transformation matrix 224. The image processing engine 226 may transmit the temporal filter indicator 304 to a processor 306, which may process the temporal filter indicator 304 to obtain a local motion estimate associated with the input image (e.g., the target image 208) and / or the transformation matrix 224. The processor 306 may include any processing device of the electronic device 100. For example, the processor 306 may include or represent the CPU 112. In other examples, the processor 306 may include or represent any other processor, such as, for example, the GPU 114, the DSP 116, the ISP 118, and / or any other processor.

[0059] The processor 306 may use the temporal filter indicator 304 to provide feedback to the video analytics engine 230 and / or determine what type of feedback to provide to the video analytics engine 230. In some cases, the processor 306 may use the motion information (e.g., local motion estimation and / or global motion estimation) identified in the temporal filter indicator 304 to provide feedback to the video analytics engine 230. For example, depending on the amount of motion indicated in the temporal filter indicator 304, the feedback may instruct the video analytics engine 230 to revert to global motion estimation (with or without local motion estimation), switch to using a lower resolution image to estimate motion, or switch to using local motion estimation (with or without global motion estimation).

[0060] In some cases, if the temporal filter indicator 304 indicates that there is no motion or the amount of motion is below a threshold, the processor 306 may instruct the video analytics engine 230 to revert from the higher power consumption pipeline to the lower power consumption pipeline. In some examples, the higher power consumption pipeline may include a mode in which the video analytics engine 230 implements global and local motion estimation and / or processes full resolution frames instead of downscaled frames. In some examples, the lower power consumption pipeline may include a mode in which the video analytics engine 230 implements global motion estimation but does not implement local motion estimation and / or processes downscaled frames instead of full resolution frames.

[0061] For example, in flow 300, processor 306 determines the amount of local motion identified in temporal filter indicator 304. If temporal filter indicator 304 indicates no local motion or local motion is below a threshold, processor 306 may generate instructions 308 for video analytics engine 230 instructing video analytics engine 230 to revert from global and local motion vector estimation to global motion vector estimation (e.g., a lower power pipeline for motion estimation). In some cases, processor 306 may implement hysteresis logic to trigger a fallback (e.g., triggering instructions 308 instructing video analytics engine 230 to revert to global motion vector estimation without local motion vector estimation) only after a certain number of frames show no local motion or local motion is below a threshold.

[0062] In some examples, if the temporal filter indicator 304 indicates no local motion or that the local motion is below a threshold, the processor 306 may reprogram or reconfigure the video analytics engine 230 to switch from a higher power pipeline invoking the processing of the motion vector estimation 214, the alignment block 216 to align the motion vectors to account for global motion, the SGM block 218, the domain change block 222, and the transformation matrix 224 to a lower power pipeline invoking the processing of the motion vector estimation 214, the domain change block 222, and the transformation matrix 224 (e.g., and excluding the alignment block 216 and the SGM block 218). In general, global motion estimation consumes less power than local motion estimation. Therefore, global motion estimation implemented in a lower power pipeline may result in less power consumption than local motion estimation in a higher power pipeline in which the local motion estimation is implemented in addition to global motion estimation.

[0063] In addition, switching from a higher power consumption pipeline to a lower power consumption pipeline can reduce power consumption of the electronic device 100 when there is no local motion or the local motion is below a threshold, and thus the higher power consumption pipeline is not required to accurately and efficiently estimate motion. Therefore, switching from a higher power consumption pipeline to a lower power consumption pipeline when there is no local motion or the local motion is below a threshold can increase motion estimation efficiency while reducing power consumption at the electronic device 100 without sacrificing the quality of motion estimation (e.g., because there is no local motion or the local motion is below a threshold).

[0064] The processor 306 may continue to process the temporal filter indicator 304 and may trigger a switch to a higher power processing flow (e.g., full resolution motion estimation) once the motion indicated in the temporal filter indicator 304 exceeds a motion threshold. To illustrate, when the temporal filter indicator 304 indicates an amount of local motion above a threshold, the processor 306 may reprogram or reconfigure the video analytics engine 230 to switch back to a higher power pipeline that includes the alignment block 216 and the SGM block 218 (e.g., in addition to the processing of the motion vector estimation 214, the domain change block 222, and the transform matrix 224). For example, when the temporal filter indicator 304 indicates an amount of local motion above a threshold, the processor 306 may generate instructions to reprogram or reconfigure the video analytics engine 230 to switch back to a higher power pipeline that also includes the alignment block 216 and the SGM block 218.

[0065] In some cases, the switch from the higher power consumption pipeline to the lower power consumption pipeline may include a switch from processing a higher resolution frame (e.g., a full resolution version of the target image 208 and the reference image 210) to processing a lower resolution frame (e.g., a lower resolution or downscaled version of the target image 208 and the reference image 210). In some examples, the switch from processing a higher resolution frame (e.g., a full resolution version of the target image 208 and the reference image 210) to processing a lower resolution frame (e.g., a lower resolution or downscaled version of the target image 208 and the reference image 210) may be facilitated by downscaled blocks (not shown) for the target image 208 and the reference image 210.

[0066] In some examples, the threshold for switching between a higher power pipeline and a lower power pipeline can be determined by evaluating image quality (IQ) information, parameters, and / or characteristics. For example, the threshold for switching between a higher power pipeline and a lower power pipeline can be determined by evaluating image reconstruction information, parameters, and / or characteristics. Alternatively or additionally, the threshold for switching between a higher power pipeline and a lower power pipeline can be determined by evaluating power consumption benefits and performance results or benefits associated with switching between a higher power pipeline and a lower power pipeline.

[0067] In some examples, if the motion indicated in the temporal filter indicator 304 is above a first threshold (e.g., the lower threshold described above) but below a second threshold (e.g., the upper threshold described above), the processor 306 may avoid triggering the video analytics engine 230 to switch to a lower power pipeline or continue to implement a higher power pipeline at full resolution. Conversely, when the motion indicated in the temporal filter indicator 304 is above the lower threshold but below the upper threshold, the processor 306 may trigger the video analytics engine 230 to switch to the higher power pipeline to process the target image 208 and the reference image 210 at a reduced image resolution or at an image resolution lower than the image resolution used when the amount of motion exceeds the upper threshold.

[0068] Figure 3B 306 and 307. FIG. 307 is a diagram illustrating another example flow 350 for a motion estimation implementation with a feedback mechanism. In this example, flow 350 provides a pipeline for motion estimation with feedback incorporated by video analytics engine 230 (e.g., as opposed to feedback between processor 306 and video analytics engine 230).

[0069] In this example, the dense motion map 302 generated by the SGM block 218 is fed into a motion map processing block 352, which runs a computational algorithm on the dense motion map 302 to estimate the size and / or amount of motion in the dense motion map 302, and generates instructions 354 to configure (or reconfigure) the process implemented by the video analysis engine 230 based on the size and / or amount of motion in the dense motion map 302. For example, when the size and / or amount of motion in the dense motion map 302 is below a threshold, the instructions 354 generated by the motion map processing block 352 can trigger a switch to a lower power pipeline (e.g., global motion vector estimation). The low power pipeline in this example includes the motion vector estimation 214, the motion vector processing 356 described below, and the domain change block 222.

[0070] In some examples, the motion vector processing 356 in the lower power pipeline may determine whether the magnitude and / or amount of motion in the motion vector generated by the motion vector estimation 214 exceeds a threshold. If the motion vector processing 356 determines that the magnitude and / or amount of motion in the motion vector exceeds the threshold, the motion vector processing 356 may generate instructions 354 to configure or reconfigure the flow implemented by the video analytics engine 230. In some examples, when the magnitude and / or amount of motion in the motion vector from the motion vector estimation 214 exceeds the threshold, the instructions 354 generated by the motion vector processing 356 may trigger a switch back to the higher power flow including the motion vector estimation 214, the alignment block 216, the SGM block 218, and the domain change block 222. Alternatively, if the size and / or amount of motion in the motion vectors from the motion vector estimation 214 does not exceed a threshold, the motion vector processing 356 may provide the motion vectors from the motion vector estimation 214 to the domain change block 222 for processing, and thereby maintain a lower power consumption pipeline including the motion vector estimation 214, the motion vector processing 356, and the domain change block 222.

[0071] Figure 4 4 is a diagram illustrating an example state machine 400 showing various example processing states and thresholds for estimating motion. In this example, the state machine 400 begins with the electronic device 100 implementing a higher resolution motion estimation pipeline 402 (e.g., a higher power consumption pipeline) that includes estimating motion vectors (e.g., motion vector estimation 214), an alignment block for aligning global motion vectors to account for global motion (e.g., alignment block 216), an SGM block for estimating local motion between frames (e.g., SGM block 218), a domain change block (e.g., domain change block 222) that uses a global stabilization matrix to warp a dense motion map (e.g., dense motion map 220) to generate a transformation matrix (e.g., transformation matrix 224), and using the transformation matrix to generate an output, such as (for example) an image stabilized output.

[0072] When implementing the higher resolution motion estimation 402, the electronic device 100 may determine whether the estimated motion in the input / target frame is below a lower threshold or above a lower threshold and below an upper threshold. For example, if the electronic device 100 determines the condition 410 that the amount of estimated motion is below the lower threshold, the electronic device 100 may switch to the lower resolution motion estimation 404 (e.g., a lower power consumption pipeline), and if the electronic device 100 determines the condition 412 that the amount of estimated motion is above the lower threshold but below the upper threshold, the electronic device 100 may switch to the intermediate resolution motion estimation 406 (e.g., a higher power consumption pipeline with a lower resolution image).

[0073] In some examples, lower resolution motion estimation 404 may include global motion estimation instead of local motion estimation. For example, in some cases, lower resolution motion estimation 404 may implement Figure 3A The motion vector estimation 214 and domain change block 222 blocks shown in FIG. Figure 3A The alignment block 216 and the SGM block 218 are shown in FIG.

[0074] In some examples, intermediate resolution motion estimation 406 may include global motion estimation and local motion estimation using the downscaled image. For example, in some cases, intermediate resolution motion estimation 406 may implement Figure 3A The motion vector estimation 214, alignment block 216, SGM block 218, and domain change block 222 blocks shown in FIG. 10 , but may process downscaled images or images compared to those generated by a higher resolution motion estimation 402 (e.g., as described above with respect to FIG. 10 ). Figure 3A The image processed by the higher power pipeline described in the embodiment of the present invention is downscaled to a lower resolution image.

[0075] When the electronic device 100 implements the lower resolution motion estimation 404, the electronic device 100 may check whether the estimated motion exceeds the upper threshold and / or the lower threshold. For example, the electronic device 100 may check whether the estimated motion satisfies the condition 412 that the amount of the estimated motion is higher than the lower threshold but lower than the upper threshold, or whether the estimated motion satisfies the condition 414 that the amount of the estimated motion is higher than the upper threshold. If the estimated motion satisfies the condition 412 that the amount of the estimated motion is higher than the lower threshold but lower than the upper threshold, the electronic device 100 may switch to the intermediate resolution motion estimation 406, and if the estimated motion satisfies the condition 414 that the amount of the estimated motion is higher than the upper threshold, the electronic device 100 may switch to the higher resolution motion estimation 402.

[0076] Similarly, when the electronic device 100 implements the intermediate resolution motion estimation 406, the electronic device 100 may check whether the estimated motion exceeds the upper threshold and / or the lower threshold. For example, the electronic device 100 may check whether the estimated motion satisfies the condition 414 that the amount of the estimated motion is higher than the upper threshold, or check whether the estimated motion satisfies the condition 410 that the amount of the estimated motion is lower than the lower threshold. If the estimated motion satisfies the condition 414 that the amount of the estimated motion is higher than the upper threshold, the electronic device 100 may switch to the higher resolution motion estimation 402, and if the estimated motion satisfies the condition 410 that the amount of the estimated motion is lower than the lower threshold, the electronic device 100 may switch to the lower resolution motion estimation 404.

[0077] Figure 5is a diagram illustrating example state transitions and corresponding thresholds. In this example, the electronic device 100 implements a lower power motion vector estimation 510 (e.g., a lower resolution motion estimation 404) until the amount of estimated motion exceeds a lower threshold 502. When the estimated motion exceeds the lower threshold 502 but remains below the upper threshold 504, the electronic device 100 may switch to motion estimation with a reduced frame 520 (e.g., an intermediate resolution motion estimation 406). Here, the electronic device 100 may implement a higher power consumption pipeline as previously described, but may reduce power consumption by processing reduced images. In addition, when using full resolution images, power consumption is reduced relative to power consumption associated with a higher power consumption pipeline.

[0078] When the estimated motion exceeds the upper threshold 504, the electronic device 100 may switch to motion estimation (e.g., higher resolution motion estimation 402) without the downscaled frame 530. The electronic device 100 may implement a higher power consumption pipeline as previously described without using downscaled images, as implemented when the lower threshold 502 is exceeded but the upper threshold 504 is not exceeded.

[0079] Figure 6 6 is a flow chart illustrating an example process 600 for saving resources when performing motion estimation. Process 600 may be implemented, for example, by electronic device 100 (or a component thereof, such as a chipset) or other electronic or computing device (or a component thereof). At block 602, electronic device 100 (or a component thereof) may determine, based on an input image and a reference image, a motion vector identifying motion between the input image and the reference image. In some examples, electronic device 100 (or a component thereof) may determine, based on the previously described description of the motion vector, a motion vector identifying motion between the input image and the reference image. Figure 3A The described motion vector estimation (eg, motion vector estimation 214) determines the motion vector. In some cases, the electronic device 100 (or a component thereof) may determine the motion vector based on a Harris corner detection algorithm.

[0080] At block 604, electronic device 100 (or a component thereof) may determine whether the motion indicated by the motion vector is below a lower threshold. In some examples, to determine whether the motion indicated by the motion vector is below a lower threshold, electronic device 100 (or a component thereof) may check whether the motion exceeds an upper threshold and / or a lower threshold.

[0081] At block 606, the electronic device 100 (or a component thereof) may refrain from determining local motion between the input image and the reference image based on a determination that the motion indicated by the motion vector is below a lower threshold. In some examples, refraining from determining local motion between the input image and the reference image may be based on a determination to implement a lower power pipeline (e.g., lower resolution motion estimation 404).

[0082] At block 608, the electronic device 100 (or a component thereof) may determine a transformation matrix (e.g., transformation matrix 224) based on the motion vectors and without using local motion between the input image and the reference image. In some examples, to determine the transformation matrix based on the motion vectors, the electronic device 100 (or a component thereof) may warp the motion vectors using a global stabilization matrix.

[0083] At block 610, electronic device 100 (or a component thereof) may adjust the input image based on the transformation matrix. In some examples, to adjust the input image based on the transformation matrix, electronic device 100 (or a component thereof) may stabilize the input image using the transformation matrix.

[0084] In some aspects, the electronic device 100 (or its components) may determine an additional motion vector identifying the motion between the second input image and the second reference image based on the second input image and the second reference image. The electronic device 100 (or its components) may determine whether the motion indicated by the additional motion vector is below or above a lower threshold. The electronic device 100 (or its components) may generate a motion map indicating the local motion between the second input image and the second reference image based on the determination that the motion indicated by the additional motion vector is above the lower threshold. In some examples, the determination that the motion indicated by the additional motion vector is above the lower threshold may also include the determination that the motion indicated by the additional motion vector is below an upper threshold greater than the lower threshold. In some aspects, the electronic device 100 (or its components) may determine an additional transformation matrix based on the motion map. The electronic device 100 (or its components) may adjust the second input image based on the additional transformation matrix. In some cases, in order to determine the additional transformation matrix based on the motion map, the electronic device 100 (or its components) may use a global stability matrix to distort the motion map.

[0085] In some aspects, electronic device 100 (or a component thereof) may align additional motion vectors based on a global motion associated with electronic device 100 before generating the motion map. Electronic device 100 (or a component thereof) may generate the motion map based on the aligned additional motion vectors. In some examples, the additional motion vectors may be aligned based on motion information obtained from a motion sensor (e.g., one or more inertial sensors 104). In some examples, the motion sensor may include a gyroscope and / or an inertial measurement unit.

[0086] In some aspects, the electronic device 100 (or a component thereof) may determine that the motion indicated by the additional motion vector is above a lower threshold and below an upper threshold that is greater than the lower threshold. In some aspects, the electronic device 100 (or a component thereof) may scale down the second input image and the second reference image before determining the additional motion vector identifying the motion between the second input image and the second reference image. In some examples, the electronic device 100 (or a component thereof) may determine the additional motion vector based on the scaled down second input image and the scaled down second reference image.

[0087] In some aspects, the electronic device 100 (or its component) may determine an additional motion vector identifying the motion between the second input image and the second reference image based on the second input image and the second reference image. The electronic device 100 (or its component) may determine whether the motion indicated by the additional motion vector is above a lower threshold and below an upper threshold that is greater than the lower threshold. Based on the determination that the motion indicated by the additional motion vector is above the lower threshold and below the upper threshold, the electronic device 100 (or its component) may reduce the second input image and the second reference image. The electronic device 100 (or its component) may generate a motion map indicating the local motion between the reduced second input image and the reduced second reference image. The electronic device 100 (or its component) may further determine an additional transformation matrix based on the motion map and may adjust the second input image based on the additional transformation matrix. In some examples, in order to determine the additional transformation matrix based on the motion map, the electronic device 100 (or its component) may use a global stability matrix to warp the motion map.

[0088] In some aspects, prior to generating the motion map, the electronic device 100 (or its components) may align additional motion vectors based on a global motion associated with the electronic device. In some cases, the electronic device 100 (or its components) may align additional motion vectors based on motion information obtained from a motion sensor. In some examples, the motion sensor may include a gyroscope or an inertial measurement unit. In some aspects, the electronic device 100 (or its components) may generate the motion map based on the aligned additional motion vectors.

[0089] In some examples, the processes described herein (e.g., process 600 and / or any other process described herein) can be performed by a computing device or apparatus. In one example, process 600 can be performed by Figure 1 In another example, the process 600 may be performed by an electronic device 100 having Figure 7 The computing system 700 shown in FIG. 1 is executed by the computing system 700. For example, Figure 7 The computing device of the computing system 700 shown in FIG. 7 can implement Figure 5 The operation and / or this article is relative to Figures 1 to 5 The components and / or operations described in any figure in the drawings.

[0090] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, an AR glasses, a networked watch or smart watch, or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a laptop, a smart TV, a camera, and / or any other computing device having resource capabilities to perform the processes described herein (including process 600 and / or other processes described herein). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0091] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0092] Process 600 is illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operations. In general, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0093] In addition, process 600 and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, through hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0094] Figure 7 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Figure 7 An example of a computing system 700 is illustrated, which may be, for example, any computing device that constitutes a computing system, a camera system, or any components thereof, where the components of the system communicate with each other using a connection 705. The connection 705 may be a physical connection using a bus, or a direct connection into the processor 710, such as in a chipset architecture. The connection 705 may also be a virtual connection, a networked connection, or a logical connection.

[0095] In some examples, computing system 700 is a distributed system, where the functionality described in the present disclosure may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some examples, one or more of the described system components represent a number of such components, each performing some or all of the functionality for which the component is described. In some examples, each component may be a physical or virtual device.

[0096] The example system 700 includes at least one processing unit (CPU or processor) 710 and connections 705 that couple various system components including system memory 715, such as read only memory (ROM) 720 and random access memory (RAM) 725, to the processor 710. The computing system 700 may include a cache 712 of high-speed memory directly connected to the processor 710, in close proximity to the processor, or integrated as part of the processor.

[0097] Processor 710 may include any general purpose processor and hardware or software services, such as services 732, 734, and 736 stored in storage device 730, that are configured to control processor 710 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 710 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0098] To enable user interaction, the computing system 700 includes an input device 745 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 700 may also include an output device 735 that can be one or more of a plurality of output mechanisms. In some instances, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 700. The computing system 700 may include a communication interface 740, which generally may govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using a wired and / or wireless transceiver, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Wireless signal transfer, Low Energy (BLE) wireless signal transfer, Wireless signal transfer, radio frequency identification (RFID) wireless signal transfer, near field communication (NFC) wireless signal transfer, dedicated short range communication (DSRC) wireless signal transfer, 702.11 Wi-Fi wireless signal transfer, wireless local area network (WLAN) signal transfer, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transfer, public switched telephone network (PSTN) signal transfer, integrated services digital network (ISDN) signal transfer, 3G / 4G / 5G / LTE cellular data network wireless signal transfer, ad hoc network signal transfer, radio wave signal transfer, microwave signal transfer, infrared signal transfer, visible light signal transfer, ultraviolet light signal transfer, wireless signal transfer along the electromagnetic spectrum, or some combination thereof. The communication interface 740 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 700 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0099] The storage device 730 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a magnetic cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory ... card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0100] The storage device 730 may include software services, servers, services, etc., which, when the code defining such software is executed by the processor 710, enables the system to perform functions. In some examples, hardware services that perform specific functions may include software components stored in a computer-readable medium connected to necessary hardware components (such as the processor 710, connection 705, output device 735, etc.) to perform functions. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact disks (CDs) or digital versatile disks (DVDs)), flash memory, memory, or memory devices. Computer-readable media may have stored thereon code and / or machine-executable instructions, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0101] In some examples, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically exclude media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0102] Specific details are provided in the above description to provide a detailed understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can also be practiced without these specific details. For the sake of clarity of explanation, in some instances, the present technology may be presented as including a separate functional block, which includes a device, device component, step or routine in a method embodied in a combination of software or hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid confusing various aspects and examples in unnecessary details. In other instances, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid confusing various aspects and examples.

[0103] The above may describe various examples as processes or methods, which are depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe an operation as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or main function.

[0104] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. These instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Parts of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary, intermediate format instructions such as assembly language, firmware, source code. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memories, USB devices with non-volatile memory, networked storage devices, etc.

[0105] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0106] Instructions, media for communicating such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0107] In the foregoing description, various aspects of the present application are described with reference to the specific examples of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative examples of the present application have been described in detail herein, it should be understood that the inventive concept can be implemented and adopted in various other ways, and the appended claims are intended to be interpreted as including these variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, without departing from the broader essence and scope of this specification, each example and aspect of the present disclosure can be utilized in any number of environments and applications beyond those described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that in an alternative example, the method can be performed in an order different from the described order.

[0108] It should be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0109] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0110] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0111] Claim language or other language in this disclosure that states "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that states "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language that states "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language that states "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0112] Various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, frames, modules, circuits, and steps have been generally described in terms of their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0113] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices, mobile phones, and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a computer-readable data storage medium including a program code, which includes instructions for executing one or more of the above methods, algorithms, and / or operations when executed. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0114] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0115] Exemplary aspects of the present disclosure include:

[0116] Aspect 1. A device for processing image data, the device comprising: at least one memory; at least one processor, the at least one processor being coupled to the at least one memory, the at least one processor being configured to: determine, based on an input image and a reference image, a motion vector indicating motion between the input image and the reference image; determine whether the motion indicated by the motion vector is below a first threshold; avoid determining local motion between the input image and the reference image based on the determination that the motion indicated by the motion vector is below the first threshold; determine a transformation matrix based on the motion vector and without using the local motion between the input image and the reference image; and adjust the input image based on the transformation matrix.

[0117] Aspect 2. The apparatus according to aspect 1, wherein in order to adjust the input image based on the transformation matrix, the at least one processor is configured to stabilize the input image using the transformation matrix.

[0118] Aspect 3. An apparatus according to any one of Aspects 1 or 2, wherein, in order to determine the transformation matrix based on the motion vector, the at least one processor is configured to warp the motion vector using a global stability matrix.

[0119] Aspect 4. An apparatus according to any one of Aspects 1 to 3, wherein the at least one processor is configured to: determine an additional motion vector identifying motion between the second input image and the second reference image based on a second input image and a second reference image; determine whether the motion indicated by the additional motion vector is lower than or higher than the first threshold; based on the determination that the motion indicated by the additional motion vector is higher than the first threshold, generate a motion map indicating local motion between the second input image and the second reference image; determine an additional transformation matrix based on the motion map; and adjust the second input image based on the additional transformation matrix.

[0120] Aspect 5. The apparatus according to aspect 4, wherein, to determine the additional transformation matrix based on the motion map, the at least one processor is configured to warp the motion map using a global stabilization matrix.

[0121] Aspect 6. An apparatus according to any one of Aspects 4 or 5, wherein the at least one processor is configured to: before generating the motion map, align the additional motion vectors based on the global motion associated with the apparatus, the additional motion vectors being aligned based on motion information obtained from a motion sensor, the motion sensor comprising at least one of a gyroscope and an inertial measurement unit; and generate the motion map based on the aligned additional motion vectors.

[0122] Aspect 7. An apparatus according to any one of Aspects 4 to 6, wherein the at least one processor is configured to determine that the motion indicated by the additional motion vector is above the first threshold and below a second threshold that is greater than the first threshold.

[0123] Aspect 8. An apparatus according to any one of Aspects 4 to 7, wherein the at least one processor is configured to: before determining the additional motion vector identifying the motion between the second input image and the second reference image, reduce the second input image and the second reference image, wherein the additional motion vector is determined based on the reduced second input image and the reduced second reference image.

[0124] Aspect 9. An apparatus according to any one of Aspects 1 to 8, wherein the at least one processor is configured to: determine an additional motion vector identifying motion between the second input image and the second reference image based on a second input image and a second reference image; determine whether the motion indicated by the additional motion vector is higher than the first threshold and lower than a second threshold that is greater than the first threshold; and based on the determination that the motion indicated by the additional motion vector is higher than the first threshold and lower than the second threshold: reduce the second input image and the second reference image; generate a motion map indicating local motion between the reduced second input image and the reduced second reference image; determine an additional transformation matrix based on the motion map; and adjust the second input image based on the additional transformation matrix.

[0125] Clause 10. The apparatus according to clause 9, wherein to determine the additional transformation matrix based on the motion map, the at least one processor is configured to warp the motion map using a global stabilization matrix.

[0126] Aspect 11. An apparatus according to any one of Aspects 9 or 10, wherein the at least one processor is configured to: before generating the motion map, align the additional motion vector based on the global motion associated with the apparatus, the additional motion vector being aligned based on motion information obtained from a motion sensor, the motion sensor comprising at least one of a gyroscope and an inertial measurement unit.

[0127] Aspect 12. The apparatus according to aspect 11, wherein to generate the motion map, the at least one processor is configured to generate the motion map based on the aligned additional motion vectors.

[0128] Clause 13. An apparatus according to any one of Clauses 1 to 12, further comprising a camera configured to capture a video sequence comprising the input image and the reference image.

[0129] Aspect 14. An apparatus according to any one of aspects 1 to 13, wherein the apparatus comprises a mobile device.

[0130] Aspect 15. A method for processing image data, the method comprising: determining, by a computing device, a motion vector indicating motion between an input image and a reference image based on the input image and a reference image; determining, by the computing device, whether the motion indicated by the motion vector is lower than a first threshold; avoiding determining local motion between the input image and the reference image based on the determination that the motion indicated by the motion vector is lower than the first threshold; determining, by the computing device, a transformation matrix based on the motion vector and without using the local motion between the input image and the reference image; and adjusting the input image based on the transformation matrix.

[0131] Clause 16. The method according to clause 15, wherein adjusting the input image based on the transformation matrix comprises using the transformation matrix to stabilize the input image.

[0132] Aspect 17. The method according to any one of aspects 15 to 16, wherein determining the transformation matrix based on the motion vector comprises warping the motion vector using a global stabilization matrix.

[0133] Aspect 18. A method according to any one of Aspects 15 to 17, the method further comprising: determining an additional motion vector identifying the motion between the second input image and the second reference image based on the second input image and the second reference image; determining whether the motion indicated by the additional motion vector is lower than or higher than the first threshold; based on the determination that the motion indicated by the additional motion vector is higher than the first threshold, generating a motion map indicating the local motion between the second input image and the second reference image; determining an additional transformation matrix based on the motion map; and adjusting the second input image based on the additional transformation matrix.

[0134] Clause 19. The method according to clause 18, wherein determining an additional transformation matrix based on the motion map comprises warping the motion map using a global stabilization matrix.

[0135] Aspect 20. A method according to any one of Aspects 18 or 19, the method further comprising: before generating the motion map, aligning the additional motion vectors based on the global motion associated with the computing device, the additional motion vectors being aligned based on motion information obtained from a motion sensor, the motion sensor comprising at least one of a gyroscope and an inertial measurement unit; and generating the motion map based on the aligned additional motion vectors.

[0136] Aspect 21. The method according to any one of aspects 18 to 20, further comprising: determining that the motion indicated by the additional motion vector is above the first threshold and below a second threshold greater than the first threshold.

[0137] Aspect 22. A method according to any one of Aspects 18 to 21, the method further comprising: before determining the additional motion vector identifying the motion between the second input image and the second reference image, reducing the second input image and the second reference image, wherein the additional motion vector is determined based on the reduced second input image and the reduced second reference image.

[0138] Aspect 23. A method according to any one of Aspects 15 to 22, the method further comprising: determining an additional motion vector identifying the motion between the second input image and the second reference image based on the second input image and the second reference image; determining whether the motion indicated by the additional motion vector is higher than the first threshold and lower than a second threshold that is greater than the first threshold; and based on the determination that the motion indicated by the additional motion vector is higher than the first threshold and lower than the second threshold: reducing the second input image and the second reference image; generating a motion map indicating the local motion between the reduced second input image and the reduced second reference image; determining an additional transformation matrix based on the motion map; and adjusting the second input image based on the additional transformation matrix.

[0139] Clause 24. The method according to clause 23, wherein determining an additional transformation matrix based on the motion map comprises warping the motion map using a global stabilization matrix.

[0140] Aspect 25. The method according to any one of Aspects 23 or 24, further comprising: before generating the motion map, aligning the additional motion vector based on the global motion associated with the computing device, the additional motion vector being aligned based on motion information obtained from a motion sensor, the motion sensor comprising at least one of a gyroscope and an inertial measurement unit.

[0141] Aspect 26. The method of aspect 25, wherein generating the motion map comprises generating the motion map based on the aligned additional motion vectors.

[0142] Aspect 27. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform the operations according to any one of Aspects 1 to 26.

[0143] Aspect 28. An apparatus for processing image data, the apparatus comprising: one or more components for performing the operations according to any one of aspects 1 to 26.

Claims

1. A device for processing image data, the device comprising: at least one memory; as well as at least one processor, the at least one processor coupled to the at least one memory, the at least one processor configured to: determining, based on an input image and a reference image, a motion vector indicating motion between the input image and the reference image; determining whether the motion indicated by the motion vector is below a first threshold; refraining from determining local motion between the input image and the reference image based on a determination that the motion indicated by the motion vector is below the first threshold; determining a transformation matrix based on the motion vector and without using the local motion between the input image and the reference image; as well as The input image is adjusted based on the transformation matrix. 2 . The apparatus of claim 1 , wherein to adjust the input image based on the transformation matrix, the at least one processor is configured to stabilize the input image using the transformation matrix. 3 . The apparatus of claim 1 , wherein to determine the transformation matrix based on the motion vector, the at least one processor is configured to warp the motion vector using a global stabilization matrix.

4. The apparatus of claim 1 , wherein the at least one processor is configured to: determining, based on a second input image and a second reference image, an additional motion vector identifying motion between the second input image and the second reference image; determining whether the motion indicated by the additional motion vector is below or above the first threshold; generating a motion map indicating local motion between the second input image and the second reference image based on a determination that the motion indicated by the additional motion vector is above the first threshold; determining an additional transformation matrix based on the motion map; as well as The second input image is adjusted based on the additional transformation matrix. 5 . The apparatus of claim 4 , wherein to determine an additional transformation matrix based on the motion map, the at least one processor is configured to warp the motion map using a global stabilization matrix.

6. The apparatus of claim 4, wherein the at least one processor is configured to: Prior to generating the motion map, aligning the additional motion vectors based on a global motion associated with the device, the additional motion vectors aligned based on motion information obtained from a motion sensor, the motion sensor comprising at least one of a gyroscope and an inertial measurement unit; and The motion map is generated based on the aligned additional motion vectors. 7 . The apparatus of claim 4 , wherein the at least one processor is configured to determine that the motion indicated by the additional motion vector is above the first threshold and below a second threshold that is greater than the first threshold.

8. The apparatus of claim 4, wherein the at least one processor is configured to: The second input image and the second reference image are downscaled before determining the additional motion vector identifying motion between the second input image and the second reference image, wherein the additional motion vector is determined based on the downscaled second input image and the downscaled second reference image.

9. The apparatus of claim 1 , wherein the at least one processor is configured to: determining, based on a second input image and a second reference image, an additional motion vector identifying motion between the second input image and the second reference image; determining whether the motion indicated by the additional motion vector is above the first threshold and below a second threshold that is greater than the first threshold; as well as Based on a determination that the motion indicated by the additional motion vector is above the first threshold and below the second threshold: downscaling the second input image and the second reference image; generating a motion map indicating local motion between the scaled down second input image and the scaled down second reference image; determining an additional transformation matrix based on the motion map; as well as The second input image is adjusted based on the additional transformation matrix.

10. The apparatus of claim 9, wherein to determine an additional transformation matrix based on the motion map, the at least one processor is configured to warp the motion map using a global stabilization matrix.

11. The apparatus of claim 9, wherein the at least one processor is configured to: Prior to generating the motion map, the additional motion vectors are aligned based on a global motion associated with the device, the additional motion vectors being aligned based on motion information obtained from a motion sensor, the motion sensor comprising at least one of a gyroscope and an inertial measurement unit. 12 . The apparatus of claim 11 , wherein to generate the motion map, the at least one processor is configured to generate the motion map based on the aligned additional motion vectors.

13. The apparatus of claim 1, further comprising a camera configured to capture a video sequence comprising the input image and the reference image.

14. The apparatus of claim 1, wherein the apparatus comprises a mobile device.

15. A method for processing image data, the method comprising: determining, by a computing device, a motion vector indicating motion between the input image and the reference image based on the input image and the reference image; determining, by the computing device, whether the motion indicated by the motion vector is below a first threshold; refraining from determining local motion between the input image and the reference image based on a determination that the motion indicated by the motion vector is below the first threshold; determining, by the computing device, a transformation matrix based on the motion vector and without using the local motion between the input image and the reference image; as well as The input image is adjusted, by the computing device, based on the transformation matrix.

16. The method of claim 15, wherein adjusting the input image based on the transformation matrix comprises using the transformation matrix to stabilize the input image.

17. The method of claim 15, wherein determining the transformation matrix based on the motion vector comprises warping the motion vector using a global stabilization matrix.

18. The method according to claim 15, further comprising: determining, based on a second input image and a second reference image, an additional motion vector identifying motion between the second input image and the second reference image; determining whether the motion indicated by the additional motion vector is below or above the first threshold; generating a motion map indicating local motion between the second input image and the second reference image based on a determination that the motion indicated by the additional motion vector is above the first threshold; determining an additional transformation matrix based on the motion map; as well as The second input image is adjusted based on the additional transformation matrix.

19. The method of claim 18, wherein determining an additional transformation matrix based on the motion map comprises warping the motion map using a global stabilization matrix.

20. The method according to claim 18, further comprising: prior to generating the motion map, aligning the additional motion vectors based on a global motion associated with the computing device, the additional motion vectors aligned based on motion information obtained from a motion sensor, the motion sensor comprising at least one of a gyroscope and an inertial measurement unit; as well as The motion map is generated based on the aligned additional motion vectors.

21. The method according to claim 18, further comprising: The motion indicated by the additional motion vector is determined to be above the first threshold and below a second threshold that is greater than the first threshold.

22. The method according to claim 18, further comprising: The second input image and the second reference image are downscaled before determining the additional motion vector identifying motion between the second input image and the second reference image, wherein the additional motion vector is determined based on the downscaled second input image and the downscaled second reference image.

23. The method according to claim 15, further comprising: determining, based on a second input image and a second reference image, an additional motion vector identifying motion between the second input image and the second reference image; determining whether the motion indicated by the additional motion vector is above the first threshold and below a second threshold that is greater than the first threshold; as well as Based on a determination that the motion indicated by the additional motion vector is above the first threshold and below the second threshold: downscaling the second input image and the second reference image; generating a motion map indicating local motion between the scaled down second input image and the scaled down second reference image; determining an additional transformation matrix based on the motion map; as well as The second input image is adjusted based on the additional transformation matrix.

24. The method of claim 23, wherein determining an additional transformation matrix based on the motion map comprises warping the motion map using a global stabilization matrix.

25. The method according to claim 23, further comprising: Prior to generating the motion map, the additional motion vectors are aligned based on a global motion associated with the computing device, the additional motion vectors being aligned based on motion information obtained from a motion sensor, the motion sensor comprising at least one of a gyroscope and an inertial measurement unit.

26. The method of claim 25, wherein generating the motion map comprises generating the motion map based on the aligned additional motion vectors.

27. A non-transitory computer readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: determining, based on an input image and a reference image, a motion vector indicating motion between the input image and the reference image; determining whether the motion indicated by the motion vector is below a first threshold; refraining from determining local motion between the input image and the reference image based on a determination that the motion indicated by the motion vector is below the first threshold; determining a transformation matrix based on the motion vector and without using the local motion between the input image and the reference image; as well as The input image is adjusted based on the transformation matrix.

28. The non-transitory computer-readable medium of claim 27, wherein to adjust the input image based on the transformation matrix, the instructions, when executed by the one or more processors, cause the one or more processors to stabilize the input image using the transformation matrix.

29. The non-transitory computer-readable medium of claim 27, wherein to determine the transformation matrix based on the motion vector, the instructions, when executed by the one or more processors, cause the one or more processors to warp the motion vector using a global stabilization matrix.

30. The non-transitory computer readable medium of claim 27, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: determining, based on a second input image and a second reference image, an additional motion vector identifying motion between the second input image and the second reference image; determining whether the motion indicated by the additional motion vector is below or above the first threshold; generating a motion map indicating local motion between the second input image and the second reference image based on a determination that the motion indicated by the additional motion vector is above the first threshold; determining an additional transformation matrix based on the motion map; as well as The second input image is adjusted based on the additional transformation matrix.

Images