Optical Image Stabilization Movement for Creating Super-Resolution Images of Scenes

Abstract

Optical image stabilization movement for creating a super-resolution image of a scene is disclosed. This disclosure describes systems and techniques for optical image stabilization movement to create a super-resolution image of a scene. The systems and techniques include a user device introducing movement into one or more components of a camera system of the user device via an optical image stabilization system. The user device then captures a corresponding plurality of frames of an image of the scene, where the corresponding plurality of frames of the image of the scene have corresponding sub-pixel offsets of the image of the scene across the plurality of frames as a result of the movement introduced into the one or more components of the camera system. The user device performs super-resolution calculations based on the corresponding sub-pixel offsets of the image of the scene across the corresponding plurality of frames and creates a super-resolution image of the scene based on the super-resolution calculations.

Description

[0001] Division Explanation

[0002] This application is a divisional application. The application number of the original application is 201980045299.3, the filing date is August 6, 2019, and the invention title is "Optical Image Stabilization Movement for Creating Super-Resolution Images of a Scene". Technical Field

[0003] This application relates to optical image stabilization movement for creating super-resolution images of a scene. Background Art

[0004] Camera technologies associated with current user devices, such as digital single-lens reflex (DLSR) cameras or smartphones with image sensors, have advanced in several aspects. As an example, a user device may include an optical image stabilization (OIS) camera module that combines a camera system with an image sensor and an OIS system to compensate for the movement of the user device while the user device is capturing an image. As another example, a user device may include a super-resolution capability to create a super-resolution image from multiple low-resolution images of a scene, where the multiple low-resolution images of the scene are multiple images captured using the native resolution capability of the camera system and the super-resolution image has a resolution higher than the native resolution capability of the camera system.

[0005] In some cases, the use of the OIS system is incompatible or conflicts with the super-resolution capability of the user device. For example, if the multiple, low-resolution images of the scene need to reflect a positional shift (or difference in pixel positions) of the image of the scene for the super-resolution algorithm to work, the stability introduced by the OIS system during the capture of the multiple low-resolution images of the scene may prevent the required positional shift. As another example, if the user device is in a stable state without movement (and the OIS system is not functioning or compensating for movement during the capture of multiple low-resolution images), the multiple low-resolution images may be the same and do not possess the required positional shift.

[0006] Furthermore, a user device may capture multiple images of a scene in various modes including a burst sequence mode that occurs before and after the shutter button is pressed and during a zero shutter lag mode that occurs almost simultaneously with the pressing of the shutter button. In such cases, the aforementioned incompatibility between the OIS system and the super-resolution capability of the user device further complicates their reconciliation. Summary of the Invention

[0007] The present disclosure describes systems and techniques for optically image-stabilized movement to create a super-resolution image of a scene. The systems and techniques include a user device using an optical image stabilization system to introduce movement into one or more components of the user device's camera system. The user device then captures a corresponding plurality of frames of an image of the scene, where the corresponding plurality of frames of the image of the scene have corresponding sub-pixel offsets of the image of the scene across the plurality of frames as a result of the movement introduced into the one or more components of the camera system. The user device performs super-resolution calculations based on the corresponding sub-pixel offsets of the image of the scene across the corresponding plurality of frames and creates a super-resolution image of the scene based on the super-resolution calculations.

[0008] Thus, the systems and techniques can create super-resolution images without degrading the performance of the optical image stabilization (OIS) system. Additionally, these systems and techniques can stabilize one or more components of the camera system without distracting the user, operate in a "zero shutter lag" mode, and do not require waiting for the user to press the shutter to move, minimizing input lag and latency that can be detrimental to obtaining high-quality images and the user experience.

[0009] In some aspects, a method for creating a super-resolution image of a scene is described. The method includes a user device introducing movement into one or more components of the user device's camera system. As part of the method, the user device captures a corresponding plurality of frames of an image of the scene, where the corresponding plurality of frames of the image of the scene have corresponding sub-pixel offsets of the image of the scene across the plurality of frames as a result of the movement introduced into the one or more components of the camera system. The method continues with the user device performing super-resolution calculations based on the corresponding sub-pixel offsets of the image of the scene across the corresponding plurality of frames and creating a super-resolution image of the scene based on the super-resolution calculations.

[0010] In still other aspects, a user device is described. The user device includes a camera system with an image sensor and an OIS system, a processor, and a display. The user device also includes a computer-readable storage medium that stores instructions for an optical image stabilization system driver manager application and a super-resolution manager application, which, when executed by the processor, perform complementary functions that guide the user device to perform a series of operations.

[0011] The series of operations includes receiving, by one or more processors, a command to direct a user device to capture an image of a scene. The series of operations also includes introducing movement into one or more components of a camera system during the capturing of the image of the scene, wherein the introduced movement causes a corresponding plurality of frames of the image of the scene to be captured, having corresponding sub-pixel offsets of the image across the plurality of frames, and performing, by one or more processors, super-resolution calculations based on the corresponding sub-pixel offsets of the image of the scene across the plurality of frames. The series of operations further includes creating, by one or more processors, a super-resolution image of the scene based on the super-resolution calculations and rendering the super-resolution image of the scene via a display.

[0012] Details of one or more embodiments are set forth in the accompanying drawings and the following description. Other features and advantages will be apparent from the description and drawings, and from the claims. This summary is provided to introduce a subject that is further described in the detailed description and drawings. Accordingly, the reader should not regard this summary as defining essential features or as limiting the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] This disclosure describes details of one or more aspects associated with optically image-stabilized movement to create a super-resolution image of a scene.

[0014] Figure 1 An example operating environment is illustrated in which aspects of optically image-stabilized movement are performed to create a super-resolution image scene.

[0015] Figure 2 Example aspects of movement introduced by an OIS system are illustrated.

[0016] Figure 3 Example aspects associated with detecting a motion condition and introducing movement into an OIS system are illustrated.

[0017] Figure 4 Example aspects of performing super-resolution calculations and creating a super-resolution image of a scene using multiple frames with sub-pixel offsets are illustrated.

[0018] Figure 5 An example method for optically image-stabilized movement to create a super-resolution image of a scene is illustrated. DETAILED DESCRIPTION

[0019] This disclosure describes systems and techniques for optically image-stabilized movement to create a super-resolution image of a scene.

[0020] Although the features and concepts of the described systems and techniques for optically image-stabilized movement to create a super-resolution image of a scene can be implemented in any number of different environments, systems, devices, and / or various configurations, aspects are described in the context of the following example devices, systems, and configurations.

[0021] Example operating environment

[0022] Figure 1 Illustrated is an example operating environment 100 in which various aspects of optically image-stabilized movement are performed to create a super-resolution image of a scene. As illustrated, user device 102 is secured to tripod 104 and renders a super-resolution image 106 of the scene it is capturing. Although illustrated as a smart phone, user device 102 can be another type of device having image capture capabilities, such as a DLSR camera or a tablet. The securing of user device 102 to tripod 104 constrains user device 102 such that user device 102 is stationary and does not move.

[0023] User device 102 includes a combination of one or more motion sensors 108 (e.g., gyroscopes, accelerometers) that detect the motion condition of user device 102. In some cases, such as when user device 102 is secured to tripod 104, the detected motion condition can be a static motion condition (i.e., user device 102 does not move). In other cases, such as when user device 102 is removed from tripod 104, the detected motion condition can be a dynamic motion condition (i.e., any motion condition that is not a static motion condition).

[0024] User device also includes an optical image stabilization (OIS) camera module 110, which includes a camera system 112 and an OIS system 114. Camera system 112 can include multiple components, such as lenses, apertures, and one or more image sensors (e.g., 40 megapixels (MP), 32MP, 16MP, 8MP). The image sensor can include a complementary metal oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor. In some cases, the image sensor can include a color filter array (CFA) that covers the pixels of the image sensor and limits the intensity of light recorded by the pixels as associated with color wavelengths. An example of such a CFA is a Bayer CFA, which filters light according to red, blue, and green wavelengths.

[0025] Typically, the OIS system 114 provides a mechanism for modifying the physical position or orientation of one or more components of the camera system 112. The OIS system 114 can include a micro motor and a magnetic induction positioning mechanism, which can change the in-plane position, out-of-plane position, pitch, yaw, or tilt of one or more components of the camera system 112. The in-plane position can be a position located within a plane defined by two axes, while the out-of-plane position can be another position located outside the plane defined by two axes.

[0026] The user device 102 also includes a combination of one or more processors 116. The processor 116 can be a single-core processor or a multi-core processor composed of various materials such as silicon, polysilicon, high-K dielectrics, copper, etc. In the case of multiple processors 116 (e.g., a combination of more than one processor), the multiple processors 116 can include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or an image processing unit (IPU). Additionally, and in this case, the multiple processors 116 can use pipeline processing to perform two or more computational operations.

[0027] The user device 102 also includes a computer-readable storage medium (CRM) 118, which includes executable instructions in the form of an OIS driver manager 120 and a super-resolution manager 122. The CRM 118 described herein excludes propagated signals. The CRM 118 can include any suitable memory or storage device, such as random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), non-volatile RAM (NVRAM), read-only memory (ROM), or flash memory that can be used to store the OIS driver manager 120 and the super-resolution manager 122.

[0028] In some cases, the OIS driver manager 120 and the super-resolution manager 122 can be executed by the processor 116 to cause the user device 102 to perform complementary functions that effectively generate OIS movement (e.g., physical movement or change in the position of one or more elements of the camera system 112), capture multiple frames of an image of a scene with sub-pixel offsets, and compute a super-resolution image (106) of the scene. In some cases, the OIS movement can correspond to a change in the position of the camera system 112 that would manifest through the natural, hand-held movement of a user operating the user device (e.g., the user's hand jitter or other natural biomechanics).

[0029] The code or instructions of the OIS driver manager 120 can be executed by the processor 116 to direct the OIS system 114 to perform OIS movement. In some cases, the OIS movement can be an asynchronous movement (e.g., a movement within a predetermined positional displacement range and without time synchronization). Such an asynchronous movement can be applied to static motion conditions.

[0030] In other cases, the OIS movement can be a synchronous movement (e.g., a movement corresponding to a positional displacement range that depends on and is temporally synchronized with the detected movement of the user device 102). Such a synchronous movement can be applicable to dynamic motion conditions and can be superimposed on another OIS movement introduced into the OIS system 114.

[0031] The code or instructions of the super-resolution manager 122, when executed by the processor 116, can direct the user device 102 to perform multiple operations that are complementary to the operations performed under the direction of the OIS driver manager 120. Such operations can include directing the camera system 112 to capture multiple frames of an image of a scene in a burst sequence, performing super-resolution calculations, and rendering a super-resolution image 106 of the scene using the display 124 of the user device.

[0032] Due to the OIS movement (e.g., the movement introduced into the camera system 112 by the OIS system 114 under the direction of the OIS driver manager 120), the camera system 112 can capture multiple variations 126, 128, and 130 of an image of a scene, where the multiple variations 126 - 130 correspond to multiple frames of an image of the scene that have corresponding sub-pixel offsets of the image across the multiple frames.

[0033] During the OIS movement, the OIS driver manager 120 can analytically or numerically calculate a coverage score by uniformly sampling multiple frames of the scene image. In some cases, the calculated coverage score can result in additional or supplementary OIS movement to define the image content of the captured multiple variations 126 - 130. This movement can ensure full coverage of the scene image and produce higher-quality images.

[0034] The super-resolution calculation can include, for example, Gaussian Radial Basis Function (RBF) calculation in combination with robust model calculation to create the super-resolution image 106 of the scene. The super-resolution calculation uses the variations 126 - 130 of the scene image (e.g., multiple frames of the scene image that have corresponding sub-pixel offsets of the image across the multiple frames) to create the super-resolution image 106 of the scene.

[0035] Figure 2 Illustrative example aspect 200 of the movement introduced by the OIS system. Aspect 200 includes Figure 1the camera system 112 and the OIS system 114 of the user equipment 102.

[0036] As illustrated, the OIS system 114 can introduce movement into the camera system 112. "In-plane" movement can be movement contained within the plane defined by the x-axis 202 and the y-axis 204. The "plane" defined by the x-axis 202 and the y-axis 204 corresponds to (e.g., is parallel to) another plane having the super-resolution image 106 of the scene. "Out-of-plane" movement can be movement outside the plane defined by the x-axis 202 and the y-axis 204, and can be movement along the z-axis 206 or another movement including pitch, yaw, or roll.

[0037] The movement introduced by the OIS system 114 can be triggered by one or more factors. For example, the movement introduced into the OIS system 114 can be in response to the user equipment 102 entering the viewfinder mode, in response to the user equipment 102 receiving a capture command, or in response to the user equipment 102 detecting a motion condition via the motion sensor 108. The movement introduced by the OIS system 114 can also be triggered by a combination of one or more of these factors.

[0038] Figure 3 Illustrate example aspect 300 associated with detecting a motion condition and introducing movement into the OIS system. Example aspect 300 can be performed by Figure 1 the user equipment 102 and Figure 2 the OIS camera module 110. The OIS drive manager 120 (executed by the processor 116) can introduce movement into the camera system 112 via the OIS system 114. In some cases, the movement introduced into the camera system 112 via the OIS system 114 can simply stabilize the camera during the capture of the scene image, while in other cases, the movement introduced into the camera system 112 via the OIS system 114 can introduce sub-pixel offsets into multiple images of the scene during the capture of multiple images of the scene. In still some other cases, the movement introduced into the camera system 112 via the OIS system 114 can be a combination of movement that stabilizes the capture of some frames of the scene image (e.g., no sub-pixel offset for clarity purposes) and also introduces an offset during the capture of other frames of the scene image (e.g., retains sub-pixel offset for super-resolution purposes).

[0039] Example aspect 300 includes detection 302 of a motion condition, introduction 304 of movement, and capture 306 of multiple frames with sub-pixel offsets (frames 308 - 312 correspond to Figure 1Variant 126 - 128). Multiple frames 306 are used as a basis for calculating and forming the super - resolution image 106. During the burst sequence, the user equipment 102 can capture multiple frames 306 using a resolution that is another resolution lower than the super - resolution image 106 of the scene.

[0040] The detection 302 of the motion condition can include detecting a static motion condition 314 or a dynamic motion condition 318. The introduction 304 of movement into the camera system 112 can include the OIS drive manager 120 introducing an asynchronous OIS movement 316 or a synchronous OIS movement 320 during the burst sequence.

[0041] In the case of the synchronous OIS movement 320, the OIS system 114 can introduce a first movement that is intended to stabilize the camera system 112 according to the detected dynamic motion condition 318 (e.g., to stabilize the camera system 112 to capture frames of an image of the scene clearly and without sub - pixel offset). The synchronous movement 320 can be a second movement that is synchronous with and superimposed on the first movement (e.g., is a movement opposite to the movement associated with the detected dynamic motion condition 318), and is intended to generate or retain a sub - pixel offset of the frames of the image across the scene. Depending on the amplitude of the first movement, the superposition of the second movement can be positive or negative in nature (e.g., "added to" the first movement or "subtracted" from the first movement). In addition, some parts of the second movement can be larger than a pixel, while other parts of the second movement can be smaller than a pixel.

[0042] The burst sequence can include capturing multiple frames 306 at set time intervals, which can range, for example, from one millisecond to three milliseconds, from one millisecond to five milliseconds, or from half a millisecond to ten milliseconds. Additionally, and in some cases, the time interval of the burst sequence can be variable based on the movement of the user equipment (e.g., the time interval during high - speed movement of the user equipment 102 can be "shorter" than another time interval during low - speed movement of the user equipment 102 to maintain an offset less than one pixel).

[0043] The introduction of motion 304 during a burst sequence causes the user equipment 102 to capture multiple frames 306 such that the multiple frames 306 have corresponding relative sub-pixel offsets. As illustrated, the image of frame 310 is horizontally offset by half a pixel and vertically offset by half a pixel relative to the image of frame 308, respectively. Additionally, and as illustrated, the image of frame 312 is horizontally offset by a quarter of a pixel relative to the image of frame 308, respectively. The corresponding relative sub-pixel offsets can include different magnitudes and combinations of sub-pixel offsets (e.g., one sub-pixel offset associated with one frame may be a quarter of a pixel horizontally and three quarters of a pixel vertically, while another sub-pixel offset associated with another frame may be zero pixels horizontally and half a pixel vertically). Generally, the techniques and systems described by this disclosure can accommodate more random sub-pixel offsets than the illustration and description of frames 308 - 312, including non-uniform sub-pixel offsets.

[0044] Figure 4 FIG. illustrates an example aspect 400 of performing super-resolution calculations and creating a super-resolution image of a scene using multiple frames 306 having sub-pixel offsets introduced by OIS motion. The example aspect 400 may use Figures 1 - 3 components, where the super-resolution calculations are performed by Figure 1 the user equipment 102, the sub-pixel offsets are the result of Figure 2 the OIS motion of Figure 3 and the multiple frames having sub-pixel offsets are

[0045] As Figure 4 illustrated, the multiple frames 306 are input into a super-resolution calculation 402 performed by the user equipment 102. The algorithm supporting the super-resolution calculation 402 may reside in the super-resolution manager 122 of the user equipment 102. In some cases, the user equipment 102 (e.g., multiple processors 116) may use pipelined processing to perform parts of the super-resolution calculation 402. Generally, the super-resolution calculation 402 may use various algorithms and techniques.

[0046] A first example of the super-resolution calculation 402 includes Gaussian Radial Basis Function (RBF) kernel calculations. To perform Gaussian RBF kernel calculations, the user equipment 102 filters the pixel signals from each of the multiple frames 306 to generate corresponding color-specific image planes corresponding to color channels. Then, the user equipment 102 aligns the corresponding color-specific image planes to a selected reference frame.

[0047] Continuing with the first example of the super-resolution calculation 402, the Gaussian RBF kernel calculations may include the user equipment 102 analyzing the local gradient structure tensor to calculate the covariance matrix. Calculating the covariance matrix may rely on the following mathematical relationships:

[0048]

[0049] In the mathematical relationship (1), Ω represents the kernel covariance matrix, e1 and e2 represent orthogonal direction vectors and two associated eigenvalues λ1 and λ2, and k1 and k2 control the desired kernel variances.

[0050] Calculating the local gradient structure tensor can rely on the following mathematical relationship:

[0051]

[0052] In the mathematical relationship (2), I x and I y represent the local image gradients in the horizontal and vertical directions, respectively.

[0053] Additionally, as part of the first example of super-resolution calculation 402, the user equipment 102 can use a statistical neighborhood model to calculate a robustness model, which includes color mean and spatial standard deviation calculations. In some cases, the robustness model calculation can include denoising calculations to compensate for color differences.

[0054] The super-resolution image calculation 402 effectively estimates the contribution of pixel pairs to the color channels associated with the corresponding color planes for each of the multiple frames 306 (e.g., for frames 308, 310, and 312).

[0055] Continuing with the first example of super-resolution calculation 402, the following mathematical relationship for normalization calculation can be used to accumulate color planes:

[0056]

[0057] In the mathematical relationship (3), x and y represent pixel coordinates, the sum Σ n operates on the contribution frames (or the sum of contribution frames), the sum Σ i is the sum of samples within the local neighborhood, c n,i represents the Bayer pixel value at a given frame n and sample i, w n,i represents the local sample weight, and represents the local robustness. The accumulated color planes can be combined to create a super-resolution image 106 of the scene.

[0058] The second example of super-resolution calculation 402 can include analyzing the effect of motion blur. Such super-resolution calculation can use a "volume of solutions" calculation method to analyze the effect of motion blur of multiple images (e.g., multiple frames 306), and the "volume of solutions" calculation method solves the uncertainty in pixel measurements due to quantization errors.

[0059] A third example of super-resolution calculation 402 may include calculations using a frame loop method. Such a method may iteratively use previous low-resolution frames, previously calculated high-resolution images, and the current low-resolution frame to create the current super-resolution image (e.g., the previous low-resolution frame and the current low-resolution frame may be Frame 308 and Frame 310, respectively). The loop method may include flow estimation to estimate a normalized, low-resolution flow map, magnifying the low-resolution flow map using a scaling factor to produce a high-resolution flow map, using the high-resolution flow map to warp the previous high-resolution image, mapping the warped, previous high-resolution image to the low-resolution space, and splicing the mapping in the low-resolution space to the current super-resolution image.

[0060] Figure 4 The super-resolution calculation 402 may also use algorithms and techniques different from or in addition to those described in the previous examples. Such algorithms and techniques may include algorithms and techniques of machine learning. In any case, and in accordance with the present disclosure, the introduction of movement into one or more components of the camera system 112 may be applicable to other algorithms and techniques.

[0061] Example method

[0062] Reference Figure 5 Describe an example method 500 for creating a super-resolution image of a scene in connection with optical image stabilization movement. Generally, any components, modules, methods, and operations described herein may be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Some operations of the example method may be described in the general context of executable instructions stored on a computer-readable storage memory local and / or remote to a computer processing system, and implementations may include software applications, programs, functions, etc. Alternatively or additionally, any functionality described herein may be performed at least in part by one or more hardware logic components, such as but not limited to field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip (SoCs), or complex programmable logic devices (CPLDs).

[0063] Figure 5 Illustrate example aspects of method 500 used as part of creating a super-resolution image of a scene. Method 500 is described in the form of a set of blocks 502-508, which specify operations that may be performed. However, the operations are not necessarily limited to Figure 5 the order shown or described herein, and the operations may be implemented in an alternative order, in a fully or partially overlapping manner, or in an iterative manner. Additionally, although it will be in the user device 102 used by Figure 1 Figures 2 - 4The operations represented by method 500 are described in the context of being performed by components of, but may be performed (or portions of the operations may be performed) by one or more other devices having computing capabilities, such as a server or a cloud computing device that includes instructions (or portions of instructions) of the super-resolution manager 122.

[0064] At block 502, the user device 102 (e.g., the processor 116 that executes the code of the OIS drive manager 120) introduces movement into one or more components of the camera system 112 (e.g., the OIS system 114 introduces movement into the camera system 112). In some cases, the user device 102 may introduce movement in response to entering the viewfinder mode, while in other cases, the user device 102 may introduce movement in response to receiving a capture command. Introducing movement may include introducing in-plane movement or out-of-plane movement into one or more components of the camera system 112.

[0065] At block 504, the user device 102 captures a corresponding plurality of frames 306 of the scene as a result of the movement introduced into one or more components of the camera system 112, and the corresponding plurality of frames 306 have corresponding sub-pixel offsets of the image of the scene 306 across the plurality of frames.

[0066] At block 506, the user device 102 (e.g., the processor 116 that executes the instructions of the super-resolution manager 122) performs a super-resolution calculation 402 based on the corresponding sub-pixel offsets of the scene image across the plurality of frames. Examples of the super-resolution calculation 402 include: (i) calculating a Gaussian radial basis function kernel and calculating a robustness model, (ii) analyzing the effects of motion blur across the plurality of frames, and (iii) using a frame cycling method that uses previous low-resolution frames and a current low-resolution frame from the plurality of frames to create a current super-resolution image.

[0067] At block 508, based on the super-resolution calculation 402, the user device 102 (e.g., the processor 116 that executes the instructions of the super-resolution manager 122) creates a super-resolution image 106 of the scene.

[0068] The example method 500 may further include varying the movement based on a coverage score calculated by sampling a plurality of frames of the image of the scene.

[0069] Although the present disclosure describes systems and techniques for optical image stabilization movement to create a super-resolution image of a scene, it is to be understood that the subject matter of the appended claims need not be limited to the specific features or methods described. Rather, the specific features and methods are disclosed as an example manner in which a super-resolution image of a scene can be created using optical image stabilization movement.

[0070] In addition to the above description, users can be provided with controls that allow them to make choices regarding whether and when the systems, programs, or features described herein can enable the collection of user information (e.g., images captured by the user, super-resolution images computed by the system, information about the user's social network, social actions or activities, profession, user preferences, or the user's current location), and whether the user has been sent content or communications from the server. Additionally, certain data can be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the user's identity can be processed so that it is not possible for the user to determine personally identifiable information, or the user's geographical location can be generalized (such as to the city, zip code, or state level) when location information is obtained so that the user's specific location cannot be determined. Thus, the user can have control over what information about the user is collected, how that information is used, and what information is provided to the user.

[0071] Below, several examples are described:

[0072] Example 1: A method for creating a super-resolution image of a scene, the method being performed by a user device and comprising: introducing movement into one or more components of the camera system of the user device by an optical image stabilization system; capturing a corresponding plurality of frames of an image of the scene, the corresponding plurality of frames of the image of the scene having corresponding sub-pixel offsets of the image of the scene across the plurality of frames as a result of the movement introduced into the one or more components of the camera system; performing a super-resolution calculation based on the corresponding sub-pixel offsets of the image of the scene across the plurality of frames; and creating the super-resolution image of the scene based on the super-resolution calculation.

[0073] Example 2. The method according to Example 1, wherein introducing the movement into the one or more components of the camera system is in response to entering a viewfinder mode.

[0074] Example 3: The method according to Example 1, wherein introducing the movement into the one or more components of the camera system is in response to receiving a capture command.

[0075] Example 4: The method according to any one of Examples 1-3, wherein introducing the movement into the one or more components of the camera system of the user device includes introducing an in-plane movement into the one or more components of the camera system, wherein the in-plane movement is a movement contained in a plane defined by two axes.

[0076] Example 5: The method according to any one of Examples 1-4, wherein introducing the movement into one or more components of the camera system of the user equipment includes introducing an out-of-plane movement into one or more components of the camera system, and wherein the out-of-plane movement is a movement out of the plane defined by two axes.

[0077] Example 6: The method according to any one of Examples 1-5, wherein performing the super-resolution calculation includes: calculating a Gaussian radial basis function kernel; and calculating a robustness model.

[0078] Example 7: The method according to any one of Examples 1-6, wherein performing the super-resolution calculation includes analyzing the influence of motion blur across the plurality of frames.

[0079] Example 8: The method according to any one of Examples 1-7, wherein performing the super-resolution uses a frame cycling method that uses previous low-resolution frames and current low-resolution frames from the plurality of frames to create a current super-resolution image.

[0080] Example 9: The method according to any one of Examples 1-8, wherein additional or supplementary movement is introduced into one or more components of the camera system based on a coverage score calculated by sampling a plurality of frames of an image of the scene.

[0081] Example 10: A user equipment, comprising: a camera system; an optical image stabilization system; one or more processors; a display; and a computer-readable storage medium storing instructions for an optical image stabilization system driver manager application and a super-resolution manager application, the instructions when executed by the one or more processors perform complementary functions of guiding the user equipment to: receive, by the one or more processors, a command that guides the user equipment to capture an image of a scene; introduce, based on the received command through the optical image stabilization system, movement into one or more components of the camera system during the capture of the image of the scene, wherein the introduced movement results in the capture of corresponding multiple frames of the image of the scene, the corresponding multiple frames having sub-pixel offsets of the images across the multiple frames; perform, by the one or more processors, super-resolution calculation based on the corresponding sub-pixel offsets of the image of the scene across the multiple frames; create, by the one or more processors, a super-resolution image of the scene based on the super-resolution calculation; and render, by the display, the super-resolution image of the scene.

[0082] Example 11: The user equipment according to Example 10, wherein the user equipment further includes one or more motion sensors for detecting static motion conditions.

[0083] Example 12. The user equipment according to Example 11, wherein the detection of the static motion condition causes the user equipment to introduce asynchronous movement into one or more components of the camera system through the optical image stabilization system.

[0084] Example 13: The user equipment according to Example 10, wherein the user equipment further includes one or more motion sensors for detecting dynamic motion conditions.

[0085] Example 14. The user equipment according to Example 13, wherein the detection of the dynamic motion condition causes the user equipment to introduce synchronous movement into one or more components of the camera system through the optical image stabilization system.

[0086] Example 15: A system that provides means for performing the method according to any one of Examples 1-9.

[0087] Example 16: A user equipment configured to perform the method according to any one of Examples 1-9.

[0088] Example 17: A computer-readable storage medium including instructions that, when executed, configure a processor to perform the method according to any one of Examples 1-9.

Claims

1. A method for creating a super-resolution image of a scene, the method comprising: Capturing at least one frame of an image of the scene using a camera system; Determining a coverage score of the at least one frame of the image of the scene; Based on the coverage score and by means of an optical image stabilization system, introducing movement into one or more components of the camera system; Capturing at least one additional frame of an image of the scene, the at least one additional frame of the image of the scene having corresponding sub-pixel offsets of the image of the scene across the at least one frame and the at least one additional frame, the corresponding sub-pixel offsets being the result of the introduced movement to one or more components of the camera system; Performing super-resolution calculation based on the corresponding sub-pixel offsets of the image of the scene across the at least one frame and the at least one additional frame; And Creating a super-resolution image of the scene based on the super-resolution calculation.

2. The method according to claim 1, wherein Capturing at least one frame of the image of the scene is in response to entering a viewfinder mode.

3. The method according to claim 1, wherein Capturing the at least one frame of the image of the scene is in response to receiving a capture command.

4. The method according to claim 1, wherein, Introducing the movement into the one or more components of the camera system includes: introducing in-plane movement into the one or more components of the camera system.

5. The method according to claim 1, wherein Introducing the movement into the one or more components of the camera system includes: introducing out-of-plane movement into the one or more components of the camera system.

6. The method according to claim 1, wherein, Performing the super-resolution calculation includes: Calculating a Gaussian radial basis function kernel; and Calculating a robustness model.

7. The method according to claim 1, wherein Performing the super-resolution calculation includes: analyzing the influence of motion blur across the at least one frame and the at least one additional frame.

8. The method according to claim 1, wherein Performing the super-resolution calculation uses a frame cycling method, the frame cycling method using previous low-resolution frames and current low-resolution frames from the at least one frame and the at least one additional frame to create a current super-resolution image.

9. The method according to claim 1, wherein Determining the coverage score of the at least one frame of the image of the scene includes: sampling the at least one frame of the image of the scene.

10. The method according to any one of claims 1-9 further comprises: Detecting a motion condition, wherein introducing the movement into one or more components of the camera system is further based on the motion condition.

11. The method according to claim 10, wherein, The motion condition is a static motion condition, and the method further includes: introducing asynchronous movement into one or more components of the camera system by means of the optical image stabilization system.

12. The method according to claim 10, wherein The motion condition is a dynamic motion condition, and the method further includes: introducing synchronous movement into one or more components of the camera system by means of the optical image stabilization system.

13. The method according to any one of claims 1-9, wherein, Performing the super-resolution calculation or creating the super-resolution image of the scene is performed at a remote server.

14. A user device, comprising: A camera system; An optical image stabilization system; One or more processors; A display; And A computer-readable storage medium storing instructions for an optical image stabilization system driver manager application and a super-resolution manager application, the instructions when run by the one or more processors perform supplementary functions that guide the device: Capturing at least one frame of an image of a scene; Determine a coverage score for at least one image of the scene; Based on the coverage score and via the optical image stabilization system, introduce movement into one or more components of the camera system; Capture at least one additional frame of an image of the scene, the at least one additional frame of the image of the scene having a corresponding sub-pixel offset of the image of the scene across the at least one frame and the at least one additional frame, the corresponding sub-pixel offset being a result of the introduced movement to one or more components of the camera system; Perform super-resolution calculations by the one or more processors based on the corresponding sub-pixel offset of the image of the scene across the at least one frame and the at least one additional frame; and Create a super-resolution image of the scene by the one or more processors based on the super-resolution calculations.

15. A system for creating a super-resolution image of a scene, the system providing means for performing the method according to any one of claims 1-13.

Images