Method for creating background blur in camera panning or motion

Abstract

A method of creating background blur in camera panning or motion is provided. The method comprises the steps that imaging equipment is provided, the imaging equipment comprises a main camera, and the main camera comprises an image sensor; selecting an object to be tracked in the scene; and moving the image sensor to optically align the selected object to the same given position on the image sensor as the selected object moves relative to the imaging device or relative to the scene, thereby creating a blurred image background and / or foreground relative to the selected object.

Description

[0001] This application is a divisional application of application number 202080004470.9 (PCT application number PCT / IB2020 / 055428), filed on June 09, 2020, having the title “System and method for creating background blur in camera pan or motion”.

[0002] Cross Reference to Related Applications This application claims priority to U.S. Provisional Patent Application No. 62 / 881,007, filed on July 31, 2019, the disclosure of which is incorporated by reference herein in its entirety. TECHNICAL FIELD

[0003] Embodiments disclosed herein generally relate to digital cameras, and in particular to effects during camera pan or motion. BACKGROUND

[0004] A manual camera pan involves a user opening the camera shutter during acquisition and keeping the subject or object in the same position in the frame for the duration of the exposure before closing the shutter, thereby acquiring an image that includes a blurred background and a relatively sharp subject as the photographer follows the subject in the viewfinder. The exposure time must be long enough to blur the background as the photographer follows the subject in the viewfinder due to camera movement. In the following description, “subject” and “object” can be used interchangeably.

[0005] The ability of the photographer to pan the camera blindly to track a selected object is also key to taking images of the selected object in motion. Poor technique can result in blurring of the selected object as well as the background and / or foreground, for example, due to hand movement. Thus, it can be quite difficult to take an image that describes the motion of the selected object by manual camera pan. To help take panned photographs, photographers can use tripods and monopods or motion-controlled camera robots that make it easy to swing the camera in one plane while remaining stable in other planes.

[0006] Generally, it is desirable to take images that depict the motion of a selected object, particularly in the field of sports photography. An improved method of taking images that depict the motion of a selected object using simple photography techniques is desirable that takes images that depict the motion of a selected object by simple photography techniques that produce smooth background / foreground blur and large field of view (FOV). SUMMARY

[0007] Embodiments disclosed herein teach methods for automatically producing still images or videos with a panning effect as well as other blur effects derived from camera or object motion. The methods use a single camera or multiple cameras and output background and foreground blur depending on the movement of one or more cameras or scenes.

[0008] In several exemplary embodiments, a method is provided that includes: in an imaging device including an image sensor, selecting an object to be tracked in a scene; recording an image or an image stream to provide a corresponding recorded image or recorded image stream; and aligning the selected object with the same given position on the image sensor as the selected object moves relative to the imaging device or relative to the scene, thereby creating a blurred image background and / or foreground relative to the selected object.

[0009] In one embodiment, the alignment includes optically aligning the selected object to the same given position on the image sensor.

[0010] In one embodiment, the imaging device includes a first camera having a first image sensor and a second camera having a second image sensor, wherein the recording and alignment are performed using the first camera, and wherein at least one parameter required for alignment is calculated using information provided by the second camera.

[0011] In one embodiment, the image sensor includes a first plurality of pixels that provide information for calculating at least one parameter required for the alignment, and a second plurality of pixels for the recording.

[0012] In one embodiment, the recording includes recording a plurality of frames, and wherein the alignment includes optically aligning the selected object to a similar position on the image sensor and further digitally aligning the selected object to the same given position on the image sensor.

[0013] In one embodiment, the recording includes recording a plurality of frames, and wherein the alignment includes optically aligning the selected object to a plurality of discrete pixels on the image sensor and further digitally aligning the selected object to the same given position on the image sensor.

[0014] In one embodiment, the recorded image stream includes a plurality of single images, and the method further includes generating artificial image data using single image data.

[0015] In one embodiment, the method further includes playing the recorded image stream at a frame rate (fps) different from the frame rate used to record the image stream.

[0016] In one embodiment, the recorded image stream is divided into two or more sequences, and the method further includes playing the individual sequences at different frame rates.

[0017] In one embodiment, the method further includes dividing the recorded image stream into two or more sequences displaying different degrees of blur.

[0018] In one embodiment, the alignment includes inferring a future location of the selected object.

[0019] In one embodiment, the selected object includes a celestial body.

[0020] In one embodiment, the at least one parameter required for alignment is selected from a group consisting of a shift parameter, a scaling parameter, and a rotation parameter.

[0021] In one embodiment, the recording includes recording a plurality of frames, and the alignment includes optically aligning the selected object to a similar position on the first image sensor and further digitally aligning the selected object to the same given position on the first image sensor.

[0022] In one embodiment, the recording includes recording a plurality of frames, and wherein the alignment includes optically aligning the selected object to a plurality of discrete pixels on the image sensor and further digitally aligning the selected object to the same given position on the image sensor.

[0023] In one embodiment, the calibration data between the first camera and the second camera is stored in a non-volatile memory.

[0024] In one embodiment, the sum of the first and second plurality of pixels is the total number of sensor pixels.

[0025] In one embodiment, the at least one parameter required for alignment is selected from a group consisting of a shift parameter, a scaling parameter, and a rotation parameter.

[0026] In one embodiment, the artificial image data is used to increase the uniformity of image blur.

[0027] In several exemplary embodiments, a system is provided comprising: an imaging device including an image sensor for recording an image or an image stream to provide a corresponding recorded image or recorded image stream; and a processor configured to select an object to be tracked in a scene, and to align the selected object with the same given position on the image sensor as the selected object moves relative to the imaging device or relative to the scene, thereby creating a blurred image background and / or foreground relative to the selected object.

[0028] In one embodiment, the processor configuration that aligns the selected object to the same given position on the image sensor includes a configuration that instructs a reflective component to scan a field of view including the object.

[0029] In one embodiment, the imaging device includes a first camera and a second camera.

[0030] In various embodiments, such as the above or below, a system is a smartphone. Attached Figure Description

[0031] The various embodiments disclosed herein are described with reference to the accompanying drawings listed later in this paragraph. The drawings and description are intended to explain and illustrate the various embodiments disclosed herein and should not be considered as limiting in any way. Similar elements in different drawings may be denoted by similar numbers. Elements in the drawings are not necessarily drawn to scale.

[0032] Figure 1A This demonstrates the phenomenon of multi-frame averaging using low-frequency shooting.

[0033] Figure 1B This demonstrates the phenomenon of multi-frame averaging using high-frequency shooting.

[0034] Figure 2A An example of a dynamic scene is shown, in which objects move relative to a fixed (non-moving) background.

[0035] Figure 2B An example of a dynamic scene is shown where the camera moves while capturing an object that has a fixed position relative to the background.

[0036] Figure 3 Exemplary blur effects of different types are shown in (a) to (h).

[0037] Figure 4 An exemplary background blur is shown that utilizes the movement of objects farther / closer to the camera.

[0038] Figure 5 An exemplary background is shown, illustrating the movement of an object using an optical axis perpendicular to the camera.

[0039] Figure 6 An exemplary background blur is shown using camera motion perpendicular to the camera's optical axis.

[0040] Figure 7 An exemplary embodiment of a method for capturing images that describe the sense of motion of the objects disclosed herein is shown in the flowchart.

[0041] Figure 8A The diagram schematically illustrates a telefield of view (FOV) with adjustable range. T An example of a dual-aperture digital camera.

[0042] Figure 8BA single-aperture folding digital camera with an adjustable field of view is schematically shown.

[0043] Figure 9 An embodiment of an electronic device including a multi-aperture camera having at least one (scanning telecam) is schematically illustrated. Detailed Implementation

[0044] Figure 1A An example of an average image of a dynamic scene, generated by averaging multiple single images captured at low frequency, is shown. The scene shows a background 102 including trees and a selected object 104. Low-frequency shooting means that a relatively long time has elapsed between two consecutively captured images. The term "relatively long time" refers to a time interval during which a significant amount of movement occurs in the scene. Capturing a single image at low frequency yields an average image with a visible "step" between object features. As an example, we refer to the visible step between the positions ad of the respective tops of two trees 102 in the background.

[0045] Artificial blurring, as known in the art, can be used to modify averaged images and obtain more continuous blurring of image features or regions. For example, artificial blurring can be used to obtain, for instance,... Figure 1B The degree of blurring is shown in the treetops. In other examples, a more continuous blur can be obtained by generating artificial image data. (Reference) Figure 1A Image data of the left tree visible in the image can be artificially generated using image data from the frame at the top left of the tree (first image data) and image data from the frame at the top right of the tree (second image data). For example, a motion model that moves an object (here: the tree) from a first position to a second position (here: from left to right) can be applied. Image data can be artificially generated by "freezing" one or more positions produced by the motion model. The motion model can be uniform motion along a straight line connecting the object feature (here: the top of the tree) from the first position to the second position (here: from left to right), or any other motion, such as uniformly accelerated motion or motion along any non-linear path.

[0046] In some examples, "blurry level" can be defined relatively and only relative to visual appearance. For example, when comparing two or more images that appear more or less blurred to a technician, it may refer to "stronger blur" versus "weaker blur" or "reduced blur level," etc. In some cases, "blurry level" can be defined by the number of images (or "frames") averaged to produce an image with a blurred background and / or foreground, as described herein. A higher degree of blur may correspond to a larger number of images being averaged for image generation. In yet another example, "blurry level" can be defined by the number of pixels averaged to produce the image. A higher degree of blur may correspond to a larger number of pixels being averaged for image generation. For example, a lower degree of blur can be obtained if 2 to 3 pixels are averaged, and a higher degree of blur can be obtained if 40 to 50 pixels are averaged. In yet another example, "blurry level" can be defined by a value obtained by calculating the sum of image-to-image deviations from the values ​​of all pixels and all averaged pixels. A higher degree of ambiguity corresponds to a higher summation value. Functions known in the art (e.g., the root mean square (rms) function) can be used to calculate the aforementioned deviation.

[0047] Figure 1B Another example is shown of an average image of a dynamic scene produced by averaging multiple single images captured at high frequency. High-frequency shooting means that a relatively short time has passed between two consecutively captured images. Capturing a single image at high frequency yields an average image with almost no visible gradations but smooth transitions between object features. In the example of the tree's location in background 102, we did not find such... Figure 1A Instead of the visible stepped patterns, it presents a smooth transition between different locations on the tree. This smooth transition can be idealized, as it can be perceived as a uniform blur.

[0048] Dynamic scenes in photography may involve camera movement, object movement, or both. Figure 2A An example of a dynamic scene is shown, featuring an object 200 (e.g., a boy on a skateboard) that moves relative to a fixed (non-moving) background 202. The scene is imaged (captured) by a camera 204. The camera 204 may include a lens, an image sensor, and a processing unit (processor), and is also seen in... Figure 9 . Figure 2B An example of another dynamic scene is shown where the camera 204 is moved while shooting a selected object (e.g., a child) 200' that has a fixed position relative to the background 202. Figure 2A Selected object 200 and background 202, or Figure 2BThe selected object 200' and background 202 are at different distances from the camera 204.

[0049] When shooting dynamic scenes, it is best to keep the subject in focus even with a blurred background and / or foreground. The embodiments disclosed herein provide this in many ways, some of which involve... Figure 3 As shown in the figure.

[0050] Figure 3 Exemplary blurring effects of different types produced by the methods disclosed herein are shown in (a) through (h): (a) a camera shift along the X direction, wherein the background tree 302 behind the object (human figure) 304 exhibits linear blur in the X direction while the object 304 remains sharp; (b) a small camera shift in the X and Y directions results in a Gaussian or "disk" type blur in the tree 302; (c) a large camera shift in the X and Y directions results in a "heart" type blur in the tree 302; (d) a movement of the object 304 relative to the tree 302 in the X direction results in a shift-type blur (similar to the shift in (a)). (e) Movement of object 304 relative to tree 302 in the Z direction results in a scaling-type blur; (f) Rotation of object 304 relative to tree 302 results in a rotation-type blur; (g) Deformation motion of object 304 relative to tree 302 (e.g., waving) results in a non-rigid object shape change / transformation or a non-rigid object / pose change type blur; and (h) Camera shifts in the X direction, wherein while object 304 remains sharp, the background tree 302 and foreground tree 306 behind object (human figure) 304 exhibit linear blur in the X direction.

[0051] In some embodiments, the methods described herein can be used not only for imaging scenes commonly referred to as dynamic scenes, but also for other purposes, such as imaging scenes in low-light conditions. "Low light" can be defined as a scene with an average brightness of less than 50 to 70 lux, for example, 20 lux or 5 lux. More generally, the methods described herein can be used to photograph scenes with long effective exposure times. A long effective exposure time can be defined as the product of the degree of dynamism or motion in the scene and the actual exposure time. A long effective exposure time can be obtained, for example, by moving an object rapidly at an angular velocity ω1 during an exposure time T1 and by photographing an object moving more slowly at an angular velocity ω2 = 1 / 2∙ω1 with a longer exposure time T2 = 2T1. For example, an object moving at a slow angular velocity ω can be photographed with a long exposure time T, such as a star visible in the night sky.

[0052] In some embodiments, a long effective exposure time is achieved if one or more objects in a scene or the entire scene move significantly during exposure. In one example, significant movement can be defined as a movement during exposure that causes smearing of a particular object point across, for example, 1 to 3 pixels on an image sensor. In another example, significant movement can be defined as a movement during exposure that causes smearing of a particular object point across, for example, 10 or more pixels. In yet another example, significant movement can be defined as a movement during exposure that causes smearing of a particular object point across, for example, 20 to 250 or more pixels.

[0053] According to the embodiments described below, background and foreground blurring can be intentionally and artificially created through the following four plots, see reference. Figures 4-6 illustrate. In the first scene, the selected object is moved closer to / away from the camera, see, for example... Figure 4 As an object moves closer to the camera, its size appears larger on the camera's image sensor; as an object moves further away from the camera, its size appears smaller on the camera's image sensor. This represents a change in scale. If the change in the object's scale is not equal to the change in scale of the rest of the scene, objects that are not moving in this way will appear blurry. In this scenario, when recording begins, mechanical zoom is used when the image is captured, and the object is scaled (also called "aligned") to its size on the sensor.

[0054] More specifically, frame (a) shows an image with an object (human figure) 400 far from the camera and a background including two trees 402a and 402b behind the object 400. Object 400 is the selected object. Frame (a) corresponds to the initial scene on the sensor at the start of image recording. Frame (b) shows an image of object 400' (representing object 400 at a different position and time at this point) closer to the camera without using alignment. Figure 4As shown, in Figures 5 and 6 below, labels 500 and 600 denote the initially selected objects, while labels 500' and 600' denote the same corresponding objects at different locations and times. Frame (c) shows the scene (b) optically and / or digitally aligned (scaled) to the size on the sensor at the start of recording, see step 708 below, such that the objects are the same size and position as in frame (a). Note that, compared to frame (a), the movement reduces the size of trees 402a' and 402b' relative to the size of object 400'. Frame (c) corresponds to the final scene on the sensor at the end of the recorded image. Frame (d) shows an overlay image of object 400 and trees 402a and 402b from frames (a) and (c) of the start step 704 (frame a) and end step 710 (frame c). Frame (e) shows the final shooting result, which includes the light captured on the sensor during recording (steps 704–708). This includes the first scene (a), the last scene (c), and all scenes in between.

[0055] In some embodiments, and optionally, alignment can be performed digitally after shooting. In other embodiments, such as in the case of multiple frames (see step 708 below), and optionally, partial alignment can be performed optically and / or digitally during shooting, and partial alignment can be performed digitally after shooting. For example, alignment may include optically aligning a selected object to a plurality of discrete pixels on an image sensor, and further digitally aligning the selected object to the same given location on the image sensor. Optically aligning the selected object to a plurality of discrete pixels may refer to optical alignment that does not include subpixel alignment.

[0056] In the second scene, the camera is stationary, and the selected object moves (displaces) perpendicular to the optical axis of the camera lens, see, for example... Figure 5 If the displacement of the selected object is not equal to the displacement of the rest of the scene on the camera sensor, the object that is not moved in this manner will be blurred. In this scenario, the selected object is aligned to its position on the sensor when recording begins. The selected object is aligned (displaced) when the image is captured using mechanical displacement. This mechanical displacement can be achieved, for example, by moving a reflective component or an optical path folding element (OPFE) (e.g., a prism), or by moving the camera lens, or by moving the lateral or tilt position of the camera module.

[0057] Figure 5The following describes the background blurring result obtained according to an embodiment of the method described herein for moving a selected object perpendicular to the optical axis of a camera lens. More specifically, frame (a) shows an image with a selected object (human figure) 500 and a background including two trees 502a and 502b behind the object 500. Frame (a) corresponds to the initial scene on the sensor when image recording begins. Frame (b) shows an image with an object 500' representing the object 500 moving to the right relative to the camera, without the use of alignment. Frame (c) shows scene (b) optically and / or digitally aligned (shifted) to align object 500' by shifting the field of view (FOV), see step 708 below, such that object 500' is in the same position as object 500 in frame (a). Note that the shift moves the trees 502a' and 502b' on the sensor to the left. Frame (c) corresponds to the final scene on the sensor at the end of image recording. Frame (d) shows an overlapping image of object 500 and trees 502a and 502b from frames (a) and (c) of the start step 704 (frame a) and end step 710 (frame c) of the shooting. Frame (e) shows the final shooting result, including the light captured on the sensor during recording (steps 704–708). This includes the first scene (a), the last scene (c), and all scenes in between.

[0058] In the third scene, the camera moves perpendicular to the optical axis while the object remains stationary relative to the background / foreground. This alters the distance between the object and the camera, for example, see [link to example]. Figure 6 Due to the difference in the baselines between the camera positions throughout the movement, all objects in the scene at a distance from the camera that differs from the selected object will be blurred. If the distance between the selected object and the camera is not equal to the distances of other objects or the rest of the scene relative to the camera sensor, their displacement on the sensor will differ. Objects that do not move in the same way will become blurred. In this scenario, mechanical displacement as described above is used to align (shift) the selected object when capturing the image.

[0059] Figure 6The background blurring result obtained according to an embodiment of the method described herein for cases where the camera moves perpendicular to the camera's optical axis is described. More specifically, frame (a) shows an image with an object (human figure) 600 and a background including two trees 602a and 602b behind the object 600. Object 600' represents the displaced object 600. Frame (a) corresponds to the initial scene on the sensor at the start of image recording. Frame (b) shows an image shifted to the right relative to the position of frame (a) due to camera movement without the use of alignment. Frame (c) shows scene (b) optically and / or digitally aligned (shifted) by moving the field of view to align with object 600' so that the selected object is located in the same position as in frame (a). Note that the shift moves the trees 602a' and 602b' on the sensor to the left. Frame (c) corresponds to the final scene on the sensor at the end of image recording. Frame (d) shows an overlay image of object 600 and trees 602a and 602b from frames (a) and (c) of the start step 704 (frame a) and end step 710 (frame c) of the shooting. Frame (e) shows the final shooting result, which includes the light captured on the sensor during recording (steps 704–708). This includes the first scene (a), the last scene (c), and all scenes in between.

[0060] In the fourth episode, any combination of the first, second, and third episodes mentioned above can be used.

[0061] Figure 7 The flowchart illustrates an exemplary embodiment of a method for acquiring images describing the sense of motion of the objects disclosed herein. The process begins by observing a scene including motion caused by the movement of an object or a camera. In step 702, an object to be tracked is selected, and in step 704, recording of an image or video stream begins, during which the object is tracked by the moving object or the moving camera. In step 706, shift, scaling, and rotation calculations are performed on the object seen in the image or video stream. As indicated by the arrows from 708 to 706, the tracking, scaling, shift, and rotation modifications are recalculated in each frame (or each X-set frame). In step 708, in some embodiments, the object is optically and / or digitally aligned to ensure it remains in the same position on the camera's image sensor. In some embodiments involving multiple frames, the remaining alignment of the object may be performed digitally in step 712 after recording has ended, or by a combination of mechanical and digital alignment. In some embodiments used to generate video comprising an image stream, object alignment may be performed such that the object appears in the same position within the field of view of all images in the stream.

[0062] In other embodiments for generating video including an image stream, object alignment is performed such that the object appears to move at an angular velocity ω, which approximately corresponds to the angular velocity at which the object would move within the scene if optical alignment had not been performed in step 708. In some embodiments, the resulting image stream can be played at the speed used to capture the video (defined by the frame rate (fps) used for the video). In other embodiments, the resulting image stream can be played at a higher fps to achieve an artistic time-lapse effect, or at a lower fps to achieve an artistic slow-motion effect. The fps may not be constant and can be modified during video generation. This can be beneficial for emphasizing specific segments within the field of view or for highlighting specific events that may have occurred during video capture.

[0063] In one example, a specific event in a video can be emphasized using the following or a similar sequence: - Play the first sequence of the video at a constant frame rate (fps), for example, using the fps of a recorded video. -Play the second sequence of the video at a gradually decreasing FPS until the event to be emphasized occurs. - A third sequence of videos played at a constant frame rate after the highlighted event. - Play the fourth sequence of video at gradually increasing FPS after the event occurs, until it reaches the FPS of the first sequence. - The fifth sequence of the video is played at a constant frame rate (fps) of the first sequence.

[0064] In step 710, the recording of the image or video stream is completed, resulting in an image with a blurred background.

[0065] In some examples, the object might be a moving object. This implies that, in order to produce the resulting image, it is not possible to use the entire image data present in each frame, but rather to use only the image data from image fragments present in each image of the image sequence.

[0066] exist Figure 7 More details of some of the steps performed are described below.

[0067] Object / subject selection step 702 The selection of the object or subject to be tracked can be accomplished in several ways. The selected object can be identified using a rectangular region of interest (ROI) or by a masked region within the image. The identified markers can be selected automatically, by the user, or by a combination of user selection and digital refinement, all of which are known in the art. Tracking methods with improved robustness, known in the art, can be used. Tracking methods can rely not only on single object features but also on multiple object features.

[0068] Recording of image or video stream, steps 704-710 Recording can be performed using a single long exposure or by taking a sequence of multiple short exposure images and averaging them, as follows: a) Single long exposure using a long shutter speed. Aperture and digital / analog gain are automatically adjusted to achieve the same level of brightness.

[0069] b) Use short exposures and average multiple images. Since noise is reduced when averaging multiple images, the exposure time can be very short. For video mode, the output can be the average of the last frame, for example, the average of the last 10 to 30 frames.

[0070] c) Using a single camera with different sensor pixel capabilities (e.g., the folding camera 804, also...) Figure 8A As shown in the diagram, some sensor pixels undergo multiple short exposures, while other sensor pixels undergo long exposures. For example, by using a quad sensor (see, for example, Japanese Patent Application No. 2019041178), a quad-Bayer pixel structure enables two exposures within a group of four pixels. In this case, the pixel selection on the image sensor is used for the long exposure, and the remaining pixels are used for the short exposure. The final image is a combination of image data from the long-exposure pixels and the short-exposure pixels.

[0071] In some embodiments, a single image may be generated and output to be displayed to a user. In other embodiments, an image stream (i.e., video) may be generated for output. In the example of generating a video stream, a specific degree of blur may be required for a particular sequence of the video. This can be achieved by adjusting the number of frames used for averaging (or averaging). In one example, for a first sequence with a first degree of blur, 10 to 30 frames may be averaged, while for a second sequence with a second degree of blur, 30 to 60 frames may be averaged. The resulting second sequence of video has a higher degree of blur than the first sequence. In another instance, 5 to 10 frames may be averaged to obtain a third sequence exhibiting a weaker degree of blur than the first and second sequences. The user or program may define the degree of blur in "post-capture" (i.e., any time after capturing the image stream). In yet another example, to emphasize a specific event in a scene, it may be desirable to gradually increase the degree of blur until the event occurs, maintain a constant degree of blur during the event, and then gradually decrease the degree of blur. Since blur depends on averaging image data appearing in different frames, the degree of blur cannot be modified continuously but only in discrete steps. In some examples, discrete blurring is achieved by adding one or more frames to an averaged sequence or set of frames, or by subtracting one or more frames from an averaged sequence or set of frames. To achieve continuous blur modification, artificial blurring known in the art can be superimposed on blurring achieved through image averaging. Artificial blurring known in the art can also be used to continuously modify the degree of blur within a single image as described herein.

[0072] Calculation of shift, scale and rotation step 706 In one exemplary embodiment, the object may be tracked, for example, using a known camera described in commonly owned international patent applications PCT / IB2016 / 052179, PCT / IB2016 / 055308, PCT / IB2016 / 057366 and PCT / IB2019 / 053315, in one of a number of ways outlined below, and displacement, scaling and rotation may be calculated based on the tracking results.

[0073] 1. Using a single camera with different sensor pixel capabilities (e.g., the folding camera 804, also...) Figure 8A As shown in the diagram, some sensor pixels undergo short, multiple exposures, while others undergo long exposures. Information from some of the pixels (e.g., 1 / 16 of the total number of pixels) on sensor 806 is used to perform object shifting, scaling, and rotation calculations. Mechanical alignment is performed on camera 804. Image recording steps 704-710 are performed on the remaining pixels of sensor 806 (i.e., 15 / 16 of the total number of pixels not used in the object shifting, scaling, and rotation calculations).

[0074] 2. Alternatively, use a single camera, in situations such as Figure 8B Object displacement calculations and image recording are performed on the same camera sensor shown (e.g., sensor 806). The object is tracked on the video stream, and displacement, scaling, and rotation are calculated. Mechanical alignment is performed simultaneously with tracking (step 708).

[0075] 3. Using, for example, those described in PCT / IB2016 / 057366 and Figure 8B and Figure 9 The dual-camera setup shown comprises a first upright camera 800 with sensor 802 and a second folded camera 804 with sensor 806. Camera 800 can be infrared (IR), visible light, structured light, or any other type of light and is used for object tracking. Mechanical alignment and image recording are performed on camera 804. Object position (movement, scaling, and rotation) is calculated using information from sensor 802 while tracking the object. To estimate the required alignment for camera 804 (for step 708), the displacement, scaling, and rotation between the sensors are calculated using feature matching and prior calibration between the cameras. (See also: Camera 804) Figure 8A ) and dual cameras including camera 800 and camera 804 (see Figure 8B It can be included in the host device (e.g., electronic device 900).

[0076] In some embodiments, the future position of a moving object can be inferred (estimated). This inference can be based on parameters detected in the captured frame (e.g., past displacement, scaling, and rotation). These provide estimates of future displacement, scaling, and rotation. The estimation can be performed, for example, by linear extrapolation of past displacement, scaling, and rotation. In another embodiment, nonlinear extrapolation can be used. The inference can also be based on machine learning or other techniques known in the art. This inference can be beneficial for meaningful mechanical alignment in step 708. For example, an angular velocity ω within the scene's field of view (FOV) can be considered. Object The moving object, whose field of view may be smaller, but still moves at a maximum angular velocity ω that allows for field-of-view scanning (which may be necessary for mechanical alignment). ScanThe sequential movement, for example, is performed by rotating an optical path folding assembly to perform the field-of-view scanning movement. In this example, the inference may be crucial for the mechanical movement that actually aligns the object with a specific location on the image sensor. In some embodiments, extrapolation can be performed on a timescale required for, for example, capturing 2 to 4 frames. In other embodiments, such as those requiring larger mechanical alignment strokes, extrapolation can be performed on a timescale required for, for example, capturing 4 to 10 or more frames.

[0077] In some embodiments, digital alignment may be performed to compensate for discrepancies between the actual movement captured and the movement extrapolated.

[0078] Other embodiments may use other single, dual, or multi-aperture cameras for object tracking, image recording, or both.

[0079] Mechanical alignment step 708 In an exemplary embodiment, alignment, displacement, rotation, and zoom of an object on the sensor can be performed optically and / or digitally (the latter using a digital video stream). Mechanical alignment of the following types can be used: For example, in Figure 8B The prism movement in the dual-camera system, as shown and described in PCT / IB2019 / 053315, is used to match object displacement. An optical path folding assembly (e.g., a prism) 808 can fold the optical path about two axes (each axis being a degree of freedom, DOF). These two degrees of freedom are a yaw rotation 810 about a yaw rotation axis 812 parallel to the first optical path 814 (X-axis) and a pitch rotation 816 about a pitch rotation axis 818 parallel to the Y-axis. The field-of-view scanning via prism movement is not instantaneous but requires a settling time. The field-of-view scanning via prism movement can scan, for example, 2° to 5° on a timescale of approximately 1 to 30 milliseconds, and 10° to 25° on a timescale of approximately 15 to 80 milliseconds.

[0080] 1. Lens movement or camera module movement to modify the camera's lateral position or tilt angle to match object displacement, such as the displacement described in PCT / IB2016 / 052179. For example, movement of the lens module can be performed using an actuator in the X direction 822, the movement of the lens module corresponding to object displacement in the Z and Y directions 820. Tilting movement can be converted into linear displacement along the optical axis of an optical element coupled to the actuator. Two actuators can be combined into a component capable of providing biaxial tilt, such as the commonly owned PCT / IB2019 / 053315.

[0081] 2. Mechanical zoom adjustment to match subject proportions. Some advanced camera designs may include different lens groups that can move relative to each other, thus changing the camera's effective focal length to produce the ability to optical zoom. Mechanical zoom can be used to optically align the camera with the subject's proportions.

[0082] 3. Mechanical alignment to match object rotation. For example, the image scrolling motion created by tilting the optical path folding assembly compensates for object rotation. Compensation for the tangential rotation of the object (on the scrolling axis) can be achieved by rotating the combined prisms about two axes (Y-818 and X-812) and by shifting the prism in the Y direction. This mechanism is described in detail in the applicant's international patent application PCT / IB2016 / 055308.

[0083] Note that other configurations may use alternative external or internal mechanical alignment components that utilize motors or motion control.

[0084] Some embodiments of the method can be configured to photograph scenes with very low light, such as the night sky. Embodiments for night sky photography can resemble a so-called "star tracker" tripod, i.e., a tripod that follows the movement of celestial bodies to photograph the sky with long exposure times ("astrophotography"). Astrophotography is typically performed under light conditions of approximately 0.1 to 0.0001 lux. For astrophotography, it may be advantageous to place the camera main unit on a stationary object (relative to the Earth) or to fix the camera main unit to a stationary object. The Earth moves relative to celestial bodies, so for a photographer on Earth, a celestial body can move at an angular velocity ω of approximately ω≈4∙10⁻³ degrees / second, which angular velocity ω originates from the Earth's rotation of 360 degrees in approximately 24 hours (translated to approximately 15 degrees per hour or 1 / 4 degree per minute). It is well known that the actual angular velocity depends on the photographer's specific location or coordinates on Earth. In the optical alignment step 708, movement can be achieved using an optical path folding assembly, lens, sensor, or camera that mimics the movement of celestial bodies. In one example, the optical alignment in step 708 may depend solely on the user's location and the camera's orientation on Earth; that is, no further information (e.g., image information from the camera) is required. The user's location and the camera's orientation can be obtained directly from a known camera host device, or indirectly, for example, through an external device (e.g., WiFi) that provides access to a device with a known location.

[0085] Generally, photographers use the "rule of 600" (sometimes also the "rule of 500"). The rule of 600 provides the maximum possible exposure time T for a given camera setting that can be used for astrophotography. Max A rough estimate. That is, in T MaxTime-based exposure of the sensor is expected to provide maximum signal capture before star trails are formed in the image. The 600 rule specifies that T... Max (In seconds) is given by the following formula:

[0086] Where CF is the crop factor of the camera sensor, and EFL is the effective focal length (in millimeters) known in the art. Typically, the effective focal length of cameras included in mobile devices ranges from EFL = 2.5 mm (for ultra-wide cameras) to EFL = 25 mm (for telephoto cameras). Typically, the crop factor of cameras included in mobile devices ranges from CF = 2.5 (for large 1 / 1” sensors) to CF = 10 (for 1 / 4” sensors. Taking extreme cases (a) CF = 2.5 and EFL = 2.5 mm (large sensor and large field of view) and (b) CF = 10 and EFL = 25 mm (small sensor and narrow field of view) as examples, T is obtained in the context of mobile astrophotography. Max The value is T 1 Max ≈100 seconds and T 2 Max ≈2.4 seconds. The second example (CF=10, EFL=25mm) could, for example, correspond to a folding telephoto camera with a high zoom factor. Utilizing the method described herein and applied in astrophotography for compensating for celestial motion by optically aligning the celestial body on the image sensor, significantly longer exposure times can be used while still obtaining a clear image of the night sky without star trails. A significantly longer exposure time could mean 1.5 to 40 times longer than the time estimated using the 600 rule. This can be particularly applicable to telephoto cameras, as can be clearly seen from the second example. In other examples, in step 712, the celestial body can be digitally aligned instead of optically. In still other examples, the celestial body can be aligned using a combination of optical and digital alignment.

[0087] Figure 9An embodiment of an electronic device, designated 900, is schematically illustrated. This electronic device includes a multi-aperture camera with at least one scanning remote camera. The electronic device 900 may be, for example, a smartphone, tablet, or laptop computer. The electronic device 900 includes a first scanning remote camera module 910 with an optical path folding assembly 912 for field-of-view scanning and a first lens module 918 forming a first image recorded by a first image sensor 916. A first lens actuator 924 can move the lens module 918 for focusing and / or optical image stabilization (OIS). In some embodiments, the electronic device 900 may further include an application processor (AP) 940, which includes an object aligner 942 and an image / video generator 944. In some embodiments, first calibration data may be stored in a first memory 922 of the camera module, such as an electrically erasable programmable read-only memory (EEPROM). In other embodiments, the first calibration data may be stored in a third memory 950 of the electronic device 900, such as non-volatile memory (NVM). The first calibration data may include calibration data between the sensor of the wide-angle camera module 930 and the sensor of the remote camera module 910. In some embodiments, the second calibration data may be stored in a second memory 938. In other embodiments, the second calibration data may be stored in a third memory 950. In other embodiments, the application processor 940 may receive calibration data stored in a first memory located on camera module 910 and a second memory located on camera module 930, respectively. The second calibration data may include calibration data between the sensor of the wide-angle camera module 930 and the sensor of the remote camera module 910. The electronic device 900 also includes a wide-angle (or ultra-wide-angle) camera module 930 having a field of view larger than that of camera module 910, the wide-angle (or ultra-wide-angle) camera module 930 including a second lens module 932 forming an image recorded by a second image sensor 934. A second lens actuator 936 is movable for focusing and / or optical image stabilization of the lens module 932.

[0088] In use, a processing unit (e.g., application processor 940) may receive first and second image data from camera modules 910 and 930, respectively, and provide camera control signals to camera modules 910 and 930. The camera control signals may include control signals to an optical path folding component actuator 914, which may rotate the optical path folding component 912 in response to the control signals to perform a field-of-view scan. The field-of-view scan may be used in step 708 to optically (or mechanically) align a selected object to a given position on the image sensor. In some embodiments, the optical path folding component actuator 914 may drive the optical path folding component 912 for optical image stabilization. In some embodiments, and for example, to perform step 706, application processor 940 may receive second image data from camera module 930. Object aligner 942 may be a processor configured to use the second image data to track a selected object and calculate control signals sent to remote camera 910 to optically align the selected object to a given position on the image sensor. In other embodiments, object aligner 942 may use the first image data to track the selected object. In other embodiments, object aligner 942 may be configured to digitally align the object to a given position on the image sensor in step 712. In some embodiments, object aligner 942 may be configured to use first image data and / or second image data to infer (estimate) the future position of a moving object. Image or video generator 944 may be configured to generate, respectively, images such as... Figure 7 The images and image streams described herein. In some embodiments, the image / video generator 944 may be configured to average first image data from a plurality of single images. In some embodiments, the image / video generator 944 may be configured to generate artificial images. In some embodiments, the image / video generator 944 may be configured to generate images including artificial blur.

[0089] Unless otherwise stated, the word "and / or" is used between the last two items in the list of options to indicate that a choice of one or more of the listed options is appropriate and selectable.

[0090] It should be understood that when a claim or specification refers to a component “a” or “an”, such reference should not be construed as meaning that there is only one such component.

[0091] It should be understood that, for clarity, certain features of the invention described in the context of a single embodiment or example may also be provided in combination in a single embodiment. Conversely, for brevity, various features of the invention described in the context of a single embodiment may also be provided individually or in any suitable sub-combination or appropriately in any other described embodiment. Certain features described in the context of various embodiments should not be considered essential features of those embodiments unless the embodiments cannot be practiced without these components.

[0092] While this disclosure describes a limited number of embodiments, it should be understood that many variations, modifications, and other applications can be made to these embodiments. Generally, this disclosure should be understood as not being limited to the specific embodiments described herein, but only to the scope of the appended claims.

[0093] All references mentioned in this specification are incorporated herein by reference in their entirety, to the same extent that each individual reference is explicitly and individually indicated to be incorporated herein by reference. Furthermore, any citation or identification of any reference in this application should not be construed as an admission that such citation is prior art that can be used in this application.

Claims

1. A method characterized by, comprising: providing an imaging device comprising a main camera comprising an image sensor; selecting an object in a scene to be tracked; and moving the image sensor to optically align the selected object to the same given position on the image sensor as the selected object moves relative to the imaging device or relative to the scene, thereby creating a blurred image background and / or foreground relative to the selected object.

2. The method of claim 1, wherein, the imaging device further comprises a second camera, wherein the aligning is performed by the main camera, and wherein at least one parameter required for the aligning is calculated using information provided by the second camera.

3. The method of claim 1, wherein, the image sensor comprises a first plurality of pixels providing information used to calculate at least one parameter required for the aligning, and a second plurality of pixels used to record images or image streams.

4. The method of claim 1, wherein, the aligning comprises extrapolating future positions of the selected object.

5. The method of claim 1, wherein, selecting an object in a scene to be tracked is performed using image data of the main camera.

6. The method of claim 2, wherein, selecting an object in a scene to be tracked is performed using image data of the second camera.

7. The method of claim 2, wherein, the aligning comprises extrapolating future positions of the selected object.

8. The method of claim 2, wherein, the at least one parameter required for the aligning is selected from the group consisting of a shift parameter, a scale parameter, and a rotation parameter.

9. The method of claim 3, wherein, the recording comprises recording a plurality of frames, and the aligning comprises optically aligning the selected object to similar positions on the image sensor, and further digitally aligning the selected object to the same given position on the image sensor.

10. The method of claim 3, wherein, the recording comprises recording a plurality of frames, and the aligning comprises optically aligning the selected object to a plurality of discrete pixels on the image sensor, and further digitally aligning the selected object to the same given position on the image sensor.

11. The method as described in claim 3, characterized in that, the recorded image stream comprises a plurality of single images, and the method further comprises generating artificial image data using single image data.

12. The method of claim 3, wherein, it is further included to play the recorded image stream at a different frame rate than the frame rate used to record the image stream.

13. The method of claim 3, wherein, the recorded image stream is divided into two or more sequences, and the method further comprises playing each sequence at a different frame rate.

14. The method of claim 3, wherein, it is further included to divide the recorded image stream into two or more sequences displaying different degrees of blurring.

15. The method of claim 11, wherein, the artificial image data is used to increase uniformity of image blurring.

16. The method of claim 1, wherein, the selected object is a celestial body.

17. The method of claim 1, wherein, the imaging device is a smartphone.

18. The method of claim 1, wherein, the imaging device is a tablet computer.

19. The method as described in claim 1, characterized in that, the main camera has an effective focal length range of 2.5mm to 25mm.

Images