System and method for providing sliding zoom view synthesis

Abstract

A method and apparatus for providing a sliding zoom effect by an electronic device are provided. A first image having a first depth map and a second image having a second depth map are obtained. A first composite image and a corresponding first composite depth map are generated using the first image and the first depth map, respectively. A second composite image and a corresponding second composite depth map are generated using the second image and the second depth map, respectively. A fused image is generated from the first composite image and the second composite image. A fused depth map is generated from the first composite depth map and the second composite depth map. A final composite image is generated based on processing the fused image and the fused depth map.

Description

[0001] priority

[0002] This application is based on and claims priority to a provisional patent application filed in the U.S. Patent and Trademark Office on October 28, 2019 and assigned serial number 62 / 926,858 and a patent application filed in the U.S. Patent and Trademark Office on March 10, 2020 and assigned serial number 16 / 814,184, the contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to a digital image processing system, and more particularly, to a method and system for providing dolly zoom view synthesis through an electronic device. Background Art

[0004] The sliding zoom effect is usually achieved by zooming the lens in or out so as to adjust the field of view (FoV) while simultaneously moving the camera away from or towards the subject so as to always keep the subject relatively the same size. This results in a continuous perspective distortion, the most immediately noticeable feature being that the background appears to change size relative to the subject. The visual appearance to the viewer is that the background increases in size and detail to simulate the overwhelming presence of the foreground, or the foreground begins to dominate, depending on how the effect is performed.

[0005] The sliding zoom effect may also be referred to as a vertigo effect, focus interference zoom, zido, zolly, Hunter Smith shot, Hitchcock shot, Hitchcock zoom, vertiginous zoom, jaw shot, reverse tracking shot, triple reverse zoom, back zoom march, smash zoom, telescoping, trombone shot, push / pull, long pull, reverse pull, trombone effect, stretch shot, forward zoom, reverse tracking, zoom in / out, trans-trav, and reverse zoom.

[0006] Figure 1A and Figure 1B is a diagram showing the effect of sliding zoom. Figure 1A In the case of a camera with FoVθ 1 At "position A" of , and object 102 is in focus. Figure 1B In the example above, the camera moves to "position B" and increases its FoV to θ 2 , where position B is closer to object 102 than position A. After this camera movement, the size of object 102 remains the same, while background objects 104 and 106 appear to shrink in size as more of the object is brought into the image due to the increased FoV.

[0007] Performing a slide zoom effect requires skill and equipment because it requires simultaneous zooming and movement. Performing a slide zoom effect on a mobile phone camera is particularly difficult because it requires fine control over image zoom, object tracking, and movement. Summary of the invention

[0008] According to one embodiment, a method for providing a sliding zoom effect by an electronic device is provided. A first image having a corresponding first depth map and a second image having a corresponding second depth map are obtained. The first image and the second image have different fields of view. A first composite image and a corresponding first composite depth map are generated using the first image and the first depth map, respectively. A second composite image and a corresponding second composite depth map are generated using the second image and the second depth map, respectively. A fused image is generated from the first composite image and the second composite image. A fused depth map is generated from the first composite depth map and the second composite depth map. A final composite image is generated based on processing the fused image and the fused depth map.

[0009] According to one embodiment, an electronic device is provided. The electronic device includes at least one camera and a processor on which instructions are executed, so that the processor obtains a first image having a corresponding first depth map and a second image having a corresponding second depth map. The first image and the second image have different fields of view. The instructions also cause the processor to: use the first image and the first depth map to generate a first composite image and a corresponding first composite depth map, respectively, and use the second image and the second depth map to generate a second composite image and a corresponding second composite depth map, respectively. The instructions also cause the processor to: generate a fused image from the first composite image and the second composite image, generate a fused depth map from the first composite depth map and the second composite depth map, and generate a final composite image based on processing the fused image and the fused depth map.

[0010] According to one embodiment, a method for performing depth-aware inpainting by an electronic device. A fused image is generated from a first image and a second image. A fused depth map is generated from the first depth map and the second depth map. Depth hole filling is performed using the fused depth map to obtain a hole-filled depth map. An occlusion mask is updated based on each of a plurality of unique depth values ​​in the hole-filled depth map. A windowed average filter is applied to occluded regions in the fused image based on the updated occlusion mask to obtain a final image. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other aspects, features and advantages of certain embodiments of the present disclosure will be more clearly understood through the following detailed description in conjunction with the accompanying drawings, in which:

[0012] Figure 1A and Figure 1B is a diagram showing a sliding zoom effect;

[0013] Figure 2 is a diagram illustrating camera translation according to an embodiment of the present disclosure;

[0014] Figure 3 is a diagram showing camera geometry under sliding zoom according to an embodiment of the present disclosure;

[0015] Figure 4A and Figure 4B is a diagram illustrating single-camera image synthesis under sliding zoom with forward distortion according to an embodiment of the present disclosure;

[0016] Figure 5A and Figure 5B is a diagram showing single camera image synthesis under sliding zoom showing epipolar lines according to an embodiment of the present disclosure;

[0017] Figure 6 is a diagram showing a dual camera system according to an embodiment of the present disclosure;

[0018] Fig. 7A and Figure 7B is a diagram showing image synthesis from a second camera under sliding zoom according to an embodiment of the present disclosure;

[0019] Fig. 8A and Figure 8B is a diagram showing image synthesis from a second camera under sliding zoom with epipolar lines according to an embodiment of the present disclosure;

[0020] Figure 9A-9F is a diagram illustrating image fusion according to an embodiment of the present disclosure;

[0021] Fig.10 is a flow chart illustrating a method for nearest neighbor hole filling according to an embodiment of the present disclosure;

[0022] Fig.11 is a flow chart illustrating a method for depth-aware inpainting according to an embodiment of the present disclosure;

[0023] Figures 12A-12D is a diagram showing a hole filling process according to a specific step of an embodiment of the present disclosure;

[0024] Figure 13A-13G is a diagram showing the result of a hole filling process according to an embodiment of the present disclosure;

[0025] Fig.14 is a diagram showing a thin-lens camera model and a circle of astigmatism according to an embodiment of the present disclosure;

[0026] Fig.15 is a flow chart illustrating a method for providing sliding zoom view composition for a setup with one or more cameras according to an embodiment of the present disclosure;

[0027] Fig.16 is a diagram showing the corresponding Fig.15 A flowchart of a sliding zoom view synthesis pipeline with a single camera setup and depth map estimation;

[0028] Figures 17A-17D The method for using the Fig.16 An example of a single camera setup where the composition pipeline performs sliding zoom view composition;

[0029] Fig.18 is a diagram showing the corresponding Fig.15 A flowchart of a sliding zoom view synthesis pipeline with a dual-camera setup and depth map estimation;

[0030] Figures 19A-19D The method according to the embodiment of the present disclosure is shown Fig.18 An example of a dual-camera setup where the synthesis pipeline performs sliding zoom view synthesis;

[0031] Fig. 20 is a flow chart illustrating a method for providing sliding zoom view synthesis for a single camera setup with a depth map input according to an embodiment of the present disclosure;

[0032] Fig.21 The present invention is based on the embodiment of the present invention. Fig. 20 A flowchart of a sliding zoom view synthesis pipeline for a single camera setup;

[0033] Fig. 22 is a flow chart illustrating a method for providing sliding zoom view synthesis for a dual camera setup with depth map input according to an embodiment of the present disclosure;

[0034] Fig.23 The present invention is based on the embodiment of the present invention. Fig. 22 A flowchart of a sliding zoom view synthesis pipeline for a dual camera setup; and

[0035] Fig.24 is a block diagram illustrating electronic devices in a network environment according to an embodiment. DETAILED DESCRIPTION

[0036] Hereinafter, embodiments of the present disclosure are described in detail with reference to the accompanying drawings. It should be noted that although the same elements are shown in different drawings, they will be represented by the same figure numerals. In the following description, specific details such as detailed configuration and components are provided only to help fully understand the embodiments of the present disclosure. Therefore, it is obvious to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. In addition, for the sake of clarity and brevity, descriptions of well-known functions and configurations are omitted. The terms described below are terms defined in consideration of functions in the present disclosure, and may be different according to the user, the user's intention or habit. Therefore, the definition of the term should be determined based on the content throughout the entire specification.

[0037] The present disclosure may have various variations and various embodiments, wherein each embodiment is described in detail below with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to the embodiments, but includes all modifications, equivalents and substitutes within the scope of the present disclosure.

[0038] Although various elements can be described using terms including ordinal numbers such as first, second, etc., structural elements are not limited by terms. Terms are only used to distinguish one element from another element. For example, without departing from the scope of the present disclosure, a first structural element may be referred to as a second structural element. Similarly, a second structural element may also be referred to as a first structural element. As used herein, the term "and / or" includes any one and all combinations of one or more associated items.

[0039] The terms used herein are only used to describe various embodiments of the present disclosure and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In the present disclosure, it should be understood that the term "including" or "having" means the presence of features, numbers, steps, operations, structural elements, parts or combinations thereof, and does not exclude the presence or possibility of one or more other features, numbers, steps, operations, structural elements, parts or combinations thereof.

[0040] Unless otherwise defined, all terms used herein have the same meaning as those understood by those skilled in the art to which the present disclosure belongs. Terms such as those defined in commonly used dictionaries are interpreted as having meanings equivalent to the contextual meanings in the relevant technical field, and are not interpreted as having ideal or overly formal meanings unless clearly defined in this description.

[0041] The electronic device according to one embodiment may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer, a portable multimedia device, a portable medical device, a camera, a wearable device, and / or a household appliance, etc. According to one embodiment of the present disclosure, the electronic device is not limited to the above-mentioned devices.

[0042] The terms used in the present disclosure are not intended to limit the present disclosure, but are intended to include various changes, equivalents or replacements of the corresponding embodiments. With regard to the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. The singular form of the noun corresponding to the project may include one or more of the things, unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include all possible combinations of the items enumerated together in the corresponding one of the phrases. As used herein, terms such as "1st", "2nd" or "first" and "second" may be used to distinguish a corresponding component from another component, but do not limit the components in other aspects (e.g., importance or order). This means that if an element (e.g., a first element) is referred to as being "coupled with," "coupled to," "connected with," or "connected to" another element (e.g., a second element) with or without the term "operably" or "communicatively," it indicates that the element may be coupled with the other element directly (e.g., by wire), wirelessly, or via a third element.

[0043] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms, such as "logic," "logic block," "component," or "circuitry." A module may be a single integral component suitable for performing one or more functions, or its smallest unit or part. For example, according to one embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0044] The present disclosure describes a system and method for modeling a sliding zoom effect based on a synthesis pipeline to simulate the effect of a single shot of a given single camera or multi-camera image capture. Using single camera / single shot image capture, the present system can simulate camera movement (along the main axis) and camera parameter (focal length / FoV) changes based on camera geometry, image fusion and hole filling. This process can be extended to multi-camera single instant shots and video sequence frames to simulate camera movement in horizontal and vertical directions.

[0045] Due to the nature of the camera geometry involved, emulating this effect requires at least a single camera with the depth of its FoV, but can be extended to two or more cameras.The present systems and methods may account for movement of the camera along a principal axis towards an object under focus.

[0046] The present disclosure describes a view synthesis pipeline based on camera geometry and depth for single / multi-camera image capture, which is used to generate views for camera movement in the horizontal direction (x direction), vertical direction (y direction) and translation direction (z direction), as well as changes in FoV / focal length. Specifically, the present system and method use the view synthesis pipeline for single camera system and dual camera system (dual camera system can be extended to more cameras) to synthesize sliding zoom effect.

[0047] In one embodiment, the system generates a sliding zoom effect using camera geometry and depth of one or more red-green-blue (RGB) cameras. In another embodiment, the system can be applied to more than one camera of a multi-camera system and for camera movement in other directions (x and y as well as z and FoV / focal length variation). The system and method can be further extended to video applications without departing from the scope of the present disclosure.

[0048] In this disclosure, the notation scheme is as follows. The matrix is ​​denoted as H, and (.) T represents transpose. The projection of point P is defined as Where P = (X, Y, Z) T , represented as a point u, is defined as where u=(x,y) T . Scalars are denoted as X or x. Correspondingly, I is used to represent images. I(x,y) or alternatively I(u) is the intensity of the image at position (x,y). Similarly, for a matrix H, H(x,y) represents the element at position (x,y) in the matrix. J n and 0 n represents an n×n identity matrix and an n×1 zero vector.

[0049] Given from different FoVθ n (θ n >θ n-1 ) and its corresponding depth D n The input image I of n (n≥1) cameras n , for each input image, project each 2D image point to a 3D coordinate using the corresponding depth value at that point, apply the desired operation, and then reproject the operated 3D point back to image coordinates.

[0050] For a single view, there may be areas of missing information after reprojection due to occlusions and fixed FoV. These areas can be filled by using an image with a larger FoV and the same image coordinate projection concept described above, followed by image fusion.

[0051] Any remaining areas where information is lost due to occlusion are then processed using a hole filling module.

[0052] After filling the holes, a shallow depth of field (SDoF) effect is applied to obtain the final composite image.

[0053] Each step in the above process is described in more detail below. The case for two cameras (ie, n=2) is considered, but the derivation can be easily extended to more cameras, or can be applied to a single camera with some preprocessing.

[0054] Image synthesis can be described as pixel motion with image distortion. Pixel movement can be derived for different methods for different objectives. For example, optical flow is used to find pixel movement between consecutive images. The present disclosure can derive pixel movement if the camera moves along the x and y directions and in z as well as changes in FoV or focal length.

[0055] Single Camera Setup

[0056] Let I 1 is the input image from camera 1, and D 1 is the corresponding depth map, so that at each pixel position u=(x,y) T At 1 (u).

[0057] Figure 2 is a diagram showing camera translation according to an embodiment of the present disclosure. Assuming a pinhole camera model, when the camera is moved from position C A 204 Move to C B 206 (such as Figure 2 As shown), any point P202 (where ) and their projections onto the camera's image plane is given in Equation (1) below.

[0058]

[0059] In this paper, the 2×1 vector u X , 3×3 matrix K X and the scalar D XEach of them is the pixel coordinates, internal parameters and depth on the image plane of point P of the camera at position X, X∈{A,B}. The 3×3 matrix R and 3×1 vector T are the relative rotation and translation of the camera at position B relative to position A. T can also be written as T=R(C A -C B ).

[0060] Camera 1 is at position C A Camera internal matrix at 204 It can be given by the following equation (2).

[0061]

[0062] here, Where W is the image width, Where H is the image height, is camera 1 at position C A 204, and s is the shear factor which is assumed to be 0. The principal point is defined as u 0 =(u 0 ,v 0 ) T Therefore, the camera internal matrix can be given by the following equation (3).

[0063]

[0064]

[0065] Figure 3 is a diagram showing the camera geometry under sliding zoom according to an embodiment of the present disclosure. Camera 1 is in the initial position have and focal length Translate a distance t to position And its focal length becomes And correspondingly, FoV becomes

[0066] The following assumptions are made for single camera sliding zoom. The translation of the camera center is on the principal axis, as shown in equation (4) below. In addition, there is no relative rotation about the camera translation. Therefore, R is the identity matrix J 3 .

[0067]

[0068] where t is the translation of the camera center along the principal axis, and and Point P is at position and The depth of the place.

[0069] Location The internal matrix at By zoom factor k and The matrix at is related and can be given by the following equation (5).

[0070]

[0071] Here, k can be expressed as Where D 0 is the sliding zoom depth (ie, the depth of the point / plane under focus). Substituting equations (3), (4), and (5) into (1) yields equation (6) below.

[0072]

[0073] When solving equation (6), we obtain in equation (7) below expression.

[0074]

[0075] Using equation (7), the camera translation step size t can be expressed as D 1 To distort I 1 (So ​​that for each pixel u = (x, y), the corresponding depth D can be obtained 1 (u)) to obtain the composite image If the camera moves in the z direction, then Equation (7) derives the pixel movement by determining the coordinate movement at the required sliding zoom depth. Similarly, D can be distorted 1 , and using the known t, the corresponding depth can be obtained In this article, forward warping with z-buffering is used for this purpose.

[0076] Figure 4A and Figure 4B is a diagram illustrating single-camera image synthesis under sliding zoom with forward warp according to an embodiment of the present disclosure. Figure 4A Shown with The input image I 1 ,and Figure 4B It is shown that after the sliding zoom step t, Composite image of

[0077] In epipolar geometry, the pixel shifts are along epipolar lines, which are related by the fundamental matrix of the two camera views. 1 will involve corresponding pixels in the pixel domain of the two images without knowing the pixel depth information. Therefore, as shown in the following equation (8), this is a necessary condition for point correspondence.

[0078]

[0079] in and

[0080] Due to artificially creating new camera positions So the fundamental matrix F 1 can be obtained in closed form. For the matrix form [] × The cross product of , is given below in equation (9).

[0081]

[0082] Can get location The epipolar lines in the pixel domain of the camera at , as shown in equation (10) below.

[0083]

[0084] All epipolar lines converge to The poles of the null space As shown in equation (11) below.

[0085]

[0086] Figure 5A and Figure 5B is a diagram showing single-camera image synthesis under sliding zoom showing epipolar lines according to an embodiment of the present disclosure. The camera at the pole is shown Figure 5A Shown with The input image I 1 502, and Figure 5B It is shown that after the sliding zoom step t, Composite image of

[0087] like Figure 5A and Figure 5B As shown in , occlusion occurs along the epipolar lines. Whenever there is a discontinuity along the epipolar lines, the pixel shifts will be different and will result in holes. Using the fundamental matrix F derived in closed form 1 , the epipolar line information can be used for hole filling and occlusion processing, as described in more detail below.

[0088] The camera movement can be given by the following equation (12). In this paper, m 1 、n 1 and t 1 Represents the camera movement in the horizontal, vertical and main axis directions respectively.

[0089]

[0090] Then equation (6) can be rewritten as equation (13) below.

[0091]

[0092] Solving equation (13), we can obtain , as set forth in equation (14) below.

[0093]

[0094] Equation (14) can be used to obtain a composite image for camera movement in the x, y, and z directions and for changes in FoV.

[0095] In Location Where Can be the initial position C 1 The actual camera FoVθ at 1 Partial FoV. Let position C 1 The actual internal matrix at is K 1 Then the internal matrix of the partial FoV can be obtained as described in the following equation (15):

[0096]

[0097] in

[0098] For the actual image position (where the camera rotation is the identity matrix J 3 ) 1 The zoom pixel coordinates A closed form equation can be obtained as set forth below in equation (16).

[0099]

[0100] The input image I can be converted into 1 and its depth map D 1 Digitally zoom to desired FoV To obtain the enlarged image and depth map.

[0101] Dual camera setup

[0102] Applying the resulting formula of equation (7) to a single camera results in many missing and occluded regions as the FoV increases or decreases. Some of these regions can be filled using projections from other available cameras with different FoVs. A second camera can be introduced into the system.2 is set to the input image from camera 2, and D 2 is set to the corresponding depth map so that each pixel position u and the corresponding depth D can be obtained 2 (u). The FoV of camera 2 is larger than that of camera 1 (i.e., θ 2 >θ 1 ).

[0103] Figure 6 is a diagram showing a dual camera system according to an embodiment of the present disclosure. The second camera is in C 2 At, with focal length f 2 It is assumed that the two cameras are well calibrated so that they are in the same plane and their principal axes are perpendicular to that plane. The baseline between the two cameras is b.

[0104] Point P is projected on the plane (u 2 ) and its projection on camera 1 after applying sliding zoom (i.e., camera 1 is translated to position yes ) can be obtained from equation (1) and can be written as the following equation (17).

[0105]

[0106] The camera center is from position C 2 To location The translation of can be given in the following equation (18).

[0107]

[0108] where b is the baseline between the two cameras.

[0109] It is assumed that there is no relative rotation between the two cameras (corrected by camera calibration), and therefore, R is the identity matrix. Cameras 1 and 2 have the same depth, since the cameras are assumed to be centered on the sample, with the major axes perpendicular to that plane. Thus, camera 1 is at position The depth of point P at Camera 2 is at position C 2 Depth D 2 It can be given in the following equation (19).

[0110]

[0111] The intrinsic matrix K of camera 2 2 With Camera 1 at position The internal matrix at is related, as shown in equation (20) below.

[0112]

[0113] Here, k′ can be given as in the following equation (21).

[0114]

[0115] in and f 2 is the focal length and and θ 2 They are respectively camera 1 (at position ) and the FOV of camera 2.

[0116] When equations (18), (19), (20) and (21) are substituted into equation (17), equation (22) is derived as follows.

[0117]

[0118] When solving equation (22), we obtain The expression of is shown in the following equation (23).

[0119]

[0120]

[0121] Using equation (23), the camera translation step size t can be expressed as D 2 To distort I 2 (So ​​that for each pixel u = (x, y), the corresponding depth D can be obtained 2 (u)) to obtain the composite image D 2 It can be twisted, and using the known t, the corresponding depth can be obtained Use forward warping with z-buffering for this step size.

[0122] Fig. 7A and Figure 7B is a diagram showing the synthesis of images from the second camera under sliding zoom according to an embodiment of the present disclosure. Specifically, Fig. 7A Shown with FoVθ 2 =77° input image I 2 ,and Figure 7B The composite image after sliding zoom step t is shown

[0123] Similar to the case of a single camera, the fundamental matrix F 2 It can be derived using a closed form equation, as shown in equation (24) below.

[0124]

[0125] in

[0126] Fundamental Matrix F 2 Can be Get, where the extreme Projection Matrix Indicates P 2 The pseudo-reversal and The pseudo-inverse of is shown in equation (25) below.

[0127]

[0128] Fig. 8A and Figure 8B is a diagram showing image synthesis from a second camera under sliding zoom with epipolar lines according to an embodiment of the present disclosure. Specifically, Fig. 8A Shown with FoVθ 2 =77° input image I 2 802, and Figure 8B The composite image after sliding zoom step t is shown The actual extreme point e of the wide image 2 is (875.11,540), and the pole of the position B image It is (815.18,540).

[0129] Closed form fundamental matrix F 2 It can be used again for occlusion processing to find the corresponding epipolar lines.

[0130] The derivation can be easily extended to include any number of additional cameras in the system.

[0131] Equation (17) can be generalized for camera movement in the x and y directions.

[0132] The camera movement can be given by the following equation (26). Assume that the baseline b is included in m 2 and / or 2 middle.

[0133]

[0134] Equation (22) can then be rewritten as the following equation (27).

[0135]

[0136] Solving equation (27), we obtain The expression of is shown in the following equation (28).

[0137]

[0138] Equation (28) can be used to obtain a composite image for camera movement in the x, y, and z directions and for changes in FoV.

[0139] Image Fusion

[0140] Composite image from the second camera Can be used to fill in the composite image from the first camera This is achieved by image fusion.

[0141] As a first step, identify the composite image In this paper, a simple scheme is implemented as follows to check the missing area at each pixel position (x, y) by using the following equation (29) The value of is used to create a binary mask B. In this paper, express A set of missing / occluded pixels.

[0142]

[0143] Equations (19) and (23) can also be used to identify these regions.

[0144] As a second step, the binary mask B is used to fuse the synthesized image. as well as To generate I F , as shown in equation (30) below.

[0145]

[0146] The depth of the synthesized image is also fused in a similar way To obtain D F .

[0147] Figure 9A-9F is a diagram illustrating image fusion according to an embodiment of the present disclosure. Fig.9A Shown with The input image I 1 ,and Fig. 9B Shown with FoVθ 2 =77° input image I 2 . Fig. 9C It is shown that after the sliding zoom step t, Composite image of and Fig.9D The composite image after sliding zoom step t is shown Fig.9E A binary mask B is shown, and Fig.9F The synthetic fused image I after sliding zoom step t is shown F .

[0148] Occlusion handling

[0149] For each synthesized sliding zoom image, regions with missing information (usually due to occlusions) are identified and filled in for satisfactory subjective viewing. Occlusions occur due to the nature of camera movement from foreground to background. Therefore, one restriction on filling occluded regions is that they should only fill the background, not the foreground.

[0150] As a first step, occluded regions are identified. F is the image generated after image fusion. M is a binary mask that depicts the occluded area. M is generated by checking the F The value of is simply generated as shown in the following equation (31). In this paper, Indicates I F A set of occluded pixels.

[0151]

[0152] Equations (19) and (23) can be substituted to identify these regions

[0153] A key piece of information is the fusion depth D of the synthesized image F , which distinguishes foreground from background. F Also since occlusions have holes, if depth is used for image inpainting, the holes in the depth itself must be filled first.

[0154] Deep hole filling

[0155] Reference now Fig.10 , a flowchart showing a method for nearest neighbor hole filling according to an embodiment of the present disclosure. The filled depth map is And do not initialize it to At 1002, for each pixel location (x, y) with M(x, y) = 1, the system determines the four nearest neighbors (upper, lower, left, right) with valid depth values. In the current implementation, the pixels are filled in raster scan order (i.e., row by row, from left to right). At 1004, the system determines the neighbor with the largest depth value (d max ). At 1006, the system sets the filled depth map equal to the maximum depth value at the given pixel location, At 1008, the system determines whether there are additional pixel positions. If there are additional pixel positions, the system returns to 1002 to repeat the method for the next pixel. If there are no additional pixel positions, the system terminates the method at 1010. Determining Fill Depth

[0156] Composite hole filling

[0157] The system can perform hole filling, also known as depth-aware inpainting, on a synthesized image based on propagation from the image background to the image foreground. The system can perform hole filling in an image based on filling holes in the image from parts of the image at the same depth or the next closest depth, in a back-to-front order.

[0158] The input is a synthetic image I F , synthetic depth map for hole filling And the original occlusion mask M. The output is set to the composite image in Sure All unique values ​​in d u is set to an array of unique depth values, where d u In descending order of depth value S. For example, d u is an array representing [5,4,3,2,1], where a depth value S of 5 is the farthest and a value S of 1 is the closest. The system starts by selecting the depth segment corresponding to the farthest depth value S.

[0159] Reference now Fig.11 , a flowchart illustrating a method for depth-aware inpainting according to an embodiment of the present disclosure. At 1102, the system determines a depth mask D for a current depth segment between a current depth value S and a subsequent depth value (S-1). n The system can determine the depth mask D based on the following equation (32): n Here, ≥ is the element-wise matrix of the 'greater than or equal to' operation, and & is the element-wise matrix AND operation.

[0160]

[0161] At 1104, the system determines the depth mask I corresponding to s The system can determine the image segment based on the following equation (33). Here, · is an element-by-element matrix product operation.

[0162] I s =I F ·D n …(33)

[0163] At 1106, the system updates the occlusion mask with the depth mask to obtain a current occlusion mask M for the current depth segment. curr The system can obtain the current occlusion mask M based on equation (34): curr .

[0164] M curr =M·D n …(34)

[0165] At 1108, the system updates the current occlusion mask based on the previous mask from the previous depth segment. For the first depth segment, the system may initialize M with zeros. Prev The system can update the current occlusion mask based on the following equation (35). Here, || is an element-by-element matrix OR operation.

[0166] M curr =M curr ||M Prev …(35)

[0167] At 1110, if M curr (x, y) = 1, then for each pixel position, the system determines the same line I s The system can also determine more than one valid pixel in the horizontal and vertical directions. At 1112, the system uses Based on the nearest valid pixel I on the same line s (x,y′) to update the synthetic view At 1114, the system updates the current occlusion mask M curr (x, y)=0, and the original occlusion mask M(x, y)=0.

[0168] At 1116, the system determines whether there are other pixel positions. If there are other pixel positions, the system returns to 1110 to determine the nearest valid pixel for the next pixel position. When there are no other pixel positions, at 1118, the system sets the current occlusion mask M to the nearest valid pixel for the next pixel position. curr Propagate to the previous occlusion mask M Prev At 1120, the system determines that u Whether there is a subsequent depth value in . When there is a subsequent depth value, the system returns to 1102 to repeat the method for the next depth value.

[0169] When there is no subsequent depth value, at 1122, the system performs the following operations according to M. A windowed average (for example, the kernel size is set to 10x10) filter is applied to the occluded regions in .

[0170] Figures 12A-12D is a diagram illustrating a hole filling process for a specific step s according to an embodiment of the present disclosure. Fig. 12A The composite image I is shown F ,and Fig. 12B shows the depth-based image segment I at s s . Fig. 12C Shows the occlusion mask M at depth value s curr ,and Fig.12D Shows the filled image after depth value s

[0171] Figure 13A-13G is a diagram showing the result of a hole filling process according to an embodiment of the present disclosure. Fig.13A Shown with The input image I 1 ,and Fig. 13B Shown with FoVθ 2 =77° input image I 2 . Fig. 13C The synthetic fused image I after sliding zoom step t is shown F . Fig.13D The fused image I F The depth map D F ,and Fig.13E Shown in hole filling The depth map after. Fig.13F Shows the composite image after hole filling using OpenCV (PSNR=16.0680) and Figure 13G A composite image showing hole filling (PSNR=20.0612) according to the present disclosure

[0172] SDoF

[0173] After image synthesis and occlusion processing, the system can also apply SDoF effects. The purpose of the bokeh algorithm is to implement the SF effect by changing the blur intensity according to the depth. Therefore, for the bokeh effect, it is necessary to determine the blur kernel size for each area based on the depth. As described below, the diameter of the blur kernel is referred to as the circle of confusion (CoC) in the literature related to depth of field. Unlike the synthetic SDoF effect for image capture, the blur intensity varies not only according to the depth, but also according to the sliding zoom translation parameter t for the sliding zoom effect.

[0174] Fig.14 is a diagram showing a thin-lens camera model and an aperture circle according to an embodiment of the present disclosure.

[0175] like Fig.14 Shown: u,u f ,u n They are located in P 0 , P f , P n The distance of the object at f 、v n They are located in P 0 , P f , P n The image distance of the object at the point; f is the focal length of the lens; D 0 ,D f ,D n They are P 0 , P f , P n is the depth at the image plane; d is the effective lens size; and c is the circle of astigmatism at the image plane.

[0176] The lens equation is stated in equation (36) below.

[0177]

[0178] c. Lens aperture A, magnification factor m, focal length object depth D 0 The relationship between and the depth D of another object can be given as follows in equation (37).

[0179]

[0180] The amplification factor m is set forth in equation (38) below.

[0181]

[0182] The above equation holds true when no zoom is applied to the camera, that is, when the focal length f of the thin lens f is fixed. Under sliding zoom conditions, the focal length changes according to the movement t along the main axis. Here, we denote the focal length relative to the step length t as f(t).

[0183] Therefore, the relationship between f(t) and t is shown in the following equation (39).

[0184]

[0185] Therefore, the amplification factor m(t) relative to t can be obtained as in the following equation (40).

[0186]

[0187] The above equation is completely consistent with the pinhole camera model under sliding zoom, that is, the magnification factor of the object under the focal length is fixed. In addition, the relative depth between objects in the scene |DD0 |Remains constant for a single image capture. Assuming the lens aperture A remains constant, the circle of astigmatism can be obtained using equation (41) below.

[0188]

[0189] where c(t) is the diameter of the circle of astigmatism of the object at depth D and camera translation t along the principal axis.

[0190] Any conventional method can be used for synthetic SDoF generation as long as the CoC of all pixels within the image is found.

[0191] Fig.15 1 is a flow chart illustrating a method for providing a sliding zoom view synthesis for a setting having one or more cameras according to an embodiment of the present disclosure. At 1502, the system obtains a first image and a first depth map corresponding to the first image, and at 1504, the system obtains a second image and a second depth map corresponding to the second image. At least one of the first image and the second image is obtained using at least one camera of the electronic device. The first image and the second image have different FoVs. At 1506, the system uses the first image and the first depth map to generate a first synthesized image and a first synthesized depth map corresponding to the first synthesized image. At 1508, the system uses the second image and the second depth map to generate a second synthesized image and a second synthesized depth map corresponding to the second synthesized image. At 1510, the system generates a fused image from the first synthesized image and the second synthesized image, and the system generates a fused depth map from the first synthesized depth map and the second synthesized depth map. At 1512, the system performs hole filling on the fused depth map and the fused image. At 1514, the system applies an SDoF effect to obtain a final synthesized image with a sliding zoom effect.

[0192] Fig.16 is a diagram showing the corresponding Fig.15 Flowchart of a sliding zoom view synthesis pipeline with a single camera setup and depth map estimation.

[0193] Two input images are provided: a first input image I from a first camera (e.g., a telephoto camera) 1 (e.g., a telephoto image), and a second input image I from a second camera (e.g., a wide-view camera) 2 (e.g., wide image). 1 and the second input image I 2 Provided to depth estimation block 1602 .

[0194] Assume that the depth map is obtained by taking a stereo input image pair (I 1 and I 2) is obtained by a stereo depth estimation module of the second camera. Specifically, the depth of the second camera is found by using stereo vision of the image pair (2 cameras passive). However, the depth can also be obtained by other means, in which case the second input image I from the second camera may not be needed. 2 For example, the depth of the second camera can be obtained by a time-of-flight (ToF) camera (1 camera is active), as described about Fig. 20 Described in more detail.

[0195] The first input image I 1 and its corresponding depth map D 1 (from depth estimation block 1602) is provided to view synthesis block 1604. 1 is provided to the first digital zoom frame 1606, and the first input image I 1 Provided to the second digital zoom frame 1608.

[0196] To perform sliding zoom view synthesis, the second digital zoom frame 1608 and the first digital zoom frame 1606 respectively zoom the first input image I 1 and the depth map D 1 Perform digital zoom up to a specific depth FoV (DFoV)θ 1 ′, where (θ 1 >θ 1 ′), to obtain the corresponding enlarged image I 1 ′ and the corresponding enlarged depth map D 1 '. The enlarged depth map D output from the first digital zoom frame 1606 is 1 ′ and the enlarged image I output from the second digital zoom frame 1608 1 ′ is input to the view synthesis box 1610.

[0197] The composite image output from the view synthesis block 1604 and the synthesized magnified image output from view synthesis block 1610 Provided to the image fusion block 1612. The synthesized depth map output from the view synthesis block 1604 and the synthesized up-scaled depth map output from the view synthesis block 1610 Provided to the depth fusion box 1614.

[0198] The fused depth map D output from the depth fusion block 1614 F The fused image I output from the image fusion block 1612 is input to the depth hole filling block 1616. F and the hole-filled depth map output from the depth hole filling block 1616 The composite image output from the depth-aware image restoration block 1618 is and the hole-filled depth map output from the depth hole filling block 1616 Input to SDoF box 1620, which generates a composite image

[0199] Figures 17A-17D The method according to the embodiment of the present disclosure is shown Fig.16 An example of a single-camera setup where the composition pipeline performs sliding zoom view composition. Fig.17A Shown with FoVθ 1 Input image I with ′=30° 1 ',and Fig. 17B Shown with FoVθ 1 =45° input image I 1 . Fig. 17C Shown is an input image I 1 ′ has FoVθ z =37.2730° composite image and Fig.17D It is shown that after applying SDoF, the effective FoVθ z =37.2730° composite image

[0200] Fig.18 is a diagram showing the corresponding Fig.15 Flowchart of a sliding zoom view synthesis pipeline with a dual-camera setup and depth map estimation.

[0201] Three input images are provided: a first input image I from a first camera (eg, a telephoto camera) 1 (e.g., a telephoto image), a second input image I from a second camera (e.g., a wide image camera) 2 (e.g., a wide image), and a third input image I from a third camera (e.g., an ultra-wide image camera) 3 (e.g., ultra-wide image). 1 and the second input image I 2 is provided to the depth estimation block 1802, and the second input image I 2 and the third input image I 3 Provided to depth estimation block 1804 .

[0202] The depth map D output from the depth estimation block 1802 1 Corresponding to the first input image I 1 , and the depth map D output from the depth estimation block 1804 2 Corresponding to the second input image I2 It can be assumed that the depth map requires a stereo input image pair (I 1 and I 2 ;I 2 and I 3 ) is obtained by a stereo depth estimation module. However, the depth map may also be obtained by other means, in which case the third input image I from the third camera may not be needed. 3 The following is about Fig. 22 Describe this situation in more detail.

[0203] The first input image I 1 and its corresponding depth map D 1 Provided to the view synthesis block 1806. The second input image I 2 and its corresponding depth map D 2 Provided to view synthesis block 1808. The synthesized image output from view synthesis block 1806 and the composite image output from view synthesis block 1808 Provided to the image fusion block 1810. The synthesized depth map output from the view synthesis block 1806 and the synthesized depth map output from the view synthesis block 1808 Provided to the depth fusion box 1812.

[0204] The fused depth map D output from the depth fusion block 1812 F The fused image I output from the image fusion block 1810 is input to the depth hole filling block 1814. F and the hole-filled depth map output from the depth hole filling block 1814 The composite image output from the depth-aware image restoration block 1816 is and the hole-filled depth map output from the depth hole filling block 1814 Input to SDoF box 1818, which generates a composite image

[0205] Figures 19A-19D The method according to the embodiment of the present disclosure is shown Fig.18 An example of a dual-camera setup where the synthesis pipeline performs sliding zoom view synthesis. Fig.19A Shown with FoVθ 1 =45° input image I 1 ,and Fig.19B Shown with FoVθ 2 =77° input image I 2 . Fig.19C Shown with FoVθ z =50° composite image and Fig.19D It is shown that after applying SDoF, the effective FoVθ z =50° composite image

[0206] Fig. 20 is a flow chart illustrating a method for providing sliding zoom view synthesis for a single camera setup with a depth map input according to an embodiment of the present disclosure.

[0207] At 2002, the system receives an image and its corresponding depth map. At 2004, the system applies digital zoom to obtain an image with a desired FoV. Digital zoom is also applied to the depth map to obtain a magnified depth. The desired FoV is less than the original FoV. At 2006, the system sets a desired camera translation along the principal axis (z) or along the x, y, and z axes. The translation along the principal axis is determined according to equation (4), while the translation along the x, y, and z axes is determined according to equation (12). At 2008, the system receives a desired focal length distance determined by user input. At 2010, for all pixels, the system obtains the necessary warp. The image and depth map are warped using a forward warp as described in equation (7) or equation (14).

[0208] At 2012, camera parameters are collected from the input image and depth map, and a camera intrinsic matrix is ​​established. The camera intrinsic parameters may include focal length, image width, and image height, and the matrix is ​​established according to equations (1)-(3). At 2014, the desired camera translation along the principal axis (z) or along the x, y, and z axes is set. The translation along the principal axis is determined according to equation (18), while the translation along the x, y, and z axes is determined according to equation (26). At 2016, the desired focal length is set by user input. At 2018, for all pixels, the necessary distortion is obtained. The image and depth map are warped using forward distortion as described in equation (23) or equation (28).

[0209] At 2020, the system fuses the synthesized image and depth according to equations (29) and (30), and at 2022, the system performs depth and image hole filling according to equations (31)-(35). At 2024, the system applies the SDoF effect, and at 2026, the system obtains the final synthesized image.

[0210] Fig.21 The present invention is based on the embodiment of the present invention. Fig. 20 Flowchart of the sliding zoom view synthesis pipeline for a single camera setup.

[0211] Provide an input image, an input image I from a camera (e.g., a telephoto camera) 1And the corresponding depth map D 1 . Input image I 1 and the depth map D 1 Provided to the view synthesis block 2102. The depth map D 1 The first digital zoom frame 2104 is further provided with the input image I 1 Further provided to the second digital zoom frame 2106 .

[0212] The enlarged depth map D output from the first digital zoom frame 2104 1 ' and the enlarged image I output from the second digital zoom frame 2106 1 ’ is input to the view synthesis box 2108.

[0213] The composite image output from the view synthesis block 2102 and the composite image output from the view synthesis block 2108 Provided to the image fusion block 2110. The synthesized depth map output from the view synthesis block 2102 and the synthesized depth map output from the view synthesis block 2108 Provided to the depth fusion box 2112.

[0214] The fused depth map D output from the depth fusion block 2112 F The fused image I output from the image fusion frame 2110 is input to the depth hole filling frame 2114. F and the hole-filled depth map output from the depth hole filling block 2114 The composite image output from the depth-aware image restoration block 2116 is and the hole-filled depth map output from the depth hole filling block 2114 Input to SDoF box 2118, which generates a composite image

[0215] Fig. 22 is a flow chart illustrating a method for providing sliding zoom view synthesis for a dual camera setup with depth map input according to an embodiment of the present disclosure.

[0216] At 2202, the system inputs a first image and its corresponding depth map. At 2204, the system collects camera parameters and establishes a camera intrinsic matrix. The camera intrinsic parameters may include focal length, image width, and image height, and the matrix is ​​established according to equations (1)-(3). At 2206, the system sets the desired camera translation along the principal axis (z) or along the x, y, and z axes. The translation along the principal axis is determined according to equation (4), while the translation along the x, y, and z axes is determined according to equation (12). At 2208, the system sets the desired focal length through user input. At 2210, for all pixels, the system obtains the necessary distortion. The image and depth map are distorted using forward distortion as described in equation (7) or equation (14).

[0217] At 2212, the system inputs a first image and its corresponding depth map. At 2214, the system collects camera parameters from the input image and depth map and establishes a camera intrinsic matrix. The camera intrinsic parameters may include focal length, image width, and image height, and the matrix is ​​established according to equations (1)-(3). At 2216, the system sets the desired camera translation along the principal axis (z) or along the x, y, and z axes. The translation along the principal axis is determined according to equation (18), while the translation along the x, y, and z axes is determined according to equation (26). At 2218, the system sets the desired focal length through user input. At 2220, for all pixels, the system obtains the necessary distortion. The image and depth map are distorted using forward distortion as described in equation (23) or equation (28).

[0218] At 2222, the system fuses the composite image and depth according to equations (29) and (30), and at 2224, the system performs depth and image hole filling according to equations (31)-(35). At 2226, the system applies the SDoF effect, and at 2228, the system obtains the final composite image.

[0219] Fig.23 The present invention is based on the embodiment of the present invention. Fig. 22 Flowchart of a sliding zoom view synthesis pipeline for a dual-camera setup.

[0220] Provide a first input image I from a first camera (e.g., a telephoto camera) 1 (e.g., telephoto image) and its corresponding first depth map D 1 , and a second input image I from a second camera (e.g., a wide image camera) 2 (e.g., wide image) and its corresponding second depth map D 2 .

[0221] The first input image I 1and the first depth map D 1 is provided to the first view synthesis frame 2302, and the second input image I 2 and the second depth map D 2 Provided to the second view synthesis frame 2304. The first synthesized image output from the first view synthesis frame 2302 and the second synthesized image output from the second view synthesis block 2304 Provided to the image fusion block 2306. The first synthesized depth map output from the first view synthesis block 2302 and the second synthesized depth map output from the second view synthesis block 2304 Provided to the depth fusion box 2308.

[0222] The fused depth map D output from the depth fusion block 2308 F The fused image I output from the image fusion block 2306 is input to the depth hole filling block 2310. F and the hole-filled depth map output from the depth hole filling block 2310 The composite image output from the depth-aware image restoration block 2312 is and the hole-filled depth map output from the depth hole filling block 2310 Input to SDoF box 2314, which generates a composite image

[0223] Fig.24 is a block diagram showing an electronic device in a network environment according to an embodiment. Fig.24, the electronic device 2401 in the network environment 2400 may communicate with the electronic device 2402 via a first network 2498 (e.g., a short-range wireless communication network), or communicate with the electronic device 2404 or the server 2408 via a second network 2499 (e.g., a long-range wireless communication network). The electronic device 2401 may communicate with the electronic device 2404 via the server 2408. The electronic device 2401 may include a processor 2420, a memory 2430, an input device 2450, a sound output device 2455, a display device 2460, an audio module 2470, a sensor module 2476, an interface 2477, a haptic module 2479, a camera module 2480, a power management module 2488, a battery 2489, a communication module 2490, and a subscriber identification module (SIM) 2496 or an antenna module 2497. In one embodiment, at least one of the components (e.g., display device 2460 or camera module 2480) may be omitted from electronic device 2401, or one or more other components may be added to electronic device 2401. Some of the components may be implemented as a single integrated circuit (IC). For example, sensor module 2476 (e.g., fingerprint sensor, iris sensor, or light sensor) may be embedded in display device 2460 (e.g., display).

[0224] The processor 2420 may execute software (e.g., program 2440) for controlling at least one other component (e.g., hardware or software component) coupled to the processor 2420 of the electronic device 2401, for example, and may perform various data processing or calculations. As at least part of the data processing or calculation, the processor 2420 may load a command or data received from another component (e.g., sensor module 2476 or communication module 2490) in the volatile memory 2432, process the command or data stored in the volatile memory 2432, and store the resulting data in the non-volatile memory 2434. The processor 2420 may include a main processor 2421 (e.g., a central processing unit (CPU) or an application processor (AP)), and an auxiliary processor 2423 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor center processor, or a communication processor (CP)), which may operate independently of the main processor 2421 or in conjunction with the main processor. Additionally or alternatively, the auxiliary processor 2423 may be adapted to consume less power than the primary processor 2421, or to perform specific functions. The auxiliary processor 2423 may be implemented separate from the primary processor 2421 or as part of the primary processor.

[0225] The auxiliary processor 2423 may control at least some of the functions or states related to at least one component (e.g., the display device 2460, the sensor module 2476, or the communication module 2490) among the components of the electronic device 2401, in place of the main processor 2421 when the main processor 2421 is in an inactive (e.g., sleeping) state, or together with the main processor 2421 when the main processor 2421 is in an active state (e.g., executing an application). The auxiliary processor 2423 (e.g., an image signal processor or a communication processor) may be implemented as a part of another component (e.g., a camera module 2480 or a communication module 2490) that is functionally related to the auxiliary processor 2423.

[0226] The memory 2430 may store various data used by at least one component of the electronic device 2401 (e.g., the processor 2420 or the sensor module 2476). The various data may include, for example, software (e.g., program 2440) and input data or output data for commands related thereto. The memory 2430 may include a volatile memory 2432 or a non-volatile memory 2434.

[0227] The program 2440 may be stored as software in the memory 2430 , and may include, for example, an operating system (OS) 2442 , middleware 2444 , and / or an application program 2446 .

[0228] The input device 2450 may receive a command or data to be used by another component (eg, the processor 2420) of the electronic device 2401 from outside (eg, a user) of the electronic device 2401. The input device 2450 may include, for example, a microphone, a mouse, or a keyboard.

[0229] The sound output device 2455 can output sound signals to the outside of the electronic device 2401. The sound output device 2455 may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as playing multimedia or recording, and the receiver may be used to receive incoming calls. The receiver may be implemented as a part of the speaker or as a separate speaker.

[0230] The display device 2460 may visually provide information to the outside of the electronic device 2401 (e.g., a user). The display device 2460 may include, for example, a display, a hologram device, or a projector, and a control circuit that controls a corresponding one of the display, the hologram device, and the projector. The display device 2460 may include a touch circuit suitable for detecting a touch, or a sensor circuit (e.g., a pressure sensor) suitable for measuring the strength of the force caused by the touch.

[0231] The audio module 2470 can convert sound into an electrical signal and vice versa. The audio module 2470 can obtain sound via the input device 2450, or output sound via the sound output device 2455 or a headset of an external electronic device 2402 directly (eg, wired) or wirelessly coupled to the electronic device 2401.

[0232] The sensor module 2476 may detect an operating state (e.g., power or temperature) of the electronic device 2401 or an environmental state (e.g., a user's state) outside the electronic device 2401, and then generate an electrical signal or data value corresponding to the detected state. The sensor module 2476 may include, for example, a gesture sensor, a gyroscope sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or a light sensor.

[0233] The interface 2477 may support one or more designated protocols for coupling the electronic device 2401 directly (e.g., wired) or wirelessly with the external electronic device 2402. The interface 2477 may include, for example, a High-Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital (SD) card interface, or an audio interface.

[0234] The connection terminal 2478 may include a connection via which the electronic device 2401 can be physically connected to the external electronic device 2402. The connection terminal 2478 may include, for example, an HDMI connection, a USB connection, an SD card connection, or an audio connection (e.g., a headphone connection).

[0235] The haptic module 2479 may convert the electrical signal into mechanical stimulation (eg, vibration or movement) or electrical stimulation that can be recognized by the user via tactile sensation or kinesthetic sensation. The haptic module 2479 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0236] The camera module 2480 may capture still images or moving images. The camera module 2480 may include one or more lenses, image sensors, image signal processors, or flashes.

[0237] The power management module 2488 may manage power supplied to the electronic device 2401. The power management module 2488 may be implemented as, for example, at least a portion of a power management integrated circuit (PMIC).

[0238] The battery 2489 may supply power to at least one component of the electronic device 2401. The battery 2489 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0239] The communication module 2490 can support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 2401 and an external electronic device (e.g., electronic device 2402, electronic device 2404, or server 2408), and perform communication via the established communication channel. The communication module 2490 may include one or more communication processors that can operate independently of the processor 2420 (e.g., AP) and support direct (e.g., wired) communication or wireless communication. The communication module 2490 may include a wireless communication module 2492 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module), or a wired communication module 2494 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate via a first network 2498 (e.g., a short-range communication network such as Bluetooth TM , Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA) standards) or a second network 2499 (e.g., a remote communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN)) to communicate with an external electronic device. These various types of communication modules can be implemented as a single component (e.g., a single IC), or can be implemented as multiple components separated from each other (e.g., multiple ICs). The wireless communication module 2492 can use subscriber information (e.g., International Mobile Subscriber Identity (IMSI)) stored in the subscriber identification module 2496 to identify and authenticate the electronic device 2401 in the communication network (e.g., the first network 2498 or the second network 2499).

[0240] The antenna module 2497 may transmit a signal or power to the outside of the electronic device 2401 (e.g., an external electronic device) or receive a signal or power from the outside. The antenna module 2497 may include one or more antennas, and therefore, at least one antenna appropriate for a communication scheme used in a communication network (such as the first network 2498 or the second network 2499) may be selected, for example, by the communication module 2490 (e.g., the wireless communication module 2492). Then, via the selected at least one antenna, a signal or power is transmitted or received between the communication module 2490 and the external electronic device.

[0241] At least some of the components described above may be coupled to each other and transmit signals (e.g., commands or data) therebetween via an inter-peripheral communication scheme (e.g., a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)).

[0242] Commands or data may be transmitted or received between the electronic device 2401 and the external electronic device 2404 via a server 2408 coupled to the second network 2499. Each of the electronic devices 2402 and 2404 may be a device of the same type or a different type as the electronic device 2401. All or some of the operations performed at the electronic device 2401 may be performed at one or more of the external electronic devices 2402, 2404, or 2408. For example, if the electronic device 2401 should automatically perform a function or service, or in response to a request from a user or another device, the electronic device 2401 may request one or more external electronic devices to perform at least a portion of the function or service instead of or in addition to performing the function or service. The one or more external electronic devices that receive the request may perform at least a portion of the requested function or service, or an additional function or additional service related to the request, and transmit the result of the execution to the electronic device 2401. The electronic device 2401 may provide the result as at least a part of the request to the response, with or without further processing of the result. For this purpose, cloud computing, distributed computing or client-server computing techniques may be used, for example.

[0243] One embodiment may be implemented as software (e.g., program 2440) that includes one or more instructions stored in a storage medium (e.g., internal memory 2436 or external memory 2438) that can be read by a machine (e.g., electronic device 2401). For example, a processor of the electronic device 2401 may call at least one of the one or more instructions stored in the storage medium and execute the instruction with or without one or more other components under the control of the processor. Thus, the machine may be operated to perform at least one function according to at least one of the called instructions. The one or more instructions may include code generated by a compiler or code executable by a translation program. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" indicates that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but this term does not distinguish between locations where data is semi-permanently stored in the storage medium and locations where data is temporarily stored in the storage medium.

[0244] According to one embodiment, the method of the present disclosure may include and be provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or distributed via an application store (e.g., Play Store TM) online distribution (e.g., downloading or uploading), or directly between two user devices (e.g., smart phones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium (e.g., a memory of a manufacturer's server, an application store's server, or a relay server).

[0245] According to one embodiment, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities. One or more of the components described above may be omitted, or one or more other components may be added. Alternatively or in addition, multiple components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may still perform one or more functions of each of the multiple components in the same or similar manner as performed by a corresponding one of the multiple components before integration. The operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or heuristically, or one or more of the operations may be performed or omitted in a different order, or one or more other operations may be added.

[0246] Although certain embodiments of the present disclosure have been described in the detailed description of the present disclosure, the present disclosure may be modified in various forms without departing from the scope of the present disclosure. Therefore, the scope of the present disclosure should not be determined based solely on the described embodiments, but should be determined based on the appended claims and their equivalents.

Claims

1. A method for providing a sliding zoom effect by an electronic device, the method comprising: obtaining a first image and a second image, the first image having a first depth map corresponding to the first image, and the second image having a second depth map corresponding to the second image, wherein the first image and the second image have different fields of view; generating a first synthesized image and a first synthesized depth map corresponding to the first synthesized image using the first image and the first depth map, respectively; generating a second synthesized image and a second synthesized depth map corresponding to the second synthesized image using the second image and the second depth map, respectively; generating a fused image from the first composite image and the second composite image; generating a fused depth map from the first synthesized depth map and the second synthesized depth map; as well as Based on processing the fused image and the fused depth map, generating a final composite image, The step of generating the final composite image comprises: performing hole filling on the fused depth map and the fused image; as well as A shallow depth of field effect is applied to obtain the final composite image with the sliding zoom effect.

2. The method of claim 1 , wherein obtaining the first image having the first depth map and the second image having the second depth map comprises: receiving the first image and the third image from a first camera; determining the first depth map of the first image by performing depth estimation using the first image and the third image; determining the second image by performing digital zoom on the first image; as well as The second depth map is determined by performing digital zoom on the first depth map.

3. The method of claim 1 , wherein obtaining the first image having the first depth map and the second image having the second depth map comprises: receiving the first image and the second image from a first camera; receiving a third image from the second camera; determining the first depth map by performing depth estimation using the first image and the second image; as well as The second depth map is determined by performing depth estimation using the second image and the third image.

4. The method of claim 1, wherein obtaining the first image having the first depth map and the second image having the second depth map comprises: receiving the first image from a first camera; deriving the first depth map; determining the second image by performing digital zoom on the first image; as well as The second depth map is determined by performing digital zoom on the first depth map.

5. The method of claim 1 , wherein obtaining the first image having the first depth map and the second image having the second depth map comprises: receiving the first image from a first camera; receiving the second image from a second camera; as well as The first depth map and the second depth map are derived.

6. The method of claim 1, wherein generating the first synthesized image and the first synthesized depth map comprises: setting a camera translation along at least one axis of the first image; Setting the focal distance via user input; as well as The first image and the first depth map are warped using a forward warp based on the camera translation and the focal distance of the first image to obtain the first synthesized image and the first synthesized depth map.

7. The method of claim 6, wherein generating the second synthesized image and the second synthesized depth map comprises: setting a camera translation along at least one axis of the second image; as well as Based on the camera translation and the focal distance of the second image, the second image and the second depth map are warped using a forward warp to obtain the second synthesized image and the second synthesized depth map.

8. The method of claim 1 , wherein performing hole filling comprises: performing depth hole filling using the fused depth map to obtain a hole-filled depth map; as well as Depth-aware image inpainting is performed using the fused image and the hole-filled depth map to obtain a hole-filled image.

9. The method of claim 8, wherein performing depth-aware image inpainting comprises: updating an occlusion mask based on each of a plurality of unique depth values ​​in the hole-filled depth map; as well as A windowed average filter is applied to the occluded regions in the fused image according to the updated occlusion mask.

10. An electronic device comprising: At least one camera; A processor, which when instructions are executed is caused to: obtaining a first image and a second image, the first image having a first depth map corresponding to the first image, and the second image having a second depth map corresponding to the second image, wherein the first image and the second image have different fields of view; generating a first synthesized image and a first synthesized depth map corresponding to the first synthesized image using the first image and the first depth map, respectively; generating a second synthesized image and a second synthesized depth map corresponding to the second synthesized image using the second image and the second depth map, respectively; generating a fused image from the first composite image and the second composite image; generating a fused depth map from the first synthesized depth map and the second synthesized depth map; and Based on processing the fused image and the fused depth map, generating a final composite image, Wherein, when generating the final composite image, the instructions further cause the processor to: performing hole filling on the fused depth map and the fused image; and A shallow depth of field effect is applied to obtain said final composite image with a sliding zoom effect.

11. The electronic device of claim 10, wherein when obtaining the first image having the first depth map and the second image having the second depth map, the instructions further cause the processor to: receiving the first image and the third image from a first camera; determining the first depth map of the first image by performing depth estimation using the first image and the third image; determining the second image by performing digital zoom on the first image; and The second depth map is determined by performing digital zoom on the first depth map.

12. The electronic device of claim 10, wherein when obtaining the first image having the first depth map and the second image having the second depth map, the instructions further cause the processor to: receiving the first image and the second image from a first camera; receiving a third image from the second camera; determining the first depth map by performing depth estimation using the first image and the second image; and The second depth map is determined by performing depth estimation using the second image and the third image.

13. The electronic device of claim 10, wherein when obtaining the first image having the first depth map and the second image having the second depth map, the instructions further cause the processor to: receiving the first image from a first camera; deriving the first depth map; determining the second image by performing digital zoom on the first image; and The second depth map is determined by performing digital zoom on the first depth map.

14. The electronic device of claim 10, wherein when obtaining the first image having the first depth map and the second image having the second depth map, the instructions further cause the processor to: receiving the first image from a first camera; receiving the second image from a second camera; and The first depth map and the second depth map are derived.

15. The electronic device of claim 10, wherein in generating the first synthesized image and the first synthesized depth map, the instructions further cause the processor to: setting a camera translation along at least one axis of the first image; setting the focus distance via user input; and The first image and the first depth map are warped using a forward warp based on the camera translation and the focal distance for the first image to obtain the first synthesized image and the first synthesized depth map.

16. The electronic device of claim 15, wherein when generating the second synthesized image and the second synthesized depth map, the instructions further cause the processor to: setting a camera translation along at least one axis of the second image; and Based on the camera translation and the focal distance for the second image, the second image and the second depth map are warped using a forward warp to obtain the second synthesized image and the second synthesized depth map.

17. The electronic device of claim 10, wherein when performing hole filling, the instructions further cause the processor to: Perform depth hole filling using the fused depth map to obtain a hole-filled depth map; and Depth-aware image inpainting is performed using the fused image and the hole-filled depth map to obtain a hole-filled image.

18. The electronic device of claim 17, wherein when performing depth-aware image inpainting, the instructions further cause the processor to: updating an occlusion mask based on each of a plurality of unique depth values ​​in the hole-filled depth map; and A windowed average filter is applied to the occluded regions in the fused image according to the updated occlusion mask.

Images