Multi-camera post-capture image processing method, system and media

Through multi-camera capture and editing technology, the flexibility of image field of view and zoom level adjustment is solved, and dynamic adjustment of image field of view and zoom level is achieved, providing more flexible image viewing and editing capabilities.

CN114189626BActive Publication Date: 2024-11-19QUALCOMM INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111332040.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2017-08-16
Filing Date
2018-07-02
Publication Date
2024-11-19
Estimated Expiration
2038-07-02

AI Technical Summary

Technical Problem

The prior art is difficult to re-compile the field of view and zoom levels of images after shooting, limiting the flexibility of users when viewing images, especially in scenarios where different perspectives or details are needed.

Method used

Through a method and system, using multiple cameras to capture images, store and edit images, realize changes in the field of view and zoom levels, generate edited images with new fields of view and pixels, support the creation and storage of images compiled by users, and allow users to Adjust the field of view and zoom levels without changing the original image.

Benefits of technology

Dynamic adjustment of image field of view and zoom levels is realized, providing more flexible image viewing and editing capabilities, allowing users to view images of different perspectives and details while maintaining the quality of the original image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114189626B_ABST
    Figure CN114189626B_ABST
Patent Text Reader

Abstract

Image processing may include storing a first image and a second image in response to a first command. The first image may be captured by a first camera and include a first pixel and have a first field of view. The second image may be captured by a second camera and include a second pixel and have a second field of view. A second command may be received and the field of view of the first image may be changed based on the second command. The edited image may have a third field of view and include pixels based on the first pixel and the second pixel.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the invention patent application with the application date of July 2, 2018, application number 201880052458.8, and invention name “Multi-camera post-capture image processing method and system”. Technical Field

[0002] More particularly, the invention relates to reconstructing digital images of scenes. Background Art

[0003] Mobile devices typically include a camera for imaging a scene. The camera includes an optical assembly having a lens and an aperture that directs light from the scene onto an image sensor. The image sensor measures the focused light, and a processor saves the measurement as an image file.

[0004] The configuration of the optical assembly determines how light from the scene is focused onto the image sensor. For example, the configuration of the optical assembly determines (a) the field of view and (b) the zoom level of the image. The field of view relates to the amount of scene area captured in the image. The zoom level relates to the quality of the captured scene area.

[0005] For example, a user may take a panoramic image of a city skyline and a telephoto image of a specific building in the skyline. Assuming both images are of equal resolution, the panoramic image will have a wide field of view, thus showing many different buildings, but the panoramic image will have a low zoom level, so that specific details (such as windows) of many different buildings are not present. In contrast, the telephoto image will have a narrow field of view, so may only show a single building, but the telephoto image will be at a high zoom level, so that specific details (such as windows) of the single building are present.

[0006] In some cases, the user may wish to reprogram the field of view and / or zoom level after capture. For example, after taking a panoramic image, the user may wish to view the details of a particular building. Or, after taking a telephoto image, the user may wish to view adjacent buildings. Summary of the invention

[0007] A method for imaging processing is disclosed. The method may include: storing a first image of a scene captured by a first camera in response to a first command, the first image including first pixels and having a first field of view; storing a second image of the scene captured by a second camera in response to the first command, the second image including second pixels and having a second field of view; receiving a second command to change the field of view of the first image; editing the first image based on the second command; and storing the edited image, the edited image having a third field of view and including pixels based on the first pixels and the second pixels.

[0008] A processing system is disclosed, which may include a memory and one or more processors coupled to the memory. The one or more processors are configured to: store a first image of a scene captured by a first camera in response to a first command, the first image including first pixels and having a first field of view; store a second image of the scene captured by a second camera in response to the first command, the second image including second pixels and having a second field of view; receive a second command to change the field of view of the first image; edit the first image based on the second command; and store an edited image, the edited image having a third field of view and including pixels based on the first pixels and the second pixels.

[0009] A non-transitory computer-readable medium is disclosed. The medium may include program code that, when executed by one or more processors, causes the one or more processors to perform operations, the program code including code for: in response to a first command, storing a first image of a scene captured by a first camera, the first image including first pixels and having a first field of view; in response to the first command, storing a second image of the scene captured by a second camera, the second image including second pixels and having a second field of view; receiving a second command to change the field of view of the first image; editing the first image based on the second command; and storing the edited image, the edited image having a third field of view and including pixels based on the first pixels and the second pixels.

[0010] According to an embodiment of the present disclosure, a method for imaging processing is disclosed. The method may include: storing a first image of a scene captured by a first camera in response to a first command, the first image including a first pixel; storing a second image of the scene captured by a second camera in response to the first command, the second image including a second pixel; storing a first user-compiled image of the scene in response to the first command, the first user-compiled image including a first user-compiled pixel derived from at least one of the following: (a) the first pixel, and (b) the second pixel.

[0011] The method may include: receiving a second command to edit the first user edited image; editing the first user edited image; saving the edited first user edited image as a second user edited image.

[0012] The method may include saving the first user-compiled image and the second user-compiled image as metadata. After accessing the metadata, the first user-compiled image may be displayed and / or the second user-compiled image may be displayed.

[0013] Rendering the first user-crafted image may include re-deriving the first user-crafted pixels from the first and / or second pixels. Rendering the second user-crafted image may include re-deriving the second user-crafted pixels from the first and / or second pixels. The metadata may map the first user-crafted image to the first and / or second image. The metadata may map the second user-crafted image to the first image and / or the second image.

[0014] According to the method, the first user-compiled image and the second user-compiled image may have the same field of view and zoom level. In this case, the first user-compiled image is derived from the first image but not the second image. Due to the photo effects applied to the second user-compiled image, the second user-compiled image may be derived from both the first and second images.

[0015] The method may include presenting a thumbnail of the first user-edited image. The thumbnail may be a downsampled sample of the first user-edited image and appear automatically in response to the first command. The first command may consist of a single capture command. The thumbnail of the second user-edited image may automatically replace the thumbnail of the first user-edited image in response to the saving of the edited first user-edited image.

[0016] A processing system is disclosed, which may include one or more processors configured to perform the method. A non-transitory computer-readable medium is disclosed. The medium may include program code, which when executed by one or more processors causes the one or more processors to perform operations. The program code may include code for performing the method.

[0017] A processing system is disclosed. The processing system may include one or more processors configured to: store a first image of a scene captured by a first camera in response to a first command, the first image including a first pixel; store a second image of the scene captured by a second camera in response to the first command, the second image including a second pixel; store a first user-compiled image of the scene in response to the first command, the first user-compiled image including a first user-compiled pixel derived from at least one of: (a) the first pixel, and (b) the second pixel.

[0018] The one or more processors may be configured to: receive a second command to edit the first user compiled image; edit the first user compiled image; and save the edited first user compiled image as a second user compiled image.

[0019] A processing system is disclosed. The processing system may include: means for storing a first image of a scene captured by a first camera in response to a first command, the first image comprising a first pixel; means for storing a second image of the scene captured by a second camera in response to the first command, the second image comprising a second pixel; means for storing a first user-compiled image of the scene in response to the first command, the first user-compiled image comprising a first user-compiled pixel derived from at least one of: (a) the first pixel, and (b) the second pixel.

[0020] The processing system may further include: means for receiving a second command to edit the first user compiled image; means for editing the first user compiled image; means for saving the edited first user compiled image as a second user compiled image.

[0021] A non-transitory computer-readable medium is disclosed. The medium may include program code that, when executed by one or more processors, causes the one or more processors to perform operations. The program code may include code for: storing a first image of a scene captured by a first camera in response to a first command, the first image including a first pixel; storing a second image of the scene captured by a second camera in response to the first command, the second image including a second pixel; storing a first user-compiled image of the scene in response to the first command, the first user-compiled image including a first user-compiled pixel derived from at least one of: (a) the first pixel, and (b) the second pixel.

[0022] The program code may include code for: receiving a second command to edit the first user compiled image; editing the first user compiled image; saving the edited first user compiled image as a second user compiled image. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] For clarity and ease of reading, some of the drawings have omitted views of certain features. Unless explicitly stated otherwise, the drawings are not drawn to scale.

[0024] Figure 1 A schematic front plan view of a mobile device and a schematic illustration of a scene.

[0025] Figure 2 A schematic rear plan view of a mobile device.

[0026] Figure 3The invention comprises a lower part and an upper part. The lower part is a schematic top view of the first and second cameras of the mobile device. The upper part is a schematic front view of the first field of view of the first camera and the second field of view of the second camera.

[0027] Figure 4 A block diagram of an electronic file.

[0028] Figures 5 to 8 Compiles a view of the image for the user.

[0029] Fig. 9 A block diagram of the instance method.

[0030] Fig.10 is a schematic front plan view of a mobile device presenting thumbnail images.

[0031] Fig. 10A is a schematic front plan view of the mobile device and indicates the studio area with dotted lines.

[0032] Fig.11 A block diagram of the instance method.

[0033] Fig.12 is a schematic diagram of a first image according to a viewing angle of a first coordinate system.

[0034] Fig.13 is a schematic diagram of a second image according to a viewing angle of a second coordinate system.

[0035] Fig.14 Schematic diagram of the image compiled for the user according to the perspective of the global coordinate system.

[0036] Fig.15 A block diagram of the instance method.

[0037] Fig.16 A block diagram of an electronic file.

[0038] Fig.16A Display segmented user-compiled videos.

[0039] Fig.17 A block diagram of the processing system. DETAILED DESCRIPTION

[0040] Part 1

[0041] Although the features, methods, devices, and systems described herein may be embodied in various forms, some exemplary and non-limiting embodiments are shown in the drawings and described below. The features described herein may be optional. Implementations may include more, different, or fewer features than the examples discussed.

[0042] The subject matter is described using illustrative examples. The invention is not limited to these examples. Changes and modifications may be made to the invention without departing from its spirit. The claims are intended to cover such changes and modifications.

[0043] Part 2

[0044] Figure 1 and 2 The front and back of a mobile device 100 are shown, respectively, and may include a body or outer frame 101, a display 102, one or more physical buttons 103, and a camera group 104 including a first camera 105 and a second camera 106. Figure 2 The first camera 105 is shown directly to the side of the second camera 106, but this arrangement (as with all features disclosed in this application) is merely exemplary. The first camera 105 can have any position relative to the second camera 106. The first and second cameras 105, 106 can be disposed on separate mobile devices. The first and second cameras 105, 106 can be located on any part of the mobile device 100 (e.g., front, back, side).

[0045] The mobile device 100 may be configured to enter a viewfinder mode. When in viewfinder mode, the mobile device 100 may present virtual buttons 107-109 and a viewfinder area 110. Button 107 may control which camera group 104 is active (e.g., button "F" for the front-facing camera group 104 (not shown for clarity) and button "B" for the side-facing camera group 104 (not shown for clarity). Figure 2 The rear-facing selfie or rear selfie camera group 104 is shown.

[0046] Button 108, sometimes referred to herein as "camera button 108," can control whether viewfinder area 110 shows images captured by first camera 105 (button "1") or second camera 106 (button "2"). Camera button 108 can thus control which camera 105, 106 of camera group 104 is active, as discussed further below. Capture button 109 can cause mobile device 100 to capture one or more images of scene 10.

[0047] Although viewfinder area 110 is shown as occupying a portion of display 102, viewfinder area 110 may be coextensive with display 102. While in viewfinder mode, at least one camera of selected camera group 104 may continuously image (e.g., capture) scene 10. These images may be presented in viewfinder area 110 to provide a sample of the view of at least one camera. When the user presses capture button 109, mobile device 100 may cause camera group 104 to formally save one or more images of scene 10 in memory (e.g., non-volatile memory).

[0048] refer to Figures 1 to 3 , the first camera 105 may have a wider angle of view A11, thereby producing a wider field of view 11; and the second camera 106 may have a narrower angle of view A21, thereby producing a narrower field of view 21. Therefore, the first camera 105 may have a smaller optical zoom, while the second camera 106 may have a larger optical zoom. The fields of view 11, 21 may have any aspect ratio. The aspect ratios may be the same or different. Figure 1 The mobile device 100 is shown presenting the narrow field of view 21 of the second camera 106 in the viewfinder area 110 .

[0049] According to some examples, mobile device 10 may be configured such that wider field of view 11 completely encloses narrower field of view 21. First camera 105 may have a lower optical zoom than second camera 106. Although fields of view 11 and 21 have been described as "wider" and "narrower," fields of view 11 and 21 may have a variety of different relationships, as discussed below. According to other examples, fields of view 11, 21 may be the same size and different.

[0050] Figure 1 1 illustrates how an image of a scene 10 captured by a first camera 105 may differ from an image of the scene 10 captured by a second camera 106. With a wider field of view 11 of the first camera 105, the first camera 105 may simultaneously capture scene objects A, B, and C. Due to the lower optical zoom of the first camera 105, the first camera 105 may not be able to resolve some details of the scene objects A, B, and C. For example, if the scene object B is a tree, the first camera 105 may not be able to resolve the outlines of the individual leaves of the tree.

[0051] With the narrower field of view 21 of the second camera 106, the second camera 106 may capture a smaller two-dimensional area of ​​the scene 10 than the first camera 105. Figure 1 In the example of , the narrower field of view 21 includes scene object B, but not scene objects A or C. Due to the higher optical zoom of second camera 106, second camera 106 can resolve more details of the captured two-dimensional area than first camera 105. For example, and if scene object B is a tree, second camera 106 may be able to resolve the outlines of individual leaves.

[0052] According to some examples, the first camera 105 is equipped with a wide-angle lens, and the second camera 106 is equipped with a telephoto lens. However, this arrangement is merely exemplary. In other examples, both the first camera 105 and the second camera 106 are equipped with telephoto lenses, but the optical zoom of the first camera 105 is set to be lower than the optical zoom of the second camera 106. According to some examples, neither the first camera 105 nor the second camera 106 is equipped with a wide-angle or telephoto lens.

[0053] Because the first and second cameras 105, 106 each have unique advantages and disadvantages, the mobile device 100 may be configured to instruct both the first camera 105 and the second camera 106 to capture the scene 10. For example, when the user presses the capture button 109, the mobile device 100 may cause the first camera 105 to capture the field of view 11 as the first image 411, and cause the second camera 106 to simultaneously capture the field of view 21 as the second image 421. As used herein, "simultaneously" means "substantially simultaneously," and includes a situation where the first camera 105 captures the field of view 11 slightly before / slightly after the second camera 106 captures the field of view 21 (e.g., 0.5 seconds or less before / after the second camera 106 captures the field of view 21).

[0054] Part 3

[0055] Figure 4 A block diagram of a digital file 400 that may be prepared by the mobile device 100. The digital file 400 may include a first image 411 captured by the first camera 105, a second image 421 captured by the second camera 106, a user-compiled image 431, a displayed image 441, a calibration 451, a map 461, and a history 471.

[0056] The first image 411 may be a full resolution image captured by the first camera 105. The first image 411 may have a first field of view 412 equal to the field of view 11, and a first zoom level 423 corresponding to the optical zoom of the first camera 105. The first image 411 may apply a first coordinate system.

[0057] The second image 421 may be a full resolution image captured by the second camera 106. The second image 421 may have a second field of view 422 equal to the field of view 21, and a second zoom level 423 corresponding to the optical zoom of the second camera 106. The second image 421 may apply a second coordinate system.

[0058] The first field of view 412 may completely enclose the second field of view 422. Alternatively, the first field of view 412 may overlap only a portion of the second field of view 422. The first zoom level 423 may be lower than the second zoom level 423. According to some examples, the first and second images 411, 421 are pre-spatially and pre-photometrically aligned.

[0059] The user-compiled image 431 may be a sample of the first image 411, a sample of the second image 421, and a combination of a sample of the first image 411 and a sample of the second image 412. As used herein, the term "sample" conveys the source. For example, if the first image 411 is globally sampled 1:1, then the user-compiled image 431 may be equivalent to the first image 411.

[0060] The user-compiled image 431 may be derived from the first image 411 , derived from the second image 421 , or derived from both the first and second images 411 , 421 .

[0061] Image pixels may be derived from a sample. The derivation may take into account adjustments to the sample. For example, if one million pixels are sampled from the first image 411, then the one million sampled pixels may be adjusted (e.g., photometrically aligned) and inserted into the user compiled image 431. In this case, the user compiled image 431 will include pixels derived from the first image 411 (e.g., derived from a sample of the first image 411). In some instances, the number of derived pixels may be the same as the number of sampled pixels. The derivation may not include adjustments. Therefore, the derived pixels may be equal to the sampled pixels. In some instances, the number of derived pixels may be less than the number of sampled pixels.

[0062] The user-compiled image 431 may have a user-compiled field of view 432 and a user-compiled zoom level 433. The user-compiled field of view 432 may be equal to the first field of view 412, the second field of view 422, or any other user-selected field of view. The user-compiled zoom level 433 may be equal to the first zoom level 413, the second zoom level 423, or any other user-selected level. The user-compiled image 433 may apply the global coordinate system G discussed below.

[0063] The user compiled image 431 may reflect any photo effects selected by the user at the time of capture. Examples of photo effects include color shift, depth of field (e.g., bokeh effect), and averaging effect. For example, an averaging effect may average the color channel values ​​of all or a selected group of pixels in two different images (e.g., first image 411 and second image 421). A depth of field (e.g., bokeh) effect may depend on the color channel values ​​of pixels in first image 411 and pixels in second image 421.

[0064] The mobile device 100 may apply post-processing effects (e.g., re-framing) to the user-framing image 431 instead of the first image 411 and the second image 421. Thus, the mobile device 100 may preserve the original source data present in the first and second images 411, 421. The mobile device 100 may be configured to initially set the user-framing image 431 equal to the image captured by the active camera plus any photo effects selected by the user at the time of capture.

[0065] The first and second images 411 , 412 may have local first and second coordinate systems, respectively, discussed below. A global coordinate system G may be applied to compare the first image 411 , first field of view 412 , and first zoom level 413 with the second image 421 , second field of view 422 , and second zoom level 423 . Figure 2 and 3The vector d is shown separating the first camera 105 from the second camera 106. Thus, the transformation from the first coordinate system to the global coordinate system G may be non-linear, and the transformation from the second coordinate system to the global coordinate system G may be non-linear.

[0066] According to some examples, the global coordinate system G is the corrected coordinate system. When this application refers to accessing features of the first image 411 and the second image 421, such access may be performed via the global coordinate system G, the first coordinate system, or the second coordinate system.

[0067] For example, if the mobile device 100 accesses the first field of view 412, the mobile device 100 may access the first field of view 412 via the first coordinate system or the global coordinate system G. If the mobile device 100 accesses the second zoom level 423, such access may be via the first coordinate system or the global coordinate system G. If the mobile device 100 derives pixels from the first image 411, these pixels may be derived via the global coordinate system G.

[0068] The global coordinate system G may be equivalent to the first coordinate system or the second coordinate system. Alternatively, the global coordinate system G may be transformed (linearly or nonlinearly) to the first coordinate system, and transformed (linearly or nonlinearly) to the second coordinate system.

[0069] The displayed image 441 may be a view presented on the mobile device display 102. The displayed image 441 may rely on pixels derived from a source, such as the first image 411, the second image 421, the user-compiled image 431. The displayed image 441 may have a displayed field of view 442 and a displayed zoom level 443. The displayed image 441 may apply a global coordinate system G.

[0070] Because displayed image 441 may have a predetermined resolution equal to studio area 601 (discussed below), displayed zoom level 443 may be a function of displayed field of view 442 and the predetermined resolution.

[0071] The map 461 may enable the mobile device 100 to transform features of the first image 411 and features of the second image 421 to the global coordinate system G. The mobile device 100 may generate the map 461 using the first image 411 , the second image 421 , and the calibration 451 .

[0072] Part 4

[0073] Figures 5 to 8 Depicts an example reprogramming of a user-programmed image 431. The reprogrammed user-programmed image 431 is also referred to as a reprogrammed image 431. Figure 5As shown, the first user programmed image 431a is equal to the second image 421 , with the user programmed field of view 432 being equal to the second field of view 422 , and the user programmed zoom level 433 being equal to the second zoom level 423 .

[0074] Figure 6 A second user-compiled image 431b (i.e., the first recompiled image) is shown. Region 601 overlaps with the first field of view 412, but does not overlap with the second field of view 422. Therefore, the mobile device 100 fills the region 601 with pixels derived from the first image 411. If the user-compiled zoom level 433 exceeds the first zoom level 413, the first image 411 may not be able to supply enough pixels to fill the region 601. In this case, the mobile device 100 may interpolate to fill the remaining pixels. The interpolation may rely on pixels derived from the first image 411 and pixels derived from the second image 421.

[0075] The region 602 intersects (i.e., overlaps) both the first and second fields of view 412, 422. Therefore, the mobile device 100 may fill the region 602 with pixels derived from the first image 411 or the second image 421 depending on the user-programmed zoom level 433. In order to enable the region 602 to have the same field of view and zoom level as the first user-programmed image 431a, the mobile device 100 may enable the aspect ratio of the user-programmed image 433 to be adjusted dynamically.

[0076] Figure 7 A third user compiled image 431c is shown (ie, a twice recompiled image). The user compiled zoom level 433 exceeds the first zoom level 413, while the user compiled field of view 432 is equal to the first field of view 412. Therefore, the third user compiled image 431c has a greater resolution than the first image 411.

[0077] Region 701 overlaps with first field of view 412, but does not overlap with second field of view 422. Therefore, mobile device 100 samples pixels for region 701 from first image 411. As stated above, as used herein, the term sampling may include 1:1 sampling. Therefore, mobile device 100 may fill region 701 with all pixels derived from a block of first image 411.

[0078] Region 702 overlaps the first and second fields of view 412, 422. Mobile device 100 may supply pixels to region 702 by sampling first image 411 or second image 421. Mobile device 100 may select between sampling first image 411 or second image 421 by comparing user programmed zoom level 433 to first zoom level 413 and second zoom level 423. To fill any unfilled pixels of region 701, mobile device 100 may interpolate pixels derived from first image 411 and second image 421.

[0079] refer to Figure 8 , the fourth user-compiled image 431d includes area 801 but does not include area 802. The user-compiled zoom level 433 is equal to the first zoom level 413. Therefore, the mobile device 100 can fill area 801 with pixels (e.g., 1:1 derived pixels) from a block of the first image 411. The user-compiled field of view 432 is different from the first field of view 412, resulting in an adjusted aspect ratio. Figure 8 Further shown is a scene object D (omitted from other views), which has been blurred by a depth of field (eg, bokeh) photographic effect.

[0080] Because the original data of the first and second images 411, 421 can be preserved during reprogramming, the mobile device 100 enables reprogramming in any order between the first to fourth user-programmed images 431a to 431d. For example, the user can start at the fourth user-programmed image 431d, reprogram to the second user-programmed image 431b, reprogram to the first user-programmed image 431a, and then end with the third user-programmed image 431c.

[0081] To reprogram the user program image 431, the mobile device 100 may begin by accessing the user program image 431 and presenting the displayed image 441 based on the accessed user program image 431. The displayed image 441 may be derived from (eg, sampled from) the user program image 431.

[0082] The user may manipulate the displayed image 441 (e.g., pan, zoom, apply photo effects). As the user manipulates the displayed image 441, the mobile device 100 may re-visualize the displayed image 441 based on the first image 411 and the second image 421. After the user has finished manipulating the displayed image 441, the mobile device 100 may re-compile the user-compiled image 431 based on the net effect of the manipulation. The mobile device 100 may set the user-compiled image 431 equal to the re-compiled image. However, the mobile device 100 may retain the original user-compiled image (e.g., as metadata or as a traditional image file (e.g., JPEG)).

[0083] Thus, when the user re-accesses the user edited image 431, the mobile device 100 may default to presenting the displayed image 441 derived from the re-edited user edited image, rather than the original user edited image. However, the mobile device 100 may enable the user to revert to the original user edited image.

[0084] Thus, the example of mobile device 100 enables dynamic reprogramming of user-programmed image 431 without requiring the user to formally access or view first image 411 or second image 421. This is because any data required by mobile device 100 to extract first image 411 and / or second image 421 can be automatically accessed and exported behind the scenes.

[0085] In addition, the example of the mobile device 100 enables the user to reprogram using the user program image 431 as a starting point. As a first example, and with reference to Figure 6 , it is contemplated that the user intended the user compiled image 431 to include all of the scene objects A, but made an error that resulted in the second user compiled image 431b. Instead of starting from scratch (e.g., starting at the first user compiled image 431a), the user may open the second user compiled image 431b and recompile from the initial conditions of the second user compiled image 431b.

[0086] As a second example, and referring to Figure 8 , it is contemplated that the user intended for the user compiled image 431 to lack any photo effects, but made the mistake of producing the fourth user compiled image 431d. Instead of starting from scratch (e.g., starting at the first user compiled image 431a, recompiled the field of view to match the fourth user compiled image 431d, and recompiled the zoom level to match the fourth user compiled image 431d), the user may start at the fourth user compiled image 431d and instruct the mobile device 100 to remove the depth of field effect.

[0087] In response, the mobile device 100 may maintain the field of view and zoom level of the fourth user compiled image 431 d , derive pixels from the first and / or second images 411 , 421 based on the maintained field of view and zoom level, and fill the recompiled image with the derived pixels.

[0088] Part 5

[0089] According to some examples, the mobile device 100 is based on Fig. 9 The presented series of initiation operations (also referred to as a compilation method or initiation method) selects pixel sources for a user to compile an image.

[0090] At block 902, the mobile device 100 may compare the user-programmed field of view 432 to the first field of view 412 and the second field of view 422. At block 904, the mobile device 100 may segment the user-programmed field of view 432 into first and second regions. The first region (e.g., Figure 7 The second region overlaps the first field of view 412 and the second field of view 422 (e.g., region 701). Figure 7 area 702).

[0091] At block 906, the mobile device 100 may select a sampling density for sampling a block of the first field of view 412 corresponding to the first region. The mobile device 100 may select the sampling density by comparing the first zoom level 413 to the user programmed zoom level 433.

[0092] If the first zoom level 413 ≤ the user programmed zoom level 433, the mobile device 100 may set the sampling density to 1:1 so that every pixel in the block of the first image 411 is sampled. Otherwise, and according to some examples, the mobile device 100 only samples some pixels in the block of the first image 411. Sampling at a ratio less than 1:1 is referred to as downsampling or downscaling. At block 908, the mobile device 100 samples the first image 411 and fills the first region with pixels derived from the samples.

[0093] At block 910, the mobile device 100 may compare the user-selected zoom level 433 to a predetermined and user-selectable value K, where the first zoom level 413≤K≤the second zoom level 423. If the user selected the zoom level 433≤K, the mobile device 100 may select a sampling density for a block of the first field of view 412 corresponding to the second region at block 912. If the user selected the zoom level 433>K, the mobile device 100 may select a sampling density for a block of the second field of view 422 corresponding to the second region at block 916.

[0094] At block 912, the mobile device 100 may select a sampling density by comparing the first zoom level 413 to the user-programmed zoom level 433. If the first zoom level 413 ≤ the user-programmed zoom level 433, the mobile device 100 may set the sampling density to 1:1 so that every pixel in the block of the first image 411 is sampled. Otherwise, the mobile device 100 may only sample some pixels in the block of the first image 411. At block 914, the mobile device 100 may sample the first image 411 and fill the second region with pixels derived from the samples.

[0095] At block 916, the mobile device 100 may select a sampling density by comparing the second zoom level 423 to the user-programmed zoom level 433. If the first zoom level 423 ≤ the user-programmed zoom level 433, the mobile device 100 may set the sampling density to 1:1 so that every pixel in the block of the second image 421 is sampled. Otherwise, the mobile device 100 may only sample some pixels in the block of the first image 421. At block 914, the mobile device 100 may sample the second image 421 and fill the second region with samples.

[0096] If the user compiled image 431 is missing pixels after completion of blocks 908, 914, and 918 (performed as appropriate), the values ​​of the missing pixels may be interpolated from the filled pixels.

[0097] Part 6

[0098] The mobile device 100 may provide a gallery mode. After the gallery mode is turned on, and Fig.10 As shown, the mobile device 100 can present a plurality of image thumbnails 1001 to the user. Each thumbnail 1001 can compile a preview (eg, a sample) of the image 431 for a different user.

[0099] When the user selects (e.g., clicks) thumbnail 1001, mobile device 100 may access the image associated with the thumbnail and enter studio mode. During studio mode, the accessed image or a downsampled sample thereof may be presented in studio area 1010 (which may be coextensive with viewfinder area 110).

[0100] Studio mode may include a plurality of user-selectable sub-modes, including: (a) reprogramming viewer mode, (b) non-reprogramming viewer mode (also referred to as anchor mode), (c) user-programmed field of view reprogramming mode, (d) user-programmed zoom reprogramming mode, and (e) photo effects reprogramming mode. Mobile device 100 may enable the user to dynamically switch between modes via one or more virtual buttons.

[0101] Reframing may adjust the user-framing image 431 (as discussed above) and / or the displayed image 441. Reframing may be the result of user interaction (eg, zooming in, zooming out, panning, applying photo effects). Reframing may include framing.

[0102] During the reorganization period, and as referred to above Fig. 9 As discussed, pixels derived from the first image 411 may be reorganized using pixels derived from the second image 421. If the mobile device 100 is touch enabled, the reorganization may be touch responsive (e.g., pinch to zoom out, separate two fingers to zoom in, swipe a single finger to pan). If the mobile device 100 is non-touch enabled, the reorganization may be performed using a suitable input device (e.g., mouse, keyboard).

[0103] Initially, the user compiled image 431 may be equal to the image captured by the active camera. As discussed above, the user may apply the camera button 108 to designate the first camera 105 or the second camera 106 as "active", and the preview in the viewfinder area 110 may be initiated by the active camera. Thus, and according to this example, if the user presses the capture button 109 while the second camera 106 is designated as active, the user compiled image 431 will initially be set equal to the second image 421.

[0104] Upon entering the studio mode, the mobile device 100 may automatically present the displayed image 441 with the displayed field of view 442 being equal to the user programmed field of view 432. The mobile device 100 may automatically resize the studio area 1010 to accommodate the aspect ratio of the displayed field of view 442.

[0105] Section 6A

[0106] Displayed image 441 may begin with displayed field of view 442 equal to user compiled field of view 443. During the reformatting viewer mode, pixels of displayed image 441 may be sampled directly from first image 411 and second image 421 without reference to user compiled image 431. According to some examples, during the reformatting viewer mode, first image 411, second image 421, or user compiled image 431 may not change.

[0107] As the user pans or zooms, reprogramming the viewer mode can be applied Fig. 9 The above pixel initiation (including interpolation when necessary) is performed, except that the displayed zoom level 443 is applied instead of the user-programmed zoom level 433.

[0108] Section 6B

[0109] Anchor mode enables a user to interact with a particular image (e.g., pan, zoom) without automatic re-framing. During anchor mode, a user may select a user-framing image 431, a first image 411, or a second image 421. Anchor mode may default to the user-framing image 431.

[0110] According to some examples, anchor mode causes mobile device 100 to supply displayed image 441 having pixels sampled from only a single image, such as first image 411, second image 421, or user-compiled image 431. According to some examples, during anchor mode, first image 411, second image 421, or user-compiled image 431 does not change.

[0111] When in anchor mode, the mobile device 100 may compare the displayed zoom level 443 with the source image zoom level. The comparison may determine whether the sampling involves downsampling or 1:1 sampling. If the displayed zoom level 443 exceeds the source image zoom level, the mobile device 100 may interpolate the sampled pixels.

[0112] Section 6C

[0113] According to some examples, field of view recomposition (FOVR) mode enables a user to adjust a user-compiled field of view 432 without affecting a user-compiled zoom level 433. While in FOVR mode, a displayed image 441 may be initiated and presented according to the techniques discussed for recompilation viewer mode.

[0114] While in FOVR mode, the user may click a virtual button that automatically sets the user-compiled field of view 432 equal to the displayed field of view 442. Alternatively, the user may draw a geometric shape (e.g., a rectangle) around the desired portion of the image. Thereafter, the mobile device 100 may set the user-compiled field of view 432 equal to the portion of the displayed field of view 442 disposed in the geometric shape. The mobile device 100 may constrain the geometric shape to a rectangle (e.g., a square).

[0115] Section 6D

[0116] According to some examples, a level of zoom recomposition (LOZR) mode enables a user to adjust a user-compiled zoom level 433 without affecting the user-compiled field of view 432. While in LOZR mode, a displayed image 441 may be initiated and presented according to the techniques discussed for the recompiled viewer mode. The mobile device 100 may be configured to prevent the user from panning outside the current user-compiled field of view 432 during LOZR mode.

[0117] To set the user programmed zoom level 433, the user may click a virtual button that automatically sets the user programmed zoom level 433 equal to the displayed zoom level 443. Alternatively, the user may enter a value. If the user chooses to enter a value, the mobile device 100 may present an input window that lists both the first and second zoom levels 413, 423, along with the current user programmed zoom level 433.

[0118] Section 6E

[0119] When recompiled, the user may apply one or more photo effects to the user compiled image 431. One example of a photo effect is a depth of field (e.g., bokeh) effect. To simulate the depth of field effect, the mobile device 100 may apply the first and second images 411, 421 to construct a stereo depth map. Another example is a color shift effect (e.g., increasing the red value of each pixel by 10%). Another example is an averaging effect, which averages the color channels of the pixels across the two images.

[0120] After the user selects an item (eg, B) present in the user compiled image 431 , the mobile device 100 may estimate a depth range of the selected item and blur items outside the estimated depth range.

[0121] When the user selects an item (e.g., C) outside the second field of view 422, the mobile device 100 may apply one-dimensional and non-stereoscopic techniques to construct a depth map and estimate the depth range. Thus, the mobile device 100 may apply a first depth of field technique for items captured in both the first and second images 411, 421, and may apply a second depth of field technique for items captured only in the first image 411.

[0122] The photo effects reprogramming mode enables the user to remove previously applied photo effects. Figure 8 , the user may wish to remove the applied depth of field (eg, bokeh) effect that causes distortion of the scene object D. To this end, the mobile device 100 may regenerate the user-compiled image 431 .

[0123] More specifically, the mobile device 100 may access the user-programmed field of view 432 and the user-programmed zoom level 433, and then regenerate the user-programmed image 431 using pixels derived from the first image 411 and the second image 412. The regeneration may rely on the reference Fig. 9 The initiation operations discussed (including interpolation).

[0124] According to this example, the first and second images 411, 412 lack photo effects. Therefore, the regenerated user-compiled image 431 will not include the undesirable depth of field effect, but will maintain the desired user-compiled field of view 432 and zoom level 433. After removing the undesirable photo effects from the user-compiled image 431, the user can apply new and different photo effects.

[0125] After the user has applied a photo effect to the user compiled image 431, other modes may continue to apply the photo effect. For example, during the recompile viewer mode, the photo effect may be maintained when presenting various displayed images 441. Techniques for maintaining one or more applied photo effects are discussed further below.

[0126] Part 7

[0127] Fig.11 An example method of constructing and re-compiling a user-compiling image 431 is shown. The method includes a series of operations. The mobile device 100 may be configured to perform some or all of these operations.

[0128] At block 1102, the mobile device 100 enters a viewfinder mode. At block 1104, the user selects a camera group 104. At block 1106, the user selects an active camera from the selected camera group 104. At block 1108, the mobile device 100 presents the view from the active camera in the viewfinder area 110. To do this, the mobile device 100 may capture images with the active camera (rather than the inactive cameras) and present the images in the viewfinder area 110. After presentation, the mobile device 100 may discard these captured images.

[0129] At block 1110, the user may press the capture button 109. In response, the mobile device 100 may simultaneously (e.g., substantially simultaneously, as discussed above) utilize both the first and second cameras 105, 106 (i.e., the active camera and the inactive camera) to capture the scene 10. At block 1112, the mobile device 100 may load the calibration 451. At block 1114, the mobile device 100 may set the user compiled image 431 equal to the image taken by the active camera.

[0130] At block 1116, the mobile device 100 may combine the first image 411, the second image 421, the user-composed image 431, the calibration 451, and the history 471 (discussed further below) into a package. The package may be a single discrete file. The package may be a zip file, a JPEG container with a media picture object extension, and the like. The calibration 451, the map 461, and the history 471 may be included in the package as, for example, a JPEG application header or image metadata. The history 471 may include a first entry (e.g., entry one: user-composed image 431 = first image 411) that identifies the user-composed image 431.

[0131] At block 1118, the mobile device 100 may add the thumbnail 1001 to the image gallery. The thumbnail 1001 may be a preview (e.g., sample, crop, export) of the user compiled image 431. At block 1120, the user may click (e.g., tap) the thumbnail 1001. At block 1122, the mobile device 100 may present the displayed image 441, where the displayed field of view 442 is equal to the user compiled field of view 432. Block 1122 may default to a non-recompiled viewer mode. Block 1122 may include Fig.15 operation, as discussed further below.

[0132] At block 1124, the mobile device 100 may determine whether the package includes or lacks a map 461. If the package lacks a map 461, the mobile device 100 may perform blocks 1124 and 1126. If the package includes a map 461, the mobile device 100 may jump to block 1128.

[0133] At block 1124, which may occur in parallel with block 1122, the mobile device 100 may generate a map 461 based on the first image 411, the second image 421, and the calibration 451. The map 461, discussed further below, may include one or more maps (e.g., transformations) that relate pixels in the first image 411 and pixels in the second image 421 to a global coordinate system. The map 461 may be used for spatial alignment and / or photometric (e.g., color, intensity) alignment. At block 1126, the mobile device 100 may add the map 461 to a package. The package may be a zip file, a JPEG container with a media picture object extension, and the like.

[0134] At block 1128, the user may re-edit the user-edited image 431 and / or the displayed image 441 using any of the techniques discussed in this application, including Fig. 9 Thus, at block 1128, the user may switch from the non-reprogramming viewer mode to any of the other modes (e.g., reprogramming viewer mode, field of view reprogramming mode, zoom level reprogramming mode, photo effects reprogramming mode).

[0135] At block 1130, the user may instruct the mobile device (eg, via a virtual button) to save the reprogrammed user program image 431. At block 1132, the mobile device 100 may add an entry to the history 471 (discussed below).

[0136] At block 1134, the mobile device 100 may rewrite the user edited image 431 in the packaging to reflect the re-edited user edited image 431. At block 1136, the mobile device 100 may update the thumbnail to reflect the re-edited user edited image 431 (eg, sampling the re-edited user edited image 431).

[0137] After block 1136, and if the user exits gallery mode, mobile device 100 may start at block 1120. If the user remains in gallery mode, mobile device 100 may start at block 1122. According to some examples, thumbnails 1001 are not updated until the user exits gallery mode.

[0138] Part 8

[0139] Calibration 451 may include some or all of the following: (a) a vector d between the first camera 105 and the second camera 106 (see Figure 3 ), (b) the auto focus position of the first camera 105 (if the first camera 105 is capable of auto focus), (c) the auto focus position of the second camera 106 (if the second camera 106 is capable of auto focus), and any other data related to aligning the first image 411 and the second image 421.

[0140] Techniques for aligning images and intersecting fields of view are known in the art. Examples of these techniques are disclosed in U.S. Publication No. 2017 / 0024846 to Nash et al., among others. According to some examples, pixels having key items (e.g., edges, brightness, color) in the untransformed first image 411 are matched with pixels having corresponding key items in the untransformed second image 421.

[0141] After these pixels have been matched, the mobile device 100 can interpolate (i.e., estimate) to determine how pixels with non-critical items should be matched. A range of known techniques can be used to identify critical items and corresponding critical items, including sum of squared differences techniques, Laplacian of Gaussian techniques, Hessian determinant techniques, and the like.

[0142] According to some examples, same channel and cross-channel horizontal and vertical color gradients are calculated for each pixel in the untransformed first image 411 and the untransformed second image 412. A weighted average of (a) the same channel and cross-channel horizontal color gradients, and (b) the same channel and cross-channel vertical color gradients is calculated for each pixel.

[0143] The weighted average of each pixel represents the edge direction for each pixel. The pixels in the untransformed second image 412 are grouped into small clusters, and the average edge direction for each small cluster is determined. The average edge direction for each small cluster of the second image 412 is then compared with the edge direction for each pixel in the selected central region of the first image 411. According to one example of the sum of squared differences technique, the clusters of the untransformed first image 411 and the selected central region are iterated until the sum of squared differences drops below a predetermined threshold.

[0144] Using this technique (or another technique), the mobile device 100 may determine which pixels in the untransformed first image 411 intersect with the second field of view 422. Each of these intersecting (also referred to as overlapping) pixels may then be mapped to a cluster of one or more pixels in the second image 421. Following this action, the mobile device 100 may prepare one or more global coordinate systems G, along with one or more transformation functions G(x) and G-1(x) that convert between the global coordinate system G and the first and second images 411, 421. According to some examples, the global coordinate system G is set to the coordinate system of the first image 411 (i.e., the first coordinate system).

[0145] The mapping 461 may include one or more transformations that spatially align the first coordinate system, the second coordinate system, and / or the global coordinate system G via one or more translations, scalings, rotations, and / or deformations. The mapping 461 may further include one or more transformations that photometrically align the untransformed first image 411 relative to the second image 421 and / or the global coordinate system G (e.g., by equalizing color, intensity, and contrast).

[0146] Fig.12 The first coordinate system is applied, and Fig.13 A second coordinate system is applied. Therefore, reference Fig.12 and 13 The characteristics discussed (eg, field of view, zoom level) may be untransformed (ie, not expressed in terms of the viewing angle of a global coordinate system G).

[0147] Fig.12 The first image 411 is shown as a matrix of first pixels 1201. Each first pixel 1201 may have a three-dimensional color vector (e.g., a red value, a green value, and a blue value). For clarity, only some of the first pixels 1201 are depicted, particularly those in first image pixel clusters 1211a, 1211b, and 1211c. The first pixels 1201 in first image pixel clusters 1211a and 1211b are outside the second field of view 422, and therefore do not intersect with the second pixel 1301 in the second image 421. The first pixel 1201 in first image pixel cluster 1211c is inside the third field of view 422, and therefore does intersect with the second pixel 1301 in the second image 421.

[0148] Fig.13 The second image 421 is shown as a matrix of second pixels 1301. Each second pixel 1301 may have a three-dimensional color vector, as described above. For clarity, only some of the second pixels 1301 are depicted, in particular those in the second image pixel clusters 1311a, 1311b.

[0149] As previously discussed, the map 461 can relate a cluster of one or more second pixels 1301 to each intersecting first pixel 1201, such as each first pixel 1201 disposed within the second field of view 422. According to this example, the second pixel cluster 1311a maps to the first pixel 1201a, and the second pixel cluster 1311b maps to the first pixel 1201b. The map 461 thus indicates that the field of view of the first pixel 1201a is coextensive with the second pixel cluster 1311a.

[0150] Because the second zoom level 423 exceeds the first zoom level 413, the field of view is mapped to more second pixels 1301 than first pixels 1201. Fig.13As shown, the second pixel clusters 1311a, 1311b may be non-rectangular. In fact, the second pixel clusters 1311a, 1311b may have any shape as determined by the alignment process.

[0151] First pixels 1201 outside the second field of view (eg, first pixels 1201 in first clusters 1211a and 1211b) may be marked as non-intersecting.Each intersecting first pixel 1201 may be linked to a two-dimensional region corresponding to an associated second pixel cluster.

[0152] For example, the first pixel 1201a has first coordinates (m, n). The first pixel 1201 can be mapped to a shape drawn around the perimeter of the second pixel group 1311a. Here, the perimeter will be represented by the following segments (e.g., vectors) expressed in the second coordinates: (0,0)-(2,0); (2,0)-(2,2); (2,2)-(1,2); (1,2)-(0,3); (0,3)-(0,0).

[0153] Mapping 461 enables programming of first pixel 1201 with second pixel 1301 during the programming process. According to some examples, mobile device 100 generates user-programmed image 431 and displayed image 441 based on mapping 461 .

[0154] For example, and refer to Fig.12 , the first displayed field of view 442a can be expressed in the first coordinate system as: (0,0)-(m-1,0); (m-1,0)-(m-1,n-1); (m-1,n-1)-(0,n-1); (0,n-1)-(0,0). After determining how the first displayed field of view 442a is mapped to the first image 411, the mobile device 100 can determine whether any of the first pixels 1201 in the first displayed field of view 442a intersect (i.e., overlap). The mobile device 100 can apply this information to visualize the displayed image 441, as shown in FIG. Fig. 9 Discussed.

[0155] Fig.12 A second displayed field of view 442b is shown, which now includes the first pixel cluster 1211c. Relative to the second displayed field of view 442b, only the first pixels 1201 in the first pixel cluster 1211c overlap. Thus, the second displayed field of view 442b can be divided into two parts: a first part covering the non-overlapping first pixels 1201, and a second part covering the overlapping first pixels 1211c.

[0156] The mobile device 100 may use this information to present the displayed image 441, as shown in FIG. Fig. 9If sampling from the second image 421, the mobile device 100 may apply the map 461 to determine which second pixels 1301 correspond to the intersecting portions of the second displayed field of view 442b.

[0157] As the user reprograms the user-programmed image 431 and / or the displayed image 441, the mobile device 100 may reprogram the mapping. For example, it is contemplated that the user wishes to change from the second user-programmed image 431b ( Figure 6 ) is reprogrammed into the first user-programmed image 431a ( Figure 5 After accessing the third user-edited image 431 c , the mobile device 100 may map the second user-edited image 431 b to the first and second images 411 , 421 .

[0158] More specifically, the mobile device 100 may map the second user-programmed field of view 432b to the first and second images 411 and 421, and then apply Fig. 9 The initiation technique is to fill the second user-edited field of view 432b with pixels derived from the first and second images 411, 421. When the user stays in the first field of view 432a of the first user-edited image 431a, the mobile device 100 can re-edit the mapping and then re-apply Fig. 9 The initiation technique uses pixels derived from the first and second images 411 and 421 to fill the first user-programmed field of view 431a.

[0159] Instead of determining how the first user compiled field of view 432 is mapped from the draft to the first and second images 411, 421, the mobile device 100 may record the user manipulations that produced the first user compiled field of view 432a, net the user manipulations, and set the first user compiled field of view equal to the second user compiled field of view 432b plus the net manipulations.

[0160] For example, it is contemplated that the first user compiled field of view 431a represents a translation to the right (e.g., units of the global coordinate system G) of one thousand units from the second user compiled field of view 432a. If the user translates two thousand units to the right, then translates one thousand units to the left, the net manipulation will be a translation to the right of one thousand units. The mobile device 100 may then calculate the first user compiled field of view 432a by adding one thousand units to the X-axis coordinate of the second user compiled field of view 432b.

[0161] Part 9

[0162] As previously discussed, the first and second images 411, 421 may each be stored as a matrix, wherein each cell in the matrix comprises a three-dimensional color vector. The matrix may further be stored in a compressed form via lossy or lossless compression. Even in the case of compression, the matrix may still consume a large amount of memory.

[0163] Map 461 enables user-compiled image 441 to be stored losslessly as a reference matrix. When stored in this manner, user-compiled image 441 may have the same post-rendering resolution as first image 411 and / or second image 412, but consumes less than 1 / 10 of the storage space (and in some cases less than 1 / 100 of the storage space).

[0164] After accessing the reference matrix, the mobile device 100 can dynamically calculate the color vector of each pixel by resampling the first image 411 and the second image 421. Thereafter, the mobile device 100 can create a temporary copy (e.g., on a volatile memory) of the user-compiled image 441. After the user exits the gallery mode, the mobile device 100 can clear the temporary copy from the memory.

[0165] Fig.14 An example user-compiled image 431x is shown that is temporarily stored as a two-dimensional matrix of three-dimensional vectors (i.e., in non-reference matrix form). The user-compiled image 431x has a user-compiled field of view 432x and a user-compiled zoom level 433x. The user-compiled field of view 432x includes a first area 1401 and a second area 1402. A boundary 1403 separates the two areas.

[0166] The second region 1402 intersects both the first field of view 412 and the second field of view 422. The first region 1401 intersects only the first field of view 412. Thus, the mobile device 100 has pixels in the first region 1401 that are derived only from the first image 411. The mobile device 100 has applied Fig. 9 The above-mentioned initiation operation is used to determine whether the pixels in the second area 1402 should be initiated from the first image 411 or the second image 421.

[0167] exist Fig.14 In the example of FIG. 4 , the user programmed zoom level 433x is the second zoom level 423. Thus, the pixels in the second area 1402 have been derived from the second image 421. The black dots indicate the derived pixels. Because the user programmed zoom level 433x exceeds the first zoom level 413, only some of the pixels in the first area 1401 are derived from the first image 411. These pixels are dotted.

[0168] The remaining undotted pixels in the first region 1401 are interpolated based on the color values ​​of the neighboring pixels (see reference Fig. 9 For example, the color vector of pixel 1404 may have been interpolated based on the color vectors in pixels 1405, 1406, and 1407. The color vector of pixel 1408 may have been interpolated based on the color vectors in pixels 1409-1412.

[0169] The user-generated image 431 may be stored as a reference matrix (e.g., metadata). The reference matrix may include: (a) global coordinates of the user-generated field of view; (b) values ​​of the user-generated zoom level; (c) identities and coordinates of post-processing effects; (d) identities of specific global coordinate systems, since the map 461 may include multiple global coordinate systems; (e) identities of the first image 411; and (f) identities of the second image 421.

[0170] Because the mobile device 100 may apply the first and second images 411 , 421 to re-appear the user-compiled image 431 , the reference matrix may represent the superposition of the first image 411 and the second image 421 , plus any post-processing effects.

[0171] Assume that G(x, y) is a nonlinear transformation from a unique coordinate system y (e.g., the first or second coordinate system) to a global coordinate system G. G(x, y) may include a correction. Therefore, G(first image 411, first coordinate system) may perform a shift, a warp, and / or a shift to place the first image 411 in the global coordinate system G. Similarly, G(second image 421, second coordinate system) may perform a shift, a warp, and / or a shift to place the first image 421 in the global coordinate system G.

[0172] According to some examples, the coordinate system of the first image 411 (ie, the first coordinate system) is used as the global coordinate system such that: G(first image 411, first coordinate system)=first image 411 in the first coordinate system.

[0173] The user-generated view 432 may be stored according to a global coordinate system. Assume that the matrix M is a matrix of coordinates (eg, points or vectors) that express the user-generated view 432 according to a global coordinate system: According to this example, each (x, y) coordinate pair represents a corner of the user-programmed field of view 432. According to another example, each coordinate pair may represent a two-dimensional vector marking the outer boundary of the user-programmed field of view 432. Fig.14 In the example of

[0174] Assume G -1 (x, y) is a nonlinear transformation from the global coordinate system to a unique coordinate system y (e.g., the first or second coordinate system). -1 (M, first coordinate system) transforms the user-programmed field of view 432 into the first coordinate system. Similarly, G -1 (M, second coordinate system) transforms the user-generated view 432 into the second coordinate system. The map 461 may include G and G -1 Both.

[0175] Thus, the mobile device 100 can re-render any particular user-edited image 431 from the reference matrix RM: RM=(M, user-edited zoom level 433, photo effect). As discussed above, the reference can further include an identity of the global coordinate system, an identity (e.g., file name) of the first image 411, and an identity (e.g., file name) of the second image.

[0176] Fig.15 is an exemplary block diagram of a series of operations (eg, a method) for applying a reference matrix. Fig.15 The operation can be Fig.11 occurs during the frame.

[0177] Blocks 1502 to 1512 may be Fig.11 At block 1502, the mobile device 100 may access a reference matrix of the user-compiled image 431. After accessing the reference matrix, the mobile device 100 may inversely transform the coordinate M to the first coordinate system and the second coordinate system at block 1504. -1 (M, first coordinate system) Place the user-programmed field of view 432 in the first coordinate system. -1 (M, second coordinate system) Place the user-programmed field of view 432 in the second coordinate system.

[0178] exist Fig.14 In the example of G, the first region 1401 is outside the second field of view 422. Therefore, G -1 (Area 1401, second coordinate system) may return a null value, while G -1 (Region 1402, second coordinate system)Region 1402 is expressed according to the second coordinate system.

[0179] At block 1506, the mobile device 100 may apply Fig. 9 At block 1508, the mobile device 100 may apply the photo effects listed in the reference matrix RM to generate the final stage of the user compiled image 431. Each of the listed photo effects may be a matrix that lists any initial conditions required for applying the photo effect (e.g., the type of photo effect, the two-dimensional coordinates of the photo effect, the degree of the photo effect).

[0180] At block 1510, the mobile device 100 may temporarily save a rendered copy of the user-compiled image 431 in a memory (e.g., volatile memory) as a two-dimensional matrix of three-dimensional color vectors. At block 1512, the mobile device 100 may present a displayed image 441 based on the temporarily saved user-compiled image 431 using the techniques described above.

[0181] refer to Fig.11, and at block 1128, the mobile device 100 may reprogram the user-programmed image 431 and / or the displayed image 441 depending on the selected mode. To present the displayed image 441, the mobile device 100 may rely on the temporarily saved user-programmed image 431. At block 1134, the mobile device 100 may erase the temporarily stored user-programmed image 431. When rewriting the user-programmed image 431x, the mobile device 100 may generate a new reference matrix.

[0182] According to some examples, mobile device 100 may retain multiple reference matrices and therefore not necessarily perform a rewrite at block 1134. Referring to the following table, each recompilation of user compiled image 431 may be stored as a new entry in history 471 (other possible features of each reference matrix, such as the identity of global coordinate system G, are not shown):

[0183]

[0184] Thus, instead of storing the user-edited image 431 independently of the history 471, as Figure 4 As shown, the user-compiled image 431 may be stored as the most recent entry (in this case, entry number n) in the history 471. The mobile device 100 may include an option to present the history 471 and enable the user to select any entry number to restore the user-compiled image 431 to a previous layout. The mobile device 100 may include an option to display any entry in the history 471 as a two-dimensional matrix with cells filled using a three-dimensional color vector.

[0185] Mobile device 100 may include an API that enables user compilation image 431 to be accessed by other programs (e.g., email program, text messaging program) on mobile device 100. When user compilation image 431 is called via the API, mobile device 100 may automatically render user compilation image 431 and supply the rendered copy to the requesting program.

[0186] Part 10

[0187] Fig.16 A video package 1601 is shown including a first video 1611, a second video 1621, a user-edited video 1631, a calibration 1651, and a map 1661. The first video 1611 may include a plurality of first frames 1612, each of which may be a different first image 411. The second video 1621 may include a plurality of second frames 1622, each of which may be a different second image 421. The user-edited video may include a plurality of user-edited frames 1632, each of which may be a different user-edited image 431.

[0188] Map 1661 may include a plurality of different maps, each of which is applied to one or more frames of first video 1611, one or more frames of second video 1621, and one or more frames of user-compiled video 1631. Map 1661 may be constructed using calibration 1651, as previously discussed with respect to map 461 and calibration 451.

[0189] like Fig.16A As shown, the mobile device 100 may enable user-selected segmentation of a user-compiled video into a plurality of segments 1601. Here, the user has segmented the user-compiled video 1631 into first, second, and third segments 1631a, 1631b, and 1631c. The length of each segment and the total number of segments may be user selectable.

[0190] The mobile device 100 may enable the user to independently select a user-programmed field of view, a user-programmed zoom level, and one or more photo effects for each segment. Alternatively, the mobile device 100 may enable the user to independently select a user-programmed field of view and one or more photo effects for each segment, and then automatically apply the zoom level. This enables the mobile device 100 to maintain each frame of the user-programmed video 1631 at a uniform resolution.

[0191] The mobile device 100 may store the user-compiled videos 1631 as a table with multiple entries. An example table is shown below:

[0192]

[0193] Each entry number may correspond to a frame of the user-programmed video 1631. Each entry may include a frame reference matrix having the following items: (a) user-programmed field of view; (b) user-programmed zoom level; (c) one or more photo effects; (d) global coordinate transformation function (and inverse global coordinate transformation function—not shown); (e) reference frames of the first video 1611 and reference frames of the second video 1621.

[0194] Items (a), (b) and (c) may be referred to above. Fig.14 and 15 1611, 1621. The first and second videos 1611, 1621 are described in detail. With respect to item (d), each user-compiled frame may be associated with a different global transformation function G(x), and therefore with a different inverse global transformation function G-1(x). Items (e) and (f) describe which frames of the first and second videos 1611, 1621 are referenced when presenting a particular user-compiled frame. Although each pair of items (e) and (f) has the same value in the above table, other examples may have different values ​​for each pair of items (e) and (f). This would be advantageous if the first and second videos have different frame rates.

[0195] To play the user-compiled video 1631 , the mobile device 100 may present a plurality of displayed frames. Each displayed frame may appear based on a corresponding user-compiled frame in the same manner that a displayed image 441 may appear based on a user-compiled image 431 .

[0196] The mobile device 100 may be configured to display the user-compiled video 1631 into a known video format (e.g., AVI format). When an external program (e.g., an email program, a text messaging program) calls the user-compiled video 1631 via the above-mentioned API, the API may automatically display the user-compiled video 1631 into a typical video format. The typical video format may store each frame as a two-dimensional matrix of cells, each cell corresponding to a different pixel, and each cell having a three-dimensional color vector. The typical video format may store compressed versions of these frames. By storing the video 1631 as multiple entries, which is relative to the traditional format, the mobile device 100 may save storage space.

[0197] Part 11

[0198] The mobile device 100 may be a smart phone, a tablet computer, a digital camera, or a laptop computer. Devices, Devices (e.g. an or ),or Devices (such as Surface Phone, or The mobile device 100 may represent a non-mobile device, such as a vehicle, or a desktop computer (such as a computer server).

[0199] like Fig.17 As shown, the mobile device 100 may include a processing system 1700. The processing system 1700 may include one or more processors 1701, a memory 1702, one or more input / output devices 1703, one or more sensors 1704, one or more user interfaces 1705, one or more motors / actuators 1706, and one or more data buses 1707.

[0200] The processor 1701 may include one or more different processors, each having one or more cores. Each of the different processors may have the same or different architectures. The processor 1701 may include one or more central processing units (CPUs), one or more graphics processing units (GPUs), application specific integrated circuits (ASICs), digital signal processors (DSPs), and the like.

[0201] The processor 1701 is configured to perform a specific function or operation at least when one of the one or more of the distinct processors is capable of executing code stored on the memory 1702 embodying the function or operation. The processor 1701 may be configured to perform any and all functions, methods, and operations disclosed herein.

[0202] Memory 1702 may include volatile memory, nonvolatile memory, and any other media capable of storing data. Each of volatile memory, nonvolatile memory, and any other type of memory may include multiple different memory devices located at multiple different locations and each having a different structure.

[0203] Examples of memory 1702 include non-transitory computer-readable media such as RAM, ROM, flash memory, EEPROM, any kind of optical storage disk such as DVD, Optical disks, magnetic storage devices, holographic storage devices, HDDs, SSDs, any media that can be used to store program code in the form of instructions or data structures, and the like. Any and all methods, functions, and operations described in this application can be fully embodied in the form of tangible and / or non-transitory machine-readable code stored in the memory 1702.

[0204] Input-output devices 1703 may include any component for transporting data, such as ports and onboard information systems. Input-output devices 1703 may enable Ethernet and the like for wired communication. The input-output device 1703 may enable electronic, optical, magnetic, and holographic communication with suitable storage 1703. The input-output device may enable communication via Cellular technologies (e.g. ), GPS and the like for wireless communication.

[0205] The sensor 1704 may capture physical measurements of the environment and report the physical measurements to the processor 1701. The sensor 1704 may include a first camera 105 and a second camera 106. Each of the first camera 105 and the second camera 106 may include one or more lenses, one or more apertures, and one or more photodiode panels arranged along an optical axis. The one or more photodiode panels may capture the luminosity of incident light, which the processor 1701 may convert into an image file. The first camera 105 may include a wide-angle lens, and the second camera 106 may include a telephoto lens.

[0206] The user interface 1705 may enable a user to interact with the imaging system 100. The user interface 1705 may include a display (e.g., an OLED touch screen, an LED touch screen), physical buttons, a speaker, a microphone, a keyboard, and the like. The user interface 1705 may include the display 102 and the physical buttons 103.

[0207] The motor / actuator 1706 may enable the processor 1701 to control mechanical or chemical forces. If the first or second camera 105, 106 includes autofocus, the motor / actuator 1706 may move the lens along the optical axis of the lens to provide autofocus.

[0208] Data bus 1707 may transport data between components of processing system 1700. Data bus 1707 may include conductive paths printed on or otherwise applied to a substrate (e.g., conductive paths on a logic board), SATA cables, coaxial cables, The data bus 1707 may be a conductive path of a logic board on which the processor 1701 and volatile memory are mounted. The data bus 1707 may include a wireless communication path. The data bus 1707 may include a series of different wires 1707 (e.g. Cable), different components of processing system 1700 are connected via connection 1707.

Claims

1. A method for image processing, comprising: receiving a first command specifying a first field of view; receiving a second command subsequent to the first command; storing a first image of a scene captured by a first camera in response to the second command, the first image comprising first pixels and having the first field of view; storing a second image of the scene captured by a second camera in response to the second command, the second image comprising second pixels and having a second field of view; storing a third image of the scene captured by the first camera in response to the second command, the third image including the first pixels and having a third field of view, wherein the first field of view is disposed within the third field of view; receiving a third command subsequent to the second command; In response to the third command, outputting a thumbnail image of the first image; receiving a fourth command subsequent to the third command; In response to the fourth command, editing the field of view of the first image; as well as An edited image is stored, the edited image having a fourth field of view and including pixels based on the first pixel and the second pixel. The method of claim 1 , wherein the first field of view is disposed within the second field of view.

3. The method according to claim 2, further comprising: The plurality of first pixels are spatially aligned with the plurality of second pixels.

4. The method according to claim 3, further comprising: The plurality of first pixels are photometrically aligned with the plurality of second pixels.

5. The method according to claim 3, further comprising: The edited image is stored in a package that includes the second image and the third image.

6. The method according to claim 1, further comprising: receiving a fifth command to edit the edited image; editing the edited image based on the fifth command to generate a second edited image; as well as The second edited image is stored. The method of claim 6 , wherein editing the edited image comprises applying a photo effect to the edited image. The method of claim 7 , wherein the photographic effect comprises a bokeh effect.

9. The method of claim 8, wherein the bokeh effect is based on a depth map, and wherein the depth map is based on the first pixel and the second pixel.

10. The method of claim 7, wherein the photographic effect comprises a color shift.

11. The method according to claim 1, The first command is a zoom command. The method of claim 11 , wherein the third image comprises the first pixel and a third pixel. The method of claim 12 , wherein the edited image comprises pixels based on the third pixel.

14. A processing system comprising: Memory; and One or more processors, coupled to the memory and configured to: receiving a first command specifying a first field of view; receiving a second command subsequent to the first command; In response to the second command, storing a first image of a scene captured by a first camera, the first image comprising first pixels and having the first field of view; In response to the second command, storing a second image of the scene captured by a second camera, the second image including second pixels and having a second field of view; storing a third image of the scene captured by the first camera in response to the second command, the third image including the first pixels and having a third field of view, wherein the first field of view is disposed within the third field of view; receiving a third command subsequent to the second command; In response to the third command, outputting a thumbnail image of the first image; receiving a fourth command subsequent to the third command; In response to the fourth command, editing the field of view of the first image; and An edited image is stored, the edited image having a fourth field of view and including pixels based on the first pixel and the second pixel.

15. The system of claim 14, wherein the first field of view is disposed within the second field of view.

16. The system of claim 15, wherein the processor is further configured to: The plurality of first pixels are spatially aligned with the plurality of second pixels.

17. The system of claim 16, wherein the processor is further configured to photometrically align the plurality of first pixels with the plurality of second pixels.

18. The system of claim 16, wherein the processor is further configured to: The edited image is stored in a package that includes the second image and the third image.

19. The system of claim 14, wherein the processor is further configured to: receiving a fifth command to edit the edited image; editing the edited image based on the fifth command to generate a second edited image; and The second edited image is stored.

20. The system of claim 19, wherein editing the edited image comprises applying a photographic effect to the edited image.

21. The system of claim 20, wherein the photographic effect comprises a bokeh effect.

22. The system of claim 21, wherein the bokeh effect is based on a depth map, and wherein the depth map is based on the first pixel and the second pixel.

23. The system of claim 20, wherein the photographic effect comprises a color shift.

24. The system of claim 14, wherein the first command is a zoom command.

25. The system of claim 24, wherein the third image comprises the first pixel and a third pixel.

26. The system of claim 25, wherein the edited image comprises pixels based on the third pixel.

27. A non-transitory computer readable medium comprising program code, which when executed by one or more processors causes the one or more processors to perform operations, the program code comprising code for: receiving a first command specifying a first field of view; receiving a second command subsequent to the first command; In response to the second command, storing a first image of a scene captured by a first camera, the first image comprising first pixels and having the first field of view; In response to the second command, storing a second image of the scene captured by a second camera, the second image including second pixels and having a second field of view; storing a third image of the scene captured by the first camera in response to the second command, the third image including the first pixels and having a third field of view, wherein the first field of view is disposed within the third field of view; receiving a third command subsequent to the second command; In response to the third command, outputting a thumbnail image of the first image; receiving a fourth command subsequent to the third command; editing the field of view of the first image in response to the fourth command; and An edited image is stored, the edited image having a fourth field of view and including pixels based on the first pixel and the second pixel.

28. A method for image processing, comprising: receiving a first image of a scene captured by a first camera, the first image comprising first pixels and having a first field of view; receiving a second image of the scene captured by a second camera, the second image including second pixels having a second field of view; storing a first user-compiled image of the scene in response to a first command, the first user-compiled image comprising a first user-compiled pixel and having a user-compiled field of view, and the first user-compiled pixel being derived from the first pixel based on an intersection of the user-compiled field of view with the first field of view; receiving a second command to edit the first user compiled image; editing the first user compiled image based on the second command; storing the edited first user compiled image as a second user compiled image; receiving a third command to change the field of view of the second user compiled image; editing the second user compiled image based on the third command; and The edited second user compiled image is stored as a third user compiled image of the scene, the third user compiled image including the second user compiled pixels and having a second user compiled field of view, and the second user compiled pixels are derived from the second pixels based on the second user compiled field of view intersecting the second field of view.

29. The method according to claim 28, further comprising: deriving a displayed image from said first user-compiled image; as well as presenting the displayed image; wherein the second command includes at least one command to adjust the displayed image; Wherein editing the first user-compiled image comprises: Third user-programmed pixels are automatically derived for the second user-programmed image in response to the at least one command.

30. The method of claim 28, wherein the second field of view is disposed within the first field of view.

31. The method of claim 30, wherein the first user compiled image comprises pixels derived from the first image.

32. The method of claim 30, wherein the first user compiled image includes pixels derived from the first pixels, and the first user compiled image does not include pixels derived from the second pixels; and in, Pixels in the second user compiled image are derived from one or more of the first pixels and one or more of the second pixels.

33. The method of claim 28, wherein editing the first user-edited image comprises deriving third user-edited pixels for the second user-edited image from both the first pixels and the second pixels without displaying the first image and without displaying the second image.

34. The method of claim 33, wherein: (a) both the first image and the second image are rectified to a global plane, or (b) The first image and the second image are not rectified to a global plane, and deriving the second user-crafted pixels from the second pixels includes accessing the second image via one or more spatial transforms.

35. The method of claim 28, further comprising generating a depth map based on one or more of the first image and the second image, wherein editing the first user-compiled image comprises: Receive photo effect selection; and In response to the photo effect selection, a photo effect is applied to the first user compiled image based on the generated depth map.

36. The method of claim 35, wherein: Editing the first user compiled image includes changing a bokeh effect applied to the first user compiled image based on the generated depth map.

37. A method according to claim 35, wherein the first user-compiled image includes pixels derived from the first pixel but does not include pixels derived from the second pixel, and due to the photographic effect, the second user-compiled image includes pixels derived from the first pixel and the second pixel.

38. The method of claim 35, wherein editing the first user-generated image comprises: receiving a selection of a first item in the first user compiled image; In response to selection of the first item, estimating a depth range of the first item based on the generated depth map; and An operation of blurring a second item in the first user compiled image that is outside the estimated depth of field range is performed.

39. The method of claim 38, further comprising: determining that the first item is in the first image and not in the second image; and One-dimensional and non-stereoscopic techniques are applied to generate the depth map including the first item.

40. The method of claim 35, wherein: The generating includes generating the depth map based on the first image and the second image.

41. The method of claim 28, wherein editing the first user compiled image comprises adjusting a depth of field effect.

42. The method of claim 28, further comprising: receiving a fourth command to store the edited first user compiled image; storing the edited first user compiled image as the second user compiled image in response to the fourth command; The first user-edited image is deleted in response to the third command.

43. The method of claim 30, wherein editing the first user compiled image comprises applying a photo effect to the first user compiled image.

44. The method of claim 43, wherein: The second command is associated with selection of the photo effect.

45. The method of claim 44, wherein the photographic effect comprises a bokeh effect.

46. ​​The method of claim 44, wherein the photographic effect comprises a color shift.

47. The method of claim 30, further comprising spatially aligning the first image and the second image.

48. The method of claim 47, further comprising photometrically aligning the first image and the second image.

49. A processing system comprising: Memory; and One or more processors, coupled to the memory and configured to: receiving a first image of a scene captured by a first camera, the first image comprising first pixels and having a first field of view; receiving a second image of the scene captured by a second camera, the second image including second pixels having a second field of view; storing a first user-compiled image of the scene in response to a first command, the first user-compiled image comprising a first user-compiled pixel and having a user-compiled field of view, and the first user-compiled pixel being derived from the first pixel based on an intersection of the user-compiled field of view with the first field of view; receiving a second command to edit the first user compiled image; editing the first user compiled image based on the second command; storing the edited first user compiled image as a second user compiled image; receiving a third command to change the field of view of the second user compiled image; editing the second user compiled image based on the third command; and The edited second user compiled image is stored as a third user compiled image of the scene, the third user compiled image including the second user compiled pixels and having a second user compiled field of view, and the second user compiled pixels are derived from the second pixels based on the second user compiled field of view intersecting the second field of view.

50. The processing system of claim 49, wherein the one or more processors are configured to: deriving a displayed image from the first user-compiled image; and presenting the displayed image; Edit the first user-compiled image by: Third user-crafted pixels are automatically derived for the second user-crafted image in response to at least one command to adjust the displayed image, the second command comprising the at least one command.

51. The processing system of claim 49, wherein: The second field of view is disposed within the first field of view.

52. The processing system of claim 51, wherein the first user compiled image comprises pixels derived from the first image.

53. The processing system of claim 51, wherein the first user compiled image includes pixels derived from the first pixel, and the first user compiled image does not include pixels derived from the second pixel; and in, Pixels in the second user compiled image are derived from one or more of the first pixels and one or more of the second pixels.

54. The processing system of claim 49, wherein the one or more processors are configured to edit the first user-compiled image by: A third user-crafted pixel is derived for the second user-crafted image from both the first pixel and the second pixel without displaying the first image and without displaying the second image.

55. The processing system of claim 49, wherein the processing system is a wireless communication device, the processing system further comprising a first camera and a second camera, wherein the one or more processors are configured to: In response to the first command, commanding the first camera to capture the first image; In response to the first command, commanding the second camera to capture the second image; and In response to the first command, the first user-programmed pixel is exported.

56. The processing system of claim 49, wherein the one or more processors are configured to generate a depth map based on one or more of the first image and the second image.

57. The processing system of claim 56, wherein the one or more processors are configured to edit the first user compiled image by changing a bokeh effect applied to the first user compiled image based on the generated depth map.

58. The processing system of claim 56, wherein the one or more processors are configured to edit the first user-compiled image by: receiving a selection of a first item in the first user compiled image; In response to selection of the first item, estimating a depth range of the first item based on the generated depth map; and A second item in the first user compiled image that is outside the estimated depth of field range is blurred.

59. The processing system of claim 58, wherein the one or more processors are configured to edit the first user-compiled image by: determining that the first item is in the first image and not in the second image; and One-dimensional and non-stereoscopic techniques are applied to generate the depth map including the first item.

60. The processing system of claim 51, wherein editing the first user compiled image comprises applying a photo effect to the first user compiled image.

61. The processing system of claim 60, wherein the second command is associated with selection of the photo effect.

62. The processing system of claim 61, wherein the photographic effect comprises a bokeh effect.

63. The processing system of claim 60, wherein the photographic effect comprises a color shift.

64. The processing system of claim 61, wherein: The one or more processors are configured to spatially align the first image and the second image.

65. The processing system of claim 64, wherein the one or more processors are configured to photometrically align the first image and the second image.

66. A non-transitory computer readable medium comprising program code, which when executed by one or more processors causes the one or more processors to perform operations, the program code comprising code for: receiving a first image of a scene captured by a first camera, the first image comprising first pixels and having a first field of view; receiving a second image of the scene captured by a second camera, the second image including second pixels having a second field of view; storing a first user-compiled image of the scene in response to a first command, the first user-compiled image comprising a first user-compiled pixel and having a user-compiled field of view, and the first user-compiled pixel being derived from the first pixel based on an intersection of the user-compiled field of view with the first field of view; receiving a second command to edit the first user compiled image; editing the first user compiled image based on the second command; storing the edited first user compiled image as a second user compiled image; receiving a third command to change the field of view of the second user compiled image; editing the second user compiled image based on the third command; and The edited second user compiled image is stored as a third user compiled image of the scene, the third user compiled image including the second user compiled pixels and having a second user compiled field of view, and the second user compiled pixels are derived from the second pixels based on the second user compiled field of view intersecting the second field of view.