Imaging system and method

Abstract

In an illustrative embodiment, a method of generating a digital image and / or modified depth information can include obtaining a plurality of images of a target over a time period via a first electronic sensor; selecting one or more pixels in a first image of the plurality of images; identifying corresponding pixels in one or more other images of the plurality of images corresponding to the selected one or more pixels, the selected one or more pixels and the corresponding pixels defining a reference set of pixels; identifying two or more images of the plurality of images having a best disparity for the respective reference set of pixels; generating modified depth information; and / or generating a final digital image via the plurality of images and the modified depth information.

Description

[0001] Cross-reference of related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 168,757, filed March 31, 2021, the disclosure of which is incorporated herein by reference in its entirety, as fully set forth herein. Technical Field

[0003] This disclosure generally relates to imaging systems and methods, including imaging systems and methods that can be used in conjunction with smartphone cameras, such as those simulating full-frame cameras and lenses. Background Technology

[0004] The background description described below is provided for context only. Therefore, nothing in this background description is expressly or implicitly acknowledged as prior art to this disclosure without otherwise conforming to the principles of the art.

[0005] Integrating a full-frame camera and lens into a smartphone may be impractical or unfeasible, but it is desirable to emulate one or more full-frame cameras and lenses for use with images captured via the smartphone's camera.

[0006] Some imaging systems may not be configured to acquire sufficient depth information and / or may not effectively or efficiently utilize depth information when capturing and / or rendering images.

[0007] Solutions / options that aim to minimize or eliminate one or more challenges or drawbacks of imaging systems and methods. The above discussion is intended to illustrate examples in the art only and is not a denial of scope. Summary of the Invention

[0008] In an exemplary illustrative embodiment, a method for generating a digital image and / or modified depth information may include: acquiring multiple images of a target over a time period via a first electronic sensor; selecting one or more pixels in a first image among the multiple images; identifying corresponding pixels in one or more other images among the multiple images that correspond to the selected one or more pixels, the selected one or more pixels and the corresponding pixels defining a reference pixel set; determining disparity between the corresponding reference pixel sets; identifying two or more images among the multiple images that have optimal disparity based on the reference pixel set; determining disparity between some, most, or all pixels of the identified image pair; combining the determined disparity from all pixels with depth information from a second electronic sensor to generate modified depth information; and / or generating a final digital image via the multiple images and the modified depth information.

[0009] Using an illustrative embodiment, a mobile electronic device may include a first electronic sensor, a second electronic sensor, and / or an electronic control unit. The electronic control unit may be configured to: acquire multiple images of a target within a time period via the first electronic sensor; select one or more pixels in a first image among the multiple images; identify corresponding pixels in one or more other images among the multiple images that correspond to the selected one or more pixels, the selected one or more pixels and the corresponding pixels defining a reference pixel set; identify two or more images among the multiple images that have optimal parallax based on the reference pixel set; generate modified depth information; and / or generate a final digital image via the multiple images and / or the modified depth information.

[0010] Using some illustrative embodiments, generating modified depth information may include: determining the disparity between two or more identified images; combining the determined disparity with depth information from a second electronic sensor; amplifying the combined disparity and depth information; shifting pixels of one of the identified images based on the amplified combined disparity and depth information; determining the remaining disparity after the shift; combining the remaining disparity with the depth information; and / or amplifying the remaining combined disparity and depth information. The modified depth information may include the amplified remaining combined disparity and depth information. The imaging system may include a mobile electronic device.

[0011] The above and other potential aspects, features, details, utility and / or advantages of the examples / embodiments of this disclosure will become apparent from reading the following description and viewing the accompanying drawings. Attached Figure Description

[0012] While the claims are not limited to the specific illustrations, an understanding of the aspects can be gained through discussion of various examples. The drawings are not necessarily drawn to scale, and some features may be exaggerated or omitted to better illustrate and explain the innovative aspects of the examples. Furthermore, the exemplary illustrations described herein are not exhaustive or otherwise limiting, and are not limited to the precise forms and configurations shown in the drawings or disclosed in the detailed description below. The exemplary illustrations are described in detail with reference to the following drawings:

[0013] Figures 1 to 3 This is a representation of an exemplary embodiment of the imaging system and method.

[0014] Figure 4 This is a flowchart of an exemplary embodiment of the imaging method.

[0015] Figure 5 It is a representation of multiple images having corresponding reference pixels identified in an exemplary embodiment of the imaging system.

[0016] Figures 6 to 10 This is a flowchart of a portion of an exemplary embodiment of the imaging method. Detailed Implementation

[0017] Reference will now be made in detail to the illustrative embodiments of this disclosure, examples of which are described herein and illustrated in the accompanying drawings. While this disclosure will be described in conjunction with embodiments and / or examples, they are not intended to limit this disclosure to those embodiments and / or examples. Rather, this disclosure covers alternatives, modifications, and equivalents.

[0018] Full-frame cameras and lenses can provide bokeh for at least some parts of an image. For example, bokeh can include the aesthetic effect of blurring out-of-focus areas of an image. Various components of full-frame cameras and lenses (such as aperture blades, flare, spherical aberration, and / or chromatic aberration) can affect how the camera / lens renders bokeh. With full-frame cameras and lenses, bokeh can be created before light hits the sensor, which can maintain consistent graininess in both in-focus and out-of-focus areas of the image.

[0019] Smartphone cameras typically have much smaller image sensors, with a significantly smaller surface area than full-frame cameras to capture light and produce a near-infinite depth of field. This allows the background image to be rendered almost as sharp as the object / target itself (e.g., so no part of the image is out of focus). While some smartphone cameras have modes that blur the background of an image (e.g., portrait mode), this blur does not mimic the optical effects associated with the combination of a full-frame camera and lens, such as optical vignetting, chromatic aberration, spherical aberration, and / or various other properties of bokeh that photographers can use to achieve a particular look in their art. Instead, smartphone blur can be uniform and completely grain-free, which may give the blur an unnatural look and is barely noticeable to the viewer's eye.

[0020] Some smartphone cameras (and the resulting images) can suffer from poor segmentation between foreground and background. Smartphones often rely on relatively low-resolution depth maps, which can lead to inaccurate edge and depth estimation errors, resulting in sharp areas in the background or blurred areas on objects / targets, especially noticeable when capturing moving scenes. To provide improved image quality compared to other designs, embodiments of this disclosure can be configured in particular to obtain enhanced / modified depth / parallax information, change the intensity or focus of objects based on depth (e.g., distance from the smartphone / sensor), and / or generate enhanced / modified images based at least in part on the enhanced / modified depth / parallax information.

[0021] In such as Figure 1In the illustrative exemplary embodiments generally shown, the imaging system 100 may include a mobile electronic device 110 (e.g., a smartphone), which may include a first electronic sensor 120, a second electronic sensor 122, and / or an electronic control unit 124 (ECU). The mobile electronic device 110 may, for example, include a cellular transceiver / radio / modem 126 that can communicate with the ECU 124 and / or a cellular network. The electronic control unit 124 may include and / or be connected to a processor 124A and / or a memory 124B. The mobile electronic device 110 may be relatively thin. For example, but not limited to, the thickness of the mobile electronic device 110 may be approximately 1 inch (25.4 mm) or less, or approximately 0.5 inches (12.7 mm) or less.

[0022] Using an illustrative embodiment, the mobile electronic device 110 can be configured to simulate one or more effects of a combination of one or more full-frame cameras and lenses, such as bokeh associated with such a combination. Effectively simulating full-frame camera and lens bokeh can include obtaining accurate depth information on a pixel-by-pixel basis (e.g., rather than applying a uniform blur). Accurate depth information can be provided via an enhanced depth map 134.

[0023] Adopting, such as in Figure 1 In the exemplary embodiments generally shown, the first electronic sensor 120 may be configured to acquire image information associated with a target. For example, but not limited to, the first electronic sensor 120 may include a color sensor (e.g., an RGB sensor). The image information may include color information associated with the target, such as RGB intensity information. In some embodiments, such as in… Figure 2 As generally shown, the first electronic sensor 120 may include an additional sensor portion 140, such as a second color sensor (e.g., a second RGB sensor).

[0024] In an illustrative embodiment, the second electronic sensor 122 may be configured to acquire initial depth information associated with a target (e.g., may include a depth / distance sensor). For example, but not limited to, the second electronic sensor 122 may include a time-of-flight (TOF) sensor (see, for example, see...). Figure 1 and Figure 2For example, the second electronic sensor 122 can emit a signal, and determine the distance between the second electronic sensor 122 and the object based on the amount of time between when the signal is emitted and when the signal returns to the second electronic sensor 122 after being reflected by the object. The target / scene may include one or more physical objects that may be positioned at different distances from the mobile electronic device 110. The second electronic sensor 122 may be configured to acquire depth information associated with some or all of the objects of the target. For example, the second electronic sensor 122 may be configured to acquire initial depth information, such as on a pixel-by-pixel basis. The depth information may include depth data 130 (e.g., feet, meters, etc. for some or all pixels) and / or a confidence map 132 (e.g., confidence values ​​on a scale of 0 to 1 for some or all pixels). The ECU 124 may be configured to provide an enhanced depth map 134 and / or a final image 136 that can be obtained at least in part based on the enhanced depth map 134.

[0025] Adopting, such as in Figure 3 In the exemplary embodiments generally shown, the second electronic sensor 122 may additionally or alternatively include one or more phase-detection autofocus (PDAF) sensors. In at least some cases, the PDAF sensor may be connected to and / or integrated with the first electronic sensor 120. For example, but not limited to, the first electronic sensor 120 may include a first RGB sensor that may include the first PDAF sensor of the second electronic sensor 122, and / or the first electronic sensor 120 may include a second RGB sensor that may include the second PDAF sensor of the second electronic sensor 122.

[0026] In an exemplary embodiment, the method 1000 of operating the imaging system 100 may be configured to: generate a digital image, obtain enhanced disparity information, obtain enhanced depth information, and / or generate an enhanced depth / disparity map. Disparity can be used to determine depth. For example, a larger disparity may correspond to a closer portion of the scene / image, and a smaller disparity may correspond to a farther portion of the scene / image.

[0027] Adopting, such as in Figure 4 In the illustrative embodiments generally shown, the method 1000 of operating the imaging system 100 may include: acquiring a plurality of images 150 of the target 102 over a period of time, such as via a first electronic sensor 120 of a mobile electronic device 110 (see, for example, [link to relevant documentation]). Figure 5 Image 150 1-8 (Box 1002). For example, but not limited to, the time period can be approximately 1 second or less. For example, image 150 1-8 It can be obtained through image bursting, and the image is 150. 1-8Captures can be taken at appropriate times (e.g., one after another). Target 102 may, for example, include a scene expected to be captured and rendered in a digital image. Method 1000 may include selecting one or more pixels (box 1004) in a first image 1501 of a plurality of images 150. The selected pixels of the image may include associated location information (e.g., X). 1-8 Y 1-8 Z 1-8 ) and / or associated color information (e.g., RGB) 1-8 The associated color information (RGB) of a pixel may be at least somewhat unique relative to other pixels in the first image 1501, especially when one or more images 150 are scaled down. The selected pixels may be automatically selected, such as via the ECU 124 of the mobile electronic device 110. For example, but not limited to, the selected pixels may include pixels at the center of the first image 1501, pixels closest to (e.g., the mobile electronic device 110), pixels on one or more parts of the face of a person or animal in the first image 1501, and / or pixels at various depths. In some cases, the mobile electronic device 110 may provide the selected pixels, such as via a rangefinder.

[0028] Using an illustrative example embodiment, method 1000 may include: one or more other images among a plurality of images 150 (e.g., image 150) 2-8 Identify the corresponding pixel (X1, Y1, Z1) for each selected pixel (X1, Y1, Z1). 2-8 Y 2-8 Z 2-8 (Box 1006). The corresponding pixel may, for example, include substantially the same color information as the corresponding selected pixel, and may or may not include the same position information. ECU 124 can be configured to automatically determine the corresponding pixel, such as by means of other images 150. 2-8 The search includes pixels whose color information is most similar (or identical) to the selected pixel. ECU124 can utilize lower-resolution and / or scaled-down versions of multiple images 150 to identify corresponding pixels in other images. The selected pixel in the first image 1501 and other images 150... 2-8 The corresponding pixels in the image can be defined and / or referred to as a corresponding set of reference pixels. Once the corresponding pixels are identified, the ECU 124 can align the image based on these pixels (e.g., alignment that minimizes total parallax) and can remove and / or reduce distortions, such as those caused by rotational motion (e.g., non-translational motion). For example, rotational motion may not be used to determine depth / parallax. Once distortions are removed / reduced, a significant amount (e.g., the majority) of the remaining parallax can correspond to the user's hand movements during that period.

[0029] In an illustrative embodiment, method 1000 may include, for example, determining disparity between corresponding pairs of reference pixels from different sets / images via ECU 124 (box 1008). Determining disparity may include determining disparity in two dimensions, such as from reference pixels from a first image 1501 to another image 150. 2-8 The number of pixels shifted. The two dimensions can include an X value (e.g., left-to-right pixel shift) and a Y coordinate (e.g., top-to-bottom pixel shift). In some cases, at least some parallax can correspond to the user's hand movement during that time period (e.g., natural tremor). Method 1000 can include identifying two or more images 150 (box 1010) with optimal / expected parallax using a corresponding set of reference pixels. The optimal / expected parallax can be at or above a minimum parallax threshold, which can correspond to a parallax amount sufficient to determine depth. For example, the optimal / expected parallax can correspond to a parallax amount that allows for the differentiation of depth levels in the image. The optimal / expected parallax can be determined at least in part based on depth information from a depth sensor (such as one that may be included with a second electronic sensor 122). The optimal / expected parallax can be below a maximum threshold, such as to filter out certain images, such as images occluded beyond an occlusion threshold (e.g., excluding a sufficient number of images of the same scene) and / or anomalous images, such as in the event that the mobile electronic device 110 is dropped, the sensor malfunctions, or some other error occurs.

[0030] In an illustrative embodiment, method 1000 may include determining the disparity between some, most, or all pixels of two or more identified images 150 having optimal / desired disparity (box 1012).

[0031] In an exemplary embodiment, method 1000 may include, for example, obtaining depth information of a target via a second electronic sensor 122 (block 1014). For example, but not limited to, the second electronic sensor 122 may include a TOF sensor.

[0032] In an exemplary embodiment, method 1000 may include combining information from two or more identified images 150 with depth information from a second electronic sensor 122, such as to generate a modified / enhanced depth map 134 (block 1016). The initial depth information (such as depth data) may be of relatively low resolution. Combining the initial depth information with parallax information from the two identified images 150 can improve this resolution and may result in a modified depth map that is more accurate / complete than if only depth data from the second electronic sensor 122 were used.

[0033] Using an illustrative embodiment, method 1000 may include generating a final digital image 136 (box 1018) via a plurality of images 150 and a modified / enhanced depth map 134. In some cases, the final digital image 136 may or may not be specifically configured for viewing on a display 112 of a mobile electronic device 110. Instead, the final digital image 136 may be configured for printing by printer 104 on a tangible medium (e.g., paper, photographic paper, canvas, metal, plastic, ceramic, etc.) such that all aspects of the printed image are clear from both near and far distances. For example, printer 104 may be configured to receive information from mobile electronic device 110 (e.g., via a wired connection, wireless connection, removable media, etc.), and this information may include the final digital image 136. Method 1000 may include printing the final digital image 136 on paper or a physical medium (such as photographic paper) via printer 104.

[0034] In an exemplary embodiment, method 1000 may not rely on the gyroscope 160 of the mobile electronic device 110. In other embodiments, method 1000 may utilize information from the gyroscope 160, such as to convert parallax information (e.g., directly) into depth information.

[0035] Figures 6 to 10 Exemplary embodiments of methods for obtaining depth information and combining depth information with color information are generally shown.

[0036] Figure 6 An exemplary embodiment of a method 2000 for selecting two (or more) images 150 with optimal parallax is generally shown, as described above in conjunction with blocks 1002 to 1010 of method 1000 (see, for example, [link to relevant documentation]). Figure 4 The baseline BL can correspond to the expected or average amount of hand shakiness / tremor between image captures. Method 2000 may include: obtaining depth information such as a first image 1501 captured at a first time (box 2002), obtaining color information RGB from the first image 1501 captured at the first time (box 2004), obtaining color information RGB from a second image 1502 captured at a second time (box 2006), obtaining color information RGB from a third image 1503 captured at a third time (box 2008), obtaining color information RGB from a fourth image 1504 captured at a fourth time (box 2010), and / or from one or more additional images 150 captured at other times (e.g., image 150). 5-8 Obtain color information RGB. Method 2000 may include: such as determining the optimal baseline BL based on the obtained depth information (box 2012). Method 2000 may include: for images 150 after the first image 1501 2-4Determining the BL (box 2014) may include eliminating virtual reflections on moving objects. Method 2000 may include identifying / selecting two images with optimal parallax from a plurality of images 150 (box 2016). The output of box 1010 of method 1000 may correspond to the output of box 2016.

[0037] Adopting, such as in Figure 7 The exemplary embodiments generally shown (e.g., in block 1016 of method 1000, see...) Figure 4 Combining depth information and image information can include obtaining an enhanced confidence map 170. A depth sensor (such as one that may be included with a second electronic sensor 122) can output depth data 130 and a confidence map 132 (e.g., based on the number of photons received). The ECU 124 can combine the depth data 130 and / or the confidence map 132 with one or more images 150 and / or their color information that may be obtained via the first electronic sensor 120 to generate the combined confidence map 170. For example, the ECU 124 can look for pixel colors that the confidence map 132 identifies as having high confidence. For example, if a pixel is essentially black (and not expected to actually reflect a large amount of light), the ECU 124 can ignore it or give it a smaller weight in terms of confidence. Additionally or alternatively, if a pixel has a low or zero confidence level, but the pixel's color is relatively light (and is expected to reflect a lot of light), the ECU 124 can use the depth data 130 to confirm the low confidence level, since the pixel corresponds to a distant part of the scene. Additionally or alternatively, if the confidence level is missing (e.g., this could occur for low or zero confidence levels), the ECU 124 can generate an estimated confidence level based on the pixel's color information (e.g., higher confidence for lighter pixels and lower confidence for darker pixels).

[0038] In such as Figures 8 to 10 In the illustrative embodiments generally shown, ECU 124 can be configured to perform multiple depth iterations 3000. For example... Figure 8Typically shown, the first iteration may include downscaling the identified image with the optimal parallax (boxes 3002, 3004) (e.g., an image with optimal / desired parallax), such as to 1 / 8 resolution. Downscaling may tend to eliminate noise. The ECU 124 may then perform parallax calculations between the downscaled images (box 3006), such as by using a pixel kernel (e.g., a 3x3 kernel, a 5x5 kernel, etc.) for some pixels or per pixel, where the pixel of interest can be located at the center of the kernel. The color information of the kernel may be more unique than that of individual pixels, which may be beneficial for locating corresponding pixels (e.g., and reducing the computational resources used). The ECU 124 may convert the depth data 130 to parallax based on the maximum parallax of the identified images (box 3008) (e.g., the maximum depth is scaled to equal the maximum parallax, and the minimum depth is scaled to the minimum parallax, which may typically be set to 0). ECU 124 can combine disparity information from the second electronic sensor 122 (e.g., converted from depth information) with disparity information determined between identified images based at least in part on the combined confidence map 170 (box 3010). For example, if the combined confidence of pixels is high, ECU 124 can utilize (e.g., from box 3008) the converted disparity data and / or weight it more. If the combined confidence of pixels is low, ECU 124 can utilize (e.g., from box 3006) the determined disparity data and / or weight it more. For moderate confidence, ECU 124 can use a weighted average of the converted disparity data and the determined disparity data.

[0039] In an exemplary embodiment, ECU 124 can filter the combined disparity data (box 3012), similar to denoising. Filtering (e.g., guided filtering) can identify pixels with the same or highly similar color information that may have the same disparity, and can remove and / or modify the disparity information of pixels whose disparity information is significantly different from other pixels of the same or similar color. ECU 124 can (e.g., after filtering) amplify the combined disparity, such as from 1 / 8 resolution to 1 / 4 resolution (box 3014).

[0040] In such as Figure 9 and Figure 10 In the exemplary embodiments generally shown, the ECU may perform additional iterations, such as until full resolution is obtained. In these additional iterations, the ECU 124 may utilize disparity information from the previous iteration to shift pixels of the second image and any additional images, and determine the remaining disparity relative to the first image (boxes 3020, 3030). The image may be scaled down before determining the remaining disparity. For example, boxes 3002 and / or 3004 may be repeated, and different scaling factors may be applied, such as for the second iteration (…). Figure 9 Apply 1 / 4 instead of 1 / 8, and / or for the third iteration ( Figure 10 Use 1 / 2 instead of 1 / 8.

[0041] In an illustrative embodiment, ECU 124 may use the remaining disparity to modify the disparity information from the previous iteration. ECU 124 may then combine, filter, and / or amplify the modified disparity information in the same or similar manner as the first iteration (boxes 3022, 3024, 3026 and boxes 3032, 3034, 3036, which may be done in a manner similar to boxes 3010 to 3014). The modified / enhanced disparity information from the last iteration (e.g., full resolution) may be the output of method 3000. In some cases, the modified / enhanced disparity information may be converted into depth information and / or an enhanced depth map 134, such as by using a conversion or stretching factor used in box 3008 to convert the depth information from the second electronic sensor 122 into disparity. ECU 124 may utilize the modified / enhanced disparity / depth map 134 to generate a final image 136 (e.g., see box 1018).

[0042] In an illustrative embodiment, ECU 124 can be configured to generate depth estimates 142 using a neural network. The neural network can generate depth estimates based on input from one or more RGB images 150. In some cases, the neural network depth estimate may be consistent across objects in a scene, but it may report incorrect depth values ​​for different parts of the same object. For example, a tree at 3 meters (9.84 feet) might be reported by the network (e.g., incorrectly) as being at 2 meters (6.56 feet), but it may report 2 meters across all pixels located within that tree. ECU 124 may use depth data from a second electronic sensor 122 (e.g., a ToF sensor, a parallax sensor, and / or a PDAF sensor) to assign real-world depth values ​​to the next neighbor (NN) on a per-object basis.

[0043] In the example, an ECU (e.g., ECU 124) may include an electronic controller and / or an electronic processor, such as a programmable microprocessor and / or a microcontroller. In embodiments, the ECU may include, for example, an application-specific integrated circuit (ASIC). The ECU may include a central processing unit (CPU), memory (e.g., a non-transitory computer-readable storage medium), and / or input / output (I / O) interfaces. The ECU may be configured to perform various functions, including those described in more detail herein, using appropriate programming instructions and / or code embodied in software, hardware, and / or other media. In embodiments, the ECU may include multiple controllers. In embodiments, the ECU may be connected to a display, such as a touchscreen display.

[0044] Various examples / embodiments of various apparatuses, systems, and / or methods are described herein. Numerous specific details are set forth to provide a thorough understanding of the overall structure, function, manufacture, and use of the examples / embodiments described in the specification and shown in the accompanying drawings. However, those skilled in the art will understand that the examples / embodiments can be practiced without such specific details. In other instances, well-known operations, components, and elements have not been described in detail so as not to obscure the examples / embodiments described in the specification. Those skilled in the art will understand that the examples / embodiments described and illustrated herein are non-limiting examples, and therefore it can be understood that the specific structural and functional details disclosed herein may be representative and do not necessarily limit the scope of the embodiments.

[0045] Throughout this specification, references to “example,” “in an example,” “using an example,” “various embodiments,” “using an embodiment,” “in an embodiment,” or “embodiment,” etc., mean that a particular feature, structure, or characteristic described in connection with an example / embodiment is included in at least one embodiment. Therefore, the appearance of the phrases “example,” “in an example,” “using an example,” “in various embodiments,” “using an embodiment,” “in an embodiment,” or “embodiment,” etc., in their appropriate places throughout this specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic can be combined in any suitable manner in one or more examples / embodiments. Therefore, a particular feature, structure, or characteristic illustrated or described in connection with one embodiment / example can be combined, in whole or in part, with features, structures, functions, and / or characteristics of one or more other embodiments / examples, without limitation, provided that such combination is not illogical or nonfunctional. Moreover, many modifications can be made to adapt a particular situation or material to the teachings of this disclosure without departing from the scope of this disclosure.

[0046] It should be understood that references to a single element are not necessarily so limiting and may include one or more such elements. Any directional references (e.g., addition, subtraction, up, down, upward, downward, left, right, left to right, top, bottom, above, below, vertical, horizontal, clockwise, and counterclockwise) are used for identification purposes only to aid the reader's understanding of this disclosure and do not impose limitations, particularly with respect to the location, orientation, or use of the examples / embodiments.

[0047] The term "attachment" (e.g., attachment, coupling, connection, etc.) should be interpreted broadly and may include intermediate components between connected elements, relative movement between elements, direct connection, indirect connection, fixed connection, movable connection, operative connection, indirect contact, and / or direct contact. Therefore, an engagement reference does not necessarily imply that two elements are directly connected / coupled and in a fixed relationship with each other. Connections of electrical components (if any) may include mechanical connections, electrical connections, wired connections, and / or wireless connections, etc. The use of "e.g." and "such as" in the specification should be interpreted broadly and is intended to provide non-limiting examples of embodiments of this disclosure, and this disclosure is not limited to such examples. The use of "and" and "or" should be interpreted broadly (e.g., should be considered as "and / or"). For example, but not limited to, the use of "and" does not necessarily require listing all elements or features, and the use of "or" is inclusive unless such a construction is illogical.

[0048] While processes, systems, and methods may be described herein by combining one or more steps in a particular order, it should be understood that this approach may be practiced with steps in different orders, with some steps performed simultaneously, with additional steps, and / or with some steps omitted.

[0049] All matters contained in the foregoing description or shown in the accompanying drawings should be interpreted as illustrative only, not restrictive. Changes may be made to details or structure without departing from this disclosure.

[0050] It should be understood that the computer / computing device, electronic control unit (ECU), system, and / or processor described herein may include conventional processing devices known in the art, capable of executing pre-programmed instructions stored in associated memory, all of which perform according to the functions described herein. With regard to the methods described herein being embodied in software, the resulting software may be stored in associated memory and may also constitute a means for performing such methods. Such a system or processor may further be of the type of ROM, RAM, RAM and ROM, and / or a combination of non-volatile and volatile memory, so that any software can be stored, and also allowing the storage and processing of dynamically generated data and / or signals.

[0051] It should be further understood that the article of manufacture according to this disclosure may include a non-transitory computer-readable storage medium on which a computer program is encoded for implementing the logic and other functions described herein. The computer program may include code to perform one or more of the methods disclosed herein. Such embodiments may be configured to execute via one or more processors (such as multiple processors integrated into a single system, or multiple processors distributed across and connected together on a communication network), and the communication network may be wired and / or wireless. Code for implementing one or more of the features described in conjunction with one or more embodiments causes multiple transistors to change from a first state to a second state when executed by a processor. Specific changing patterns (e.g., which transistors change state and which do not change state) may be specified at least in part by logic and / or code.

Claims

1. A method for generating a digital image, the method comprising: Multiple images of the target are acquired over a period of time via the first electronic sensor; Select one or more pixels from the first image among the plurality of images; Identify a corresponding pixel in one or more other images among the plurality of images that corresponds to one or more selected pixels, wherein the one or more selected pixels and the corresponding pixel define a reference pixel set; Based on the reference pixel set, identify two or more images among the plurality of images that have the best parallax; Modified depth information is generated from two or more identified images; as well as A final digital image is generated using the modified depth information; The generated modified depth information includes: Reduce the size of two or more identified images; For two or more identified images, determine disparity information, the determination including: determining the disparity via scaled-down versions of the two or more identified images; The determined parallax information is combined with the depth information from the second electronic sensor; Amplify the parallax and depth information of the combination; Based on the magnified combined parallax and depth information, the pixels of the two or more identified images are shifted; Determine the remaining disparity information after the shift; Combine the remaining disparity information with the depth information; and The parallax and depth information of the remaining combinations are magnified; and The modified depth information includes the magnified parallax and depth information of the remaining combined data.

2. The method according to claim 1, wherein, Each of the selected one or more pixels includes associated position and color information.

3. The method according to claim 2, wherein, The associated location information includes two-dimensional location information.

4. The method according to claim 2, wherein, The associated location information includes three-dimensional location information.

5. The method according to claim 1, wherein, Identifying the two or more images includes: determining the two-dimensional parallax between corresponding reference pixel pairs.

6. The method according to claim 1, wherein, Identifying the two or more images includes determining the three-dimensional parallax between corresponding reference pixel pairs.

7. The method according to claim 1, wherein, The optimal parallax is the smallest parallax that can be used to distinguish depth levels.

8. The method according to claim 7, wherein, The optimal parallax is less than the maximum threshold corresponding to the expected error or fault.

9. The method according to claim 1, comprising: Depth information is obtained via the second electronic sensor; The depth information includes depth data and a confidence map; and The modified depth information is different from the depth information obtained via the second electronic sensor.

10. The method according to claim 1, wherein, The modified depth information includes a modified depth map and / or a modified disparity map.

11. The method according to claim 1, wherein, Generating the modified depth information includes filtering the combined disparity and depth information.

12. The method according to claim 1, wherein, The remaining disparity information is determined via the pixel kernel.

13. An imaging system, comprising: Mobile electronic devices, including: First electronic sensor; The second electronic sensor; and Electronic control unit; A printer configured to receive information from the mobile electronic device; The electronic control unit is configured as follows: Multiple images of the target are acquired within a time period via the first electronic sensor; Select one or more pixels from the first image among the plurality of images; Identify a corresponding pixel in one or more other images among the plurality of images that corresponds to one or more selected pixels, wherein the one or more selected pixels and the corresponding pixel define a reference pixel set; Based on the reference pixel set, identify two or more images among the plurality of images that have the best parallax; Generate modified depth information; and A final digital image is generated using the modified depth information, wherein the final digital image is configured for printing via the printer; The generated modified depth information includes: Reduce the size of two or more identified images; For two or more identified images, determine disparity information, the determination including: determining the disparity via scaled-down versions of the two or more identified images; The determined parallax information is combined with the depth information from the second electronic sensor; Amplify the parallax and depth information of the combination; Based on the magnified combined parallax and depth information, the pixels of the two or more identified images are shifted; Determine the remaining disparity information after the shift; Combine the remaining disparity information with the depth information; and The parallax and depth information of the remaining combinations are magnified; and The modified depth information includes the magnified parallax and depth information of the remaining combined data.

14. A mobile electronic device, comprising: First electronic sensor; Second electronic sensor; as well as Electronic control unit; The electronic control unit is configured as follows: Multiple images of the target are acquired within a time period via the first electronic sensor; Select one or more pixels from the first image among the plurality of images; Identify a corresponding pixel in one or more other images among the plurality of images that corresponds to one or more selected pixels, wherein the one or more selected pixels and the corresponding pixel define a reference pixel set; Based on the reference pixel set, identify two or more images among the plurality of images that have the best parallax; Generate modified depth information; and A final digital image is generated using the modified depth information; The generated modified depth information includes: Reduce the size of two or more identified images; For two or more identified images, determine disparity information, the determination including: determining the disparity via scaled-down versions of the two or more identified images; The determined disparity information is combined with the depth information from the second electronic sensor; Amplify the parallax and depth information of the combination; Based on the magnified combined parallax and depth information, the pixels of the two or more identified images are shifted; Determine the remaining disparity information after the shift; Combine the remaining disparity information with the depth information; and The parallax and depth information of the remaining combinations are amplified; The modified depth information includes the magnified parallax and depth information of the remaining combined data.

Images