Image processing method, graphical interface and related device

Abstract

The invention discloses an image processing method, a graphical interface and a related device, an electronic device can acquire a first image through a first camera and acquire a second image through a second camera, the definition of a first object acquired in the first image is lower than that of a first object acquired in the second image, and then the first image is acquired through the first camera. According to the method, the second image can be spatially aligned to the first image, the first image is deblurred, and finally, the target image is obtained according to the first image, the aligned second image and the deblurred first image. The pixel points in the target image come from the pixel points, except the first object, collected in the first image, the pixel points, collected in the aligned second image, in the first object and the pixel points, collected in the deblurred first image. The target image is an image with a larger depth of field on the basis of the first image. Therefore, the depth of field expansion of the image can be realized through the method.

Description

Technical Field

[0001] The present application relates to the field of terminals, and in particular to image processing methods, graphical interfaces and related devices. Background Art

[0002] Depth of field (DOF) refers to the relatively clear imaging range of the camera in front of and behind the focal plane, which is usually determined by the object distance, lens focal length, and lens aperture value. In terminal devices, such as mobile phones, since the aperture and lens focal length cannot be adjusted, the image captured by the main camera may only have part of the content relatively clear, and the rest of the content is usually blurred. For example, when shooting a scene that includes a foreground and background, no matter how the focus position is adjusted, it is difficult to ensure that all objects in the scene are captured clearly in one image. Summary of the invention

[0003] The present application provides an image processing method, a graphical interface and related devices, which realize the depth of field expansion of images using images captured by multiple cameras.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device comprising a first camera and a second camera, the method comprising: the electronic device captures a first image through the first camera, and captures a second image through the second camera, the clarity of the first object captured in the first image is lower than the clarity of the first object captured in the second image; the electronic device spatially aligns the second image with the first image to obtain an aligned second image; the electronic device deblurs the first image to obtain a deblurred first image; the electronic device obtains a target image based on the first image, the aligned second image, and the deblurred first image, the pixel points in the icon image are derived from the pixel points other than the first object captured in the first image, the pixel points in the first object captured in the aligned second image, and the pixel points in the deblurred first image.

[0005] By implementing the method provided in the first aspect, images captured by multiple cameras can be used to achieve depth of field expansion of images. This method fully considers the characteristics of images captured by different cameras, and uses a clear picture captured by one camera to correct the blurred parts of the image captured by another camera, thereby greatly expanding the depth of field of the image captured by the camera and improving the image quality. In addition, this method also fully considers the physical position differences of different cameras, combines the aligned image and the deblurred image to expand the image depth of field, and improves the image quality of the image after the depth of field expansion as much as possible, so that the details in the target image are also clear and complete, achieving a more complete and high-quality image deblurring effect.

[0006] In combination with the first aspect, in one implementation, the electronic device deblurs the first image to obtain the deblurred first image, specifically including: the electronic device inputs the first image into a deblurring model to obtain the deblurred first image, and the deblurring model is a model trained using clear images and blurred images.

[0007] The deblurring model is used to realize image deblurring, which can quickly realize image deblurring.

[0008] In combination with the first aspect, in one implementation, the electronic device obtains a target image based on the first image, the aligned second image and the deblurred first image, specifically including: the electronic device obtains a mask map of the first image, a mask map of the aligned second image, and a mask map of the deblurred first image based on the first image, the aligned second image and the deblurred first image, the mask map being used to indicate pixels retained in the image; the electronic device generates a target image based on the mask map of the first image, the mask map of the aligned second image and the mask map of the deblurred first image.

[0009] It can be seen that by selecting the best pixels on different images to stitch together the target image, the target image can fuse the clearest pixels in each image, thereby achieving the extension of the depth of field of the image.

[0010] In combination with the first aspect, in one implementation, the mask map of the first image, the mask map of the aligned second image, and the mask map of the deblurred first image are data obtained after inputting the first image, the aligned second image, and the deblurred first image into a fusion model. The fusion model is trained using multiple sets of data, wherein one set of data includes a shallow depth of field image, a large depth of field image, and an image after deblurring the shallow depth of field image captured by a camera, and an image aligned to the shallow depth of field image captured by another camera.

[0011] In combination with the first aspect, in one implementation, an electronic device obtains a target image based on a first image, an aligned second image, and a deblurred first image, specifically including: the electronic device downsamples the first image, the aligned second image, and the deblurred first image at a first magnification to obtain a downsampled image; the electronic device obtains a mask image based on the downsampled image; the electronic device upsamples the mask image at a first magnification to obtain an upsampled mask image, and the upsampled mask image is used to indicate pixels retained in the first image, the aligned second image, and the deblurred first image; the electronic device generates a target image based on the mask image, the first image, the aligned second image, and the deblurred first image.

[0012] Downsampling an image allows an electronic device to generate a target image using an image with a lower resolution, thereby reducing the amount of computation of the electronic device and speeding up the speed at which the electronic device generates the target image.

[0013] In a second aspect, an embodiment of the present application also provides an image processing method, which is applied to an electronic device comprising a first camera and a second camera, the method comprising: the electronic device captures a first image through the first camera, and captures a second image through the second camera, the clarity of the first object captured in the first image is lower than the clarity of the first object captured in the second image; the electronic device spatially aligns the second image with the first image to obtain an aligned second image; the electronic device obtains a target image based on the first image and the aligned second image, the pixel points in the target image are derived from the pixel points other than the first object captured in the first image and the pixel points of the first object captured in the aligned second image.

[0014] By implementing the method provided in the first aspect, images captured by multiple cameras can be used to achieve depth of field expansion of images. This method fully considers the characteristics of images captured by different cameras, and uses a clear picture captured by one camera to correct the blurred parts of the image captured by another camera, thereby greatly expanding the depth of field of the image captured by the camera and improving the image quality. In addition, this method also fully considers the physical position differences of different cameras, and combines the aligned images to expand the image depth of field. While achieving the image depth of field expansion, it minimizes the amount of equipment calculation as much as possible, so that the electronic device can output the target image as soon as possible, and the details in the target image are also clear and complete, achieving a more complete and high-quality image deblurring effect.

[0015] In combination with the first aspect and the second aspect, in one implementation, before the electronic device captures a first image through the first camera and captures a second image through the second camera, the method also includes: the electronic device detects a first operation, and the first operation is used to trigger the electronic device to capture images through the first camera and the second camera.

[0016] In this way, the user's operations during shooting are reduced, so that the user can obtain a clear image with only one shooting operation. In addition, the user does not need to repeatedly adjust the focus position during shooting. After locking the shooting scene, the shooting operation can be performed quickly, which simplifies the user operation.

[0017] In combination with the first aspect and the second aspect, in one implementation, before the electronic device detects the first operation, the method further includes: the electronic device displays a preview interface, where the preview interface is used to display an image captured by the first camera.

[0018] In this way, the user can lock the picture the user wants to shoot through the image captured by the camera, thereby expanding the depth of field of the picture the user wants to shoot.

[0019] In combination with the first aspect and the second aspect, in one implementation, the preview interface includes one or more shooting mode options, and before the electronic device detects the first operation, the method also includes: the electronic device detects a second operation acting on a first shooting mode option among the one or more shooting mode options.

[0020] That is to say, the image processing method provided in the embodiment of the present application can be presented as a shooting mode, and the user can achieve the shooting of a large depth of field image by switching to this shooting mode.

[0021] In combination with the first aspect and the second aspect, in one implementation, the aligned second image is a globally aligned second image, and the electronic device spatially aligns the second image with the first image to obtain the aligned second image, specifically including: the electronic device calculates the homography matrix using matching feature points in the first image and the second image; the electronic device converts the second image into a globally aligned second image based on the homography matrix and the first image as a reference.

[0022] By using feature points to align the second image with the first image, the overall pixel points of the second image can be aligned with the overall pixel points of the first image, reducing the difference in viewing angles between the first image and the second image due to the fact that they are captured by different cameras.

[0023] In combination with the first aspect and the second aspect, in one implementation, the aligned second image is a locally aligned second image, and the electronic device spatially aligns the second image to the first image to obtain the aligned second image, specifically including: the electronic device calculates the homography matrix using matching feature points in the first image and the second image; the electronic device converts the second image into a globally aligned second image based on the homography matrix and with the first image as a reference; the electronic device converts the globally aligned second image into a locally aligned second image based on the optical flow offset between the first image and the globally aligned second image and with the first image as a reference.

[0024] Combining the optical flow method and feature point matching to achieve alignment between the second image and the first image can achieve pixel alignment between the second image and the first image as much as possible, and minimize the difference in viewing angle between the first image and the second image due to the fact that they are collected by different cameras.

[0025] In combination with the first aspect and the second aspect, in one implementation, after the electronic device converts the second image into a globally aligned second image according to the homography matrix, the method also includes: the electronic device performs color alignment on the globally aligned second image with the first image according to the first image to obtain the color-aligned second image; the electronic device converts the globally aligned second image into a locally aligned second image based on the optical flow offset between the first image and the globally aligned second image and with the first image as a reference, specifically including: the electronic device converts the color-aligned second image into a locally aligned second image based on the optical flow offset between the first image and the color-aligned second image and with the first image as a reference.

[0026] Performing color alignment on the second image with the first image can make the hue and brightness of the first image and the second image consistent, ensuring the overall hue of the target image is consistent. In addition, performing color alignment before performing local alignment can improve the accuracy of local alignment.

[0027] In combination with the first aspect and the second aspect, in one implementation, the electronic device obtains a target image based on the first image and the aligned second image, specifically including: the electronic device performs color alignment on the aligned second image with the first image based on the first image to obtain the color-aligned second image; the electronic device obtains the target image based on the first image and the color-aligned second image, the target image including pixel points collected in the first image except for the first object and pixel points of the first object collected in the color-aligned second image.

[0028] Before generating the target image, the aligned second image is color-aligned with the first image, so that the hue and brightness of the first image and the aligned second image are kept consistent, thereby ensuring the overall hue of the target image is consistent.

[0029] In combination with the first aspect and the second aspect, in one implementation, the electronic device performs color alignment on the aligned second image with the first image based on the first image to obtain the color-aligned second image, specifically including: the electronic device adjusts the color of the aligned second image based on the color of the first image as a reference based on the histogram of the first image and the aligned second image to obtain the color-aligned second image.

[0030] In combination with the first aspect and the second aspect, in one implementation, the first camera is a main camera, and the second camera includes any one or more of a wide-angle camera, an ultra-wide-angle camera, or a telephoto camera.

[0031] In combination with the first aspect and the second aspect, in one implementation, the method further includes: the electronic device displays the target image.

[0032] In this way, after executing the shooting operation, the user can quickly view the large depth of field picture obtained for the current shooting scene, or the electronic device can display the target image through the gallery application after detecting the user's operation of entering the gallery to view. In this way, the user can choose whether to continue shooting pictures or browse the currently obtained target image.

[0033] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, one or more processors, and one or more programs; when the one or more processors execute the one or more programs, the electronic device implements the method described in the first aspect or any one of the implementations in the first aspect, the second aspect or any one of the implementations in the second aspect.

[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on an electronic device, cause the electronic device to execute a method as described in the first aspect or any one of the implementations of the first aspect, or the second aspect or any one of the implementations of the second aspect.

[0035] In the fifth aspect, an embodiment of the present application provides a computer program product, characterized in that when the computer program product runs on a computer, the computer executes the method described in the first aspect or any one of the implementations of the first aspect, the second aspect or any one of the implementations of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a shooting scene provided in an embodiment of the present application;

[0037] Figure 2 It is a schematic diagram of the principle of multi-focus fusion algorithm in image fusion;

[0038] Figure 3 A schematic diagram of adjusting an image using a traditional deblurring algorithm;

[0039] Figure 4A A schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of the present application;

[0040] Figure 4B A schematic diagram of the back side of an electronic device 100 provided in an embodiment of the present application;

[0041] Figure 5 A software structure block diagram of the electronic device 100 provided in an embodiment of the present application;

[0042] Figure 6A-6F Some user interfaces provided for embodiments of the present application;

[0043] Figure 7A flowchart of an image processing method provided in an embodiment of the present application;

[0044] Figure 8 A schematic diagram of an image collected by multiple cameras by an electronic device 100 provided in an embodiment of the present application;

[0045] Fig. 9 A schematic diagram of the process of global alignment of images provided in an embodiment of the present application;

[0046] Fig.10 A schematic diagram of images obtained by photographing the same object from different viewing angles provided in an embodiment of the present application;

[0047] Fig.11 A schematic diagram of the process of local alignment of images provided in an embodiment of the present application;

[0048] Fig.12 A schematic diagram of the alignment effect of an image only after global alignment and an image after global alignment and local alignment provided in an embodiment of the present application;

[0049] Fig.13 A schematic diagram of the process of image color alignment provided in an embodiment of the present application;

[0050] Fig.14 A schematic diagram of a histogram involved in color alignment of a first image and a second image provided in an embodiment of the present application;

[0051] Fig.15 A schematic diagram of the principle of image fusion provided in an embodiment of the present application;

[0052] Fig.16 A flowchart of another image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0054] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0055] The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and finally presented as content that can be recognized by the user. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0056] Figure 1 A schematic diagram of a shooting scene provided in an embodiment of the present application.

[0057] like Figure 1 As shown, the shooting scene usually includes multiple objects placed in front and behind, and these objects are at different object distances from the camera of the electronic device 100, resulting in that when the electronic device 100 uses the camera to shoot, it is difficult to ensure that the foreground and background objects are clearly imaged in the same image at the same time.

[0058] In order to make the images taken by electronic devices present an overall clear picture, there are three algorithms:

[0059] 1) Multi-focus fusion algorithm

[0060] In order to make all objects in the image clear, you can shoot multiple times, selecting a different focus position each time, so as to obtain multiple images with different parts captured clearly, and then merge these multiple images, "stitching" the clear parts of these multiple images together to synthesize a fully clear image.

[0061] For example, Figure 1 Take the image captured under the shooting scene shown as an example, Figure 2 A schematic diagram showing the principle of multi-focus fusion algorithm in image fusion.

[0062] like Figure 2 As shown, two images can be taken, one of which focuses on the foreground to make the foreground clear, and the other focuses on the background to make the background clear, and then the two images are fused to obtain an image with a clear foreground and a clear background.

[0063] However, using this algorithm to restore a clear image requires the user to adjust the focus position during the shooting process and take multiple images, which makes the user's operation cumbersome. In addition, during the shooting process, the user will see repeated focusing images in the shooting preview interface, which affects the user's preview effect and reduces the user's experience.

[0064] 2) Traditional deblurring algorithm

[0065] Considering that image blur is usually caused by user shaking during the shooting process, traditional deblurring algorithms use blind deconvolution to estimate the user's shaking trajectory at the moment of shooting the image, and then use the shaking trajectory to reconstruct a clear image.

[0066] However, this method is suitable for the situation where the overall image is blurred due to user jitter, and is not very effective for adjusting the image mentioned in this application where only part of the object is unclear, and the details of the adjusted image may still be relatively blurred.

[0067] For example, Figure 1 Take the image captured under the shooting scene shown as an example, Figure 3 A schematic diagram showing the use of a traditional deblurring algorithm to adjust an image.

[0068] in, Figure 3 The left side in the middle shows an image obtained by adjusting an image in which only part of the object is clear using a traditional deblurring algorithm. From the local comparison diagram on the right, it can be seen that although the adjusted picture is clearer than the picture before adjustment, the overall adjustment effect is not ideal and there are still relatively blurred parts in the picture.

[0069] 3) Deep learning deblurring algorithm

[0070] This method uses a dataset containing clear images and blurred images for training, extracts the features of blurred images relative to clear images, and obtains a trained model. After that, by inputting the blurred image to be processed into the model, a clear restored image can be output.

[0071] Among them, since the clear image and the blurred image are images of the same scene, if the clear image and the blurred image are taken separately, there may be shooting differences between the clear image and the blurred image due to differences in factors such as the time and location of the two shots, which in turn affects the effect of model training. Therefore, in the deep learning deblurring algorithm, the blurred image in the training data is the image obtained by blurring the clear image.

[0072] It can be seen that although this method can blur the image, it only uses the image taken by one camera, does not combine the multi-camera system to achieve image deblurring, and the adjustment effect is not ideal.

[0073] An embodiment of the present application provides an image processing method, which is applied to an electronic device including a first camera and a second camera. In the method, the electronic device can shoot the same scene through the first camera and the second camera to obtain a first image captured by the first camera and a second image captured by the second camera, wherein the clarity of the first object captured in the first image is lower than the clarity of the first object captured in the second image. After that, the second image can be spatially aligned with the first image to obtain an aligned second image, and the first image can be deblurred to obtain a deblurred first image. Finally, a target image is obtained according to the first image, the aligned second image, and the deblurred first image, wherein the pixels in the target image are derived from the pixels other than the first object captured in the first image, the pixels in the first object captured in the aligned second image, and the pixels in the deblurred first image.

[0074] Clarity is the sharpness of changes in the edges of image details. At the edges of image details, the sharper (faster) and more dramatic (higher contrast) the changes in optical density or brightness with position, the clearer the edges of the details are and the more recognizable they are.

[0075] The first camera may refer to a main camera, and the second camera may include any one or more of a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, etc. Optionally, the pixel of the first camera may be higher than the pixel of the second camera, so that the pixel value of the first image may be higher than the pixel value of the second image.

[0076] If the second camera is a wide-angle camera or an ultra-wide-angle camera, the equivalent focal length of the first camera is smaller than the equivalent focal length of the second camera, and the aperture of the first camera is larger than the aperture of the second camera, so the depth of field of the first image is shallower than that of the second image. In layman's terms, the clear range of the first image is smaller than the clear range of the second image. In this way, the clearer image in the second image can be combined to replace the blurrier image in the first image, ensuring that the details of the image are also clear, thereby expanding the depth of field of the first image and improving the deblurring effect of the image.

[0077] If the second camera is a telephoto camera, the focal length of the first camera is shorter than that of the second camera, and the second camera can capture details of the object relative to the first camera, so the picture captured by the second image can include the details of the picture captured by the first image. In this way, the details captured in the second image can be combined to replace the blurry picture in the first image, so as to achieve the depth of field extension of the first image, ensure that the details of the image are also clear, and improve the deblurring effect of the image.

[0078] In addition, due to the difference in physical positions between the first camera and the second camera, there may be a difference in viewing angles between the first image and the second image. Therefore, before using the second image, the second image can be spatially aligned with the first image, and then the aligned second image can be used to expand the depth of field of the first image, so that the blurred areas in the first image become clear, thereby achieving the effect of expanding the depth of field.

[0079] Furthermore, it is also taken into consideration that the aligned second image may have artifacts. Therefore, in order to avoid the artifacts affecting the image quality of the target image and causing poor deblurring effect of the target image, the method further combines the deblurred first image to solve the artifact problem, so that the target image is clear and free of artifacts, further improving the deblurring effect and enhancing the image quality.

[0080] In addition, before the electronic device captures an image through the first camera and the second camera, the electronic device can detect the user's shooting operation, and trigger the first camera and the second camera to capture an image through the shooting operation. In this way, the user's operations during the shooting process are reduced, so that the user can obtain a fully clear image with only one shooting operation. In addition, during the shooting process, the user does not need to repeatedly adjust the focus position. After locking the shooting scene, the shooting operation can be quickly performed, which simplifies the user operation.

[0081] An embodiment of the present application also provides an image processing method, which is applied to an electronic device including a first camera and a second camera. In the method, the electronic device can shoot the same scene through the first camera and the second camera to obtain a first image captured by the first camera and a second image captured by the second camera, wherein the clarity of the first object captured in the first image is lower than the clarity of the first object captured in the second image. Thereafter, the second image can be spatially aligned with the first image to obtain an aligned second image. Finally, a target image is obtained based on the first image and the aligned second image, wherein the pixels in the target image are derived from the pixels other than the first object captured in the first image and the pixels in the first object captured in the aligned second image.

[0082] In general, the image processing method provided in the embodiment of the present application utilizes images captured by multiple cameras in a multi-camera system to achieve depth of field expansion of images. The method fully considers the characteristics of images captured by different cameras, and uses the clear picture captured by one camera to correct the blurred parts of the image captured by another camera, greatly expanding the depth of field of the image captured by the camera and improving the image quality. In addition, the method also fully considers the physical position differences of different cameras. The former image processing method combines the aligned image and the deblurred image to expand the image depth of field, and improves the image quality of the image after the depth of field expansion as much as possible. The latter image processing method combines the aligned image to expand the image depth of field. While achieving the image depth of field expansion, it reduces the amount of equipment calculation as much as possible, so that the electronic device can output the target image as soon as possible. In summary, the image processing method provided in the embodiment of the present application can make the details in the final target image clear and complete, and achieve a more complete and high-quality image deblurring effect.

[0083] Figure 4A A schematic diagram of the hardware structure of the electronic device 100 is shown.

[0084] The electronic device 100 may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, an in-vehicle device, a smart home device and / or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific type of the electronic device.

[0085] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0086] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0087] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0088] In some embodiments, the processor 110 can be used to align multiple images captured by the camera 193, and deblur one of the images that requires depth of field expansion (hereinafter referred to as the original image), fuse the deblurred original image, the aligned image, and the original image to achieve depth of field expansion for the original image and obtain the target image.

[0089] In some other embodiments, the processor 110 may be used to align multiple images captured by the camera 193, fuse the deblurred original images, and the aligned images to extend the depth of field of the original images to obtain a target image.

[0090] For detailed descriptions on image alignment, deblurring, and image fusion, please refer to the subsequent content, which will not be expanded here.

[0091] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0092] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0093] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0094] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0095] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0096] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0097] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0098] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0099] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, demodulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0100] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0101] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0102] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), miniled, microled, micro-oled, quantum dot light emitting diodes (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0103] In some embodiments, the display screen 194 can be used to display a shooting preview interface and an image captured by the electronic device 100, as well as a target image obtained by processing the captured image. Figure 6A-6F .

[0104] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0105] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0106] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0107] In some embodiments, the camera 193 may include multiple cameras, such as a first camera and a second camera, and the electronic device 100 may collect images through the first camera and the second camera. Exemplarily, the electronic device 100 may collect images through the first camera and the second camera after detecting a user's shooting operation.

[0108] For example, Figure 4B A schematic diagram of the back side of the electronic device 100 provided in an embodiment of the present application.

[0109] like Figure 4B As shown, in the electronic device 100, the camera 193 may include a camera 193-1 and a camera 193-2. Among them, the camera 193-2 is located below the camera 193-1. Exemplarily, the camera 193-1 may be a main camera, and the camera 193-2 may be a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, etc. Among them, when the camera 193-1 is the main camera, after the electronic device 100 starts the camera application, the electronic device 100 may display the image captured by the camera 193-1 in the shooting preview interface by default.

[0110] Understandably, Figure 4B This is just an illustrative example, and the electronic device 100 may also include more cameras, which is not limited in the embodiments of the present application.

[0111] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0112] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0113] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0114] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0115] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data.

[0116] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110 .

[0117] In some embodiments, the internal memory 121 may be used to store images captured by the camera, algorithms for implementing image alignment and image deblurring, and images obtained by processing the images captured by the camera using these algorithms, and the like.

[0118] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are stored in the external non-volatile memory.

[0119] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0120] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0121] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0122] In some embodiments, the electronic device 100 can detect the user's shooting operation through the touch sensor 180K, and through the shooting operation, start the camera to capture images.

[0123] The electronic device may be a portable terminal device equipped with iOS, Android, Microsoft or other operating systems, such as a mobile phone, a tablet computer, a wearable device, etc., and may also be a non-portable terminal device such as a laptop computer with a touch-sensitive surface or a touch panel, a desktop computer with a touch-sensitive surface or a touch panel. The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0124] Figure 5 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0125] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0126] The application layer can include a series of application packages.

[0127] like Figure 5 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0128] The camera application is an image capturing application on electronic devices such as smart phones, tablet computers, etc. In the embodiment of the present application, the electronic device 100 can start multiple cameras to capture images through a single shooting operation by the user during the running of the camera application, and use the images captured by other cameras (such as the second camera) to extend the depth of field of the image captured by one of the cameras (such as the first camera), thereby obtaining an image with a larger depth of field.

[0129] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0130] like Figure 5 As shown, the application framework layer may include an image adjustment module, a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0131] The image adjustment module may include: a spatial alignment module, a color alignment module, a deblurring module, and a fusion module. Among them:

[0132] The spatial alignment module can be used to spatially align images captured by multiple cameras, that is, by finding a spatial transformation to map one image to another image, so that the points corresponding to the same position in the space in the two images correspond one to one, thereby achieving the purpose of information fusion and facilitating subsequent image fusion processing. Specifically, if the electronic device 100 includes a first camera and a second camera, the spatial alignment module can be used to spatially align the second image captured by the second camera to the first image captured by the first camera to obtain the second image after spatial alignment. Among them, the spatial alignment can include global alignment, which can refer to feature-based image alignment, and the global alignment of the second image to the first image can be achieved by calculating the homography matrix of the second image to the first image. The homography matrix describes the position mapping relationship between the object in the world coordinate system and the pixel coordinate system. Further optionally, the spatial alignment can also include local alignment, which can refer to image alignment based on optical flow, and the local alignment of the second image to the first image can be achieved by calculating the optical flow offset of the second image to the first image after global alignment.

[0133] The color alignment module can be used to perform color alignment on images (such as the first image and the second image) captured by multiple cameras, so that the hue and brightness of the images captured by the multiple cameras remain consistent. Among them, histogram processing technology can be used to achieve color alignment of multiple images.

[0134] The deblurring module may be used to deblur an image (eg, the first image). The deblurring of the image may be achieved by using a traditional deblurring algorithm, a deep learning deblurring algorithm, or the like.

[0135] The fusion module can be used to fuse the original image (eg, the first image), the aligned image (eg, the aligned second image), and the image obtained by deblurring the original image (eg, the deblurred first image) to obtain a target image.

[0136] In some embodiments, the fusion mode can be used to fuse the original image (eg, the first image) and the aligned image (eg, the aligned second image) to obtain a target image. It is understood that in this embodiment, the deblurring module is an optional step.

[0137] It is understandable that the image adjustment module may include more or fewer modules, and the embodiments of the present application are not limited to this.

[0138] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0139] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0140] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0141] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0142] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0143] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0144] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0145] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0146] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0147] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0148] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0149] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0150] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0151] A 2D graphics engine is a drawing engine for 2D drawings.

[0152] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0153] The following is an illustrative description of the workflow of the software and hardware of the electronic device 100 in conjunction with capturing a photo scene.

[0154] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0155] Figure 6A-6F Some user interfaces provided for embodiments of the present application.

[0156] Fig. 6A An exemplary user interface 21 for an application menu on the electronic device 100 is exemplarily shown.

[0157] like Fig. 6A As shown, the user interface 21 may include: a status bar 211, a calendar and weather indicator 212, and one or more application icons 213. Among them:

[0158] Status bar 211 may include one or more signal strength indicators of mobile communication signals, one or more signal strength indicators of wireless fidelity (WiFi) signals, a battery status indicator, and a time indicator. Calendar and weather indicator 212 may be used to indicate the current time and weather type. One or more application icons 213 may be used to display icons of one or more applications. For example, see Fig. 6A , one or more application icons 213 may include: a gallery icon 213A and a camera icon 213B. The gallery icon 213A is an icon of a gallery application, which is an application for managing pictures and videos on electronic devices such as smartphones and tablets, and can also be called an "album". The camera icon 213B is an icon of a camera application, which is an application for taking images on electronic devices such as smartphones and tablets. The present embodiment of the application does not limit the names of the gallery application and the camera application.

[0159] like Fig. 6A As shown, the electronic device 100 can detect a user operation on the camera icon 213B, such as a click operation. In response to the operation, the electronic device 100 can open the camera application and display the following information: Figure 6B The user interface 31 shown may be a default photo-taking interface provided by a camera application.

[0160] It is understandable that the electronic device 100 can also open the camera application in other ways. For example, the electronic device 100 can detect the user's voice command "open camera" to trigger the opening of the camera application. The embodiment of the present application does not limit the method of opening the camera application.

[0161] like Figure 6B As shown, the user interface 31 may include: a preview box 311, a camera mode option 312, a gallery shortcut control 313, a shutter control 314, and a camera flip control 315. Among them:

[0162] The preview frame 311 can be used to display the image captured by the camera 193, such as the camera 193-1, in real time. The electronic device 100 can refresh the display content therein in real time so that the user can preview the image currently captured by the camera 193.

[0163] One or more shooting mode options may be displayed in the camera mode option 312, and the one or more shooting mode options may include a portrait mode option 312A, a video mode option 312B, a photo mode option 312C, a large depth of field mode option 312D, and a more mode option 312E. When a user operation on a shooting mode option is detected, the electronic device 100 may turn on the shooting mode selected by the user. In particular, when a user operation on the more mode option 312E is detected, the electronic device 100 may further display other more shooting mode options, such as a slow motion shooting mode option, etc., so as to show the user more abundant shooting functions. Not limited to Figure 6B As shown, the camera mode option 312 may not display the more mode option 312E, and the electronic device 100 may detect the user's sliding operation to the left / right in the camera mode option 312 to display other more shooting mode options.

[0164] Among them, the large depth of field mode option 312D can be used to switch the current shooting mode to the large depth of field mode. In the large depth of field mode, the electronic device 100 can convert the shallow depth of field image shot by the user into a large depth of field image to achieve the depth of field extension of the image. In a specific implementation, in the large depth of field mode, after detecting the user's shooting operation, the electronic device 100 can trigger multiple cameras, such as camera 193-1 and camera 193-2 to shoot images, thereby obtaining multiple images, and then the electronic device 100 obtains an image with a large depth of field through these multiple images.

[0165] For details about the relevant user interface in the large depth of field mode, please refer to the subsequent content.

[0166] It is understandable that it is not limited to Figure 6BThe camera mode option 312 shown may include more or fewer shooting mode options. In addition, the embodiment of the present application does not limit the name of the large depth of field mode.

[0167] The gallery shortcut control 313 can be used to open the gallery application. In response to a user operation acting on the gallery shortcut control 313, such as a click operation, the electronic device 100 can open the gallery application. In this way, the user can easily view the captured pictures and videos without having to exit the camera application first and then open the gallery application. The gallery application can support users to perform various operations on pictures and videos stored on the electronic device 100, such as browsing, editing, deleting, selecting, etc. In addition, the electronic device 100 can also display thumbnails of the saved images in the gallery shortcut control 313.

[0168] The shutter control 314 may be used to monitor a user operation that triggers shooting. In response to the operation, the electronic device 100 may save the image in the preview box 311 as a picture in a gallery application, or start recording a video.

[0169] Furthermore, in the large depth of field mode, the electronic device 100 can also process pictures taken by the user through the shutter control 314, convert the shallow depth of field pictures taken into pictures with a large depth of field, and save them as pictures in the gallery application, or the electronic device 100 can also process the videos recorded by the user through the shutter control 314, convert the image frames with a shallow depth of field contained in the recorded video into image frames with a large depth of field.

[0170] The camera flip control 315 can be used to monitor the user operation that triggers the camera flip. In response to the operation, the electronic device 100 can flip the camera, for example, switch the rear camera to the front camera. By default, when the electronic device 100 enters the camera application, the electronic device 100 can start the rear camera by default, such as camera 193-1, and display the image captured by the camera in the preview box 311.

[0171] It is understandable that the user interface 31 is not limited to being opened by the camera application, and the user can also open the user interface 31 through other applications, for example, the user clicks the shooting icon in "WeChat" to open the user interface 31. Among them, "WeChat" is a social application that supports users to share photos or videos taken with others.

[0172] When the electronic device 100 detects Figure 6B In response to a user operation of the large depth of field mode option 312D shown, such as a click operation, the electronic device 100 switches to the large depth of field mode and displays the following Figure 6C The user interface 31 is shown.

[0173] In addition, it can be seen that Figure 6C The user interface 31 shown is an interface displayed in horizontal screen after the user changes the posture of holding the electronic device 100. It is understandable that the user can also display the user interface 31 in vertical screen without changing the posture of holding the electronic device 100, and take pictures or record videos in the large depth of field mode.

[0174] like Figure 6C As shown, in the large depth of field mode, the large depth of field mode option 312D is selected, and the electronic device 100 can display the image captured by the main camera, such as the camera 193-1, in real time in the preview box 311. Figure 1 Taking the shooting scene shown as an example, the preview frame 311 displays two objects placed front and back, and one object is clearer and the other is more blurred.

[0175] In addition, the electronic device 100 can also detect the user operation of changing the focus in the preview frame 311, change the intersection of the displayed picture in the preview frame 311, and then change the clarity of different objects in the picture.

[0176] like Figure 6C As shown, the electronic device 100 can detect a user operation on the shutter control 314 to take a photo or record a video. For example, the electronic device 100 can detect a click operation on the shutter control 314, and in response to the operation, trigger a photo, and take a picture through a main camera, such as camera 193-1. For another example, the electronic device 100 can detect a long press operation on the shutter control 314, and in response to the operation, start video recording, and record a video through a main camera, such as camera 193-1.

[0177] Furthermore, when the electronic device 100 takes a picture through the camera 193-1, the electronic device 100 can also take a picture through other cameras, such as the camera 193-2, at the same time, so that the electronic device 100 can obtain pictures collected by multiple cameras at the same time through a single photo operation. When the electronic device 100 records a video through the camera 193-1, the electronic device 100 can also record a video through other cameras, such as the camera 193-2, at the same time, so that the electronic device 100 can obtain videos recorded by multiple cameras at the same time through a single video recording operation.

[0178] In the internal implementation, the electronic device 100 can use the pictures taken by the camera 193-2 to process the pictures taken by the camera 193-1, convert the pictures with shallow depth of field taken by the camera 193-1 into pictures with large depth of field, and save the adjusted pictures to the gallery application; or, the electronic device 100 can use the image frames in the video recorded by the camera 193-2 to process the image frames in the video recorded by the camera 193-1, convert the image frames with shallow depth of field contained in the video recorded by the camera 193-1 into image frames with large depth of field, and save the adjusted video to the gallery.

[0179] In addition, in the large depth of field mode, the preview frame 311 can only display the image captured by the main camera, such as the camera 193-1, or, further optionally, the preview frame 311 can also display the image captured by other cameras, such as the camera 193-2.

[0180] Furthermore, the thumbnails displayed in the gallery shortcut control 313 may be updated to Fig.6D The thumbnails in the gallery shortcut control 313 are shown. Fig.6D As shown, the electronic device 100 can detect a user operation acting on the gallery shortcut control 313, such as a click operation. In response to the operation, the electronic device 100 can open the gallery application and display the following Fig. 6E The user interface 41 shown can be used to display a picture obtained after the electronic device 100 captures an image in a large depth of field mode.

[0181] like Fig. 6E As shown, the user interface 41 may include a preview box 411 and a thumbnail selection area 412. The preview box 411 may be used to display the image corresponding to the thumbnail selected in the thumbnail selection area 412, and the thumbnail selection area 412 may be used to display the image originally taken by the electronic device 100 in the large depth of field mode and the adjusted image. Exemplarily, the thumbnail selection area 412 may include thumbnails 412A, thumbnails 412B, and thumbnails 412C. Thumbnail 412A is a thumbnail of a large depth of field image 1 obtained by adjusting the image originally taken by the camera by the electronic device 100, and thumbnails 412B and thumbnails 412C are thumbnails of the images originally taken by the camera by the electronic device 100, wherein thumbnail 412B may be a thumbnail of image 2 taken by the electronic device 100 using the main camera, such as camera 193-1, and thumbnail 412C may be a thumbnail of image 3 taken by the electronic device 100 using another camera, such as camera 193-2. Among them, picture 1 is the picture obtained by adjusting picture 2 using picture 3. Fig. 6E The picture displayed in the preview box 411 is picture 1, and the current thumbnail 412A is in a selected state.

[0182] like Fig. 6E As shown, the electronic device 100 can detect the selection operation acting on the thumbnail 412B, select the thumbnail 412B, and display it in the preview box 411 as shown in FIG. Fig. 6F Picture 2 shown.

[0183] contrast Fig. 6E and Fig. 6F It can be seen that the depth of field of Picture 1 is greater than that of Picture 2. The overall picture of Picture 1 is clearer, while some content in the picture of Picture 2 is blurred.

[0184] It is understandable that the user interface 41 may not include the thumbnail selection area 412, and the electronic device 100 may only retain the adjusted pictures in the large depth of field mode, not retain the pictures taken by the camera, or only retain the pictures taken by the main camera, etc. In addition, if the electronic device 100 records a video in the large depth of field mode, the electronic device 100 may also play the adjusted video or the originally recorded video in the gallery, etc.

[0185] from Figure 6A-6F It can be seen that the method provided in the embodiment of the present application can be presented as a shooting mode in a camera application. In this shooting mode in the camera application, the electronic device 100 can trigger multiple cameras to capture images through a single shooting operation, and use the images taken by other cameras to adjust the image taken by one of the cameras, thereby improving the blurred parts of the image, achieving a better image depth of field expansion, and facilitating user operation.

[0186] The principles of the image processing method provided by the embodiments of the present application are described below.

[0187] Figure 7 A flowchart of an image processing method provided in an embodiment of the present application.

[0188] like Figure 7 As shown, the image processing method may include the following steps:

[0189] S101. The electronic device 100 detects a user's shooting operation.

[0190] The shooting operation can be used to trigger the electronic device 100 to capture images through the first camera and the second camera.

[0191] Before the electronic device 100 detects the user's shooting operation, the electronic device 100 may display a user interface provided by the first application, and the user interface may be used to display a camera, such as an image captured by the first camera. The electronic device 100 detecting the user's shooting operation may mean that the electronic device 100 detects the user's operation of the shooting control in the user interface. In this way, the user can lock the picture that the user wants to shoot through the image captured by the camera, and achieve depth of field expansion of the picture that the user wants to shoot.

[0192] It can be understood that the shooting operation may refer to a click operation performed by a user on the electronic device 100, or may refer to a voice operation performed by a user on the electronic device 100, and the embodiment of the present application does not limit the form of the shooting operation.

[0193] For example, the first application may refer to a camera application, or may refer to other applications including a shooting function. Taking the camera application as an example, the user interface may refer to Figure 6C The user interface 31 shown in FIG. Figure 6C Shutter control 314 is shown.

[0194] In addition, the user interface provided by the first application may include one or more shooting mode options. Before the electronic device 100 detects the user's shooting operation, it may also include the electronic device 100 detecting the user's operation of switching shooting modes, selecting the shooting mode option corresponding to the specified shooting mode, and the shooting operation is an operation initiated by the user in the specified shooting mode. Among them, the option corresponding to the specified shooting mode may refer to Figure 6B The large depth of field mode option 312D shown in the figure may be a switching operation of the shooting mode. Figure 6B User operations are shown acting on the extended depth of field mode option 312D.

[0195] In the embodiment of the present application, the shooting operation may also be referred to as the first operation, the shooting mode option corresponding to the specified shooting mode may also be referred to as the first shooting mode option, and the operation of switching the shooting mode may also be referred to as the second operation.

[0196] S102. The electronic device 100 captures a first image through the first camera and a second image through the second camera.

[0197] The electronic device 100 may include multiple cameras, such as a first camera, a second camera, and the like.

[0198] The focal lengths of the first camera and the second camera may be different, so that the objects included in the first image and the second image overlap but are not completely the same.

[0199] Specifically, the first camera may refer to a main camera, and the second camera may include any one or more of a wide-angle camera, an ultra-wide-angle camera, and a telephoto camera.

[0200] It is understandable that the second camera may include one or more cameras. If the second camera includes multiple cameras, the second image may include multiple images, and the electronic device 100 may combine the multiple images to improve the first image and expand the depth of field of the first image. The embodiment of the present application does not limit the number of cameras included in the second camera.

[0201] If the second camera is a wide-angle camera or an ultra-wide-angle camera, the depth of field of the first image is shallower than that of the second image, and since the focal length of the wide-angle camera is shorter and the field of view is wider, the second image may have perspective distortion. If the second camera is a telephoto camera, the image captured by the second image is the details of the object captured by the first image.

[0202] Optionally, the pixel of the first camera is higher than the pixel of the second camera, so that the pixel value of the first image can be higher than the pixel value of the second image.

[0203] It can be seen that by using the clear picture captured in the second image to improve the blurred picture captured in the first image, an image with a high pixel value and a large depth of field can be obtained.

[0204] It should be understood that the first camera and the second camera are located on the same side of the electronic device 100 and are relatively close to each other. In this way, after detecting the user's shooting operation, the first camera and the second camera can shoot the same object at the same time.

[0205] That is, the clarity of an object (e.g., the first object) captured in the first image is lower than the clarity of the object captured in the second image. In this way, the electronic device 100 can use the pixel points of the object captured in the second image to improve the pixel points of the object captured in the first image, thereby expanding the image clarity range in the first image and achieving the effect of extending the depth of field of the first image.

[0206] For example, Figure 5 Taking the electronic device 100 shown as an example, the first camera may refer to the camera 193 - 1 in the electronic device 100 , and the second camera may refer to the camera 193 - 2 in the electronic device 100 .

[0207] Figure 8 FIG. 1 shows a schematic diagram of images collected by the electronic device 100 through multiple cameras. Figure 8 The image shown in (a) may refer to the first image, Figure 8 The image shown in (b) may refer to the second image.

[0208] S103. The electronic device 100 aligns the second image with the first image to obtain an aligned second image.

[0209] Since the first image and the second image are images captured by different cameras, there is a difference in shooting angle of view and depth on the screen between the first image and the second image. Therefore, the second image needs to be spatially aligned with the first image so that the position, direction, and scale of the first image and the second image are consistent, which facilitates subsequent image fusion.

[0210] Further optionally, since the first image and the second image are images captured by different cameras, there may be color differences between the first image and the second image. Therefore, the second image can be color aligned with the first image so that the first image and the second image are consistent in color tone, which facilitates subsequent image fusion.

[0211] The spatial alignment may include: global alignment. Global alignment may refer to feature-based image alignment.

[0212] For example, Fig. 9 A schematic diagram of the process of global alignment of images provided in an embodiment of the present application.

[0213] Specifically, Fig. 9 As shown, the global alignment of the second image to the first image may include the following steps:

[0214] S201. The electronic device 100 extracts feature points of the first image and the second image.

[0215] Feature points refer to points where the grayscale value of an image changes dramatically or points with large curvature on the edge of an image (i.e., the intersection of two edges). Commonly used feature points include: corner points, edge points, spots, etc. Feature points can reflect the essential features of an image and identify the target object in the image.

[0216] In some embodiments, feature points in an image may be extracted through image texture information, image region grayscale statistics, or through frequency change, wavelet transform, and other transformations followed by spatial transformation. Exemplarily, feature points in an image may be extracted through a scale-invariant feature transform (SIFT) algorithm, a speeded up robust features (SURF) algorithm, or a feature detection (oriented FAST and rotatedBRIEF, ORB) algorithm.

[0217] S202. The electronic device 100 performs feature point matching on the feature points extracted from the first image and the second image.

[0218] If the same object in the scene is captured in the two images, the two images contain the same feature points. Performing feature point matching on the feature points extracted from the first image and the second image is to determine the same feature points in the first image and the second image.

[0219] For example, key point matching may be performed using a fast library for approximate nearest neighbors (FLANN) algorithm.

[0220] S203. The electronic device 100 calculates a homography matrix using the matched feature points.

[0221] The homography matrix reflects the position mapping relationship between the object in the world coordinate system and the pixel coordinate system.

[0222] Exemplarily, the homography matrix may be calculated by a random sample consensus (RSC) algorithm.

[0223] S204. The electronic device 100 uses the homography matrix to convert the second image into a globally aligned second image.

[0224] The homography matrix can be used to map pixels on the second image to the first image, and transform the second image into a globally aligned second image by rotating, displacing, scaling, etc., wherein the pixels in the globally aligned second image are globally registered with the pixels in the first image.

[0225] That is, the aligned second image may refer to a globally aligned second image. In this case, when the electronic device 100 aligns the second image to the first image, it may calculate the homography matrix using the matched feature points in the first image and the second image, and then convert the second image to the globally aligned second image based on the homography matrix and the first image as a reference, thereby achieving alignment of the second image to the first image.

[0226] In some embodiments, before calculating the homography matrix, the electronic device 100 may first downsample the first image and the second image, that is, reduce the resolution of the image, for example, reduce the resolution of the image to 1 / 4 of the original, and then use the first image and the second image with reduced resolution to calculate the homography matrix, and finally use the calculated homography matrix to globally align the second image with the original resolution to the first image to obtain the globally aligned second image. In other words, the images used in the above steps S201-S203 may be the first image and the second image with reduced resolution. In this way, the calculation efficiency can be improved as much as possible without affecting the accuracy of the calculation results, saving time and power consumption.

[0227] Further optionally, the spatial alignment may also include: local alignment. The local alignment may refer to image alignment based on optical flow.

[0228] This is because considering the difference in physical positions between the first camera and the second camera, the first image and the second image may have a difference in perspective, and using only global alignment may not be able to align all objects in space. Fig.10 For example, Fig.10 The figure shows a schematic diagram of images obtained by photographing the same object at different viewing angles.

[0229] in, Fig.10 (a) and (b) in the figure are images of the same object taken from the left and right perspectives, respectively. It can be seen that the position of the object in the image from the left perspective is further to the left than that from the right perspective.

[0230] Therefore, in order to achieve a more precise spatial alignment between the first image and the second image and improve the quality of image depth of field extension, the relative offset movement between the first camera and the second camera can be determined by calculating the optical flow of the first image and the second image to achieve local alignment of the second image to the first image.

[0231] Optical flow is the instantaneous speed of the pixels of a moving object in space on the observation imaging plane. The optical flow method uses the change of pixels in the image in the time domain and the correlation between images to find the correspondence between two images, thereby calculating the motion information of the object between the two images.

[0232] For example, Fig.11 A schematic diagram of the process of local alignment of images provided in an embodiment of the present application.

[0233] Specifically, Fig.11 As shown, locally aligning the second image to the first image may include the following steps:

[0234] S301. The electronic device 100 determines an optical flow according to the first image and the globally aligned second image.

[0235] Exemplarily, the optical flow between the first image and the globally aligned second image may be calculated using a gradient-based method, a matching-based method, an energy-based method, a phase-based method, and a neural dynamics method.

[0236] Taking the gradient-based method as an example, the spatiotemporal differential (i.e., the spatiotemporal gradient function) of the time-varying image grayscale (or its filtered form) can be used to calculate the velocity vector of the pixel, thereby determining the optical flow between the first image and the globally aligned second image. For example, the Horn-Schunck algorithm and the Lucas-Kanade (LK) algorithm can be used to determine the optical flow.

[0237] S302. The electronic device 100 converts the globally aligned second image into a locally aligned second image based on the optical flow and the first image as a reference.

[0238] After the optical flow is determined, the first image is used as a reference, and the optical flow method and the backward alignment method are used to locally align the globally aligned second image using the determined optical flow, so as to obtain the locally aligned second image.

[0239] It should be noted that, compared with layout alignment, global alignment is equivalent to coarse alignment, and local alignment is equivalent to fine alignment. Local alignment can, on the basis of global alignment, further achieve finer registration of pixels in the first image and the second image, so that the aligned second image is closer to the image captured under the shooting angle of the first camera, thereby minimizing the perspective error between the first image and the second image due to the fact that they are taken by different cameras, thereby making the target icon finally fused by the electronic device 100 more accurate and achieving a finer deblurring effect.

[0240] For example, Fig.12 Schematic diagram of the alignment effect of an image only after global alignment and an image after global alignment and local alignment provided in an embodiment of the present application.

[0241] in, Fig.12 (a) in the figure is the image obtained after the image to be processed is globally aligned with the reference image. Fig.12 (b) is the image obtained after the image to be processed is globally and locally aligned to the reference image. Fig.12 The error graph shown in (c) is Fig.12 The image obtained by pixel subtraction between the image shown in (a) and the reference image, Fig.12 The error diagram shown in (d) is Fig.12The image obtained by pixel-wise subtraction of the image shown in (b) and the reference image. In the error image, the color of the pixel is black or closer to black, which means that the alignment effect of the pixel is better, that is, in the aligned image, the alignment effect of the pixel with the reference image is better.

[0242] contrast Fig.12 As can be seen from (c) and (d) in the figure, there are more black areas in the error map corresponding to the images after global and local alignment, indicating that compared with only global alignment, the images after global and local alignment are better aligned with the reference image.

[0243] That is to say, the aligned second image may refer to a locally aligned second image. In this case, the aligned second image may refer to a locally aligned second image. When the electronic device 100 aligns the second image to the first image, it may first calculate the homography matrix using the matching feature points in the first image and the second image, and then convert the second image into a globally aligned second image based on the homography matrix and the first image as a reference. Finally, based on the optical flow offset between the first image and the globally aligned second image and the first image as a reference, convert the globally aligned second image into a locally aligned second image.

[0244] It is understandable that, similar to the description of reducing the image resolution mentioned in the global alignment, in the process of local alignment of images, the image with reduced resolution can also be used to determine the optical flow, and then the determined optical flow is used to perform local alignment at the original resolution. That is, in step S301, the image used to calculate the optical flow can be the first image with reduced resolution and the second image with reduced resolution and global alignment. Exemplarily, the reduced resolution can be 1 / 4 of the original image resolution.

[0245] Due to the different characteristic parameters of the first camera and the second camera, there may be problems of color inconsistency and color perception inconsistency, resulting in different colors of the first image and the second image. Therefore, before using the second image to adjust the first image, it is also necessary to align the color of the second image with the color of the first image to ensure that the overall color tone of the target image remains consistent.

[0246] The color alignment of the first image and the second image can be achieved by using a color correction model, a histogram processing technology, and the like.

[0247] For example, taking the histogram processing technology as an example, Fig.13 A schematic diagram of the process of image color alignment provided in an embodiment of the present application.

[0248] Specifically, Fig.13As shown, color alignment of the second image to the first image may include the following steps:

[0249] S401. The electronic device 100 determines a histogram of the first image on RGB channels as a reference histogram.

[0250] The histogram of the first image reflects the pixel distribution of the first image, wherein the horizontal axis of the histogram may be grayscale and the vertical axis may be the number of pixels. In other words, the histogram of the first image on the RGB channel may be determined by counting the number of pixels of different grayscales in the three color channels of red, green, and blue.

[0251] S402. The electronic device 100 determines a histogram of the second image on RGB channels as an original histogram.

[0252] The histogram of the second image reflects the pixel distribution of the second image. The histogram of the second image in the RGB channel can be determined by counting the number of pixels of different grayscales in the three color channels of red, green and blue.

[0253] S403. The electronic device 100 equalizes the reference histogram and the original histogram, and converts the reference histogram and the original histogram into the same normalized uniform histogram.

[0254] Exemplarily, Formula 1 shows the relationship between the reference histogram, the original histogram, and the uniform histogram:

[0255] Z b =T(X a ),Z b =G(Y c ) Formula 1

[0256] Among them, X a is the original histogram function expression, Y c is the functional expression of the reference histogram, Z b is the function expression of uniform histogram, T() is the transformation function from original histogram to uniform histogram, and G() is the transformation function from reference histogram to uniform histogram.

[0257] S404. The electronic device 100 performs an inverse equalization operation on the reference histogram, and calculates a transformation function from the original histogram to the reference histogram.

[0258] Exemplarily, in combination with the above formula 1, formula 2 shows the process of performing an inverse operation of equalization on the reference histogram:

[0259] Y c =G -1 (Z b) Formula 2

[0260] Among them, Y c is the functional expression of the reference histogram, Z b is the function expression of uniform histogram, G -1 () is the inverse function of the transformation function from reference histogram to uniform histogram.

[0261] Further, Formula 3 shows the transformation function from the original histogram to the reference histogram:

[0262] Y c =G -1 (T(X a )) Formula 3

[0263] Among them, X a is the original histogram function expression, Y c is the functional expression of the reference histogram, T() is the transformation function from the original histogram to the uniform histogram, G -1 () is the inverse function of the transformation function from the reference histogram to the uniform histogram. It can be seen that G -1 (T()) is the transformation function from the original histogram to the reference histogram.

[0264] S405. The electronic device 100 uses a transformation function from the original histogram to the reference histogram to adjust the color of the second image to obtain a color-aligned second image.

[0265] After the color alignment, the hue and brightness of the second image can be consistent with those of the first image.

[0266] That is, the electronic device 100 performs color alignment on the first image and the second image, and can adjust the color of the aligned second image based on the histograms of the first image and the second image and the color of the first image as a reference to obtain the color-aligned second image.

[0267] Fig.14 A schematic diagram of a histogram involved in color alignment of a first image and a second image provided in an embodiment of the present application.

[0268] like Fig.14 As shown in , the first image and the second image both correspond to histograms of three channels: R (red), G (green), and B (blue). Fig.14 It can be seen that, on any channel, the histogram of the second image after color alignment is a histogram obtained by transforming the histogram of the second image to the histogram of the first image.

[0269] It should be noted that the embodiment of the present application does not limit the timing of executing color alignment.

[0270] There are three situations for executing color alignment:

[0271] 1) The electronic device 100 may first perform color alignment of the first image and the second image, and then perform global alignment of the first image and the second image.

[0272] In this way, when the electronic device 100 performs global alignment on the first image and the second image, the second image used is the second image after color alignment.

[0273] 2) The electronic device 100 may first perform global alignment of the first image and the second image, and then perform color alignment of the first image and the second image.

[0274] In this way, when the electronic device 100 performs global alignment on the first image and the second image, the second image used is the second image that has not been color aligned. If the electronic device 100 further includes local alignment of the first image and the second image after performing global alignment of the first image and the second image, when the electronic device 100 performs local alignment on the first image and the second image, the second image used is the second image that has been color aligned and globally aligned.

[0275] 3) After performing spatial alignment of the first image and the second image, the electronic device 100 performs color alignment of the first image and the second image. The spatial alignment may include global alignment, or the spatial alignment may include global alignment and local alignment.

[0276] In this way, when the electronic device 100 performs global alignment and local alignment on the first image and the second image, the second images used are all second images that have not been color aligned, and when performing image fusion later, the aligned second images used are second images that have been color aligned and spatially aligned.

[0277] Preferably, the electronic device 100 may perform color alignment on the second image with respect to the first image before performing local alignment. Then, when performing local alignment, the second image used is the color-aligned second image, thereby improving the accuracy of local alignment.

[0278] S104. The electronic device 100 deblurs the first image to obtain a deblurred first image.

[0279] Among them, the electronic device 100 can deblur the first image by using a traditional deblurring algorithm, a deep learning deblurring algorithm, etc. to obtain a deblurred first image.

[0280] Exemplarily, if the electronic device 100 uses a deep learning deblurring algorithm to deblur the first image, the electronic device 100 can input the first image and the aligned second image into the deblurring model, thereby outputting an image with a smaller degree of blur than the first image, that is, the deblurred first image. The deblurring model can be a model trained using a data set containing clear images and blurred images, and a relatively clear image can be input by inputting a blurred image into the deblurring model. The deblurring model can adopt a deformable convolution and a Unet neural network structure.

[0281] In some embodiments, the electronic device 100 may also obtain the deblurred first image using only the first image. For example, the electronic device 100 may output an image with a smaller blur than the first image, that is, the deblurred first image, by inputting the first image into the deblurring model.

[0282] It is understandable that the electronic device 100 may also combine more or fewer other images to achieve deblurring of the first image, and the embodiment of the present application is not limited to this.

[0283] In some embodiments, before deblurring the first image using the deblurring model, the electronic device 100 may also downsample the first image to reduce the resolution of the first image, for example, reducing the resolution of the first image to 1 / 4 of the original, and then inputting the first image with reduced resolution into the deblurring model to output the deblurred first image with the same reduced resolution. In this way, the computational efficiency of the electronic device 100 when deblurring the first image can be improved. Afterwards, if the deblurred first image at the original resolution is required, the deblurred first image can be restored to the resolution of the first image by upsampling the deblurred first image with reduced resolution.

[0284] It is understandable that other methods may be used to deblur the first image to obtain the deblurred first image, and the embodiment of the present application is not limited to this.

[0285] It is understandable that the embodiment of the present application does not limit the execution order of step S103 and step S104. Step S103 can be executed first, and then step S104, or step S104 can be executed, and then step S103, or step S103 and step S104 can be executed simultaneously.

[0286] S105. The electronic device 100 obtains a target image with a greater depth of field based on the first image according to the first image, the aligned second image, and the deblurred first image.

[0287] The electronic device 100 can adjust the depth of field of the first image by combining the aligned second image and the deblurred first image to obtain a target image with a larger depth of field.

[0288] It should be understood that if, when aligning the second image, the electronic device 100 performs global alignment of the second image to the first image, the aligned second image may refer to the globally aligned second image, and the electronic device 100 also performs local alignment of the second image to the first image, the aligned second image may refer to the locally aligned second image, and if the electronic device 100 performs color alignment of the second image to the second image, and the color alignment is performed after the spatial alignment, the aligned second image may refer to the color-aligned second image.

[0289] Exemplarily, the target image may refer to Figure 8 The image shown in (c) in .

[0290] Among them, considering that if the second camera is a wide-angle camera or an ultra-wide-angle camera, the depth of field of the second image is greater than that of the first image, and if the second camera is a telephoto camera, the second image can include the details captured by the first image. Therefore, adjusting the depth of field of the first image in combination with the aligned second image can use the clear picture captured in the second image to replace the blurred picture captured in the first image, thereby achieving the expansion of the depth of field of the first image.

[0291] In addition, considering that the first image and the second image are images captured by different cameras, even if the second image is aligned to the first image, the difference in viewing angles of different cameras may cause defects in the aligned second image. For example, assuming that the first image includes object 1 and object 2, due to the difference in physical positions of the first camera and the second camera, object 2 may be blocked by object 1 in the shooting angle of the second camera. Therefore, the second image only includes object 1. Then, when aligning the second image, the pixel area where object 2 is located may be filled with the pixel points of object 1, resulting in repeated pixel points of object 1 in the aligned second image, that is, the artifact of object 1. Therefore, using only the aligned second image to adjust the first image cannot solve the occlusion problem in the multi-camera system well, and may cause the details of the target image to be not processed properly, such as the details are still blurry, or there are artifacts, etc. Therefore, adjusting the depth of field of the first image in combination with the deblurred first image can further adjust the details in the image and improve the image deblurring effect.

[0292] In other words, if the first image and the second image both capture the first object, and the clarity of the first object captured in the first image is lower than the clarity of the first object captured in the second image, then the pixels in the target image are derived from the pixels other than the first object captured in the first image, the pixels of the first object captured in the aligned second image, and the pixels in the deblurred first image.

[0293] Among them, the electronic device 100 can obtain a mask map of the first image, a mask map of the aligned second image, and a mask map of the deblurred first image according to the first image, the aligned second image, and the deblurred first image, and the mask map is used to determine the pixels to be retained and the pixels to be deleted in the image. In this way, the electronic device 100 can generate a target image using the mask maps of these images.

[0294] In some embodiments, the electronic device 100 may use a fusion network to obtain mask images of the first image, the aligned second image, and the deblurred first image. The fusion network is a neural network model trained using multiple sets of images, wherein a set of images may include: a shallow depth of field image and a large depth of field image captured by a camera, an image after deblurring the shallow depth of field image, and an image captured by another camera and aligned to the shallow depth of field image.

[0295] Exemplarily, the electronic device 100 may input the first image, the aligned second image, and the deblurred first image into a fusion network, and output mask images of the respective images. Afterwards, the electronic device 100 uses the mask image of the first image, the mask image of the aligned second image, and the mask image of the deblurred first image to fuse and obtain a target image, wherein the pixels in the target image are composed of the pixels in the first image, the aligned second image, and the deblurred first image.

[0296] Further optionally, in order to improve the computing efficiency of the electronic device 100, the electronic device 100 may first downsample the first image, the aligned second image, and the deblurred first image to reduce the resolution of the image, for example, reducing the resolution of the image to 1 / 4 of the original, and then input the first image with reduced resolution, the second image with reduced resolution and aligned, and the first image with reduced resolution and deblurred into the fusion network, output the mask map of the first image with reduced resolution, the mask map of the second image with reduced resolution and aligned, and the mask map of the first image with reduced resolution and deblurred, and then restore the output mask map to the mask map corresponding to the image at the original resolution. Finally, use these mask maps with adjusted resolution to process the first image at the original resolution, the aligned second image, and the deblurred first image to obtain the target image.

[0297] For example, Fig.15 A schematic diagram of the principle of image fusion provided in an embodiment of the present application.

[0298] like Fig.15 As shown, I m represents the first image, I wecf represents the second image after alignment, I deblur represents the first image after deblurring, I m After downsampling by 4 times, the obtained I m The corresponding image at 1 / 4 resolution is I w,e,c,f After downsampling by 4 times, the obtained I w,e,c,f The corresponding image at 1 / 4 resolution is I deblur After downsampling by 4 times, the obtained I deblur The corresponding image at 1 / 4 resolution, Conv(·) represents the fusion network, express The mask map of express The mask map of express The mask map, M m Indicates that After upsampling 4 times, the obtained I m The mask map at the corresponding resolution, M w,e,c,f Indicates that After upsampling 4 times, the obtained I w,e,c,f The mask map at the corresponding resolution, M deblur Indicates that After upsampling 4 times, the obtained I deblur The mask map at the corresponding resolution, I o Represents the target image.

[0299] from Fig.15 It can be seen that by downsampling the first image, the aligned second image and the deblurred first image and inputting them into the fusion network, the mask images corresponding to the three images can be obtained. After that, the mask image is upsampled to make the resolution of the mask image consistent with the resolution of the first image, the aligned second image and the deblurred first image. Finally, the mask image with adjusted resolution can be used to fuse the first image, the aligned second image and the deblurred first image to obtain the target image.

[0300] The target image retains the picture content of the first image, integrates the clear pixels in the aligned second image, and uses the pixels in the deblurred first image to correct the details in the first image, thereby improving the image deblurring effect.

[0301] In some embodiments, after the electronic device 100 obtains the target image, the electronic device 100 may display the target image. In this way, the user can quickly view the large depth of field image obtained for the current shooting scene after performing the shooting operation, or the electronic device 100 can display the target image through the gallery application after detecting the user's operation of entering the gallery to view. In this way, the user can choose whether to continue shooting pictures or browse the currently obtained target image.

[0302] In addition, in addition to displaying the target image, the electronic device 100 can also display any one or more of the first image, the second image, the aligned second image, and the deblurred first image, which is not limited in this embodiment of the present application.

[0303] In general, the image processing method provided in the embodiment of the present application reduces the differences in viewing angle, color, parameters and other factors between images taken by different cameras through the steps of global image alignment, local image alignment and color alignment, and expands the image depth of field in combination with the deblurred image, thereby solving the problem of object occlusion in images taken by different cameras, breaking through the current technical limitation of using only one camera to collect images for image deblurring, and the solution can process images of any size, has a wide range of application scenarios, facilitates user operations, and improves user experience.

[0304] It is understandable that the image processing method provided in the embodiment of the present application can be applied to the photo scene to extend the depth of field of the picture taken by the user. Alternatively, the image processing method provided in the embodiment of the present application can also be applied to the video recording scene to extend the depth of field of the video recorded by the user. In this case, the first image, the second image and the target image mentioned above can refer to one of the frames in the video. In this way, the electronic device 100 can deblur the video by extending the depth of field of each frame of the video, thereby improving the user's video recording experience.

[0305] Fig.16 A flowchart of another image processing method provided in an embodiment of the present application.

[0306] like Fig.16 As shown, the image processing method may include the following steps:

[0307] S501. The electronic device 100 detects a shooting operation of a user.

[0308] The shooting operation can be used to trigger the electronic device 100 to collect images through the first camera and the second camera. In the embodiment of the present application, the shooting operation can also refer to the first operation.

[0309] In some implementations, before the electronic device 100 detects the first operation, the electronic device 100 may further display a preview interface, where the preview interface may be used to display an image captured by the first camera.

[0310] Further, the preview interface may include one or more shooting mode options. Before the electronic device 100 detects the first operation, the electronic device 100 may detect a second operation acting on a first shooting mode option among the one or more shooting mode options.

[0311] S502. The electronic device 100 captures a first image through the first camera and a second image through the second camera.

[0312] S503. The electronic device 100 aligns the second image with the first image to obtain an aligned second image.

[0313] Exemplarily, the alignment may include spatial alignment.

[0314] In some embodiments, the aligned second image may refer to a globally aligned second image, and the electronic device 100 aligns the second image to the first image to obtain the aligned second image, which may mean that the electronic device 100 calculates the homography matrix using the matching feature points in the first image and the second image, and then converts the second image into the globally aligned second image based on the homography matrix and the first image.

[0315] In other embodiments, the aligned second image may refer to a locally aligned second image. Then, the electronic device 100 aligns the second image to the first image to obtain the aligned second image, which may mean that the electronic device 100 calculates the homography matrix using the matching feature points in the first image and the second image, and then converts the second image into a globally aligned second image based on the homography matrix and with the first image as a reference. Finally, based on the optical flow offset between the first image and the globally aligned second image and with the first image as a reference, the globally aligned second image is converted into a locally aligned second image.

[0316] Exemplarily, the alignment may also include color alignment.

[0317] In some embodiments, the electronic device 100 may perform color alignment of the second image to the first image between global alignment and local alignment of the first image and the second image. In this case, the electronic device 100 may, after converting the second image into a globally aligned second image according to the homography matrix, perform color alignment of the globally aligned second image to the first image according to the first image to obtain a color-aligned second image. In this way, when the electronic device 100 performs local alignment of the images, the color-aligned second image may be converted into a locally aligned second image based on the optical flow offset of the first image and the color-aligned second image, with the first image as a reference.

[0318] In other embodiments, the electronic device 100 may perform color alignment of the second image to the first image after spatially aligning the second image to the first image. In this case, the aligned second image may refer to a color-aligned second image, and the electronic device 100 may perform color alignment of the second image to the first image after completing the spatial alignment to obtain a color-aligned second image.

[0319] The color of the second image after spatial alignment can be adjusted by calculating the histogram of the first image and the second image after spatial alignment based on the color of the first image, thereby obtaining the second image after color alignment.

[0320] S504. The electronic device 100 obtains a target image with a larger depth of field based on the first image according to the first image and the aligned second image.

[0321] The electronic device 100 can obtain a mask map of the first image and a mask map of the aligned second image based on the first image and the aligned second image, and the mask map indicates the pixels to be retained and the pixels to be deleted in the image. In this way, the electronic device 100 can generate a target image based on the mask map. The mask map can be the data output by the electronic device 100 after inputting the first image and the aligned second image into the fusion model.

[0322] It can be understood that steps S501-S503 are the same as the above steps S101-S103, and the detailed description of steps S501-S503 can refer to the relevant content of the above steps S101-S103. Step 504 is similar to the description of step S105, and both use the first image and the aligned second image to obtain the target image. The difference is that step S105 also uses the deblurred first image to obtain the target image. That is to say, the pixels in the target image obtained in step S504 are derived from the pixels other than the first object collected in the first image and the pixels of the first object collected in the aligned second image. In this way, the image processing steps of the electronic device 100 can be reduced and the speed of obtaining the target image can be accelerated. It should be understood that in this case, the principle of image fusion performed by the electronic device 100 is similar to the above principle, except that the input and output of the fusion network only involve the first image and the aligned second image, and no longer involve the deblurred first image.

[0323] It should be understood that each step in the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0324] The present application also provides an electronic device, which may include: a memory and a processor. The memory may be used to store a computer program; the processor may be used to call the computer program in the memory, so that the electronic device executes the method executed by the electronic device 100 in any of the above embodiments.

[0325] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method performed by the electronic device 100 in any of the above embodiments.

[0326] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0327] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0328] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0329] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0330] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0331] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute a method executed by any one of the electronic devices 100 in any of the above embodiments.

[0332] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes any method executed by the electronic device 100 in any of the above embodiments.

[0333] It should be understood that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (digitalsignal processor, DSP), an application specific integrated circuit (AP 800plication specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined to perform. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0334] In addition, an embodiment of the present application further provides a device. The device may be a component or a module, and the device may include one or more processors and a memory connected to each other. The memory is used to store a computer program. When the computer program is executed by one or more processors, the device executes the method in each of the above method embodiments.

[0335] Among them, the device, computer-readable storage medium, computer program product or chip provided in the embodiments of the present application are used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0336] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0337] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0338] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0339] In short, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image processing method, characterized in that: The method is applied to an electronic device including a first camera and a second camera, and the method includes: The electronic device collects a first image through a first camera and collects a second image through a second camera, and a clarity of a first object collected in the first image is lower than a clarity of the first object collected in the second image; The electronic device spatially aligns the second image with the first image to obtain an aligned second image; The electronic device deblurs the first image to obtain a deblurred first image; The electronic device obtains a target image based on the first image, the aligned second image and the deblurred first image, and the pixels in the icon image are derived from the pixels other than the first object collected in the first image, the pixels in the first object collected in the aligned second image, and the pixels in the deblurred first image.

2. The method according to claim 1, characterized in that The electronic device obtains the target image according to the first image, the aligned second image, and the deblurred first image, specifically including: The electronic device obtains, according to the first image, the aligned second image and the deblurred first image, a mask map of the first image, a mask map of the aligned second image, and a mask map of the deblurred first image, wherein the mask map is used to indicate pixel points retained in the image; The electronic device generates the target image according to the mask map of the first image, the mask map of the aligned second image, and the mask map of the deblurred first image.

3. The method according to claim 2, characterized in that The electronic device obtains the target image according to the first image, the aligned second image, and the deblurred first image, specifically including: The electronic device performs downsampling of the first image, the aligned second image and the deblurred first image at a first magnification to obtain a downsampled image; The electronic device obtains a mask image according to the downsampled image; The electronic device performs upsampling on the mask image at the first magnification to obtain an upsampled mask image, wherein the upsampled mask image is used to indicate pixel points retained in the first image, the aligned second image, and the deblurred first image; The electronic device generates the target image according to the mask image, the first image, the aligned second image, and the deblurred first image.

4. An image processing method, characterized in that: The method is applied to an electronic device including a first camera and a second camera, and the method includes: The electronic device collects a first image through a first camera and collects a second image through a second camera, and a clarity of a first object collected in the first image is lower than a clarity of the first object collected in the second image; The electronic device spatially aligns the second image with the first image to obtain an aligned second image; The electronic device obtains a target image based on the first image and the aligned second image, and the pixels in the target image are derived from the pixels other than the first object collected in the first image and the pixels of the first object collected in the aligned second image.

5. The method according to any one of claims 1 to 4, characterized in that: Before the electronic device collects the first image through the first camera and collects the second image through the second camera, the method further includes: The electronic device detects a first operation, where the first operation is used to trigger the electronic device to capture images through the first camera and the second camera.

6. The method according to claim 5, characterized in that Before the electronic device detects the first operation, the method further includes: The electronic device displays a preview interface, where the preview interface is used to display the image captured by the first camera.

7. The method according to claim 6, characterized in that The preview interface includes one or more shooting mode options. Before the electronic device detects the first operation, the method further includes: The electronic device detects a second operation acting on a first shooting mode option among the one or more shooting mode options.

8. The method according to any one of claims 1 to 7, characterized in that: The aligned second image is a globally aligned second image, and the electronic device spatially aligns the second image with the first image to obtain the aligned second image, specifically including: The electronic device calculates a homography matrix using the matched feature points in the first image and the second image; The electronic device converts the second image into the globally aligned second image according to the homography matrix and based on the first image.

9. The method according to any one of claims 1 to 7, characterized in that: The aligned second image is a locally aligned second image, and the electronic device spatially aligns the second image with the first image to obtain the aligned second image, specifically including: The electronic device calculates a homography matrix using the matched feature points in the first image and the second image; The electronic device converts the second image into a globally aligned second image based on the homography matrix and the first image as a reference; The electronic device converts the globally aligned second image into the locally aligned second image based on the optical flow offset between the first image and the globally aligned second image and taking the first image as a reference.

10. The method according to claim 9, characterized in that After the electronic device converts the second image into the globally aligned second image according to the homography matrix, the method further includes: The electronic device performs color alignment on the globally aligned second image with the first image according to the first image to obtain a color-aligned second image; The electronic device converts the globally aligned second image into the locally aligned second image based on the optical flow offset between the first image and the globally aligned second image and taking the first image as a reference, specifically including: The electronic device converts the color-aligned second image into the locally aligned second image based on the optical flow offset between the first image and the color-aligned second image and taking the first image as a reference.

11. The method according to any one of claims 1 to 10, characterized in that: The electronic device obtains a target image according to the first image and the aligned second image, specifically comprising: The electronic device performs color alignment on the aligned second image with the first image according to the first image to obtain a color-aligned second image; The electronic device obtains a target image based on the first image and the color-aligned second image, wherein the target image includes pixel points other than the first object collected in the first image and pixel points of the first object collected in the color-aligned second image.

12. The method according to any one of claims 1 to 11, characterized in that: The first camera is a main camera, and the second camera includes any one or more of a wide-angle camera, an ultra-wide-angle camera, or a telephoto camera.

13. The method according to any one of claims 1 to 12, characterized in that: The method further comprises: The electronic device displays the target image.

14. An electronic device, characterized in that: The electronic device comprises a memory, one or more processors, and one or more programs; when the one or more processors execute the one or more programs, the electronic device implements the method as claimed in any one of claims 1 to 13.

15. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 13.

Images