An image processing method, an electronic device, and a storage medium

Abstract

The application provides an image processing method, an electronic device and a storage medium, and relates to the technical field of image processing. In the method, in response to a camera application being in a starting state, a first preview image is displayed on a preview interface of the camera application; in response to an operation of increasing a first zoom ratio of the camera application to a second zoom ratio, a second preview image is displayed on the preview interface of the camera application, and a third preview image is displayed on a first window of the preview interface; and in response to a triggering operation of a photographing function of the camera application, a thumbnail of a photographing image is displayed on a second window of the preview interface. The target object in the second preview image is relatively clear, the clarity of the target object in the photographing image can remain unchanged or be higher compared with the second preview image, and the clarity of other objects in the photographing image except the target object is higher, so that the real scene seen by the user can be restored as much as possible, and the image shooting effect is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image processing method, an electronic device, and a storage medium. BACKGROUND

[0002] With the wide popularity of electronic devices such as mobile phones, tablet computers, and the rapid development of photographing technology, the photographing function of electronic devices is gradually enriched, and more and more users choose to use electronic devices to record the wonderful moments in life. For example, an electronic device can provide a moon photographing mode, in which the electronic device can photograph a relatively clear moon.

[0003] However, other objects in the image where the moon is located can not be imaged or imaged clearly, that is, the image photographed by the electronic device can not match the real scene seen by the user, affecting the user experience. SUMMARY

[0004] To solve the above problems, the present application provides an image processing method, an electronic device, and a storage medium, which aims to improve the image photographing effect of the electronic device and restore the real scene seen by the user as much as possible.

[0005] In a first aspect, the present application provides an image processing method, which can be applied to an electronic device, which can be a mobile phone, a tablet computer, a notebook computer, or the like, including a camera application. In the method, after the camera application is started, for example, the user clicks the icon of the camera application, or the user issues a gesture instruction and a voice instruction to start the camera application, the electronic device can display a first preview image on the preview interface of the camera application; then, when the first zoom ratio of the camera application is increased to a second zoom ratio, for example, the user slides the zoom control of the camera application to increase the zoom ratio, in the case that the second zoom ratio is greater than the first zoom ratio, the electronic device can display a second preview image on the preview interface of the camera application, and display a third preview image in a first window of the preview interface. Illustratively, the third preview image can be displayed in the form of a picture-in-picture window on the preview interface, and the first window can be a picture-in-picture window. Illustratively, when the electronic device is a foldable screen mobile phone, the second preview image can be displayed in the left window of the preview interface, and the third preview image can be displayed in the right window (i.e., the first window) of the preview interface. Through the preview interface of the camera application, it can be seen that the clarity of a target object in the second preview image is greater than the clarity of the target object in the third preview image; the clarity of other objects in the second preview image, except the target object, is less than the clarity of other objects in the third preview image, except the target object. Illustratively, the target object can be the moon, a lamp, or the like, and the other objects can be buildings, people, plants, and animals.

[0006] After the photographing function of the camera application is triggered, for example, the user clicks the shooting control of the camera application, or the user issues a gesture instruction and a voice instruction to trigger the photographing function, the electronic device can display a thumbnail of the photographed image in the second window of the preview interface. For example, the second window can be a thumbnail display area of the preview interface, which can be in any position of the lower left corner of the preview interface. The photographed image can show that the clarity of the target object is greater than or equal to the clarity of the target object in the second preview image, and the clarity of other objects except the target object is greater than the clarity of other objects except the target object in the second preview image.

[0007] In this way, compared with the third preview image, the second preview image has higher clarity of the target object, and the third preview image has higher clarity of other objects except the target object. Compared with the second preview image, the photographed image can have the same or higher clarity of the target object, and the clarity of other objects except the target object is higher. That is, after the photographing function of the camera application is triggered, a photographed image with clear target object and other objects can be obtained. Therefore, the image processing method provided in the present application can obtain a relatively clear photographed image, improve the image shooting effect of the electronic device, and restore the real scene as much as possible for the user.

[0008] In a possible implementation, after the first zoom ratio of the camera application is increased to the second zoom ratio, for example, from 2x to 10x, the electronic device can compare the second zoom ratio with the zoom ratio threshold value, and determine that the second zoom ratio is greater than or equal to the zoom ratio threshold value. For example, the zoom ratio threshold value can be 10x. At this time, the electronic device can detect the object contained in the first preview image. Then, when the first preview image contains the target object, the electronic device can obtain the second preview image collected by the first camera and the third preview image collected by the second camera. Subsequently, the electronic device can display the second preview image in the preview interface of the camera application, and display the third preview image in the first window.

[0009] In this way, when it is determined that the first preview image contains the target object, two images with different clarities can be displayed to the user, so that the user can see a relatively clear target object and a relatively clear other object, thereby improving the user experience.

[0010] In a possible implementation, the second preview image can be obtained through the following steps: the electronic device can capture the second preview image based on the first exposure parameter, where the first exposure parameter is lower than the exposure parameter of the first preview image. In this way, the second preview image is obtained by performing the exposure reduction processing, which can make the object in the bright area of the second preview image clearer. For example, when the target object is a brighter object, the clarity of the target object can be improved.

[0011] In a possible implementation, the third preview image can be obtained through the following steps: the electronic device can capture the third preview image based on the second exposure parameter, where the second exposure parameter is higher than the exposure parameter of the second preview image. For example, when the second preview image is obtained by performing the exposure reduction processing, the third preview image can be obtained by performing the normal exposure processing. In this way, compared with the second preview image, the object in the dark area of the third preview image is clearer. For example, when the target object is a brighter object, the clarity of the object other than the target object in the third preview image can be improved.

[0012] In a possible implementation, the electronic device can capture the first preview image through the first camera. In a case where it is determined that the first preview image contains the target object, the electronic device can start the second camera. Then, the electronic device can obtain the second preview image captured by the first camera and the third preview image captured by the second camera. For example, the first camera can be a long-focus camera, and the second camera can be a main camera. In this way, when it is determined that the shooting scene contains the target object, the second camera can be started based on the fact that the first camera has been started, and the second preview image and the third preview image with different exposure amounts can be obtained. When the image has bright and dark areas, the user can see the clearer shooting scene through the two images.

[0013] In a possible implementation, the first preview image can be captured by the first camera. In a case where it is determined that the first preview image contains the target object, the electronic device can reduce the exposure parameter of the first camera. Then, the electronic device can capture the second preview image through the first camera based on the reduced exposure parameter. In this way, the second preview image is captured by performing the exposure reduction processing, which can improve the clarity of the object with high brightness (i.e., the bright area in the image) compared with the first preview image. When the target object is the moon, a lamp, or other objects with high brightness, the user can see the target object more clearly, thereby improving the user experience.

[0014] In a possible implementation, when the first zoom ratio is adjusted to the second zoom ratio, the electronic device can pop up a first window to display a fourth preview image in the preview interface, where the resolution of the first preview image is the same as that of the fourth preview image. For example, the fourth preview image can be displayed in a picture-in-picture window. The first preview image and the fourth preview image can be obtained by the same camera. When it is determined that the first preview image contains the target object, the electronic device can update the first preview image to a second preview image in the preview interface, and update the fourth preview image to a third preview image in the first window. In this way, when it is determined that the target object exists in the shooting scene, the image is updated so that the user can see the target object and other objects through the preview interface, thereby improving the user experience.

[0015] In a possible implementation, the first camera can obtain a first raw image and a second preview image, where the resolution of the target object and the resolution of the other objects in the first raw image and the second preview image are the same. The second camera can obtain a second raw image and a third preview image, where the resolution of the target object and the resolution of the other objects in the second raw image and the third preview image are the same. The electronic device can generate a photographed image by performing fusion processing on the first raw image and the second raw image. In this way, the moon in the first raw image is clearer, and the other objects in the second raw image are clearer. The photographed image obtained by fusing the first raw image and the second raw image is clearer, which can improve the image shooting effect of the electronic device and restore the real scene as much as possible.

[0016] In a possible implementation, the electronic device can determine a first exposure sequence based on the first raw image and the second raw image, where the first exposure sequence includes a plurality of groups of exposure parameters. The exposure parameters in one group of exposure parameters can include an exposure time, a sensitivity, an aperture coefficient, and the like. The values of at least two groups of exposure parameters are different, for example, the values of the exposure time are different, the values of the sensitivity are different, or the values of each parameter are different. Then, the electronic device can obtain a plurality of third raw images by the first camera according to the first exposure sequence, where it should be understood that the plurality of third raw images correspond to the plurality of groups of exposure parameters in a one-to-one manner. Subsequently, the electronic device generates a photographed image by performing fusion processing on the plurality of third raw images and the first raw image. In this way, the plurality of third raw images with different exposure parameters and the first raw image are fused to adjust the exposure parameters of the first raw image, so that the other objects become clearer, and a photographed image with better quality is obtained.

[0017] In a possible implementation, the first exposure sequence can include a first set of exposure parameters and a second set of exposure parameters, wherein a value of the first set of exposure parameters is greater than a value of the second set of exposure parameters. Exemplarily, the first set of exposure parameters includes exposure parameters corresponding to long-frame images, and the second set of exposure parameters includes exposure parameters corresponding to medium-frame images. In this way, the third original image corresponding to the first set of exposure parameters can make other objects in the dark region of the first original image clearer.

[0018] In a possible implementation, the first exposure sequence can include a first set of exposure parameters, a second set of exposure parameters, and a third set of exposure parameters, wherein a value of the first set of exposure parameters is greater than a value of the second set of exposure parameters, and a value of the second set of exposure parameters is greater than a value of the third set of exposure parameters. Exemplarily, the first set of exposure parameters includes exposure parameters corresponding to long-frame images, the second set of exposure parameters includes exposure parameters corresponding to medium-frame images, and the third set of exposure parameters includes exposure parameters corresponding to short-frame images. In this way, the third original image corresponding to the first set of exposure parameters can make objects (for example, objects other than the target object) in the dark region of the first original image clearer, and the third original image corresponding to the third set of exposure parameters can make objects (for example, the target object) in the bright region of the first original image clearer.

[0019] In a possible implementation, a value of the second set of exposure parameters is obtained based on the first original image, for example, exposure parameters of the first original image are taken as the second set of exposure parameters, and a value of the first set of exposure parameters and a value of the third set of exposure parameters are obtained based on the first original image and the second original image. Exemplarily, the value of the first set of exposure parameters and the value of the third set of exposure parameters can be determined based on the exposure parameters of the first original image, for example, the first set of exposure parameters is obtained by increasing the exposure parameters of the first original image, and the third set of exposure parameters is obtained by decreasing the exposure parameters of the first original image. In this way, the exposure parameters of the first original image are taken as the basis, and the target object in the first original image is relatively clear, so that the clarity of the target image in the finally generated photograph image can be avoided to be reduced.

[0020] In a possible implementation, the obtaining of the third preview image can include: performing, by the electronic device, spatial alignment processing on the second original image based on the first original image to obtain the third preview image. In this way, considering that different camera positions on the electronic device cause a displacement deviation of the content contained in the original image collected, the second original image is aligned with the first original image in space, so that the user can see the spatially aligned second preview image and the third preview image in the preview interface of the electronic device, and the use experience is improved.

[0021] In a possible implementation, a first vertex of the first original image is an origin of a first coordinate system, a second vertex of the second original image is an origin of a second coordinate system, the first vertex is located at a same position in the first original image as the second vertex is located at in the second original image, and the first vertex and the second vertex can be, for example, top-left vertices of the images. The electronic device can determine a first coordinate point of the target object in the first original image in the first coordinate system, and determine a second coordinate point of the target object in the second original image in the second coordinate system, where the second coordinate point is located at a same position in the target object as the first coordinate point is located at in the target object, and the first coordinate point and the second coordinate point can be, for example, center points of the target object. Then, the electronic device can calculate a coordinate deviation value of the first coordinate point and the second coordinate point. Subsequently, the electronic device can perform spatial alignment processing on the second original image based on the coordinate deviation value to obtain the third preview image, for example, by performing movement, cropping, or the like on the second original image to obtain the third preview image that is spatially aligned with the first original image. In this way, the user can see the spatially aligned second preview image and the third preview image in the preview interface of the electronic device, and the use experience is improved.

[0022] In a possible implementation, the target object can include the moon, the clarity of the moon in the second preview image is greater than the clarity of the moon in the third preview image, and the clarity of objects other than the moon in the second preview image is less than the clarity of the objects other than the moon in the third preview image. It should be understood that the moon itself has a high brightness, and the user can see the second preview image containing a clear moon in the preview interface, and can see the third preview image containing clear objects in the first window, and the use experience is improved.

[0023] In a second aspect, the present application provides an electronic device, which includes a memory and a processor; the memory stores computer program code, and the computer program code includes computer instructions; the one or more processors invoke the computer instructions to enable the electronic device to perform the image processing method of the first aspect.

[0024] In a third aspect, the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the image processing method of the first aspect is implemented.

[0025] According to the above technical solution, the present application has the following beneficial effects:

[0026] In a process of starting the camera application, a preview interface of the camera application displays a first preview image; when a zoom ratio of the camera application is adjusted from a first zoom ratio to a second zoom ratio, and the second zoom ratio is a value greater than or equal to a zoom ratio threshold, the preview interface displays a second preview image, and a first window of the preview interface displays a third preview image; after a photographing function of the camera application is triggered, a thumbnail of a photographing image is displayed in a second window of the preview interface. Compared with the second preview image and the third preview image, the moon in the second preview image is clearer, and other objects in the third preview image are clearer except the moon; compared with the second preview image and the photographing image, the photographing image can keep the same or higher clarity of a target object, and other objects in the photographing image are clearer except the target object, so that a photographing image with clear moon and other objects can be obtained. In this way, the image processing method provided in the present application can obtain a clearer photographing image, improve the image photographing effect of the electronic device, and restore the real scene as much as possible. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 A schematic diagram of an image containing a moon provided by an embodiment of the present application;

[0028] Figure 2a A schematic diagram of a running process of a camera application in a related technology provided by an embodiment of the present application;

[0029] Figure 2b A schematic diagram of a photographing scene in a camera technology provided by an embodiment of the present application;

[0030] Figure 3a A schematic diagram of an application scene provided by an embodiment of the present application;

[0031] Figure 3b A schematic diagram of another application scene provided by an embodiment of the present application;

[0032] Figure 4 A schematic diagram of a system structure of an electronic device provided by an embodiment of the present application;

[0033] Figure 5a A signaling interaction diagram of a preview stage of a normal photographing mode provided by an embodiment of the present application;

[0034] Figure 5b A signaling interaction diagram of a preview stage of a moon-viewing photographing mode provided by an embodiment of the present application;

[0035] Figure 5c A signaling interaction diagram of a photographing stage of a moon-viewing photographing mode provided by an embodiment of the present application;

[0036] Figure 6 A schematic diagram of a camera application running process provided for an embodiment of the present application;

[0037] Figure 7 A schematic diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0038] For the sake of clear and concise description of the following embodiments, first, the glossary involved in the embodiments of the present application is explained. It should be understood that the explanation is for a clearer understanding of the embodiments of the present application, and does not necessarily constitute a limitation on the embodiments of the present application.

[0039] Exposure: refers to the strength and length of time of the light sensed by the camera, which affects the brightness of the image captured by the camera. When taking a picture through the camera of the electronic device, the exposure may be too high or too low, which will directly lead to overexposure or underexposure of the subject and the background. Overexposure, then the image will be too bright, and the bright part details cannot be reflected; underexposure, then the image will be dark, and the dark part details cannot be reflected.

[0040] Exposure parameter: refers to the factor affecting the brightness of the image. In the embodiments of the present application, the exposure parameter can include aperture, exposure time and sensitivity.

[0041] Long frame image: in the embodiments of the present application, the image with long exposure time is called long frame image.

[0042] Short frame image: in the embodiments of the present application, the image with short exposure time is called short frame image.

[0043] Medium frame image: in the embodiments of the present application, the image with exposure time between long frame image and short frame image is called medium frame image. That is, the exposure time of long frame image is greater than that of medium frame image, and the exposure time of medium frame image is greater than that of short frame image.

[0044] It should be noted that the length of the exposure time here is relative, and the specific exposure time value can be determined according to actual use requirements, which is not limited by the embodiments of the present application.

[0045] Next, the technical advantages of the image processing method, electronic device and storage medium provided by the present application are compared and explained in combination with related technologies. For the convenience of understanding, an example scene is used for explanation. In this example scene, the electronic device can be a mobile phone.

[0046] In related technologies, the user may have the demand of using the mobile phone to take pictures of the moon, and the mobile phone can take pictures of the moon as Figure 1The image shown in (a) is not clear, and the moon is overexposed and out of focus. Currently, in order to make the phone take a clear moon, more and more electronic devices provide a moon shooting mode (also known as a shooting moon mode, a moon mode, or a super moon mode, etc.). The image taken by the phone in the moon shooting mode can be seen in Figure 1 The image shown in (b) is more clear than Figure 1 The moon in (a) is more clear than

[0047] Figure 1 (a) and (b) are taken of the same shooting scene, and by combining the two, it can be seen that the imaging clarity of the moon is different, Figure 1 The moon in (a) is overexposed and out of focus, Figure 1 The moon in (b) is more clear; Figure 1 (a) and Figure 1 The imaging clarity of other objects in (b) except the moon is also different, Figure 1 In (a), other objects (which can be understood as foreground) such as trees and fence nets can be seen, Figure 1 In (b), only a clear moon can be seen, and other objects except the moon cannot be imaged.

[0048] Next, the shooting process of the phone in the moon shooting mode in the related art is introduced. Figure 2a and Figure 2b

[0049] As shown in Figure 2a , the phone starts the camera application, and in the case of using a long-focus camera for shooting, the long-focus camera starts a data stream (also known as long-focus starting flow) to obtain a long-focus flow, and then sends the image obtained based on the long-focus flow to the camera preview, and when the camera preview displays a picture-in-picture window, the image obtained based on the long-focus flow can also be sent to the picture-in-picture window. After the user triggers the shooting function, the phone can perform frame processing on the single-frame image included in the long-focus flow to obtain a shooting image.

[0050] As shown in Figure 2b , the phone displays a camera preview interface 210 (which can be referred to as a preview interface), which includes a preview area 211, a picture-in-picture window area 212, and a shooting control 213. In the moon shooting mode, the images displayed in the preview area 211 and the picture-in-picture window 212 are both obtained based on the long-focus flow, and the moon in the image is relatively clear. After the user clicks the camera shooting control 213, the phone can perform frame processing on the single-frame image included in the long-focus flow to obtain an image as shown in Figure 1 (b).

[0051] ​As can be seen from the above, the image obtained by the mobile phone in the moon shooting mode may not be able to image or image other objects in the image in addition to the moon, and the user can see the moon and other objects in the scene, which means that the image photographed by the electronic device in the moon shooting mode cannot restore the real scene, and the image effect is poor.

[0052] Therefore, in order to solve the above problems, the embodiments of the present application provide an image processing method, an electronic device and a storage medium. In the method, during the startup of the camera application, the electronic device can display a first preview image in the preview interface of the camera application; when the zoom ratio of the camera application is adjusted from a first zoom ratio to a second zoom ratio, and the second zoom ratio is a value greater than or equal to a zoom ratio threshold, the electronic device can display a second preview image in the preview interface, and display a third preview image in a first window of the preview interface; and then after the photographing function of the camera application is triggered, a thumbnail of a photographed image can be displayed in a second window of the preview interface. Compared with the second preview image and the third preview image, the moon in the second preview image has higher clarity, and other objects in the third preview image have higher clarity. Compared with the second preview image and the photographed image, the clarity of the target object in the photographed image can remain unchanged or be higher, and the clarity of other objects in the photographed image except the target object is higher, so that a photographed image with clear moon and other objects can be obtained. In this way, the image processing method provided by the present application can obtain a clear photographed image, improve the image shooting effect of the electronic device, and restore the real scene as much as possible as seen by the user.

[0053] In order to make the person skilled in the art more clearly understand the scheme of the present application, the following will combine Figure 3a and Figure 3b First, the application scenario of the technical scheme of the present application is described.

[0054] Still taking the electronic device as a mobile phone as an example, the image processing method provided by the embodiments of the present application is exemplarily described. In this scenario, the mobile phone includes a camera application, and the user can use the photographing function of the camera application for shooting. The mobile phone can include a main camera (which can be referred to as a second camera), a long-focus camera (which can be referred to as a first camera), and a super-wide-angle camera, and the like. The user can use any camera to shoot in the process of using the mobile phone.

[0055] In one possible implementation, the user uses the long-focus camera of the mobile phone for shooting. For example, the user uses the long-focus camera of the mobile phone to shoot a scene including a moon and other objects. Figure 3aAs shown, after the phone starts the camera application, a camera preview interface 310 is displayed, which includes a zoom control 311 and a shooting control 312. The zoom control 311 is used to adjust the zoom ratio of the camera, and the shooting control 312 is used to implement the shooting function of the camera application. The camera preview interface 310 also includes a preview area 313, which displays a preview image A obtained based on the telephoto camera.

[0056] In some embodiments, the user can slide the zoom control 311 to the right to change the zoom ratio from 2x to 10x. In response to the user's adjustment operation on the zoom ratio, the phone displays a preview image B in the preview area 313, which has a smaller field of view than the preview image A.

[0057] In some embodiments, when the zoom ratio of the camera application is greater than or equal to a first zoom threshold, the phone can display a picture-in-picture window area 314 in the camera preview interface 310, so that the user can grasp the shooting position of the telephoto camera and position the shooting object more quickly and accurately. For example, the first zoom threshold is 10x. Then the user can slide the zoom control 311 to the right to change the zoom ratio from 2x to 10x. In response to the user's adjustment operation on the zoom ratio, the phone displays the picture-in-picture window area 314 in the camera preview interface 310 and displays a selection frame mark 315 in the area, which displays the same image range as the preview area.

[0058] In some embodiments, when the zoom ratio of the camera application is greater than or equal to a second zoom threshold (which can be referred to as a zoom threshold), the phone can detect whether the image captured by the telephoto camera contains the moon. When the phone detects the moon, it enters the moon shooting mode. For example, the second zoom threshold can be the same as the first zoom threshold, which is also 10x. When the zoom ratio is determined to be 10x, which is equal to the second zoom threshold, the phone can perform moon detection and enter the moon shooting mode when the moon is detected.

[0059] In some embodiments, the user can also switch the shooting mode of the camera application to the moon shooting mode. In response to the user's operation of switching to the moon shooting mode, the phone enters the moon shooting mode.

[0060] In the moon shooting mode, on the one hand, the phone performs a low-exposure processing on the preview image based on the telephoto camera to obtain a preview image C, and on the other hand, the phone starts the main camera and performs a normal exposure processing on the preview image based on the main camera to obtain a preview image D. Then the phone displays the preview image C in the preview area 313 and the preview image D in the picture-in-picture window area 314.

[0061] As Figure 3aAs shown, in some embodiments, when a user needs to photograph the moon, they can click the shooting control 312 to trigger the camera application's shooting function. In response to the user's click on the shooting control 312, the phone processes the image captured by the telephoto camera based on the image parameters of the image captured by the main camera to obtain image E (also called a photographed image). The camera preview interface 310 also includes a thumbnail display area 316, where the phone can display a thumbnail F of image E. After the user clicks on the thumbnail F of image E, the phone responds to the user's click by displaying an image viewing interface 320, which shows image E to the user. Compared to... Figure 1 In (b), the moon in image E is clear, and other objects such as the fence are also clearly imaged.

[0062] It should be noted that the above-mentioned mobile phone capturing images simultaneously through the telephoto camera and the main camera is only an example. The mobile phone can also capture images simultaneously through two other different cameras, and this application does not limit this.

[0063] like Figure 3b As shown, when the phone does not detect the moon, it remains in normal shooting mode. On one hand, the phone displays the preview image G obtained from the telephoto camera in the preview area 313, and on the other hand, the phone displays the preview image H obtained from the telephoto camera in the picture-in-picture window area 314, and displays the selection box mark 315 in this area. The image range displayed by the selection box mark 315 is the same as the image range displayed in the preview area. Both the preview image G and the preview image H are images obtained by normal exposure processing.

[0064] It should be noted that the above example of the second magnification threshold being the same as the first magnification threshold is only for illustration; the second magnification threshold may also be different from the first magnification threshold.

[0065] Next, taking a mobile phone as an example, which runs the layered Android system, we will illustrate the software structure of an electronic device.

[0066] like Figure 4 As shown, the layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system, from top to bottom, consists of the application layer, application framework layer, hardware abstraction layer, driver layer, and hardware layer.

[0067] The application layer may include a series of application packages. In this embodiment, the application package may include application packages for applications such as camera and gallery.

[0068] The application framework layer provides an application programming interface (API) and a programming framework for application programs of the application layer. The application framework layer includes some pre-defined functions. In the embodiments of the present application, the application framework layer can include a camera access interface, which can include camera management and camera devices. The camera access interface is used to provide an application programming interface and a programming framework for a camera application.

[0069] The hardware abstraction layer is an interface layer between the application framework layer and the driver layer, and provides a virtual hardware platform for the operating system. In the embodiments of the present application, the hardware abstraction layer can include a multi-camera module, a camera algorithm library and an image processing module.

[0070] In some embodiments, the multi-camera module can provide virtual hardware of the camera device 1, the camera device 2 or more camera devices. One camera device corresponds to one camera. In the embodiments of the present application, the camera device 1 corresponds to a telephoto camera, and the camera device 2 corresponds to a main camera.

[0071] The camera algorithm library can include running code and data for implementing the image processing method provided in the embodiments of the present application. The image processing module is used to receive a raw image output by a sensor, and is also used to call the camera algorithm library to process the raw image to obtain a preview image, a photographed image (which can be referred to as a photographed image) and the like for display to a user. In the embodiments of the present application, the camera algorithm library can include an SAT packaging module, an ROI packaging module, a smartAE packaging module, an AI packaging module and an AE packaging module, and the like. The calling manner of each module can be referred to the introduction of the embodiments below.

[0072] The driver layer is a layer between hardware and software. The driver layer includes drivers of various hardware. The driver layer can include a camera device driver and the like. The camera device driver is used to drive a sensor (which can also be referred to as an image sensor) of a camera to convert an image light signal into an image electrical signal, and to drive an image signal processor to pre-process the image electrical signal to obtain a raw image. In the embodiments of the present application, the sensor 1 is a sensor of a telephoto camera, and the sensor 2 is a sensor of a main camera.

[0073] Next, taking a mobile phone as an example of an electronic device, the mobile phone includes multiple cameras such as a telephoto camera and a main camera, and a user uses the telephoto camera for shooting. In combination with the system structure shown in FIG. 1, Figure 4 the image processing method provided in the embodiments of the present application is introduced. Figure 5a - Figure 5c and Figure 6 the image processing method provided in the embodiments of the present application is introduced.

[0074] It should be noted that the above Figure 4The camera access interface in the application framework layer and the camera device driver in the driver layer are described in textual form in the following embodiments, and are not included in the examples. Figure 5a - Figure 5c As shown in the image.

[0075] It should be noted that the following embodiments are implemented based on the fact that the camera application is successfully launched and the preview area of ​​its camera preview interface displays the preview image captured by the telephoto camera.

[0076] In some embodiments, combined with Figure 4 As shown, in response to a user's action of launching the camera application, such as clicking the camera application icon, or using preset gesture or voice commands to launch the camera application, the camera (also known as the camera application) calls the camera access interface corresponding to the camera application in the application framework layer. Then, the camera access interface calls the camera device 1 in the multi-camera module of the hardware abstraction layer to send a launch command to the telephoto camera, thus launching the camera device driver. The camera device driver uses the telephoto camera and image signal processor to obtain the original image. Subsequently, the image processing module of the hardware abstraction layer receives the original image and processes it to obtain a preview image. For example, the image processing module performs spatial alignment processing on the original image by calling the SAT encapsulation module of the camera algorithm library to obtain the preview image, and the camera application displays the preview image.

[0077] The image processing method may include a preview stage for normal shooting mode, a preview stage for moon-viewing shooting mode, and a shooting stage for moon-viewing shooting mode.

[0078] like Figure 5a As shown, the preview stage in normal shooting mode includes the following steps:

[0079] S501: The camera application adjusts the first zoom level to the second zoom level.

[0080] In some embodiments, the zoom ratio of the camera application can be manually adjusted by the user. When taking pictures with a mobile phone, the user may need to obtain preview images within different framing ranges, and can manually adjust the first zoom ratio to the second zoom ratio.

[0081] For example, such as Figure 3a As shown, the user can trigger the zoom control 311 on the camera preview interface 310. For example, the user can slide the zoom control 311 to the right to adjust the zoom ratio from 2x to 10x, thus reducing the field of view.

[0082] In some embodiments, the zoom level of the camera application can also be automatically adjusted by the camera application from a first zoom level to a second zoom level.

[0083] Exemplarily, in the case of switching the shooting mode, the camera application automatically enlarges the zoom ratio. As in the normal shooting mode, the user switches to the moon shooting mode, in order to obtain a clear moon, the camera application can automatically adjust the first zoom ratio to the second zoom ratio, for example, adjust 2x to 5x.

[0084] S502: The camera application determines that the second zoom ratio is greater than or equal to the first ratio threshold, and sends a display request of the picture-in-picture small window to the image processing module.

[0085] In the embodiments of the present application, the first ratio threshold is a ratio threshold at which the camera application displays the picture-in-picture small window, and the zoom ratio of the camera application is greater than or equal to the first ratio threshold, so that the mobile phone can display the picture-in-picture small window on the camera preview interface.

[0086] It should be noted that the first ratio threshold can be set in advance according to requirements, for example, the first ratio threshold can be 10 times (or written as 10X), 15 times, 20 times, or 30 times, etc., which is not limited in the present application.

[0087] The second zoom ratio is the adjusted zoom ratio, and determining that it is greater than or equal to the first ratio threshold indicates that the camera preview interface of the camera application needs to display the picture-in-picture small window, so that the camera application can send a display request of the picture-in-picture small window to the image processing module of the hardware abstraction layer through the camera access interface of the application framework layer.

[0088] S503: In response to the display request of the picture-in-picture small window, the image processing module generates a first long-focus stream and a first picture-in-picture stream by calling the SAT encapsulation module of the camera algorithm library to process the fourth raw image stream.

[0089] In some embodiments, the SAT encapsulation module is used for spatial alignment processing of the image, such as cropping, zooming, panning, rotating, etc.

[0090] It should be understood that after the camera application starts the long-focus camera, the long-focus camera continuously collects and generates the fourth raw image stream, which includes a frame of fourth raw image collected continuously. In the process of receiving the fourth raw image stream, the image processing module can call the camera algorithm library to perform spatial alignment processing on a frame of fourth raw image to obtain a frame of long-focus preview image, that is, a long-focus preview stream.

[0091] For example, the image processing module can call the SAT encapsulation module to perform cropping and zooming processing on the fourth raw image stream to obtain the long-focus preview stream, and then the image processing module displays the long-focus preview stream to the preview area of the camera preview interface of the camera application. Figure 3a As shown in the figure, the preview image B is obtained by cropping and zooming processing, and the range of the viewfinder is smaller.

[0092] In some embodiments, after receiving the display request of the picture-in-picture sub-window, the image processing module calls the SAT packaging module, the SAT packaging module copies the fourth original image stream to obtain the same fourth original image stream; then, the SAT packaging module performs spatial alignment processing on one of the fourth original image streams to obtain a long-focus preview stream, and simultaneously, the SAT packaging module performs spatial alignment processing on the other fourth original image stream to obtain a first picture-in-picture stream, which includes a frame of picture-in-picture sub-window image.

[0093] Taking a frame of image 1 included in one of the fourth original image streams as an example, the SAT packaging module first performs cropping processing on the image 1 according to the second preset magnification to obtain an image 2 satisfying the field of view angle corresponding to the second preset magnification, and then performs magnification processing on the image 2 to obtain an image 3 (i.e., a long-focus image) satisfying the size of the preview interface of the mobile phone camera, which is used as the preview image of the preview area of the camera preview interface.

[0094] Exemplarily, when the second zoom magnification is 10x, the SAT packaging module can determine the cropping frame corresponding to the second zoom magnification with the center point of the image 1 as the center, i.e., keeping the center points of the image 1 before and after cropping unchanged, and then perform cropping processing on the image 1 based on the cropping frame to obtain the image 2, and subsequently, perform magnification processing on the image 2 to obtain the image 3, so that the user can see the preview image after 5x zooming in the preview area of the camera preview interface, which can be seen from the preview image B shown in Figure 3a

[0095] Similarly, taking the same frame of image 1 included in the other fourth original image stream as an example, the SAT packaging module can first perform cropping processing on the image 1 according to the first magnification threshold to obtain an image 4 satisfying the field of view angle corresponding to the first magnification threshold, and then perform reduction processing on the image 4 to obtain an image 5 (i.e., a picture-in-picture image) satisfying the size of the picture-in-picture sub-window area; subsequently, the SAT packaging module determines a selection box corresponding to the second preset magnification with the image center point of the image 5 as the center, and marks the selection box on the image 5, so that the image processing module displays the image 5 with the selection box mark in the picture-in-picture sub-window area, and the image content (i.e., the range of the image) in the selection box mark is the same as the content contained in the preview image of the preview area, which can be seen from the images G and H shown in Figure 3b

[0096] It should be noted that in the normal shooting mode, the first long-focus stream and the first picture-in-picture stream are both subjected to normal exposure processing.

[0097] S504: The image processing module sends the first long-focus stream and the first picture-in-picture stream to the camera application.

[0098] The image processing module sends the first long-focus stream and the first picture-in-picture stream to the camera application through the camera application access interface. ​​

[0099] S505: The camera application displays the first long-focus image in the preview area and displays the first picture-in-picture image in the picture-in-picture sub-window area.

[0100] Based on the above example, the first long-focus stream includes one frame of first long-focus images (also referred to as first preview images), the first picture-in-picture stream includes one frame of picture-in-picture images (also referred to as fourth preview images), and the camera application displays one frame of first long-focus images in the preview area and displays one frame of first picture-in-picture images in the picture-in-picture sub-window area (also referred to as the first window).

[0101] In some embodiments, as shown in FIG. 3B, the preview area 313 and the picture-in-picture sub-window area 314 both display images obtained based on the long-focus camera, and the images are normally exposed and the objects contained in the images can be seen. Figure 3b

[0102] It should be noted that in the normal shooting mode, the long-focus camera continuously generates a stream, i.e., continuously sends the fourth raw image stream including one frame of fourth raw images to the image processing module, as shown in FIG. 3B. Figure 5a

[0103] As shown in FIG. 4B, the preview phase of the moon shooting mode phase includes the following steps: Figure 5b

[0104] S506: The camera application determines that the second zoom ratio is greater than or equal to the second zoom ratio threshold, and sends a moon recognition request to the image processing module.

[0105] In the embodiments of the present application, the second zoom ratio threshold is a zoom ratio threshold for the camera application to determine whether to automatically enter the moon mode. When the zoom ratio of the camera application is greater than or equal to the second zoom ratio threshold, the mobile phone can recognize whether the image contains a moon.

[0106] In some embodiments, the second zoom ratio threshold can be set in advance according to requirements, for example, the second zoom ratio threshold can be 10 times (or written as 10X), 15 times, 20 times, or 30 times, etc.

[0107] In some embodiments, the first zoom ratio threshold and the second zoom ratio threshold can be the same. For example, the first zoom ratio threshold and the second zoom ratio threshold are both 10x, indicating that when the zoom ratio of the camera application is 10x, the camera preview interface displays the picture-in-picture sub-window area on the basis of the display preview area, and the mobile phone performs moon recognition on the image to determine whether to enter the moon shooting mode.

[0108] S507: The image processing module calls the AI encapsulation module of the camera algorithm library to perform object recognition on the tiny stream.

[0109] ​​​In the embodiments of the present application, the tiny stream is a low-resolution processing stream obtained by calling the camera algorithm library to downsample the fourth original image stream in the process in which the image processing module receives the fourth original image stream.

[0110] In some embodiments, the camera algorithm library module can include a downsampling algorithm, and the image processing module calls the downsampling algorithm to downsample the fourth original image stream to obtain the tiny stream. The tiny stream includes images with low resolution, and thus can be processed quickly to improve the recognition speed of the object. It should be noted that the downsampling algorithm is only a general term for algorithms that can downsample images, and the specific algorithm is not limited in the embodiments of the present application.

[0111] For example, the size of the image in the fourth original image stream can be 3024*4032 (p), and the size of the downsampled image in the tiny stream can be 378*502 (p).

[0112] In some embodiments, the AI packaging module has an AI recognition function. Thus, the AI packaging module can identify each object contained in the image in the tiny stream through the AI recognition function to detect whether the moon exists.

[0113] For example, the AI packaging module can be packaged with an AI recognition algorithm, which is used to identify each object of the image, such as whether the image contains the moon, the sun, plants, portraits, etc., to further determine the shooting scene.

[0114] In addition, in some embodiments, the camera algorithm library can also be provided with a neural network model, which can perform object recognition. Thus, the image processing module can identify each object of the image in the tiny stream by calling the neural network model to determine whether the moon exists. The manner of moon recognition is not limited in the present application.

[0115] S508: The AI packaging module determines that the tiny stream contains the moon, and sends the recognition result that the image contains the moon to the multi-camera module and the camera application, respectively.

[0116] In some embodiments, the AI packaging module determines that the tiny stream contains the moon, which indicates that the preview image seen by the user contains the moon. The AI packaging module sends this result to the multi-camera module and sends this result to the camera application through the camera access interface. It should be noted that the order of sending the result is not limited in the present application. The result can be sent to the multi-camera module first, and then sent to the camera application. The result can also be sent simultaneously, and the present application does not limit this.

[0117] In some embodiments, when the first zoom threshold is the same as the second zoom threshold, the phone can display the picture-in-picture window region on the basis of the display preview region, and perform moon recognition on the image to determine whether to enter the moon shooting mode. That is, after performing S501, S502 and S506-S507 can be performed in sequence or simultaneously. If after performing S502 and before performing S503, the phone determines that the image contains a moon, S503-S505 can not be performed, and S508-S516 can be directly performed.

[0118] In addition, in some embodiments, the first zoom threshold is less than the second zoom threshold, and when the adjusted zoom of the camera application is greater than or equal to the first zoom threshold and less than the second zoom threshold, the phone can perform S502-S505, and then when the zoom is adjusted to be greater than the second zoom threshold again, the phone continues to perform S506-S516.

[0119] S509: The camera application enters the moon shooting mode.

[0120] After the camera application receives the result that the image contains a moon, the camera application can automatically enter the moon shooting mode.

[0121] In some embodiments, when the camera application enters the moon shooting mode, the camera preview interface can prompt the user about the current shooting mode through prompt information. The prompt information can be in at least one of the forms of text, pattern, control, etc. For example, a moon pattern can be displayed on the camera preview interface to prompt the user that the camera application has entered the moon shooting mode.

[0122] If the image contains a moon, the phone can automatically enter the moon shooting mode, otherwise the phone remains in the normal shooting mode. Therefore, in some embodiments, S503-S505 described above can also be performed after the AI encapsulation module determines that the tiny stream does not contain a moon.

[0123] S510: The camera application displays the second long-focus image in the preview region.

[0124] In some embodiments, the exposure of the second long-focus image (also referred to as the second preview image) is less than that of the first long-focus image, that is, the clarity of the moon in the second long-focus image is greater than that in the first long-focus image, and the clarity of other objects in the second long-focus image is less than that in the first long-focus image. For details, please refer to Figure 3a As shown in the figure, image B is the first long-focus image, and image C is the second long-focus image.

[0125] It should be understood that, since the long-focus camera has a large brightness of the moon when collecting the fourth original image, the moon region in the first long-focus image finally displayed to the user is displayed fuzzy, so in order to obtain a clear moon, the multiple frames of fourth original images included in the fourth original image stream need to be subjected to a decrease in exposure processing.

[0126] As Figure 3a described, the image C is the second long-focus image, which is an image obtained by performing decrease in exposure processing, and the moon is clearer, but other objects in the image are not imaged.

[0127] In some embodiments, the manner of obtaining the second long-focus image can include the following steps:

[0128] Step 1: The image processing module calls the ROI encapsulation module to select the moon region of the first long-focus image as the ROI region.

[0129] In the embodiments of the present application, the ROI encapsulation module is used to outline the region to be processed, i.e., the region of interest (which can be referred to as the ROI region), in the form of a box, a circle, an ellipse, an irregular polygon, etc.

[0130] It should be understood that, in the moon-viewing shooting mode, the camera application needs to show the user a clear moon, so the moon region needs to be taken as the ROI region.

[0131] In some embodiments, the camera algorithm library stores some first long-focus images included in the first long-focus stream, and the image processing module can select one frame therefrom and call the ROI encapsulation module to process it.

[0132] Step 2: The image processing module calls the AE encapsulation module to obtain AE statistical data based on the ROI region.

[0133] In the embodiments of the present application, the AE encapsulation module is used to count the AE statistical data of the ROI region in the image and determine the exposure parameter based on the AE statistical data to achieve automatic exposure. The AE statistical data can include the brightness of the pixels in the ROI region, the exposure time, the ISO sensitivity value, etc. The present application does not limit this.

[0134] In some embodiments, the AE encapsulation module includes an AE statistical module, which counts the ROI region to obtain the AE statistical data.

[0135] Step 3: The image processing module calls the AE encapsulation module to obtain the first exposure parameter based on the AE statistical data.

[0136] The parameters in the first exposure parameter can include the exposure time, the sensitivity, the aperture coefficient, etc. The present application does not limit this.

[0137] In some embodiments, the exposure time can be adjusted by controlling the shutter speed. The faster the shutter speed, the shorter the exposure time, and the less the exposure amount; conversely, the slower the shutter speed, the longer the exposure time, and the more the exposure amount, and the image brightness increases.

[0138] The larger the aperture (the smaller the numerical value), such as F2.8, the more the exposure amount, and the image brightness increases. The smaller the aperture (the larger the numerical value), such as F16, the less the exposure amount, and the image brightness decreases.

[0139] The sensitivity is used to measure the sensitivity of the light-sensitive components of the camera to light. The higher the sensitivity, the stronger the resolution capability to light, the more light is sensed, and the image brightness increases; conversely, the lower the sensitivity, the weaker the resolution capability to light, the less light is sensed, and the image brightness decreases. In some embodiments, the sensitivity is commonly expressed by the ISO sensitivity value, which can be divided into several levels, such as 50, 100, 200, 400, 800, 1600, 3200, … The higher the ISO sensitivity value, the stronger the light sensitivity of the light-sensitive components.

[0140] For example, the first exposure parameter can have a lower exposure time, a lower sensitivity, and / or a lower aperture coefficient, etc. compared with the AE statistical data of the first telephoto image, that is, the exposure parameter is reduced to obtain an image with low exposure amount, and the clarity of the moon is improved.

[0141] In some embodiments, the AE packaging module can include an AE algorithm module, which calculates the first exposure parameter based on the AE statistical data. For example, the AE algorithm module can encapsulate the mean method AE, the histogram improved mean method AE, the N-section statistical method, etc.

[0142] In some embodiments, the AE packaging module can query a pre-stored exposure table to determine the corresponding exposure parameter as the first exposure parameter based on the AE statistical data.

[0143] Step 4: The image processing module sends the first exposure parameter to the telephoto camera.

[0144] The image processing module sends the first exposure parameter to the telephoto camera through the camera device driver.

[0145] Step 5: The telephoto camera acquires a first raw image stream according to the first exposure parameter.

[0146] The sensor of the telephoto camera acquires image light signals and converts them into image electrical signals according to the first exposure parameter. The sensor transmits the image electrical signals to the image signal processor for preprocessing, and obtains a first raw image stream including a plurality of first raw images.

[0147] Step 6: The long-focus camera sends the first raw image stream to the image processing module.

[0148] The long-focus camera sends the first raw image stream to the image processing module through the camera device driver.

[0149] Step 7: The image processing module calls the camera algorithm library to process the first raw image stream to obtain a second long-focus stream.

[0150] In some embodiments, the image processing module calls the SAT encapsulation module to perform spatial alignment processing on the first raw image stream to obtain the second long-focus stream.

[0151] Step 8: The image processing module sends the second long-focus stream to the camera application.

[0152] The image processing module sends the second long-focus stream to the camera application through the camera application access interface

[0153] It should be noted that the implementation of steps 7-8 can refer to the processing process of the fourth raw image stream in S503-S504, which will not be described herein.

[0154] S511: The multi-camera module initiates a start request to the main camera.

[0155] After the multi-camera module receives the result that the image contains the moon, the start request is sent to the camera device driver of the driver layer, and the camera device driver can start the main camera.

[0156] On the basis of starting the long camera, a main camera is also started, that is, in the embodiment of the present application, during the process of the camera application in the moon-viewing shooting mode, the long-focus camera and the main camera of the mobile phone are in a started state.

[0157] As shown in Figure 6 After the moon is recognized, the mobile phone starts the long-focus stream and the main camera stream at the same time.

[0158] S512: The main camera captures a second raw image stream.

[0159] The second raw image stream including multiple frames of second raw images is obtained based on the sensor and the image signal processor of the main camera.

[0160] In some embodiments, the main camera captures the second raw image stream based on a second exposure parameter. The second exposure parameter is generated by the image processing module calling the ROI encapsulation module to select the entire region of the image as the ROI region, and then calling the AE encapsulation module, that is, the image captured by the main camera is balanced in brightness of each object in the entire image, and the image is normally exposed and processed, and the image normally converges.

[0161] S513: The main camera sends the second raw image stream to the image processing module.

[0162] S514: The image processing module calls the SAT packaging module to perform spatial alignment processing on the second raw image stream to generate a second picture-in-picture stream.

[0163] It should be understood that the main camera and the long-focus camera are installed at different positions on the mobile phone, and the images obtained by shooting will have a displacement deviation. In order to enable the user to see two spatially aligned images and improve the user experience, the SAT packaging module is required to take the first raw image stream collected by the long-focus camera as a reference to perform spatial alignment processing on the second raw image stream collected by the main camera.

[0164] In some embodiments, the displacement deviation of the moon in the first raw image stream and the moon in the second raw image stream can be calculated.

[0165] Exemplarily, one frame of the first raw image corresponds to a coordinate system (which can be referred to as a first coordinate system), and one frame of the second raw image corresponds to another coordinate system (which can be referred to as a second coordinate system). Both coordinate systems take the top-left corner vertex of the image as the coordinate system origin. The SAT packaging module first determines the coordinates (x1, y1) of the center point of the moon in the first raw image, and then determines the coordinates (x2, y2) of the center point of the moon in the second raw image. Then, the coordinate deviation of the two coordinates is calculated. Subsequently, after the SAT packaging module determines the cropping frame of the two images based on the first magnification threshold, the cropping frame of the second raw image is first moved based on the coordinate deviation of the two coordinates, and then the second raw image is cropped based on the moved cropping frame to obtain an image that satisfies the field of view angle corresponding to the first magnification threshold. Then, the image is reduced to obtain a picture-in-picture image that satisfies the size of the picture-in-picture small window region. Subsequently, a selection box corresponding to the second preset magnification is determined, and the selection box is marked on the picture-in-picture image with the image center point of the picture-in-picture image as the center. The image processing module sends the picture-in-picture image 2 in the picture-in-picture small window region to display the selection box mark. See Figure 3a As shown, image D is the picture-in-picture image.

[0166] It should be noted that the first vertex of the first raw image can be taken as the origin of the first coordinate system, and the second vertex of the second raw image can be taken as the origin of the second coordinate system. The position of the first vertex in the first raw image is equivalent to the position of the second vertex in the second raw image. For example, the first vertex and the second vertex are both top-left corner vertices.

[0167] Exemplarily, the coordinate 1 of the center point of the moon in the first original image (which can be referred to as a first coordinate point) is (300, 400), and the coordinate 2 of the center point of the moon in the second original image (which can be referred to as a second coordinate point) is (400, 300). The coordinate value of the coordinate 2 can be subtracted from the coordinate value of the coordinate 1, so that the offset of the horizontal coordinates is +100 and the offset of the vertical coordinates is -100. Then, the cropping frame of the second original image can be offset by 100 pixel points to the left and 100 pixel points downward.

[0168] It should be noted that the first coordinate point and the second coordinate point are center points, which are only examples. Other coordinate points of the moon can also be used, and the position of the second coordinate point on the moon is equivalent to that of the first coordinate point.

[0169] In addition, in some embodiments, the displacement deviation of the first original image and the second original image can also be calculated in a manner of automatic focusing laser and calibration distance calculation, which is not limited in the present application.

[0170] S515: The image processing module sends the second picture-in-picture stream to the camera application.

[0171] S516: The camera application displays the second picture-in-picture image in the picture-in-picture window area.

[0172] In some embodiments, as shown in Figure 3a , the second long-focus image (i.e., preview image C) displayed in the preview area has a lower exposure, and the moon is clearer, but other objects cannot be imaged. The second picture-in-picture image (i.e., preview image D, which can be referred to as a third preview image) displayed in the picture-in-picture window area has a higher exposure, and other objects can be imaged, but the clarity of the moon is poor.

[0173] As shown in Figure 6 , in the moon shooting mode, on the one hand, the long-focus stream is subjected to a lower exposure processing to obtain a clearer moon displayed in the camera preview (i.e., the preview area of the camera preview interface); on the other hand, the mobile phone no longer uses the long-focus stream (which can be equivalent to the first picture-in-picture stream) to display in the picture-in-picture window, but starts the main camera to obtain the main camera stream (which can also be referred to as the second picture-in-picture stream) and performs normal exposure processing on it. The obtained image is displayed in the picture-in-picture window (i.e., the picture-in-picture window area of the camera preview interface).

[0174] In some embodiments, as shown in Figure 5bAs shown, in the moon shooting mode, the long-focus camera generates a first raw image stream, and the main camera generates a second raw image stream. The mobile phone processes the first raw image stream and the second raw image stream respectively, and each processing does not affect the other. After the image processing module detects the moon by calling the AI encapsulation module, the mobile phone executes steps S1-S8 and S510. After the multi-camera module receives the recognition result that the image contains the moon, the mobile phone executes S511-S516.

[0175] As shown in FIG. 1, the moon shooting mode includes the following steps: Figure 5c

[0176] S517: The camera application receives a trigger operation of the shooting function by the user.

[0177] After the user determines that the camera of the mobile phone is aimed at the object to be photographed through the preview image displayed on the screen of the mobile phone, the user can trigger the shooting function of the camera application.

[0178] In some embodiments, as shown in FIG. 1, the trigger operation of the shooting function can be a click operation of the user on the shooting control 312. Figure 3a

[0179] In some embodiments, the trigger operation of the shooting function can be a voice instruction and a gesture instruction of the user on the camera application, and the camera application can be triggered to take a photo. The present application does not limit this.

[0180] S518: The camera application sends a shooting request to the image processing module.

[0181] The camera application sends a shooting request to the image processing module through the camera access interface.

[0182] S519: The image processing module calls the smartAE encapsulation module of the camera algorithm library to calculate a first exposure sequence.

[0183] In some embodiments, during the preview stage of the moon shooting mode, the image processing module can call the camera algorithm library to cache a plurality of second picture-in-picture images included in the second picture-in-picture image stream (i.e., images taken by the main camera), and cache the AE statistical data of each frame of the second picture-in-picture image. Similarly, in some embodiments, during the preview stage of the moon shooting mode, the image processing module can also call the camera algorithm library to cache a plurality of second long-focus images included in the second long-focus image stream.

[0184] ​​Exemplarily, the image processing module can cache the multiple frames of the second picture-in-picture image and the AE statistical data of each frame of the second picture-in-picture image into a cache queue. The cache queue can be a ZSL queue or the like, and the second picture-in-picture images can be sequentially stored in the ZSL queue in the order of frame output. When all the storage positions in the ZSL queue are occupied, the ZSL queue can replace the historical second picture-in-picture images in the ZSL queue with the latest collected second picture-in-picture images in real time. The caching process of the second long-focus image is the same, and thus will not be described herein again.

[0185] In some embodiments, the smartAE packaging module can determine a frame of the second picture-in-picture image from the multiple frames of the second picture-in-picture image in one cache queue, and determine the AE statistical data of the frame of the second picture-in-picture image as the reference exposure parameter. The smartAE packaging module can also determine a frame of the second long-focus image from the multiple frames of the second long-focus image in another cache queue as the reference frame image. Then, the smartAE packaging module can determine the frame number of the third original image, the exposure parameter of each frame of the third original image, and the frame output order of the third original image based on the reference exposure parameter and the reference frame, and arrange the exposure parameters of the third original image in the frame output order to generate the first exposure sequence.

[0186] The first exposure sequence includes multiple sets of exposure parameters (one frame of the third original image corresponds to one set of exposure parameters). Exemplarily, the multiple sets of exposure parameters can include the exposure parameters of the middle frame image and the exposure parameters of the long frame image. Exemplarily, the multiple sets of exposure parameters can include the exposure parameters of the middle frame image (which can be referred to as the second set of exposure parameters), the exposure parameters of the long frame image (which can be referred to as the first set of exposure parameters), and the exposure parameters of the short frame image (which can be referred to as the third set of exposure parameters).

[0187] In some embodiments, the reference frame image corresponds to the middle frame image.

[0188] As can be seen from the above, at least two sets of exposure parameters corresponding to the long frame image and the middle frame image in the multiple sets of exposure parameters have different values. For example, the two sets of exposure parameters can have different values of exposure time, different values of sensitivity, or different values of each parameter, which is not limited in the present application.

[0189] Exemplarily, taking the exposure time in the exposure parameter as an example, the smartAE packaging module can calculate the exposure time corresponding to the long frame image and the short frame image according to the exposure time of the reference frame image based on the exposure time of the reference exposure parameter, and determine the number of the long frame image, the middle frame image, and the short frame image required.

[0190] For example, in a moonlight shooting scene, a long frame image, five middle frame images, and two short frame images are needed. The sequence and exposure time can be represented by the first exposure sequence, which can be represented as {N, S, N, S, N, N, N, L, N, S = 5, N = 10, S = 5, N = 10, N = 10, N = 10, L = 50}, where N represents a middle frame image, L represents a long frame image, and S represents a short frame image. N = 10 represents that the exposure time of the middle frame image is 10 ms, S = 5 represents that the exposure time of the short frame image is 5 ms, and L = 50 represents that the exposure time of the long frame image is 50 ms.

[0191] In some embodiments, the reference frame image can be the last frame of the second long focal image in the cache queue when the smartAE encapsulation module receives the shooting instruction.

[0192] In some embodiments, considering the time delay between the user triggering the shooting function and the image processing module receiving the shooting instruction, the last frame of the second long focal image in the cache queue is not the frame image that the user actually wants. Therefore, a frame of the second long focal image with an earlier timestamp can be selected from the cache queue as the reference frame image. In this way, by selecting the second long focal image output closer to the time when the user issues the shooting instruction as the reference frame image, subsequent processing based on the reference frame image can obtain an image that better meets the user's needs, thereby improving the user experience.

[0193] In addition, in some embodiments, the image processing module can call the camera algorithm library to cache the multiple frames of the second raw image included in the second raw image stream, or call the camera algorithm library to cache the multiple frames of the first raw image included in the first raw image stream. Subsequently, when processing, the reference exposure parameter can be determined based on the second raw image captured by the main camera, and the reference frame image can be determined based on the first raw image captured by the long focal camera.

[0194] S520: The image processing module sends the first exposure sequence to the long focal camera.

[0195] The image processing module sends the first exposure sequence to the sensor of the long focal camera through the camera device driver.

[0196] S521: The long focal camera sequentially captures multiple frames of third raw images according to the first exposure sequence.

[0197] The sensor of the long focal camera captures according to the output sequence of the first exposure sequence and the exposure parameter corresponding to each frame, and the image signal processor sequentially outputs multiple frames of third raw images after preprocessing. The number of frames of the multiple frames of third raw images is the same as the number of frames of the first exposure sequence, and the multiple frames of third raw images correspond one-to-one to the multiple sets of exposure parameters of the first exposure sequence.

[0198] S522: The long-focus camera sends the multiple third raw images to the image processing module.

[0199] The long-focus camera sends the multiple third raw images to the image processing module through the camera device driver.

[0200] S523: The image processing module processes the reference frame image based on the multiple third raw images to generate a photographed image.

[0201] In some embodiments, the camera algorithm library can include an exposure fusion algorithm. Based on the above example, the multiple third raw images include a long-frame image and a medium-frame image, and the image processing module fuses the multiple third raw images into the reference frame image by calling the exposure fusion algorithm, so that the dark area in the reference frame image is brightened by long exposure, that is, the brightness of other objects in the image is improved, so that the imaging of other objects is clearer, for example, the foreground is clearer.

[0202] In some embodiments, based on the above example, the multiple third raw images include a long-frame image, a medium-frame image, and a short-frame image, and the image processing module fuses the multiple third raw images into the reference frame image by calling the exposure fusion algorithm, so that the imaging of other objects is clearer while the bright area is restored by short exposure, that is, the brightness of the lunar region in the reference frame image is further reduced, so that the lunar texture and edge are clearer.

[0203] As an example, the image processing module can determine a mapping parameter according to the exposure parameter of the third raw image, the mapping parameter being used to adjust the exposure parameter of the reference frame image; adjust the exposure parameter of the reference frame image according to the mapping parameter; and fuse the reference frame image with the adjusted reference frame image to obtain the photographed image.

[0204] As shown in Figure 6 , the main camera AE statistics, that is, the AE statistics of the second picture-in-picture image photographed by the main camera, and the preview image, that is, the reference frame image, the mobile phone can perform multi-frame processing based on the main camera AE statistics and the reference frame image to obtain the final photographed image, so as to display it to the user.

[0205] S524: The image processing module sends the photographed image to the camera application.

[0206] The image processing module sends the photographed image to the camera application through the camera access interface.

[0207] S525: The camera application displays a thumbnail of the photographed image.

[0208] The camera application displays a thumbnail of the photographed image to the user.

[0209] In some embodiments, as Figure 3aAs shown, after the user clicks the shooting control 312, the mobile phone displays a thumbnail of the shot image (i.e., thumbnail F) in the thumbnail display area 316 (which can also be referred to as a second window).

[0210] Next, the components of the electronic device are introduced.

[0211] It should be noted that the electronic device is a mobile phone in the above embodiments, which is only an example. In some embodiments, the electronic device can be a terminal device such as a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The specific form of the electronic device is not specially limited in the present application, as long as it can realize the photographing function.

[0212] As shown in FIG. 7, the electronic device 700 can include a processor 710, an internal memory 720, a camera 730, and a display screen 740. Figure 7

[0213] It can be understood that the structure shown in the present embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device can include more or fewer components than shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0214] The processor 710 can include one or more processing units, for example: the processor 710 can include an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP), a digital signal processor (DSP), etc. Different processing units can be independent devices, or can be integrated in one or more processors.

[0215] The processor 710 can also be provided with a memory for storing instructions and data.

[0216] The internal memory 720 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 710 executes various functional applications and data processing of the electronic device 700 by running the instructions stored in the internal memory 720.​

[0217] In some embodiments, the internal memory 720 stores instructions for performing the image processing method. The processor 710 can implement the image processing method provided by the embodiments of the present application by executing the instructions stored in the internal memory 720.

[0218] The electronic device 700 implements the display function through the image processor, the display screen 740, and the application processor, etc. The image processor is a microprocessor for image processing, connected to the display screen 740 and the application processor. The image processor is used to perform mathematical and geometric calculations for graphics rendering. The processor 710 can include one or more image processors that execute program instructions to generate or alter display information. The display screen 740 is used to display images, videos, etc.

[0219] In some embodiments, the display screen 740 is used to display the running interface of the camera application of the electronic device 700, such as a camera preview interface, an image viewing interface, etc.

[0220] The electronic device 700 can implement the shooting function through the ISP, the camera 730, the video codec, the image processor, the display screen 740, and the application processor, etc.

[0221] In some embodiments, the shooting function includes the photographing function and the video recording function of the camera application of the electronic device 700.

[0222] The ISP is used to process the data fed back by the camera 730. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also optimize the algorithm of the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene.

[0223] In some embodiments, the ISP can be arranged in the camera 730. The camera 730 is used to capture still images or videos. Objects generate optical images through lenses and project them onto photosensitive elements (i.e., sensors). The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the image light signal into an image electrical signal, and then transmits the image electrical signal to the ISP to convert it into a digital image signal (i.e., a raw image). The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into a standard RGB, YUV, etc. format image signal.

[0224] In some embodiments, the electronic device 700 can include N cameras 730, N being a positive integer greater than 1. Exemplarily, the electronic device 700 includes a telephoto camera and a main camera.

[0225] The embodiments of the present application further provide a computer readable storage medium storing a computer program, and the computer program can implement one or more steps in any of the above image processing methods when executed by a computer.

[0226] The computer readable storage medium can be a non-transitory computer readable storage medium, for example, the non-transitory computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0227] The embodiments of the present application further provide a computer program product containing instructions. The computer program product can implement one or more steps in any of the above image processing methods when executed by a computer.

[0228] The electronic device, the computer readable storage medium, and the computer program product provided by the embodiments of the present application are all used to execute the corresponding image processing method provided above, and thus the beneficial effects that can be achieved are referable to the beneficial effects of the corresponding image processing method provided above, which will not be described here again.

[0229] The terms "first", "second", and "third" and the like in the specification of the present application, the claims, and the accompanying drawings are used to distinguish different objects, and are not used to limit a specific sequence.

[0230] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as "exemplary" or "for example" in the embodiments of the present application should not be construed as being superior or superior to other embodiments or design solutions. Rather, the use of "exemplary" or "for example" is intended to present relevant concepts in a concrete manner.

[0231] The above-described and above-embodied examples are only used to illustrate the technical solutions of the present application, and not to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features. Such modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that, include: In response to the camera application being launched, a first preview image is displayed on the preview interface of the camera application; In response to the operation of increasing the first zoom level to the second zoom level, it is determined that the second zoom level is greater than or equal to the zoom level threshold, and the object contained in the first preview image is detected. The target object contained in the first preview image is determined, and a second preview image and a third preview image are obtained by the first camera. A second preview image is displayed on the preview interface, and a third preview image is displayed in the first window; wherein, the clarity of the target object in the second preview image is greater than the clarity of the target object in the third preview image; the clarity of other objects in the second preview image besides the target object is less than the clarity of other objects in the third preview image besides the target object. In response to a trigger operation of the camera application's photo-taking function, a thumbnail of the captured image is displayed in the second window of the preview interface; the sharpness of the target object in the captured image is greater than or equal to the sharpness of the target object in the second preview image, and the sharpness of other objects in the captured image besides the target object is greater than the sharpness of other objects in the second preview image besides the target object.

2. The method according to claim 1, characterized in that, The steps for obtaining the second preview image include: The second preview image is obtained based on the first exposure parameter, which is lower than the exposure parameter of the first preview image.

3. The method according to claim 1, characterized in that, The steps for obtaining the third preview image include: The third preview image is obtained based on the second exposure parameter, which is higher than the exposure parameter of the second preview image.

4. The method according to claim 1, characterized in that, The first preview image is acquired through the first camera; the step of determining that the first preview image contains the target object, acquiring the second preview image acquired through the first camera, and the third preview image acquired through the second camera includes: Once it is determined that the first preview image contains the target object, the second camera is activated; Acquire a second preview image captured by the first camera and a third preview image captured by the second camera.

5. The method according to claim 1, characterized in that, The first preview image was captured by the first camera; The step of determining that the first preview image contains the target object and acquiring the second preview image captured by the first camera includes: Once it is determined that the first preview image contains the target object, the exposure parameters of the first camera are reduced; Based on the reduced exposure parameters, the second preview image is obtained by capturing it through the first camera.

6. The method according to claim 1, characterized in that, The operation of increasing the first zoom level of the camera application to the second zoom level, displaying a second preview image on the preview interface of the camera application, and displaying a third preview image in the first window of the preview interface, includes: In response to the operation of increasing the first zoom level to the second zoom level, the first window pops up in the preview interface to display a fourth preview image; the clarity of the first preview image is the same as that of the fourth preview image. If the first preview image is determined to contain the target object, the first preview image is updated to the second preview image in the preview interface, and the fourth preview image is updated to the third preview image in the first window.

7. The method according to any one of claims 1-6, characterized in that, The steps for obtaining the photographed image include: The captured image is generated by fusing the first original image and the second original image. Both the first original image and the second preview image are acquired by the first camera; both the second original image and the third preview image are acquired by the second camera.

8. The method according to claim 7, characterized in that, The process of generating the captured image by fusing the first original image and the second original image includes: Based on the first original image and the second original image, a first exposure sequence is determined; the first exposure sequence includes multiple sets of exposure parameters, and the multiple sets of exposure parameters include at least two sets of exposure parameters with different values. Based on the first exposure sequence, multiple frames of third original images are acquired by the first camera; each of the multiple frames of third original images corresponds one-to-one with the multiple sets of exposure parameters. The captured image is generated by fusing the multiple frames of the third original image and the first original image.

9. The method according to claim 8, characterized in that, The first exposure sequence includes a first set of exposure parameters and a second set of exposure parameters; the value of the first set of exposure parameters is greater than the value of the second set of exposure parameters.

10. The method according to claim 8, characterized in that, The first exposure sequence includes a first set of exposure parameters, a second set of exposure parameters, and a third set of exposure parameters; the value of the first set of exposure parameters is greater than the value of the second set of exposure parameters; the value of the second set of exposure parameters is greater than the value of the third set of exposure parameters.

11. The method according to claim 10, characterized in that, The values ​​of the second set of exposure parameters are obtained based on the first original image, and the values ​​of the first set of exposure parameters and the third set of exposure parameters are obtained based on the first original image and the second original image.

12. The method according to claim 7, characterized in that, The steps for obtaining the third preview image include: Based on the first original image, the second original image is spatially aligned to obtain the third preview image.

13. The method according to claim 12, characterized in that, The first vertex of the first original image is the origin of the first coordinate system; the second vertex of the second original image is the origin of the second coordinate system; the position of the first vertex in the first original image is the same as the position of the second vertex in the second original image. The step of obtaining the third preview image by spatially aligning the second original image based on the first original image includes: Determine the first coordinate point of the target object contained in the first original image in the first coordinate system; Determine the second coordinate point of the target object contained in the second original image in the second coordinate system; the position of the second coordinate point on the target object is the same as the position of the first coordinate point on the target object. Calculate the coordinate deviation between the first coordinate point and the second coordinate point; Based on the coordinate deviation value, the second original image is spatially aligned to obtain the third preview image.

14. The method according to any one of claims 1-13, characterized in that, The target object includes the moon; the clarity of the moon in the second preview image is greater than that of the moon in the third preview image; the clarity of other objects in the second preview image besides the moon is less than that of other objects in the third preview image besides the moon.

15. An electronic device, characterized in that, Including memory and processor; The memory is coupled to the processor and is used to store computer program code, the computer program code including computer instructions, wherein one or more of the processors invoke the computer instructions to cause the electronic device to perform the image processing method as described in any one of claims 1-14.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the image processing method as described in any one of claims 1-14.

Images