Image processing method, electronic equipment and storage medium

Abstract

The invention provides an image processing method, electronic equipment and a storage medium, and relates to the technical field of image processing. In the method, a first preview image is displayed on a preview interface of a camera application in response to the fact that the camera application is in a starting state; in response to an operation that the first zoom magnification of the camera application is adjusted to a second zoom magnification, displaying a second preview image in a preview interface of the camera application, and displaying a third preview image in a first window of the preview interface; and in response to a trigger operation on a photographing function of the camera application, displaying a thumbnail of the photographed image in a second window of the preview interface. The target object in the second preview image is clear, compared with the second preview image, the definition of the target object in the photographed image can be kept unchanged or higher, and the definition of other objects except the target object in the photographed image is higher, so that a real scene seen by a user can be restored as much as possible, and the image photographing effect is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and in particular, to an image processing method, an electronic device, and a storage medium. Background Art

[0002] With the wide popularization of electronic devices such as mobile phones and tablet computers, as well as the rapid development of photography technologies, the shooting functions of electronic devices have gradually become richer, and more and more users choose to use electronic devices to record wonderful moments in life. For example, an electronic device can provide a moon-viewing shooting mode, in which the electronic device can capture a relatively clear image of the moon.

[0003] However, other objects in the image where the moon is located are likely to fail to be imaged or the imaging is not clear, that is, there may be a situation where the image captured by the electronic device does not match the real scene seen by the user, affecting the user experience. Summary of the Invention

[0004] To solve the above problems, this application provides an image processing method, an electronic device, and a storage medium, aiming to improve the image shooting effect of the electronic device and restore the real scene seen by the user as much as possible.

[0005] In a first aspect, this application provides an image processing method. Exemplarily, this method can be applied to an electronic device, which can be a device including a camera application such as a mobile phone, a tablet computer, a laptop computer, etc. In this method, after the camera application is started, for example, when the user clicks on the icon of the camera application, or the user issues a gesture instruction and a voice instruction, etc. to start the camera application, the electronic device can display a first preview image on the preview interface of the camera application; then, when the first zoom ratio of the camera application is adjusted to a second zoom ratio, for example, when the user slides the zoom control of the camera application to increase its zoom ratio, in the case where the second zoom ratio is greater than the first zoom ratio, the electronic device can display a second preview image on the preview interface of the camera application, and display a third preview image in a first window on this preview interface. Exemplarily, the third preview image can be displayed on the preview interface in the form of a picture-in-picture small window, then the first window can be the picture-in-picture small window. Also exemplarily, when the electronic device is a foldable mobile phone, the second preview image can be displayed in the left window of the preview interface, and the third preview image can be displayed in the right window (that is, the first window) of the preview interface. Through the preview interface of the camera application, it can be seen that the clarity of the target object in the second preview image is greater than that of the target object in the third preview image; the clarity of other objects in the second preview image except the target object is less than that of other objects in the third preview image except the target object. Exemplarily, the target object can be the moon, a lamp, etc., and other objects can be buildings, people, animals and plants, etc.

[0006] After the shooting function of the camera application is triggered, for example, when the user clicks the shooting control of the camera application, or when the user issues gesture commands and voice commands to trigger the shooting function, the electronic device can display a thumbnail of the captured image in the second window of the preview interface. Exemplarily, the second window can be the thumbnail display area of the preview interface, which can be located at any position such as the lower left corner of the preview interface. From the captured image, it can be seen that the clarity of the target object therein is greater than or equal to the clarity of the target object in the second preview image, and the clarity of other objects except the target object is greater than the clarity of other objects except the target object in the second preview image.

[0007] In this way, when comparing the second preview image and the third preview image, the clarity of the target object in the second preview image is higher, and the clarity of other objects except the target object in the third preview image is higher. When comparing the second preview image and the captured image, the clarity of the target object in the captured image can remain the same or be higher, and the clarity of other objects except the target object in the captured image is higher. That is, after the shooting function of the camera application is triggered, a captured image with both the target object and other objects being relatively clear can be obtained. Therefore, the image processing method provided in this application can obtain a relatively clear captured image, improve the image shooting effect of the electronic device, and restore the real scene seen by the user as much as possible.

[0008] In a possible implementation manner, after the first zoom ratio of the camera application is adjusted to a second zoom ratio, for example, from 2x to 10x, the electronic device can compare the second zoom ratio with the zoom ratio threshold. When it is determined that the second zoom ratio is greater than or equal to the zoom ratio threshold, exemplarily, the zoom ratio threshold can be 10x, then at this time, the electronic device can detect the objects included in the first preview image; then, when the electronic device detects that the first preview image includes the target object, it can obtain a second preview image captured by the first camera and a third preview image captured by the second camera; subsequently, the electronic device can display the second preview image in the preview interface of the camera application and display the third preview image in the first window.

[0009] In this way, when it is determined that the first preview image includes the target object, two images with different clarities can be presented to the user, which can not only enable the user to see the relatively clear target object but also enable the user to see the relatively clear other objects, improving the user experience.

[0010] In a possible implementation, the above-mentioned second preview image can be obtained through the following steps: The electronic device can collect the second preview image based on the first exposure parameter, where the first exposure parameter is lower than the exposure parameter of the first preview image. In this way, obtaining the second preview image through reduced exposure processing can make the objects in the bright area of the second preview image clearer. For example, when the target object is a relatively bright object, the clarity of the target object can be improved.

[0011] In a possible implementation, the above-mentioned third preview image can be obtained through the following steps: The electronic device can collect the third preview image based on the second exposure parameter, where the second exposure parameter is higher than the exposure parameter of the second preview image. For example, when obtaining the second preview image through reduced exposure processing, the third preview image can be obtained through normal exposure processing. In this way, compared with the second preview image, the objects in the dark area of the third preview image are clearer. For example, when the target object is a relatively bright object, the clarity of the objects other than the target object in the third preview image can be improved.

[0012] In a possible implementation, the electronic device can collect the first preview image through the first camera. When it is determined that the first preview image contains the target object, the electronic device can activate the second camera; subsequently, the electronic device can obtain the second preview image collected through the first camera and the third preview image collected through the second camera. Exemplarily, the first camera can be a telephoto camera, and the second camera can be a main camera. In this way, when it is determined that the shooting scene contains the target object, on the basis that the first camera has been activated, another second camera can be activated to obtain the second preview image and the third preview image with different exposure amounts. When there are bright and dark areas in the image, it is convenient for the user to see a clearer shooting scene through the two images.

[0013] In a possible implementation, the above-mentioned first preview image can be collected through the first camera. When it is determined that the first preview image contains the target object, the electronic device can reduce the exposure parameter of the first camera; subsequently, based on the reduced exposure parameter, the electronic device collects the second preview image through the first camera. In this way, collecting the second preview image through reduced exposure processing can improve the clarity of the objects with high self-brightness (i.e., the bright area in the image) compared with the first preview image. When the target object is an object with high self-brightness such as the moon or a lamp, the user can see a clearer target object, thereby improving the user experience.

[0014] In a possible implementation, when the first zoom magnification is increased to the second zoom magnification, the electronic device can pop up a first window on the preview interface to display a fourth preview image, where the clarity of the first preview image is the same as that of the fourth preview image. Exemplarily, a picture-in-picture window can be popped up to display the fourth preview image, and the first preview image and the fourth preview image can be obtained through the same camera. When it is determined that the first preview image contains a target object, the electronic device can update the first preview image to a second preview image on the preview interface and update the fourth preview image to a third preview image in the first window. In this way, when it is determined that there is a target object in the shooting scene, the updated images enable the user to see the target object and other objects through the preview interface, improving the user experience.

[0015] In a possible implementation, the first original image and the second preview image can be acquired through the first camera, indicating that the clarity of the target object and other objects in both is the same. The second original image and the third preview image can be acquired through the second camera, indicating that the clarity of the target object and other objects in both is also the same. The electronic device can perform a fusion process on the first original image and the second original image to generate a captured image. In this way, the moon has higher clarity in the first original image, and other objects except the moon have higher clarity in the second original image. Based on the two, a captured image with clear moon and other objects can be obtained, that is, the image capture effect of the electronic device can be improved, and the real scene seen by the user can be restored as much as possible.

[0016] In a possible implementation, the electronic device can determine a first exposure sequence based on the first original image and the second original image. The first exposure sequence includes multiple sets of exposure parameters. The parameters in a set of exposure parameters can include exposure time, ISO, aperture coefficient, etc. At least two sets of exposure parameters in the multiple sets of exposure parameters have different values. For example, the values of the exposure time are different, the values of the ISO are different, or the values of each parameter are different. Then, the electronic device can acquire multiple frames of third original images through the first camera according to the first exposure sequence. It should be understood that the multiple frames of third original images correspond to the multiple sets of exposure parameters one by one. Subsequently, the electronic device performs a fusion process on the multiple frames of third original images and the first original image to generate a captured image. In this way, by fusing multiple frames of third original images with different exposure parameters and the first original image, the exposure parameters of the first original image can be adjusted to make other objects in it clearer, and thus a captured image with better quality can be obtained.

[0017] In a possible implementation, the above first exposure sequence may include a first set of exposure parameters and a second set of exposure parameters, where the value of the first set of exposure parameters is greater than the value of the second set of exposure parameters. Exemplarily, the first set of exposure parameters includes the exposure parameters corresponding to the long-frame image, and the second set of exposure parameters includes the exposure parameters corresponding to the medium-frame image. In this way, the other objects in the dark area of the first original image can be made clearer by using the third original image corresponding to the first set of exposure parameters.

[0018] In a possible implementation, the above first exposure sequence may include a first set of exposure parameters, a second set of exposure parameters, and a third set of exposure parameters, where the value of the first set of exposure parameters is greater than the value of the second set of exposure parameters; the value of the second set of exposure parameters is greater than the value of the third set of exposure parameters. Exemplarily, the first set of exposure parameters includes the exposure parameters corresponding to the long-frame image, the second set of exposure parameters includes the exposure parameters corresponding to the medium-frame image, and the third set of exposure parameters includes the exposure parameters corresponding to the short-frame image. In this way, the objects (such as other objects except the target object) in the dark area of the first original image can be made clearer by using the third original image corresponding to the first set of exposure parameters, and the objects (such as the target object) in the bright area of the first original image can also be made clearer by using the third original image corresponding to the third set of exposure parameters.

[0019] In a possible implementation, the value of the above second set of exposure parameters is obtained based on the first original image. For example, the exposure parameters of the first original image are used as the second set of exposure parameters, and the values of the first set of exposure parameters and the third set of exposure parameters are obtained based on the first original image and the second original image. Exemplarily, the values of the first set of exposure parameters and the third set of exposure parameters can be determined based on the exposure parameters of the first original image. For example, the first set of exposure parameters is obtained by increasing the exposure parameters of the first original image, and the third set of exposure parameters is obtained by decreasing the exposure parameters of the first original image. In this way, based on the exposure parameters of the first original image, and the target object in the first original image is relatively clear, so it is possible to avoid a decrease in the clarity of the target image in the finally generated captured image.

[0020] In a possible implementation, the step of obtaining the above third preview image may include: the electronic device performs spatial alignment processing on the second original image based on the first original image to obtain the third preview image. In this way, considering the problem that different cameras are located at different positions on the electronic device, resulting in a displacement deviation in the content included in the captured original images, based on the first original image, the second original image is spatially aligned with it, so that the user can see the spatially aligned second preview image and third preview image on the preview interface of the electronic device, improving their usage experience.

[0021] In a possible implementation, the first vertex of the first original image is the origin of the first coordinate system, and the second vertex of the second original image is the origin of the second coordinate system. The position of the first vertex in the first original image is the same as the position of the second vertex in the second original image. Exemplarily, both the first vertex and the second vertex can be the upper left corner vertices of the image. The electronic device can determine the first coordinate point corresponding to the target object included in the first original image in the first coordinate system; then determine the second coordinate point corresponding to the target object included in the second original image in the second coordinate system, where the position of the second coordinate point on the target object is the same as the position of the first coordinate point on the target object. Exemplarily, both the first coordinate point and the second coordinate point can be the center points of the target object. Then, the electronic device can calculate the coordinate deviation value between the first coordinate point and the second coordinate point. Subsequently, the electronic device can perform spatial alignment processing on the second original image based on the coordinate deviation value to obtain a third preview image. For example, the second original image is moved, cropped, etc. to obtain a third preview image that is spatially aligned with the first original image. In this way, the user can see the second preview image and the third preview image that are spatially aligned on the preview interface of the electronic device, improving their usage experience.

[0022] In a possible implementation, the above target object can include the moon. Then, the clarity of the moon in the second preview image is greater than the clarity of the moon in the third preview image, and the clarity of other objects except the moon in the second preview image is less than the clarity of other objects except the moon in the third preview image. It should be understood that the moon itself has a higher brightness. The user can see the second preview image containing a clear moon on the preview interface and can also see the third preview image containing clear other objects in the first window, which can improve their usage experience.

[0023] In a second aspect, the present application provides an electronic device, which includes a memory and a processor; the memory stores computer program code, and the computer program code includes computer instructions; one or more processors call the computer instructions to enable the electronic device to execute the image processing method in the first aspect above.

[0024] In a third aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the image processing method in the first aspect above is implemented.

[0025] It can be seen from the above technical solutions that the present application has the following beneficial effects:

[0026] During the startup process of the camera application, the preview interface of the camera application first displays a first preview image; when the zoom ratio of the camera application is adjusted from a first zoom ratio to a second zoom ratio, and the second zoom ratio is greater than or equal to the zoom ratio threshold value, a second preview image can be displayed on the preview interface, and a third preview image is displayed in a first window of the preview interface; after the camera function of the camera application is triggered, a thumbnail of the captured image can be displayed in a second window of the preview interface. Among them, compared with the third preview image, the moon in the second preview image is clearer, and other objects except the moon in the third preview image are clearer. Compared with the second preview image and the captured image, the clarity of the target object in the captured image can remain unchanged or be higher, and the clarity of other objects except the target object in the captured image is higher, so that a captured image with a clear moon and other objects can be obtained. In this way, the image processing method provided in this application can obtain a relatively clear captured image, improve the image shooting effect of the electronic device, and restore the real scene seen by the user as much as possible. Description of the Drawings

[0027] Figure 1 It is a schematic diagram of an image containing a moon provided by an embodiment of the present application;

[0028] Figure 2a It is a schematic diagram of the operation process of a camera application in a related technology provided by an embodiment of the present application;

[0029] Figure 2b It is a schematic diagram of a shooting scene in camera technology provided by an embodiment of the present application;

[0030] Figure 3a It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0031] Figure 3b It is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0032] Figure 4 It is a schematic diagram of the system structure of an electronic device provided by an embodiment of the present application;

[0033] Figure 5a It is a signaling interaction diagram in the preview stage of a normal shooting mode provided by an embodiment of the present application;

[0034] Figure 5b It is a signaling interaction diagram in the preview stage of a full moon shooting mode provided by an embodiment of the present application;

[0035] Figure 5c It is a signaling interaction diagram in the shooting stage of a full moon shooting mode provided by an embodiment of the present application;

[0036] Figure 6 A schematic diagram of the running process of a camera application provided by an embodiment of this application;

[0037] Figure 7 A schematic diagram of the composition of an electronic device provided by an embodiment of this application. Detailed implementation manners

[0038] For the sake of clear and concise description of the following embodiments, the vocabulary involved in the embodiments of this application will be described first. It should be understood that this description is for a clearer understanding of the embodiments of this application and does not necessarily constitute a limitation on the embodiments of this application.

[0039] Exposure amount: It refers to the intensity of the light sensed by the camera and the length of time, which affects the brightness of the image captured by the camera. When shooting through the camera of an electronic device, the exposure amount may be too high or too low, which will directly cause overexposure or underexposure of the object being photographed and the background. If overexposed, the captured image will be too bright to show the details of the bright part; if underexposed, the captured image will be too dark to show the details of the dark part.

[0040] Exposure parameters: They refer to the factors that affect the image brightness. In the embodiments of this application, the exposure parameters may include aperture, exposure time, and sensitivity.

[0041] Long-frame image: In the embodiments of this application, an image with a long exposure time is called a long-frame image.

[0042] Short-frame image: In the embodiments of this application, an image with a short exposure time is called a short-frame image.

[0043] Medium-frame image: In the embodiments of this application, an image with an exposure time between that of a long-frame image and a short-frame image is called a medium-frame image. That is, the exposure time of a long-frame image is greater than that of a medium-frame image, and the exposure time of a medium-frame image is greater than that of a short-frame image.

[0044] It should be noted that the length of the exposure time here is relative, and the specific value of the exposure time can be determined according to actual usage requirements, which are not limited in the embodiments of this application.

[0045] Next, in combination with related technologies, the technical advantages of an image processing method, an electronic device, and a storage medium provided by this application will be compared and described. For the convenience of understanding, it will be described through an example scenario. In this example scenario, the electronic device can be a mobile phone.

[0046] In related technologies, users may have the need to use a mobile phone to photograph the moon, and the mobile phone can capture an image such as Figure 1The image shown in (a) has an unclear moon image, with overexposure and defocus. Currently, in order to enable a mobile phone to capture a clear moon, more and more electronic devices provide a moon shooting mode (which can also be called a moon shooting mode, a moon mode, or a super moon mode, etc.). The image captured by the mobile phone in the moon shooting mode can be seen in Figure 1 the image shown in (b). Compared with Figure 1 (a), the moon is clearer.

[0047] Figure 1 (a) and (b) in Figure 1 are obtained by shooting the same shooting scene. By combining the two, it can be seen that the imaging clarity of the moon in the two is different. Figure 1 The moon in (a) is overexposed and defocused, Figure 1 while the moon in Figure 1 (b) is clearer; Figure 1 In (a), other objects such as trees and fence nets can be seen (which can be understood as the foreground), while Figure 1 in (b), only the relatively clear moon can be seen, and other objects except the moon cannot be imaged.

[0048] Next, in combination with Figure 2a and Figure 2b the shooting process of the mobile phone in the moon shooting mode in the related technology will be introduced.

[0049] As Figure 2a shown, when the mobile phone starts the camera application and shoots using the telephoto camera, the telephoto camera starts the data stream (which can also be called the telephoto starts the stream) to obtain the telephoto stream, and then sends the image obtained based on the telephoto stream to the camera preview. Moreover, when the camera preview displays a picture-in-picture small window, an image can also be obtained based on the telephoto stream and sent to the picture-in-picture small window. After the user triggers the shooting function, the mobile phone can perform frame processing on the single-frame image included in the telephoto stream to obtain the shooting image.

[0050] As Figure 2b shown, the mobile phone displays the camera preview interface 210 (which can be called the preview interface). The camera preview interface 210 includes a preview area 211, a picture-in-picture small window area 212, and a shooting control 213. In the moon shooting mode, the images displayed in the preview area 211 and the picture-in-picture small window 212 are both obtained based on the telephoto stream, and the moon in the image is relatively clear. When the user clicks the camera shooting control 213, the mobile phone can perform frame processing on the single-frame image included in the telephoto stream to obtain the image shown in Figure 1 (b).

[0051] As can be seen from the above, for the image captured by the mobile phone in the moon-viewing shooting mode, other objects in the image except the moon may not be imaged clearly or may not be imaged at all. However, the user can see the moon and other objects in this scenario, which means that the image captured by the electronic device in the moon-viewing shooting mode cannot restore the real scene, and the image effect is poor.

[0052] Therefore, to solve the above problems, the embodiments of the present application provide an image processing method, an electronic device, and a storage medium. In this method, during the startup process of the camera application, the electronic device can first display a first preview image on the preview interface of the camera application; when the zoom ratio of the camera application is adjusted from a first zoom ratio to a second zoom ratio, and the second zoom ratio is greater than or equal to the zoom ratio threshold, the electronic device can display a second preview image on the preview interface and display a third preview image in the first window of the preview interface; subsequently, after the shooting function of the camera application is triggered, a thumbnail of the captured image can be displayed in the second window of the preview interface. Among them, compared with the third preview image, the moon in the second preview image is clearer, and other objects except the moon in the third preview image are clearer. Compared with the captured image, the target object in the captured image can remain clear or be clearer, and other objects except the target object in the captured image are clearer, indicating that a captured image with both the moon and other objects being relatively clear can be obtained. In this way, the image processing method provided by the present application can obtain a relatively clear captured image, improve the image shooting effect of the electronic device, and restore the real scene seen by the user as much as possible.

[0053] To enable those skilled in the art to understand the solution of the present application more clearly, the following Figure 3a and Figure 3b First, the application scenario of the technical solution of the present application will be described.

[0054] Taking the electronic device as a mobile phone as an example, the image processing method provided by the embodiments of the present application will be described exemplarily. In this scenario, the mobile phone includes a camera application, and the user can use the shooting function of the camera application to take pictures. The mobile phone may include multiple cameras such as a main camera (which can be called the second camera), a telephoto camera (which can be called the first camera), and an ultra-wide-angle camera. The user may take pictures through any camera during the use of the mobile phone.

[0055] In a possible implementation manner, the user uses the telephoto camera of the mobile phone to take pictures. As Figure 3aAs shown, after the mobile phone starts the camera application, a camera preview interface 310 is displayed. The camera preview interface 310 includes a zoom control 311 and a shooting control 312. The zoom control 311 is used to adjust the zoom ratio of the camera, and the shooting control 312 is used to implement the shooting function of the camera application. The camera preview interface 310 also includes a preview area 313, which shows a preview image A obtained based on the telephoto camera.

[0056] In some embodiments, the user can slide the zoom control 311 to the right to change the zoom ratio from 2× to 10×. In response to the user's operation of adjusting the zoom ratio, the mobile phone displays a preview image B in the preview area 313. Compared with the preview image A, the viewing range of the preview image B becomes smaller.

[0057] In some embodiments, when the zoom ratio of the camera application is greater than or equal to a first magnification threshold, the mobile phone can display a picture-in-picture window area 314 in the camera preview interface 310 so that the user can master the viewing position of the telephoto camera and locate the shooting object faster and more accurately. Exemplarily, the first magnification threshold is 10x. Then the user can slide the zoom control 311 to the right to change the zoom ratio from 2× to 10×. In response to the user's operation of adjusting the zoom ratio, the mobile phone displays a picture-in-picture window area 314 in the camera preview interface 310 and shows a selection box marker 315 in this area. The image range shown by the selection box marker 315 is the same as the image range shown by the preview area.

[0058] In some embodiments, when the zoom ratio of the camera application is greater than or equal to a second magnification threshold (which can be referred to as the zoom ratio threshold), the mobile phone can detect whether the image captured by the telephoto camera contains the moon. When the mobile phone detects the moon, it enters the full moon shooting mode. Exemplarily, the second magnification threshold can be the same as the first magnification threshold, which is also 10x. Then it is determined that the zoom ratio is 10x, which is equal to the second magnification threshold. At this time, the mobile phone can perform moon detection. When the moon is detected, it enters the full moon shooting mode.

[0059] In some embodiments, it can also be that the user switches the shooting mode of the camera application to the full moon shooting mode. In response to the user's operation of switching to the full moon shooting mode, the mobile phone enters the full moon shooting mode.

[0060] In the full moon shooting mode, on the one hand, the mobile phone performs reduced exposure processing on the preview image obtained based on the telephoto camera to obtain a preview image C. On the other hand, the mobile phone starts the main camera and performs normal exposure processing on the preview image obtained based on the main camera to obtain a preview image D. Subsequently, the mobile phone displays the preview image C in the preview area 313 and displays the preview image D in the picture-in-picture window area 314.

[0061] As Figure 3aAs shown, in some embodiments, when the user needs to take a picture of the moon, the user can click on the shooting control 312 to trigger the shooting function of the camera application. In response to the user's click operation on the shooting control 312, the mobile phone processes the image captured by the telephoto camera based on the image parameters of the image captured by the main camera to obtain image E (which can also be referred to as the captured image). The camera preview interface 310 further includes a thumbnail display area 316, and the mobile phone can display the thumbnail F of image E in the thumbnail display area 316. After the user clicks on the thumbnail F of image E, the mobile phone responds to the user's click operation and displays an image viewing interface 320, which shows image E to the user. Compared with Figure 1 in (b) thereof, while the moon in image E is clear, other objects such as the fence net are also clearly imaged.

[0062] It should be noted that the above example of the mobile phone capturing images simultaneously through the telephoto camera and the main camera is only an example. The mobile phone can also capture images simultaneously through two other different cameras, and the present application does not limit this.

[0063] As Figure 3b shown, when the mobile phone does not detect the moon, it remains in the normal shooting mode. On the one hand, the mobile phone shows the preview image G obtained based on the telephoto camera in the preview area 313. On the other hand, the mobile phone shows the preview image H obtained based on the telephoto camera in the picture-in-picture window area 314 and shows a display selection box marker 315 in this area. The image range shown by the selection box marker 315 is the same as the image range shown in the preview area. Both the preview image G and the preview image H are images obtained by normal exposure processing.

[0064] It should be noted that the above example of the second magnification threshold being the same as the first magnification threshold is only an example. The second magnification threshold can also be different from the first magnification threshold.

[0065] Next, taking the electronic device as a mobile phone running an Android system with a layered architecture as an example, the software structure of the electronic device will be exemplarily described.

[0066] As Figure 4 shown, the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system from top to bottom is the application layer, the application framework layer, the hardware abstraction layer, the driver layer, and the hardware layer.

[0067] The application layer can include a series of application packages. In the embodiments of the present application, the application packages can include the application packages of applications such as the camera and the gallery.

[0068] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions. In the embodiments of the present application, the application framework layer may include a camera access interface, and the camera access interface may include camera management and camera devices. The camera access interface is used to provide application programming interfaces and programming frameworks for camera applications.

[0069] The hardware abstraction layer is an interface layer located between the application framework layer and the driver layer, and provides a virtual hardware platform for the operating system. In the embodiments of the present application, the hardware abstraction layer may include a multi-camera module, a camera algorithm library, and an image processing module.

[0070] In some embodiments, the multi-camera module may provide virtual hardware for camera device 1, camera device 2, or more camera devices. One camera device corresponds to one camera. In the embodiments of the present application, camera device 1 corresponds to a telephoto camera, and camera device 2 corresponds to a main camera.

[0071] The camera algorithm library may include the running code and data for implementing the image processing method provided in the embodiments of the present application. The image processing module is used to receive the raw image output by the sensor, and is also used to call the camera algorithm library to process the raw image to obtain a preview image, a captured image (which can be called a photographed image), etc. shown to the user. In the embodiments of the present application, the camera algorithm library may include multiple modules such as a SAT encapsulation module, an ROI encapsulation module, a smart AE encapsulation module, an AI encapsulation module, and an AE encapsulation module. The calling methods of each module can be referred to the introduction in the following embodiments.

[0072] The driver layer is a layer between hardware and software. The driver layer includes drivers for various hardware. The driver layer may include a camera device driver, etc. The camera device driver is used to drive the sensor of the camera (which can also be called an image sensor) to convert the image optical signal into an image electrical signal, and drive the image signal processor to preprocess the image electrical signal to obtain a raw image. In the embodiments of the present application, sensor 1 is the sensor of the telephoto camera, and sensor 2 is the sensor of the main camera.

[0073] Next, taking an electronic device as a mobile phone, the mobile phone includes multiple cameras such as a telephoto camera and a main camera, and the user uses the telephoto camera to take pictures as an example. Combining Figure 4 with the system structure shown, Figure 5a - Figure 5c and Figure 6 the image processing method provided in the embodiments of the present application will be introduced.

[0074] It should be noted that the above Figure 4The camera access interface in the application framework layer and the camera device driver in the driver layer included are described in the following embodiments in text form and are not shown in Figure 5a - Figure 5c It is shown in

[0075] It should be noted that the following embodiments are implemented on the basis that the camera application is successfully started and the preview area of its camera preview interface displays the preview image captured by the telephoto camera.

[0076] In some embodiments, in combination with Figure 4 As shown, in response to the user's operation of starting the camera application, such as clicking the icon of the camera application, or preset gesture instructions and voice instructions for starting the camera application, etc., the camera (also referred to as the camera application) calls the camera access interface corresponding to the camera application in the application framework layer; then, the camera access interface calls the camera device 1 in the multi-camera module of the hardware abstraction layer to send a start instruction for the telephoto camera to start the camera device driver; the camera device drives the telephoto camera and the image signal processor to obtain the original image. Subsequently, the image processing module of the hardware abstraction layer receives the original image and processes it to obtain the preview image. For example, the image processing module performs spatial alignment processing on the original image by calling the SAT encapsulation module of the camera algorithm library to obtain the preview image, and the camera application displays the preview image.

[0077] This image processing method may include the preview stage in the normal shooting mode, the preview stage in the full moon shooting mode, and the shooting stage in the full moon shooting mode.

[0078] As Figure 5a shown, the preview stage in the normal shooting mode includes the following steps:

[0079] S501: The camera application adjusts the first zoom ratio to the second zoom ratio.

[0080] In some embodiments, the zoom ratio of the camera application can be adjusted manually by the user. During the process of using the mobile phone to take pictures, the user may need to obtain preview images within different viewing ranges, so the first zoom ratio can be manually adjusted to the second zoom ratio.

[0081] Exemplarily, as Figure 3a shown, the user can trigger the zoom control 311 of the camera preview interface 310. For example, the user swipes the zoom control 311 to the right to adjust the zoom ratio from 2x to 10x, and the viewing range becomes smaller.

[0082] In some embodiments, the zoom ratio of the camera application can also be automatically adjusted from the first zoom ratio to the second zoom ratio by the camera application.

[0083] Exemplarily, in the case of switching the shooting mode, the camera application will automatically increase the zoom ratio. For example, in the normal shooting mode, when the user switches to the moon-viewing shooting mode, in order to obtain a clear image of the moon, the camera application can automatically adjust the first zoom ratio to the second zoom ratio. For example, it can be adjusted from 2x to 5x.

[0084] S502: The camera application determines that the second zoom ratio is greater than or equal to the first magnification threshold, and sends a display request for the pip window to the image processing module.

[0085] In the embodiments of the present application, the first magnification threshold is the magnification threshold for the camera application to display the pip window. If the zoom ratio of the camera application is greater than or equal to the first magnification threshold, the mobile phone can display the pip window in the camera preview interface.

[0086] It should be noted that the first magnification threshold can be set in advance according to requirements. For example, the first magnification threshold can be 10 times (or written as 10X), 15 times, 20 times, 30 times, etc. The present application does not make any limitations in this regard.

[0087] The second zoom ratio is the adjusted zoom ratio. Determining that it is greater than or equal to the first magnification threshold indicates that the pip window needs to be displayed in the camera preview interface of the camera application. Then, the camera application can send a display request for the pip window to the image processing module in the hardware abstraction layer through the camera access interface of the application framework layer.

[0088] S503: In response to the display request for the pip window, the image processing module processes the fourth raw image stream by calling the SAT encapsulation module in the camera algorithm library to generate the first telephoto stream and the first pip stream.

[0089] In some embodiments, the SAT encapsulation module is used to perform spatial alignment processing on images, such as cropping, magnifying, translating, rotating, etc.

[0090] It should be understood that after the camera application starts the telephoto camera, the telephoto camera continuously captures and generates the fourth raw image stream, and the fourth raw image stream includes a frame of the fourth raw image continuously captured by it. During the process of receiving the fourth raw image stream, the image processing module can call the camera algorithm library to perform spatial alignment processing on each frame of the fourth raw image to obtain each frame of the telephoto preview image, that is, the telephoto preview stream.

[0091] For example, the image processing module can call the SAT encapsulation module to perform cropping processing and magnifying processing on the fourth raw image stream to obtain the telephoto preview stream, and then the image processing module sends the telephoto preview stream to the preview area of the camera preview interface of the camera application. As Figure 3a shown, the preview image B is obtained through cropping processing and magnifying processing, and the viewing range becomes smaller.

[0092] In some embodiments, after the image processing module receives a request to display the picture-in-picture small window, the SAT encapsulation module is called, and the SAT encapsulation module copies the fourth original image stream to obtain the same fourth original image stream; then, the SAT encapsulation module performs spatial alignment processing on one fourth original image stream to obtain a telephoto preview stream, and at the same time, the SAT encapsulation module performs spatial alignment processing on another fourth original image stream to obtain a first picture-in-picture stream, which includes frame by frame of picture-in-picture small window images.

[0093] Taking a frame of image 1 included in the fourth original image stream as an example, the SAT encapsulation module first crops image 1 according to the second preset magnification to obtain image 2 that satisfies the field of view angle corresponding to the second preset magnification, and then enlarges it to obtain image 3 (that is, a telephoto image) that satisfies the size of the mobile phone camera preview interface, and uses it as the preview image of the preview area of ​​the camera preview interface.

[0094] For example, when the second zoom ratio is 10x, the SAT encapsulation module can take the center point of image 1 as the center, that is, keep the center points of image 1 and image 2 unchanged before and after cropping, determine the cropping frame corresponding to the second zoom ratio, and then determine the upper left vertex, height and width of image 1 based on the cropping frame to crop image 2, and then enlarge it to obtain image 3. The user can see the preview image after 5x zoom in the preview area of ​​the camera preview interface, which can be seen in Figure 3a Preview image B shown.

[0095] Similarly, taking the same frame of image 1 included in another fourth original image stream as an example, the SAT encapsulation module can first crop image 1 according to the first magnification threshold to obtain image 4 that meets the field of view angle corresponding to the first magnification threshold, and then reduce it to obtain image 5 (that is, picture-in-picture image) that meets the size of the picture-in-picture small window area; then determine the selection box corresponding to the second preset magnification, mark the selection box on image 5 with the image center point of image 5 as the center, and the image processing module sends the image 5 to the picture-in-picture small window area to display the selection box mark. The image content in the selection box mark (that is, the framing range) is the same as the content contained in the preview image in the preview area, which can be seen in FIG. Figure 3b Image G and image H are shown.

[0096] It should be noted that, in the normal shooting mode, both the first telephoto stream and the first picture-in-picture stream are subjected to normal exposure processing.

[0097] S504: The image processing module sends the first telephoto stream and the first picture-in-picture stream to the camera application.

[0098] The image processing module sends the first telephoto stream and the first picture-in-picture stream to the camera application through the camera application access interface.

[0099] S505: The camera application displays the first telephoto image in the preview area and displays the first picture-in-picture image in the picture-in-picture window area.

[0100] Based on the above example, the first telephoto stream includes frames of the first telephoto image (which can also be called the first preview image), and the first picture-in-picture stream includes frames of the picture-in-picture image (which can also be called the fourth preview image). The camera application displays frames of the first telephoto image in the preview area and displays frames of the first picture-in-picture image in the picture-in-picture window area (which can also be called the first window).

[0101] In some embodiments, as Figure 3b shown, both the preview area 313 and the picture-in-picture window area 314 display images obtained from the telephoto camera, and the images are processed with normal exposure, and the objects included in the images can be seen.

[0102] It should be noted that in the ordinary shooting mode, the telephoto camera continuously streams, that is, continuously sends a fourth raw image stream including frames of the fourth raw image to the image processing module, which can be referred to Figure 5a shown.

[0103] As Figure 5b shown, the preview stage of the full moon shooting mode includes the following steps:

[0104] S506: The camera application determines that the second zoom ratio is greater than or equal to the second ratio threshold and sends a moon recognition request to the image processing module.

[0105] In the embodiments of the present application, the second ratio threshold is the ratio threshold for the camera application to determine whether to automatically enter the full moon mode. If the zoom ratio of the camera application is greater than or equal to the second ratio threshold, the mobile phone can recognize whether there is a moon in the image.

[0106] In some embodiments, the second ratio threshold can be set in advance according to requirements. For example, the second ratio threshold can be 10 times (or written as 10X), 15 times, 20 times, or 30 times, etc.

[0107] In some embodiments, the first ratio threshold and the second ratio threshold can be the same. Exemplarily, both the first ratio threshold and the second ratio threshold are 10x, indicating that when the zoom ratio of the camera application is 10x, the camera preview interface displays the picture-in-picture window area on the basis of displaying the preview area, and the mobile phone performs moon recognition on the image to determine whether to enter the full moon shooting mode.

[0108] S507: The image processing module calls the AI encapsulation module of the camera algorithm library to perform object recognition on the tiny stream.

[0109] In the embodiments of the present application, the tiny stream is a low-resolution processed stream obtained by calling the camera algorithm library to downsample the fourth original image stream during the process of the image processing module receiving the fourth original image stream, and can be referred to as the tiny stream.

[0110] In some embodiments, the camera algorithm library module may include a downsampling algorithm. The image processing module calls the downsampling algorithm to downsample the fourth original image stream to obtain the tiny stream. Since the tiny stream includes images with low resolution, it can be processed quickly, improving the recognition speed of objects. It should be noted that the downsampling algorithm is only a general term for algorithms that can downsample images, and the specific algorithm is not limited in the embodiments of the present application.

[0111] Exemplarily, the size of the images in the fourth original image stream may be 3024*4032(p), then the size of the downsampled images in the tiny stream may be 378*502(p).

[0112] In some embodiments, the AI encapsulation module has an AI recognition function. Thus, the AI encapsulation module can identify each object included in the images in the tiny stream through the AI recognition function to detect whether there is a moon.

[0113] Exemplarily, the AI encapsulation module may be encapsulated with an AI recognition algorithm, which is used to recognize each object in the image, such as whether there are a moon, a sun, plants, human figures, etc. in the image, and then determine the shooting scene.

[0114] In addition, in some embodiments, the camera algorithm library may also be provided with a neural network model, which can perform object recognition. Thus, the image processing module can identify each object in the images in the tiny stream by calling the neural network model to determine whether there is a moon. The present application does not make specific restrictions on the way of moon recognition.

[0115] S508: The AI encapsulation module determines that there is a moon in the tiny stream, and sends the recognition result that there is a moon in the image to the multi-camera module and the camera application respectively.

[0116] In some embodiments, if the AI encapsulation module determines that there is a moon in the tiny stream, it indicates that there is a moon in the preview image seen by the user. The AI encapsulation module sends this result to the multi-camera module and sends this result to the camera application through the camera access interface. It should be noted that the present application does not limit the sending order of the results. It may be to send this result to the multi-camera module first and then to the camera application, or to send the results simultaneously. The present application does not make a limit on this.

[0117] In some embodiments, when the first magnification threshold is the same as the second magnification threshold, the mobile phone can display a pip window area on the basis of the display preview area while performing moon recognition on the image to determine whether to enter the full moon shooting mode. That is, after executing S501, S502 and S506 - S507 can be executed successively or simultaneously. If after executing S502 and before executing S503, the mobile phone determines that there is a moon in the image, then S503 - S505 can be skipped and S508 - S516 can be directly executed.

[0118] In addition, in some embodiments, when the first magnification threshold is less than the second magnification threshold and the zoom magnification of the adjusted camera application is greater than or equal to the first magnification threshold and less than the second magnification threshold, the mobile phone can execute S502 - S505. When the zoom magnification is subsequently adjusted to be greater than the second magnification threshold, the mobile phone continues to execute S506 - S516.

[0119] S509: The camera application enters the full moon shooting mode.

[0120] After receiving the result that there is a moon in the image, the camera application can automatically enter the full moon shooting mode.

[0121] In some embodiments, when the camera application enters the full moon shooting mode, the user can be prompted about the current shooting mode through a prompt message on the camera preview interface. The prompt message can be represented in at least one form of text, pattern, control, etc. Exemplarily, a moon pattern can be displayed on the camera preview interface to prompt the user that the camera application has entered the full moon shooting mode.

[0122] If there is a moon in the image, the mobile phone can automatically enter the full moon shooting mode; otherwise, the mobile phone remains in the normal shooting mode. Therefore, in some embodiments, the above S503 - S505 can also be executed after the AI encapsulation module determines that there is no moon in the tiny stream.

[0123] S510: The camera application displays the second telephoto image in the preview area.

[0124] In some embodiments, the exposure of the second telephoto image (which can also be called the second preview image) is less than that of the first telephoto image. That is, the clarity of the moon in the second telephoto image is greater than that in the first telephoto image, and the clarity of other objects except the moon in the second telephoto image is less than that of other objects in the first telephoto image. Refer to Figure 3a As shown, Image B is the first telephoto image and Image C is the second telephoto image.

[0125] It should be understood that when the telephoto camera captures the fourth original image, the brightness of the moon is relatively high, resulting in a blurred display of the moon area in the first telephoto image finally presented to the user. Therefore, in order to obtain a clear moon, it is necessary to perform a reduced exposure process on multiple frames of the fourth original image included in the fourth original image stream.

[0126] As Figure 3a described, the image C is the second telephoto image, which is an image obtained by performing a reduced exposure process. The moon is clearer, but other objects in the image are not imaged.

[0127] In some embodiments, the method for obtaining the second telephoto image may include the following steps:

[0128] Step 1: The image processing module calls the ROI encapsulation module to select the moon area of the first telephoto image as the ROI area.

[0129] In the embodiments of the present application, the ROI encapsulation module is used to outline the area to be processed, that is, the region of interest (which can be referred to as the ROI area) in the image in the form of a square, circle, ellipse, irregular polygon, etc.

[0130] It should be understood that in the full moon shooting mode, the camera application needs to present a clear moon to the user. Therefore, the moon area needs to be used as the ROI area.

[0131] In some embodiments, the camera algorithm library stores some of the first telephoto images included in the first telephoto stream. The image processing module can select one frame from them and then call the ROI encapsulation module to process it.

[0132] Step 2: The image processing module calls the AE encapsulation module to obtain AE statistical data based on the ROI area.

[0133] In the embodiments of the present application, the AE encapsulation module is used to statistically obtain AE statistical data for the ROI area in the image and determine the exposure parameters based on the AE statistical data to achieve automatic exposure. The AE statistical data may include the brightness of the pixels in the ROI area, the exposure time, the ISO sensitivity value, etc. The present application does not limit this.

[0134] In some embodiments, the AE encapsulation module includes an AE statistical module, and the AE statistical module statistically obtains AE statistical data for the ROI area.

[0135] Step 3: The image processing module calls the AE encapsulation module to obtain the first exposure parameter based on the AE statistical data.

[0136] The parameters in the first exposure parameter may include the exposure time, the sensitivity, the aperture coefficient, etc. The present application does not limit this.

[0137] In some embodiments, the exposure time can be adjusted by controlling the shutter speed. The faster the shutter speed, the shorter the exposure time and the less the exposure amount; on the contrary, the slower the shutter speed, the longer the exposure time and the more the exposure amount, and the image brightness increases.

[0138] The larger the aperture (the smaller the value), such as F2.8, the more the exposure amount and the increase in image brightness. The smaller the aperture (the larger the value), such as F16, the less the exposure amount and the decrease in image brightness.

[0139] The sensitivity is used to measure the sensitivity of the camera's photosensitive component to light. The higher the sensitivity, the stronger the ability to analyze light, the more light is sensed, and the image brightness increases; on the contrary, the lower the sensitivity, the weaker the ability to analyze light, the less light is sensed, and the image brightness decreases. In some embodiments, the sensitivity is commonly represented by the ISO sensitivity value, and the ISO sensitivity value can be divided into several levels, such as 50, 100, 200, 400, 800, 1600, 3200,.... The higher the ISO sensitivity value, the stronger the photosensitive ability of the photosensitive component.

[0140] Exemplarily, compared with the AE statistical data of the first telephoto image, the first exposure parameter can have a lower exposure time, a lower sensitivity, and / or a lower aperture coefficient, etc., that is, reducing the exposure parameter to obtain an image with a low exposure amount and improving the clarity of the moon.

[0141] In some embodiments, the AE encapsulation module may include an AE algorithm module, and the AE algorithm module calculates the first exposure parameter based on the AE statistical data. Exemplarily, the AE algorithm module can encapsulate AE algorithms such as the mean method AE, the mean method AE improved by the histogram, and the N - segment statistical method.

[0142] In some embodiments, the AE encapsulation module can query the pre - stored exposure table and determine the corresponding exposure parameter as the first exposure parameter based on the AE statistical data.

[0143] Step 4: The image processing module sends the first exposure parameter to the telephoto camera.

[0144] The image processing module sends the first exposure parameter to the telephoto camera through the camera device driver.

[0145] Step 5: The telephoto camera acquires the first raw image stream according to the first exposure parameter.

[0146] The sensor of the telephoto camera acquires the image optical signal according to the first exposure parameter and converts it into an image electrical signal. The sensor then transmits the image electrical signal to the image signal processor for pre - processing to obtain the first raw image stream including multiple frames of the first raw images.

[0147] Step 6: The telephoto camera sends the first raw image stream to the image processing module.

[0148] The telephoto camera sends the first raw image stream to the image processing module through the camera device driver.

[0149] Step 7: The image processing module calls the camera algorithm library to process the first raw image stream to obtain the second telephoto stream.

[0150] In some embodiments, the image processing module calls the SAT encapsulation module to perform spatial alignment processing on the first raw image stream to obtain the second telephoto stream.

[0151] Step 8: The image processing module sends the second telephoto stream to the camera application.

[0152] The image processing module sends the second telephoto stream to the camera application through the camera application access interface.

[0153] It should be noted that the implementation manners of steps 7 - 8 can refer to the processing process of the fourth raw image stream in S503 - S504, and details are not described herein again in this application.

[0154] S511: The multi - camera module sends a startup request to the main camera.

[0155] After the multi - camera module receives the result that there is a moon in the image, it sends the startup request to the camera device driver in the driver layer, and the camera device driver can start the main camera.

[0156] On the basis of starting the telephoto camera, another main camera is started. That is, in the embodiments of this application, during the process that the camera application is in the full - moon shooting mode, both the telephoto camera and the main camera of the mobile phone are in the startup state.

[0157] As Figure 6 shown, after recognizing the moon, the mobile phone starts both the telephoto stream and the main - camera stream simultaneously.

[0158] S512: The main camera captures the second raw image stream.

[0159] Based on the sensor and the image signal processor of the main camera, a second raw image stream including multiple frames of second raw images is obtained.

[0160] In some embodiments, the main camera captures the second raw image stream based on the second exposure parameter. The second exposure parameter is generated by the image processing module calling the ROI encapsulation module to select the entire area of the image as the ROI area and then calling the AE encapsulation module. That is, the image captured by the main camera equalizes the brightness of each object in the entire image, performs normal exposure processing on the image, and the image shown is normally converged.

[0161] S513: The main camera sends the second raw image stream to the image processing module.

[0162] S514: The image processing module calls the SAT encapsulation module to perform spatial alignment processing on the second raw image stream to generate a second picture-in-picture stream.

[0163] It should be understood that the installation positions of the main camera and the telephoto camera on the mobile phone are different, and the images captured will have displacement deviations. In order to enable the user to see two spatially aligned images and improve their usage experience, the SAT encapsulation module needs to perform spatial alignment processing on the second raw image stream collected by the main camera based on the first raw image stream collected by the telephoto camera.

[0164] In some embodiments, the displacement deviation between the moon in the first raw image stream and the moon in the second raw image stream can be calculated.

[0165] Exemplarily, one frame of the first raw image corresponds to a coordinate system (which can be called the first coordinate system), and one frame of the second raw image corresponds to another coordinate system (which can be called the second coordinate system). Both coordinate systems have the upper left vertex of the image as the origin of the coordinate system. The SAT encapsulation module first determines the coordinates (x1, y1) of the center point of the moon in the first raw image, then determines the coordinates (x2, y2) of the center point of the moon in the second raw image, and then calculates the coordinate deviation between the two coordinates. So that after the SAT encapsulation module respectively determines the cropping frames of the two images based on the first magnification threshold, first moves the cropping frame of the second raw image based on the coordinate deviation of the two coordinates, and then crops the second raw image based on the moved cropping frame to obtain an image that meets the field of view angle corresponding to the first magnification threshold, and then performs a shrinking process on it to obtain a picture-in-picture image that meets the size of the picture-in-picture small window area; subsequently, determine the selection box corresponding to the second preset magnification, and mark the selection box on the picture-in-picture image with the image center point of the picture-in-picture image as the center. The image processing module sends and displays the picture-in-picture image 2 in the picture-in-picture small window area with the selection box marked. See Figure 3a As shown, image D is the picture-in-picture image.

[0166] It should be noted that the origin of the first coordinate system can be the first vertex of the first raw image, and the origin of the second coordinate system can be the second vertex of the second raw image. The position of the first vertex in the first raw image is the same as the position of the second vertex in the second raw image. For example, both the first vertex and the second vertex are the upper left vertices.

[0167] Exemplarily, the coordinates 1 of the center point of the moon (which can be called the first coordinate point) in the first original image are (300, 400), and the coordinates 2 of the center point of the moon (which can be called the second coordinate point) in the second original image are (400, 300). The coordinate values of coordinate 2 can be subtracted from the coordinate values of coordinate 1. Then, the horizontal offset between the two is +100, and the vertical offset is -100. Subsequently, the cropping frame of the second original image can be offset 100 pixel points to the left and then 100 pixel points downwards.

[0168] It should be noted that the above first coordinate point and second coordinate point being the center points are only examples, and they can also be other coordinate points of the moon. The position of the second coordinate point on the moon only needs to be the same as the position of the first coordinate point on the moon.

[0169] In addition, in some embodiments, the displacement deviation between the first original image and the second original image can also be calculated by means of autofocus laser and calibrated distance calculation, and the present application does not limit this.

[0170] S515: The image processing module sends the second picture-in-picture stream to the camera application.

[0171] S516: The camera application displays the second picture-in-picture image in the picture-in-picture small window area.

[0172] In some embodiments, as Figure 3a shown, the exposure of the second telephoto image (i.e., preview image C) displayed in the preview area is relatively low, and the moon is clearer, but other objects cannot be imaged. The exposure of the second picture-in-picture image (i.e., preview image D, which can be called the third preview image) displayed in the picture-in-picture small window area is higher, and other objects can be imaged, but the clarity of the moon is poor.

[0173] As Figure 6 shown, in the full moon shooting mode, on the one hand, the telephoto stream undergoes reduced exposure processing to obtain a clearer moon and is displayed in the camera preview (i.e., the preview area of the camera preview interface); on the other hand, the mobile phone no longer sends the telephoto stream (which can be equivalent to the first picture-in-picture stream) to the picture-in-picture small window, but instead activates the main camera to obtain the main camera stream (which can also be called the second picture-in-picture stream) and performs normal exposure processing on it, and the obtained image is displayed in the picture-in-picture small window (i.e., the picture-in-picture small window area of the camera preview interface).

[0174] In some embodiments, as Figure 5bAs shown, in the full moon shooting mode, the telephoto camera generates the first original image stream, and the main camera generates the second original image stream. The mobile phone processes the first and second original image streams separately without interference. After the image processing module calls the AI encapsulation module to detect the moon, the mobile phone executes steps 1 - 8 and S510. After the multi-camera module receives the recognition result that the image contains the moon, the mobile phone executes S511 - S516.

[0175] As Figure 5c shown, the shooting stage of the full moon shooting mode includes the following steps:

[0176] S517: The camera application receives the user's trigger operation on the shooting function.

[0177] After the user determines that the camera of the mobile phone is aligned with the object to be photographed through the preview image displayed on the mobile phone screen, the user can trigger the shooting function of the camera application.

[0178] In some embodiments, as Figure 3a shown, the trigger operation of the shooting function can be the user's click operation on the shooting control 312.

[0179] In some embodiments, the trigger operation of the photographing function can be the user's voice command and gesture command for the camera application, etc., as long as it can trigger the camera application to take a photo. The present application does not make any limitations in this regard.

[0180] S518: The camera application sends a shooting request to the image processing module.

[0181] The camera application sends a shooting request to the image processing module through the camera access interface.

[0182] S519: The image processing module calls the smartAE encapsulation module in the camera algorithm library to calculate the first exposure sequence.

[0183] In some embodiments, during the preview stage of the full moon shooting mode, the image processing module can call the camera algorithm library to cache multiple frames of the second picture-in-picture images included in the second picture-in-picture image stream (i.e., the images obtained by shooting with the main camera), and cache the AE statistical data of each frame of the second picture-in-picture images. Similarly, in some embodiments, during the preview stage of the full moon shooting mode, the image processing module can also call the camera algorithm library to cache multiple frames of the second telephoto images included in the second telephoto image stream.

[0184] Exemplarily, the image processing module may cache multiple frames of the second picture-in-picture images and the AE statistical data of each frame of the second picture-in-picture images into a cache queue. The cache queue may be a ZSL queue or the like. The second picture-in-picture images may be sequentially stored into the ZSL queue according to the frame output order. When all the storage positions in the ZSL queue are occupied, the ZSL queue uses the newly acquired second picture-in-picture images to sequentially replace the historical second picture-in-picture images in the ZSL queue in real time. The caching process of the second telephoto image is the same, and will not be elaborated in this application.

[0185] In some embodiments, the smart AE encapsulation module may determine one frame of the second picture-in-picture image from multiple frames of the second picture-in-picture images in a cache queue, and use the AE statistical data of this frame of the second picture-in-picture image as the reference exposure parameter; and determine one frame of the second telephoto image as the reference frame image from multiple frames of the second telephoto images in another cache queue. Then, the smart AE encapsulation module determines the number of frames of the third original image to be captured by the telephoto camera, the exposure parameter of each frame of the third original image, and the frame output order based on the reference exposure parameter and the reference frame; subsequently, the exposure parameters of each frame of the third original image are arranged in the frame output order to generate the first exposure sequence.

[0186] The first exposure sequence includes multiple groups of exposure parameters (one frame of the third original image corresponds to one group of exposure parameters). Exemplarily, the multiple groups of exposure parameters may include the exposure parameters of the medium frame image and the long frame image. Another example is that the multiple groups of exposure parameters may include the exposure parameters of the medium frame image (which can be called the second group of exposure parameters), the exposure parameters of the long frame image (which can be called the first group of exposure parameters), and the exposure parameters of the short frame image (which can be called the third group of exposure parameters).

[0187] In some embodiments, the above reference frame image corresponds to the medium frame image.

[0188] As can be seen from the above, at least two groups of exposure parameters corresponding to the long frame image and the medium frame image have different values among the multiple groups of exposure parameters. For example, when comparing the two groups of exposure parameters, it may be that the values of the exposure time are different, the values of the sensitivity are different, or the values of each parameter are different. This application does not make any limitations in this regard.

[0189] Exemplarily, taking the exposure time in the exposure parameters as an example, the smart AE encapsulation module may calculate the exposure times corresponding to the long frame image and the short frame image required based on the exposure time of the reference frame image according to the exposure time of the reference exposure parameter, and determine the quantities corresponding to the required long frame image, medium frame image, and short frame image respectively.

[0190] For example, in the scene of shooting at the full moon, one long-frame image, five medium-frame images, and two short-frame images are required. The order and exposure time can be represented by the first exposure sequence, which can be expressed as {N, S, N, S, N, N, N, L, N, S = 5, N = 10, S = 5, N = 10, N = 10, N = 10, L = 50}, where N represents a medium-frame image, L represents a long-frame image, S represents a short-frame image, N = 10 means the exposure time of the medium-frame image is 10 ms, S = 5 means the exposure time of the short-frame image is 5 ms, and L = 50 means the exposure time of the long-frame image is 50 ms.

[0191] In some embodiments, the reference frame image can be the last second long-focus image in the cache queue when the smart AE encapsulation module receives the shooting instruction.

[0192] In some embodiments, considering the delay duration between when the user triggers the shooting function and when the image processing module receives the shooting instruction, the last second long-focus image in the cache queue is not the image that the user actually wants. Then, a second long-focus image with an earlier timestamp can be selected in the cache queue and used as the reference frame image. In this way, by selecting the second long-focus image output closer to the time when the user issues the shooting instruction as the reference frame image and performing subsequent processing based on this, an image that better meets the user's needs can be obtained, thereby enhancing the user experience.

[0193] In addition, in some embodiments, the image processing module can call the camera algorithm library to cache multiple second original images included in the second original image stream, or can also call the camera algorithm library to cache multiple first original images included in the first original image stream. Similarly, during subsequent processing, the reference exposure parameters can be determined based on the second original images collected by the main camera, and the reference frame image can be determined based on the first original images collected by the long-focus camera.

[0194] S520: The image processing module sends the first exposure sequence to the long-focus camera.

[0195] The image processing module sends the first exposure sequence to the sensor of the long-focus camera through the camera device driver.

[0196] S521: The long-focus camera sequentially acquires multiple third original images according to the first exposure sequence.

[0197] The sensor of the long-focus camera acquires images according to the frame output order of the first exposure sequence and the exposure parameters corresponding to each frame. The image signal processor then performs preprocessing and outputs multiple third original images in sequence. The number of frames of the multiple third original images is the same as the number of frames of the first exposure sequence, and the multiple third original images correspond one by one to the multiple groups of exposure parameters of the first exposure sequence.

[0198] S522: The telephoto camera sends multiple frames of third original images to the image processing module.

[0199] The telephoto camera sends multiple frames of third original images to the image processing module through the camera device driver.

[0200] S523: The image processing module processes the reference frame image based on multiple frames of third original images to generate a captured image.

[0201] In some embodiments, the camera algorithm library may include an exposure fusion algorithm. Based on the above example, if the multiple frames of third original images include long-frame images and medium-frame images, then with the reference frame image as a benchmark, the image processing module fuses the multiple frames of third original images into the reference frame image by invoking the exposure fusion algorithm, so that the dark areas in the reference frame image are brightened by long exposure, that is, the brightness of other objects in the image is increased, making the imaging of other objects clearer, for example, the foreground is clearer.

[0202] In some embodiments, based on the above example, if the multiple frames of third original images include long-frame images, medium-frame images, and short-frame images, then with the reference frame image as a benchmark, the multiple frames of third original images are fused into the reference frame image through the exposure fusion algorithm. While making the imaging of other objects clearer; the bright areas are restored by short exposure, that is, the brightness of the moon area in the reference frame image is further reduced, making the moon texture and edges clearer.

[0203] As an example, the image processing module may determine a mapping parameter according to the exposure parameters of the third original image, and this mapping parameter is used to adjust the exposure parameters of the reference frame image; adjust the exposure parameters of the reference frame image according to the mapping parameter; fuse the reference frame image with the adjusted reference frame image to obtain a captured image.

[0204] As Figure 6 shown, the main camera AE statistical data is the AE statistical data of the second picture-in-picture image captured by the main camera, the preview image is the reference frame image, and the mobile phone can perform multi-frame processing based on the main camera AE statistical data and the reference frame image to obtain the final captured image for display to the user.

[0205] S524: The image processing module sends the captured image to the camera application.

[0206] The image processing module sends the captured image to the camera application through the camera access interface.

[0207] S525: The camera application displays a thumbnail of the captured image.

[0208] The camera application shows the user a thumbnail of the captured image.

[0209] In some embodiments, as Figure 3aAs shown, after the user clicks the shooting control 312, the mobile phone displays the thumbnail of the captured image (i.e., thumbnail F) in the thumbnail display area 316 (which can also be referred to as the second window).

[0210] Next, the composition of the electronic device will be introduced.

[0211] It should be noted that the electronic device being a mobile phone in the above embodiments is only an example. In some embodiments, the electronic device can be a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), or other terminal devices. The specific form of the above electronic device is not particularly limited in this application, as long as it can implement the photographing function.

[0212] As Figure 7 As shown, the electronic device 700 may include a processor 710, an internal memory 720, a camera 730, and a display screen 740.

[0213] It can be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0214] The processor 710 may include one or more processing units. For example, the processor 710 may include an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP), a digital signal processor (DSP), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0215] The processor 710 may also be provided with a memory for storing instructions and data.

[0216] The internal memory 720 may be used to store computer-executable program code, and the executable program code includes instructions. The processor 710 executes various functional applications and data processing of the electronic device 700 by running the instructions stored in the internal memory 720.

[0217] In some embodiments, the internal memory 720 stores instructions for executing an image processing method. The processor 710 can implement the image processing method provided by the embodiments of the present application by executing the instructions stored in the internal memory 720.

[0218] The electronic device 700 implements a display function through an image processor, a display screen 740, an application processor, etc. The image processor is a microprocessor for image processing, connected to the display screen 740 and the application processor. The image processor is used to perform mathematical and geometric calculations for graphics rendering. The processor 710 may include one or more image processors, which execute program instructions to generate or change display information. The display screen 740 is used to display images, videos, etc.

[0219] In some embodiments, the display screen 740 is used to display the running interface of the camera application of the electronic device 700, such as a camera preview interface, an image viewing interface, etc.

[0220] The electronic device 700 can implement a shooting function through an ISP, a camera 730, a video codec, an image processor, a display screen 740, an application processor, etc.

[0221] In some embodiments, the shooting function includes a photographing function and a video recording function of the camera application of the electronic device 700.

[0222] The ISP is used to process the data fed back by the camera 730. For example, when taking a photo, the shutter is opened, and light passes through the lens and is transmitted to the camera photosensitive element. The optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene.

[0223] In some embodiments, the ISP can be provided in the camera 730. The camera 730 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element (i.e., the sensor). The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the image optical signal into an image electrical signal, and then transmits the image electrical signal to the ISP to convert it into a digital image signal (i.e., the original image). The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into a standard RGB, YUV, etc. format image signal.

[0224] In some embodiments, the electronic device 700 may include N cameras 730, where N is a positive integer greater than 1. Exemplarily, the electronic device 700 includes a telephoto camera and a main camera.

[0225] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a computer, it can implement one or more steps in any of the above image processing methods.

[0226] The computer-readable storage medium may be a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0227] Another embodiment of the present application also provides a computer program product containing instructions. When the computer program product is executed by a computer, it can implement one or more steps in any of the above image processing methods.

[0228] The electronic device, computer-readable storage medium, and computer program product provided in this embodiment are all used to execute the corresponding image processing method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding image processing method provided above, and will not be elaborated here.

[0229] The terms "first", "second", "third", etc. in the description, claims, and drawings of the present application are used to distinguish different objects, rather than to limit a specific order.

[0230] In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, using words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0231] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that, Including: In response to the camera application being in the startup state, displaying a first preview image on the preview interface of the camera application; In response to an operation of adjusting the first zoom ratio of the camera application to a second zoom ratio, displaying a second preview image on the preview interface of the camera application, and displaying a third preview image in a first window of the preview interface; the second zoom ratio is greater than or equal to a zoom ratio threshold; the clarity of the target object in the second preview image is greater than the clarity of the target object in the third preview image; The clarity of other objects except the target object in the second preview image is less than the clarity of other objects except the target object in the third preview image; In response to a trigger operation on the photographing function of the camera application, displaying a thumbnail of the photographed image in a second window of the preview interface; the clarity of the target object in the photographed image is greater than or equal to the clarity of the target object in the second preview image, and the clarity of other objects except the target object in the photographed image is greater than the clarity of other objects except the target object in the second preview image.

2. The method according to claim 1, characterized in that, The step of, in response to an operation of adjusting the first zoom ratio of the camera application to a second zoom ratio, displaying a second preview image on the preview interface of the camera application, and displaying a third preview image in a first window of the preview interface, includes: In response to the operation of adjusting the first zoom ratio to the second zoom ratio, determining that the second zoom ratio is greater than or equal to the zoom ratio threshold, and detecting the objects included in the first preview image; Determining that the first preview image includes the target object, obtaining a second preview image collected by a first camera, and a third preview image collected by a second camera; Displaying the second preview image on the preview interface, and displaying the third preview image in the first window.

3. The method according to claim 1, characterized in that The obtaining step of the second preview image includes: Collecting the second preview image based on a first exposure parameter, and the first exposure parameter is lower than the exposure parameter of the first preview image.

4. The method according to claim 1, characterized in that, The obtaining step of the third preview image includes: Collecting the third preview image based on a second exposure parameter, and the second exposure parameter is higher than the exposure parameter of the second preview image.

5. The method according to claim 2, wherein The first preview image is collected by the first camera; the step of determining that the first preview image includes the target object, obtaining a second preview image collected by the first camera, and a third preview image collected by the second camera, includes: Determining that the first preview image includes the target object, and starting the second camera; Obtaining a second preview image collected by the first camera, and a third preview image collected by the second camera.

6. The method according to claim 2, characterized in that, The first preview image is collected by the first camera; The step of determining that the first preview image includes the target object, obtaining a second preview image collected by the first camera, includes: Determining that the first preview image includes the target object, and reducing the exposure parameter of the first camera; Based on the reduced exposure parameters, the second preview image is acquired by the first camera.

7. The method according to claim 1, characterized in that, In response to the operation of zooming in the first zoom ratio of the camera application to the second zoom ratio, the second preview image is displayed on the preview interface of the camera application, and the third preview image is displayed in the first window of the preview interface, including: In response to the operation of zooming in the first zoom ratio to the second zoom ratio, the first window pops up on the preview interface to display the fourth preview image; the clarity of the first preview image is the same as that of the fourth preview image; It is determined that the first preview image contains the target object, the first preview image is updated to the second preview image on the preview interface, and the fourth preview image is updated to the third preview image in the first window.

8. The method according to any one of claims 1-7, characterized in that, The steps for obtaining the captured image include: The captured image is generated by performing fusion processing on the first original image and the second original image. The first original image and the second preview image are both acquired by the first camera; the second original image and the third preview image are both acquired by the second camera.

9. The method according to claim 8, characterized in that, The generating the captured image by performing fusion processing on the first original image and the second original image includes: Based on the first original image and the second original image, a first exposure sequence is determined; the first exposure sequence includes multiple sets of exposure parameters, and the multiple sets of exposure parameters at least include two sets of exposure parameters with different values; According to the first exposure sequence, multiple frames of third original images are acquired by the first camera; the multiple frames of third original images correspond to the multiple sets of exposure parameters one by one; The captured image is generated by performing fusion processing on the multiple frames of third original images and the first original image.

10. The method according to claim 9, wherein The first exposure sequence includes a first set of exposure parameters and a second set of exposure parameters; the value of the first set of exposure parameters is greater than the value of the second set of exposure parameters.

11. The method according to claim 9, wherein The first exposure sequence includes a first set of exposure parameters, a second set of exposure parameters, and a third set of exposure parameters; the value of the first set of exposure parameters is greater than the value of the second set of exposure parameters; the value of the second set of exposure parameters is greater than the value of the third set of exposure parameters.

12. The method according to claim 11, wherein The value of the second set of exposure parameters is obtained based on the first original image, and the values of the first set of exposure parameters and the third set of exposure parameters are obtained based on the first original image and the second original image.

13. The method according to claim 8, wherein The steps for obtaining the third preview image include: Based on the first original image, spatial alignment processing is performed on the second original image to obtain the third preview image.

14. The method according to claim 13, characterized in that, The first vertex of the first original image is the origin of the first coordinate system; the second vertex of the second original image is the origin of the second coordinate system; the position of the first vertex in the first original image is the same as the position of the second vertex in the second original image; The performing spatial alignment processing on the second original image based on the first original image to obtain the third preview image includes: Determine the first coordinate point corresponding to the target object included in the first original image in the first coordinate system; Determine the second coordinate points corresponding to the target object included in the second original image in the second coordinate system; the position of the second coordinate point on the target object is the same as the position of the first coordinate point on the target object; Calculate the coordinate deviation value between the first coordinate point and the second coordinate point; Based on the coordinate deviation value, perform spatial alignment processing on the second original image to obtain the third preview image.

15. The method according to any one of claims 1 to 14, characterized in that The target object includes the moon; the clarity of the moon in the second preview image is greater than the clarity of the moon in the third preview image; the clarity of other objects except the moon in the second preview image is less than the clarity of other objects except the moon in the third preview image.

16. An electronic device, characterized in that, Comprising a memory and a processor; The memory is coupled to the processor, and the memory is used to store computer program code, the computer program code includes computer instructions, and one or more of the processors call the computer instructions to cause the electronic device to execute the image processing method according to any one of claims 1-15.

17. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the image processing method according to any one of claims 1-15 is implemented.

Images