Image processing method, electronic device, chip system and storage medium

Abstract

The present application relates to the field of image processing, and particularly relates to an image processing method, an electronic device, a chip system and a storage medium. The method comprises: after first processing, displaying to a user a first image that has been subjected to the first processing; during second processing, displaying to the user the first image that has been subjected to the first processing; after second processing, displaying to the user a second image that has been subjected to the second processing; and simultaneously displaying the first image and the second image for comparison by the user. By means of the method, a corresponding interface can be displayed to a user at each stage of an image enhancement processing process, and the changes in the interface indicates the stage of image processing to the user, which is conducive to improving the user experience.

Description

Image processing method, electronic device, chip system and storage medium TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to an image processing method, an electronic device, a chip system and a storage medium. BACKGROUND

[0002] With the increasing demand of users for image quality, the processing of images by electronic devices has become more complex, and accordingly, the process of image processing performed by an electronic device takes a long time. In this case, the user may need to wait for a period of time before being able to view the high-quality image after image enhancement processing. However, during this process, the electronic device usually displays an interface, which brings a "stuttering" feeling to the user and affects the user experience.

[0003] SUMMARY

[0004] The present application provides an image processing method, an electronic device, a chip system and a storage medium, which helps to improve the user experience during image enhancement processing.

[0005] To achieve the above object, the present application adopts the following technical solutions:

[0006] In a first aspect, an image processing method is provided, applied to an electronic device, and the method comprises:

[0007] displaying a first interface; the first interface is a shooting interface of a camera application, and the first interface comprises a first control, which is used to instruct photographing;

[0008] receiving a first operation of a user on the first control;

[0009] in response to the first operation, displaying a second interface; wherein the second interface comprises a thumbnail of a first image, the first image being obtained by performing first processing on a first original image, and the first original image being obtained by the camera based on capturing in response to the first operation;

[0010] receiving a second operation of a user on the thumbnail of the first image;

[0011] in response to the second operation, displaying a third interface at a first time; wherein the third interface is an interface of a gallery application, and the third interface comprises the first image;

[0012] At a second moment, a fourth interface is displayed; the fourth interface is an interface of a gallery application, the fourth interface includes a second image and a second control, the second image is obtained by performing a second processing on the first original image, an algorithm complexity of the first processing is lower than an algorithm complexity of the second processing, an image quality of the second image is higher than an image quality of the first image, and the second moment is after the first moment;

[0013] A third operation of the user on the second control is received;

[0014] In response to the third operation, a fifth interface is displayed; the fifth interface includes the first image and the second image.

[0015] For example, in the example of FIG. 7, the original image RAW2 is the first original image, the image after the first-stage processing JPEG21 is the first image, the image after the second-stage processing JPEG22 is the second image, and the image data after the first-stage processing Packet2 is the first image data. The operation of the user clicking the photograph control 301 shown in (b) of FIG. 7 can be recorded as a first operation. The interface shown in (b) of FIG. 7 can be recorded as a first interface, the interface shown in (c) of FIG. 7 can be recorded as a second interface, the interface shown in (d) of FIG. 7 can be recorded as a third interface, and the interface shown in (e) of FIG. 7 can be recorded as a fourth interface. The operation of the user clicking the thumbnail in the first display box 302 in (c) of FIG. 7 can be recorded as a second operation. The moment t76 can be recorded as a first moment, and the moment t77 can be recorded as a second moment.

[0016] For example, in the example of FIG. 11, the interface shown in (a) of FIG. 11 can be recorded as a first interface, the photograph control 301 on the first interface can be recorded as a first control, and the operation of the user clicking the photograph control 301 on the first interface can be recorded as a first operation. The interface shown in (b) of FIG. 11 can be recorded as a second interface. The thumbnail in the display box 302 on the second interface can be recorded as a thumbnail of a first image. The operation of the user clicking the thumbnail in the display box 302 can be recorded as a second operation. The interface shown in (c) of FIG. 11 can be recorded as a third interface, and the interface shown in (d) of FIG. 11 can be recorded as a fourth interface. The control 1002 on the fourth interface can be recorded as a second control. The operation of the user clicking the second control can be recorded as a third operation. The interface shown in (e) of FIG. 11 can be recorded as a fifth interface.

[0017] In the embodiments of the present application, after the first processing, the first image after the first processing is displayed to the user; during the second processing, the first image after the first processing is displayed to the user; after the second processing, the second image after the second processing is displayed to the user; and the first image and the second image can be displayed simultaneously. Through the above-mentioned manner, the corresponding interface can be displayed to the user at each stage in the image enhancement processing process, so as to prompt the user about the stage of image processing through the change of the interface, thereby improving the user experience.

[0018] In addition, the electronic device can process the original image, thereby improving the image quality in the photo, presenting the user with a high-quality portrait photo with clear, realistic and delicate picture quality, and improving the user's visual experience. Moreover, considering the possible computing power, power consumption, storage and other pressures in the image processing process, in the embodiments of the present application, the electronic device processes the image in stages, performs image processing with lower algorithm complexity on the original image after the photographing is completed, and performs image processing with higher algorithm complexity after the camera application is exited, thereby greatly reducing the time consumption of one shooting, enabling the user to timely capture and continuously shoot, and reducing the power consumption and storage pressure of the electronic device during the user's continuous shooting of images.

[0019] In an implementation form of the first aspect, the displaying the first interface comprises:

[0020] In a case where the preset function is turned on, the first interface is displayed, and the first interface further includes a first icon corresponding to the preset function, the first icon is in a first state, and the first state indicates that the preset function is turned on.

[0021] The preset function refers to a function for image enhancement processing. In different shooting scenes, the image enhancement processing function indicated by the preset function can be different, and the corresponding image processing flow and the involved image processing algorithm can also be different. For example, in a portrait shooting scene, the preset function can refer to a super portrait function. In a long-focus shooting scene, the preset function can refer to a long-focus shooting function.

[0022] In one example, the two-stage image processing process is introduced by taking the first stage using an AIGC algorithm and the second stage using a GAN model as an example.

[0023] In the portrait shooting scene, in the first stage, the original image RAW is sequentially subjected to denoising fusion processing, light and shadow enhancement processing and skin quality optimization processing to obtain a first-stage-processed YUV image, and the first-stage-processed YUV image is converted into a JPEG image (one-segment image) and sent to display. In parallel, the original image RAW is subjected to image segmentation processing to obtain a segmented image of a face. The DDR takes the result after the light and shadow enhancement processing and the segmented image of the face as intermediate results in the first-stage processing, and takes the original image RAW and the first-stage-processed YUV image as image data Packet and stores them in the UFS. In the second stage, the DDR obtains the image data Packet from the UFS, and performs second-stage image processing according to the result after the light and shadow enhancement processing. As shown in (a) of FIG. 15, the AIGC algorithm and the skin quality optimization processing are sequentially performed according to the result after the light and shadow enhancement processing and the segmented image of the face to obtain a second-stage-processed YUV image, and the second-stage-processed YUV image is converted into a JPEG image (two-segment image) and sent to display.

[0024] In the long-focus shooting scene, in the first stage, the original image RAW is sequentially subjected to denoising fusion processing, light and shadow enhancement processing and TGB domain detail processing to obtain a first-stage-processed YUV image, and the first-stage-processed YUV image is converted into a JPEG image (one-segment image) and sent to display. In parallel, the original image RAW is subjected to image segmentation processing to obtain a segmented image of a target object. The DDR takes the result after the light and shadow enhancement processing and the segmented image as intermediate results in the first-stage processing, and takes the original image RAW and the first-stage-processed YUV image as image data Packet and stores them in the UFS.

[0025] In the second stage, the DDR obtains the image data Packet from the UFS, and performs second-stage image processing according to the result after the light and shadow enhancement processing. As shown in (a) of FIG. 19, the AIGC algorithm processing is sequentially performed according to the result after the light and shadow enhancement processing and the segmented image to obtain a second-stage-processed YUV image, and the second-stage-processed YUV image is converted into a JPEG image (two-segment image) and sent to display.

[0026] In the embodiments of the present application, in the case where the preset function is turned on, the image processing in stages is supported. By turning on / off the preset function, the image processing can be flexibly performed, which is beneficial to improving the user experience.

[0027] In an implementation form of the first aspect, after the super-clear portrait is turned on, the electronic device selects a corresponding second-stage image processing algorithm according to the current shooting parameter.

[0028] In an implementation, the process of selecting the image processing algorithm of the second stage can include: if the current shooting parameter meets a first condition, using a first algorithm in the second stage; if the shooting parameter meets a second condition, using a second algorithm in the second stage; and if the shooting parameter meets a third condition, using a third algorithm in the second stage.

[0029] The first algorithm, the second algorithm and the third algorithm are different image processing algorithms in the second stage, the complexity of the first algorithm is higher than that of the second algorithm, the complexity of the second algorithm is higher than that of the third algorithm, and the complexity of the third algorithm is higher than that of the image processing algorithm in the first stage.

[0030] In a case, the algorithm complexity of the third algorithm has less impact on memory, and in this case, the second stage can not trigger dumping when executing the third algorithm.

[0031] In an implementation of the first aspect, after the long-focus image function is started, the electronic device selects a corresponding image processing algorithm of the second stage according to a current shooting scene.

[0032] It can be understood that different image processing algorithms can be set in the second stage for different shooting scenes. For example, for a portrait shooting scene, the image processing algorithm of the second stage can include AIGC, a GAN model and a skin beautifying algorithm. For a long-focus shooting scene, the image processing algorithm of the second stage can include an AIGC algorithm and a GAN model. Moreover, the parameters of the AIGC algorithm in the portrait shooting scene and the AIGC algorithm in the long-focus shooting scene are different, and the parameters of the GAN model obtained in the portrait shooting scene and the GAN model obtained in the long-focus shooting scene are also different.

[0033] In an implementation, in the long-focus shooting scene, the process of selecting the image processing algorithm of the second stage can include: if the current shooting parameter meets a fifth condition, using a fourth algorithm in the second stage; and if the shooting parameter meets a sixth condition, using a fifth algorithm in the second stage. The algorithm complexity of the fourth algorithm is higher than that of the fifth algorithm, and the algorithm complexity of the fifth algorithm is higher than that of the image processing algorithm in the first stage.

[0034] In a case, the algorithm complexity of the fifth algorithm has less impact on memory, and in this case, the second stage can not trigger dumping when executing the fifth algorithm.

[0035] In an implementation of the first aspect, the method further includes:

[0036] In response to the first operation, performing the first processing on the first original image to obtain the first image;

[0037] displaying the second interface;

[0038] save the first image and image data corresponding to the first image from the first memory to a second memory;

[0039] in response to the second operation, obtain the first image and image data corresponding to the first image from the second memory;

[0040] display the third interface at the first time;

[0041] perform the second processing according to the image data corresponding to the first image to obtain the second image;

[0042] display the fourth interface at the second time.

[0043] For example, after the first processing is completed, the DDR stores the first-processed data in the UFS; when the second processing is needed, the DDR obtains the first-processed data from the UFS, so that the CPU performs the second-stage image processing in the background.

[0044] Alternatively, the first-processed data can be stored in the memory (such as the DDR). However, as the number of captured images increases and / or the image quality of the captured images improves, this storage manner will occupy a large amount of space of the DDR. Generally, the DDR has a high efficiency in processing images, but if a large amount of space of the DDR is occupied, the processing of other images by the DDR will be affected, thereby affecting the timely snapshot or continuous shooting of the user. In addition, the power consumption of the DDR is relatively high, and if a large amount of space of the DDR is occupied, the power consumption can be increased.

[0045] In the above manner, the first-processed data can be stored in the second memory. For example, after the first processing is completed, the DDR stores the first-processed data in the UFS; when the second processing is needed, the DDR obtains the first-processed data from the UFS, so that the CPU performs the second processing in the background. In this way, the space occupation of the first memory can be effectively reduced, thereby meeting the needs of the user for continuous shooting and timely snapshot. In addition, reducing the space occupation of the first memory releases the storage space of the first memory, which is helpful to reduce the power consumption.

[0046] In an implementation form of the first aspect, after the second interface is displayed, the method further includes:

[0047] in response to a fourth operation of the user exiting the camera application on the second interface, a sixth interface is displayed; the sixth interface is an interface of a desktop application, and the sixth interface includes a second icon of a gallery application;

[0048] a fifth operation of the user on the second icon is received;

[0049] In response to the fifth operation, a seventh interface is displayed; wherein the seventh interface is an interface of a gallery application, and the sixth interface includes a thumbnail of the first image.

[0050] A sixth operation of a user on the thumbnail of the first image is received;

[0051] In response to the sixth operation, at a third time, the third interface is displayed;

[0052] At a fourth time, the fourth interface is displayed; wherein the fourth time is after the third time.

[0053] For example, in the example of FIG. 8, the original image RAW1 is a first original image, the first-stage processed image JPEG11 is a first image, the second-stage processed image JPEG12 is a second image, and the first-stage processed image data Packet1 is first image data. The interface shown in (d) of FIG. 8 can be referred to as a sixth interface, the icon 601 in the interface shown in (d) of FIG. 8 can be referred to as a second icon, and the operation of the user clicking the second icon can be referred to as a fifth operation. The interface shown in (e) of FIG. 8 can be referred to as a seventh interface, the thumbnail 602 can be referred to as a thumbnail of the first image, and the operation of the user clicking the thumbnail 602 can be referred to as a sixth operation. The interface shown in (f) of FIG. 8 can be referred to as a third interface, and the interface shown in (g) of FIG. 8 can be referred to as a fourth interface. The time t87 can be referred to as a third time, and the time t88 can be referred to as a fourth time.

[0054] In the embodiments of the present application, when the electronic device exits the camera application and enters the gallery application, the second processing of the image is performed, that is, the image processing with high algorithm complexity is performed after the camera application is exited, the time consumption of one shooting is greatly reduced, the user can timely capture and continuously shoot, and the power consumption and storage pressure of the electronic device are reduced in the process of the user continuously shooting images.

[0055] In a manner of implementation of the first aspect, the method further includes:

[0056] In response to the first operation, the first processing is performed on the first original image to obtain the first image, and the second interface is displayed;

[0057] The first image and image data corresponding to the first image are stored from a first storage to a second storage;

[0058] In response to the sixth operation, the first image and the image data corresponding to the first image are acquired from the second storage;

[0059] At the third time, the third interface is displayed;

[0060] performing the second processing on image data corresponding to the first image to obtain the second image;

[0061] displaying the fourth interface at the fourth time.

[0062] In the above manner, the first processed data can be stored in the second memory. In this way, the space occupation of the first memory can be effectively reduced, thereby meeting the user's demand for continuous shooting and timely snapshot. In addition, reducing the space occupation of the first memory releases the storage space of the first memory, which helps to reduce power consumption.

[0063] In an implementation form of the first aspect, after displaying the second interface, the method further includes:

[0064] in response to a fourth operation of the user exiting the camera application on the second interface, displaying a sixth interface; wherein the sixth interface is an interface of a desktop application, and the sixth interface includes a second icon representing a gallery application;

[0065] receiving a fifth operation of the user on the second icon;

[0066] in response to the fifth operation, displaying a seventh interface; wherein the seventh interface is an interface of the gallery application, and the sixth interface includes a thumbnail of the first image;

[0067] receiving a sixth operation of the user on the thumbnail of the first image;

[0068] in response to the sixth operation, displaying the fourth interface.

[0069] For example, in the example of FIG. 9, the original image RAW1 is the first original image, the first stage processed image JPEG11 is the first image, the second stage processed image JPEG12 is the second image, and the first stage processed image data Packet1 is the first image data. The interface shown in (b) of FIG. 9 is the first interface, and the interface shown in (c) of FIG. 9 is the second interface. The operation of the user clicking the display box 301 in (c) of FIG. 9 is the fourth operation. The interface shown in (d) of FIG. 9 is the sixth interface, the icon 601 in the sixth interface can be referred to as the second icon, and the operation of the user clicking the second icon can be referred to as the fifth operation. The interface shown in (e) of FIG. 9 is the seventh interface, the thumbnail 801 in the seventh interface can be referred to as the thumbnail of the first image, and the clicking operation of the user on the thumbnail 801 can be referred to as the sixth operation. The interface shown in (f) of FIG. 9 is the fourth interface.

[0070] In the embodiments of the present application, the electronic device sequentially performs the second-stage image processing of each image in the background after exiting the camera application. The second-stage image processing flow with high algorithm complexity runs in the background, and compared with running in the foreground, running in the background has lower demand for memory and can reduce memory occupancy, thereby improving the performance of the electronic device. Moreover, when the user views a certain image in the gallery application, if the electronic device has completed the second-stage image processing task of the image in the background, the electronic device directly displays the image processed in the second stage, which reduces the transition time between displaying the image processed in the first stage and displaying the image processed in the second stage, thereby improving the user experience.

[0071] In an implementation form of the first aspect, the method further includes:

[0072] in response to the first operation, performing the first processing on the first original image to obtain the first image, and displaying the second interface;

[0073] storing the first image and first image data corresponding to the first processing from a first memory to a second memory;

[0074] in response to the fourth operation, obtaining the first image data from the second memory;

[0075] performing the second processing according to the first image data to obtain the second image;

[0076] in response to the sixth operation, displaying the fourth interface.

[0077] In the above manner, the data processed in the first stage can be stored in the second memory. In this way, the space occupancy of the first memory can be effectively reduced, thereby meeting the user's demand for continuous shooting and timely snapshot. In addition, reducing the space occupancy of the first memory and releasing the storage space of the first memory help to reduce power consumption.

[0078] In an implementation form of the first aspect, the method further includes:

[0079] displaying an eighth interface, the eighth interface being a shooting interface of the camera application;

[0080] if the current shooting scene is a first preset scene, displaying a ninth interface, the ninth interface being a shooting interface of the camera application, the ninth interface including the first icon, the first icon being in a second state, the second state indicating that the preset function is turned off;

[0081] receiving a seventh operation of the user on the first icon;

[0082] in response to the seventh operation, turning on the preset function;

[0083] display the first interface.

[0084] Optionally, the first preset scene is a portrait shooting mode.

[0085] In an implementation form of the first aspect, the method further includes:

[0086] when the first interface is displayed, receiving an eighth operation of the first icon by the user;

[0087] in response to the eighth operation, closing the preset function;

[0088] displaying a tenth interface; the tenth interface is a shooting interface of the camera application, and the tenth interface includes the first icon, and the first icon is in the second state.

[0089] For example, in the example of FIG. 10, the icon 902 is the first icon. The preset function refers to a super portrait function. The interface shown in (a) of FIG. 10 can be recorded as an eighth interface, and the interfaces shown in (b) and (c) of FIG. 10 can be recorded as a ninth interface. The click operation of the first icon by the user can be recorded as a seventh operation. (d) of FIG. 10 can be recorded as the first interface.

[0090] Generally, the image enhancement algorithm can be applied in most portrait shooting scenes. In the embodiment of the present application, in the portrait shooting scene, the user can start the preset function (such as the super clear portrait function), thereby providing a flexible selection mode for the user.

[0091] In an implementation form of the first aspect, the method further includes:

[0092] displaying an eighth interface, the eighth interface being a shooting interface of the camera application;

[0093] if the current shooting scene is a second preset scene, starting the preset function;

[0094] displaying the first interface.

[0095] In an implementation form of the first aspect, the method further includes:

[0096] when the first interface is displayed, receiving an eighth operation of the first icon by the user;

[0097] in response to the eighth operation, closing the preset function;

[0098] displaying an eleventh interface; the eleventh interface is a shooting interface of the camera application, the eleventh interface includes the first icon, the first icon is in a second state, and the second state indicates that the preset function is in a closed state.

[0099] In an implementation form of the first aspect, the method further comprises:

[0100] When the first interface is displayed, if the current shooting scene jumps out of the second preset scene, a twelfth interface is displayed at a fifth time point after the second preset scene is jumped out; the twelfth interface is a shooting interface of a camera application, and the twelfth interface does not include the first icon.

[0101] In an implementation form of the first aspect, a magnification of the image shot in the second preset scene is greater than a preset magnification.

[0102] In an implementation form of the first aspect, the second preset scene is a long-focus shooting scene.

[0103] For example, in the example shown in FIG. 16, the icon 1601 is the first icon. The interface shown in (a) of FIG. 16 can be recorded as an eighth interface, and the interface shown in (b) of FIG. 16 can be recorded as the first interface. The operation of the user clicking the icon 1601 on the first interface can be recorded as an eighth operation. The interface shown in (c) of FIG. 16 can be recorded as an eleventh interface.

[0104] The interface shown in (a) of FIG. 17 can be recorded as the first interface, and the interface shown in (c) of FIG. 17 can be recorded as the twelfth interface.

[0105] For the long-focus shooting scene, whether the two-stage image enhancement can be performed depends on the magnification, brightness, and other image attribute parameters. In the embodiment of the present application, in the long-focus shooting scene, whether the long-focus image function is enabled can be detected by the electronic device, thereby improving the intelligent degree of the electronic device and effectively improving the effect of image enhancement. In addition, in this case, the long-focus image function can be closed by the user, thereby providing the user with a flexible selection mode.

[0106] In an implementation form of the first aspect, the method further comprises:

[0107] obtaining an image attribute parameter of the first original image, the image attribute parameter being related to a shooting scene when the first original image is shot;

[0108] determining a first processing model from a plurality of image processing models according to the image attribute parameter, the first processing model being used to improve image quality of the first original image;

[0109] performing image enhancement on the first original image according to the first processing model to obtain the second image.

[0110] For example, the shooting scene can include a first preset scene and a second preset scene. The first preset scene can represent a portrait shooting scene, and the second preset scene can represent a long-focus shooting scene.

[0111] In one example, in the portrait shooting scene, the image processing algorithm of the first stage is determined according to the image attribute parameter of the first original image. If the image processing algorithm of the first stage adopts a denoising fusion network, different processing modes each correspond to a first processing model according to the image attribute parameter.

[0112] The processing mode can include a double-shot mode and a single-shot mode. The single-shot mode means that the electronic device finally displays only the second segment image, but does not display the first segment image. The double-shot mode means that the electronic device can finally display the first segment image and the second segment image.

[0113] In another example, for the long-focus shooting scene, due to the difference in the performance of the camera module, the performance parameters of the electronic device are also different, so the long-focus shooting scene can be divided into a first performance scene and a second performance scene, which are used to represent the long-focus scene mode of the electronic device under different device performance parameters. For example, in the long-focus shooting scene, it is determined whether it is the first performance scene or the second performance scene according to the performance parameter of the electronic device; and different performance scenes each correspond to a first processing model according to the image attribute parameter of the first original image.

[0114] In the embodiments of the present application, the first processing model can be determined from a plurality of image processing models according to the image attribute parameter, and the first image is further enhanced according to the first processing model, and finally a second image with improved image quality is obtained. That is, on the basis of the image obtained in the original shooting process, the first processing model adapted to the corresponding scene mode is selected to further enhance the image, so as to improve the image quality.

[0115] In some implementations of the first aspect, the saved segment image (the first processed image) and / or the saved segment image (the second processed image) can be selected.

[0116] In one implementation, the save control on the comparison interface can be set in the granularity of a single image, that is, one image corresponds to one save control, and in response to the operation of the save control, the image corresponding to the save control can be saved, and other images corresponding to other save controls are not saved if the other save controls are not operated. While the single image is stored in the gallery and displayed on the thumbnail interface of the gallery, the second segment image and the first segment image can also be displayed on the thumbnail interface of the gallery.

[0117] Further, in order to further improve the user experience, if the electronic device is triggered in response to the other save control corresponding to a certain image being touched, a pop-up window is popped up in the comparison interface to prompt the user whether to save the image.

[0118] In some embodiments, after the two-segment image is saved, the image interface of the gallery application can display the aggregated display thumbnail and the two-segment image thumbnail, wherein different marks are respectively arranged on the aggregated display thumbnail and the two-segment image thumbnail to distinguish them. In other embodiments, the image interface can not display the aggregated display thumbnail, but display the two-segment image thumbnail and the one-segment image thumbnail respectively, wherein the two-segment image thumbnail is marked with an AI enhancement mark to facilitate the user to distinguish the two-segment image and the one-segment image.

[0119] The principle and process of saving the one-segment image alone are the same as those of saving the two-segment image alone, and will not be described in detail here.

[0120] In another implementation, the other save control on the comparison interface can be set with multiple images as a granularity, that is, multiple images share one other save control, and in response to the operation of the other save control, the multiple images corresponding to the other save control can be saved. The deployment position of the other save control is not limited, and can be deployed in the navigation menu at the bottom of the comparison interface or in the navigation bar at the top of the comparison interface.

[0121] It should be noted that the deletion operation of the one-segment image and the two-segment image can be set with a single image as a granularity, or with multiple images as a granularity. The implementation is the same as that of the save operation, which will not be described here.

[0122] In other implementation, if the other save control is not provided in the comparison interface, as shown in (c) of FIG. 18. In this case, when the user clicks the back control 1806, the electronic device saves the two-segment image by default in response to the user operation, that is, the one-segment image and the two-segment image are saved in the electronic device. Of course, in response to the user operation, the electronic device can also delete the one-segment image by default and save the two-segment image. For this, the embodiments of the present application are not limited.

[0123] In a second aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, any method of the first aspect can be implemented.

[0124] In a third aspect, a chip system is provided, which includes a processor coupled with a memory, and the processor is configured to execute a computer program stored in the memory to implement the method of any one of the first aspect.

[0125] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, any method of the first aspect can be implemented.

[0126] In a fifth aspect, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, any method of the first aspect can be implemented. BRIEF DESCRIPTION OF DRAWINGS

[0127] FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application;

[0128] FIG. 2 is a software structure block diagram of the electronic device 100 according to an embodiment of the present application;

[0129] FIG. 3 is a schematic diagram of a shooting scene according to an embodiment of the present application;

[0130] FIG. 4 is an image processing timing diagram in a shooting process according to an embodiment of the present application;

[0131] FIG. 5 is a software structure block diagram of the electronic device 100 according to another embodiment of the present application;

[0132] FIG. 6 is a schematic diagram of a scene of exiting a camera and entering a gallery according to an embodiment of the present application;

[0133] FIG. 7 is an image processing timing diagram according to an embodiment of the present application;

[0134] FIG. 8 is an image processing timing diagram according to another embodiment of the present application;

[0135] FIG. 9 is an image processing timing diagram according to another embodiment of the present application;

[0136] FIG. 10 is a schematic diagram of opening / closing of super-clear portrait according to an embodiment of the present application;

[0137] FIG. 11 is a schematic diagram of an image processing interface of super-clear portrait according to an embodiment of the present application;

[0138] FIG. 12 is an image processing timing diagram of super-clear portrait according to an embodiment of the present application;

[0139] FIG. 13 is a schematic diagram of image saving according to an embodiment of the present application;

[0140] FIG. 14 is a schematic diagram of image saving according to another embodiment of the present application;

[0141] FIG. 15 is a schematic diagram of a two-stage image processing flow according to an embodiment of the present application;

[0142] FIG. 16 is a schematic diagram of opening / closing of a long-focus image function according to an embodiment of the present application;

[0143] FIG. 17 is a schematic diagram of long-focus image function opening / closing according to another embodiment of the present application;

[0144] FIG. 18 is a schematic diagram of image contrast according to an embodiment of the present application;

[0145] FIG. 19 is a schematic diagram of a two-stage image processing flow according to another embodiment of the present application;

[0146] FIG. 20 is a schematic diagram of an image processing flow of an escape scheme according to an embodiment of the present application;

[0147] FIG. 21 is a selection flow of a first processing model under a portrait shooting scenario according to an embodiment of the present application;

[0148] FIG. 22 is a selection flow of a first processing model under a long-focus shooting scenario according to an embodiment of the present application. DETAILED DESCRIPTION

[0149] In the following description, specific details are set forth in order to provide a thorough understanding of embodiments of the application. However, persons having ordinary skill in the art will appreciate that embodiments of the application can be practiced without the specific details, and that the scope of the application is not limited to the particular details described herein.

[0150] It should be understood that the term "comprising" when used in this specification and the appended claims specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0151] It should also be understood that "one or more" in the embodiments of the present application means one, two, or more than two; "and / or" describes the association relationship of associated objects, which means that there can be three relationships; for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0152] In addition, in the description of the specification and the appended claims, the terms "first", "second", "third", "fourth", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0153] Reference throughout this application to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase "in one embodiment" or "in some embodiments" in various places throughout this specification are not necessarily all referring to the same embodiment, but can refer to one or more but not all embodiments. The terms "including," "comprising," "carrying," "having," "containing," and variations thereof are meant to encompass the item listed thereafter, but do not exclude additional, unrecited items. Although the terms "comprise," "comprises," "comprising," "include," "includes," "including," and / or "contain," "contains," or "containing," or any other variation thereof, are used herein in certain instances, these terms are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0154] The term "user interface" in the embodiments of the present application is a medium interface for interaction and information exchange between an application program or an operating system and a user, which realizes conversion between an internal form of information and a form acceptable by the user. The user interface of an application program is source code written in a specific computer language such as Java or extensible markup language (XML), and the interface source code is parsed, rendered, and finally presented as content recognizable by the user such as pictures, texts, and buttons on a terminal device. A control, also called a widget, is a basic element of a user interface, and typical controls include a toolbar, a menu bar, a text box, a button, a scrollbar, a picture, and a text. Properties and content of controls in an interface are defined by tags or nodes, such as XML tags. <textview> 、 <imgview> 、 <videoview>The interface is defined by nodes that specify the controls contained in the interface. One node corresponds to one control or property in the interface, and the nodes are parsed and rendered to present the content visible to the user. In addition, many applications, such as hybrid applications, also contain web pages in the interface. A web page, also referred to as a page, can be understood as a special control embedded in the interface of an application. The web page is a source code written in a specific computer language, such as hyper text markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed by a browser or a web page display component similar to the function of a browser to present content recognizable to the user. The specific content contained in the web page is also defined by tags or nodes in the web page source code, such as HTML defines the content by tags, and the content is displayed by a browser or a web page display component similar to the function of a browser. 、 、 <video> 、 <canvas>to define the elements and attributes of a web page.

[0155] A common form of user interface is a graphic user interface (GUI), which refers to a user interface that displays in a graphical manner. It can be an icon, window, control, etc. interface element displayed in the display screen of an electronic device, wherein the control can include an icon, button, menu, tab, text box, dialog box, status bar, navigation bar, Widget, etc. visual interface element.

[0156] The image processing method provided by the embodiments of the present application can be applied to an electronic device with a screen and a photographing function. The electronic device includes a terminal device, which can also be referred to as a terminal, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The terminal device can be a mobile phone, a smart television, a wearable device, a tablet computer (Pad), a smart screen, a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiments of the present application do not limit the specific technology and specific device form of the electronic device.

[0157] Referring to FIG. 1, a structural schematic diagram of an electronic device provided by an embodiment of the present application is shown. The electronic device 100 can include a processor 110, an external memory interface 120, a first memory 121, a second memory 122, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a touch sensor 180K, an ambient light sensor 180L, etc.

[0158] It can be understood that the structure shown in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0159] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices, or can be integrated in one or more processors.

[0160] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0161] The processor 110 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can hold instructions or data that the processor 110 has just used or cycled through. If the processor 110 needs to use the instructions or data again, it can be called directly from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thereby improving the efficiency of the system.

[0162] The ISP can be used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electric signal, and the camera photosensitive element transmits the electric signal to the ISP for processing and is converted into an image visible to the naked eye. The ISP can also optimize the algorithm of the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be provided in the camera 193.

[0163] The DSP is used to process digital signals, and in addition to being able to process digital image signals, it can also process other digital signals.

[0164] The video codec is used to compress or decompress digital video. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0165] The NPU is a neural-network (NN) calculation processor that learns from the structure of a biological neural network, such as the transmission mode between human brain neurons, to quickly process input information and can also continuously self-learn. Through the NPU, the electronic device 100 can implement intelligent cognitive applications such as image recognition, face recognition, voice recognition, and text understanding.

[0166] In the embodiments of the present application, the electronic device 100 implements the image processing method provided by the embodiments of the present application, first relying on the ISP, the image collected by the camera 193, and second relying on the video codec and the image calculation and processing capability provided by the GPU. Among them, the electronic device 100 can implement neural network algorithms such as face recognition and image processing through the calculation and processing capability provided by the NPU.

[0167] The external memory interface 120 is generally used to connect an external memory (e.g., a USB flash drive, an external hard disk, etc.). In embodiments of the present application, the external memory refers to a storage device that can be used independently of a computer or other device, and is generally used for long-term storage and backup of data. Unlike the internal memory, the external memory can be easily connected and disconnected.

[0168] The first memory 121 and the second memory 122 generally refer to internal memory, and in embodiments of the present application, the internal memory refers to memory directly integrated inside a device, such as read-only memory (ROM), Embedded Multi Media Card (eMMC), Universal Flash Storage (UFS), etc. The internal memory can also be referred to as "memory", and can be used to store computer executable program code, including instructions. The first memory 121 or the second memory 122 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function (such as a sound playing function, an image playing function, etc.), etc.

[0169] The first memory 121 or the second memory 122 can include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs). The random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, e.g., the fifth generation of DDR SDRAM, commonly referred to as DDR5 SDRAM), etc. The non-volatile memory can include a magnetic disk storage device, a flash memory.

[0170] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (for example, machine instructions) of an operating system or other programs running in the background, and can also be used to store data of users and application programs, and the like. The non-volatile memory can also store executable programs and store data of users and application programs, and the like, and can be loaded into the random access memory in advance for direct reading and writing by the processor 110.

[0171] In the embodiments of the present application, the code for implementing the image processing method described in the embodiments of the present application can be stored on the non-volatile memory. When the camera application is running, the electronic device 100 can load the executable code stored in the non-volatile memory into the random access memory.

[0172] The camera 193 is used to capture still images or videos. Objects generate optical images through lenses and project the optical images to photosensitive elements. The photosensitive elements can be charge coupled devices (CCD) or complementary metal-oxide-semiconductor (CMOS) phototransistors. The photosensitive elements convert the optical signals into electrical signals, and then transmit the electrical signals to the ISP to convert into digital image signals. The ISP outputs the digital image signals to the DSP for processing. The DSP converts the digital image signals into image signals in standard RGB, YUV, or the like. In some embodiments, the electronic device 100 can include one or N cameras 193, where N is a positive integer greater than 1.

[0173] The display screen 194 is used to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), or the like. In some embodiments, the electronic device 100 can include one or N display screens 194, where N is a positive integer greater than 1. In some embodiments, the electronic device 100 displays a user interface through the display screen 194, such as a preview interface for taking a photo or a captured image, and the like.

[0174] The electronic device 100 implements display functions through a GPU, the display 194, and an application processor, etc. The GPU is a microprocessor for image processing, connecting the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0175] The electronic device 100 can implement a photographing function through an ISP, the camera 193, a video codec, a GPU, the display 194, and an application processor, etc.

[0176] The electronic device 100 also includes various sensors that can convert various different physical signals into electrical signals. For example, the pressure sensor 180A is used to sense a pressure signal and can convert the pressure signal into an electrical signal. The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. The barometric pressure sensor 180C is used to measure air pressure. The magnetic sensor 180D includes a Hall sensor. The acceleration sensor 180E can detect the magnitude of acceleration of the electronic device 100 in various directions (typically three axes). The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance by infrared or laser. The proximity light sensor 180G can include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The ambient light sensor 180L is used to sense ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display 194 according to the sensed ambient light brightness. The fingerprint sensor 180H is used to collect a fingerprint. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locking, fingerprint photographing, fingerprint answering a call, etc. The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to implement a temperature processing strategy. The bone conduction sensor 180M can obtain a vibration signal.

[0177] The touch sensor 180K is also referred to as a "touch panel". The touch sensor 180K can be disposed on the display 194, and the touch sensor 180K and the display 194 together form a touch screen, also referred to as a "touch panel". The touch sensor 180K is used to detect a touch operation acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the touch event type. Visual output related to the touch operation can be provided through the display 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, which is different from the position where the display 194 is located.

[0178] In the embodiments of the present application, the electronic device 100 can detect the click, sliding and other operations of the user on the display screen 194 by using the touch sensor 180K, so as to realize the photographing process as shown in FIGS. 3, 6, 10, 11, 13, 14, 16, 17 and 18.

[0179] The wireless communication function of the electronic device 100 can be realized by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor, etc.

[0180] The electronic device 100 can realize the audio function by the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D and the application processor, etc. For example, music playing, recording, etc.

[0181] In the embodiments of the present application, the electronic device 100 can enable the microphone 170C to collect the sound signal at the same time in the process of enabling the camera to collect the image, and convert the sound signal into the electrical signal and store it. In this way, the user can get the video with sound.

[0182] The above is a specific description of the embodiments of the present application taking the electronic device 100 as an example. It should be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. The electronic device 100 can have more or fewer components than those shown in the figure, can combine two or more components, or can have a different component configuration. The various components shown in the figure can be realized in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0183] In addition, an operating system runs on the above components. For example, iOS operating system, Android open source operating system, Windows operating system, etc. Application programs can be installed and run on the operating system.

[0184] The operating system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application take the operating system with a layered architecture as an example to illustrate the software structure of the electronic device 100.

[0185] FIG. 2 is a software structure block diagram of the electronic device 100 provided by the embodiments of the present application. In order to facilitate the description, the hardware layer is also shown in FIG. 2 to explain the cooperation relationship between the related hardware and software.

[0186] It should be noted that the embodiments of the present application take a common operating system as an example to illustrate, and as long as the functions of each functional module are similar to the embodiments of the present application, the scheme of the present application can also be realized in other operating systems.

[0187] As shown in FIG. 2, the layered architecture divides software into several layers, each of which has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the operating system is divided into four layers, from top to bottom, the application layer, the application framework layer, the hardware abstraction layer (HAL), and the driver layer.

[0188] The application layer can include a series of application packages. As shown in FIG. 2, the application package can include camera, gallery and other applications.

[0189] The camera is used to provide a shooting function. The gallery application is used to provide image browsing and editing functions. It can be understood that the application layer can also include image editors and other applications. In the embodiments of the present application, in response to the operation of the user, the electronic device can acquire a shooting image through the camera and display the shooting image after image processing through the gallery.

[0190] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications of the application layer. The application framework layer includes some pre-defined functions. In the embodiments of the present application, the application framework layer can include a camera access interface, wherein the camera access interface can include camera management and camera devices. The camera access interface is used to provide application programming interfaces and programming frameworks for camera applications. It can be understood that the application framework layer can also include window managers, content providers, view systems, phone managers, resource managers, notification managers, etc.

[0191] The HAL layer is an interface layer between the application framework layer and the driver layer. The HAL layer can abstract the hardware layer and provide a unified hardware interface for the upper layer. The HAL layer can provide a virtual hardware platform for the operating system. In the embodiments of the present application, the hardware abstraction layer can include a camera hardware abstraction layer and a camera algorithm library.

[0192] The camera hardware abstraction layer can provide virtual hardware for camera device 1, camera device 2 or more camera devices. The camera algorithm library can include running code and data for implementing the image processing method provided in the embodiments of the present application. For example, the camera algorithm library can include an image processing module, which can be used to perform scaling, splicing, and clarity enhancement operations on images.

[0193] The driver layer is a layer between hardware and software. The driver layer includes drivers for various hardware. The driver layer can include camera device drivers, digital signal processor drivers, and image processor drivers, etc.

[0194] Among them, the camera device driver is used to drive the sensor of the camera to collect images and drive the image signal processor to pre-process the images. The DSP driver is used to drive the DSP to process the images. The image processor driver is used to drive the image processor to process the images.

[0195] It can be understood that the driver layer can also include display drivers, audio drivers, sensor drivers, etc., which will not be described in detail here.

[0196] The image processing method in the photographing scenario will be described below in combination with Figure 2 and the description of the system structure.

[0197] In the photographing scenario, in response to the operation of the user opening the camera application (such as the operation of clicking the camera application icon), the camera application calls the camera access interface of the application framework layer, starts the camera application, and then sends an instruction to start the camera through the camera device (such as camera device 1, camera device 2, and / or other camera devices) in the camera hardware abstraction layer.

[0198] After the camera is started, the electronic device displays a photographing preview interface, as shown in (a) of Figure 3. The user can operate a photographing control in the photographing preview interface, which can include a photographing control, a video recording control, and / or a photographing mode control of a certain effect (such as a portrait photographing mode control, a film effect photographing mode control, or a slow-motion photographing mode control, etc.). As shown in (a) of Figure 3, the photographing preview interface includes a photographing control 301 (i.e., a first control). When the user clicks the photographing control 301, in response to the user operation, the touch sensor receives the touch and / or click operation, and after a series of processing through the driver layer and the application framework layer, it can be identified that the control corresponding to the operation is the photographing control. Then, the camera hardware abstraction layer sends a photographing instruction corresponding to the photographing control to the camera device driver. The camera device driver can start the corresponding camera sensor and collect light signals through the sensor. One camera device in the camera hardware abstraction layer corresponds to one or more camera sensors in the hardware layer. The light-sensitive elements in the camera transmit the collected light signals to the image signal processor for preprocessing, and the image obtained after the light signals are converted into electrical signals can be called a raw image (i.e., a RAW image). The image signal processor stores the above-mentioned raw image in the first memory 121. The above-mentioned process can be referred to as camera image collection.

[0199] The first memory 121 can be a Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM, referred to as DDR for short) as described in the embodiment of FIG. 1. It should be noted that the first memory 121 can also include DDR, that is, the first memory 121 can include DDR and other memories, and the other memories can be one or more. For ease of illustration, the embodiment of the present application is described by taking DDR as an example of the first memory 121.

[0200] In the image processing process, the camera hardware abstraction layer calls the camera algorithm library. Based on the digital signal processor and the image processor, the camera algorithm library can realize the image processing capability of the original image. Specifically, the camera algorithm library acquires the original image from the DDR based on the digital signal processor and the image processor, and performs image processing on the original image, and sends the processed image to the camera hardware abstraction layer. Then, the camera hardware abstraction layer can send the processed image for display. The camera hardware abstraction layer can send the processed image to the gallery application for display. As shown in (b) of FIG. 3, the display box 302 (i.e., the first display box) in the shooting interface displays a thumbnail of a shooting image. When the user clicks the display box 302, in response to the user operation, the electronic device displays an interface as shown in (c) of FIG. 3, that is, the interface of the gallery application.

[0201] The camera hardware abstraction layer can also store the processed image in the memory of the electronic device.

[0202] Referring to FIG. 4, it is a timing diagram of image processing in a shooting process according to an embodiment of the present application. FIG. 4 shows the timing flow of image processing in the above shooting scenario. As shown in FIG. 4, in the above shooting scenario, the electronic device displays an interface as shown in (a) of FIG. 4, that is, a shooting preview interface. The user clicks the shooting control 301 (i.e., the first control) in the interface shown in (a) of FIG. 4 to perform the first shooting. In response to the user operation, at t41, the electronic device acquires the original image RAW1 through the camera and performs image processing on the original image RAW1.

[0203] During the image processing of the original image RAW1 by the electronic device, the electronic device displays the interface shown in (b) of FIG. 4. As shown in (b) of FIG. 4, the shooting control 301 is in a non-operable state, in other words, during the image processing of the original image RAW1 by the electronic device, the user cannot shoot the next image.

[0204] The obtained original image is stored in the DDR. During image processing of the original image RAW1 by the electronic device, the electronic device acquires the original image RAW1 from the DDR, and then performs image processing on RAW1 through a camera algorithm library of the HAL to obtain an image in a luminance and chrominance (YUV) format. For ease of description, in this embodiment of this application, the image format is represented as the YUV format.

[0205] Generally, the electronic device only displays the image in the YUV format, and does not directly send the image in the YUV format to the gallery. Therefore, in order to conform to the image storage format, the YUV image can be converted into a format acceptable to the gallery of the electronic device before being stored in the gallery. For example, the YUV image can be converted into a Joint Photographic Experts Group (JPEG) format.

[0206] When the image processing of RAW1 is completed and the YUV image corresponding to RAW1 is obtained, the electronic device converts the YUV image corresponding to RAW1 into an image in the JPEG format to obtain image JPEG1. At time t42, the electronic device displays image JPEG1, and displays the interface shown in (c) of FIG. 4, in which a thumbnail of image JPEG1 is displayed in display box 302 (i.e., a first display box). As shown in (c) of FIG. 4, photographing control 301 (i.e., a first control) is in an operable state.

[0207] It should be noted that the image format for display can also be other formats, for example, a Portable Network Graphics (PNG) format, a High Efficiency Image File Format (HEIF) format, a Graphics Interchange Format (GIF) format, a Bitmap (BMP) format, a Tagged Image File Format (TIFF) format, and a RAW format. For ease of description, in this embodiment of this application, the image in the JPEG format is taken as an example for description.

[0208] The user clicks the photographing control 301 (i.e., the first control) in the interface shown in (c) of FIG. 4 to perform a second shooting. In response to the user operation, at time t43, the electronic device collects the raw image RAW2 through the camera and performs image processing on the raw image RAW2. During the image processing performed by the electronic device on the raw image RAW2, the electronic device displays the interface shown in (d) of FIG. 4. As shown in (d) of FIG. 4, the photographing control 301 again becomes inoperable. Similarly, the specific implementation content of the interface shown in (d) of FIG. 4 can be referred to the related description of the interface shown in (b) of FIG. 4 of the electronic device, and will not be described herein.

[0209] When the image processing on RAW2 is completed and the YUV image corresponding to RAW2 is obtained, the electronic device converts the YUV image corresponding to RAW2 into a JPEG format image to obtain the image JPEG2. At time t44, the electronic device displays the image JPEG2 and displays the interface shown in (e) of FIG. 4, in which the thumbnail of the image JPEG2 is displayed in the display box 302 (the first display box). As shown in (e) of FIG. 4, the photographing control 301 (the first control) again becomes operable. At this time, the user can shoot the next image, and the same is true for the next.

[0210] In the above-mentioned photographing scenario, the time sequence of the above-mentioned times t41, t42, t43 and t44 is in chronological order. It can be understood that, because the image processing needs a certain time, and the two shootings can also be separated by a certain time, the above-mentioned times are not necessarily adjacent times.

[0211] It should be noted that the first shooting and the second shooting in the example of FIG. 4 are only used to illustrate the image processing time sequence relationship corresponding to the two adjacent shootings. In other words, in actual application, the first shooting in any two adjacent shootings can be referred to as the first shooting, and the second shooting in any two adjacent shootings can be referred to as the second shooting. It can be understood that, before the first shooting described in the embodiment of the present application, the user can have performed other shooting behaviors; after the second shooting described in the embodiment of the present application, the user can also perform other shooting behaviors. Therefore, the first shooting can also be the Mth shooting, and the second shooting can also be the (M+1)th shooting, where M is a positive integer.

[0212] As shown in FIG. 4, in the above-mentioned photographing scenario, each time the user clicks the photographing control 301, the electronic device immediately performs image processing on the raw image RAW collected this time, and the user can perform the next shooting only after the completion of the image processing this time. In other words, the entire flow of image processing is executed in the foreground.

[0213] However, as users' demands for image quality increase, the image processing algorithms in camera algorithm libraries have become more complex, resulting in longer processing times for electronic devices. In this situation, if the entire image processing flow is executed in the foreground, users will need to wait for a period of time after each shot before they can take the next one, failing to meet their needs for timely and continuous image capture. Furthermore, when users are taking multiple images in a concentrated period, the electronic device needs to execute complex image processing flows serially and multiple times, increasing power consumption and storage pressure.

[0214] Based on this, this application provides an image processing method. In this application embodiment, the electronic device can process the original image to improve the image quality in the photo, presenting the user with a clear, realistic, and detailed high-quality portrait photo, thus enhancing the user's visual experience. Furthermore, considering the potential pressure on computing power, power consumption, and storage during image processing, this application embodiment performs image processing in stages. After taking a picture, the original image undergoes image processing with lower algorithm complexity first, and then higher algorithm complexity image processing is performed after exiting the camera application. This significantly reduces the time spent on a single shot, enabling users to capture images promptly and continuously. Moreover, it also reduces the power consumption and storage pressure on the electronic device during continuous image capture.

[0215] The following describes the flow of the image processing method according to an embodiment of this application.

[0216] For example, see Figure 5, which is a software structure block diagram of an electronic device 100 provided in another embodiment of this application.

[0217] In response to a user's action of opening the camera application (such as clicking the camera application icon), the camera application calls the camera access interface of the application framework layer to launch the camera application, and then sends a command to start the camera by calling the camera device (camera device 1 and / or other camera devices) in the camera hardware abstraction layer.

[0218] After the camera is activated, the user can operate the photo-taking control in the shooting preview interface. When the user clicks the photo-taking control, the touch sensor receives the touch and / or click operation in response. After a series of processing steps in the driver layer and application framework layer, the control corresponding to the operation can be identified as the photo-taking control. Then, the camera hardware abstraction layer sends the shooting command corresponding to the photo-taking control to the camera device driver. The camera device driver can then activate the corresponding camera sensor and collect light signals through the sensor. One camera device in the camera hardware abstraction layer corresponds to one camera sensor in the hardware layer. The photosensitive element in the camera transmits the collected light signals to the image signal processor for preprocessing, converting them into electrical signals to obtain the raw image (i.e., RAW image).

[0219] In the first-stage image processing flow, the step labeled as ① in FIG. 5 is included, and specifically:

[0220] The original image (i.e., RAW image) processed by the image signal processor of the hardware layer is transmitted to the camera algorithm library through the camera device driver and the camera hardware abstraction layer. The camera hardware abstraction layer calls the camera algorithm library. The camera algorithm library performs first-stage image processing on the original image (i.e., RAW image) based on the digital signal processor and the image processor, to obtain a first-stage processed YUV image. The camera algorithm library stores the image data corresponding to the first-stage image processing (which can include the original image, the first-stage processed YUV image, intermediate processing data, etc.) to the DDR. The DDR stores the received image data to the second memory 122.

[0221] The second memory 122 can be a non-volatile memory, such as an embedded multi media card (eMMC) or a universal flash storage (UFS), etc. It should be noted that the second memory 122 can also be a UFS, i.e., the second memory 122 can include a UFS and other memories, and the other memories can be one or more. For ease of illustration, the UFS is taken as the second memory 122 in the embodiments of the present application for illustration.

[0222] The camera algorithm library sends the first-stage processed YUV image to the camera hardware abstraction layer. Then, the camera hardware abstraction layer can convert it into a JPEG format and then send it for display, such as in the display box 302 of the camera interface. The camera hardware abstraction layer can also send the first-stage processed JPEG image (hereinafter referred to as: one-segment image) to the gallery application for display.

[0223] When the electronic device exits the camera and enters the gallery, the gallery application can display the first-stage processed JPEG image (one-segment image). The gallery application successively calls the camera algorithm library through the camera access interface of the application framework layer and the camera hardware abstraction layer in the HAL layer, so that the camera algorithm library performs second-stage image processing.

[0224] In the second-stage image processing flow, the step labeled as ② in FIG. 5 is included, and specifically:

[0225] The gallery application calls the camera algorithm library in sequence through the camera access interface of the application framework layer, the camera hardware abstraction layer in the HAL layer. The camera algorithm library sends instructions to the DDR based on the digital signal processor and the image processor, and correspondingly, the DDR receives the instructions, obtains the image data corresponding to the first-stage image processing from the UFS, and sends the image data corresponding to the first-stage image processing to the digital signal processor and the image processor; the camera algorithm library performs second-stage image processing on the image data based on the digital signal processor and the image processor, and obtains a second-stage processed YUV image. The camera algorithm library stores the second-stage processed YUV image to the DDR. The DDR stores the received second-stage processed YUV image to the UFS.

[0226] The camera algorithm library sends the second-stage processed YUV image to the camera hardware abstraction layer. Then, the camera hardware abstraction layer can convert it into a JPEG format and send it to the gallery application for display.

[0227] In the embodiment of the present application, the algorithm complexity of the first-stage image processing is lower than that of the second-stage image processing. The image quality of the second-stage processed image is higher than that of the first-stage processed image.

[0228] The image quality can include one or more of the sharpness, color difference, distortion degree, contrast, color fidelity, and sharpness of the image. The color difference can evaluate the color fidelity of the image; the contrast can measure the difference in brightness of different regions in the image; and the sharpness can measure the index of the edge sharpness of the image.

[0229] One way of exiting the camera and entering the gallery is that, as shown in (b) of FIG. 3, the user clicks the display box 302 of the shooting interface of the camera application, and in response to the user operation, the electronic device exits the camera and enters the gallery, and displays the interface as shown in (c) of FIG. 3.

[0230] Another way of exiting the camera and entering the gallery is that, as shown in (a) of FIG. 6, it is the shooting interface of the camera application. In response to the user's operation of exiting the camera (such as sliding the screen from bottom to top, clicking the HOME key, etc.), the electronic device exits the camera and enters the desktop, and displays the desktop interface as shown in (b) of FIG. 6. As shown in (b) of FIG. 6, the desktop interface includes an icon 601 of the gallery application, and when the user clicks the icon 601, in response to the user operation, the electronic device enters the gallery application, and displays the gallery interface as shown in (c) of FIG. 6. As shown in (c) of FIG. 6, the gallery interface includes a plurality of thumbnails, and when the user clicks a thumbnail 602, in response to the user operation, the electronic device displays the interface as shown in (d) of FIG. 6.

[0231] Compared with the image processing flow shown in FIG. 4, the two-stage image processing flow described in FIG. 5, in the shooting process, the first-stage image processing is performed, that is, the image processing with lower algorithm complexity is performed, which greatly reduces the time consumption of one shooting, so that the user can timely capture and continuously shoot, and in the process of the user continuously shooting images, the power consumption of the electronic device can also be reduced; the second-stage image processing, that is, the image processing with higher algorithm complexity, is performed after the camera application is exited, which can provide the user with high-quality images, thereby meeting the user's requirement for the shooting image quality and improving the user experience.

[0232] For the segmented image processing flow of the embodiment of the present application, in one implementation manner, the image processing of the two stages is performed in the foreground. Specifically, when the camera application is in the foreground, after the user shoots an image, the electronic device performs the first-stage image processing on the shot image; when the user exits the camera application and enters the gallery application, at this time, the gallery application is in the foreground, and the electronic device performs the second-stage image processing on the shot image. However, in the case of the user continuously shooting multiple images, since the algorithm complexity of the second-stage image processing is high, a large amount of memory will be occupied, thereby affecting the performance of the electronic device.

[0233] To solve the above problem, in another implementation manner, the first-stage image processing flow is performed in the foreground, and the second-stage image processing flow is performed in the background.

[0234] Specifically, when the camera application is in the foreground, after the user shoots an image, the electronic device performs the first-stage image processing on the shot image. When the user exits the camera application, the electronic device can perform the second-stage image processing on the shot image in the background. When the user opens the gallery application and views a certain image picA, the electronic device sends the image processed in the second stage to the gallery to display to the user.

[0235] Optionally, the data processed in the first stage can be stored in the memory (such as DDR). However, when the number of shot images increases and / or the image quality of the shot images improves, this storage manner will occupy a large amount of space of the DDR. Generally, the DDR has high efficiency in processing images, but if a large amount of space of the DDR is occupied, the processing of other images by the DDR will be affected, thereby affecting the timely capture or continuous shooting of the user. In addition, the power consumption of the DDR is relatively high, and if a large amount of space of the DDR is occupied, the power consumption can be increased.

[0236] To solve the above problems, optionally, the first stage processed data can be stored in the second memory 122 (such as UFS). For example, after completing the first stage image processing, the DDR stores the first stage image processed data into the UFS; when the second stage image processing is needed, the DDR obtains the first stage processed data from the UFS, so that the CPU performs the second stage image processing in the background.

[0237] In the above manner, the space occupation of the first memory can be effectively reduced, thereby meeting the user's demand for continuous shooting and timely snapshot. In addition, reducing the space occupation of the first memory releases the storage space of the first memory, which helps to reduce the power consumption.

[0238] For example, referring to FIGS. 7, 8 and 9, the image processing timing diagram provided by the embodiments of the present application is shown. The image processing timing shown in FIGS. 7 and 8 is the image processing flow of the first implementation manner, i.e., the image processing flows of the two stages are both executed in the foreground. The image processing shown in FIG. 9 is the image processing flow of the second implementation manner, i.e., the image processing flow of the first stage is executed in the foreground, and the image processing flow of the second stage is executed in the background.

[0239] In the manner of exiting the camera and entering the gallery shown in FIG. 3, the electronic device displays the interface shown in (a) of FIG. 7, i.e., the shooting preview interface. The user clicks the shooting control 301 (the first control) in the interface shown in (a) of FIG. 7 to perform the first shooting. In response to the user operation, at t71, the electronic device collects the raw image RAW1 through the camera, performs the first stage image processing on the raw image RAW1, and stores the first stage processed image data Packet1 from the DDR to the UFS. The Packet1 can include the raw image RAW1, the YUV image processed by RAW1 in the first stage, and the intermediate processing result in the first stage processing. At t72, the electronic device displays the interface shown in (b) of FIG. 7, in which the first stage processed image JPEG11 corresponding to the first shooting is displayed in the display frame 302 (the first display frame).

[0240] The user clicks the photographing control 301 in the interface shown in (b) in FIG. 7 to perform the second shooting. In response to the user operation, at time t73, the electronic device collects the raw image RAW2 through the camera, performs the first-stage image processing on the raw image RAW2, and transfers the first-stage processed image data Packet2 from the DDR to the UFS. The Packet2 can include the raw image RAW2, the YUV image processed from RAW2 in the first stage, and the intermediate processing result in the first-stage processing. At time t74, the electronic device displays the interface shown in (c) in FIG. 7, in which the first-stage processed image JPEG21 corresponding to the second shooting is displayed in the display box 302.

[0241] The user clicks the display box 301 in the interface shown in (c) in FIG. 7. In response to the user operation, at time t75, the electronic device exits the camera and enters the gallery. The DDR obtains the first-stage processed image data Packet2 corresponding to the second shooting from the UFS, converts the first-stage processed YUV image in Packet2 into the image JPEG21, and sends the image JPEG21 to the gallery application. Correspondingly, at time t76, the electronic device displays the interface shown in (d) in FIG. 7, in which the first-stage processed image JPEG21 corresponding to the second shooting is displayed, and "AI enhancement in progress..." is displayed.

[0242] In addition, the DDR sends the image data Packet2 to the camera algorithm library for the second-stage image processing, and sends the processed image JPEG22 to the gallery application. Correspondingly, at time t77, the electronic device displays the interface shown in (e) in FIG. 7, in which the second-stage processed image JPEG22 corresponding to the second shooting is displayed.

[0243] The transition from the interface shown in (d) in FIG. 7 to the interface shown in (e) in FIG. 7 needs a period of time.

[0244] It should be noted that in the example of FIG. 7, the original image RAW2 is a first original image, the image after the first-stage processing JPEG21 is a first image, the image after the second-stage processing JPEG22 is a second image, and the image data after the first-stage processing Packet2 is first image data. The operation of the user clicking the photographing control 301 shown in (b) of FIG. 7 can be recorded as a first operation. The interface shown in (b) of FIG. 7 can be recorded as a first interface, the interface shown in (c) of FIG. 7 can be recorded as a second interface, the interface shown in (d) of FIG. 7 can be recorded as a third interface, and the interface shown in (e) of FIG. 7 can be recorded as a fourth interface. The operation of the user clicking the thumbnail of the first display box 302 in (c) of FIG. 7 can be recorded as a second operation. The time t76 can be recorded as a first time, and the time t77 can be recorded as a second time.

[0245] In the manner of exiting the camera and entering the gallery shown in FIG. 6, the electronic device displays the interface shown in (a) of FIG. 8, i.e., a photograph preview interface. The user clicks the photographing control 301 in the interface shown in (a) of FIG. 8 to take a first photograph. In response to the user operation, at the time t81, the electronic device collects an original image RAW1 through the camera, performs first-stage image processing on the original image RAW1, and stores the image data Packet1 after the first-stage processing from the DDR to the UFS. The Packet1 can include the original image RAW1, the YUV image after the first-stage processing on RAW1, and the intermediate processing result in the first-stage processing. At the time t82, the electronic device displays the interface shown in (b) of FIG. 8, in which the display box 302 (a first display box) displays the first-stage processed image JPEG11 corresponding to the first photograph.

[0246] The user clicks the photographing control 301 in the interface shown in (b) of FIG. 8 to take a second photograph. In response to the user operation, at the time t83, the electronic device collects an original image RAW2 through the camera, performs first-stage image processing on the original image RAW2, and stores the image data Packet2 after the first-stage processing from the DDR to the UFS. The Packet2 can include the original image RAW2, the YUV image after the first-stage processing on RAW2, and the intermediate processing result in the first-stage processing. At the time t84, the electronic device displays the interface shown in (c) of FIG. 8, in which the display box 302 displays the first-stage processed image JPEG21 corresponding to the second photograph.

[0247] The user operates to exit the camera application, and in response to the user operation, the electronic device displays an interface shown in (d) of FIG. 8, which is a desktop, at time t85. When the user clicks an icon 601 (second icon) of the gallery application on the desktop, the electronic device displays an interface shown in (e) of FIG. 8, which is an interface of the gallery application, in response to the user operation. When the user clicks a thumbnail 602 on the interface of the gallery application, the electronic device acquires first-stage processed image data Packet1 corresponding to the first photograph from the UFS, converts a first-stage processed YUV image in Packet1 into an image JPEG11, and transmits the image JPEG11 to the gallery application, in response to the user operation, at time t86. Accordingly, the electronic device displays an interface shown in (f) of FIG. 8, which displays the first-stage processed image JPEG11 corresponding to the first photograph and displays "AI enhancement in progress…", in response to the user operation, at time t87.

[0248] In addition, the DDR transmits the image data Packet1 to the camera algorithm library to perform second-stage image processing, and transmits a processed image JPEG12 to the gallery application. Accordingly, the electronic device displays an interface shown in (g) of FIG. 8, which displays the second-stage processed image JPEG12 corresponding to the second photograph, in response to the user operation, at time t88.

[0249] It should be noted that, in the example of FIG. 8, the original image RAW1 is a first original image, the first-stage processed image JPEG11 is a first image, the second-stage processed image JPEG12 is a second image, and the first-stage processed image data Packet1 is first image data. The interface shown in (d) of FIG. 8 can be referred to as a sixth interface, the icon 601 in the interface shown in (d) of FIG. 8 can be referred to as a second icon, and the operation of clicking the second icon by the user can be referred to as a fifth operation. The interface shown in (e) of FIG. 8 can be referred to as a seventh interface, the thumbnail 602 can be referred to as a thumbnail of the first image, and the operation of clicking the thumbnail 602 by the user can be referred to as a sixth operation. The interface shown in (f) of FIG. 8 can be referred to as a third interface, and the interface shown in (g) of FIG. 8 can be referred to as a fourth interface. Time t87 can be referred to as a third time, and time t88 can be referred to as a fourth time.

[0250] In the image processing timing shown in FIGS. 7 and 8, during the shooting, the second stage image processing with high algorithm complexity is not performed, the memory occupancy rate during the shooting is reduced, and the image data processed by the first stage is stored in the UFS, without occupying the resources of the DDR, thereby meeting the needs of the user for timely snapshot and continuous shooting. However, in this implementation manner, when the user enters the gallery, that is, when the gallery application is in the foreground, the electronic device performs the second stage image processing in the foreground. In this manner, when the user views a certain image, the electronic device first displays the image processed by the first stage, and it can take a period of time to present the image processed by the second stage.

[0251] However, it can be understood that this implementation manner is equivalent to performing the second stage image processing on demand, that is, only when the user enters the gallery to view a certain image, the electronic device performs the second stage image processing on the image. In this manner, the second stage image processing with high algorithm complexity does not need to be performed on each shot image, which is beneficial to reducing the system power consumption.

[0252] In some cases, when the algorithm complexity of the second stage image processing is low, the interface shown in (b) of FIG. 7 appears for a very short time, so that the user is difficult to perceive, and in some cases, the electronic device can not display the interface shown in (b) of FIG. 7, that is, after the user clicks the first control, the interface shown in (c) of FIG. 7 is directly displayed.

[0253] In other cases, when the algorithm complexity of the second stage image processing is high, when the user views a certain image for the first time in the gallery application, the user needs to wait for a period of time before the electronic device presents the image with high quality.

[0254] In another implementation manner, the first stage image processing is performed in the foreground, and the second stage image processing is performed in the background. Specifically, when the camera application is opened, the electronic device performs the first stage image processing on the raw image of each shot in the foreground, and stores the raw image (RAW image) and the corresponding image processed by the first stage in the second memory 122 shown in FIG. 1 from the memory. Among them, the image data is stored in the second memory 122 in the shooting order. After exiting the camera, the electronic device performs the second stage image processing on the corresponding image of each shot in the background in the shooting order, and stores the image processed by the second stage in the second memory 122. When the user opens the gallery application and views the image of a certain shot, the electronic device obtains the image processed by the second stage corresponding to the shot from the second memory 122, sends it to the gallery application, and displays it to the user.

[0255] In the application scenarios, the electronic device can perform the second-stage image processing task on the image data corresponding to each shot in sequence in the background after exiting the camera, and can perform the second-stage image processing on the image data corresponding to each shot in sequence in the background after opening the gallery application.

[0256] For example, referring to FIG. 9, an image processing timing diagram provided by another embodiment of the present application is shown. The processing flow of the interfaces shown in (a)-(c) in FIG. 9 is the same as that in the embodiment of FIG. 8, and will not be described here again.

[0257] As shown in the interface in (c) in FIG. 9, the user performs an operation of exiting the camera (such as sliding the screen from bottom to top, clicking the HOME key, etc.). In response to the user operation, at t95, the electronic device exits the camera and displays the interface shown in (d) in FIG. 9, that is, the desktop interface. After exiting the camera, the electronic device performs the second-stage image processing task on the image data corresponding to each shot in sequence in the background, and correspondingly, the DDR obtains the first-stage processed image data (such as Packet11 and Packet21) corresponding to each shot from the UFS as needed, performs the second-stage image processing through the camera algorithm library, and stores the second-stage processed image data (such as Packet12 and Packet22) in the UFS.

[0258] As shown in the interface in (e) in FIG. 9, the user clicks the thumbnail 801 of the first shot image. In response to the user operation, at t96, the DDR obtains the second-stage processed image data Packet12 corresponding to the first shot from the UFS, and converts the first-stage processed YUV image in the image data Packet12 into the image JPEG12, and sends the image JPEG12 to the gallery application. Correspondingly, at t97, the electronic device displays the interface shown in (f) in FIG. 9, in which the second-stage processed image JPEG12 (second image) corresponding to the first shot is displayed.

[0259] It should be noted that in the above application scenarios, the thumbnail in the gallery application interface can be a thumbnail of the first-stage processed image, or a thumbnail of the second-stage processed image. For example, when the electronic device displays the gallery application interface, for an image whose second-stage processing has been completed, the gallery application interface displays a thumbnail of the second-stage processed image thereof; for an image whose second-stage processing has not been completed, the gallery application interface displays a thumbnail of the first-stage processed image thereof. For another example, whether the second-stage image processing has been completed or not, when the electronic device displays the gallery application interface, the gallery application interface displays a thumbnail of the first-stage processed image.

[0260] It should be noted that in the example of FIG. 9, the original image RAW1 is a first original image, the image after the first-stage processing JPEG11 is a first image, the image after the second-stage processing JPEG12 is a second image, and the image data after the first-stage processing Packet1 is first image data. The interface shown in (b) of FIG. 9 is a first interface, and the interface shown in (c) of FIG. 9 is a second interface. The operation of clicking the display box 301 in (c) of FIG. 9 is a fourth operation. The interface shown in (d) of FIG. 9 is a sixth interface, and the icon 601 in the sixth interface can be referred to as a second icon. The operation of clicking the second icon can be referred to as a fifth operation. The interface shown in (e) of FIG. 9 can be referred to as a seventh interface, and the thumbnail 801 in the seventh interface can be referred to as a thumbnail of the first image. The clicking operation on the thumbnail 801 can be referred to as a sixth operation. The interface shown in (f) of FIG. 9 can be referred to as a fourth interface.

[0261] It can be understood that, in the process of sequentially executing the second-stage image processing tasks in the background of the electronic device, if the user needs to view the image of a certain shot, the electronic device preferentially executes the second-stage image processing task corresponding to the shot. For example, the electronic device sequentially executes the second-stage image processing tasks corresponding to the first shot, the second shot, and the third shot in the background. After executing the second-stage image processing task corresponding to the first shot, if the user needs to view the image of the third shot, the electronic device preferentially executes the second-stage image processing task corresponding to the third shot, and sends the image after the second-stage processing corresponding to the third shot to the gallery for display, and then executes the second-stage image processing task corresponding to the second shot.

[0262] Compared with FIG. 7 and FIG. 8, in the image processing flow shown in FIG. 9, the electronic device sequentially executes the second-stage image processing of each image in the background after exiting the camera application. The second-stage image processing flow with high algorithm complexity runs in the background, which has lower demand for memory than running in the foreground, and can reduce the memory occupancy rate, thereby improving the performance of the electronic device. Moreover, when the user views a certain image in the gallery application, if the electronic device has already completed the second-stage image processing task of the image in the background, the electronic device directly displays the image after the second-stage processing, which reduces the transition time between displaying the image after the first-stage processing and displaying the image after the second-stage processing, thereby improving the user experience.

[0263] In the embodiments of the present application, the first-stage image processing provides basic image processing capability, the second-stage image processing improves the image effect, and the first stage and the second stage cooperate to dump to reduce system power consumption. In this way, the requirement of the user for high-quality images can be met without improving the system power consumption.

[0264] In some implementations, the electronic device enables the second-stage image processing by default. In other words, no matter what shooting scene or shooting mode is currently in, the electronic device needs to perform the second-stage image processing for each shooting.

[0265] In some other implementations, the electronic device can enable the second-stage image processing on demand. For example, in some specific shooting scenes, the electronic device enables the second-stage image processing; and for non-specific shooting scenes, the electronic device only performs the first-stage image processing without enabling the second-stage image processing. In this way, it is conducive to reducing the power consumption of the electronic device.

[0266] The specific shooting scene can refer to a specific shooting mode, such as a portrait shooting mode. The specific shooting scene can also refer to a specific shooting parameter, such as a magnification of a picture exceeding a preset magnification, a brightness of a picture reaching a preset brightness, a number of persons included in a picture reaching a preset number, and the like.

[0267] The image processing method of the embodiments of the present application will be described below taking the portrait shooting scene and the long-focus shooting scene as examples.

[0268] Referring to FIG. 10, it is a schematic diagram of opening / closing of super-clear portrait provided by the embodiments of the present application. As an example but not limitation, as shown in (a) of FIG. 10, in a shooting interface, when a user selects a portrait mode by operation (such as clicking the "portrait" mode or sliding the mode bar), in response to the user operation, the electronic device detects whether the current shooting scene conforms to a specific shooting scene, wherein the specific shooting scene can include a portrait shooting scene and a long-focus shooting scene; if it conforms to the portrait shooting scene (specific shooting scene), an interface as shown in (b) of FIG. 10 is displayed.

[0269] As shown in the interface of (b) of FIG. 10, a prompt card 901 and an icon 902 of super-clear portrait are displayed to prompt the user to open the super-clear portrait. The icon 902 is in a first state, which indicates that the super-clear portrait is not currently opened.

[0270] It can be understood that after the super-clear portrait is opened, the electronic device enables the second-stage image processing. In addition, the user can click the selection control 903 in the prompt card 901 to modify the storage settings of the image.

[0271] When the user clicks the prompt card 901, the electronic device displays an interface as shown in (c) of FIG. 9 in response to the user operation. In the interface shown in (c) of FIG. 10, the prompt card 901 disappears, and the icon 902 is still in the first state. When the user clicks the icon 902, the electronic device enables the second-stage image processing in response to the user operation, and displays an interface as shown in (d) of FIG. 10. In the interface shown in (c) of FIG. 10, the icon 902 changes to the second state, which indicates that the super-clear portrait function is currently enabled.

[0272] In the example of FIG. 10, the icon 902 is a first icon. The preset function refers to the super portrait function. The interface shown in (a) of FIG. 10 can be referred to as an eighth interface, and the interfaces shown in (b) and (c) of FIG. 10 can be referred to as a ninth interface. The clicking operation of the user on the first icon can be referred to as a seventh operation. (d) of FIG. 10 can be referred to as a first interface.

[0273] It should be noted that the image enhancement algorithm can be applied in most portrait shooting scenes, and therefore, as shown in FIG. 10, in the embodiment of the present application, the super-clear portrait function can be enabled by the user in the portrait shooting scene, thereby providing a flexible selection mode for the user.

[0274] In the embodiment of the present application, the camera algorithm library in the HAL can provide a plurality of second-stage image processing algorithms. After the super-clear portrait function is enabled, the electronic device selects a corresponding second-stage image processing algorithm according to the current shooting parameter.

[0275] In an implementation manner, the process of selecting the second-stage image processing algorithm can include: if the current shooting parameter meets a first condition, a first algorithm is used in the second stage; if the shooting parameter meets a second condition, a second algorithm is used in the second stage; and if the shooting parameter meets a third condition, a third algorithm is used in the second stage.

[0276] The first algorithm, the second algorithm, and the third algorithm are different image processing algorithms in the second stage, the complexity of the first algorithm is higher than that of the second algorithm, the complexity of the second algorithm is higher than that of the third algorithm, and the complexity of the third algorithm is higher than that of the first-stage image processing algorithm.

[0277] In a case where the algorithm complexity of the third algorithm has a relatively small impact on the memory, the second stage can not trigger dumping when the third algorithm is executed. In other words, in the shooting process, the electronic device sequentially executes the first stage and the second stage, and sends the image processed by the second stage to display, so as to be displayed in the display frame 302 of the shooting interface.

[0278] For example, the first algorithm can employ a full-amount artificial intelligence generated content (AIGC) model. The second algorithm can employ a light-amount AIGC model. The third algorithm can employ a generative adversarial network (GAN) model or a stable diffusion (SD) model. Among them, the algorithm complexity of the full-amount AIGC model is higher than that of the light-amount AIGC model. Specifically, the light-amount AIGC model can refer to using a simplified full-amount AIGC model, usually a model with fewer parameters and a simpler structure.

[0279] It should be noted that the image enhancement technology based on deep learning (for example: image enhancement technology based on AIGC, GAN, SD, etc. model) can be used to improve the image quality, and different image enhancement technologies have their own advantages and disadvantages. Therefore, the selection of using image enhancement technology based on which model is very important for the overall shooting performance and effect of the electronic device.

[0280] Specifically, the image enhancement technology based on the GAN model usually generates more rich and real image details using a convolutional neural network (CNN). However, since the GAN model pays more attention to generating images with a sense of reality during the training process, the similarity with the original image is sacrificed to a certain extent, resulting in a relatively low similarity in content between the generated image and the original image, i.e. low fidelity. Secondly, since the generator and the discriminator two networks are trained simultaneously during the training process, the training process becomes complex and unstable. Therefore, based on this technology, sometimes some unexpected facial features, textures, etc. are generated.

[0281] The image enhancement technology based on the SD model can capture more diversified and higher quality prior information in the data by gradually adding noise to the data and then learning how to recover the original data from the noisy data. This process enables the model to capture more diversified and higher quality prior information in the data, so the diffusion model can generate more natural, delicate and diverse human face images. However, too strong generation prior information is risky for the image enhancement task, which may cause the model to be unstable during the generation process, such as reducing the diversity of the generated results or generating images that do not meet expectations. Therefore, the image generated based on this technology may deviate from the content of the original input image, reducing the effect of image enhancement.

[0282] The image enhancement technology based on the AIGC model can improve the image quality in the photo, generate high-quality portrait photos with clear, realistic and delicate quality for users, and improve the visual experience of the users. However, the AIGC model usually requires a large amount of computing resources, including high-performance hardware (such as GPU or TPU). In addition, due to the high complexity of the model, the generation time of the content may be long. The image enhancement technology based on the light AIGC model is a simplified AIGC model, which usually has fewer parameters and simpler structure, and can run on ordinary computing devices without high-performance hardware. In addition, due to the small size of the model, the generation time of the content is short, and the quality of the generated content may not be as good as that of the full AIGC, but it is sufficient to meet the needs in some application scenarios.

[0283] It should be noted that GAN and SD are specific generation models mainly used in the field of image generation, while AIGC is a more general concept covering the application of generating various types of content using artificial intelligence technology. GAN and SD can be part of AIGC, but AIGC is not limited to GAN and SD. The AIGC model provided in the embodiments of the present application can include SD models, GAN models, etc.

[0284] Based on the differences between the above technologies, the electronic device in the embodiments of the present application can select a corresponding image processing algorithm according to the current shooting parameters.

[0285] For example, referring to Table 1, the decision logic table for selecting the image processing algorithm in the second stage in the portrait shooting scene.

[0286] Table 1

[0287] As shown in Table 1, the decision of selecting the image processing algorithm in the second stage in the portrait shooting scene includes:

[0288] If the number of people identified from the image is 1 (i.e. single person shooting scene), the brightness reaches the first brightness range, and the pixel number corresponding to the size of the face identified from the image is within the first pixel range, the second algorithm is used in the second stage.

[0289] It should be noted that in the embodiments of the present application, the brightness is obtained according to the average illumination intensity (lux index) in the specified area (such as the whole image, the face area or the body area).

[0290] Specifically, the first luminance range indicates that the luminance in the specified region is greater than a first luminance range threshold and less than a second luminance range threshold. The first luminance range threshold can be one of 250, 260, 270, 280, 290, 300, 310, or 320, and the second luminance range threshold can be one of 350, 360, 370, 380, 390, or 400, without limitation of embodiments of the present application. Generally, the first luminance range can be considered as a range of high luminance or medium-high luminance.

[0291] The second luminance range indicates that the luminance in the specified region is less than a third luminance range threshold. The third luminance range threshold can be one of 250, 260, 270, 280, 290, 300, 310, or 320, without limitation of embodiments of the present application. Generally, the third luminance range can be considered as a range of low luminance or low luminance.

[0292] The third luminance range can include the first luminance range and the second luminance range. Generally, the third luminance range can be considered as a range of full luminance or low-medium-high luminance.

[0293] Specifically, the first pixel range indicates that the size of the face region is greater than a first pixel range threshold and less than a second pixel range threshold. The first pixel range threshold can be one of 50, 100, 150, 200, 250, or 300, and the second pixel range threshold can be one of 1000, 1200, 1500, 1800, or 2000, without limitation of embodiments of the present application.

[0294] The second pixel range indicates that the size of the face region is greater than a third pixel range threshold and less than a fourth pixel range threshold. The third pixel range threshold can be one of 50, 100, 150, 200, 250, or 300, and the fourth pixel range threshold can be one of 1000, 1200, 1500, 1800, 2000, or 3000, without limitation of embodiments of the present application.

[0295] For example, if the number of people identified from the image is 1 (i.e., a single-person shooting scene), the luminance reaches the first luminance range, and the size of the face identified from the image corresponds to a pixel number within the first pixel range, the second stage adopts the second algorithm.

[0296] If the number of people identified from the image is 1 (i.e., a single-person shooting scene), the luminance reaches the second luminance range (low brightness) but does not reach the first luminance range, and the size of the face identified from the image corresponds to a pixel number within the first pixel range, the second stage adopts the first algorithm.

[0297] If the number of people in the image is in the range of [2, 10] (i.e., a multi-person shooting scene), the brightness reaches the third brightness range, and the number of pixels corresponding to the size of the face identified from the image is in the first pixel range, the second stage uses the second algorithm.

[0298] If the number of people in the image is in the range of [1, 50] (i.e., a single-person or multi-person shooting scene), the brightness reaches the third brightness range, and the number of pixels corresponding to the size of the face identified from the image is in the second pixel range, the second stage uses the third algorithm.

[0299] It can be understood that the selection process of the above-mentioned second stage algorithm can be performed when the super-clear portrait function is turned on, or can be performed when the second stage is performed. If it is performed when the second stage is performed, since the face recognition can also be involved in the image processing process in the first stage in the portrait shooting mode, the face recognition result after the first stage processing can be obtained when the second stage algorithm is selected, and the algorithm of the second stage is selected according to the face recognition result.

[0300] It should be noted that the above-mentioned Table 1 is only an example of the selection of the second stage algorithm, and the conditions for selecting the second stage algorithm in the portrait shooting scene and the algorithms involved in the second stage are not limited in the embodiments of the present application.

[0301] Exemplarily, the image processing flow of the super-clear portrait is explained in combination with FIG. 11 and FIG. 12.

[0302] As an example but not limitation, as shown in the interface of (a) in FIG. 11, when the super-clear portrait is turned on, the user clicks the shooting control 301, and in response to the user operation, as shown in FIG. 12, at t11, the electronic device collects the raw image RAW through the camera, and processes the raw image RAW through the image processing algorithm of the first stage in the camera algorithm library to obtain the processed YUV image; then the YUV image after the first stage processing is converted into a JPEG image (a segment of image) and sent to display, at t12, the interface shown in (b) of FIG. 11 and (b) of FIG. 12 is displayed. The segment of image can also be sent to the gallery application.

[0303] As shown in FIG. 12, after the first stage processing is completed, the DDR stores the image data Packet of this shooting into the UFS, wherein the image data Packet includes the raw image RAW, the YUV image after the first stage processing, and the intermediate result in the first stage processing.

[0304] In the interface shown in (b) of FIG. 11, the user can click the display box 302, and in response to the user operation, at time t13, the electronic device exits the camera and enters the gallery, the DDR obtains the first-stage-processed image data Packet from the UFS, converts the first-stage-processed YUV image in the Packet into a JPEG image to obtain a segment of the image, and sends the segment of the image to the gallery application. Correspondingly, at time t15, the electronic device displays the interface shown in (c) of FIG. 11. In the interface shown in (c) of FIG. 11, the segment of the image is displayed, and the prompt "AI enhancement in progress…” is displayed.

[0305] It can be understood that the first-stage-processed image data Packet can also include a segment of the image. Correspondingly, after the DDR obtains the image data Packet from the UFS, the electronic device does not need to perform image format conversion, and directly displays the segment of the image.

[0306] In addition, as shown in FIG. 12, the DDR performs second-stage image processing on the image data Packet, and then converts the second-stage-processed YUV image into a JPEG image to obtain a second segment of the image, and then displays the second segment of the image. Correspondingly, at time t16, the electronic device displays the interface shown in (d) of FIG. 11. In the interface shown in (d) of FIG. 11, the second segment of the image is displayed.

[0307] In the interface shown in (d) of FIG. 11, the icon 1002 is displayed. When the user clicks the icon 1002, in response to the user operation, the electronic device displays the comparison interface shown in (e) of FIG. 11. In the comparison interface, the first-stage-processed JPEG image (the "original image” shown in (e) of FIG. 11) and the second-stage-processed JPEG image (the "AIGC” shown in (e) of FIG. 11) are displayed. It can be seen from the comparison that the second-stage-processed image has higher definition than the first-stage-processed image. The user can delete the image that does not need to be saved by operating the control 1003.

[0308] In the example shown in FIG. 11, the first-stage-processed JPEG image (the "original image” shown in (e) of FIG. 11) is the first image, that is, the segment of the image. The second-stage-processed JPEG image (the "AIGC” shown in (e) of FIG. 11) is the second image, that is, the second segment of the image.

[0309] In some implementations, the segment of the image and / or the second segment of the image can be saved. The saving process will be described below.

[0310] In an implementation, the save controls on the comparison interface can be set in granularity of a single image, that is, one image corresponds to one save control, and in response to an operation on the save control, the image corresponding to the save control can be saved, while other images corresponding to other save controls are not saved if the other save controls are not operated. While the single image is separately stored in the gallery and displayed on the thumbnail interface of the gallery, the two-segment image and the one-segment image can also be displayed in aggregation on the thumbnail interface of the gallery.

[0311] Further, in order to further improve the user experience, if the electronic device responds to the touch control of the save control corresponding to a certain image, a pop-up window is popped up in the comparison interface to prompt the user whether to save the image.

[0312] Taking the two-segment image and the one-segment image as an example, as shown in the comparison interface in (a) of FIG. 13, two windows are set in the comparison interface to display the one-segment image and the two-segment image respectively. Save controls are also provided in each window. The window corresponding to the one-segment image includes save control 1301, and the window corresponding to the two-segment image includes save control 1302.

[0313] Taking the separate saving of the two-segment image as an example, specifically, the electronic device responds to the touch control of the save control 1301 corresponding to the two-segment image, as shown in (b) of FIG. 13, a pop-up window is popped up in the comparison interface, the upper interface in the pop-up window displays "whether to save the AI enhanced image for independent display", the lower left of the pop-up window is provided with a "cancel" button, and the lower right is provided with a "save" button. The electronic device responds to the touch control of the "save" button in the pop-up window to save a two-segment image in the image interface. After the electronic device saves the two-segment image, it can jump from the comparison interface to other interfaces. The other interfaces can be a large image page after the two-segment image is saved separately, or an image interface.

[0314] In some embodiments, after the two-segment image is saved, the image interface of the gallery application can display the aggregated thumbnail and the thumbnail of the two-segment image, wherein different identifiers are provided on the aggregated thumbnail and the thumbnail of the two-segment image respectively for distinction. In other embodiments, the image interface can not display the aggregated thumbnail, but display the thumbnail of the two-segment image and the thumbnail of the one-segment image respectively, wherein the thumbnail of the two-segment image is provided with an AI enhanced identifier to facilitate the user to distinguish the two-segment image and the one-segment image.

[0315] The principle and process of separately saving the one-segment image are the same as those of separately saving the two-segment image described above, and will not be described in detail here.

[0316] In another implementation, the save-as control on the comparison interface can be set in granularity of multiple images, i.e., multiple images share one save-as control, and in response to an operation on the save-as control, the multiple images corresponding to the save-as control can be saved. The deployment position of the save-as control is not limited, and the save-as control can be deployed in the navigation menu at the bottom of the comparison interface or in the navigation bar at the top of the comparison interface.

[0317] In an example, taking the case of saving two-paragraph images and one-paragraph images at the same time as an example, as shown in (a) of FIG. 14, a save-as control 1401 is further provided above the two-paragraph images. In a case where the thumbnail 1402 of the one-paragraph image and the thumbnail 1403 of the two-paragraph image are both selected, for example, the thumbnail 1402 of the one-paragraph image and the thumbnail 1403 of the two-paragraph image are clicked, and selection marks (for example, number marks) are displayed on the thumbnail of the one-paragraph image and the thumbnail of the two-paragraph image. In response to the save-as control 1401 being touched, in this case, the electronic device saves the two-paragraph images and the one-paragraph images at the same time, as shown in (b) of FIG. 14, a pop-up window is popped up in the comparison interface, the content of the pop-up window is "Do you want to save the one-paragraph image for independent display?", and "After saving, the AI enhanced image and the original image will be displayed in the gallery at the same time, and the associated effect will no longer be displayed.", the "cancel" button is provided at the lower left of the pop-up window, and the "save" button is provided at the lower right of the pop-up window. In response to the "save" button being touched, the electronic device saves the one-paragraph images and the two-paragraph images in the gallery, and displays the thumbnail of the two-paragraph images and the thumbnail of the one-paragraph images in the image interface of the gallery, and does not display the thumbnail of the two-paragraph images and the one-paragraph images after aggregation. The AI enhanced mark is further provided in the thumbnail of the two-paragraph images, so as to facilitate the user to distinguish the two-paragraph images and the two-paragraph images.

[0318] In (a) of FIG. 14, in a case where only the thumbnail of one image is selected in the comparison interface (for example, only the thumbnail of the two-paragraph image is selected, or only the thumbnail of the one-paragraph image is selected), after the save-as control 1401 is touched, the electronic device can also save only the image corresponding to the selected thumbnail.

[0319] It should be noted that for the deletion operation of the one-paragraph images and the two-paragraph images, the operation can be set in granularity of a single image or in granularity of multiple images. The implementation manner is the same as that of the save operation, and will not be described herein.

[0320] In other implementation manners, if the save-as control is not provided in the comparison interface, as shown in (c) of FIG. 18. In this case, when the user clicks the return control 1806, in response to the user operation, the electronic device saves the two-paragraph images by default, i.e., the one-paragraph images and the two-paragraph images are saved in the electronic device. Of course, in response to the user operation, the electronic device can also delete the one-paragraph images by default and save the two-paragraph images. For this, the embodiments of the present application do not make specific limitation.

[0321] In one example, the two-stage image processing process is introduced by taking the first stage using the AIGC algorithm and the second stage using the GAN model as an example. As shown in (a) of FIG. 15, it is the processing flow in the case of using the AIGC algorithm in the second stage. In the first stage, the original image RAW is sequentially subjected to denoising fusion processing, light and shadow enhancement processing, and skin quality optimization processing to obtain the YUV image after the first stage processing, and the YUV image after the first stage processing is converted into a JPEG image (one segment) and sent to the display. In parallel, the original image RAW is subjected to image segmentation processing to obtain the segmented image of the face. Among them, the DDR takes the result after the light and shadow enhancement processing and the segmented image of the face as the intermediate result in the first stage processing process, and the original image RAW and the YUV image after the first stage processing as the image data Packet stored in the UFS.

[0322] In the second stage, the DDR obtains the image data Packet from the UFS and performs the second stage image processing according to the result after the light and shadow enhancement processing. As shown in (a) of FIG. 15, the AIGC algorithm and the skin quality optimization processing are sequentially performed according to the result after the light and shadow enhancement processing and the segmented image of the face to obtain the YUV image after the second stage processing, and the YUV image after the second stage processing is converted into a JPEG image (two segments) and sent to the display.

[0323] As shown in (b) of FIG. 15, it is the processing flow in the case of using the GAN model in the second stage. In the first stage, the original image RAW is sequentially subjected to denoising fusion processing and light and shadow enhancement processing to obtain the YUV image after the first stage processing. In parallel, the original image RAW is subjected to image segmentation processing to obtain the segmented image of the face.

[0324] Immediately after the first stage, the second stage is performed. In the second stage, the second stage image processing is performed according to the result after the light and shadow enhancement processing. As shown in (b) of FIG. 15, the GAN and the skin quality optimization processing are sequentially performed according to the result after the light and shadow enhancement processing and the segmented image of the face to obtain the YUV image after the second stage processing, and the YUV image after the second stage processing is converted into a JPEG image (two segments) and sent to the display.

[0325] The denoising fusion processing process can include denoising processing and image fusion processing. Denoising processing refers to the process of removing or reducing noise data in the RAW image. During the camera module shooting process, due to external interference (for example, an imaging assembly is provided in an electronic device, and device noise exists during the operation of the imaging assembly), noise data will be generated on the RAW image to reduce image quality. Through denoising processing, the picture quality can be improved.

[0326] Light and shadow enhancement processing refers to optimizing light and shadow effects of an image. For example, by adjusting parameters such as the type (for example, natural light, artificial light), direction (for example, direct light, side light, backlight), intensity, and color of light sources in the image, a preset light and shadow effect is achieved. For another example, by adjusting parameters such as the depth or range of shadows in the image, and the brightness or position of highlights in the image, the shape of objects in the image is more accurately described, and the texture and spatial sense of the objects are enhanced.

[0327] Image segmentation processing is used to segment a face region from an image.

[0328] Skin optimization processing refers to improving the visual effect of skin and enhancing the beauty of skin when an object skin is displayed in an image (for example, a face image of a user is obtained in a portrait shooting mode).

[0329] As can be seen, in the process shown in (a) of FIG. 15, the dumping process is included, that is, the second stage of image processing is not immediately performed after the first stage of image processing, and the first and second segments are displayed separately. In the process shown in (b) of FIG. 15, there is no need for dumping, that is, the second stage of image processing is immediately performed after the first stage of image processing, and only the second segment is displayed.

[0330] It can be understood that "dumping" in the embodiments of the present application refers to that two stages of image processing are discontinuous in time. Specifically, if the second stage of image processing is not immediately performed after the first stage of image processing, that is, two stages of image processing are discontinuous in time, it is represented as "dumping"; if the second stage of image processing is immediately performed after the first stage of image processing, that is, two stages of image processing are continuous in time, it is represented as not "dumping".

[0331] As can be seen from the example of FIG. 15, in the image processing method of the embodiments of the present application, the second stage of image processing algorithm is automatically selected according to the shooting scene. Correspondingly, in the case that the complexity of the second stage of image processing algorithm is high, dumping is needed; in the case that the complexity of the second stage of image processing algorithm is low, dumping is not needed. Through the above manner, the image can be more flexibly segmented and processed, so that the requirement of the user for high-quality images is met without improving the system power consumption.

[0332] It should be noted that FIG. 15 is only an example of the first and second stages of image processing method, and the specific image processing method used in the first and second stages is not limited in the embodiments of the present application.

[0333] Referring to FIG. 16, it is a schematic diagram of starting / closing the long-focus image function according to an embodiment of the present application. As an example but not limitation, when the user opens the camera, the electronic device displays the shooting interface as shown in (a) of FIG. 16 in response to the user operation, which is a shooting preview interface, and detects whether the current shooting scene meets a specific shooting scene, wherein the specific shooting scene can include a portrait shooting scene and a long-focus shooting scene; if the long-focus shooting scene is not met at this time, the electronic device still displays the interface as shown in (a) of FIG. 16.

[0334] In the interface shown in (a) of FIG. 16, when the user zooms in the image (such as clicking the magnification bar 1602 or double-finger zooming in the preview frame), the electronic device adjusts the display magnification of the frame in response to the user operation, and detects again whether the current shooting scene meets the specific shooting scene; if the long-focus shooting scene is met at this time, the electronic device automatically starts the long-focus image function, and displays the interface as shown in (b) of FIG. 16. In the interface shown in (b) of FIG. 16, the current magnification is displayed in the magnification bar 1602 as 30 times, and the icon 1601 is also displayed in the interface, and the icon 1601 is in the third state, indicating that the long-focus image function is currently started.

[0335] It can be understood that after starting the long-focus image function, the electronic device enables the second-stage image processing.

[0336] In some implementations, when the user closes the long-focus image function, the electronic device can detect the current shooting scene every preset time; when the current shooting scene is detected as a long-focus shooting scene, the electronic device can automatically start the long-focus image function again, and switch the icon 1601 to the third state.

[0337] It should be noted that in the example shown in FIG. 16, the icon 1601 is a first icon. The interface shown in (a) of FIG. 16 can be recorded as an eighth interface, and the interface shown in (b) of FIG. 16 can be recorded as a first interface. The operation of the user clicking the icon 1601 on the first interface can be recorded as an eighth operation. The interface shown in (c) of FIG. 16 can be recorded as an eleventh interface.

[0338] It should be noted that for the long-focus shooting scene, whether two-stage image enhancement can be performed depends on the magnification, brightness and other image attribute parameters. Therefore, as shown in FIG. 16, in the embodiment of the present application, whether the long-focus image function is started can be detected by the electronic device under the long-focus shooting scene, thereby improving the intelligent degree of the electronic device and effectively improving the image enhancement effect. In addition, in this case, the user can close the long-focus image function, thereby providing the user with a flexible selection method.

[0339] Referring to FIG. 17, it is a schematic diagram of enabling / disabling the long-focus image function according to another embodiment of the present application. As an example but not limitation, when the long-focus image function is enabled, the electronic device displays an interface as shown in (a) of FIG. 17. In the interface shown in (a) of FIG. 17, the icon 1601 is in the third state, indicating that the long-focus image function is currently enabled.

[0340] When the user zooms out the image (e.g., clicks the zoom bar 1602 or double-taps the preview frame to zoom out the image), in response to the user operation, the electronic device adjusts the display magnification of the image, and displays an interface as shown in (b) of FIG. 17. In the interface shown in (b) of FIG. 17, the zoom bar 1602 displays that the current magnification is 1x. At this time, the electronic device detects again whether the current shooting scene meets the specific shooting scene; if the long-focus shooting scene is not met at this time, and the state of not meeting the scene shooting scene lasts for a period of time, the electronic device automatically disables the long-focus image function, and displays an interface as shown in (c) of FIG. 17. In the interface shown in (c) of FIG. 17, the icon 1601 is hidden.

[0341] It should be noted that the interface shown in (a) of FIG. 17 can be referred to as a first interface, and the interface shown in (c) of FIG. 17 can be referred to as a twelfth interface.

[0342] In an implementation manner, the step of detecting whether the current shooting scene is the long-focus shooting scene includes: if the current shooting parameter meets the fourth condition, determining that the current shooting scene is the long-focus shooting scene; and if the current shooting parameter does not meet the fourth condition, determining that the current shooting scene is not the scene shooting scene.

[0343] The fourth condition can include a limitation condition of one shooting parameter or multiple shooting parameters. For example, the fourth condition can include that the magnification reaches a preset magnification and the brightness reaches a preset brightness.

[0344] When the user clicks the icon 1601 in the interface shown in (b) of FIG. 16, in response to the user operation, the electronic device displays the interface shown in (c) of FIG. 16. In the interface shown in (c) of FIG. 16, the icon 1601 changes to the fourth state, indicating that the long-focus image function is currently disabled.

[0345] As shown in FIG. 17, in the embodiment of the present application, when the image attribute parameter does not meet the scene shooting scene in the long-focus shooting scene, the electronic device can automatically disable the long-focus image function, thereby improving the intelligent degree of the electronic device.

[0346] In the embodiment of the present application, the camera algorithm library in the HAL can provide multiple second-stage image processing algorithms. After the long-focus image function is enabled, the electronic device selects a corresponding second-stage image processing algorithm according to the current shooting scene.

[0347] It can be understood that different image processing algorithms can be set in the second stage for different shooting scenes. For example, for a portrait shooting scene, the image processing algorithm in the second stage can include AIGC, GAN model and skin beautifying algorithm. For a long-focus shooting scene, the image processing algorithm in the second stage can include AIGC algorithm and GAN model. Moreover, the parameters of the AIGC algorithm in the portrait shooting scene and the AIGC algorithm in the long-focus shooting scene are different, and the parameters of the GAN model obtained in the portrait shooting scene and the GAN model obtained in the long-focus shooting scene are also different.

[0348] In an implementation manner, in the long-focus shooting scene, the process of selecting the image processing algorithm in the second stage can include: if the current shooting parameter meets the fifth condition, the fourth algorithm is used in the second stage; and if the shooting parameter meets the sixth condition, the fifth algorithm is used in the second stage. Wherein, the algorithm complexity of the fourth algorithm is higher than that of the fifth algorithm, and the algorithm complexity of the fifth algorithm is higher than that of the image processing algorithm in the first stage.

[0349] In a case, the algorithm complexity of the fifth algorithm has less influence on the memory, and in this case, the dumping can not be triggered when the fifth algorithm is executed in the second stage. In other words, in the shooting process, the electronic device executes the first stage and the second stage in sequence, and sends the image processed in the second stage to display, so as to be displayed in the display frame 302 of the shooting interface.

[0350] For example, referring to Table 2, the decision logic table for selecting the image processing algorithm in the second stage in the long-focus shooting scene.

[0351] Table 2

[0352] As shown in Table 2, the decision for selecting the image processing algorithm in the second stage in the long-focus shooting scene includes:

[0353] If the magnification is in the first magnification range and the brightness reaches the fourth brightness range or the fifth brightness range, the fifth algorithm is used in the second stage.

[0354] If the magnification is in the second magnification range and the brightness reaches the fourth brightness range or the fifth brightness range, the fourth algorithm is used in the second stage.

[0355] Wherein, the magnification is the shooting magnification, which refers to the ratio of the size of the object being shot to the size of the actual image taken through the lens. It represents the proportion of the image formed on the photosensitive element and the original object.

[0356] It should be noted that in the embodiments of the present application, the brightness is obtained according to the average illumination intensity (lux index) in the specified area (such as the whole image, the face area or the body area).

[0357] Specifically, the fourth luminance range indicates that the luminance in the specified area is greater than a fifth luminance range threshold. The fifth luminance range threshold can be one of 350, 360, 370, 380, 390, or 400, and the embodiments of the present application are not limited thereto. Generally, the fourth luminance range can be considered as a high luminance or an extremely high luminance range.

[0358] The fifth luminance range indicates that the luminance in the specified area is greater than a sixth luminance range threshold and less than a seventh luminance range threshold. The sixth luminance range threshold can be one of 250, 260, 270, 280, 290, 300, 310, or 320, and the seventh luminance range threshold can be one of 350, 360, 370, 380, 390, or 400, and the embodiments of the present application are not limited thereto. Generally, the fifth luminance range can be considered as a relatively high luminance or a medium-high luminance range.

[0359] It can be understood that the two-stage algorithm can also be determined according to the magnification when the luminance is greater than a preset luminance threshold. When the luminance is greater than the preset luminance threshold, it can be considered that the luminance reaches a medium-high luminance or above.

[0360] For example, if the magnification is in the range of [30, 50] (in the first magnification range) and the luminance reaches a medium-high (fifth luminance range) or an extremely high luminance (fourth luminance range), the fifth algorithm is used in the second stage.

[0361] If the magnification is in the range of [50, 100] (in the second magnification range) and the luminance reaches a medium-high (fifth luminance range) or an extremely high luminance (fourth luminance range), the fourth algorithm is used in the second stage.

[0362] It should be noted that the above Table 2 is only an example of the second-stage algorithm selection, and the conditions for selecting the second-stage algorithm in the long-focus shooting scene and the algorithms involved in the second stage are not limited in the embodiments of the present application.

[0363] In one example, the image processing flow in the case of starting the long-focus image function is similar to the image processing flow in the case of starting the super-clear portrait, and details can be referred to the description in the embodiments of FIGS. 11-12, which will not be repeated here.

[0364] In another example, as shown in FIG. 9, after each shooting, the electronic device performs the first stage of processing, and then the DDR stores the image data Packet of this shooting into the UFS. When the camera is exited, the electronic device performs the second stage of image processing task on the image data Packet corresponding to each shooting in sequence in the background. Correspondingly, the DDR obtains the first stage of processed image data corresponding to each shooting from the UFS as needed, performs the second stage of image processing through the camera algorithm library, and stores the second stage of processed image data in the UFS.

[0365] After entering the gallery application, as shown in (a) of FIG. 18, when the user clicks the thumbnail 1801 of a certain shooting image, the DDR obtains the second stage of processed YUV image corresponding to the thumbnail 1801 from the UFS in response to the user operation, converts the YUV image into a JPEG image, and sends the JPEG image to display. Correspondingly, the electronic device displays the interface shown in (b) of FIG. 18. The interface shown in (b) of FIG. 18 displays the second stage of processed JPEG image (two-section image) corresponding to the thumbnail 1801.

[0366] When the user clicks the icon 1802 in the interface shown in (b) of FIG. 18, the electronic device displays the interface shown in (c) of FIG. 18 in response to the user operation. In the interface shown in (c) of FIG. 18, the first stage of processed JPEG image (original image) and the second stage of processed JPEG image (AIGC) are displayed. As shown in (c) of FIG. 18, the region 1805 in the second stage of processed image has more and clearer details than the region 1804 in the first stage of processed image.

[0367] The user can delete the image that does not need to be saved by operating the control 1803. Correspondingly, after the user performs the deletion operation, the electronic device deletes the corresponding image in the UFS.

[0368] In one example, as shown in (a) of FIG. 19, the processing flow in the case of using the AIGC algorithm in the second stage. In the first stage, the original image RAW is sequentially subjected to denoising fusion processing, light and shadow enhancement processing, and TGB domain detail processing to obtain the first stage of processed YUV image, and the first stage of processed YUV image is converted into a JPEG image (one-section image) and sent to display. In parallel, the original image RAW is subjected to image segmentation processing to obtain a segmented image of the target object. Among them, the DDR stores the light and shadow enhancement processing result and the segmented image as intermediate results in the first stage of processing, and the original image RAW and the first stage of processed YUV image as image data Packet into the UFS.

[0369] In the second stage, the DDR obtains image data Packet from the UFS, and performs second-stage image processing according to the result of the light and shadow enhancement processing. As shown in (a) of FIG. 19, the AIGC algorithm processing is sequentially performed according to the result of the light and shadow enhancement processing and the segmented image, to obtain a second-stage-processed YUV image, and the second-stage-processed YUV image is converted into a JPEG image (two-stage image) and sent to the display.

[0370] As shown in (b) of FIG. 19, the processing flow in the case of using a GAN model in the second stage. In the first stage, the original image RAW is sequentially subjected to denoising fusion processing and light and shadow enhancement processing, to obtain a first-stage-processed YUV image. In parallel, the original image RAW is subjected to image segmentation processing, to obtain a segmented image of a face.

[0371] Immediately after the first stage, the second stage is performed. In the second stage, second-stage image processing is performed according to the result of the light and shadow enhancement processing. As shown in (b) of FIG. 19, the GAN and skin optimization processing are sequentially performed according to the result of the light and shadow enhancement processing and the segmented image of the face, to obtain a second-stage-processed YUV image, and the second-stage-processed YUV image is converted into a JPEG image (two-stage image) and sent to the display.

[0372] The denoising fusion processing process can include denoising processing and image fusion processing. The denoising processing refers to a process of removing or reducing noise data in the RAW image. During the process of capturing by the camera module, due to external interference (for example, an imaging assembly is provided in an electronic device, and the imaging assembly has device noise during operation), noise data is generated on the RAW image to reduce image quality. The denoising processing can improve the picture quality.

[0373] The light and shadow enhancement processing refers to optimization of the light and shadow effect of the image. For example, by adjusting the type (for example, natural light, artificial light), direction (for example, direct light, side light, backlight), intensity, and color of the light source in the image, a preset light and shadow effect is achieved. For another example, by adjusting the depth or range of the shadow in the image, and the brightness or position of the highlight in the image, the shape of the object in the image is more accurately described, and the texture and spatial sense of the object are enhanced.

[0374] RGB domain detail enhancement refers to a series of processing of the image in the RGB color space, including sharpening processing, contrast enhancement processing, color adjustment processing, denoising processing, and various processing modes.

[0375] The image segmentation processing is used to segment the shooting object from the image. In the portrait shooting scene, the shooting object is a face. In the long-focus shooting scene, the shooting object can be a person, a building, a plant, or a scene.

[0376] As can be seen, in the flow shown in (a) of FIG. 19, the dumping process is included, and the first-stage image and the second-stage image are displayed respectively. In the flow shown in (b) of FIG. 19, the dumping is not needed, and only the second-stage image is displayed.

[0377] As can be seen from the example of FIG. 19, in the image processing method of the embodiment of the present application, the image processing algorithm of the second stage is automatically selected according to the shooting scene. Specifically, in the case where the complexity of the image processing algorithm of the second stage is high, the image data processed in the first stage needs to be dumped; in the case where the complexity of the image processing algorithm of the second stage is low, the dumping is not needed. Through the above manner, the image can be processed more flexibly in segments, so that the requirement of the user for high-quality images can be met without improving the system power consumption.

[0378] As can be seen by comparison with FIG. 15, the image processing algorithm in the long-focus shooting scene shown in FIG. 19 is different from the image processing algorithm in the portrait shooting scene shown in FIG. 15. Specifically, in the portrait shooting scene, after the light and shadow enhancement processing in the first-stage image processing flow, the skin quality optimization processing is performed; and in the long-focus shooting scene, after the light and shadow enhancement processing in the first-stage image processing flow, the RGB domain detail enhancement processing is performed. In addition, in the portrait shooting scene, after the AIGC processing in the second-stage image processing flow, the skin quality optimization processing is performed; and in the long-focus shooting scene, after the AIGC processing in the second-stage image processing flow, the second-stage image is generated. Since the image processing requirements in the two shooting scenes are different, different image processing flows can be set for different shooting scenes.

[0379] It should be noted that FIG. 19 is only an example of the image processing method of the first stage and the second stage, and the image processing method actually used in the first stage and the second stage in the embodiment of the present application is not limited.

[0380] In some cases, due to the power-off of the electronic device, performance factors and the like, the first-stage image processing process can be interrupted, and the first-stage image and the intermediate result in the first-stage image processing process cannot be obtained. In this case, the second-stage image processing process can not be normally run, resulting in poor image effect presented to the user.

[0381] In order to solve the above problem, an escape scheme is provided in the embodiment of the present application.

[0382] Taking a portrait shooting scene as an example, if the first-stage image processing process fails to be completed normally, as shown in FIG. 20, in the first stage, the DDR dumps the original image RAW into the UFS. When entering the second stage, the DDR obtains the corresponding original image RAW from the UFS, and sequentially executes the first-stage image algorithm (such as denoising fusion processing, light and shadow enhancement processing, and image segmentation processing) and the second-stage image processing algorithm (such as the AIGC algorithm and skin quality optimization processing).

[0383] It should be noted that in some application scenarios, in order to ensure the shooting quality, the electronic device collects multiple RAW images each time the user clicks a shooting control, in other words, one shooting corresponds to multiple original images RAW. In this case, if the first-stage image processing process fails to be completed normally, the DDR dumps the multiple original images RAW of this shooting into the UFS. In addition, in this case, usually one of the multiple original images RAW is selected as a reference frame, and the reference frame is used for image segmentation processing. When performing image segmentation processing, the reference frame is first converted into a YUV format, and then the YUV format image is subjected to image segmentation processing. Correspondingly, if the first-stage image processing process fails to be completed normally, the DDR also needs to dump the YUV format image used for image segmentation processing into the UFS. Since the memory and temperature of the electronic device have a greater impact on the performance of image processing, based on this consideration, in some implementations, when the memory of the electronic device reaches a certain occupancy rate, or the temperature of the electronic device reaches a preset temperature, the escape scheme is triggered.

[0384] In the embodiments of the present application, by setting the escape scheme, the normal operation of the second-stage image processing process can be ensured in the case that the first-stage image processing process fails to be completed normally, thereby meeting the user's demand for high-quality images.

[0385] In some embodiments, the second-stage image processing algorithm that is adapted to the shooting scene can also be selected according to the shooting scene. The specific method can include:

[0386] obtaining an image attribute parameter of the first original image, the image attribute parameter being related to a shooting scene when the first original image is shot;

[0387] determining a first processing model from a plurality of image processing models according to the image attribute parameter;

[0388] performing image processing on the first original image according to the first processing model to obtain a second image (two-section image).

[0389] In the embodiments of the present application, the shooting scene can include a first preset scene and a second preset scene. The first preset scene can represent a portrait shooting scene, and the second preset scene can represent a long-focus shooting scene.

[0390] For example, the image processing on the first original image according to the first processing model to obtain the second image can include:

[0391] In the first preset scene, that is, the portrait shooting scene, the first image data is input into the first processing model, and the output image is subjected to skin quality optimization processing to obtain the second image. As shown in FIG. 15, the first image data can include image data subjected to light and shadow enhancement processing and a segmentation result of image segmentation.

[0392] In the second preset scene, that is, the scene shooting scene, the first image data is input into the first processing model to obtain the second image. As shown in FIG. 19, the first image data can include image data subjected to light and shadow enhancement processing and a segmentation result of image segmentation.

[0393] First, several model types of the first processing model are introduced.

[0394] Type 1, diffusion model (PSD) in face scene: used for denoising processing of an image containing a face region, to achieve the denoising effect of the face image and improve the picture clarity.

[0395] Type 2, speech-driven 3D facial animation generation model based on Transformers (FaceFormer): predicts a three-dimensional (3D) face sequence from a speech sequence recorded during the image shooting process.

[0396] The camera captures the audio content contained in the captured scene during the process of capturing the scene. The Transformer model includes an encoder and a decoder. The encoder converts the input original audio sequence into a corresponding speech feature representation. The decoder predicts the future facial action sequence autoregressively with the output of the encoder (speech feature representation), style embedding layer (style information for representing the speaker identity) and past facial action sequence as conditions. The Transformer structure in the decoder can capture the complex relationship between audio and facial action, and generate facial animation synchronized with the audio. Finally, the FaceFormer outputs a predicted facial action sequence corresponding to the length of the input audio. This sequence contains 3D face vertex position information for each frame, which can be used to render an animated 3D face model.

[0397] Type 3, face super-resolution (FaceSR) model: a technology that uses low-resolution face images and face-inherent attribute parameters (i.e., the first segmentation result) to reconstruct high-resolution face images, which converts the input first image with lower resolution into a second image with higher resolution by using face key point features and face segmentation maps as constraints.

[0398] Type 4, patch super-resolution (PatchSR) model: the first image is segmented into multiple image blocks (also referred to as image sub-regions), and each image block is converted from a low-resolution image block to a high-resolution image block using the above-mentioned SR model, and finally the multiple high-resolution image blocks are spliced to obtain the second image.

[0399] Type 5, face attribute controllable reenactment generative network (FaceGAN): a technology that applies a generative adversarial network (GAN) to face image processing. Face reenactment is a popular face animation method in which the identity of a person in an image comes from the source image itself, while the facial movements of the person come from a driving image. The driving image refers to a plurality of image frames obtained by multiple frames of acquisition during the process of the camera capturing the view, and finally a reference frame is determined as the shooting result from the plurality of image frames. Therefore, during the playing process of the plurality of image frames, the face of the person will have a motion change. FaceGAN sets an action unit (AU) as a motion clue and manipulates the source facial feature points (i.e., object face key points) to generate a new set of feature points, which represent the desired motion with the source facial structure. The modified feature points and the source image (the first image) will be provided to a reconstruction module to generate a reenacted face, and finally a second image containing the reenacted face is obtained.

[0400] Type 6, diffusion model for natural images (Nature Stable Diffusion, NatureSD): in long-focus mode, natural images (also referred to as landscape images) can usually be captured in long-focus mode. Therefore, in the process of image processing of natural images using a diffusion model (SD model), the model is trained to learn to extract features from natural images and generate new images. The core idea is to regard the generation process of an image as a diffusion process, gradually generate the final image by adding noise and backpropagation step by step.

[0401] Type 7, Patch-based NSD model (Patch Nature SD): the core idea is to use the NSD model to segment the input image into multiple image blocks, and use the NSD model separately on each image block for image processing, and finally splice the processed image blocks to obtain the final image.

[0402] Type 8, RGB super resolution (RGBSR) model for RGB color space image: a technique for reconstructing a high-resolution RGB format image using a low-resolution RGB format image and a second segmentation result.

[0403] Generally, the image processing requirements for portrait and long-focus landscape images are different, so different first processing models can be used for different shooting scenes. For example, for a portrait shooting scene (first preset scene), at least one of the above types 1-5 can be used. For a long-focus shooting scene (second preset scene), at least one of the above types 6-8 can be used.

[0404] The way of determining the first processing model under the portrait shooting scene and the long-focus shooting scene will be described below respectively.

[0405] As an example of a portrait shooting scene, referring to FIG. 21, a selection process of the first processing model under the portrait shooting scene provided by the embodiment of the application is shown.

[0406] As an example but not limitation, as shown in FIG. 21, under the portrait shooting scene, the image processing algorithm of the first stage is determined according to the image attribute parameters of the first original image. If the image processing algorithm of the first stage uses a denoising fusion network, different processing modes each correspond to a first processing model according to the image attribute parameters.

[0407] The processing mode can include a single-get mode and a double-get mode. The single-get mode means that the electronic device finally displays only the second segment image but not the first segment image. The double-get mode means that the electronic device can finally display the first segment image and the second segment image.

[0408] For example: if in the single-get mode, in the display interface of the gallery application, when the user clicks the thumbnail of a certain image picA, the electronic device displays the second segment image corresponding to the image picA in response to the user operation. At this time, the electronic device only stores the second segment image of the image picA, and does not store the first segment image of the image picA. If in the double-get mode, the display interface of the gallery application displays a thumbnail of a certain image picA, and the thumbnail includes a comparison control. When the user clicks the comparison control on the thumbnail, the electronic device displays the first segment image and the second segment image corresponding to the image picA in response to the user operation, for comparing the image quality and clarity between the two. In the above method, different storage schemes are formulated for the processed images according to different image display requirements, and the memory allocation of the electronic device is reasonably planned.

[0409] In an implementation manner, a first decision tree is provided, taking a portrait shooting mode as a root node, taking the first stage and the second stage as leaf nodes in the first decision tree, and selecting a matched first processing model from the first decision tree for image enhancement according to an attribute parameter range corresponding to an image attribute parameter in the portrait shooting mode, to finally obtain a two-section image.

[0410] Specifically, taking a portrait shooting scene as a root node, there is a first decision tree corresponding thereto, which contains a plurality of algorithm links, each algorithm link is connected with a leaf corresponding to an image processing model, and thus according to an image attribute parameter, a corresponding algorithm link is selected, so that the first stage and the second stage corresponding to the algorithm link are executed.

[0411] The first decision tree includes the following algorithm link structures.

[0412] Algorithm link 1: the attribute parameter range is that the number of faces is 1, all face sizes reach a first size threshold (for example, the face size is greater than or equal to 200 pixel points), the average illumination intensity reaches a first intensity threshold and is less than or equal to a second intensity threshold, the first stage uses a denoising fusion network to perform image processing on a RAW image (a first original image), in the case that the user selects a double-get display mode, the first image obtained in the first stage corresponds to a first resolution, the second stage selects a PSD model as a first processing model to perform image enhancement on the first image, to obtain a second image corresponding to a second resolution, and the second resolution is greater than the first resolution.

[0413] Algorithm link 2: the attribute parameter range is that the number of faces is 1, all face sizes reach the first size threshold, the average illumination intensity reaches the first intensity threshold and is less than or equal to the second intensity threshold, the first stage is to use the denoising fusion network to perform image processing on the RAW image, in the case that the user selects a single-get mode, the first stage is defaulted (in this embodiment, it means that the first image is not stored after the stage ends, but the image processing process of the stage is normally executed), the second stage selects the PSD model as the first processing model to perform image enhancement on the first image, to obtain the second image corresponding to the second resolution, and the first image is covered (the covering means that the electronic device finally only stores the second image without storing the first image).

[0414] Algorithm link 3: the attribute parameter range is that the number of faces is 1, all face sizes reach the first size threshold, and the average illumination intensity is less than the first intensity threshold, the first stage is to use the denoising fusion network to perform image processing on the RAW image, in the case that the user selects the double-get mode, the first image obtained in the first stage corresponds to the first resolution, the second stage selects a FaceFormer model as the first processing model to perform image enhancement on the first image, to obtain the second image corresponding to the second resolution.

[0415] Algorithm link 4: attribute parameter range: face number = 1, all face sizes reach the first size threshold, average light intensity is less than the first intensity threshold, the first stage is to use the denoising fusion network to process the RAW image, in the case that the user selects the single mode, the first stage is default, the second stage selects the FaceFormer model as the first processing model to perform image enhancement on the first image to obtain a second image corresponding to a second resolution, and the first image is covered, and the default means that the original image processing mode of the electronic device is used to process the RAW image to obtain the first image, for example, after the camera captures the shooting picture, the directly generated image is taken as the first image.

[0416] Algorithm link 5: attribute parameter range: face number ≥ 2, all face sizes reach the first size threshold, average light intensity is less than or equal to the second intensity threshold, the first stage is to use the denoising fusion network to process the RAW image, in the case that the user selects the double mode, the first image obtained in the first stage corresponds to the first resolution, the second stage selects the FaceFormer model as the first processing model to perform image enhancement on the first image to obtain a second image corresponding to a second resolution.

[0417] Algorithm link 6: attribute parameter range: face number ≥ 2, all face sizes reach the first size threshold, average light intensity is less than or equal to the second intensity threshold, the first stage is to use the denoising fusion network to process the RAW image, in the case that the user selects the single mode, the first stage is default, the second stage selects the FaceFormer model as the first processing model to perform image enhancement on the first image to obtain a second image corresponding to a second resolution, and the first image is covered.

[0418] Algorithm link 7: attribute parameter range: face number ≥ 1, at least one face size does not reach the first size threshold, the first stage is to use the denoising fusion network to process the RAW image, in the case that the user selects the double mode, the first image obtained in the first stage corresponds to the first resolution, the second stage selects the FaceSR model as the first processing model to perform image enhancement on the first image to obtain a second image corresponding to a second resolution.

[0419] Algorithm link 8: attribute parameter range: face number ≥ 1, at least one face size does not reach the first size threshold, the first stage is to use the denoising fusion network to process the RAW image, in the case that the user selects the double mode, the first stage is default, the second stage selects the FaceSR model as the first processing model to perform image enhancement on the first image to obtain a second image corresponding to a second resolution, and the first image is covered.

[0420] Algorithm link 9: attribute parameter range: focal change ratio meets the first focal change condition (for example: focal change ratio = 3), face number = 1, average light intensity is less than or equal to the third intensity threshold, the first stage is to use the denoising fusion network to process the RAW image, in the case that the user selects the double get mode, the first image obtained in the first stage is down-sampled to obtain a down-sampled image of the first resolution, and the second stage selects the FaceSR model as the first processing model to perform image enhancement on the down-sampled image to obtain a second image corresponding to the second resolution.

[0421] Algorithm link 10: attribute parameter range: focal change ratio meets the first focal change condition, face number = 1, average light intensity is less than or equal to the third intensity threshold, the first stage is to use the denoising fusion network to process the RAW image, in the case that the user selects the single get mode, the first stage is default, and the second stage selects the FaceSR model as the first processing model to perform image enhancement on the first image to obtain a second image corresponding to the second resolution.

[0422] It is worth noting that the focal change ratio belongs to the lens performance parameter of the camera module in the electronic device, and the corresponding focal change ratio is different for different lens performance.

[0423] Algorithm link 11: attribute parameter range: does not meet the above conditions, and only generates a first image corresponding to a first resolution according to the first stage.

[0424] As an example of a long-focus shooting scene, referring to FIG. 22, a selection process of the first processing model under the long-focus shooting scene provided by the embodiment of the application is shown.

[0425] For the long-focus shooting scene, due to the difference in the performance of the camera module, the performance parameters of the electronic device are also different, so the long-focus shooting scene can be divided into a first performance scene and a second performance scene, which are used to represent that the electronic device is in a long-focus scene mode under different device performance parameters.

[0426] As an example but not limitation, as shown in FIG. 22, in the long-focus shooting scene, it is determined whether it is the first performance scene or the second performance scene according to the performance parameters of the electronic device; and then the first processing model corresponding to each of the different performance scenes is determined according to the image attribute parameters of the first original image.

[0427] In an implementation manner, a second decision tree is provided, taking the long-focus shooting scene as a root node, and taking the above-mentioned first stage and second stage as leaf nodes in the second decision tree, for the attribute parameter range corresponding to the image attribute parameters in the long-focus shooting scene, a matched first processing model is selected from the second decision tree for image enhancement, and finally a second image is obtained.

[0428] Specifically, taking a long-focus shooting scene as a root node, there is a second decision tree, which includes a plurality of algorithm links, each algorithm link is connected with a leaf corresponding to an image processing model, and therefore, according to the image attribute parameters, a corresponding algorithm link is selected, so that the first stage and the second stage corresponding to the algorithm link are executed.

[0429] The second decision tree includes the following algorithm link structure:

[0430] For the first performance scene, the algorithm link 1, the algorithm link 2 and the algorithm link 3 are included, and for the second performance scene, the algorithm link 4, the algorithm link 5, the algorithm link 6 and the algorithm link 7 are included, which will be described in detail below.

[0431] The algorithm link 1: the attribute parameter range is that no portrait is detected in the first image, and the first image does not belong to a text scene and a running scene, if the first image includes a texture feature, and the magnification range is in a first numerical value range (for example, greater than or equal to 30 times and less than or equal to 50 times), the second stage selects the NatureSD model as the first processing model to perform image enhancement on the first image.

[0432] The algorithm link 2: the attribute parameter range is that the magnification range is greater than the first numerical threshold, and other attribute parameter ranges in the algorithm link 1 are not met, a general decision tree is used, for example, the first decision tree or the second decision tree described above.

[0433] The algorithm link 3: the attribute parameter range does not meet any attribute parameter range of the algorithm link 1 and the algorithm link 2, and a general decision tree is used, for example, the first decision tree described above.

[0434] The algorithm link 4: the attribute parameter range is that no portrait is detected in the first image, and the first image does not belong to a text scene and a running scene, if the first image includes a texture feature, and the magnification range is in a first numerical value range, the first stage is to use a denoising fusion network to perform image processing on the first image to obtain a first processing image, and the second stage selects the RGBSR model as the first processing model to perform image enhancement on the first processing image.

[0435] The algorithm link 5: the attribute parameter range is that no portrait is detected in the first image, and the first image does not belong to a text scene and a running scene, if the first image includes a texture feature, and the magnification range is in a second numerical value range (the second numerical value range is greater than the first numerical value range), the first stage is to use a denoising fusion network to perform image processing on the first image to obtain a first processing image, and the second stage selects the NatureSD model as the first processing model to perform image enhancement on the first processing image.

[0436] Algorithm link 6: in the case that the range of the magnification is greater than the first numerical threshold and does not meet the range of other attribute parameters in algorithm link 4 and algorithm link 5, a common decision tree, for example, the first decision tree described above, is used.

[0437] Algorithm link 7: in the case that the range of the attribute parameter does not meet the range of any attribute parameter in algorithm link 4, algorithm link 5 and algorithm link 6, a common decision tree, for example, the first decision tree described above, is used.

[0438] In the embodiment of the present application, the input of the NatureSD model is the output of the denoising fusion network algorithm in Quadra format or the denoising fusion network algorithm in Hex format in the long-focus mode. The image size of the first image is determined by the shooting ratio, for example: 4:3 or 9:16. Before image enhancement by the first processing model, the first image is expanded to a picture with an aspect ratio of 1:1 (the AI WAR algorithm supports external specification of the cropping range) in the field of view (FOV) of the first image, and then scaled to a resolution size of 512*512, to obtain a second adjusted image. The input of the NatureSD algorithm is an 8-bit YUV image with a resolution of 512*512, and the output is an 8-bit YUV image with a target size as the second image, wherein the target size is consistent with the image size of the first image.

[0439] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0440] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in each of the above method embodiments can be implemented.

[0441] The embodiment of the present application also provides a computer program product, which, when running on a terminal device, enables the terminal device to implement the steps in each of the above method embodiments.

[0442] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the first device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium can not be an electrical carrier signal and a telecommunication signal.

[0443] The embodiment of the present application also provides a chip system, which comprises a processor coupled with a memory. The processor executes a computer program stored in the memory to implement the steps of any method embodiment of the present application. The chip system can be a single chip or a chip module composed of multiple chips.

[0444] In the above embodiments, the description of each embodiment has its own focus. The parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0445] Those skilled in the art can appreciate that the units and method steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0446] Finally, it should be noted that: the above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. An image processing method, characterized by, The method is applied to an electronic device, and the method comprises: displaying a first interface; the first interface is a shooting interface of a camera application, and the first interface comprises a first control for indicating photographing; receiving a first operation of a user on the first control; in response to the first operation, displaying a second interface; wherein the second interface comprises a thumbnail of a first image, the first image being obtained by performing first processing on a first original image, the first original image being captured by the camera in response to the first operation; receiving a second operation of a user on the thumbnail of the first image; in response to the second operation, displaying a third interface at a first time; wherein the third interface is an interface of a gallery application, and the third interface comprises the first image; at a second time, displaying a fourth interface; wherein the fourth interface is an interface of the gallery application, and the fourth interface comprises a second image and a second control, the second image being obtained by performing second processing on the first original image, the algorithm complexity of the first processing being lower than that of the second processing, and the image quality of the second image being higher than that of the first image, the second time being after the first time; receiving a third operation of a user on the second control; in response to the third operation, displaying a fifth interface; wherein the fifth interface comprises the first image and the second image.

2. The method of claim 1, wherein, The method further comprises: in a case where a preset function is turned on, displaying the first interface, and the first interface further comprises a first icon corresponding to the preset function, the first icon being in a first state, and the first state indicating that the preset function is turned on.

3. The method of claim 1, wherein, The method further comprises: in response to the first operation, performing the first processing on the first original image to obtain the first image; displaying the second interface; saving the first image and image data corresponding to the first image from a first storage to a second storage; in response to the second operation, obtaining the first image and the image data corresponding to the first image from the second storage; at the first time, displaying the third interface; performing the second processing on the image data corresponding to the first image to obtain the second image; at the second time, displaying the fourth interface.

4. The method of claim 1, wherein, After displaying the second interface, the method further comprises: in response to a fourth operation of a user on the second interface for exiting the camera application, displaying a sixth interface; wherein the sixth interface is an interface of a desktop application, and the sixth interface comprises a second icon of the gallery application; receiving a fifth operation of a user on the second icon; in response to the fifth operation, displaying a seventh interface; wherein the seventh interface is an interface of the gallery application, the sixth interface comprises a thumbnail of the first image; receiving a sixth operation of a user on the thumbnail of the first image; in response to the sixth operation, displaying the third interface at a third time; at a fourth time, displaying the fourth interface; wherein the fourth time is after the third time.

5. The method of claim 4, wherein, The method further comprises: in response to the first operation, performing the first processing on the first original image to obtain the first image, and displaying the second interface; storing the first image and image data corresponding to the first image from a first storage to a second storage; in response to the sixth operation, obtaining the first image and image data corresponding to the first image from the second storage; at the third moment, displaying the third interface; performing the second processing according to the image data corresponding to the first image to obtain the second image; at the fourth moment, displaying the fourth interface.

6. The method of claim 1, wherein, After displaying the second interface, the method further comprises: in response to a fourth operation of the user exiting the camera application on the second interface, displaying a sixth interface; wherein the sixth interface is an interface of a desktop application, and the sixth interface comprises a second icon representing a gallery application; receiving a fifth operation of the user on the second icon; in response to the fifth operation, displaying a seventh interface; wherein the seventh interface is an interface of the gallery application, and the sixth interface comprises a thumbnail of the first image; receiving a sixth operation of the user on the thumbnail of the first image; in response to the sixth operation, displaying the fourth interface.

7. The method of claim 4, wherein, The method further comprises: in response to the first operation, performing the first processing on the first original image to obtain the first image, and displaying the second interface; storing the first image and image data corresponding to the first processing from a first storage to a second storage; in response to the fourth operation, obtaining the first image data from the second storage; performing the second processing according to the first image data to obtain the second image; in response to the sixth operation, displaying the fourth interface.

8. The method according to any one of claims 1 to 7, characterized in that, The method further comprises: displaying an eighth interface, wherein the eighth interface is a shooting interface of a camera application; if a current shooting scene is a first preset scene, displaying a ninth interface, wherein the ninth interface is a shooting interface of the camera application, the ninth interface comprises the first icon, and the first icon is in a second state, and the second state represents that the preset function is closed; receiving a seventh operation of the user on the first icon; in response to the seventh operation, starting the preset function; displaying the first interface.

9. The method of claim 8, wherein, The method further comprises: when the first interface is displayed, receiving an eighth operation of the user on the first icon; in response to the eighth operation, closing the preset function; displaying a tenth interface, wherein the tenth interface is a shooting interface of the camera application, and the tenth interface comprises the first icon, and the first icon is in the second state.

10. The method according to claim 8 or 9, characterized in that, The first preset scene is a shooting scene of a portrait shooting mode.

11. The method according to any one of claims 1 to 10, characterized in that, The method further comprises: displaying an eighth interface, wherein the eighth interface is a shooting interface of a camera application; if a current shooting scene is a second preset scene, starting the preset function; displaying the first interface.

12. The method of claim 11, wherein, The method further comprises: when the first interface is displayed, receiving an eighth operation of the user on the first icon; in response to the eighth operation, closing the preset function; The eleventh interface is a shooting interface of the camera application, and the eleventh interface includes the first icon, and the first icon is in a second state, and the second state indicates that the preset function is in a closed state.

13. The method of claim 11, wherein, The method further includes: When the first interface is displayed, if a current shooting scene jumps out of the second preset scene, a twelfth interface is displayed at a fifth time point after the second preset scene is jumped out; the twelfth interface is a shooting interface of the camera application, and the twelfth interface does not include the first icon.

14. The method according to any one of claims 11 to 13, characterized in that, The magnification of the image shot in the second preset scene is greater than a preset magnification.

15. The method according to any one of claims 11 to 14, characterized in that, The second preset scene is a long-focus shooting scene.

16. The method according to any one of claims 1 to 15, characterized in that, The method further includes: An image attribute parameter of the first original image is acquired, the image attribute parameter is related to a shooting scene when the first original image is shot; A first processing model is determined from a plurality of image processing models according to the image attribute parameter, the first processing model is used to improve the image quality of the first original image; The first original image is subjected to image enhancement according to the first processing model, and the second image is obtained.

17. An electronic device, comprising: The electronic device includes a processor, and the processor is used to run a computer program stored in a memory to implement the method in any one of claims 1 to 16.

18. A chip system, characterized by The chip system includes a processor, and the processor is coupled with a memory, and the processor is used to run a computer program stored in the memory to implement the method in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by one or more processors to implement the method in any one of claims 1 to 16.

Images