Photo editing method and electronic equipment

Abstract

The embodiment of the invention provides a photo editing method and electronic equipment, and the method can carry out the multi-exposure special effect processing of a dynamic photo based on the main body detection information, motion detection and other metadata carried in the dynamic photo, and obtains a multi-exposure photo. Compared with a dynamic photo, a plurality of copies of the main body appear in the multi-exposure photo generated by the multi-exposure special effect processing, and the plurality of copies are fixed at different positions, so that a user can simultaneously see the plurality of copies moving to different positions at different moments in the multi-exposure photo, and the image originality is sufficient. Moreover, the user can obtain the photo with multiple exposure special effects through one-time editing operation, which is simple and efficient, and is beneficial to arousing the enthusiasm of the user to create by using the dynamic photo.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to a photo editing method and an electronic device. Background Art

[0002] Multiple exposure, also known as multiple exposure, is a technical method of using two or more independent exposures to overlap different images and record them on a photo. Multiple exposure is one of the most creative and expressive techniques in digital photography. At present, many SLR cameras support multiple exposure shooting mode, which can support users to create different overlapping special effects they want. On SLR cameras, the overlapping mode of multiple exposure generally has four options: add, average, brighten, and darken. Among them, add means that the exposures overlap each other without modification; the gain effect is not adjusted; average means that the gain of each exposure is divided by the total number of exposures before overlapping exposure; brighten means that the camera will compare the loudness of each exposure and only use the brightest; darken means that the camera will compare the pixels of each exposure and only use the darkest. Creators can choose different multiple exposure modes according to different shooting scenes to get ideal fusion photos with special visual effects.

[0003] However, shooting with a SLR camera requires high photographic literacy and shooting scenes from the user, and the user needs to perform multiple shooting operations to get a satisfactory photo. Summary of the invention

[0004] The embodiments of the present application provide a photo editing method and an electronic device, which can support users to obtain photos with multiple exposure effects through one shooting and one creation. The operation is simple and convenient, which can enhance the user's creative desire.

[0005] In a first aspect, an embodiment of the present application provides a photo editing method, which may include: an electronic device displays a dynamic photo, and detects a first editing operation for the dynamic photo, and in response to the first editing operation, generates a multiple exposure photo using the dynamic photo. Then, the electronic device may display the multiple exposure photo.

[0006] The dynamic photo is composed of the cover photo and the first video. The first video is composed of a preview stream of photos several seconds before and after the first photo moment. The first photo moment is the moment that triggers the shooting of the dynamic photo. The resolution of the cover photo is greater than that of the first video.

[0007] Among them, the multiple exposure photos include multiple clones of the first moving subject in the dynamic photo, and the multiple clones are frozen at different positions. The paths strung together at the different positions are consistent with the movement path of the first moving subject recorded in the dynamic photo.

[0008] In the first aspect, the electronic device may be a device with a display such as a mobile phone, a tablet computer, a smart screen, etc. The electronic device may display the dynamic photo based on a user operation of opening the dynamic photo. The user operation of opening the dynamic photo may be a user operation of clicking on the thumbnail of the dynamic photo in the gallery. The embodiment of the present application does not limit the triggering conditions for the electronic device to display the dynamic photo.

[0009] In the first aspect, the first editing operation can be used to trigger multiple exposure special effects editing of the dynamic photo. It can be, for example, clicking Figure 2 The user operation of the special effect editing option "multiple exposure" in the user interface 21 is shown. The embodiment of the present application does not display the form of the first editing operation, and the first editing operation can even be an editing operation in the form of a voice command.

[0010] The photo editing method provided in the first aspect enables the user to simultaneously see multiple avatars moving to different positions at different times in a multiple exposure photo, which is very creative. Moreover, the user can obtain a photo with multiple exposure special effects through a single editing operation (such as the first editing operation), which is simple and efficient, and is conducive to stimulating the user's enthusiasm for creating with dynamic photos.

[0011] In conjunction with the first aspect, in some embodiments, the clarity of the multiple clones in the multiple exposure photos may be consistent or close to the clarity of the first moving subject in the dynamic photo. Close means that the clarity difference is small and does not exceed a specific threshold. The setting of the specific threshold may depend on which algorithm is used to evaluate clarity. That is, the clones are clear in the multiple exposure photos, not blurred.

[0012] In combination with the first aspect, in some embodiments, in order to present the same shooting scene, the background in the multiple exposure photos and the background of the cover photo in the dynamic photo can be the same.

[0013] In combination with the first aspect, in some embodiments, in the multiple exposure photos, the images of the multiple avatars may respectively come from the images of the first moving subject in multiple key frames, and the multiple key frames are video frames in the first video.

[0014] The multiple key frames may be selected from the first video, and may have the following characteristics: there is a first moving subject in the key frames, and the positions of the first moving subject in the multiple key frames do not overlap with each other. In this way, it can be ensured that there are multiple clones in the multiple exposure photos, and the multiple clones are independent and distinct from each other.

[0015] The frozen position of each clone in the multiple exposure photo can be aligned with the position of the first moving subject in a key frame. In this way, the frozen positions of multiple clones in the multiple exposure photos can be strung together to reproduce the movement trajectory of the subject in the dynamic photo, forming a strong connection between the two and showing a very creative image expression.

[0016] In combination with the first aspect, in some embodiments, the dynamic photo may further carry first metadata for multiple exposure special effects processing, and generate multiple exposure photos based on the dynamic photo. The first metadata may include subject detection information and optical flow detection information, the subject detection information is obtained by performing subject detection on the photo preview stream when taking the dynamic photo, and the first optical flow detection information is obtained by performing optical flow detection on the photo preview stream when taking the dynamic photo.

[0017] In conjunction with the first aspect, in some embodiments, the specific implementation of generating a multiple exposure photo using a dynamic photo may include the following steps:

[0018] S1. Based on the optical flow detection information in the first metadata, the video frames in the first video are aligned based on the reference frame to obtain a steady-state video stream.

[0019] The reference frame belongs to the first video, and the reference frame includes an intermediate video frame in the first video.

[0020] S2. Perform subject detection on the steady-state video stream based on the subject detection information in the first metadata.

[0021] S3. Select multiple key frames from the steady-state video stream according to the subject detection result of the steady-state video stream.

[0022] The electronic device can use the subject detection information to select multiple video frames with the first subject and non-overlapping subject positions from the first video of the dynamic photo as key frames. Of course, in actual applications, there may be more constraints for key frame selection.

[0023] S4. Segment the subject of the key frame and determine the subject outline of the key frame.

[0024] S5. Fusing the main body contours of multiple key frames to obtain a first fused contour, where the first fused contour includes contours of multiple avatars in the dynamic photo.

[0025] S6. Fusion multiple key frames according to the first fusion profile to generate an initial multiple-exposure photo.

[0026] S7. Use the cover photo in the dynamic photo to convert the initial multiple exposure photo into a higher resolution multiple exposure photo.

[0027] In combination with the first aspect, in some embodiments, after displaying the multiple exposure photos, the electronic device may also detect a first user operation on the multiple exposure photos, and in response to the first user operation, play a second video, where the second video is generated using the dynamic photos; wherein, when the second video is played, more and more clones of the first moving subject gradually appear in the video screen, and the clones that appear in sequence are successively frozen in the video screen, and the appearance paths of the clones are consistent with the movement path of the first moving subject recorded in the dynamic photos. In this way, when the second video is played, a visual effect of a ghost clone walking around will be presented, which can also be called a "light step" effect, and the image expression is unique.

[0028] The first user operation may be, for example, a user operation of long pressing a multiple exposure photo.

[0029] In combination with the first aspect, in some embodiments, the clarity of the avatar in the second video may be consistent with or close to the clarity of the first moving subject in the first video.

[0030] In combination with the first aspect, in some embodiments, like the multiple exposure photo, the second video is also generated using the dynamic photo.

[0031] In the second video, the images of the multiple clones can come from the images of the first moving subject in multiple key frames. These multiple key frames are key video frames in the low-resolution video (first video) in the dynamic photo. These multiple key frames can be selected from the first video, and have the following characteristics: there is a first moving subject in the key frames, and the positions of the first moving subject in these multiple key frames do not overlap with each other. In this way, it can be ensured that the clones are frozen in the second video playback screen, and these multiple clones are independent and distinct from each other.

[0032] The second video frame may include multiple video frames, and the number of clones contained in each of the multiple video frames increases in accordance with the playback order of the multiple video frames, wherein a video frame in a later playback order includes a clone in a video frame in a previous playback order, and the same clone in a video frame in a later playback order is positionally aligned with that in a video frame in a previous playback order.

[0033] In addition, in order to present the same shooting scene, the background of the second video can be the same as the background of the first video. Specifically, the background of the first video frame in the second video is the same as the background of the second video frame in the first video frame, and the first video frame and the second video frame can have the following connection: the frozen position of the last avatar in the first video frame is the same as the position of the first moving subject in the second video frame.

[0034] In combination with the first aspect, in some embodiments, after generating a multiple exposure photo and a second video with a "light step on water" special effect based on a dynamic photo, the electronic device can associate the two together and save them as a multiple exposure fusion photo. The multiple exposure fusion photo consists of a high-definition multiple exposure photo and a video with a "light step on water" special effect. Specifically, the electronic device can save the multiple exposure fusion photo to the gallery. In this way, the user can view the high-definition multiple exposure photos in the multiple exposure fusion photos when opening the gallery, and can further view the "light step on water" special effect video by long pressing the multiple exposure photo to trigger the playback of the "light step on water" video. In a specific implementation, the electronic device can associate and save the multiple exposure photos and the second video with the same file name, which is similar to the association and saving of high-definition cover photos and low-definition short videos in dynamic photos.

[0035] In a second aspect, an embodiment of the present application provides an electronic device, which may include a processor and a memory; wherein the memory is coupled to the processor, and the memory is used to store a computer program, and when the processor executes the computer program, the electronic device executes the method described in the first aspect or any possible implementation method of the first aspect.

[0036] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described in the first aspect or any possible implementation manner of the first aspect is implemented.

[0037] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A dynamic photo is shown as an example;

[0039] Figure 2 A human-computer interaction method provided by an embodiment of the present application is shown;

[0040] Figure 3 A human-computer interaction method provided by an embodiment of the present application is shown;

[0041] Figure 4 The image processing framework provided by the embodiment of the present application is shown;

[0042] Figure 5 The detailed process of multiple exposure special effects processing provided by the present application is shown;

[0043] Figure 6A-6C The process of multiple exposure fusion in the embodiment of the present application is exemplarily shown;

[0044] Figure 7 An example is shown using Figure 6A-6C The process of generating the “Light Footsteps” special effects video by fusing frames in;

[0045] Figure 8 The overall process of the photo editing method provided by the embodiment of the present application is shown;

[0046] Fig. 9 An electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0047] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0048] A multiple exposure shooting function similar to that on SLR cameras has appeared on some mobile phones. It can be released as a multiple exposure mode in the mobile phone camera or a certain image processing software. In this way, users can use the mobile phone camera to take multiple photos independently or select multiple independently taken photos from the gallery, and fuse and superimpose these multiple photos to get a photo with multiple exposure effects. Users can manually adjust the fusion position, transparency, etc., so that they can customize the multiple exposure effect. The multiple exposure shooting mode on the mobile phone is similar to the multiple exposure shooting mode on the SLR camera in terms of algorithm logic and shooting strategy, and it also requires multiple shots to get a photo with multiple exposure effects.

[0049] An embodiment of the present application provides a photo editing method that can be applied to electronic devices such as mobile phones. The method can support users to obtain photos with multiple exposure effects through one shooting and one creation. The operation is simple and convenient, which can enhance the user's creative desire.

[0050] The photo editing method provided in the embodiment of the present application can be implemented based on a moving photo.

[0051] Dynamic photos are a special form of photos, which can be composed of a high-definition cover photo and a video of several seconds (such as 3 seconds). The high-definition cover photo is generally a high-resolution photo obtained through the photo stream after the user presses the shutter, and the resolution can be, for example, 4096*3072; the short video can be generated by the preview stream 1.5 seconds before and after the user presses the shutter, and its resolution is lower, usually only 1920*1080. Compared with ordinary static photos, dynamic photos provide more editing possibilities because they include a short video of 1.5 seconds before and after the photo is taken.

[0052] Figure 1 The dynamic photo 11 is shown as an example. The dynamic photo 11 is marked by a dynamic photo identifier 12 in the gallery 13. Figure 1 As shown, the dynamic photo 11 includes a cover photo 11a and a short video 11b. Among them, the short video 11b records a short movement process of the person 15 playing basketball. The person 15 successively goes through multiple different positions during the movement process and makes different actions in these multiple different positions. These different actions are connected in series according to their respective action times to form a short and coherent basketball action. In the short video 11b, one frame of the image records a moment of the action of the person 15, and multiple frames of the image are arranged in the order of the action time, so that when the user plays the dynamic photo 11, he can see the coherent action image of the person 15 playing basketball.

[0053] by Figure 1 Taking the dynamic photo 11 shown as an example, the following Figure 2 and Figure 3 A human-computer interaction method provided in an embodiment of the present application is described, and the method can be used by a user to perform multiple exposure special effects editing on dynamic photos.

[0054] like Figure 2 As shown, the electronic device can display a user interface 21. The user interface 21 is a photo browsing interface of the gallery, in which a dynamic photo 11 is displayed, specifically a cover photo 11a of the dynamic photo 11. The user interface 21 may also include but is not limited to one or more of the following functional options: one or more of a sharing option 22a, a favorite option 22b, an editing option 22c, and a deletion option 22d. Among them, the sharing option 22a can be used by the user to forward the dynamic photo 11 to a friend or other device; the favorite option 22b can be used by the user to save or move the dynamic photo to a favorite folder; the editing option 22c can be used by the user to change the cover photo of the dynamic photo, etc.; the deletion option 22d can be used by the user to delete the dynamic photo 11 from the gallery.

[0055] In the embodiment of the present application, the user interface 21 may further include: a dynamic photo button 22e. The user can open some special effect editing options of dynamic photos by clicking the dynamic photo button 22e, such as loop playback, long exposure, and multiple exposure. Among them, loop playback can be used for users to convert dynamic photos into loop videos; long exposure can be used for users to obtain photos with long exposure effects. Moving objects in long exposure photos are generally blurred, such as fireworks across the night sky in dynamic photos leaving only a bright light trail, and waterfalls falling straight down become a white phantom; multiple exposure can be used for users to obtain photos with multiple exposure special effects. Figure 2 Just as an example, the special effects editing options of the dynamic photo can also be directly displayed in the user interface 21 like the sharing option 22a, the collection option 22b and other functional options, without the user having to click the dynamic photo button 22e. The special effects editing options of the dynamic photo can also only have multiple exposures.

[0056] The electronic device may detect a user operation on the special effect editing option "multiple exposure", such as a user operation of clicking "multiple exposure". In response to this, the electronic device may generate a multiple exposure photo 25 using the dynamic photo 11, and Figure 2 As shown, a multiple exposure photo 25 is shown. In the multiple exposure photo 25, the images of the multiple clones of the person 15 are from multiple frames of images in the short video of the dynamic photo 11, and the multiple frames of images respectively record the person 15 moving to different positions at multiple times.

[0057] Comparison Figure 2 It can be seen from the dynamic photo 11 on the left and the multiple exposure photo 25 on the right that multiple clones of the person 15 appear in the multiple exposure photo 25. These multiple clones are fixed at different positions. The paths formed by stringing together these different fixed positions are consistent with the movement path of the person 15 recorded in the dynamic photo 11, so that the user can see multiple clones moving to different positions at different times in the multiple exposure photo 25 at the same time, which is very creative. Moreover, the user can get a photo with multiple exposure effects through one editing operation (such as clicking "multiple exposure"), which is simple and efficient, and is conducive to stimulating the user's enthusiasm for creating with dynamic photos.

[0058] The multiple exposure special effect is different from the long exposure special effect. The moving subject (such as person 15) in the multiple exposure special effect is not blurred, but exists in the form of independent and distinct clones. It can present a completely different image experience from the long exposure special effect. The long exposure special effect is obtained by directly superimposing the pictures of multiple moments recorded in the dynamic photo, while the multiple exposure special effect in the embodiment of the present application is obtained by subject segmentation of multiple key frames in the dynamic photo and image fusion based on the subject segmentation results of these multiple key frames. Therefore, the multiple exposure special effect can present a different visual effect from the long exposure special effect.

[0059] Further, such as Figure 3 As shown, the electronic device can also detect user operations on the multiple exposure photos 25, such as long pressing the multiple exposure photos 25. In response to this, the electronic device can play a video 23, and more and more clones of the person 15 gradually appear in the playback screen when playing the video 23. The clones that appear one after another are frozen in the video screen one after another, and the appearance paths of these clones are consistent with the movement paths of the person 15 recorded in the dynamic photos 11. In this way, when the video 23 is played, a visual effect of a ghost clone walking will be presented, and the image expression is ingenious. Based on this, the visual effects presented by the video 23 can also be called the "Lingbo Weibu" effect.

[0060] In addition, in video 23, these clones show their own freeze-frame actions at their respective freeze-frame positions. The freeze-frame actions of these clones are coherent according to the order in which the clones appear, such as a coherent dunk action.

[0061] In specific implementation, like the multiple exposure photo 25 , the clone images in each video frame in the video 23 also come from the dynamic photo 11 . Figure 3 The dynamic playback process of video 23 is not presented, and only multiple video frames in video 23 are displayed in the playback order, such as video frame 23a, video frame 23b, and video frame 23c. The number of clones contained in each of these multiple video frames increases in accordance with their playback order, wherein the video frame in the later playback order includes the clones in the video frame in the earlier playback order, and the same clone in the video frame in the later playback order is aligned with the position in the video frame in the earlier playback order.

[0062] For example, Figure 3 As shown, there is one clone A in video frame 23a, two clones A and B in video frame 23b, and three clones A, B and C in video frame 23c. Among them, the clone A in video frame 23b is aligned with the clone A in video frame 23a, and the clones A and B in video frame 23c are aligned with the clones A and B in video frame 23b respectively. In this way, when playing video 23, the visual effect of clones A, B, and C appearing one after another and clones A, B, and C being frozen in different positions one after another can be presented.

[0063] Figure 1-Figure 3 It is only used to explain the embodiments of the present application and should not constitute a limitation. The moving subject in the dynamic photo can also be an animal, a sphere, a car, a ship, etc., not limited to a person. The photo editing method provided in the embodiment of the present application can also process such a dynamic photo to obtain a multiple exposure photo. The multiple exposure photo includes multiple clones of the moving subject, and these multiple clones are fixed in different positions, so that the user can see multiple subjects moving to different positions at different times in the multiple exposure photos at the same time, and the image is full of creativity.

[0064] The photo editing method provided in the embodiment of the present application can be based on Figure 4The image processing framework shown. This image processing framework can also be called an image processing framework for multiple exposure special effects. In this image processing framework, when shooting dynamic photos, the electronic device will perform subject detection, motion detection and other processing on the preview stream in real time, and save these detection results as auxiliary information for multiple exposure special effects processing; when performing multiple exposure special effects processing on dynamic photos, the electronic device can use this auxiliary information to select key frames in the video stream of the dynamic photos, and combine segmentation, fusion and other algorithms to achieve the visual effects of "multiple exposures" of the moving subject in the same scene, and generate a special effects video of "Light Steps on Water" accordingly, so as to achieve a single shot and generate special effects with one click.

[0065] like Figure 4 As shown, the framework may include the following steps:

[0066] When the dynamic photo mode is turned on, once the user triggers the photo shooting, the camera of the electronic device (such as a mobile phone) can start to collect and record the image signal to form the original image data (also called RAW data). At the same time, the camera can also perform subject detection and optical flow detection on the photo preview stream in real time to obtain the subject detection information and optical flow detection information of the photo preview stream. In addition, when collecting the photo preview stream, the electronic device can also detect the device motion in real time to obtain the device motion information. These detection results can constitute metadata and be used as input data for subsequent multiple exposure special effects processing.

[0067] Dynamic photo mode is a shooting mode. When the dynamic photo mode is turned on, an electronic device (such as a mobile phone) can obtain a special photo consisting of a high-definition cover photo and a low-definition short video through the subsequent two image processing branches 2.1 and 2.2.

[0068] 2.1. Generate high-definition cover photos

[0069] The electronic device can trigger a frame decision when the user takes a photo, that is, set different exposure parameters (such as ISO, exposure gain, exposure time, etc.) according to different photo scenes (such as portrait shooting scenes, dark night shooting scenes, outdoor shooting scenes, etc.) to output different exposure frames, and then perform multi-frame noise reduction and image fusion processing through algorithms such as AIRAW, and then combine the YUV domain post-processing algorithm to obtain a high-resolution cover photo. The embodiment of the present application does not limit the specific implementation of how to use the RAW image at the time of taking a photo to obtain a high-definition cover photo.

[0070] 2.2. Generate low-definition short videos

[0071] The electronic device can frame the original preview data within a few seconds (such as 1.5 seconds) before and after the user takes a photo in real time, and convert the original preview data of the digital signal into a preview stream in YUV format through algorithms such as raw2yuv to obtain a low-resolution video stream.

[0072] 3. Data storage

[0073] The electronic device can encapsulate the low-resolution video stream into a short video and store it in correspondence with the high-definition cover photo to complete the storage of the dynamic photo itself. In addition, the electronic device can also store the relevant metadata collected when the dynamic photo was taken (including but not limited to: subject detection information, device motion information, optical flow detection information, etc.) in the dynamic photo to provide auxiliary information for the subsequent special effects editing of the dynamic photo. Users can view the dynamic photos they have taken through the gallery.

[0074] 4. Gallery special effects editing

[0075] The electronic device can detect a user operation for triggering multiple exposure special effects editing, such as Figure 2 In response to the user operation, the electronic device may execute subsequent steps to complete the special effects editing of the dynamic photo.

[0076] 5. Special effects editing preprocessing

[0077] Electronic devices can parse dynamic photos and extract high-definition cover frames, low-definition short videos and related metadata.

[0078] 6. Multiple exposure special effects processing

[0079] The electronic device can load the corresponding input data and run the multiple exposure special effect algorithm to generate multiple exposure photos with multiple exposure special effects and videos with the “light footsteps on water” special effect. Figure 4 As shown in the figure, the operation of the multiple exposure special effects algorithm may include subject detection, registration alignment, subject segmentation, key frame selection, and multiple exposure fusion. Figure 5 The technical implementation here is explained in detail, but it will not be expanded here.

[0080] 7. Data storage and display

[0081] The electronic device can store the generated multiple exposure photos and their corresponding "Light Steps on Water" special effects videos, and can also be stored in the gallery path. In this way, the user can view the multiple exposure photos by entering the gallery, and long press the multiple exposure photos to play the "Light Steps on Water" special effects video.

[0082] Figure 4In the image processing framework shown, steps 1 to 3 are the image processing flow for obtaining dynamic photos, and steps 4 to 6 are the image processing flow for multiple exposure special effects editing. Steps 1 to 3 can be specifically performed by the image signal processor (ISP) of the camera, and steps 4 to 6 can be performed by a graphics processing unit (GPU) with stronger computing power. In practical applications, electronic devices can also obtain dynamic photos through other means, such as downloading dynamic photos from the Internet, obtaining dynamic photos shared by friends, etc., and the dynamic photos are not required to be taken by electronic devices. The dynamic photos in the embodiments of the present application may carry metadata such as subject detection information, device motion information, and optical flow detection information for multiple exposure special effects editing. Device motion information can also be called Gyro motion information, which can be detected by motion sensors such as gyroscopes.

[0083] Figure 5 The detailed process of multiple exposure special effects processing provided by the present application is shown below.

[0084] Dynamic photo shooting process:

[0085] The dynamic photo shooting process can be executed after the user triggers the dynamic photo shooting.

[0086] It mainly includes three branches: 1. When the user triggers the photo-taking, the frame decision is triggered according to the photo-taking scene, and multi-frame noise reduction and image fusion processing are performed through algorithms such as AIRAW, and then combined with the YUV domain post-processing algorithm to obtain a high-resolution cover photo; 2. The low-resolution preview stream of a few seconds before and after the photo is taken (such as 1.5 seconds) is saved as a short video; 3. Subject detection, device motion (Gyro) detection, key point detection, face detection, saliency detection, etc. are performed on this preview stream of a few seconds, and metadata such as subject detection information, device motion information, and optical flow detection information are obtained based on these detection results.

[0087] Therefore, each dynamic photo shooting will get a high-resolution (such as 4096*3072) cover frame and a short low-resolution (such as 1920*1440) video stream, as well as related metadata during the shooting process.

[0088] Multiple exposure special effects processing flow:

[0089] The multiple exposure special effects processing flow can be executed after the user triggers the multiple exposure special effects editing. The user operation that triggers the multiple exposure special effects editing can be, for example, Figure 2 The user operation of clicking the "Multi-Exposure" special effect editing option.

[0090] It may mainly include:

[0091] 1. Registration and Alignment

[0092] The electronic device can align the video frames in the low-resolution video stream in the dynamic photo based on the reference frame to eliminate video jitter and obtain a steady-state video stream. This process can also be called a video stabilization process. Figure 5 As shown, in addition to the low-resolution video stream, the input of the registration alignment may also include optical flow detection information of certain feature points, which can be used to track these feature points and determine the affine transformation matrix representing the inter-frame motion, and then use the radial transformation matrix to achieve image stabilization. Prior to this, the electronic device can first screen which video frames have excessive device jitter based on the Gyro motion information, and discard the video frames with severe jitter, and do not use it or them for subsequent special effects processing.

[0093] The optical flow detection information of some of the above-mentioned feature points can come from the metadata carried in the dynamic photo, and the optical flow detection information can be obtained by performing optical flow detection on the photo preview stream when taking dynamic photos. These feature points are feature points on static scenes, and the optical flow registration algorithm will filter out feature points on moving objects. The above-mentioned gyro motion information also comes from the metadata carried in the dynamic photo, and the gyro motion information can be obtained by detecting the device motion through motion sensors such as gyroscopes when taking dynamic photos.

[0094] The reference frame used for registration and alignment is a frame in the low-resolution video stream. It can be an intermediate video frame in the low-resolution video stream, or a video frame corresponding to the cover frame. The embodiment of the present application does not limit the selection strategy of the reference frame.

[0095] The resolution of the steady-state video stream will be slightly lower than the original low-resolution video stream in the dynamic photo, because the corners of the video frame will be cropped and some pixels will be lost during the video stabilization process.

[0096] 2. Subject Detection

[0097] The electronic device may utilize the metadata related to the low-resolution video stream, specifically the subject detection information therein, to further perform subject detection on the steady-state video stream.

[0098] 3. Keyframe selection

[0099] First, based on the above subject detection results, if multiple subjects are detected, the electronic device can first perform subject selection to select the target subject to be frozen from the multiple subjects. Other subjects will not be subject to subsequent subject segmentation.

[0100] The strategies selected by the subject may include, but are not limited to:

[0101] (1) Select subjects with large movements;

[0102] (2) Select the subject with the largest size;

[0103] (3) The number of selected subjects is less than or equal to 2;

[0104] (4) In the absence of user intervention, the subject selection priority can be: people > cats and dogs > other objects, where “>” indicates a higher priority; in actual applications, the subject selection priority can be different, and this is only used as an example.

[0105] The above subject selection strategies can be selected or adopted in combination or after slight modification according to actual application requirements.

[0106] Secondly, based on the subject detection result, the electronic device can further select multiple key frames from the steady-state video stream, such as selecting no more than 10 key frames. The selected multiple key frames can have the following characteristics: there is a moving subject in the key frame, and the positions of the moving subject in multiple key frames do not overlap with each other. In this way, multiple clones of the moving subject can appear in the multiple exposure photos, and these multiple clones are distinct from each other and do not overlap.

[0107] Here, the moving subject refers to the subject selected by the above subject selection, or a part of the subject. For example, the moving subject can be a whole person or a part of the human body. In a sports scene where the position of the whole person changes, such as a person running from the left side of the court to the right side, the moving subject is the whole person; in a sports scene where only a part of the human body changes position but the position of the whole person does not change or almost does not change, such as a person standing in place and waving his arms, the moving subject is the part of the human body, such as the waving arm. The description of this paragraph is also applicable to other objects (such as animals) that have both overall movement and local movement.

[0108] In order to select key frames with the above characteristics, the electronic device can:

[0109] (1) If the scene is one in which the subject moves as a whole, such as a person running from the left side of a court to the right side, multiple frames in which the subject does not overlap can be selected as key frames.

[0110] (2) If the scene is one in which only part of the subject moves, such as a person standing still and waving his arms, then video frames are selected at equal intervals as key frames.

[0111] 4. Subject segmentation

[0112] Electronic devices can perform optical flow calculations on these key frames, such as using an AI optical flow algorithm to estimate the optical flow and obtain the position of the moving subject in each key frame. Based on the position of the moving subject in each key frame and the subject detection information in the metadata, electronic devices can segment the subject of these multiple key frames and obtain the subject mask of each key frame.

[0113] 5. Multiple Exposure Fusion

[0114] Electronic devices can use these multiple key frames and their main mask images for superposition and fusion to obtain an initial cover frame with multiple exposure special effects and a video with a "light footsteps" special effect.

[0115] Figure 6A-6C The process of multiple exposure fusion is shown exemplarily.

[0116] like Fig. 6A As shown, key frame 1, key frame 2 and key frame 3, and the fusion mask Figure 3 (marked as "fusion mask3" in the figure) are superimposed and fused to obtain fusion frame 3, which is an example of the initial cover frame mentioned above. Fusion frame 3 includes the images of the main character 15 in key frame 1, key frame 2 and key frame 3 at the same time, thus presenting a multiple exposure effect of three clones of character 15 in fusion frame 3. "Fusion mask3" is used to constrain the freeze positions of these three clones in fusion frame 3. "Fusion mask3" can be obtained by fusing the main mask map of key frame 1, the main mask map of key frame 2 and the main mask map of key frame 3.

[0117] like Figure 6B As shown, key frame 1, key frame 2, and blend mask Figure 2 (marked as "fusion mask2" in the figure) are superimposed and fused to obtain fusion frame 2. Fusion frame 2 includes the images of the main person 15 in key frame 1 and key frame 2 at the same time, thus presenting a multiple exposure effect of two clones of person 15 in fusion frame 2. "Fusion mask2" is used to constrain the freeze positions of the two clones in fusion frame 2. "Fusion mask2" can be obtained by fusing the main mask map of key frame 1 with the main mask map of key frame 2.

[0118] like Figure 6C As shown, keyframe 1 and mask Figure 1 (marked as "mask1" in the figure) are superimposed and fused to obtain fused frame 1. Fused frame 1 includes the image of the subject 15 in key frame 1. "mask1" is used to constrain the fixed position of the subject image in fused frame 1. "mask1" is the subject mask image of key frame 1.

[0119] Further, such as Figure 7As shown, the electronic device can use these fused frames to generate a video 23 with a special effect of "light footsteps on water". These fused frames can be organized in the order of increasing number of avatars to form the video 23.

[0120] Before performing the above-mentioned superposition and fusion, the electronic device can also perform edge smoothing on the subject detection results of the multiple key frames to avoid the problem of abrupt and unnatural fusion edges. The electronic device can also set the transparency of the subject of certain key frames so that individual clones in the multiple exposure cover frame and / or the "Light Footsteps" special effects video present a certain degree of transparency. For example, the historical clone that appears first can be set to semi-transparent, and the clone that appears last can be set to opaque, so as to make a distinction.

[0121] 6. Super-resolution migration

[0122] The resolution of the initial cover frame with multiple exposure effects is low. To this end, the electronic device can use the high-definition cover frame in the dynamic photo to perform super-resolution detail migration on the initial cover frame with multiple exposure effects, so as to improve the clarity of the fusion result of the multiple exposure effects, and output a high-definition cover frame with multiple exposure effects. Specifically, the electronic device can first align the high-definition cover frame in the dynamic photo and the initial cover frame with multiple exposure effects, and then use the high-definition cover frame in the dynamic photo to convert the initial cover frame with multiple exposure effects into a high-definition multiple exposure photo. The method used for clarity reconstruction may, for example, be a reference-based image super-resolution (refSR) method, in which the reference image is a high-definition cover frame in the dynamic photo.

[0123] 7. Save the multiple exposure fusion results

[0124] Finally, the electronic device can encode and store the high-resolution cover frame with multiple exposure effects and the video with the "Light Footsteps" effect. Specifically, they can be placed in the storage path of the gallery, so that the user can view the multiple exposure photos and the video with the "Light Footsteps" effect when opening the gallery.

[0125] The multiple exposure photos and their corresponding "Light Footsteps" special effects video can be collectively referred to as multiple exposure fusion photos. The storage format of multiple exposure fusion photos can be similar to the storage format of dynamic photos, wherein the multiple exposure photos (stored by encoding high-resolution cover frames with multiple exposure special effects) and their corresponding "Light Footsteps" special effects video can have the same file name, such as boy.jpg and boy.mov, to form an association between the two.

[0126] In the embodiment of the present application, the moving subject during the multiple exposure special effects editing can be selected by the user, and is not limited to being automatically identified by the electronic device from the captured image based on the subject detection algorithm. Figure 2 In the example, after the user clicks on the "Multiple Exposure" special effect editing option, the electronic device may prompt the user to select the subject in the multiple exposure special effect in the user interface 21, record the subject selected by the user, and use the subject selected by the user for subject detection and subject segmentation when processing the multiple exposure special effects.

[0127] In the embodiment of the present application, the key frame can be selected by the user, so as to support freezing the wonderful moment that the user wants most. Figure 2 In the example, when the user clicks the special effect editing option "Multiple Exposure", the electronic device can display a short video sequence of the dynamic photo 11 in the user interface 21, such as Figure 1 In the video sequence 11b, the user can select several image frames from the video sequence as key frames.

[0128] In the embodiment of the present application, users can share multiple exposure fusion photos (including cover frames with multiple exposure special effects and videos with "light steps" special effects) with other devices. Users can also share dynamic photos with other devices. Since dynamic photos carry metadata, the recipient can edit dynamic photos with multiple exposure special effects on other devices. The recipient's device can support multiple exposure special effects editing by running the photo editing method provided in the embodiment of the present application.

[0129] In an embodiment of the present application, the multiple exposure special effects processing process can be run on a mobile phone or other end side, or on a cloud server with more powerful computing power. Users can choose to upload dynamic photos to a cloud server for multiple exposure special effects processing. Mobile phones and other electronic devices can also upload compressed raw images of dynamic photos, metadata of dynamic photos, etc. to the cloud server to improve the image processing quality with the help of more powerful computing power in the cloud. For example, deblurring the subject in a steady-state video stream, using a more powerful optical flow detection model and subject segmentation model to achieve more accurate subject segmentation, and using an image restoration algorithm (such as an Inpainting model) to smooth specific objects in multiple exposure cover frames to further enhance the specific object (such as the subject, obscured background, etc.).

[0130] In the embodiment of the present application, the moving subject in the dynamic photo may refer to a subject that moves relative to the camera of the electronic device, and is not limited to an absolute motion in the earth coordinate system. The moving subject may remain absolutely still in the earth coordinate system, but it moves relative to the camera of the electronic device.

[0131] Based on the human-computer interaction method for multiple exposure special effects editing, the image processing framework for multiple exposure special effects, and the detailed process of multiple exposure special effects processing introduced above, the photo editing method provided in the embodiments of the present application will be summarized below.

[0132] Figure 8 The overall process of the photo editing method provided by the embodiment of the present application is shown below.

[0133] S11. The electronic device displays the dynamic photo.

[0134] The electronic device may be a mobile phone, a tablet computer, a smart screen, or other device with a display. The subsequent embodiments will describe the composition of the electronic device in detail, which will not be expanded here. The electronic device may display the dynamic photo based on the user operation of opening the dynamic photo. The user operation of opening the dynamic photo may be a user operation of clicking on the thumbnail of the dynamic photo in the gallery. The embodiments of the present application do not limit the triggering conditions for the electronic device to display the dynamic photo.

[0135] For how to display dynamic photos, please refer to the previous article. Figure 2 The relevant description will not be repeated here.

[0136] The dynamic photo can be composed of a cover photo and a first video, wherein the first video is composed of a preview stream of photos taken a few seconds before and after the first photo moment. The first photo moment can be the moment when the dynamic photo is triggered (such as the moment when the shutter is pressed). The cover photo is high-definition, while the resolution of the first video is generally low. For a detailed description of dynamic photos, please refer to the previous article in conjunction with Figure 1 The relevant description will not be repeated here.

[0137] Before S11, the dynamic photo can be taken by an electronic device. When taking the dynamic photo, the electronic device can also perform real-time detection such as subject detection and optical flow detection on the photo preview stream, and save these detection results as metadata carried in the dynamic photo. In this way, when the user triggers the multiple exposure special effects editing of the dynamic photo, the electronic device can directly extract metadata from the dynamic photo and use the metadata to perform multiple exposure special effects processing, which improves the efficiency of multiple exposure special effects image processing and reduces the delay in image output.

[0138] S12. The electronic device detects the first editing operation on the photo in this state.

[0139] The first editing operation can be used to trigger the multiple exposure special effects editing of the dynamic photo. It can be, for example, clicking Figure 2The user operation of the special effect editing option "Multiple Exposure" in the user interface 21 is specifically described in the previous text, which will not be expanded here. The embodiment of the present application does not display the form of the first editing operation, and the first editing operation can even be an editing operation in the form of a voice command.

[0140] S13. In response to the first editing operation, the electronic device generates a multiple exposure photo using the dynamic photo.

[0141] S14. The electronic device displays the multiple exposure photo.

[0142] The electronic device may display the multiple exposure photo after generating the multiple exposure photo. The electronic device may also save the multiple exposure photo after generating the multiple exposure photo, such as saving it to a gallery, and then display the multiple exposure photo when a user operation of viewing the multiple exposure photo (such as clicking on its thumbnail in the gallery) is detected later. The embodiment of the present application does not limit the triggering conditions for displaying the multiple exposure photo.

[0143] The multiple exposure photo may include multiple clones of the first moving subject in the dynamic photo, and the multiple clones are frozen at different positions, and the path formed by stringing together the different positions is consistent with the movement path of the first moving subject recorded in the dynamic photo.

[0144] So, like Figure 2 As shown, users can simultaneously see multiple clones moving to different positions at different times in the multiple exposure photos, which is very creative. Moreover, users can obtain photos with multiple exposure effects through one editing operation (such as the first editing operation), which is simple and efficient, and is conducive to stimulating users' enthusiasm for creating with dynamic photos.

[0145] In the embodiment of the present application, the clone in the multiple exposure photo is relative to the subject in the dynamic photo, and the clone can be regarded as the reproduction or clone of the moving subject in the dynamic photo in the multiple exposure photo.

[0146] The clarity of these clones in the multiple exposure photos is consistent or close to the clarity of the first moving subject in the dynamic photo. Close means that the difference in clarity is small and does not exceed a specific threshold. The setting of the specific threshold may depend on which algorithm is used to evaluate clarity. In other words, the clones are clear and distinct in the multiple exposure photos, not blurred, and the multiple exposure photos can present a visual effect that is completely different from the long exposure effect, that is, the multiple exposure effect mentioned above.

[0147] In addition, in order to present the same shooting scene, the background in the multiple exposure photos is the same as the background of the cover photo in the dynamic photo.

[0148] Multiple exposure photos are created using dynamic photos, which will be explained next.

[0149] like Fig. 6A As shown, in the multiple exposure photos, the images of the multiple clones (such as clones A, B, and C) can come from the images of the first moving subject (such as person 15) in multiple key frames (key frame 1, key frame 2, and key frame 3) of the dynamic photo. These multiple key frames are video frames in the low-resolution video (first video) in the dynamic photo. These multiple key frames can be selected from the first video, and have the following characteristics: there is a first moving subject in the key frames, and the positions of the first moving subject in these multiple key frames do not overlap with each other. In this way, it can be ensured that there are multiple clones in the multiple exposure photos, and these multiple clones are independent and distinct from each other.

[0150] The frozen position of each clone in the multiple exposure photo is aligned with the position of the first moving subject in a key frame. Fig. 6A As shown, in the multiple exposure photos, the frozen position of avatar A in the multiple exposure photos is aligned with the position of the main person 15 in key frame 1, the frozen position of avatar B in the multiple exposure photos is aligned with the position of the main person 15 in key frame 2, and the frozen position of avatar C in the multiple exposure photos is aligned with the position of the main person 15 in key frame 3. In this way, the frozen positions of these multiple avatars can be strung together to reproduce the motion trajectory of the subject in the dynamic photo, forming a strong connection between the two, showing a very creative image expression.

[0151] The key frame may be a key video frame selected by the electronic device from the first video of the dynamic photo. This will be expanded below. Of course, the key frame may also be selected by the user to support the presentation of the wonderful moments that the user wants in the multiple exposure photos.

[0152] In an embodiment of the present application, the dynamic photo may carry metadata such as subject detection information and optical flow detection information for use in multiple exposure special effects processing to generate multiple exposure photos based on the dynamic photo.

[0153] Reference to the previous article Figure 4 and Figure 5 The specific implementation of the electronic device generating multiple exposure photos using dynamic photos may include the following steps:

[0154] Step 1. Select multiple key frames from the first video (ie, the low-resolution video) in the dynamic photo.

[0155] Specifically, the electronic device may select the multiple key frames from the first video using the subject detection information in the metadata.

[0156] The subject detection information in the metadata can be obtained by performing subject detection on the photo preview stream when taking dynamic photos, and it can be used to indicate whether there is a subject in each video frame of the first video and where the subject is located. In this way, step 2. subject segmentation is performed on the key frame to determine the subject outline of the key frame.

[0157] Specifically, the electronic device can perform optical flow calculation on multiple key frames to obtain the position of the first moving subject in the multiple key frames and the motion trajectory of the first moving subject. Then, the electronic device can perform subject segmentation on the multiple key frames based on the optical flow calculation results and the subject detection information in the metadata to obtain the outline of the first moving subject in each key frame. The outline can be represented by a mask.

[0158] Step 3. The subject contours of multiple key frames are fused to obtain a first fused contour, where the first fused contour includes contours of multiple avatars in the dynamic photo.

[0159] The first fused contour can be Fig. 6A As shown in “Fusion mask3” in .

[0160] Step 4. Fusion multiple key frames according to the first fusion profile to generate an initial multiple exposure photo.

[0161] When fusing multiple keyframes to generate an initial multiple exposure photo, the first fused contour can be used to constrain the freeze positions of the multiple clones in the multiple exposure photo. For details on the implementation of step 4, please refer to Figure 5 The relevant contents of “multiple exposure fusion” in the embodiment will not be repeated here.

[0162] Step 5. Use the cover photo in the dynamic photo to convert the initial multiple exposure photo into a higher resolution multiple exposure photo.

[0163] The resolution of the initial multiple exposure photo is low. The electronic device can use the high-definition cover photo in the dynamic photo to perform super-resolution detail migration on the initial multiple exposure photo with multiple exposure special effects, so as to improve the clarity of the fusion result of the multiple exposure special effects and output a high-definition multiple exposure photo.

[0164] For step 5, please refer to Figure 5 The relevant contents of “super-resolution migration” in the embodiments will not be repeated here.

[0165] In an embodiment of the present application, before selecting multiple key frames from the first video in the dynamic photo, the electronic device can also align the video frames in the first video based on the reference frame to eliminate video jitter and obtain a steady-state video stream. In a specific implementation, the electronic device can perform the alignment based on the optical flow detection information in the metadata, which records the position of specific feature points in each video frame of the first video. In this way, the electronic device can perform the alignment with reference to the position of the specific feature point in the reference frame.

[0166] For other details on how to generate multiple exposure photos based on dynamic photos, you can also refer to the previous article combined with Figure 4 and Figure 5 The relevant content will not be repeated here.

[0167] Furthermore, the electronic device may also detect a first user operation on the multiple exposure photo. Figure 3 As described in the related content of , the first user operation may include a user operation of long pressing the multiple exposure photo. In response to the first user operation, the electronic device may play a second video, which may be, for example, Figure 3 Video 23 in the video is the “Lingbo Weibu” special effects video.

[0168] When the second video is played, more and more clones of the first moving subject gradually appear in the video screen, and the clones appear one after another and are frozen in the video screen one after another. Moreover, the appearance paths of these clones are consistent with the movement path of the first moving subject recorded in the dynamic photo. In this way, when the second video is played, a visual effect of ghost clones walking will be presented, which can also be called the "light footsteps" effect, and the image expression is unique.

[0169] In addition, the clarity of the clone in the second video can be consistent or close to the clarity of the first moving subject in the first video of the dynamic photo.

[0170] Like the multiple exposure photo, the second video is also generated using the dynamic photo.

[0171] In the second video, the images of the multiple clones are respectively from the images of the first moving subject in multiple key frames. These multiple key frames are key video frames in the low-resolution video (first video) in the dynamic photo. These multiple key frames can be selected from the first video, and have the following characteristics: there is a first moving subject in the key frames, and the positions of the first moving subject in these multiple key frames do not overlap with each other. In this way, it can be ensured that the clones are frozen in the second video playback screen, and these multiple clones are independent and distinct from each other.

[0172] like Figure 3As shown, the second video frame (such as video 23) may include multiple video frames, such as video frames 23a, 23b and 23c, and the number of clones contained in each of the multiple video frames is increasing according to the playback order of the multiple video frames, wherein the video frame in the later playback order includes the clone in the video frame in the earlier playback order, and the same clone in the video frame in the later playback order is aligned with the position of the video frame in the earlier playback order.

[0173] In addition, in order to present the same shooting scene, the background of the second video is the same as that of the first video. Specifically, the background of the first video frame in the second video is the same as the background of the second video frame in the first video frame, and the first video frame and the second video frame may have the following connection: the frozen position of the last avatar in the first video frame is the same as the position of the first moving subject in the second video frame.

[0174] For example, Figure 3 and Figure 1 As shown, the first video frame may be Figure 3 In the example video frame 23c of the video 23, the second video frame may be Figure 1 In the example, in video frame 16c of video 11b, the frozen position of the last avatar C in video frame 23c is the same as the position of the main character 15 in video frame 16c. In this way, the background of video frame 23c can adopt the background of video frame 16c, and the two backgrounds are the same.

[0175] For example, Figure 3 and Figure 1 As shown, the first video frame may be Figure 3 In the example video frame 23b of the video 23, the second video frame may be Figure 1 In the example, in video frame 16b of video 11b, the frozen position of the last avatar B in video frame 23b is the same as the position of the main character 15 in video frame 16b. In this way, the background of video frame 23b can adopt the background of video frame 16b, and the two backgrounds are the same.

[0176] The examples are only used to explain the embodiments of the present application and should not be construed as limiting.

[0177] After generating a multiple exposure photo and a second video with the “Lingbo Weibu” special effect based on the dynamic photo, the electronic device can associate the two together and save them as a multiple exposure fusion photo. The multiple exposure fusion photo consists of a high-definition multiple exposure photo and a video with the “Lingbo Weibu” special effect. Specifically, the electronic device can save the multiple exposure fusion photo to the gallery. In this way, the user can view the high-definition multiple exposure photos in the multiple exposure fusion photos when opening the gallery, and can further view the “Lingbo Weibu” special effects video by long pressing the multiple exposure photo to trigger the playback of the “Lingbo Weibu” video. In a specific implementation, the electronic device can associate and save the multiple exposure photos and the second video with the same file name, which is similar to the associated saving of high-definition cover photos and low-definition short videos in dynamic photos.

[0178] Fig. 9 An electronic device 300 provided in an embodiment of the present application is exemplarily shown.

[0179] The electronic device 300 may have both human-computer interaction capability and computing capability. The device type of the electronic device 300 may be any of a mobile phone, a tablet computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), and smart home devices such as smart large screens, wearable devices such as smart watches and smart glasses, extended reality (XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), in-vehicle devices or smart city devices, etc.

[0180] like Fig. 9As shown, the electronic device 300 may include: a processor 110, a memory 120, a display 130, a display driver integrated circuit (DDIC) 140, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a gyroscope sensor 180B, an acceleration sensor 180E, and a touch sensor 180K, etc. The various parts in the electronic device 300 may be connected through a bus.

[0181] Among them, the processor 110 can be responsible for providing computing power and can be used as a computing module of the electronic device 300; the display 130, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193 and other input and output components can be responsible for providing human-computer interaction capabilities and can be used as a human-computer interaction module of the electronic device 300. When the computing module in the electronic device 300 has strong computing power, the electronic device 300 can independently perform the interface display method provided in the embodiment of the present application. When the computing module in the electronic device 300 does not have strong computing power, the electronic device 300 can also only perform the human-computer interaction steps in the interface display method provided in the embodiment of the present application, such as displaying the first user interface, moving the first UI element on the wallpaper image, and other steps, and the image processing steps such as foreground patching and layered rendering in the method can be performed by the cloud side server.

[0182] The processor 110 may be one or more, and they may be integrated into an integrated circuit of a system on chip (SOC). SOC is a system-level chip. The processor 110 may include a central processing unit (CPU), a graphic processing unit (GPU), a neural-network processing unit (NPU), etc. Among them, the CPU may include an application processor (AP), a baseband processor chip (BP), etc., wherein the AP may be responsible for running the operating system, user interface, and application program on the electronic device 300; the BP may be responsible for receiving and sending wireless signals and managing radio frequency services. The GPU may be responsible for graphics rendering, coloring according to the rendering instructions and data from the CPU, filling, rendering, and outputting materials, etc. The NPU quickly processes input information by drawing on the biological neural network structure, such as the transmission mode between neurons in the human brain, and can also continuously self-learn. The NPU can be used to run artificial intelligence algorithms, such as editing instruction recommendation algorithms, image processing algorithms, image understanding algorithms, etc. The CPU and GPU can be used to render and synthesize the images to be sent to the display 130.

[0183] The processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input and output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0184] The processor 110 may be provided with a cache memory for storing instructions or data just used or cyclically used by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the cache memory, which can reduce the waiting time of the processor 110 and improve the program running efficiency.

[0185] The memory 120 may include a program storage area and a user data storage area, wherein the program storage area may store an operating system and one or more application programs (such as game applications), and the data storage area may store data (such as photos, contacts) created by the user when using the electronic device 300. The memory 120 may be a high-speed random access memory or a non-volatile memory, such as a disk, a flash memory, a universal flash storage (UFS), etc. The memory 120 may also be an external memory card, such as a Micro SD card.

[0186] The memory 120 may also store code instructions of the image editing method provided in the embodiment of the present application. When the processor 110 reads the code instructions from the memory 120 and runs the code instructions, the electronic device 300 can execute the steps performed by the human-computer interaction module and / or the computing module in the image editing method provided in the embodiment of the present application.

[0187] The memory 120 may also be integrated with the processor 110 in an integrated circuit of a SOC.

[0188] like Fig. 9 As shown, the electronic device 300 can realize the display function through the SOC, the DDIC 140, and the display 130, etc.

[0189] The display 130 has multiple refresh rates. The refresh rate indicates the number of times the display screen refreshes the display screen in 1 second. For example, a 60 Hz refresh rate indicates that the display screen refreshes the display screen 60 times in 1 second. The display 130 may use an LTPO display panel, allowing the refresh rate to be reduced to a low refresh rate, such as 10 Hz or 1 Hz, thereby supporting the reduction of the power consumption of the display screen.

[0190] Among them, the display driver integrated circuit (DDIC) 140 can be used as the control core of the display 130, driving the display 130 to work and receiving data from the SOC (processor 110), such as image data and some instructions. The DDIC 140 can send driving signals and data to the display panel of the display 130 in the form of electrical signals, and then realize the control of the screen brightness and color, so that image information such as letters and pictures can be displayed on the screen, completing the screen refresh.

[0191] The image data of the screen to be displayed sent by the SOC to the DDIC 140 can be sent to the frame buffer for storage to complete the display (or image sending). Then, the DDIC 140 takes out the image data from the frame buffer and drives the display 130 to display it.

[0192] The wireless communication function of the electronic device 300 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0193] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 300 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0194] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 300. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0195] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display 130. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0196] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 300. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0197] In some embodiments, the antenna 1 of the electronic device 300 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 300 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0198] The electronic device 300 can realize the shooting function through ISP, camera 193, video codec, GPU, display 130 and application processor.

[0199] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0200] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 300 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0201] Video codecs are used to compress or decompress digital videos. The electronic device 300 may support one or more video codecs. Thus, the electronic device 300 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0202] The electronic device 300 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0203] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0204] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 300 can listen to music or listen to a hands-free call through the speaker 170A.

[0205] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 300 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0206] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 300 can be provided with at least one microphone 170C. In other embodiments, the electronic device 300 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 300 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0207] The earphone interface 170D is used to connect a wired earphone and can be a USB interface, or a 3.5 mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0208] The button 190 includes a power button, a volume button, etc. The button 190 can be a mechanical button or a touch button. The electronic device 300 can receive the button input and generate a key signal input related to the user settings and function control of the electronic device 300. The motor 191 can generate a vibration prompt. The SIM card interface 195 is used to connect the SIM card. The SIM card can be connected to and separated from the electronic device 300 by inserting the SIM card interface 195 or pulling it out from the SIM card interface 195.

[0209] Fig. 9 The illustrated structure does not constitute a specific limitation on the electronic device 300. The electronic device 300 may include more or fewer parts than shown in the figure, or combine some parts, or split some parts, or arrange the parts differently. The various parts shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0210] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the steps performed by the human-computer interaction module in the above-mentioned method embodiments, or the steps performed by the human-computer interaction module and the computing module.

[0211] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps performed by the computing module in the above-mentioned method embodiments can be implemented.

[0212] An embodiment of the present application also provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps performed by the human-computer interaction module in the above-mentioned various method embodiments, or the steps performed by the human-computer interaction module and the computing module.

[0213] The embodiment of the present application also provides a computer program product. When the computer program product is run on a server, the server can implement the steps performed by the computing module in the above-mentioned various method embodiments.

[0214] The embodiment of the present application also provides a chip system, which includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the steps performed by the human-computer interaction module in any method embodiment of the present application, or the steps performed by the human-computer interaction module and the computing module. The chip system can be a single chip, or a chip module composed of multiple chips.

[0215] The embodiment of the present application also provides a chip system, which includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the steps performed by the computing module in any method embodiment of the present application. The chip system can be a single chip or a chip module composed of multiple chips.

[0216] The term "user interface (UI), or interface for short) in the specification and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. A control, also called a widget, is a basic element of a user interface. Typical controls include a toolbar, menu bar, text box, button, scrollbar, picture and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through

[0217] <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as <head> and <head>. A node corresponds to a control or attribute in the interface. After parsing and rendering, the node is presented as user-visible content. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code. For example, HTML is defined by <head> and <body>. 、 、 <video> 、

[0218] <canvas>To define the elements and attributes of a web page.

[0219] The most common form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operation that is displayed in a graphical manner. It can be an icon, window, control or other interface element displayed on the display screen of an electronic device, where a control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.

[0220] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0221] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0222] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A photo editing method, characterized in that: include: The electronic device displays a dynamic photo, the dynamic photo consisting of a cover photo and a first video, the first video consisting of a photo preview stream several seconds before and after a first photo moment, the first photo moment being a moment that triggers the taking of the dynamic photo; the resolution of the cover photo is greater than the resolution of the first video; The electronic device detects a first editing operation on the dynamic photo; The electronic device generates a multiple exposure photo using the dynamic photo in response to the first editing operation; The electronic device displays the multiple exposure photo; the multiple exposure photo includes multiple clones of the first moving subject in the dynamic photo, the multiple clones are frozen at different positions, and the path strung together by the different positions is consistent with the movement path of the first moving subject recorded in the dynamic photo.

2. The method according to claim 1, characterized in that The clarity of the clone in the multiple-exposure photo is consistent with or close to the clarity of the first moving subject in the dynamic photo.

3. The method according to any one of claims 1 to 2, characterized in that: In the multiple exposure photos, the images of the multiple clones are respectively from the images of the first moving subject in multiple key frames, and the multiple key frames are video frames in the first video.

4. The method according to any one of claims 1 to 3, characterized in that The background in the multiple-exposure photo is the same as the background of the cover photo in the dynamic photo.

5. The method according to any one of claims 1 to 4, characterized in that After displaying the multiple exposure photos, the method further includes: The electronic device detects a first user operation on the multiple exposure photo; the first user operation includes a user operation of long pressing the multiple exposure photo; The electronic device plays a second video in response to the first user operation, where the second video is generated using the dynamic photo; wherein, when the second video is played, more and more clones of the first moving subject gradually appear in the video screen, and the clones that appear one after another are successively frozen in the video screen, and the appearance paths of the clones are consistent with the movement path of the first moving subject recorded in the dynamic photo.

6. The method according to claim 5, characterized in that The clarity of the avatar in the second video is consistent with or close to the clarity of the first moving subject in the first video.

7. The method according to claim 5 or 6, wherein in the second video, the images of the multiple avatars are respectively from the images of the first moving subject in multiple key frames, and the multiple key frames are video frames in the first video.

8. The method according to claim 7, wherein the second video frame comprises a plurality of video frames, and the number of clones contained in each of the plurality of video frames increases in the order in which the plurality of video frames are played, wherein: The video frame in the later playback order includes the clone in the video frame in the earlier playback order, and the same clone in the video frame in the later playback order is aligned with the position in the video frame in the earlier playback order.

9. The method according to claim 3 or 7, characterized in that: The key frame contains the first moving subject, and positions of the first moving subject in the plurality of key frames do not overlap with each other.

10. The method according to any one of claims 1 to 9, characterized in that The dynamic photo also carries the first metadata, which includes subject detection information and optical flow detection information. The subject detection information is obtained by performing subject detection on the photo preview stream when taking the dynamic photo, and the first optical flow detection information is obtained by performing optical flow detection on the photo preview stream when taking the dynamic photo.

11. The method according to claim 10, characterized in that The step of generating the multiple exposure photo by using the dynamic photo includes: Based on the optical flow detection information in the first metadata, the video frames in the first video are aligned based on the reference frame to obtain a steady-state video stream; performing subject detection on the steady-state video stream based on the subject detection information in the first metadata; selecting the plurality of key frames from the steady-state video stream according to a subject detection result of the steady-state video stream; Performing subject segmentation on the key frame to determine the subject outline of the key frame; Fusing the subject contours of the multiple key frames to obtain a first fused contour, wherein the first fused contour includes contours of the multiple avatars in the dynamic photo; fusing the plurality of key frames according to the first fusion profile to generate an initial multiple exposure photo; The initial multiple-exposure photo is converted into a higher-definition multiple-exposure photo using the cover photo in the dynamic photo.

12. The method according to claim 11, characterized in that The reference frame belongs to the first video, and the reference frame includes an intermediate video frame in the first video.

13. The method according to any one of claims 5 to 8, characterized in that The background of the first video frame in the second video is the same as the background of the second video frame in the first video frame, and the frozen position of the last avatar in the first video frame is the same as the position of the first moving subject in the second video frame.

14. The method according to any one of claims 5 to 8, characterized in that Also includes: The electronic device associates the multiple exposure photos with the second video and saves them in the gallery; The associated saving includes: the file name of the multiple exposure photo is the same as that of the second video.

15. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 14.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

Images