Dynamic photo editing method and device, electronic equipment and storage medium

Abstract

The invention discloses a dynamic photo editing method and device and electronic equipment, and belongs to the technical field of artificial intelligence. The method comprises the steps that an editing interface of a dynamic photo is displayed, the dynamic photo comprises a video, the editing interface comprises a video frame display area, and the video frame display area comprises a first video frame of the video; the first video frame comprises at least one video object; in response to an elimination object selection input for the first video frame, eliminating an elimination object selected by the elimination object selection input in each video frame of the video; wherein the elimination object comprises at least one video object.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a dynamic photo editing method, device, electronic device and storage medium. Background Art

[0002] Currently, with the development of electronic devices, the functions of electronic devices are also increasing. For example, the electronic device can eliminate the shooting object selected by the user in the video.

[0003] In the related art, an electronic device can display multiple video frames of a video based on user input. Then, if the user needs to eliminate the photographic object contained in each of the multiple video frames, such as a person or an object, the user can select the photographic object in the first video frame of the multiple video frames, so that the electronic device can eliminate the photographic object in the first video frame; then, the user can select the photographic object again in the second video frame of the multiple video frames, so that the electronic device can eliminate the photographic object in the second video frame, and so on, until the user selects the photographic object in all video frames, so that the electronic device can eliminate the photographic object contained in each video frame of all video frames to obtain the video required by the user.

[0004] However, in the above method, since the electronic device requires the user to select a shooting object in each of the multiple video frames, the process of the electronic device determining the shooting object in the multiple video frames is cumbersome and time-consuming. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a dynamic photo editing method, device, electronic device and storage medium, which can improve the efficiency of electronic devices in eliminating the shooting objects in the video.

[0006] In a first aspect, an embodiment of the present application provides a dynamic photo editing method, which includes: displaying an editing interface for dynamic photos, the dynamic photos include videos, the editing interface includes a video frame display area, the video frame display area includes a first video frame of the video; the first video frame includes at least one video object; in response to an elimination object selection input for the first video frame, eliminating the elimination object selected by the elimination object selection input in each video frame of the video; wherein the elimination object includes at least one video object.

[0007] In some embodiments of the present application, the above-mentioned editing interface also includes a thumbnail picture display area, the thumbnail picture display area includes thumbnail pictures corresponding to at least two video frames of the video; the thumbnail picture display area includes a first thumbnail picture, the first thumbnail picture is a thumbnail picture corresponding to the first video frame; the first thumbnail picture also includes a thumbnail picture identifier, the thumbnail picture identifier is used to indicate the display position of the thumbnail picture corresponding to the video frame displayed in the video frame display area.

[0008] In some embodiments of the present application, the dynamic photo editing method provided by the embodiments of the present application also includes: displaying a first prompt information, wherein the first prompt information is used to prompt the user to select an object to be eliminated.

[0009] In some embodiments of the present application, the above-mentioned editing interface includes a first control; in response to the elimination object selection input of the first video frame, before eliminating the elimination object selected by the elimination object selection input in each video frame of the video, a first input to the first control is received; in response to the first input, the dynamic photo elimination function is started, and thumbnail images and thumbnail image identifiers of at least two video frames of the video are displayed in the thumbnail image display area, and a third control is displayed on the editing interface.

[0010] In some embodiments of the present application, after receiving the first input to the first control, it also includes: displaying a prompt window, the prompt window including second prompt information and a second control, the second prompt information being used to prompt that the dynamic photo elimination function has been activated; in response to the second input to the second control, closing the prompt window.

[0011] In some embodiments of the present application, when the second video frame of a video is displayed in the video frame display area and the thumbnail picture identifier is displayed on the second thumbnail picture, a selection input for a first thumbnail picture among at least two thumbnail pictures is received; in response to the selection input, the thumbnail picture identifier is moved to the first thumbnail picture, and the second video frame displayed in the video frame display area is updated to the first video frame; wherein the second thumbnail picture is the thumbnail picture corresponding to the second video frame.

[0012] In some embodiments of the present application, the elimination object selection input includes a selection input for a first video object in a first video frame; upon receiving the elimination object selection input for the first video object in the first video frame, a third prompt message is displayed, and the third prompt message is used to prompt the user to continue selecting the elimination object; upon receiving the elimination object selection input for the second video object in the first video frame, the first video object and the second video object are determined as elimination objects.

[0013] In some embodiments of the present application, when an object elimination selection input is received for a first video object in a first video frame, a first object elimination identifier is displayed on the first video object, and the first object elimination identifier is used to indicate the object outline of the first video object; when an object elimination selection input is received for a second video object in the first video frame, a second object elimination identifier is displayed on the second video object, and the second object elimination identifier is used to indicate the object outline of the second video object.

[0014] In some embodiments of the present application, a video includes N video frames, where N is an integer greater than 1; before selecting and inputting the selected elimination object in each video frame of the elimination video, a mask image corresponding to the first video frame is obtained based on the first video frame; based on the mask image corresponding to the first video frame and the image feature information of the elimination object in the first video frame, a display area of ​​the elimination object in each of the M video frames of the video is identified, and an elimination object identifier is displayed on the elimination object in each of the M video frames; wherein M is a positive integer, M≤N-1; each of the M video frames includes the elimination object.

[0015] In some embodiments of the present application, the elimination object is a first video object in a video; based on the first video frame, a mask image corresponding to the first video frame is obtained, including: determining the object contour of the first video object from the first video frame based on the input trajectory of the elimination object selection input; performing image segmentation on the first video frame based on the object contour of the first video object; and performing binarization processing on the first video frame after image segmentation to obtain a mask image corresponding to the first video frame.

[0016] In some embodiments of the present application, the elimination objects include K video objects, where K is an integer greater than 1; based on the first video frame, a first mask image corresponding to the first video frame is obtained, including: based on the K video objects, performing image segmentation processing on the first video frame to obtain K mask images corresponding to the K video objects; and merging the K mask images to obtain a mask image corresponding to the first video frame.

[0017] In some embodiments of the present application, based on the first mask image corresponding to the first video frame and the image feature information of the eliminated object in the first video frame, the display area of ​​the eliminated object in each video frame of the M video frames of the video is determined, including: performing panoramic segmentation processing on each video frame of the M video frames to obtain a mask image corresponding to each video object in each video frame and category information of each mask image, the category information is used to characterize the object category of the eliminated object in each video frame; based on the image feature information of the eliminated object in the first video frame and the category information of the mask image corresponding to the first video frame, comparing the image similarity between the mask image corresponding to the first video frame and the mask image corresponding to each video object in each video frame, and determining the mask image corresponding to each video frame of the M video frames; based on the mask image corresponding to each video frame of the M video frames, determining the display area of ​​the eliminated object in each video frame of the M video frames.

[0018] In some embodiments of the present application, the editing interface also includes a thumbnail image display area, and the thumbnail image display area includes thumbnail images corresponding to at least two video frames of the video; the first thumbnail image also includes a thumbnail image identifier; the dynamic photo editing method provided by the embodiment of the present application also includes: in the process of eliminating the display area of ​​the object in each of the M video frames of the identified video, displaying a fourth prompt information, the fourth prompt information is used to prompt the identification progress of eliminating the display area of ​​the object in each of the M video frames of the identified video; in the case of eliminating the display area of ​​the object in the i-th video frame of the identified video, moving the thumbnail image identifier to the i-th thumbnail image, and updating the first video frame displayed in the video frame display area to the i-th video frame; wherein the i-th thumbnail image is the thumbnail image corresponding to the i-th video frame, i is a positive integer, i≤M.

[0019] In some embodiments of the present application, eliminating the elimination object selected by the elimination object selection input in each video frame of the video includes: eliminating the elimination object selected by the elimination object selection input in each video frame of the video in response to a third input to a third control.

[0020] In some embodiments of the present application, the dynamic photo editing method provided by the embodiments of the present application also includes: in the process of eliminating the object selection input selected for elimination in each video frame of the elimination video, displaying the elimination effect of the elimination object.

[0021] In some embodiments of the present application, the editing interface also includes a fourth control; in the process of eliminating the object selected by the elimination object selection input in each video frame of the video, in response to the fourth input to the fourth control, the processing of eliminating the object selected by the elimination object selection input in each video frame of the video is executed in the background, and the editing interface is updated to a preview interface of the dynamic photo.

[0022] In some embodiments of the present application, in the process of selecting and inputting the selected elimination object in each video frame of the elimination video, a fifth prompt information is displayed, and the fifth prompt information is used to prompt the elimination progress of the elimination object; in the case of eliminating the elimination object in the i-th video frame of the elimination video, the elimination mark is moved to the i-th thumbnail picture; wherein the i-th thumbnail picture is the thumbnail picture corresponding to the i-th video frame, i is a positive integer, i≤M.

[0023] In some embodiments of the present application, the elimination of the elimination object selection input selected elimination object in each video frame of the video includes: adding preset noise in the first video frame to obtain a first noisy video frame; based on the first noisy video frame, the mask image corresponding to the first video frame, the video background image corresponding to the first video frame, the positive sample vector, the negative sample vector, and the prediction time step, predicting the image noise of the object area of ​​the elimination object in the first noisy video frame to obtain a second noisy video frame, the positive sample vector is used to indicate that the video object contained in the mask image corresponding to the first video frame is a shooting foreground, and the negative sample vector is used to characterize that the video object contained in the video background image corresponding to the first video frame is a shooting background, The background image of the video is an image that does not include the elimination object selected by the elimination object selection input; the second noisy video frame is denoised to obtain a third video frame corresponding to the first video frame; based on the mask image corresponding to the first video frame, the image to be filled is determined, and the image to be filled is an image of an area in the third video frame corresponding to the background image area in the mask image of the first video frame, and the background image area is an image area corresponding to the elimination object selected by the elimination object selection input in the mask image of the first video frame; the image to be filled is filled into the video background image corresponding to the first video frame, and the elimination object selected by the elimination object selection input in the first video frame is eliminated, so as to eliminate the elimination object selected by the elimination object selection input in each video frame of the video.

[0024] In some embodiments of the present application, the dynamic photo editing method provided by the embodiments of the present application also includes: obtaining a positive sample vector, negative sample data and a first training image; randomly determining a first training video object from the first training image, and obtaining a first training mask image corresponding to the first training image based on the first training video object; marking the image category corresponding to the first training mask image based on the image features, positive sample vectors, and negative sample vectors of the first training video object; the image category is any one of the shooting foreground or shooting background; inputting the first training mask image, the noisy training image corresponding to the first training image, and the training background image corresponding to the first training image into the initial model; determining the image category corresponding to the first training mask image through the cross-attention unit in the initial model; predicting the training image to be filled corresponding to the first training mask image based on the image category, the first training mask image, the noisy training image corresponding to the first training image, and the training background image corresponding to the first training image, and adding the training image to be filled to the training background image to obtain a second training image; calculating the loss value between the second training image and the reference image, and adjusting the model parameters of the original model according to the loss value to obtain a video elimination model.

[0025] In some embodiments of the present application, the object elimination selection input includes at least one of the following: fingertip touch input and knuckle touch input.

[0026] In some embodiments of the present application, when the elimination object selection input includes a knuckle touch input, the knuckle touch input includes an input of tapping the editing interface with the knuckle and circling an area along a reference direction from the tapping position.

[0027] In some embodiments of the present application, after eliminating the elimination object selected by the elimination object selection input in each video frame of the video, the thumbnail images corresponding to at least two video frames of the video are hidden, and a fifth control and a sixth control are displayed in the editing interface; when the fifth input of the fifth control is received, the thumbnail images and thumbnail image identifiers of at least two video frames of the video are displayed in the thumbnail image display area, and the third control is displayed in the editing interface; when the sixth input of the third control is received, the elimination object selected by the elimination object selection input in each video frame of the video is eliminated; when the seventh input of the sixth control is received, the video after elimination processing is stored.

[0028] In a second aspect, embodiments of the present application provide a dynamic image editing device, comprising: a display module configured to display an editing interface for a dynamic image, wherein the dynamic image includes a video, the editing interface includes a video frame display area, the video frame display area includes a first video frame of the video, the first video frame including at least one video object; and an elimination module configured to, in response to an elimination object selection input for the first video frame, eliminate an elimination object selected by the elimination object selection input in each video frame of the video, wherein the elimination object includes at least one video object.

[0029] In some embodiments of the present application, the editing interface also includes a thumbnail picture display area, the thumbnail picture display area includes thumbnail pictures corresponding to at least two video frames of the video; the thumbnail picture display area includes a first thumbnail picture, the first thumbnail picture is a thumbnail picture corresponding to the first video frame; the first thumbnail picture also includes a thumbnail picture identifier, the thumbnail picture identifier is used to indicate the display position of the thumbnail picture corresponding to the video frame displayed in the video frame display area.

[0030] In some embodiments of the present application, the above-mentioned display module is also used to display a first prompt information, and the first prompt information is used to prompt the user to select an object to be eliminated.

[0031] In some embodiments of the present application, the editing interface includes a first control; the dynamic photo editing device further includes: a receiving module configured to receive a first input to the first control in response to an input for selecting an object to be removed from the first video frame, before removing the object selected by the input for removing from each video frame of the video. A processing module configured to activate a dynamic photo removal function in response to the first input received by the receiving module. The display module further configured to display thumbnail images and thumbnail image identifiers of at least two video frames of the video in a thumbnail display area, and to display a third control on the editing interface.

[0032] In some embodiments of the present application, the display module is further configured to, after the receiving module receives a first input to the first control, display a prompt window including a second prompt message and a second control, wherein the second prompt message indicates that the dynamic photo removal function has been activated. The processing module is further configured to close the prompt window in response to a second input to the second control.

[0033] In some embodiments of the present application, the receiving module is further configured to receive a selection input for a first thumbnail image from the at least two thumbnail images when the second video frame of the video is displayed in the video frame display area and the thumbnail image identifier is displayed on the second thumbnail image. The processing module is further configured to, in response to the selection input received by the receiving module, move the thumbnail image identifier to the first thumbnail image and update the second video frame displayed in the video frame display area to the first video frame; wherein the second thumbnail image is the thumbnail image corresponding to the second video frame.

[0034] In some embodiments of the present application, the input for selecting an object to be eliminated includes an input for selecting a first video object in a first video frame; and the dynamic photo editing device further includes a determination module. The display module is further configured to, upon receiving the input for selecting the first video object to be eliminated in the first video frame, display a third prompt message, the third prompt message prompting the user to continue selecting an object to be eliminated. The determination module is configured to, upon receiving the input for selecting a second video object to be eliminated in the first video frame, determine the first video object and the second video object as objects to be eliminated.

[0035] In some embodiments of the present application, the above-mentioned display module is also used to display a first elimination object identifier on the first video object when receiving an elimination object selection input for the first video object in the first video frame, and the first elimination object identifier is used to indicate the object outline of the first video object; and to display a second elimination object identifier on the second video object when receiving an elimination object selection input for the second video object in the first video frame, and the second elimination object identifier is used to indicate the object outline of the second video object.

[0036] In some embodiments of the present application, the video includes N video frames, where N is an integer greater than 1; the dynamic photo editing device also includes: a processing module, which is used to obtain a mask image corresponding to the first video frame based on the first video frame before the elimination module eliminates the elimination object in each video frame of the video and selects the elimination object selected by input; and based on the mask image corresponding to the first video frame and the image feature information of the elimination object in the first video frame, identify the display area of ​​the elimination object in each video frame of the M video frames of the video, and display an elimination object identifier on the elimination object in each video frame of the M video frames; wherein M is a positive integer, M≤N-1; each video frame of the M video frames includes the elimination object.

[0037] In some embodiments of the present application, the elimination object is a first video object in the video; the processing module is specifically used to determine the object contour of the first video object from the first video frame based on the input trajectory of the elimination object selection input; and based on the object contour of the first video object, perform image segmentation on the first video frame; and perform binarization processing on the first video frame after image segmentation to obtain a mask image corresponding to the first video frame.

[0038] In some embodiments of the present application, the elimination objects include K video objects, where K is an integer greater than 1; the above-mentioned processing module is specifically used to perform image segmentation processing on the first video frame based on the K video objects to obtain K mask images corresponding to the K video objects; and merge the K mask images to obtain a mask image corresponding to the first video frame.

[0039] In some embodiments of the present application, the above-mentioned processing module is specifically used to: perform panoramic segmentation processing on each video frame of M video frames to obtain a mask image corresponding to each video object in each video frame and category information of each mask image, where the category information is used to characterize the object category of the eliminated object in each video frame; based on the image feature information of the eliminated object in the first video frame and the category information of the mask image corresponding to the first video frame, compare the image similarity between the mask image corresponding to the first video frame and the mask image corresponding to each video object in each video frame, and determine the mask image corresponding to each video frame of the M video frames; based on the mask image corresponding to each video frame of the M video frames, determine the display area of ​​the eliminated object in each video frame of the M video frames.

[0040] In some embodiments of the present application, the editing interface further includes a thumbnail image display area, the thumbnail image display area includes thumbnail images corresponding to at least two video frames of the video; the first thumbnail image also includes a thumbnail image identifier; the display module is further used to display a fourth prompt message during the process of identifying the display area of ​​the object to be eliminated in each of the M video frames of the video, the fourth prompt message being used to prompt the identification progress of the display area of ​​the object to be eliminated in each of the M video frames of the video. The processing module is further used to move the thumbnail image identifier to the i-th thumbnail image when the display area of ​​the object is eliminated in the i-th video frame of the identified video, and update the first video frame displayed in the video frame display area to the i-th video frame; wherein the i-th thumbnail image is the thumbnail image corresponding to the i-th video frame, i is a positive integer, and i≤M.

[0041] In some embodiments of the present application, the elimination module is specifically configured to eliminate the elimination object selected by the elimination object selection input in each video frame of the video in response to a third input to the third control.

[0042] In some embodiments of the present application, the dynamic photo editing device further includes: a processing module for displaying an elimination animation of the elimination object during the process of eliminating the object selection input selected for elimination in each video frame of the elimination video.

[0043] In some embodiments of the present application, the above-mentioned editing interface also includes a fourth control; the dynamic photo editing device also includes: a processing module for performing processing of eliminating the object selected by the elimination object selection input in each video frame of the elimination video in response to the fourth input to the fourth control in the background, and updating the editing interface to a preview interface of the dynamic photo.

[0044] In some embodiments of the present application, the display module is further configured to display a fifth prompt message during the process of selecting and inputting the selected object to be eliminated in each video frame of the video, the fifth prompt message being used to indicate the progress of the object elimination. The processing module is further configured to move the elimination marker to the i-th thumbnail image when eliminating the object to be eliminated in the i-th video frame of the video; wherein the i-th thumbnail image is the thumbnail image corresponding to the i-th video frame, where i is a positive integer and i≤M.

[0045] In some embodiments of the present application, the above-mentioned elimination module is specifically used to: add preset noise to the first video frame to obtain a first noisy video frame; based on the first noisy video frame, the mask image corresponding to the first video frame, the video background image corresponding to the first video frame, the positive sample vector, the negative sample vector, and the prediction time step, predict the image noise of the object area of ​​the elimination object in the first noisy video frame to obtain a second noisy video frame, the positive sample vector is used to indicate that the video object contained in the mask image corresponding to the first video frame is the shooting foreground, the negative sample vector is used to characterize that the video object contained in the video background image corresponding to the first video frame is the shooting background, and the video background image does not include the elimination object. The method comprises the following steps: first, performing denoising on the second noisy video frame to obtain a third video frame corresponding to the first video frame; determining an image to be filled based on the mask image corresponding to the first video frame, the image to be filled being an image of an area in the third video frame corresponding to the background image area in the mask image of the first video frame, the background image area being an image area corresponding to the object to be eliminated selected by the object to be eliminated selection input in the mask image of the first video frame; filling the image to be filled into the video background image corresponding to the first video frame, eliminating the object to be eliminated selected by the object to be eliminated selection input in the first video frame, so as to eliminate the object to be eliminated selected by the object to be eliminated selection input in each video frame of the video.

[0046] In some embodiments of the present application, the above-mentioned processing module is also used to: obtain a positive sample vector, negative sample data and a first training image; randomly determine a first training video object from the first training image, and obtain a first training mask image corresponding to the first training image based on the first training video object; mark the image category corresponding to the first training mask image based on the image features, positive sample vector and negative sample vector of the first training video object; the image category is any one of the shooting foreground or shooting background; input the first training mask image, the noisy training image corresponding to the first training image, and the training background image corresponding to the first training image into the initial model; determine the image category corresponding to the first training mask image through the cross-attention unit in the initial model; predict the training image to be filled corresponding to the first training mask image based on the image category, the first training mask image, the noisy training image corresponding to the first training image and the training background image corresponding to the first training image, and add the training image to be filled to the training background image to obtain a second training image; calculate the loss value between the second training image and the reference image, and adjust the model parameters of the original model according to the loss value to obtain a video elimination model.

[0047] In some embodiments of the present application, the object elimination selection input includes at least one of the following: fingertip touch input and knuckle touch input.

[0048] In some embodiments of the present application, when the elimination object selection input includes a knuckle touch input, the knuckle touch input includes an input of tapping the editing interface with the knuckle and circling an area along a reference direction from the tapping position.

[0049] In some embodiments of the present application, the above-mentioned processing module is also used to hide the thumbnail images corresponding to at least two video frames of the video after eliminating the elimination object selected by the elimination object selection input in each video frame of the video, and display the fifth control and the sixth control in the editing interface; when the fifth input of the fifth control is received, the thumbnail images and thumbnail image identifiers of at least two video frames of the video are displayed in the thumbnail image display area, and the third control is displayed in the editing interface; when the sixth input of the third control is received, the elimination object selected by the elimination object selection input in each video frame of the video is eliminated; when the seventh input of the sixth control is received, the video after elimination processing is stored.

[0050] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0051] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0052] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0053] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the method described in the first aspect.

[0054] In an embodiment of the present application, an editing interface for a dynamic photo is displayed, the dynamic photo includes a video, the editing interface includes a video frame display area, the video frame display area includes a first video frame of the video; the first video frame includes at least one video object; in response to an elimination object selection input for the first video frame, the elimination object selected by the elimination object selection input is eliminated from each video frame of the video; wherein the elimination object includes at least one video object. In this solution, since the electronic device can automatically determine the elimination object selected by the selection input from each video frame of the video through the user's elimination object selection input for the first video frame, the user does not need to select the elimination object in each video frame of the video in turn, which simplifies the steps for the user to select the elimination object in the video, thereby improving the efficiency of the electronic device in eliminating the elimination object in the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic flow chart of a dynamic photo editing method provided in some embodiments of the present application;

[0056] Figure 2A Schematic diagram of an example of an album interface provided in some embodiments of the present application;

[0057] Figure 2B Schematic diagram of an example image editing interface provided by some embodiments of the present application;

[0058] Figure 3A Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0059] Figure 3B Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0060] Figure 4A Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0061] Figure 4B Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0062] Figure 5A Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0063] Figure 5B Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0064] Figure 6 Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0065] Figure 7 Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0066] Figure 8A Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0067] Figure 8B Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0068] Figure 8C Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0069] Figure 9 Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0070] Figure 10 Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0071] Figure 11 Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0072] Figure 12 Schematic diagram of mask image examples provided for some embodiments of the present application;

[0073] Figure 13 Schematic diagram of mask image examples provided for some embodiments of the present application;

[0074] Figure 14A Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0075] Figure 14B Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0076] Figure 15 Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0077] Figure 16 Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0078] Figure 17A Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0079] Figure 17B Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0080] Figure 18 Schematic diagram of an example of a noisy image provided in some embodiments of the present application;

[0081] Figure 19A schematic diagram of the structure of an elimination model provided in some embodiments of the present application;

[0082] Figure 20 Schematic diagram of an example of an image elimination interface provided by some embodiments of the present application;

[0083] Figure 21 A schematic flow chart of a dynamic photo editing method provided in some embodiments of the present application;

[0084] Figure 22 is a schematic structural diagram of a dynamic photo editing device provided by some embodiments of the present application;

[0085] Figure 23 is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application;

[0086] Figure 24 This is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0087] The following will be combined with the accompanying drawings in the embodiments of this application to clearly describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0088] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0089] The terms "at least one" and "at least one of" in the specification and claims of this application refer to any one, any two, or a combination of more than two of the objects included. For example, at least one of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c", and "a, b, and c", where a, b, and c can be single or multiple. Similarly, "at least two" means two or more, and its meaning is similar to "at least one".

[0090] The terms used in the embodiments of this application are only used to explain the specific embodiments of this application and are not intended to limit this application. The following is a detailed explanation of the professional terms provided in the embodiments of this application.

[0091] Generative Adversarial Network (GAN) Model: A generative adversarial network (GAN) is a deep learning model consisting of a generator network and a discriminator network. The generator network generates new data, while the discriminator network attempts to distinguish between generated data and real data. During training, the two networks compete with each other, ultimately enabling the generator network to generate new data that is similar to the training data.

[0092] Diffusion Model: Diffusion Models is a type of generative model based on deep learning.

[0093] The core idea of ​​the diffusion model is to simulate the physical diffusion process, gradually converting data into noise, and then learning the inverse process to gradually recover the original data from the noise, thereby achieving high-quality generation effects. The diffusion model has achieved remarkable results in many fields such as image generation, speech synthesis, and text generation, bringing new breakthroughs to the development of artificial intelligence technology. This generative model based on the diffusion process can generate high-quality images based on text descriptions. Its core idea is to generate images that match the text description from random noise through a gradual denoising method. The diffusion process is divided into two stages:

[0094] Forward Process: Gradually add noise to real data (such as images) until the data becomes completely random noise. This process is fixed and usually controlled by a predefined noise schedule.

[0095] Reverse Process: Starting with random noise, the process gradually removes the noise to ultimately recover the true data. The reverse process is learned by a neural network, with the goal of predicting the noise at each step and gradually removing it.

[0096] The relationship between text description and image in the diffusion model: The diffusion model learns the correspondence between images and text quality inspection. When the user enters a text description, the model can generate the corresponding text.

[0097] 3D Variational Autoencoder (VAE): A generative model for processing 3D data, using an encoder and decoder to achieve data compression and generation. In video generation, it effectively models spatiotemporal information and generates high-quality video sequences. It combines 3D convolution with a variational autoencoder (VAE). The encoder compresses input data (e.g., video frames) into a low-dimensional representation in a latent space using 3D convolutional layers; the decoder restores the latent representation to the original data using 3D deconvolutional layers. In this paper, the encoder compresses a video with F frames, H width, W height, and 3 channels into a low-dimensional vector with f frames, h width, w height, and c channels, denoted as the video latent, with a shape of [f, h, w, c]. The decoder then restores the low-dimensional vector (latent) back to a 3-channel video with H width, W height, and F frames. Here, p=F / f and q=H / h=W / w are the compression ratios of the encoder on the temporal and spatial dimensions of the video, respectively.

[0098] Resize: Based on OpenCV's common image resizing operations, you can enlarge and reduce images.

[0099] MLP: (Multi-layer Perceptron) is mainly used for feature transformation and nonlinear enhancement.

[0100] FFN: (Feed-Forward Network, FFN) is one of the core components of the Transformer module, responsible for further nonlinear transformation and enhancement of the features output by the self-attention mechanism. FFN is essentially a multi-layer perceptron (MLP). FFN usually consists of two fully connected layers (linear layers) and an activation function. Its structure can be expressed as follows:

[0101] FFN(x)=W2·Activation(W1·x+b1)+b2 (1)

[0102] Where x is the input feature (usually the output of the self-attention mechanism). W1 and b1 are the weights and bias of the first linear layer. W2 and b2 are the weights and bias of the second linear layer. Activation is the activation function, typically GELU (Gaussian Error Linear Unit) or ReLU.

[0103] Self-Attention: Self-Attention is the core mechanism of the Transformer architecture, which is used to capture the dependencies between different positions in the input sequence.

[0104] The input feature X generates three matrices: Query (Q), Key (K) and Value (V) through linear transformation: Q = XW Q ,K=XW K ,V=XW V .

[0105] The attention scores can be calculated using the following formula (2):

[0106]

[0107] Where dk is the dimension of the key, which is used to scale the dot product result to prevent vanishing or exploding gradients. The softmax function normalizes the attention scores into a probability distribution, representing the degree to which each position pays attention to other positions. The final output is a weighted sum of the value matrix, with the weights determined by the attention scores.

[0108] Cross-Attention: Cross-Attention is an extension of Self-Attention, used to handle relationships between two different input sequences. Given two input sequences X and Y, it generates a query (Q) and a key-value pair (K, V): Q = XW_Q, K = YW_K, V = YW_V. The remaining steps are similar to those for Self-Attention.

[0109] Text-Embedding: The process of converting textual information (such as category labels or descriptive text) into numerical vectors that models can understand and process. This is primarily achieved by using pre-trained language models (such as CLIP and BERT) to encode text into fixed-length vectors. In conditional diffusion models, text-embedding serves as conditional information, guiding the model to generate images that match the text description.

[0110] Timestep: In a diffusion model, this variable represents the progress of the diffusion process, often used to control the addition or removal of noise. The diffusion process is typically divided into multiple timesteps (e.g., 1000), with each timestep corresponding to a specific noise level. In the forward pass, the image is gradually corrupted by noise until it becomes completely random. In the reverse pass, the model gradually recovers the original image from the noise.

[0111] QK-Norm: QK-Norm is a technique for normalizing the query and key in Self-Attention, aiming to improve training stability and performance.

[0112] Gate: A gating mechanism used to dynamically control the output of the attention mechanism and enhance the model's expressiveness

[0113] Schedule: A strategy used in a diffusion model to control the process of adding or removing noise, usually involving changes in the noise level. In the forward process, the schedule defines the noise level (such as variance) at each time step. Common schedules include linear scheduling and cosine scheduling, which determine the dynamic characteristics of the diffusion process and affect the training and generation effects of the model.

[0114] MSE Loss: (Mean Squared Error Loss) is a commonly used indicator to measure the difference between the predicted value and the true value.

[0115] Video elimination: refers to removing unwanted content from videos, such as advertisements, bad language, people, animals, etc., to obtain a more refined video.

[0116] Dimension: Dimension refers to the number of aspects or attributes contained in a thing or concept.

[0117] Binarization: Binarization is to set the grayscale value of the pixels on the image to 0 or 255, which means that the entire image will appear in obvious black and white.

[0118] Mask image: A mask image refers to a picture that represents the image to be retained with a color, which is generally a color that is not in the original image.

[0119] Weight: Weight is a relative concept, referring to a certain indicator. The weight of an indicator refers to its relative importance in the overall evaluation.

[0120] Cinematic Photos: Cinematic Photos combine a static image with a brief dynamic effect, providing a more vivid visual experience. When the shutter is pressed, a Cinematic Photo captures not only the still image at the moment, but also the dynamic images and sounds from a few seconds before and after the shutter is clicked, forming a short video clip (usually 1-10 seconds). When viewing, users can trigger the dynamic effect by touching or long-pressing the photo, bringing it to life.

[0121] The logo in this application refers to text, symbols, images, etc. used to indicate information, and controls or other containers can be used as carriers for displaying information, including but not limited to text logos, symbol logos, and image logos.

[0122] Below, in conjunction with the accompanying drawings, the dynamic photo method, device, electronic device and storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0123] With the explosive growth of internet video content and increasing user demand for video editing and post-processing, traditional manual video editing methods are becoming increasingly inadequate, especially when dealing with complex video scenes, where efficiency and effectiveness often fall short of expectations. To address this, video de-scrambling technology has emerged, leveraging artificial intelligence (AI) technologies such as deep learning and computer vision to make object removal and repair more efficient and automated.

[0124] In recent years, breakthroughs in technologies such as GANs and diffusion models have provided strong technical support for video deconstruction. Trained with extensive data, these models are able to understand and predict the texture, shape, and motion within video frames. This allows them to accurately fill in background information while removing objects, ensuring smooth and natural transitions. Furthermore, advances in video deconstruction technology in handling dynamic backgrounds, complex lighting changes, and multi-object occlusions have given it broad application prospects across multiple fields.

[0125] Specifically, video de-masking technology has a wide range of applications, including film and television post-production, advertising video editing, personal video editing, virtual reality content creation, and privacy protection. For example, in film production, producers can remove certain unnecessary objects without affecting the integrity of the background; in privacy protection, it can remove elements in the video that may reveal privacy, thereby protecting personal information to the greatest extent possible.

[0126] For example, the dynamic photo editing method, device, electronic device, and storage medium provided in the embodiments of the present application can be used to eliminate people, objects, or animals in dynamic photos, such as scene 1 and scene 2.

[0127] Scenario 1: Taking a mobile phone as an example, if a user needs to remove an elephant from a dynamic photo, the user can select a dynamic photo including an elephant from the photo album application, and select the elephant as the removal object from each video frame in the dynamic photo. Then, the user can trigger the electronic device to remove the elephant from the dynamic photo.

[0128] Scenario 2: If the user needs to remove the elephant and hot air balloon from the dynamic photo, the user can select the dynamic photo including the elephant and hot air balloon from the album application, and select the elephant and hot air balloon as the removal objects from each video frame in the dynamic photo. Then the user can trigger the electronic device to remove the elephant and hot air balloon from the dynamic photo.

[0129] The execution subject of the dynamic photo editing method provided in the embodiment of the present application can be an image elimination device, which can be an electronic device or a functional module in an electronic device. The following uses an electronic device as an example to illustrate the technical solution provided in the embodiment of the present application.

[0130] The present invention provides a method for editing dynamic photos. Figure 1 FIG. 1 shows a flow chart of a dynamic photo editing method provided by an embodiment of the present application. Figure 1 As shown, the dynamic photo editing method provided in the embodiment of the present application may include the following steps 201 and 202.

[0131] Step 201: The electronic device displays an editing interface for dynamic photos.

[0132] In some embodiments of the present application, the dynamic photo includes a video, the editing interface includes a video frame display area, and the video frame display area includes a first video frame of the video; the first video frame includes at least one video object.

[0133] In some embodiments of the present application, the dynamic photos may be dynamic photos stored in an album application; or the dynamic photos may be dynamic photos downloaded through a browser application. The specific details can be determined based on actual usage requirements and are not limited by the embodiments of the present application.

[0134] Exemplarily, the dynamic photos may be Live Photos.

[0135] In some embodiments of the present application, a user may long-press a dynamic photo stored in an album application so that the electronic device may display an editing interface corresponding to the dynamic photo.

[0136] In some embodiments of the present application, the above-mentioned editing interface also includes: a control display area, and the control display area includes: artificial intelligence (AI) elimination control, AI image expansion control, AI ultra-clear control and effect master control.

[0137] Among them, the AI ​​elimination control is used to eliminate the objects that users need to eliminate in dynamic photos, the AI ​​image expansion control is used to expand the image content in dynamic photos, the AI ​​ultra-clear control is used to change the resolution of dynamic photos, and the Effect Master control is used to add image effects to dynamic photos.

[0138] In some embodiments of the present application, the above-mentioned image effects include at least one of the following: filter, white balance, graffiti, mosaic, etc. The specific effect can be determined according to actual use requirements and is not limited by the embodiments of the present application.

[0139] In some embodiments of the present application, the first video frame may be any video frame in a video.

[0140] Exemplarily, the first video frame may be the first video frame in a video.

[0141] In some embodiments of the present application, the video object may be any of the following: a person, an animal, or an object.

[0142] For example, taking the electronic device as a mobile phone as an example, Figure 2A As shown, the mobile phone can use the user to Figure 2A Take the color dynamic photo 50 in the example, long press input, such as Figure 2B As shown, an editing interface 10 for a dynamic photo is displayed, and the video frame display area of ​​the editing interface 10 displays the first video frame 12 of the dynamic photo 50, and the first video frame 12 includes an elephant 40, a hot air balloon 41 and a little boy 42. The control display area of ​​the editing interface 10 includes: an AI elimination control 14, an AI image expansion control 15, an AI ultra-clear control 16 and an effect master control 17; wherein, the AI ​​elimination control 14 is used to eliminate the video object selected by the user in the dynamic photo, the AI ​​image expansion control 15 is used to enlarge the image size of the dynamic photo, the AI ​​ultra-clear control 16 is used to adjust the clarity of the dynamic photo to ultra-clear, and the effect master control 17 is used to add image effects to the dynamic photo, such as filters, mosaics, etc.

[0143] In some embodiments of the present application, the above-mentioned editing interface also includes a thumbnail image display area, which includes thumbnail images corresponding to at least two video frames of the video; the thumbnail image display area includes a first thumbnail image, which is a thumbnail image corresponding to the first video frame; the first thumbnail image also includes a thumbnail image identifier, which is used to indicate the display position of the thumbnail image corresponding to the video frame displayed in the video frame display area.

[0144] In some embodiments of the present application, the thumbnail image identifier may be any of the following: a text identifier, a pattern identifier, a symbol identifier, a color identifier, etc. The specific identifier may be determined according to actual use requirements and is not limited in the embodiments of the present application.

[0145] In some embodiments of the present application, the editing interface includes a first control.

[0146] In some embodiments of the present application, the first control may be a control in a control display area in an editing interface.

[0147] Exemplarily, the first control may be the AI ​​elimination control.

[0148] Exemplarily, the dynamic photo editing method provided in the embodiment of the present application also includes the following steps 301 and 302.

[0149] Step 301: The electronic device receives a first input to a first control.

[0150] In some embodiments of the present application, the first input is used to trigger the electronic device to activate a dynamic photo removal function.

[0151] In some embodiments of the present application, the above-mentioned first input can be a click input of the user on the above-mentioned first control through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible input. The specific input can be determined according to actual usage requirements and is not limited in the embodiments of the present application.

[0152] In some embodiments of the present application, the above-mentioned specific gesture can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture.

[0153] In some embodiments of the present application, the click input may be a single click input, a double click input, or any number of click inputs, and may also be a long press input or a short press input. For example, the first input may be a single click input of the user on the first control.

[0154] Step 302: The electronic device activates a dynamic photo removal function in response to the first input, displays thumbnail images and thumbnail image identifiers of at least two video frames of the video in the thumbnail image display area, and displays a third control in the editing interface.

[0155] In some embodiments of the present application, the electronic device can update the editing interface according to the first input, and display the video frame display area and the thumbnail image display area in the updated editing interface.

[0156] Exemplarily, the third control may be a “start eliminating” control.

[0157] Exemplarily, the third control is used to control the electronic device to start eliminating the elimination objects in all video frames.

[0158] For example, in combination Figure 2B ,like Figure 3A As shown, the user can click and input the AI ​​elimination control 14 in the editing interface 10 so that the electronic device can start the dynamic photo elimination function, such as Figure 3B As shown, an updated editing interface 20 is displayed, in which the first video frame 12 is displayed in the video frame display area of ​​the updated editing interface 20, the thumbnail image display area of ​​the updated editing interface 20 includes thumbnail images of 5 video frames, the thumbnail images of the 5 video frames include the thumbnail image 13 corresponding to the first video frame 12, and a "Start Elimination" control is displayed in the updated editing interface 20.

[0159] In some embodiments of the present application, the dynamic picture editing method provided by the embodiments of the present application further includes the following step 401.

[0160] Step 401: The electronic device receives an elimination object selection input for an elimination object in a first video frame.

[0161] In some embodiments of the present application, the above-mentioned elimination object selection input is used to determine the elimination object from the first video frame.

[0162] In some embodiments of the present application, the elimination object may be any of the following: a person, an animal, or an object in the first video frame, etc. The specific elimination object may be determined according to actual use requirements and is not limited in the present embodiment.

[0163] In some embodiments of the present application, the elimination object may be one or more objects, which may be determined based on actual use requirements and are not limited by the embodiments of the present application.

[0164] In some embodiments of the present application, the above-mentioned elimination object selection input includes at least one of the following: fingertip touch input, knuckle touch input.

[0165] In some embodiments of the present application, when the elimination object selection input includes a knuckle touch input, the knuckle touch input includes an input of tapping the editing interface with the knuckle and circling an area along a reference direction from the tapping position.

[0166] Step 202: In response to the elimination object selection input for the first video frame, the electronic device eliminates the elimination object selected by the elimination object selection input in each video frame of the video.

[0167] In the embodiment of the present application, the above-mentioned elimination object includes at least one video object.

[0168] In some embodiments of the present application, the video object may be a plurality of video objects in one video frame; or, the video object may be a plurality of video objects in different video frames.

[0169] In some embodiments of the present application, the electronic device may automatically determine the elimination object from each video frame of the video based on the elimination object selection input of the first video frame, and eliminate the elimination object in each video frame of the video.

[0170] For example, in combination with the above scenario 1, Figure 3B ,like Figure 4A As shown, the user can circle the elephant in the first video frame 12 and input 21 to determine the elimination area 22, so that the mobile phone can determine the elephant as the elimination target and automatically determine the elephants in other videos of the video as elimination targets. Then, as shown in FIG. Figure 4BAs shown in the figure, the mobile phone can eliminate the elephants in all video frames of the video and obtain the video after elimination. Figure 4B Here, only one video frame is used as an example.

[0171] For example, in combination with the above scenario 1, Figure 3B ,like Figure 5A As shown, the user can circle the elephant and the hot air balloon in the first video frame 12 and input 23 to determine the elimination area 24, so that the mobile phone can determine the elephant and the hot air balloon as the elimination objects, and automatically determine the elephants and hot air balloons in other videos of the video as the elimination objects. Then, as shown in FIG. Figure 5B As shown in the figure, the mobile phone can eliminate the elephant and the hot air balloon in all the video frames of the video and obtain the video after elimination. Figure 5B Here, only one video frame is used as an example.

[0172] In some embodiments of the present application, after eliminating the elimination object in all video frames of the video, the electronic device can update the preview image of the dynamic photo according to the content of the updated video frame.

[0173] For example, before the electronic device performs the removal process on the video, the preview image of the dynamic photo includes an elephant and a hot air balloon. If the electronic device performs the removal process on the elephant in the video, the preview image of the dynamic photo does not include the elephant.

[0174] As another example, before the electronic device removes the elephant and the hot air balloon from the video, the preview image of the dynamic photo includes the elephant and the hot air balloon. If the electronic device removes the elephant and the hot air balloon from the video, the preview image of the dynamic photo does not include the elephant and the hot air balloon.

[0175] It should be noted that the specific process of the electronic device eliminating the elimination object in all video frames of the video can be found in the following embodiments, and will not be described again here to avoid repetition.

[0176] An embodiment of the present application provides a method for editing a dynamic photo, in which an electronic device can display an editing interface for a dynamic photo, wherein the dynamic photo includes a video, the editing interface includes a video frame display area, and the video frame display area includes a first video frame of the video; the first video frame includes at least one video object; in response to an elimination object selection input for the first video frame, the elimination object selected by the elimination object selection input is eliminated from each video frame of the video; wherein the elimination object includes at least one video object. In this solution, since the electronic device can automatically determine the elimination object selected by the selection input from each video frame of the video through the user's elimination object selection input for the first video frame, the user does not need to select the elimination object in each video frame of the video in turn, which simplifies the steps for the user to select the elimination object in the video, thereby improving the efficiency of the electronic device in eliminating the elimination object in the video.

[0177] In some embodiments of the present application, the dynamic photo editing method provided by the embodiments of the present application also includes the following step 501.

[0178] Step 501: The electronic device displays a first prompt message.

[0179] In the embodiment of the present application, the first prompt information is used to prompt the user to select an object to be eliminated.

[0180] In some embodiments of the present application, the electronic device may display the first prompt information through a pop-up window in the above-mentioned editing interface; or, the electronic device may display the first prompt information in a blank area in the above-mentioned editing interface.

[0181] It should be noted that the blank area mentioned above may be an area in the editing interface that does not include controls and video frames.

[0182] For example, the first prompt information may be “circle the passerby or object you want to eliminate”.

[0183] For example, in combination Figure 3B ,like Figure 6 As shown, the mobile phone can display a first prompt message "Circle the passerby or object you want to eliminate" in a blank area of ​​the editing interface 20.

[0184] In the embodiment of the present application, the electronic device prompts the user to circle the elimination object to be eliminated through the first prompt information, thereby avoiding the phenomenon that the user is unclear about the elimination steps.

[0185] In some embodiments of the present application, after the above-mentioned step 301, the dynamic photo editing method provided by the embodiment of the present application also includes the following steps 601 and 602.

[0186] Step 601: The electronic device displays a prompt window.

[0187] In some embodiments of the present application, the prompt window includes a second prompt message and a second control, and the second prompt message is used to prompt that the dynamic photo removal function has been started.

[0188] Exemplarily, the electronic device may display a prompt window via a pop-up window in the editing interface.

[0189] For example, the second prompt information may be “circle the passers-by or objects you want to remove and remove them, and they will still be saved as dynamic photos after removal.”

[0190] In some embodiments of the present application, the second control is used to close the prompt window.

[0191] Exemplarily, the second control may be a “know” control.

[0192] In some embodiments of the present application, the electronic device may display a circle selection animation in a prompt window to prompt the user how to operate.

[0193] Step 602: The electronic device closes the prompt window in response to the second input to the second control.

[0194] In the embodiment of the present application, the second input is used to trigger the electronic device to close the prompt window.

[0195] In some embodiments of the present application, the above-mentioned second input can be a click input of the above-mentioned second control by the user through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible input. The specific input can be determined according to actual usage requirements and is not limited in the embodiments of the present application.

[0196] Exemplarily, the second input may be a single-click input of the user on the second control.

[0197] For example, in combination Figure 3A ,like Figure 7 As shown, after the user clicks the AI ​​erasure control 14, the mobile phone can display a prompt window 25 on the editing interface 10, which includes a window name "Dynamic Photo AI Erasure", a circle selection animation 26, a second prompt message "Circle the passers-by or objects you want to erase to erase, and the erased photos will still be saved as dynamic photos" and a second control 27. Figure 7 The user can click the second control 27 to input, such as Figure 3B As shown, the mobile phone can display the updated editing interface 20.

[0198] In the embodiment of the present application, the electronic device can remind the user again of the specific process of the elimination operation through a prompt window to prevent the user from forgetting.

[0199] In some embodiments of the present application, the dynamic photo editing method provided in the embodiments of the present application also includes the following steps 701 and 702.

[0200] Step 701: When the second video frame of the video is displayed in the video frame display area and the thumbnail image identifier is displayed on the second thumbnail image, the electronic device receives a selection input for a first thumbnail image among at least two thumbnail images.

[0201] In some embodiments of the present application, the second thumbnail image is a thumbnail image corresponding to the second video frame.

[0202] In some embodiments of the present application, the second video frame may be any video frame among all video frames of the video except the first video frame.

[0203] In some embodiments of the present application, the second video frame may not contain an elimination object that the user needs to eliminate.

[0204] In some embodiments of the present application, the selection input is used to select a first thumbnail image from at least two thumbnail images.

[0205] In some embodiments of the present application, the above-mentioned selection input can be a click input of the user on the above-mentioned first thumbnail image through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible input. The specific input can be determined according to actual usage requirements and is not limited in the embodiments of the present application.

[0206] Exemplarily, the selection input may be a single-click input of the user on the first thumbnail picture.

[0207] Step 702: In response to the selection input, the electronic device moves the thumbnail image identifier to the first thumbnail image and updates the second video frame displayed in the video frame display area to the first video frame.

[0208] For example, Figure 8A As shown, the video frame display area in the editing interface 20 of the mobile phone displays the second video frame 28 of the video. The second video frame does not include the elephant and the hot air balloon. At this time, Figure 8B As shown, the user can click and input the first thumbnail image 30 corresponding to the first video frame, as shown in FIG. Figure 8C As shown, the mobile phone can move the thumbnail image identifier from the second thumbnail image 29 corresponding to the second video frame to the first thumbnail image 30 corresponding to the first video frame, and update the second video frame 28 displayed in the video frame display area 11 to the first video frame 12.

[0209] In some embodiments of the present application, after the electronic device updates the second video frame displayed in the video frame display area to the first video frame, the electronic device may display the first prompt information again in the video frame display area.

[0210] In an embodiment of the present application, when there is no elimination object that the user needs to eliminate in the above-mentioned second video frame, the user can update the video frame displayed in the video frame display area by changing the thumbnail image, so that the user can select the elimination object that the user needs to eliminate from the updated video frame.

[0211] In some embodiments of the present application, the above-mentioned elimination object selection input includes a selection input of the first video object in the first video frame; the dynamic photo editing method provided by the embodiment of the present application also includes the following steps 801 and 802.

[0212] Step 801: Upon receiving an input for selecting an object to be eliminated from a first video object in a first video frame, the electronic device displays third prompt information.

[0213] In some embodiments of the present application, the third prompt information is used to prompt the user to continue selecting objects to be eliminated.

[0214] In some embodiments of the present application, the electronic device may display the third prompt information in a pop-up window in the editing interface; or the electronic device may display the third prompt information in a blank area in the editing interface. The specific details may be determined based on actual usage and are not limited by the embodiments of the present application.

[0215] For example, the third prompt message may be: "Please continue to circle the passers-by or objects you want to eliminate."

[0216] For example, in combination Figure 4A ,like Figure 9 As shown, the mobile phone may display a third prompt message “Please continue to circle the passers-by or objects you want to eliminate” in the blank area of ​​the editing interface 20 .

[0217] Step 802: Upon receiving an input for selecting an object to be eliminated for the second video object in the first video frame, the electronic device determines the first video object and the second video object as objects to be eliminated.

[0218] In some embodiments of the present application, the first video object may be any one of the following: a person, an animal, or an object.

[0219] In some embodiments of the present application, the second video object may be any one of the following: a person, an animal, or an object.

[0220] In this way, the electronic device can prompt the user to select multiple elimination objects through the third prompt information, so that the user can select multiple elimination objects at one time, which improves the flexibility of the electronic device in performing video elimination.

[0221] In some embodiments of the present application, after the electronic device determines the first video object and the second video object as objects to be eliminated, it may mark the first video object and the second video object.

[0222] In some embodiments of the present application, the dynamic photo editing method provided by the embodiments of the present application also includes the following steps 901 and 902.

[0223] Step 901: Upon receiving an input for selecting an object to be eliminated for a first video object in a first video frame, the electronic device displays a first object to be eliminated identifier on the first video object.

[0224] In some embodiments of the present application, the first elimination object identifier is used to indicate an object outline of the first video object.

[0225] Exemplarily, the electronic device may display the first elimination object identifier on the object outline of the first video object.

[0226] In some embodiments of the present application, the first elimination object identifier may be any of the following: a color identifier, a text identifier, a pattern identifier, a symbol identifier, etc. The specific identifier may be determined according to actual use requirements and is not limited in the embodiments of the present application.

[0227] For example, in combination Figure 4A ,like Figure 10 As shown, the mobile phone may display a marking frame 70 on the outline of the elephant 40 .

[0228] Step 902: Upon receiving an input for selecting an object to be eliminated for a second video object in a first video frame, the electronic device displays a second object to be eliminated identifier on the second video object.

[0229] In some embodiments of the present application, the second elimination object identifier is used to indicate an object outline of the second video object.

[0230] Exemplarily, the electronic device may display the second eliminated object identifier on the object outline of the second video object.

[0231] In some embodiments of the present application, the second elimination object identifier may be any of the following: a color identifier, a text identifier, a pattern identifier, a symbol identifier, etc. The specific identifier may be determined according to actual use requirements and is not limited in the embodiments of the present application.

[0232] For example, in combination Figure 5A ,like Figure 11 As shown, the mobile phone may display a marking frame 60 on the outline of the elephant 40 and a marking frame 61 on the outline of the hot air balloon 41 .

[0233] In the embodiment of the present application, by adding elimination object identifiers to the first video object and the second video object, the user can clearly know the elimination object, which makes it easier for the user to find the elimination object from the video frame.

[0234] In some embodiments of the present application, before "eliminating the elimination object in each video frame of the video and selecting and inputting the selected elimination object" in the above step 202, the dynamic photo editing method provided by the embodiment of the present application also includes the following step 1001.

[0235] Step 1001: The electronic device receives a third input to a third control.

[0236] In some embodiments of the present application, the third input is used to trigger the electronic device to perform an elimination operation.

[0237] In some embodiments of the present application, the third control is used to trigger the electronic device to perform an elimination operation.

[0238] In some embodiments of the present application, the above-mentioned third input can be a click input of the above-mentioned third control by the user through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible input. The specific input can be determined according to actual usage requirements and is not limited in the embodiments of the present application.

[0239] Exemplarily, the third input may be a single-click input of the user on the third control.

[0240] In some embodiments of the present application, the above step 202 of "eliminating the elimination object selected by inputting the elimination object in each video frame of the video" can be specifically implemented through the following step 202a.

[0241] Step 202a: The electronic device eliminates the elimination object selected by the elimination object selection input in each video frame of the video in response to the third input to the third control.

[0242] It should be noted that the specific process of selecting and inputting the selected object to be eliminated in each video frame of the above-mentioned elimination video can be found in the following embodiment in detail, and will not be described again here to avoid repetition.

[0243] In an embodiment of the present application, the electronic device can automatically determine the elimination object from each video frame of the video through the user's input to the third control, and directly eliminate the elimination object in each video frame of the video, thereby improving the efficiency of the electronic device in performing video elimination.

[0244] In some embodiments of the present application, the above-mentioned video includes N video frames, where N is an integer greater than 1; before "selecting and inputting the selected object to be eliminated in each video frame of the video" in the above-mentioned step 202, the dynamic photo editing method provided by the embodiment of the present application also includes the following steps 1101 and 1102.

[0245] Step 1101: The electronic device obtains a mask image corresponding to the first video frame based on the first video frame.

[0246] It should be noted that the specific process of the above step 1101 can be found in the following embodiment, and will not be described again here to avoid repetition.

[0247] Step 1102: The electronic device identifies the display area of ​​the elimination object in each of the M video frames of the video based on the mask image corresponding to the first video frame and the image feature information of the elimination object in the first video frame, and displays an elimination object identifier on the elimination object in each of the M video frames.

[0248] In some embodiments of the present application, M is a positive integer, M≤N-1; each of the M video frames includes an elimination object.

[0249] For example, Figure 6 As shown, assuming that the video includes 5 video frames, the user selects the elephant 40 in the first video frame, and the elephant 40 is included in 3 of the remaining 4 video frames, the mobile phone can determine that the elephant 40 is included in 3 video frames from the 4 video frames based on the mask image corresponding to the first video frame and the image feature information of the elephant 40, identify the display area of ​​the elephant 40 in each of the 3 video frames, and display the elimination object mark of the elephant 40 in each of the 3 video frames.

[0250] It can be understood that the above display area includes elimination objects.

[0251] In some embodiments of the present application, the electronic device can input the mask image corresponding to the first video frame and the image feature information of the eliminated object in the first video frame into the first model, identify the display area of ​​the eliminated object in each of the M video frames of the video through the first model, and display an eliminated object identifier on the eliminated object in each of the M video frames.

[0252] In some embodiments of the present application, the first model may be at least one of the following: a neural network model, an AI model, or a large model, etc. The specific model may be determined based on actual usage requirements and is not limited by the embodiments of the present application.

[0253] In an embodiment of the present application, the electronic device determines the display area of ​​the elimination object in each of the M video frames of the video through the mask image corresponding to the first video frame and the image feature information of the elimination object in the first video frame, so as to determine the elimination object based on the display area, thereby improving the accuracy of the electronic device in identifying the elimination object.

[0254] In some embodiments of the present application, the elimination object is the first video object in the video; the step 1101 can be specifically implemented through the following steps 1101a to 1101c.

[0255] Step 1101a: The electronic device determines an object contour of a first video object from a first video frame based on eliminating an input trajectory of an object selection input.

[0256] In some embodiments of the present application, the electronic device may determine the object contour of the first video object from the first video frame according to an image area enclosed by an input trajectory of the eliminated object selection input.

[0257] Exemplarily, the electronic device may convert the coordinate points of the input trajectory of the object selection input on the screen into a polygonal mask through a convex hull algorithm or an Alpha Shape algorithm to generate a closed area, and then segment the closed area through an edge detection algorithm to obtain the object contour of the first video object.

[0258] In some embodiments of the present application, the edge detection algorithm may be any one of the following: Sobel algorithm, Canny algorithm, or EdgeFlow algorithm, etc. The specific algorithm may be determined according to actual use requirements and is not limited in the embodiments of the present application.

[0259] Step 1101b: The electronic device performs image segmentation on the first video frame based on the object contour of the first video object.

[0260] In some embodiments of the present application, the electronic device may perform image segmentation on the object contour using a first segmentation model to obtain a segmented first video frame.

[0261] In some embodiments of the present application, the first segmentation model may be any of the following: a Mask Region-based Convolutional Neural Network (Mask R-CNN) model, a You Only Look At Common Things (YOLACT) model, or a Structure of the Observed Learning Outcome (SOLO) model. The specific model may be determined based on actual usage requirements and is not limited in the present embodiment.

[0262] Step 1101c: The electronic device performs binarization processing on the first video frame after image segmentation to obtain a mask image corresponding to the first video frame.

[0263] In some embodiments of the present application, the electronic device may set the image area corresponding to the object contour of the first video object in the first video frame to 0, and set the area in the first video frame other than the image area corresponding to the object contour of the first video object to 1, so as to obtain a mask image corresponding to the first video frame.

[0264] For example, Figure 12As shown, assuming that the user needs to remove the elephant in the first video frame, after determining the image outline of the elephant in the first video frame, the mobile phone can segment the image outline of the elephant through Mask R-CNN to obtain the segmented first video frame, and then set the image area corresponding to the image outline of the person in the first video frame to 0, that is, Figure 12 As shown, the image area corresponding to the image outline of the elephant is white, and the area other than the image area corresponding to the image outline of the elephant in the first video frame is set to 1, that is, Figure 12 As shown, the image area except the outline of the elephant in the first video frame is black, so the mask image corresponding to the first video frame is obtained.

[0265] It should be noted that the above is only an example explanation using one video frame. For all video frames of the video, the electronic device can obtain the mask image corresponding to each video frame through the above method. To avoid repetition, it will not be repeated here.

[0266] In an embodiment of the present application, the electronic device segments and binarizes the first video frame according to the object contour of the first video object, and can determine the elimination object that the user needs to eliminate, thereby improving the accuracy of the electronic device in determining the elimination object.

[0267] In some embodiments of the present application, the above-mentioned elimination objects include K video objects, where K is an integer greater than 1; the above-mentioned step 1101 can be specifically implemented through the following steps 1101d to 1101e.

[0268] Step 1101d: The electronic device performs image segmentation processing on the first video frame based on the K video objects to obtain K mask images corresponding to the K video objects.

[0269] It can be understood that the above K video objects are multiple video objects selected by the user.

[0270] It should be noted that the specific implementation process of the above step 1101d can be found in the above embodiment in detail, and will not be described here to avoid repetition.

[0271] Step 1101e: The electronic device merges K mask images to obtain a mask image corresponding to the first video frame.

[0272] In some embodiments of the present application, the electronic device may fuse K mask images to obtain a mask image corresponding to the first video frame.

[0273] For example, in combination Figure 12 ,like Figure 13As shown, assuming that the two video objects include an elephant and a hot air balloon, after obtaining the mask image of the hot air balloon, the mobile phone can fuse the mask image of the elephant and the mask image of the hot air balloon to obtain the mask image corresponding to the first video frame, that is, the elephant and the hot air balloon are indicated in white in the fused mask image.

[0274] It should be noted that the above is only an example explanation using one video frame. For all video frames of the video, the electronic device can obtain the mask image corresponding to each video frame through the above method. To avoid repetition, it will not be repeated here.

[0275] In an embodiment of the present application, after the user selects multiple video objects, the electronic device can obtain the mask images corresponding to each video object respectively and merge them, so that the electronic device can determine that there are multiple objects that need to be eliminated in the first video frame, thereby improving the flexibility of the electronic device in determining the objects to be eliminated.

[0276] In some embodiments of the present application, the above step 1102 can be specifically implemented through the following steps 1102a to 1102c.

[0277] Step 1102a: The electronic device performs panoptic segmentation processing on each of the M video frames to obtain a mask image corresponding to each video object in each video frame and category information of each mask image.

[0278] In some embodiments of the present application, the above-mentioned category information is used to characterize the object category of the eliminated object in each of the M video frames.

[0279] It should be noted that the granularity at which the electronic device performs panoptic segmentation processing on each of the M video frames is a video object.

[0280] For example, assuming that a video frame includes two people, an object, and a sea, performing panoramic segmentation on the video frame means segmenting the two people, the object, and the sea from the first video frame respectively.

[0281] In some embodiments of the present application, the electronic device may perform panoramic segmentation processing on each of the M video frames using a second segmentation model to obtain a mask image corresponding to each video object in each video frame.

[0282] In some embodiments of the present application, the above-mentioned second segmentation model can be any one of the following: a panoramic feature pyramid network (Panoptic Full Pyramid Network, Panoptic FPN) model or a unified panoramic segmentation network (AUnified Panoptic Segmentation Network, UPSNet) model, etc., which can be determined according to actual usage requirements and is not limited by the embodiments of the present application.

[0283] In some embodiments of the present application, after obtaining the mask image corresponding to each video object in each video frame, the electronic device can input the mask image corresponding to each video object in each video frame into the convolutional network model in sequence to identify the category information corresponding to the mask image corresponding to each video object in each video frame.

[0284] Step 1102b: The electronic device compares the image similarity between the mask image corresponding to the first video frame and the mask image corresponding to each video object in each video frame based on the image feature information of the elimination object in the first video frame and the category information of the mask image corresponding to the first video frame, and determines the mask image corresponding to each video frame of the M video frames.

[0285] In some embodiments of the present application, the above-mentioned image feature information includes at least one of the following: color feature, texture feature, edge feature, shape feature, etc.

[0286] In some embodiments of the present application, the electronic device can perform image similarity comparison with the mask image corresponding to each video object in each video frame in turn through the image feature information of the eliminated object in the first video frame and the category information of the mask image corresponding to the first video frame, thereby determining the mask image whose image similarity is greater than or equal to a preset similarity threshold as the mask image corresponding to each video frame of the M video frames.

[0287] In some embodiments of the present application, the preset similarity threshold may be any one of the following: 80%, 90%, or 100%, which may be determined based on actual use requirements and is not limited in the present application.

[0288] In some embodiments of the present application, the preset similarity threshold may be preset by the electronic device or user-defined, and may be determined based on actual usage requirements and is not limited by the embodiments of the present application.

[0289] For example, taking a video frame as an example, assuming that there are 5 mask images corresponding to each video object in the video frame, the mobile phone can compare the similarity with the 5 mask images in turn based on the image feature information of the eliminated object in the first video frame and the category information of the mask image corresponding to the first video frame. Assuming that the similarity between the second mask image among the 5 mask images and the mask image corresponding to the first video frame is greater than 80%, and the others are less than 80%, the mobile phone can determine the second mask image among the 5 mask images as the mask image corresponding to the video frame.

[0290] Step 1102c: The electronic device determines a display area of ​​each of the M video frames from which the object is to be eliminated based on the mask image corresponding to each of the M video frames.

[0291] In some embodiments of the present application, taking a video frame as an example, the electronic device can overlay the mask image corresponding to the video frame on the video frame, and then determine the image area in the video frame corresponding to the white area in the mask image as the display area of ​​the eliminated object, so as to obtain the display area of ​​the eliminated object in each of the M video frames.

[0292] In some embodiments of the present application, the electronic device automatically determines the object to be eliminated from each video frame of M video frames through a mask image, without the need for the user to manually select in each video frame in turn, thereby improving the efficiency of the electronic device in determining the object to be eliminated.

[0293] In some embodiments of the present application, the above-mentioned editing interface also includes a thumbnail image display area, which includes thumbnail images corresponding to at least two video frames of the video; the first thumbnail image also includes a thumbnail image identifier; the dynamic photo editing method provided in the embodiment of the present application also includes the following steps 12001 and 12002.

[0294] Step 1201: In the process of eliminating the display area of ​​the object in each of M video frames of the identified video, the electronic device displays fourth prompt information.

[0295] In some embodiments of the present application, the fourth prompt information is used to prompt the recognition progress of eliminating the display area of ​​the object in each of the M video frames of the recognition video.

[0296] In some embodiments of the present application, the fourth prompt information may be preset by the electronic device; or the fourth prompt information may be user-defined. The specific information may be determined based on actual use requirements and is not limited by the embodiments of the present application.

[0297] Exemplarily, the fourth prompt information may be: intelligent recognition frame by frame...10%.

[0298] In some embodiments of the present application, the fourth prompt information can change in real time according to the recognition progress.

[0299] For example, assuming that the M video frames of the video are 5 video frames, when the mobile phone recognizes the second video frame, the fourth prompt information may be "Intelligent recognition frame by frame... 40%", and when the mobile phone recognizes the 4th video frame, the fourth prompt information may be "Intelligent recognition frame by frame... 80%".

[0300] In some embodiments of the present application, the electronic device may display the fourth prompt information in the editing interface through a pop-up window; or, the electronic device may display the fourth prompt information in a blank area in the editing interface.

[0301] Step 1202: When the display area of ​​the object is eliminated in the i-th video frame of the identified video, the electronic device moves the thumbnail image identifier to the i-th thumbnail image and updates the first video frame displayed in the video frame display area to the i-th video frame.

[0302] In some embodiments of the present application, the above-mentioned i-th thumbnail picture is a thumbnail picture corresponding to the i-th video frame, i is a positive integer, i≤M.

[0303] It is understandable that when the recognized video frame changes, the thumbnail image in the editing interface must also be updated in real time.

[0304] For example, in combination Figure 10 ,like Figure 14A As shown, when the mobile phone has finished recognizing the first video frame and is about to recognize the second video frame 30, the mobile phone can Figure 14A The thumbnail image mark 63 in the video is moved to the thumbnail image 31 corresponding to the second video frame 30, and "Intelligent identification of frames... 40%" is displayed in the blank area of ​​the editing interface 20. Figure 14B As shown, when the mobile phone has finished identifying the third video frame and is about to identify the fourth video frame 32, the mobile phone can move the thumbnail image identifier 63 to the corresponding thumbnail image 33 corresponding to the fourth video frame 32, and display "Intelligent recognition frame by frame...80%" in the blank area of ​​the editing interface 20.

[0305] For example, suppose the phone has currently recognized the third of five video frames. At this point, the phone will move the thumbnail marker to the thumbnail corresponding to the third frame and display "Frame-by-frame intelligent recognition... 60%" in a blank area of ​​the editing interface. If the phone has currently recognized the fifth of five video frames, the phone will move the thumbnail marker to the thumbnail corresponding to the fifth frame and display "Frame-by-frame recognition completed" in a blank area of ​​the editing interface.

[0306] In the embodiment of the present application, the electronic device can prompt the user of the current processing progress of the electronic device by moving the thumbnail image logo and displaying the recognition progress.

[0307] In some embodiments of the present application, after identifying the elimination object in each video frame of the video, the electronic device may eliminate the elimination object in each video frame of the video.

[0308] In some embodiments of the present application, the dynamic photo editing method provided by the embodiments of the present application also includes the following step 1301.

[0309] Step 1301: In the process of selecting and inputting the selected object to be eliminated in each video frame of the elimination video, the electronic device displays an elimination animation of the elimination object.

[0310] In some embodiments of the present application, the elimination effect may be any of the following: particle diffusion elimination or gradual elimination, etc. The specific elimination effect may be determined according to actual use requirements and is not limited in the present application.

[0311] In some embodiments of the present application, the electronic device can display an elimination animation of the elimination object according to a preset direction.

[0312] For example, the preset direction may be from top to bottom, from bottom to top, from left to right, or from right to left, etc. The specific direction may be determined according to actual use requirements and is not limited in the embodiment of the present application.

[0313] For example, combining Figure 14B As shown, Figure 15 As shown, during the process of eliminating the object in each video frame of the video, the mobile phone can play the particle diffusion elimination dynamic from left to right to eliminate the elephant 40.

[0314] In an embodiment of the present application, in the process of selecting and inputting the selected elimination object in each video frame of the elimination video, playing the elimination animation of the elimination object can improve the fun of the elimination process of the electronic device.

[0315] In some embodiments of the present application, the above-mentioned editing interface also includes a fourth control.

[0316] Exemplarily, the dynamic photo editing method provided in the embodiment of the present application may further include the following step 1401.

[0317] Step 1401: The electronic device receives a fourth input to a fourth control.

[0318] In some embodiments of the present application, the fourth input is used to trigger the electronic device to run an elimination operation in the background through a fourth control.

[0319] In some embodiments of the present application, the fourth input may be a click input of the fourth control by the user through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible input, which may be determined according to actual usage requirements and is not limited in the embodiments of the present application.

[0320] Exemplarily, the fourth input may be a single-click input of the fourth control.

[0321] In some embodiments of the present application, the fourth control is used to trigger the electronic device to perform an elimination operation in the background.

[0322] Exemplarily, the dynamic photo editing method provided in the embodiment of the present application also includes the following step 1402.

[0323] Step 1402. In the process of eliminating the object selected by the elimination object selection input in each video frame of the elimination video, the electronic device responds to the fourth input to the fourth control, executes the processing of eliminating the object selected by the elimination object selection input in each video frame of the elimination video in the background, and updates the editing interface to the preview interface of the dynamic photo.

[0324] In some embodiments of the present application, the electronic device can jump from the editing interface to the preview interface of the dynamic photo based on the user's fourth input to the fourth control, so that the user can view other photos in the album application.

[0325] For example, Figure 16 As shown, the mobile phone can display the fourth control 34 in the editing interface 20. Figure 16 In the example, “background elimination” is used, and the user can click and input the fourth control 34. Figure 1 As shown, the mobile phone can jump from the editing interface 20 to the preview interface.

[0326] In some embodiments of the present application, after the electronic device completes the elimination operation in the background, the electronic device may display a prompt message in the current interface displayed by the electronic device to prompt the user to complete the elimination operation.

[0327] In an embodiment of the present application, the electronic device updates from the editing interface to the preview interface of the dynamic photo through the user's input, and can perform the elimination operation in the background without affecting the user's execution of other operations through the electronic device, thereby improving the flexibility of the electronic device in performing operations.

[0328] In some embodiments of the present application, the dynamic photo editing method provided by the embodiments of the present application also includes the following steps 1501 and 1502.

[0329] Step 1501: During the process of selecting and inputting the selected object to be eliminated in each video frame of the eliminated video, the electronic device displays fifth prompt information.

[0330] In some embodiments of the present application, the fifth prompt information is used to prompt the elimination progress of the elimination object.

[0331] Exemplarily, the fifth prompt information may be: eliminating frame by frame...10%.

[0332] In some embodiments of the present application, the fifth prompt information can change in real time according to the recognition progress.

[0333] For example, assuming that the M video frames of the video are 5 video frames, when the mobile phone eliminates the elimination object in the second video frame, the fifth prompt information may be "Eliminating frame by frame...40%", and when the mobile phone eliminates the elimination object in the fourth video frame, the fifth prompt information may be "Eliminating frame by frame...80%".

[0334] In some embodiments of the present application, the electronic device may display the fifth prompt information in the editing interface through a pop-up window; or, the electronic device may display the fifth prompt information in a blank area in the editing interface.

[0335] Step 1502: When the elimination object in the i-th video frame is eliminated, the electronic device moves the elimination mark to the i-th thumbnail picture.

[0336] In some embodiments of the present application, the above-mentioned i-th thumbnail picture is a thumbnail picture corresponding to the i-th video frame, i is a positive integer, i≤M.

[0337] It is understandable that when the deleted video frame changes, the thumbnail image in the editing interface must also be updated in real time.

[0338] For example, in combination Figure 14A ,like Figure 17A As shown, after the mobile phone finishes erasing the first of the five video frames, the mobile phone can automatically move the thumbnail image mark 63 to the thumbnail image 31 of the second video frame 30 of the five video frames, and display the second video frame 30 in the editing interface 20, and display "erasing frame by frame... 40%" in the blank area of ​​the editing interface 20. Figure 17B As shown, after the mobile phone completes eliminating the third video frame among the five video frames, the mobile phone can automatically move the thumbnail image identifier 63 to the video thumbnail image 33 of the fourth video frame 32 among the five video frames, and display the fourth video frame 32 in the editing interface 20, and display "Erasing frame by frame...80%" in the blank area of ​​the editing interface 20.

[0339] In the embodiment of the present application, the electronic device can prompt the user of the current processing progress of the electronic device by moving the thumbnail image identifier and displaying the elimination progress.

[0340] In some embodiments of the present application, the “eliminating the elimination object in each video frame of the video and selecting and inputting the selected elimination object” in the above step 202 can be specifically implemented through the following steps 1601 to 1605 .

[0341] Step 1601: The electronic device adds preset noise to a first video frame to obtain a first noisy video frame.

[0342] In some embodiments of the present application, the electronic device may randomly sample a noise from Gaussian noise, namely the above-mentioned preset noise, and then add the randomly sampled noise to the first video frame to obtain a first noisy video frame.

[0343] Step 1602: The electronic device predicts the image noise of the object area of ​​the object to be eliminated in the first noisy video frame based on the first noisy video frame, the mask image corresponding to the first video frame, the video background image corresponding to the first video frame, the positive sample vector, the negative sample vector, and the prediction time step to obtain a second noisy video frame.

[0344] In some embodiments of the present application, the above-mentioned positive sample vector is used to indicate that the video object contained in the mask image corresponding to the first video frame is the shooting foreground, and the above-mentioned negative sample vector is used to characterize that the video object contained in the video background image corresponding to the first video frame is the shooting background, and the above-mentioned video background image is an image that does not include the elimination object selected by the elimination object selection input.

[0345] In some embodiments of the present application, the above-mentioned prediction time step may be preset by the electronic device or customized by the user. The specific time step may be determined according to actual use requirements and is not limited by the embodiments of the present application.

[0346] Exemplarily, the electronic device may obtain the second noise video frame by using the following formula (3).

[0347]

[0348] Where, latent_noise_pred is the second noise video frame, w is a hyperparameter, latent_input is the first noise video frame, the video background image corresponding to the first video frame, and the mask image corresponding to the first video frame; 1embedding is the negative sample vector, 0embedding is the positive sample vector, and t is the prediction time step.

[0349] It can be understood that the electronic device uses a combination of positive and negative classes to achieve the purpose of elimination. First, for the VEM model that has been trained, the negative class embedding vector is generated once, and then the positive class embedding vector is generated once. The result of the negative class embedding vector is subtracted from the result of the positive class embedding vector as the input of the model for the next time, thereby suppressing the area of ​​interest, the positive class vector corresponds to the object to be eliminated, and the background area is generated. The negative class vector corresponds to the background area, and finally the process of guiding the model to generate the background for the eliminated area is realized.

[0350] Step 1603: The electronic device denoises the second noisy video frame to obtain a third video frame corresponding to the first video frame.

[0351] Exemplarily, the electronic device may obtain the third video frame corresponding to the first video frame by using the following formula (4).

[0352] latent_input_t-1=Denoise(latent_input_t, latent_noise_pred,t) (4)

[0353] Where latent_input_t-1 is the third video frame, Denoise is the standard denoising process in the diffusion model, which is derived from the forward denoising process, latent_input_t is the first noisy video frame, the video background image corresponding to the first video frame, and the mask image corresponding to the first video frame; latent_noise_pred is the second noisy video frame, and t is the preset time step.

[0354] For example, Figure 18 As shown, the electronic device adds preset noise to the first video frame to obtain a first noise video frame, such as Figure 4B As shown, the electronic device denoises the second noisy video frame to remove the elephant in the first video frame, and obtains a third video frame corresponding to the first video frame.

[0355] Step 1604: The electronic device determines an image to be filled based on the mask image corresponding to the first video frame.

[0356] In some embodiments of the present application, the above-mentioned image to be filled is an image of an area in the third video frame corresponding to the background image area in the mask image of the first video frame, and the background image area is the image area corresponding to the elimination object selected by the elimination object selection input in the mask image of the first video frame.

[0357] Exemplarily, the electronic device can overlay the mask image corresponding to the first video frame on the third video frame, and then determine the white area in the mask image corresponding to the first video frame and the area where the position overlaps with that in the third video frame as the first area, and the image in the first area is the image to be filled.

[0358] In some embodiments of the present application, after the electronic device determines the image to be filled, the electronic device may segment the image to be filled from the third video frame.

[0359] Step 1605: The electronic device fills the image to be filled into the video background image corresponding to the first video frame, and eliminates the elimination object selected by the elimination object selection input in the first video frame, so as to eliminate the elimination object selected by the elimination object selection input in each video frame of the video.

[0360] In some embodiments of the present application, the electronic device may add the image to be filled in the blank area of ​​the video background image, thereby eliminating the elimination object selected by the elimination object selection input in the first video frame.

[0361] For example, the above steps 1601 to 1605 are explained in detail below through a specific example, which can be implemented through the following steps 40 to 44.

[0362] Step 40: The electronic device obtains a mask image of the first object to be eliminated, and obtains a background area by performing element-wise dot product between the mask image and the first video frame.

[0363] Exemplarily, the electronic device may use a standard diffusion model sampling generation method combined with positive / negative class embedding for generation.

[0364] Step 41: During inference, the electronic device randomly samples a latent_nosie from Gaussian noise with a shape of (f, h, w, 4). It then uses the mask image to obtain the background area and converts it into latent_background through a 3D VAE. The resized mask image is then combined as the model input latent_mask, and the result is latent_input = [latent_nosie, latent_background, latent_mask].

[0365] Step 42: The electronic device may predict noise in the video frame according to the time step 0 to 1, and denoise the video frame after the noise is predicted according to the time step 1 to 0.

[0366] For example, the specific process of predicting the noise can be detailed in the above formula (3). The specific process of denoising can be detailed in the above formula (4). To avoid repetition, it will not be described here.

[0367] Repeat the above steps until the time step is 0, completing the denoising process. At that time, the final generated image latent is output.

[0368] Step 43: The electronic device uses the decoder of the 3D VAE to decode the inferred image latent to obtain the patched video content.

[0369] Step 44: The electronic device pastes the generated video content back to the blank position in the original background image according to the mask image, completing the final restoration process.

[0370] It should be noted that the above is only explained through one video frame. For all video frames of the video, the electronic device can be implemented through the above embodiment. To avoid repetition, it will not be described here.

[0371] In an embodiment of the present application, the electronic device can denoise the second noisy video frame through a diffusion model to obtain a third video frame, and determine the image to be filled from the third video frame, thereby improving the accuracy of the electronic device in determining the image to be filled.

[0372] In some embodiments of the present application, the dynamic photo editing method provided by the embodiments of the present application also includes the following steps 1701 to 1707.

[0373] Step 1701: The electronic device obtains a positive sample vector, a negative sample vector, and a first training image.

[0374] In some embodiments of the present application, the positive sample vector and the negative sample vector may be preset by the electronic device; or customized by the user.

[0375] In some embodiments of the present application, the data represented by the positive sample vector includes at least one of the following: a person, a cat, a dog, a table, a stool, etc. The specific data may be determined according to actual use requirements and is not limited in the embodiments of the present application.

[0376] In some embodiments of the present application, the data represented by the negative sample vector includes at least one of the following: beach, sea water, sky, etc. The specific data may be determined according to actual use requirements and is not limited in the embodiments of the present application.

[0377] In some embodiments of the present application, the first training image may be preset by the electronic device; or customized by the user.

[0378] Step 1702: The electronic device randomly determines a first training video object from the first training image, and obtains a first training mask image corresponding to the first training image based on the first training video object.

[0379] In some embodiments of the present application, the electronic device may perform panorama segmentation on the first training image to obtain multiple training video objects, and then randomly select the first one from the multiple training video objects as the first training video object.

[0380] It should be noted that the specific process of obtaining the first training mask image corresponding to the first training image based on the first training video object can be found in the above embodiment, and will not be described again here to avoid repetition.

[0381] Step 1303: The electronic device marks the image category corresponding to the first training mask image based on the image features, positive sample vector, and negative sample vector of the first training video object.

[0382] In some embodiments of the present application, the above-mentioned image category is any one of shooting foreground or shooting background.

[0383] In some embodiments of the present application, the electronic device can obtain the image features of the first training video object through the above-mentioned first model to obtain the category of the first training video object. Then, if the positive sample vector contains the same data as the category, the first training mask image is marked as the shooting foreground; if the negative sample vector contains the same data as the category, the first training mask image is marked as the shooting background.

[0384] For example, assuming that the electronic device determines that the category of the first training video object is "person", the electronic device detects that the data "person" exists in the positive sample vector, and at this time the first training mask image can be marked as the shooting foreground; assuming that the electronic device determines that the category of the first training video object is "beach", the electronic device detects that the data "beach" exists in the negative sample vector, and at this time the first training mask image can be marked as the shooting background.

[0385] Step 1304: The electronic device inputs the first training mask image, the noisy training image corresponding to the first training image, and the training background image corresponding to the first training image into the initial model.

[0386] In some embodiments of the present application, the electronic device may splice the first training mask image, the noisy training image corresponding to the first training image, and the training background image corresponding to the first training image to obtain spliced ​​training data, and then input the spliced ​​training data into the initial model.

[0387] Step 1305: The electronic device determines the image category corresponding to the first training mask image through the cross attention unit in the initial model.

[0388] In some embodiments of the present application, the cross-attention unit in the initial model can determine the image category corresponding to the first training mask image according to the image category marked by the first training mask image.

[0389] In some embodiments of the present application, the above-mentioned cross-attention unit can be added to the DiT structure of the initial model.

[0390] For example, Figure 19 As shown, the above DiT structure may include: input module, scale / shift module, QK-Norm module, RoPE-3D module, Self-Attention module, QK-Norm module, cross attention module, i.e. the cross attention unit, Figure 19 In the figure, Cross-Attention is used to represent the FNN module, Gate module, and output module, Text Embedding, and Time Embedding. The input module is connected to the Scale / Shift module, the Scale / Shift module is connected to the QK-Norm module, the QK-Norm module is connected to the RoPE-3D module, the RoPE-3D module is connected to the Self-Attention module, the Self-Attention module is connected to the Gate module, the Gate module is connected to the QK-Norm module, the QK-Norm module is connected to the Cross-Attention module, the Cross-Attention module is connected to the Text Embedding and Scale / Shift modules respectively, the Scale / Shift module is connected to the FNN module, the FNN module is connected to the Gate module, the Time Embedding is connected to the Scale / Shift module and the Gate module respectively, and the Gate module is connected to the output module.

[0391] Step 1306: The electronic device predicts a training image to be filled corresponding to the first training mask image based on the image category, the first training mask image, the noisy training image corresponding to the first training image, and the training background image corresponding to the first training image, and adds the training image to be filled to the training background image to obtain a second training image.

[0392] It should be noted that the above prediction process can be found in the above embodiments in detail, and will not be described here in detail to avoid repetition.

[0393] Step 1307: The electronic device calculates a loss value between the second training image and the reference image, and adjusts the model parameters of the original model according to the loss value to obtain a video elimination model.

[0394] It should be noted that the calculation of the loss value between the second training image and the reference image can be found in the description of the relevant technology. To avoid repetition, it will not be repeated here.

[0395] In some embodiments of the present application, the electronic device may adjust the model parameters of the original model by back propagation to obtain a video elimination model.

[0396] For example, Figure 19 As shown, the model training method provided in the embodiment of the present application is specifically explained below. Specifically, it can be shown in the following steps 1 to 9.

[0397] Step 1: The electronic device may define a list of objects of interest and a list of background objects.

[0398] For example, the above-mentioned list of objects of interest may include people, cats, dogs, etc. The above-mentioned list of background objects may include beaches, sea water, sky, etc.

[0399] Step 2: The electronic device randomly samples a video and a mask image of an object in the video. It determines whether the object is in the list of objects of interest based on its category. If it is in the list of objects of interest, the text embedding is marked as a positive vector. If it is in the list of background objects, the text embedding is marked as a negative vector.

[0400] Step 3: The electronic device may obtain the background area according to the mask image.

[0401] Step 4: The electronic device crops the input video and background area to a specific training size (e.g., 512*512 or 768*768) and passes the image through the 3D VAE encoder to obtain the corresponding latent vector. The low-dimensional vector obtained after the video frame passes through the 3D VAE is recorded as latent_video, and the vector corresponding to the background area is recorded as latent_background. Their shapes are both [f, h, w, c]. Before training, Gaussian noise data (according to the diffusion model schedule) needs to be added as the input latent_video.

[0402] Step 5: The electronic device resizes the mask area to the same shape as latent_video through a resize operation, but its number of channels is 1, and its shape is [f, h, w, 1], which is recorded as latent_mask.

[0403] Step 6: The electronic device concatenates all latent_x (x is video, background, mask) as the input of the model, which is recorded as latent_input = [latent_video, latent_background, latent_mask].

[0404] Step 7: After the electronic device receives latent_input as the input of Noisy Input, it trains the model using the standard diffusion model training process. The main body of the model adopts the standard DiT structure, and adds Cross-Attention to the DiT structure to receive the text embedding to distinguish whether the area corresponding to the mask image is an object or a context. The training output of the VEM model is recorded as — train_output = VEM(latent_input, t, [positive / negative] class embedding)

[0405] Step 8: The electronic device calculates the loss between the training output of the initial model and the ground truth result, and performs gradient backpropagation to update the model parameters to achieve the effect of the training model.

[0406] Step 9: The electronic device repeatedly iterates steps 2-8 until the model loss drops below a certain threshold, and the model training is considered complete.

[0407] In the embodiment of the present application, the positive vector corresponds to the object of interest, and the negative vector corresponds to the background object. The model is trained by the positive vector and the negative vector, and there is no need to use a text encoder to map the text input by the user, which can save computational complexity and simplify interaction. Moreover, the positive and negative training method is used, and there is no need to construct paired data, which greatly simplifies the training cost. In addition, the positive and negative classes are used separately during inference to suppress the model from outputting the object of interest and control the model from outputting the background object, thereby improving elimination accuracy.

[0408] In some embodiments of the present application, after the above-mentioned step 202, the dynamic photo editing method provided by the embodiment of the present application also includes the following steps 1401 to 1404.

[0409] Step 1401: The electronic device hides thumbnail images corresponding to at least two video frames of a video, and displays a fifth control and a sixth control in an editing interface.

[0410] In some embodiments of the present application, the electronic device can cancel the display in the editing interface, that is, the thumbnail images corresponding to at least two video frames of the hidden video.

[0411] In some embodiments of the present application, the electronic device may display the fifth control and the sixth control in the editing interface through a pop-up window; or, the electronic device may display the fifth control and the sixth control in a blank area in the editing interface.

[0412] In some embodiments of the present application, the fifth control is used to continue editing the video.

[0413] In some embodiments of the present application, the sixth control is used to save the video after the elimination operation is performed.

[0414] For example, in combination Figure 8B ,like Figure 20 As shown, the mobile phone can display the fifth control 35 in the blank area of ​​the editing interface 20. Figure 20 The sixth control 36 is shown as "Continue editing". Figure 20 Indicated by exit elimination.

[0415] Step 1402: When receiving the fifth input of the fifth control, the electronic device displays thumbnail images and thumbnail image identifiers of at least two video frames of the video in the thumbnail image display area, and displays the third control in the editing interface.

[0416] In some embodiments of the present application, the fifth input is used to trigger the electronic device to update the editing interface.

[0417] In some embodiments of the present application, the fifth input may be a click input of the fifth control by the user through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible input, which may be determined according to actual usage requirements and is not limited in the embodiments of the present application.

[0418] Exemplarily, the fifth input may be a single-click input of the fifth control by the user.

[0419] For example, in combination Figure 20 If the user clicks and inputs the fifth control, the mobile phone may display the third control in the editing interface 10, ie, start eliminating the controls, and cancel the display of the fifth and sixth controls.

[0420] Step 1403: When receiving the sixth input of the third control, the electronic device eliminates the elimination object selected by the elimination object selection input in each video frame of the video.

[0421] In some embodiments of the present application, the sixth input is used to trigger the electronic device to perform image elimination processing on the elimination object selected by the elimination object selection input in each video frame of the video again.

[0422] In some embodiments of the present application, the sixth input may be a click input of the fifth control by the user through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible input, which may be determined according to actual usage requirements and is not limited in the embodiments of the present application.

[0423] Exemplarily, the sixth input may be a single-click input of the user on the third control.

[0424] It should be noted that the specific process of the electronic device eliminating the elimination object in each video frame of the video through the user's input to the third control and selecting the elimination object to be input can be found in the above embodiment in detail. To avoid repetition, it will not be repeated here.

[0425] Step 1404: Upon receiving the seventh input of the sixth control, the electronic device stores the video after the elimination process.

[0426] In some embodiments of the present application, the seventh input is used to control the electronic device to store the video after the elimination process.

[0427] In some embodiments of the present application, the seventh input may be a click input of the sixth control by the user through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible input, which may be determined according to actual usage requirements and is not limited in the embodiments of the present application.

[0428] Exemplarily, the seventh input may be a single-click input of the sixth control by the user.

[0429] In some embodiments of the present application, after the electronic device stores the video after the elimination process, a prompt message may be displayed in the editing interface to remind the user that the video after the elimination process has been stored in the album application.

[0430] In an embodiment of the present application, after eliminating the object required by the user, the electronic device can provide the user with multiple controls, each control corresponding to an option. The user can control the electronic device to perform different operations through different controls, thereby improving the flexibility of the electronic device in operating.

[0431] For example, Figure 21 As shown, the dynamic photo editing method provided by the embodiment of the present application is specifically explained below through a specific example. Specifically, it can be implemented according to steps 101 to 1000.

[0432] Step 101: The user selects a video from the album.

[0433] Step 102: The electronic device enters the AI ​​elimination sub-page, activates the dynamic recruitment elimination capability, displays the decoded video frame, and provides user-selectable interactive capabilities.

[0434] It should be noted that the main process of short video elimination is: the user selects a video from the album as input, and clicks the AI ​​elimination function button of the album so that the electronic device can enter the short video elimination interface and activate the dynamic photo elimination capability. The electronic device can decode the video corresponding to the dynamic photo to obtain the video content of each video frame and display it to the user. The user can select any frame to circle or click on the elimination object, so that the electronic device can segment and obtain the edge contour of the elimination object, and segment the same elimination object in the remaining video frame content based on the segmentation tracking algorithm, so as to display the contour of the elimination object circled by the user in all video frames and determine the target area corresponding to the elimination object. Finally, the background of the target area is filled in by the video elimination model to obtain the eliminated video content, and the video content is posted back to the original video frame to eliminate the elimination object that the user needs to eliminate in all video frames, and the eliminated video frame is displayed to the user.

[0435] Step 103: The user selects a first video frame in the video frames, so that the electronic device can display the first video frame in the editing interface.

[0436] Step 104: The user clicks or frames the object to be eliminated so that the electronic device can mark the object.

[0437] For example, the user can choose to circle an object in the first video frame, such as a person, animal, bag, etc. The user can circle the precise object outline or the rough outline. The subsequent processing flow will perform instance segmentation on the outline to obtain the precise object outline, thereby reducing the difficulty of use for the user.

[0438] Step 105 : The electronic device determines whether to adjust the border of the segmented object. If so, the process returns to step 104 ; if not, the process proceeds to step 106 .

[0439] Step 106: The electronic device displays the marked object to be eliminated and tracks the mask image of the object to be eliminated in the remaining video frames.

[0440] Step 107: The electronic device determines whether to move to another video frame to add more objects. If so, the process returns to step 104; if not, the process proceeds to step 108.

[0441] Step 108: The electronic device obtains a complete mask image of the object to be eliminated, and the user clicks an elimination control.

[0442] Step 109: The electronic device eliminates the object to be eliminated selected by the user through the video elimination model.

[0443] Step 1000: The electronic device plays the delay elimination animation and displays the video after the delay is eliminated.

[0444] In an embodiment of the present application, since the user can input the selection of the elimination object for the first video frame, the electronic device can automatically determine the elimination object selected by the selection input from each video frame of the video, so the user does not need to select the elimination object in each video frame of the video in turn, which simplifies the steps for the user to select the elimination object in the video, thereby improving the efficiency of the electronic device in eliminating the elimination object in the video.

[0445] It should be noted that the dynamic photo editing method provided in the embodiment of the present application can be executed by a dynamic photo editing device. In the embodiment of the present application, the dynamic photo editing device executing the dynamic photo editing method is taken as an example to illustrate the dynamic photo editing device provided in the embodiment of the present application.

[0446] Figure 22 FIG. 1 shows a possible structural diagram of a dynamic photo editing device involved in an embodiment of the present application. Figure 22 As shown, the dynamic photo editing device 70 may include: a display module 71 and an elimination module 72.

[0447] The display module 71 is configured to display an editing interface for a dynamic photo, wherein the dynamic photo includes a video. The editing interface includes a video frame display area, wherein the video frame display area includes a first video frame of the video; the first video frame includes at least one video object. The removal module 72 is configured to, in response to an object removal selection input for the first video frame, remove an object selected by the object removal selection input from each video frame of the video; wherein the object removal includes at least one video object.

[0448] In one possible implementation, the editing interface also includes a thumbnail picture display area, the thumbnail picture display area includes thumbnail pictures corresponding to at least two video frames of the video; the thumbnail picture display area includes a first thumbnail picture, the first thumbnail picture is a thumbnail picture corresponding to the first video frame; the first thumbnail picture also includes a thumbnail picture identifier, the thumbnail picture identifier is used to indicate the display position of the thumbnail picture corresponding to the video frame displayed in the video frame display area.

[0449] In a possible implementation, the display module 71 is further configured to display a first prompt message, where the first prompt message is configured to prompt the user to select an object to be eliminated.

[0450] In one possible implementation, the editing interface includes a first control; the dynamic photo editing device 70 further includes: a receiving module configured to receive a first input to the first control in response to an input for selecting an object to be removed from the first video frame, before removing the object selected by the input from each video frame of the video. A processing module configured to activate the dynamic photo removal function in response to the first input received by the receiving module. The display module further configured to display thumbnails and thumbnail identifiers of at least two video frames of the video in a thumbnail display area, and to display a third control on the editing interface.

[0451] In one possible implementation, the display module 71 is further configured to, after the receiving module receives the first input to the first control, display a prompt window including second prompt information and a second control, wherein the second prompt information indicates that the dynamic photo removal function has been activated. The processing module is further configured to close the prompt window in response to the second input to the second control.

[0452] In one possible implementation, the receiving module is further configured to receive a selection input for a first thumbnail image from the at least two thumbnail images when the second video frame of the video is displayed in the video frame display area and the thumbnail image identifier is displayed on the second thumbnail image. The processing module is further configured to, in response to the selection input received by the receiving module, move the thumbnail image identifier to the first thumbnail image and update the second video frame displayed in the video frame display area to the first video frame; wherein the second thumbnail image is the thumbnail image corresponding to the second video frame.

[0453] In one possible implementation, the input for selecting an object to be eliminated includes input for selecting a first video object in a first video frame; the dynamic photo editing device 70 further includes a determination module. The display module 71 is further configured to, upon receiving input for selecting the first video object to be eliminated in the first video frame, display a third prompt message, the third prompt message prompting the user to continue selecting an object to be eliminated. The determination module is configured to, upon receiving input for selecting a second video object to be eliminated in the first video frame, determine the first video object and the second video object as the objects to be eliminated.

[0454] In one possible implementation, the above-mentioned display module 71 is also used to display a first elimination object identifier on the first video object when receiving an elimination object selection input for the first video object in the first video frame, and the first elimination object identifier is used to indicate the object outline of the first video object; and to display a second elimination object identifier on the second video object when receiving an elimination object selection input for the second video object in the first video frame, and the second elimination object identifier is used to indicate the object outline of the second video object.

[0455] In one possible implementation, the video includes N video frames, where N is an integer greater than 1; the dynamic photo editing device also includes: a processing module, which is used to obtain a mask image corresponding to the first video frame based on the first video frame before the elimination module eliminates the elimination object in each video frame of the video and selects the elimination object selected by input; and based on the mask image corresponding to the first video frame and the image feature information of the elimination object in the first video frame, identify the display area of ​​the elimination object in each video frame of the M video frames of the video, and display an elimination object identifier on the elimination object in each video frame of the M video frames; wherein M is a positive integer, M≤N-1; each video frame of the M video frames includes the elimination object.

[0456] In one possible implementation, the elimination object is a first video object in the video; the processing module is specifically used to determine the object contour of the first video object from the first video frame based on the input trajectory of the elimination object selection input; and based on the object contour of the first video object, perform image segmentation on the first video frame; and perform binarization processing on the segmented first video frame to obtain a mask image corresponding to the first video frame.

[0457] In one possible implementation, the objects to be eliminated include K video objects, where K is an integer greater than 1; the above-mentioned processing module is specifically used to perform image segmentation processing on the first video frame based on the K video objects to obtain K mask images corresponding to the K video objects; and merge the K mask images to obtain a mask image corresponding to the first video frame.

[0458] In one possible implementation, the above-mentioned processing module is specifically used to: perform panoramic segmentation processing on each video frame of M video frames to obtain a mask image corresponding to each video object in each video frame and category information of each mask image, where the category information is used to characterize the object category of the eliminated object in each video frame; based on the image feature information of the eliminated object in the first video frame and the category information of the mask image corresponding to the first video frame, compare the image similarity between the mask image corresponding to the first video frame and the mask image corresponding to each video object in each video frame, and determine the mask image corresponding to each video frame of the M video frames; based on the mask image corresponding to each video frame of the M video frames, determine the display area of ​​the eliminated object in each video frame of the M video frames.

[0459] In one possible implementation, the editing interface further includes a thumbnail image display area, the thumbnail image display area includes thumbnail images corresponding to at least two video frames of the video; the first thumbnail image also includes a thumbnail image identifier; the display module 71 is further used to display a fourth prompt message during the process of identifying the display area of ​​the object to be eliminated in each of the M video frames of the video, the fourth prompt message being used to prompt the progress of identifying the display area of ​​the object to be eliminated in each of the M video frames of the video. The processing module is further used to move the thumbnail image identifier to the i-th thumbnail image when identifying the display area of ​​the object to be eliminated in the i-th video frame of the video, and update the first video frame displayed in the video frame display area to the i-th video frame; wherein the i-th thumbnail image is the thumbnail image corresponding to the i-th video frame, i is a positive integer, and i≤M.

[0460] In a possible implementation, the elimination module 72 is specifically configured to eliminate the elimination object selected by the elimination object selection input in each video frame of the video in response to a third input to the third control.

[0461] In a possible implementation, the dynamic photo editing device further includes: a processing module for displaying an elimination animation of the elimination object during the process of eliminating the object selection input selected for elimination in each video frame of the elimination video.

[0462] In one possible implementation, the above-mentioned editing interface also includes a fourth control; the dynamic photo editing device also includes: a processing module for performing processing of eliminating the object selected by the elimination object selection input in each video frame of the elimination video in response to the fourth input to the fourth control in the background, and updating the editing interface to a preview interface of the dynamic photo.

[0463] In one possible implementation, the display module 71 is further configured to display a fifth prompt message during the process of selecting and inputting the selected object to be eliminated in each video frame of the video, the fifth prompt message being used to indicate the progress of the object elimination. The processing module is further configured to, when eliminating the object to be eliminated in the i-th video frame of the video, move the elimination marker to the i-th thumbnail image; wherein the i-th thumbnail image is the thumbnail image corresponding to the i-th video frame, where i is a positive integer and i≤M.

[0464] In one possible implementation, the elimination module 72 is specifically used to: add preset noise to the first video frame to obtain a first noisy video frame; based on the first noisy video frame, the mask image corresponding to the first video frame, the video background image corresponding to the first video frame, the positive sample vector, the negative sample vector, and the prediction time step, predict the image noise of the object area of ​​the elimination object in the first noisy video frame to obtain a second noisy video frame, the positive sample vector is used to indicate that the video object contained in the mask image corresponding to the first video frame is a shooting foreground, the negative sample vector is used to characterize that the video object contained in the video background image corresponding to the first video frame is a shooting background, and the video background image does not include the elimination object. The object selection input selects an image of the elimination object; the second noisy video frame is denoised to obtain a third video frame corresponding to the first video frame; based on the mask image corresponding to the first video frame, the image to be filled is determined, the image to be filled is an image of an area in the third video frame corresponding to the background image area in the mask image of the first video frame, and the background image area is an image area corresponding to the elimination object selected by the elimination object selection input in the mask image of the first video frame; the image to be filled is filled into the video background image corresponding to the first video frame, and the elimination object selected by the elimination object selection input in the first video frame is eliminated, so as to eliminate the elimination object selected by the elimination object selection input in each video frame of the video.

[0465] In one possible implementation, the processing module is further used to: obtain a positive sample vector, negative sample data, and a first training image; randomly determine a first training video object from the first training image, and obtain a first training mask image corresponding to the first training image based on the first training video object; mark the image category corresponding to the first training mask image based on the image features, positive sample vector, and negative sample vector of the first training video object; the image category is either a shooting foreground or a shooting background; input the first training mask image, the noisy training image corresponding to the first training image, and the training background image corresponding to the first training image into the initial model; determine the image category corresponding to the first training mask image through the cross-attention unit in the initial model; predict the training image to be filled corresponding to the first training mask image based on the image category, the first training mask image, the noisy training image corresponding to the first training image, and the training background image corresponding to the first training image, and add the training image to be filled to the training background image to obtain a second training image; calculate the loss value between the second training image and the reference image, and adjust the model parameters of the original model according to the loss value to obtain a video elimination model.

[0466] In a possible implementation, the object selection elimination input includes at least one of the following: a fingertip touch input and a knuckle touch input.

[0467] In a possible implementation, when the elimination object selection input includes a knuckle touch input, the knuckle touch input includes an input of tapping the editing interface with the knuckle and circling an area along a reference direction from the tapping position.

[0468] In one possible implementation, the above-mentioned processing module is also used to hide the thumbnail images corresponding to at least two video frames of the video after eliminating the elimination object selected by the elimination object selection input in each video frame of the video, and display the fifth control and the sixth control in the editing interface; when the fifth input of the fifth control is received, the thumbnail images and thumbnail image identifiers of at least two video frames of the video are displayed in the thumbnail image display area, and the third control is displayed in the editing interface; when the sixth input of the third control is received, the elimination object selected by the elimination object selection input in each video frame of the video is eliminated; when the seventh input of the sixth control is received, the video after elimination processing is stored.

[0469] An embodiment of the present application provides a dynamic photo editing device. Since the user can input the selection of the elimination object for the first video frame, the electronic device can automatically determine the elimination object selected by the selection input from each video frame of the video. Therefore, the user does not need to select the elimination object in each video frame of the video in turn, which simplifies the steps for the user to select the elimination object in the video, thereby improving the efficiency of the electronic device in eliminating the elimination object in the video.

[0470] The dynamic photo editing device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a device other than a terminal. For example, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0471] The dynamic photo editing device in the embodiment of the present application can be a device having an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0472] The dynamic photo editing device provided in the embodiment of the present application can implement each process implemented in the above embodiment. To avoid repetition, it will not be repeated here.

[0473] Alternatively, as Figure 23 As shown, an embodiment of the present application further provides an electronic device 90, including a processor 91 and a memory 92, wherein the memory 92 stores a program or instruction that can be run on the processor 91, and when the program or instruction is executed by the processor 91, the various steps of the above-mentioned image method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0474] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0475] Figure 24 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0476] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101 , a network module 102 , an audio output unit 103 , an input unit 104 , a sensor 105 , a display unit 106 , a user input unit 107 , an interface unit 108 , a memory 109 , and a processor 110 .

[0477] Those skilled in the art will understand that the electronic device 100 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 24 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0478] The display unit 106 is configured to display an editing interface for a dynamic photo, wherein the dynamic photo includes a video. The editing interface includes a video frame display area, wherein the video frame display area includes a first video frame of the video; the first video frame includes at least one video object. The processor 110 is configured to, in response to an object selection input for the first video frame, remove the object selected by the object selection input from each video frame of the video; wherein the object includes at least one video object.

[0479] In some embodiments of the present application, the display unit 106 is further configured to display a first prompt message, where the first prompt message is configured to prompt the user to select an object to be eliminated.

[0480] In some embodiments of the present application, the above-mentioned editing interface includes a first control; the above-mentioned user input unit 107 is also used to respond to the elimination object selection input of the first video frame, and receive the first input to the first control before eliminating the elimination object selected by the elimination object selection input in each video frame of the video; the above-mentioned processor 110 is also used to respond to the first input, start the dynamic photo elimination function, display thumbnail images and thumbnail image identifiers of at least two video frames of the video in the thumbnail image display area, and display a third control on the editing interface.

[0481] In some embodiments of the present application, the above-mentioned display unit 106 is also used to, after receiving the first input to the first control, further include: displaying a prompt window, the prompt window including second prompt information and a second control, the second prompt information being used to prompt that the dynamic photo elimination function has been started; the above-mentioned processor 110 is also used to close the prompt window in response to the second input to the second control.

[0482] In some embodiments of the present application, when the second video frame of the video is displayed in the video frame display area and the thumbnail image identifier is displayed on the second thumbnail image, the above-mentioned user input unit 107 is also used to receive a selection input for a first thumbnail image among at least two thumbnail images; the above-mentioned processor 110 is also used to move the thumbnail image identifier to the first thumbnail image in response to the selection input, and update the second video frame displayed in the video frame display area to the first video frame; wherein the second thumbnail image is a thumbnail image corresponding to the second video frame.

[0483] In some embodiments of the present application, the elimination object selection input includes a selection input for a first video object in a first video frame; the above-mentioned display unit 106 is also used to display a third prompt information when receiving the elimination object selection input for the first video object in the first video frame, and the third prompt information is used to prompt the user to continue selecting the elimination object; the above-mentioned processor 110 is also used to determine the first video object and the second video object as elimination objects when receiving the elimination object selection input for the second video object in the first video frame.

[0484] In some embodiments of the present application, the above-mentioned display unit 106 is also used to display a first elimination object identifier on the first video object when receiving an elimination object selection input for the first video object in the first video frame, and the first elimination object identifier is used to indicate the object outline of the first video object; and to display a second elimination object identifier on the second video object when receiving an elimination object selection input for the second video object in the first video frame, and the second elimination object identifier is used to indicate the object outline of the second video object.

[0485] In some embodiments of the present application, a video includes N video frames, where N is an integer greater than 1; the above-mentioned processor 110 is also used to obtain a mask image corresponding to the first video frame based on the first video frame before selecting the elimination object in each video frame of the video; based on the mask image corresponding to the first video frame and the image feature information of the elimination object in the first video frame, identify the display area of ​​the elimination object in each video frame of the M video frames of the video, and display an elimination object identifier on the elimination object in each video frame of the M video frames; wherein M is a positive integer, M≤N-1; each video frame of the M video frames includes the elimination object.

[0486] In some embodiments of the present application, the elimination object is the first video object in the video; the above-mentioned processor 110 is specifically used to determine the object contour of the first video object from the first video frame based on the input trajectory of the elimination object selection input; based on the object contour of the first video object, perform image segmentation on the first video frame; and perform binarization processing on the first video frame after image segmentation to obtain a mask image corresponding to the first video frame.

[0487] In some embodiments of the present application, the elimination objects include K video objects, where K is an integer greater than 1; the above-mentioned processor 110 is specifically used to perform image segmentation processing on the first video frame based on the K video objects to obtain K mask images corresponding to the K video objects; and merge the K mask images to obtain a mask image corresponding to the first video frame.

[0488] In some embodiments of the present application, the above-mentioned processor 110 is specifically used to perform panoramic segmentation processing on each video frame of M video frames to obtain a mask image corresponding to each video object in each video frame and category information of each mask image, where the category information is used to characterize the object category of the eliminated object in each video frame; based on the image feature information of the eliminated object in the first video frame and the category information of the mask image corresponding to the first video frame, compare the image similarity of the mask image corresponding to the first video frame with the mask image corresponding to each video object in each video frame, and determine the mask image corresponding to each video frame of the M video frames; based on the mask image corresponding to each video frame of the M video frames, determine the display area of ​​the eliminated object in each video frame of the M video frames.

[0489] In some embodiments of the present application, the editing interface also includes a thumbnail image display area, and the thumbnail image display area includes thumbnail images corresponding to at least two video frames of the video; the first thumbnail image also includes a thumbnail image identifier; the above-mentioned display unit 106 is also used to display a fourth prompt information in the process of eliminating the display area of ​​the object in each of the M video frames of the identified video, and the fourth prompt information is used to prompt the identification progress of eliminating the display area of ​​the object in each of the M video frames of the identified video; the above-mentioned processor 110 is specifically used to move the thumbnail image identifier to the i-th thumbnail image when eliminating the display area of ​​the object in the i-th video frame of the identified video, and update the first video frame displayed in the video frame display area to the i-th video frame; wherein the i-th thumbnail image is the thumbnail image corresponding to the i-th video frame, i is a positive integer, i≤M.

[0490] In some embodiments of the present application, the processor 110 is specifically configured to eliminate the elimination object selected by the elimination object selection input in each video frame of the video in response to a third input to the third control.

[0491] In some embodiments of the present application, the processor 110 is further configured to display an elimination animation of the object during the process of selecting and inputting the selected object to be eliminated in each video frame of the elimination video.

[0492] In some embodiments of the present application, the editing interface also includes a fourth control; the above-mentioned processor 110 is also used to, in the process of eliminating the object selected by the elimination object selection input in each video frame of the elimination video, respond to the fourth input to the fourth control, execute the processing of eliminating the object selected by the elimination object selection input in each video frame of the elimination video in the background, and update the editing interface to a preview interface of the dynamic photo.

[0493] In some embodiments of the present application, the above-mentioned display unit 106 is also used to display a fifth prompt message during the process of selecting and inputting the selected elimination object in each video frame of the elimination video, and the fifth prompt message is used to prompt the elimination progress of the elimination object; the above-mentioned processor 110 is also used to move the elimination mark to the i-th thumbnail picture when eliminating the elimination object in the i-th video frame of the video; wherein the i-th thumbnail picture is the thumbnail picture corresponding to the i-th video frame, i is a positive integer, i≤M.

[0494] In some embodiments of the present application, the above-mentioned processor 110 is specifically used to add preset noise to the first video frame to obtain a first noisy video frame; based on the first noisy video frame, the mask image corresponding to the first video frame, the video background image corresponding to the first video frame, the positive sample vector, the negative sample vector, and the prediction time step, predict the image noise of the object area of ​​the object to be eliminated in the first noisy video frame to obtain a second noisy video frame, the positive sample vector is used to indicate that the video object contained in the mask image corresponding to the first video frame is a shooting foreground, the negative sample vector is used to characterize that the video object contained in the video background image corresponding to the first video frame is a shooting background, and the video background image does not include the elimination object. The object selection input selects an image of the elimination object; the second noisy video frame is denoised to obtain a third video frame corresponding to the first video frame; based on the mask image corresponding to the first video frame, the image to be filled is determined, the image to be filled is an image of an area in the third video frame corresponding to the background image area in the mask image of the first video frame, and the background image area is an image area corresponding to the elimination object selected by the elimination object selection input in the mask image of the first video frame; the image to be filled is filled into the video background image corresponding to the first video frame, and the elimination object selected by the elimination object selection input in the first video frame is eliminated, so as to eliminate the elimination object selected by the elimination object selection input in each video frame of the video.

[0495] In some embodiments of the present application, the processor 110 is further configured to obtain a positive sample vector, negative sample data, and a first training image; randomly determine a first training video object from the first training image, and obtain a first training mask image corresponding to the first training image based on the first training video object; mark the image category corresponding to the first training mask image based on the image features, positive sample vector, and negative sample vector of the first training video object; the image category is either a shooting foreground or a shooting background; input the first training mask image, the noisy training image corresponding to the first training image, and the training background image corresponding to the first training image into the initial model; determine the image category corresponding to the first training mask image through the cross-attention unit in the initial model; predict the training image to be filled corresponding to the first training mask image based on the image category, the first training mask image, the noisy training image corresponding to the first training image, and the training background image corresponding to the first training image, and add the training image to be filled to the training background image to obtain a second training image; calculate the loss value between the second training image and the reference image, and adjust the model parameters of the original model according to the loss value to obtain a video elimination model.

[0496] In some embodiments of the present application, the above-mentioned display unit 106 is also used to hide the thumbnail images corresponding to at least two video frames of the video after eliminating the elimination object selected by the elimination object selection input in each video frame of the video, and display the fifth control and the sixth control in the editing interface; when the fifth input of the fifth control is received, the thumbnail images and thumbnail image identifiers of at least two video frames of the video are displayed in the thumbnail image display area, and the third control is displayed in the editing interface; when the sixth input of the third control is received, the elimination object selected by the elimination object selection input in each video frame of the video is eliminated; when the seventh input of the sixth control is received, the video after elimination processing is stored.

[0497] An embodiment of the present application provides an electronic device. Since the user can input the selection of the elimination object for the first video frame, the electronic device can automatically determine the elimination object selected by the selection input from each video frame of the video. Therefore, the user does not need to select the elimination object in each video frame of the video in turn, which simplifies the steps for the user to select the elimination object in the video, thereby improving the efficiency of the electronic device in eliminating the elimination object in the video.

[0498] The electronic device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0499] The beneficial effects of various implementations in this embodiment can be specifically referred to the beneficial effects of the corresponding implementations in the above method embodiment. To avoid repetition, they will not be described here.

[0500] It should be understood that in an embodiment of the present application, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0501] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0502] Processor 110 may include one or more processing units. Optionally, processor 110 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 110.

[0503] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0504] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0505] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0506] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0507] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image elimination method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0508] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0509] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0510] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A dynamic photo editing method, characterized in that: include: Displaying an editing interface for a dynamic photo, the dynamic photo including a video, the editing interface including a video frame display area, the video frame display area including a first video frame of the video; the first video frame including at least one video object; In response to the elimination object selection input for the first video frame, the elimination object selected by the elimination object selection input in each video frame of the video is eliminated; wherein the elimination object includes at least one video object.

2. The method according to claim 1, characterized in that The editing interface further includes a thumbnail display area, wherein the thumbnail display area includes thumbnails corresponding to at least two video frames of the video; The thumbnail display area includes a first thumbnail, and the first thumbnail is a thumbnail corresponding to the first video frame; The first thumbnail also includes a thumbnail identifier, and the thumbnail identifier is used to indicate the display position of the thumbnail corresponding to the video frame displayed in the video frame display area.

3. The method according to claim 2, characterized in that The method further comprises: The first prompt information is displayed, where the first prompt information is used to prompt the user to select an elimination object.

4. The method according to claim 2, characterized in that The editing interface includes a first control; Before eliminating the elimination object selected by the elimination object selection input in response to the elimination object selection input of the first video frame in each video frame of the video, the method further includes: receiving a first input to the first control; In response to the first input, a dynamic photo removal function is started, thumbnails and thumbnail identifiers of at least two video frames of the video are displayed in the thumbnail display area, and a third control is displayed in the editing interface.

5. The method according to claim 4, characterized in that After receiving the first input to the first control, the method further includes: Displaying a prompt window, the prompt window including a second prompt message and a second control, wherein the second prompt message is used to prompt that the dynamic photo removal function has been activated; In response to a second input to the second control, the prompt window is closed.

6. The method according to claim 2 or 4, characterized in that In a case where the second video frame of the video is displayed in the video frame display area, and the thumbnail identifier is displayed on the second thumbnail, the method further includes: receiving a selection input of a first thumbnail among the at least two thumbnails; In response to the selection input, the thumbnail marker is moved to the first thumbnail, and the second video frame displayed in the video frame display area is updated to the first video frame; The second thumbnail is a thumbnail corresponding to the second video frame.

7. The method according to claim 1, characterized in that The elimination object selection input includes a selection input of a first video object in the first video frame; The method further comprises: When receiving an input for selecting an object to be eliminated for the first video object in the first video frame, displaying third prompt information, wherein the third prompt information is used to prompt the user to continue selecting an object to be eliminated; In a case where an elimination object selection input for the second video object in the first video frame is received, the first video object and the second video object are determined as elimination objects.

8. The method according to claim 1, characterized in that In case that an object elimination selection input is received for the first video object in the first video frame, the method further comprises: Displaying a first elimination object identifier on the first video object, where the first elimination object identifier is used to indicate an object outline of the first video object; In case that an object elimination selection input for a second video object in the first video frame is received, the method further comprises: A second elimination object identifier is displayed on the second video object, where the second elimination object identifier is used to indicate an object outline of the second video object.

9. The method according to claim 1, characterized in that The video includes N video frames, where N is an integer greater than 1; Before the step of eliminating the elimination object in each video frame of the video and selecting and inputting the selected elimination object, the method further comprises: Based on the first video frame, obtaining a mask image corresponding to the first video frame; Based on the mask image corresponding to the first video frame and the image feature information of the elimination object in the first video frame, identifying a display area of ​​the elimination object in each of the M video frames of the video, and displaying an elimination object identifier on the elimination object in each of the M video frames; Wherein, M is a positive integer, M≤N-1; each of the M video frames includes the elimination object.

10. The method according to claim 9, characterized in that The elimination object is the first video object in the video; The obtaining, based on the first video frame, a mask image corresponding to the first video frame includes: determining an object contour of the first video object from the first video frame based on the input trajectory of the elimination object selection input; performing image segmentation on the first video frame based on an object contour of the first video object; The first video frame after image segmentation is binarized to obtain a mask image corresponding to the first video frame.

11. The method according to claim 9, characterized in that The elimination objects include K video objects, where K is an integer greater than 1; The obtaining, based on the first video frame, a first mask image corresponding to the first video frame includes: Based on the K video objects, performing image segmentation processing on the first video frame to obtain K mask images corresponding to the K video objects; The K mask images are merged to obtain a mask image corresponding to the first video frame.

12. The method according to claim 9, characterized in that The determining, based on the first mask image corresponding to the first video frame and the image feature information of the elimination object in the first video frame, a display area of ​​the elimination object in each of the M video frames of the video includes: Performing panoptic segmentation processing on each of the M video frames to obtain a mask image corresponding to each video object in each video frame and category information of each mask image, wherein the category information is used to characterize an object category of the eliminated object in each video frame; comparing, based on image feature information of the object to be eliminated in the first video frame and category information of the mask image corresponding to the first video frame, the mask image corresponding to the first video frame with the mask image corresponding to each video object in each video frame for image similarity, and determining a mask image corresponding to each of the M video frames; Based on the mask image corresponding to each of the M video frames, a display area of ​​the elimination object in each of the M video frames is determined.

13. The method according to claim 9, characterized in that The editing interface further includes a thumbnail display area, wherein the thumbnail display area includes thumbnails corresponding to at least two video frames of the video; the first thumbnail also includes a thumbnail mark; The method further comprises: During the process of identifying the display area of ​​the elimination object in each of the M video frames of the video, displaying fourth prompt information, wherein the fourth prompt information is used to prompt the identification progress of the display area of ​​the elimination object in each of the M video frames of the video; When the display area of ​​the elimination object in the i-th video frame of the video is identified, the thumbnail identifier is moved to the i-th thumbnail, and the first video frame displayed in the video frame display area is updated to the i-th video frame; The i-th thumbnail is the thumbnail corresponding to the i-th video frame, i is a positive integer, and i≤M.

14. The method according to claim 4, characterized in that The step of eliminating the elimination object in each video frame of the video and selecting and inputting the selected elimination object comprises: In response to a third input to the third control, the elimination object selected by the elimination object selection input in each video frame of the video is eliminated.

15. The method according to claim 1, wherein The method further comprises: In the process of eliminating the elimination object selected by the elimination object selection input in each video frame of the video, the elimination animation of the elimination object is displayed.

16. The method according to claim 1, wherein The editing interface also includes a fourth control; In the process of eliminating the elimination object selected by the elimination object selection input in each video frame of the video, the method further includes: In response to the fourth input to the fourth control, a process of eliminating the elimination object selected by the elimination object selection input in each video frame of the video is performed in the background, and the editing interface is updated to a preview interface of the dynamic photo.

17. The method according to claim 9, characterized in that The method further comprises: During the process of eliminating the elimination object selected by the elimination object selection input in each video frame of the video, fifth prompt information is displayed, wherein the fifth prompt information is used to prompt the elimination progress of the elimination object; When the elimination object in the i-th video frame of the video is eliminated, moving the elimination mark to the i-th thumbnail; The i-th thumbnail is the thumbnail corresponding to the i-th video frame, i is a positive integer, and i≤M.

18. The method according to claim 9, characterized in that The step of eliminating the elimination object in each video frame of the video and selecting and inputting the selected elimination object comprises: Adding preset noise to the first video frame to obtain a first noisy video frame; Predicting image noise of an object region of the object to be eliminated in the first noisy video frame based on the first noisy video frame, a mask image corresponding to the first video frame, a video background image corresponding to the first video frame, a positive sample vector, a negative sample vector, and a prediction time step, to obtain a second noisy video frame, wherein the positive sample vector is used to indicate that the video object contained in the mask image corresponding to the first video frame is a shooting foreground, the negative sample vector is used to indicate that the video object contained in the video background image corresponding to the first video frame is a shooting background, and the video background image is an image that does not include the object to be eliminated selected by the object to be eliminated selection input; De-noising the second noisy video frame to obtain a third video frame corresponding to the first video frame; Determining an image to be filled based on the mask image corresponding to the first video frame, the image to be filled being an image of an area in the third video frame corresponding to a background image area in the mask image of the first video frame, the background image area being an image area in the mask image of the first video frame corresponding to the elimination object selected by the elimination object selection input; Fill the image to be filled into the video background image corresponding to the first video frame, eliminate the elimination object selected by the elimination object selection input in the first video frame, and eliminate the elimination object selected by the elimination object selection input in each video frame of the video.

19. The method according to claim 9, characterized in that The method further comprises: Obtain a positive sample vector, a negative sample vector, and a first training image; Randomly determining a first training video object from the first training image, and obtaining a first training mask image corresponding to the first training image based on the first training video object; Based on the image features of the first training video object, the positive sample vector, and the negative sample vector, marking the image category corresponding to the first training mask image; the image category is either a shooting foreground or a shooting background; Inputting the first training mask image, the noisy training image corresponding to the first training image, and the training background image corresponding to the first training image into an initial model; Determining, by a cross-attention unit in the initial model, an image category corresponding to the first training mask image; Based on the image category, the first training mask image, the noisy training image corresponding to the first training image, and the training background image corresponding to the first training image, predicting a to-be-filled training image corresponding to the first training mask image, and adding the to-be-filled training image to the training background image to obtain a second training image; A loss value between the second training image and the reference image is calculated, and the model parameters of the original model are adjusted according to the loss value to obtain the video elimination model.

20. The method according to claim 1, wherein The elimination object selection input includes at least one of the following: fingertip touch input and knuckle touch input.

21. The method according to claim 20, characterized in that In a case where the elimination object selection input includes the knuckle touch input, the knuckle touch input includes an input of tapping the editing interface with a knuckle and circling an area along a reference direction from the tapped position.

22. The method according to claim 15, wherein After selecting and inputting the selected object to be eliminated from each video frame of the video, the method further includes: Hiding thumbnails corresponding to at least two video frames of the video, and displaying a fifth control and a sixth control in the editing interface; In the case of receiving a fifth input of the fifth control, displaying thumbnails and thumbnail identifiers of at least two video frames of the video in the thumbnail display area, and displaying a third control on the editing interface; In case of receiving a sixth input to the third control, eliminating the elimination object selected by the elimination object selection input in each video frame of the video; When a seventh input of the sixth control is received, the video after the elimination process is stored.

23. A dynamic photo editing device, characterized in that: include: A display module is configured to display an editing interface for a dynamic photo, wherein the dynamic photo includes a video, the editing interface includes a video frame display area, and the video frame display area includes a first video frame of the video; the first video frame includes at least one video object; The elimination module is configured to eliminate the elimination object selected by the elimination object selection input in each video frame of the video in response to the elimination object selection input of the first video frame; wherein the elimination object includes at least one video object.

24. The device according to claim 23, characterized in that The editing interface further includes a thumbnail display area, wherein the thumbnail display area includes thumbnails corresponding to at least two video frames of the video; The thumbnail display area includes a first thumbnail, and the first thumbnail is a thumbnail corresponding to the first video frame; The first thumbnail also includes a thumbnail identifier, and the thumbnail identifier is used to indicate the display position of the thumbnail corresponding to the video frame displayed in the video frame display area.

25. The device according to claim 24, characterized in that The display module is further configured to display first prompt information, where the first prompt information is configured to prompt the user to select an object to be eliminated.

26. The device according to claim 24, characterized in that The editing interface includes a first control; the dynamic photo editing device also includes: a receiving module configured to receive a first input to the first control before eliminating the elimination object selected by the elimination object selection input in each video frame of the video in response to the elimination object selection input of the first video frame; a processing module, configured to activate a dynamic photo removal function in response to the first input received by the receiving module; The display module is further configured to display thumbnails and thumbnail identifiers of at least two video frames of the video in the thumbnail display area, and to display a third control in the editing interface.

27. The device according to claim 26, characterized in that The display module is further configured to display a prompt window after the receiving module receives the first input to the first control, wherein the prompt window includes second prompt information and a second control, and the second prompt information is used to prompt that the dynamic photo removal function has been activated; The processing module is further configured to close the prompt window in response to a second input to the second control.

28. The method according to claim 24 or 26, characterized in that The receiving module is further configured to receive a selection input for a first thumbnail among the at least two thumbnails when a second video frame of the video is displayed in the video frame display area and the thumbnail identifier is displayed on the second thumbnail; The processing module is further configured to, in response to the selection input received by the receiving module, move the thumbnail identifier to the first thumbnail, and update the second video frame displayed in the video frame display area to the first video frame; The second thumbnail is a thumbnail corresponding to the second video frame.

29. The device according to claim 23, characterized in that The elimination object selection input includes a selection input of a first video object in the first video frame; the dynamic photo editing device further includes a determination module; The display module is further configured to display third prompt information when receiving an input for selecting an object to be eliminated for the first video object in the first video frame, wherein the third prompt information is configured to prompt the user to continue selecting an object to be eliminated; The determining module is configured to determine the first video object and the second video object as elimination objects when receiving an elimination object selection input for the second video object in the first video frame.

30. The device according to claim 23, wherein The display module is further configured to display a first elimination object identifier on the first video object when receiving an elimination object selection input for the first video object in the first video frame, wherein the first elimination object identifier is used to indicate an object outline of the first video object; When an elimination object selection input for a second video object in the first video frame is received, a second elimination object identifier is displayed on the second video object, where the second elimination object identifier is used to indicate an object outline of the second video object.

31. The device according to claim 23, characterized in that The video includes N video frames, where N is an integer greater than 1; and the dynamic photo editing device further includes: A processing module, configured to obtain, based on the first video frame, a mask image corresponding to the first video frame before the elimination module eliminates the elimination object in each video frame of the video and selects the elimination object selected by input; and identifying a display area of ​​the elimination object in each of the M video frames of the video based on the mask image corresponding to the first video frame and image feature information of the elimination object in the first video frame, and displaying an elimination object identifier on the elimination object in each of the M video frames; Wherein, M is a positive integer, M≤N-1; each of the M video frames includes the elimination object.

32. The device according to claim 31, characterized in that The elimination object is the first video object in the video; The processing module is specifically used to determine the object contour of the first video object from the first video frame based on the input trajectory of the elimination object selection input; and perform image segmentation on the first video frame based on the object contour of the first video object; and perform binarization processing on the first video frame after image segmentation to obtain a mask image corresponding to the first video frame.

33. The device according to claim 31, characterized in that The elimination objects include K video objects, where K is an integer greater than 1; The processing module is specifically configured to perform image segmentation processing on the first video frame based on the K video objects to obtain K mask images corresponding to the K video objects; and merge the K mask images to obtain a mask image corresponding to the first video frame.

34. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the dynamic photo editing method as described in any one of claims 1 to 22.

35. A readable storage medium storing a program or instruction, wherein the program or instruction, when executed by a processor, implements the steps of the dynamic photo editing method according to any one of claims 1 to 22.

Images