Media editing method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to a media editing method and device, equipment and a storage medium. The method provided by the invention comprises the following steps: presenting a media editing interface of first media content; based on the received adjustment request, identifying a to-be-edited target area in the first media content in the media editing interface; presenting at least one adjustment element associated with the target area in the media editing interface based on the received editing operation; in the media editing interface, second media content is presented, and the second media content is generated by adjusting the first media content according to the direction information and / or the position information indicated by the at least one adjusting element.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for media editing. Background Art

[0002] With the development of computer technology, the field of media editing has also undergone many technological changes, from traditional manual editing to the introduction of multiple models for assisted editing. Traditional media editing technology can usually only edit preset areas. Summary of the invention

[0003] In a first aspect of the present disclosure, a media editing method is provided. The method includes: presenting a media editing interface of a first media content; based on a received adjustment request, identifying a target area to be edited in the first media content in the media editing interface; based on a received editing operation, presenting at least one adjustment element associated with the target area in the media editing interface; and presenting second media content in the media editing interface, the second media content being generated by adjusting the first media content according to direction information and / or position information indicated by the at least one adjustment element.

[0004] In a second aspect of the present disclosure, a device for media editing is provided. The device includes: a first presentation module configured to present a media editing interface of a first media content; an identification module configured to identify a target area to be edited in the first media content in the media editing interface based on a received adjustment request; a second presentation module configured to present at least one adjustment element associated with the target area in the media editing interface based on a received editing operation; and a third presentation module configured to present second media content in the media editing interface, the second media content being generated by adjusting the first media content according to direction information and / or position information indicated by at least one adjustment element.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0010] FIG. 2A to FIG. 2H Schematic diagrams showing multiple example media editing interfaces according to some embodiments of the present disclosure;

[0011] Figure 3 A structural diagram of a model for media editing according to some embodiments of the present disclosure is shown;

[0012] Figure 4 A flowchart showing a process for media editing according to some embodiments of the present disclosure is shown;

[0013] Figure 5 A block diagram showing an apparatus for media editing according to some embodiments of the present disclosure; and

[0014] Figure 6 A block diagram of an electronic device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0015] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0016] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0017] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0018] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0019] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0020] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of executing a subsequent action executed in response to the event or condition is not necessarily strongly related to the time when the event occurs or the condition is satisfied. For example, in some cases, the subsequent action may be executed immediately when the event occurs or the condition is satisfied; while in other cases, the subsequent action may be executed some time after the event occurs or the condition is satisfied.

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section / subsection. In addition, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0023] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0024] As briefly mentioned above, with the development of computer technology, the field of media editing has also undergone many technological changes, from traditional manual editing to the introduction of multiple models for assisted editing. Traditional media editing technology is usually only able to edit preset areas.

[0025] To this end, an embodiment of the present disclosure proposes a solution for media editing. According to various embodiments of the present disclosure, a media editing interface of a first media content is presented; based on a received adjustment request, a target area to be edited in the first media content is identified in the media editing interface; based on a received editing operation, at least one adjustment element associated with the target area is presented in the media editing interface; and in the media editing interface, second media content is presented, the second media content being generated by adjusting the first media content according to direction information and / or position information indicated by at least one adjustment element.

[0026] In this way, the embodiments of the present disclosure can receive the user's adjustment operation (e.g., dragging) on ​​the specified area of ​​the media content to achieve automatic adjustment of the content in the area (e.g., position adjustment or orientation adjustment). Thus, the embodiments of the present disclosure can support controllable editing of media content and improve the editing efficiency of media content.

[0027] Example embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0028] Example Environment

[0029] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, example environment 100 may include electronic device 110 .

[0030] In this example environment 100, the electronic device 110 may run an application 120 for media editing. The application 120 may be any suitable type of application for editing media content, examples of which may include but are not limited to: a picture editing application, a video editing application, a content sharing application, etc. The user 140 may interact with the application 120 via the electronic device 110 and / or its attached devices.

[0031] exist Figure 1 In the environment 100, if the application 120 is in an active state, the electronic device 110 may present an interface 150 corresponding to the application 120. The interface 150 may include various pages provided by the application 120, such as an editing page for media content. For example, the editing interface for media content may display the media content to be edited and a plurality of controls for performing media editing, and the user 140 may select a corresponding control to edit the media content.

[0032] In some embodiments, the electronic device 110 communicates with the server 130 to provide services for the application 120. The server 150 can provide management, configuration, maintenance of applications or websites, and identification of target objects in picture content.

[0033] The electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable game terminals, VR / AR devices, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDA), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, game devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface for the user (such as "wearable" circuits, etc.).

[0034] The server 130 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The server 130 may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, etc. The server 130 may provide background services for the application 120 that supports virtual scenes in the electronic device 110.

[0035] A communication connection may be established between the server 130 and the electronic device 110. The communication connection may be established in a wired manner or a wireless manner. The communication connection may include, but is not limited to, a Bluetooth connection, a mobile network connection, a Universal Serial Bus (USB) connection, a Wireless Fidelity (WiFi) connection, etc., and the embodiments of the present disclosure are not limited in this respect. In the embodiments of the present disclosure, the server 130 and the electronic device 110 may implement signaling interaction through the communication connection between the two.

[0036] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0037] Various example implementations of the present disclosure will be described in detail below.

[0038] Sample Media Editing

[0039] The following will refer to FIG. 2A to FIG. 2H An example media editing process according to an embodiment of the present disclosure is described. FIG. 2A to FIG. 2H Example live broadcast interfaces 200A to 200H according to some embodiments of the present disclosure are shown. Such example media editing interfaces can be implemented by appropriate electronic devices or combinations of electronic devices (e.g., Figure 1 The electronic device 110, the server 130, or a combination of one or more of these devices).

[0040] In some scenarios, the electronic device 110 may present Figure 2A The display interface 200A of the first media content shown in the figure, when presenting the display interface 200A, the electronic device 110 can automatically select the first media content 201. The interface 200A includes the first media content 201 and a jump control 202 for jumping to the media editing interface. The electronic device 110 can jump to the media editing interface in response to the user 140 selecting the jump control 202.

[0041] In some embodiments, the electronic device 110 presents a media editing interface for the first media content.

[0042] As an example, Figure 2B As shown, the electronic device 110 may present a media editing interface 200B on the screen. The media editing interface 200B includes a plurality of controls for editing media content. The controls for media content may include, for example, an identification control 203.

[0043] In some embodiments, the electronic device 110 identifies the target area in the first media content in the media editing interface based on the received adjustment request. Figure 2B As shown, the electronic device 110 may identify a target area corresponding to a target object in the first media content in response to receiving the adjustment request.

[0044] In some embodiments, the adjustment request is used to adjust the orientation and / or position of the target object in the target area in the first media content. As an example, the electronic device 110 can adjust the orientation and / or position of the target object in the first media content in the first media content based on the adjustment request. For example, the electronic device 110 can adjust the orientation of the body of the character object, or can change the position of the object object in the first media content.

[0045] In some embodiments, the electronic device 110 may present candidate regions of the first media content corresponding to the target object in response to the selection of the identification control in the media editing interface. Further, the electronic device 110 may determine the target region to be edited based on the candidate regions.

[0046] As an example, Figure 2B As shown, the electronic device 110 can identify the target object in the first media content according to the selection of the identification control 203 by the user 140 to identify the candidate area corresponding to the target object (the area marked by the dotted line). Further, the electronic device 110 can use the candidate area corresponding to the target object as the target area. In some scenes, the target object can be, for example, an entity such as a person, plant or animal in the first media content. In other scenes, when jumping to the editing interface 200B, the electronic device 110 can automatically identify the candidate area where the target object in the first media content is located. In this way, the target object in the media content can be quickly selected, thereby improving the efficiency of media content editing.

[0047] In some embodiments, the electronic device 110 may provide an editing control for adjusting the candidate region in the media editing interface, the editing control including an erase control and / or a brush control, the erase control is used to delete a portion of the candidate region, and the brush control is used to adjust the boundary of the candidate region. Figure 2C As shown, the editing controls in the media editing interface include a brush control 204 and an erase control 205. The brush control 204 can be used to adjust the boundaries of the candidate region, for example, to expand or reduce the candidate region by adjusting the boundaries of the candidate region. The brush control 204 can also select other regions outside the candidate region to expand the candidate region.

[0048] In some embodiments, the electronic device 110 may use the editing control to adjust the candidate area to determine the target area to be edited.

[0049] In some scenarios, the electronic device 110 may present a first indicator associated with the brush control 204 in the media editing interface in response to the user 140 selecting the brush control 204. The electronic device 110 may receive the user 140 adjusting the candidate area through the first indicator and Figure 2B The candidate regions in are adjusted as follows Figure 2C The area marked by the dotted line.

[0050] In other scenarios, the electronic device 110 may present a second indicator associated with the erase control 205 in the media editing interface in response to the user 140 selecting the erase control 205. The electronic device 110 may receive a deletion operation of the user 140 on a portion of the candidate area through the second indicator, and Figure 2B The candidate regions in are adjusted as follows Figure 2D The area marked by the dotted line.

[0051] In some embodiments, the electronic device 110 presents at least one adjustment element associated with the target area in the media editing interface based on the received editing operation. Figure 2E As shown, the electronic device 110 can add at least one candidate area in the media editing interface 200E based on the received editing operation. Figure 2E Adjustment elements shown.

[0052] In some scenarios, such as Figure 2E As shown, the media editing interface 200E also includes a change selection area control 207. In response to the user 140 selecting the change selection area control 207, the electronic device 110 can switch the area currently being edited.

[0053] In some embodiments, the adjustment element includes an arrow element pointing from the starting position to the ending position, and the arrow element is used to indicate that the target key point corresponding to the starting position in the media content is adjusted from the starting position to the ending position. Figure 2E It can be represented by an arrow in the figure, or by other elements that can indicate direction.

[0054] In some embodiments, the electronic device 110 may determine a starting position and an ending position based on the received editing operation.

[0055] As an example, Figure 2B and 2E As shown, the electronic device 110 may receive an adjustment operation for the candidate area. Figure 2B The candidate regions in are adjusted to Figure 2EAfter determining the target area, the electronic device 110 may trigger an event of adding an arrow in response to the user 140 selecting the steering control 206. Further, the electronic device 110 may receive an editing operation of the user 140 on the target area and determine the starting position and the ending position of the editing operation.

[0056] Furthermore, the electronic device 110 may draw at least one adjustment element in the media editing interface based on the start position and the end position.

[0057] As an example, Figure 2E As shown, after determining the start position and end position of the editing operation, the electronic device 110 can draw at least one adjustment element at the target area in the media editing interface. The adjustment element points from the start position of the editing operation to the end position to indicate the direction in which the first media content is adjusted.

[0058] As another example, Figure 2E As shown, the media editing interface 200E also includes a generation control 208, which is used to trigger an adjustment event for adjusting the first media content based on the adjustment element. The generation control 208 includes text prompt information to indicate that the user 140 can trigger the adjustment event by selecting the generation control 208. The generation control 208 also includes count information for indicating the number of adjustment elements. Figure 2E As shown, the media editing interface 200E includes two adjustment elements, and the count information of the generated control 208 is the same as the number of the adjustment elements, that is, the count information is 2.

[0059] In some embodiments, the electronic device 110 may adjust the starting position and / or ending position of the adjustment element based on the modification operation for the adjustment element. As an example, the electronic device 110 may determine the starting position and ending position of the adjustment element indicated by the modification operation based on the modification operation to modify the position and orientation of the adjustment element.

[0060] In some embodiments, the electronic device 110 presents the second media content in the media editing interface, where the second media content is generated by adjusting the first media content according to the direction information and / or position information indicated by the at least one adjustment element. Figure 2E and 2F As shown, the electronic device 110 can be based on Figure 2E The adjustment element in the media content is adjusted from the starting position to the ending position of the adjustment element to generate Figure 2F The second media content 209.

[0061] In some embodiments, the electronic device 110 may display the first media content and the second media content in comparison in response to the selection of the comparison control in the media editing interface. Figure 2F As shown, the electronic device 110 may switch the second media content 209 to the first media content 201 in response to the user 140 selecting the comparison control 210 , thereby achieving a differential comparison between the first media content and the second media content.

[0062] In some embodiments, the second media content includes a first portion corresponding to the target area and a second portion corresponding to the remaining area, and the content of the second portion is the same as the content of the third portion corresponding to the remaining area in the first media content. As an example, when the electronic device 110 adjusts the first media content, it only adjusts the content in the target area, and the content outside the target area is not affected by the adjustment event.

[0063] In some embodiments, the electronic device 110 may determine an additional region to be edited in the second media content in response to receiving a second adjustment request for the second media content. Figure 2G As shown, after generating the second media content, the electronic device 110 may determine an additional area to be edited in the second media content in response to the user 140 operating the second media content through the brush control 204 .

[0064] Further, the electronic device 110 may present at least one additional adjustment element associated with the additional area in the media editing interface based on the received additional editing operation. Figure 2G As shown, the electronic device 110 may receive an additional editing operation of the user 140 in the second area, and based on the additional editing operation, draw an additional adjustment element corresponding to the additional editing operation in the second area.

[0065] Finally, the electronic device 110 may present the third media content in the media editing interface, where the third media content is generated by adjusting the second media content according to the additional direction information and / or additional position information indicated by the at least one additional adjustment element. As an example, the electronic device 110 may adjust the second media content based on the start position and the end position of the additional adjustment element to generate the third media content.

[0066] In this way, the user can expand or modify the target area to be edited in the first media content through the adjustment request. Thus, the embodiments of the present disclosure can enable the user to adjust the media content more conveniently.

[0067] The following will further combine Figure 3 To describe the process of generating media content. Figure 3An example architecture of an example generative model 300 is shown, in accordance with some embodiments of the present disclosure.

[0068] like Figure 3 As shown, the generation model 300 may obtain a reference image 305. Taking the use of the first media content 201 to generate the second media content 209 as an example, the reference image 305 may be the edited first media content 201.

[0069] As an example, the generation model 300 may include an image appearance encoding unit 310 , which may extract image features of the reference image 305 .

[0070] As another example, the generation model 300 may further include a text adapter 315 to encode appearance description text of the reference image 305 .

[0071] Additionally, the generation model 300 may also obtain mask information 320-2 corresponding to the target area to be edited, and apply the mask information 320-2 to the mask image 320-1 generated by the reference image 305. In addition, the generation model 300 may also obtain preset noise 320-3.

[0072] Furthermore, the generation model 300 may also generate a control signal based on the reference point 330 and the target point 335. Specifically, the generation model 300 may include a point encoding unit 340 to encode the reference point 330 and the target point 335.

[0073] As an example, the reference point 330 may be the starting position of the adjustment element (eg, arrow element) discussed above; the target point may be the ending position of the adjustment element discussed above.

[0074] As an example, the generation model 300 may utilize the backbone network to process the above input information to generate an input image 345. Thus, the generation model 300 may generate an output image 345 (eg, second media content) based on the reference image 305 and the control signal for indicating the reference point 330 and the target point 335.

[0075] In some embodiments, the generation model 300 can be trained using training video data. Specifically, the first image frame and the second image frame can be extracted from the training video data. Further, a training control signal can be generated based on the first image frame and the second image frame. In some embodiments, the training control signal can be determined based on the first position information of a group of key points in the first image frame and the second position information of a group of key points in the second image frame, and the group of key points can be associated with a preset object in the training video data.

[0076] For example, a first group of positions of a set of key points of a preset object can be determined based on a first image frame, and a second group of positions of corresponding key points can be determined based on a second image frame. Further, a training control signal can be constructed based on the first group of positions and the corresponding second group of positions.

[0077] Similar to Figure 3 In the process shown, the generation model 300 can generate an intermediate image frame based on the first image frame, the training mask information corresponding to the preset object and the training control signal. Further, the generation model 300 can be trained based on the intermediate image frame and the second image frame.

[0078] In this way, the embodiments of the present disclosure can utilize the position changes of key points in the video to train a generation model for adjusting the orientation or position of an entity, thereby reducing the training cost of the model.

[0079] Based on the process described above, the embodiments of the present disclosure can receive the user's adjustment operation (e.g., dragging) on ​​the specified area of ​​the media content to achieve automatic adjustment of the content in the area (e.g., position adjustment or orientation adjustment). Thus, the embodiments of the present disclosure can support controllable editing of media content and improve the editing efficiency of media content.

[0080] Example Process

[0081] Figure 4 4 is a flowchart showing an example process 400 of media editing according to some embodiments of the present disclosure. The process 400 may be implemented at the terminal device 110. Figure 1 4. The process 400 is described below.

[0082] like Figure 4 As shown, in box 410, the electronic device 110 presents a media editing interface for the first media content.

[0083] In block 420 , the electronic device 110 identifies a target area to be edited in the first media content in the media editing interface based on the received adjustment request.

[0084] In block 430 , the electronic device 110 presents at least one adjustment element associated with the target area in the media editing interface based on the received editing operation.

[0085] In box 440, the electronic device 110 presents the second media content in the media editing interface, where the second media content is generated by adjusting the first media content according to the direction information and / or position information indicated by the at least one adjustment element.

[0086] In some embodiments, identifying a target area to be edited in the first media content in a media editing interface based on a received adjustment request includes: presenting a candidate area corresponding to the target object in the first media content in response to a selection of an identification control in the media editing interface; and determining the target area to be edited based on the candidate area.

[0087] In some embodiments, based on the candidate area, determining the target area to be edited includes: providing editing controls for adjusting the candidate area in the media editing interface, the editing controls including erase controls and / or brush controls, the erase controls are used to delete part of the candidate area, and the brush controls are used to adjust the boundaries of the candidate area; and using the editing controls to adjust the candidate area to determine the target area to be edited.

[0088] In some embodiments, based on the received editing operation, presenting at least one adjustment element associated with the target area in the media editing interface includes: determining a starting position and an ending position based on the received editing operation; and drawing at least one adjustment element in the media editing interface based on the starting position and the ending position.

[0089] In some embodiments, the adjustment element includes an arrow element pointing from the starting position to the ending position, and the arrow element is used to indicate that the target key point corresponding to the starting position in the media content is adjusted from the starting position to the ending position.

[0090] In some embodiments, the method 400 further includes: adjusting a starting position and / or an ending position of the adjustment element based on a modification operation on the adjustment element.

[0091] In some embodiments, the second media content includes a first portion corresponding to the target area and a second portion corresponding to the remaining area, wherein content of the second portion is the same as content of a third portion of the first media content corresponding to the remaining area.

[0092] In some embodiments, the adjustment request is a first adjustment request, and method 400 also includes: in response to a continued second adjustment request based on the received second media content, determining an additional area to be edited in the second media content; based on the received additional editing operation, presenting at least one additional adjustment element associated with the additional area in the media editing interface; and presenting third media content in the media editing interface, the third media content being generated by adjusting the second media content according to additional direction information and / or additional position information indicated by at least one additional adjustment element.

[0093] In some embodiments, the adjustment request is used to adjust the orientation and / or position of the target object in the target area in the first media content.

[0094] In some embodiments, method 400 further includes: in response to selection of a comparison control in the media editing interface, displaying the first media content and the second media content in comparison.

[0095] In some embodiments, the second media content is generated based on the following process: generating a control signal based on direction information and / or position information; and providing the first media content, mask information corresponding to the target area, and the control signal to a generation model to generate the second media content.

[0096] In some embodiments, the generative model is trained based on the following process: extracting a first image frame and a second image frame from training video data; generating a training control signal based on the first image frame and the second image frame, the training control signal being determined based on first position information of a group of key points in the first image frame and second position information of a group of key points in the second image frame, and the group of key points being associated with a preset object in the training video data; generating an intermediate image frame using the generative model based on the first image frame, training mask information corresponding to the preset object, and the training control signal; and training the generative model based on the intermediate image frame and the second image frame.

[0097] Example devices and equipment

[0098] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 5 A schematic structural block diagram of an apparatus 500 for media editing according to some embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or included in a terminal device 110. Each module / component in the apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0099] like Figure 5 As shown, the device 500 includes a first presentation module 510, which is configured to present a media editing interface of a first media content; an identification module 520, which is configured to identify a target area to be edited in the first media content in the media editing interface based on a received adjustment request; a second presentation module 530, which is configured to present at least one adjustment element associated with the target area in the media editing interface based on a received editing operation; and a third presentation module 540, which is configured to present second media content in the media editing interface, the second media content being generated by adjusting the first media content according to direction information and / or position information indicated by at least one adjustment element.

[0100] In some embodiments, the identification module 520 is further configured to: present candidate regions corresponding to the target object in the first media content in response to selection of an identification control in the media editing interface; and determine a target region to be edited based on the candidate regions.

[0101] In some embodiments, the identification module 520 is further configured to: provide editing controls for adjusting the candidate area in the media editing interface, the editing controls including erase controls and / or brush controls, the erase controls are used to delete part of the candidate area, and the brush controls are used to adjust the boundaries of the candidate area; and use the editing controls to adjust the candidate area to determine the target area to be edited.

[0102] In some embodiments, the second presentation module 530 is further configured to: determine a starting position and an ending position based on the received editing operation; and draw at least one adjustment element in the media editing interface based on the starting position and the ending position.

[0103] In some embodiments, the adjustment element includes an arrow element pointing from the starting position to the ending position, and the arrow element is used to indicate that the target key point corresponding to the starting position in the media content is adjusted from the starting position to the ending position.

[0104] In some embodiments, the second presentation module 530 is further configured to adjust the starting position and / or ending position of the adjustment element based on the modification operation on the adjustment element.

[0105] In some embodiments, the second media content includes a first portion corresponding to the target area and a second portion corresponding to the remaining area, wherein content of the second portion is the same as content of a third portion of the first media content corresponding to the remaining area.

[0106] In some embodiments, the adjustment request is a first adjustment request, and the device 500 also includes a third media content presentation module, which is configured to determine an additional area to be edited in the second media content in response to a continued second adjustment request received for the second media content; based on the received additional editing operation, present at least one additional adjustment element associated with the additional area in the media editing interface; and present third media content in the media editing interface, where the third media content is generated by adjusting the second media content according to additional direction information and / or additional position information indicated by at least one additional adjustment element.

[0107] In some embodiments, the adjustment request is used to adjust the orientation and / or position of the target object in the target area in the first media content.

[0108] In some embodiments, the apparatus 500 further includes a comparison module configured to display the first media content and the second media content in comparison in response to a selection of a comparison control in the media editing interface.

[0109] In some embodiments, the second media content is generated based on the following process: generating a control signal based on direction information and / or position information; and providing the first media content, mask information corresponding to the target area, and the control signal to a generation model to generate the second media content.

[0110] In some embodiments, the generative model is trained based on the following process: extracting a first image frame and a second image frame from training video data; generating a training control signal based on the first image frame and the second image frame, the training control signal being determined based on first position information of a group of key points in the first image frame and second position information of a group of key points in the second image frame, and the group of key points being associated with a preset object in the training video data; generating an intermediate image frame using the generative model based on the first image frame, training mask information corresponding to the preset object, and the training control signal; and training the generative model based on the intermediate image frame and the second image frame.

[0111] The units included in the device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units in the device 500 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0112] Figure 6 1 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 6 The electronic device 600 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 6 The electronic device 600 shown can be used to implement Figure 1 An electronic device 110.

[0113] like Figure 6As shown, the electronic device 600 is in the form of a general electronic device. The components of the electronic device 600 may include, but are not limited to, one or more processors or processing units 610, a memory 620, a storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. The processing unit 610 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 620. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 600.

[0114] The electronic device 600 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 600.

[0115] The electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 6 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 625 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0116] The communication unit 640 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0117] The input device 650 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 660 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 600, or communicate with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0118] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0119] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0120] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0121] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0122] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0123] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for media editing, comprising: A media editing interface presenting the first media content; Based on the received adjustment request, identifying, in the media editing interface, a target area to be edited in the first media content; Based on the received editing operation, presenting at least one adjustment element associated with the target area in the media editing interface; as well as In the media editing interface, second media content is presented, where the second media content is generated by adjusting the first media content according to the direction information and / or position information indicated by the at least one adjustment element.

2. The method according to claim 1, wherein identifying a target area to be edited in the first media content in the media editing interface based on the received adjustment request comprises: In response to a selection of an identification control in the media editing interface, presenting a candidate region corresponding to a target object in the first media content; as well as Based on the candidate regions, the target region to be edited is determined.

3. The method according to claim 2, wherein determining the target area to be edited based on the candidate area comprises: Providing an editing control for adjusting the candidate area in the media editing interface, the editing control comprising an erase control and / or a brush control, the erase control being used to delete a portion of the candidate area, and the brush control being used to adjust a boundary of the candidate area; as well as The candidate area is adjusted using the editing control to determine the target area to be edited.

4. The method according to claim 1, wherein based on the received editing operation, presenting at least one adjustment element associated with the target area in the media editing interface comprises: Determining a start position and an end position based on the received editing operation; as well as Based on the starting position and the ending position, at least one adjustment element is drawn in the media editing interface.

5. The method according to claim 4, wherein the adjustment element comprises an arrow element pointing from the starting position to the ending position, and the arrow element is used to indicate that a target key point corresponding to the starting position in the media content is adjusted from the starting position to the ending position.

6. The method according to claim 4, further comprising: Based on the modification operation on the adjustment element, the starting position and / or the ending position of the adjustment element is adjusted.

7. The method according to claim 1, wherein the second media content includes a first portion corresponding to the target area and a second portion corresponding to the remaining area, wherein content of the second portion is the same as content of a third portion of the first media content corresponding to the remaining area.

8. The method according to claim 1, wherein the adjustment request is a first adjustment request, the method further comprising: In response to receiving a second adjustment request for the second media content, determining an additional region to be edited in the second media content; Based on the received additional editing operation, presenting at least one additional adjustment element associated with the additional area in the media editing interface; as well as In the media editing interface, third media content is presented, where the third media content is generated by adjusting the second media content according to the additional direction information and / or additional position information indicated by the at least one additional adjustment element. 9 . The method according to claim 1 , wherein the adjustment request is used to adjust the orientation and / or position of the target object in the target area in the first media content.

10. The method according to claim 1, further comprising: In response to a selection of a comparison control in the media editing interface, the first media content and the second media content are displayed in comparison.

11. The method of claim 1 , wherein the second media content is generated based on the following process: generating a control signal based on the direction information and / or the position information; and The first media content, mask information corresponding to the target area, and the control signal are provided to a generation model to generate the second media content.

12. The method of claim 11, wherein the generative model is trained based on the following process: extracting a first image frame and a second image frame from the training video data; generating a training control signal based on the first image frame and the second image frame, wherein the training control signal is determined based on first position information of a group of key points in the first image frame and second position information of the group of key points in the second image frame, wherein the group of key points is associated with a preset object in the training video data; Generate an intermediate image frame using the generation model based on the first image frame, the training mask information corresponding to the preset object, and the training control signal; as well as The generation model is trained based on the intermediate image frame and the second image frame.

13. An apparatus for media editing, comprising: A first presentation module, configured to present a media editing interface of the first media content; an identification module, configured to identify, in the media editing interface, a target area to be edited in the first media content based on the received adjustment request; A second presentation module, configured to present at least one adjustment element associated with the target area in the media editing interface based on the received editing operation; as well as The third presentation module is configured to present second media content in the media editing interface, where the second media content is generated by adjusting the first media content according to the direction information and / or position information indicated by the at least one adjustment element.

14. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 12 when executed by the at least one processing unit.

15. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 12 when executed by a processor.

Images