Video editing method, electronic equipment, storage medium and program product

Abstract

The invention relates to a video editing method, electronic equipment, a storage medium and a program product, and the method comprises the steps: responding to the setting of multiple segments of original video materials in a video editing interface, and displaying the video frames of the multiple segments of original video materials in a video track assembly in the video editing interface; in response to a coloring function component in the video editing interface, executing a coloring function operation on any video frame in each section of original video material in the video track component, generating and displaying a coloring video material corresponding to the original video material and / or generating and displaying a replacement video material corresponding to each section of original video material, and exporting the multiple segments of coloring video materials and / or the multiple segments of replacement video materials corresponding to the multiple segments of original video materials as a mixed cutting video. Therefore, by generating the coloring video material or replacing the video material, the characters in the mixed and clipped video are kept consistent, the efficiency of creating the mixed and clipped video by a creator is improved, and the understanding cost of the characters in the mixed and clipped video is greatly reduced for a viewer.

Description

Technical Field

[0001] This disclosure relates to the field of video editing, and more particularly to a video editing method, electronic device, storage medium, and program product. Background Technology

[0002] In film, television, or animation production, creators (such as action directors) can quickly edit existing video footage to transform their ideas (especially action scenes) into visually montage videos, creating a preliminary video draft that fully expresses their creative vision. This allows them to visually demonstrate the creator's envisioned action rhythm and storyline, ensuring that other members of the production team share a common expectation for the final result. It also provides a blueprint for the actual production stage, saving time and resources.

[0003] For creators, a montage video requires editing multiple clips before it can be used. However, each clip may come from different short or long videos, program segments, or film clips. This results in characters in different clips representing the same role (the character the creator intends to portray), but appearing as different characters in different segments of the montage video. For viewers (such as other members of the production team), it is difficult to associate the different characters in different segments of the same montage video. The current solution is to manually add character identification text to each clip in the video editor (such as adding text to the subtitle track) and align the characters' movements. This operation is repetitive, time-consuming, and has low creative efficiency. At the same time, viewers still need to find the names and characters and match them in the fast-paced montage video, which still presents a high understanding cost. Summary of the Invention

[0004] In view of this, this disclosure proposes a video editing method, electronic device, storage medium, and program product that can maintain consistency of characters in a mashup video by generating colored video footage or replacing video footage, thereby improving the efficiency of creators in creating mashup videos and significantly reducing the understanding cost of characters in mashup videos for viewers.

[0005] According to one aspect of this disclosure, a video editing method is provided, comprising: responding to a user setting multiple original video clips in a video editing interface, displaying video frames of the multiple original video clips in a video track component in the video editing interface; responding to performing a coloring function operation on any video frame of each original video clip in the video track component through a coloring function component in the video editing interface, generating and displaying colored video clips corresponding to each original video clip, and / or generating and displaying replacement video clips corresponding to each original video clip, so as to export the multiple colored video clips and / or the multiple replacement video clips corresponding to the multiple original video clips as a mashup. The video; wherein the coloring function operation includes: using a coloring mask to identify the area where a person is located in a video frame, setting the name of the person identified by the coloring mask, or, further including setting at least one replacement image of the person identified by the coloring mask; each video frame of the coloring video material is covered with a coloring mask corresponding to the person and the name of the person corresponding to the person is displayed following the coloring mask, and different people use different colored coloring masks; in the replacement video material, the person whose replacement image is set is displayed as the person image in the replacement image; in the mixed video, the same colored coloring mask and the person name are used to identify the same person, or the same person has the same person image.

[0006] In one possible implementation, the step of responding to perform a coloring function operation on any video frame of each original video material in the video track component through the coloring function component in the video editing interface to generate and display the colorized video material corresponding to each original video material includes: responding to the user selecting any target video frame of any original video material on the video track, generating and displaying the coloring mask corresponding to the person in the target video frame; responding to the user setting the name of the person identified by the coloring mask and triggering the confirmation generation control, generating the coloring mask corresponding to the person in each video frame of the original video material according to the coloring mask corresponding to the person in the target video frame and the original video material to which the target video frame belongs; generating the corresponding colorized video material according to the coloring mask corresponding to the person in each video frame of the original video material and the name of the person identified by the coloring mask, and playing the colorized video material in the player component of the video editing interface.

[0007] In one possible implementation, the step of responding to perform a coloring function operation on any video frame of each original video material in the video track component through the coloring function component in the video editing interface, and generating and displaying the replacement video material corresponding to each original video material, includes: responding to the user selecting any target video frame of any original video material on the video track, generating and displaying a coloring mask corresponding to the person in the target video frame; responding to the user setting at least one replacement image of the person identified by the coloring mask and triggering a confirmation generation control, generating a coloring mask corresponding to the person in each video frame of the original video material based on the coloring mask corresponding to the person in the target video frame and the original video material to which the target video frame belongs; and generating the replacement video material corresponding to the original video material based on the coloring mask corresponding to the person in each video frame of the original video material and at least one replacement image set for the person; displaying the video frame of the replacement video material in the video track component and playing the replacement video material in the player component of the video editing interface; wherein the video frame of the replacement video material corresponding to any original video material is displayed on different tracks in the video track component.

[0008] In some embodiments, the coloring function operation further includes: setting the mask color corresponding to each character; wherein, in response to the user selecting any target video frame of any original video material on the video track, generating and displaying the color mask corresponding to the character in the target video frame includes: in response to the user selecting any target video frame of any original video material on the video track, pausing the display of the target video frame in the player component; in response to the user identifying the character area of ​​a single character on the displayed target video frame using a character selection control, generating the color mask corresponding to the single character based on the initial character area of ​​the single character identified by the character selection control and the mask color set for the single character, and displaying it on the target video frame; or, by intelligently identifying the character areas of each character present in the target video frame, generating the color mask corresponding to each character and displaying it on the target video frame, wherein different characters in the target video frame use different colored color masks.

[0009] In one possible implementation, the method further includes: when the character area indicated by the shading mask generated for any character is missing a portion, in response to the user selecting the mask color corresponding to that character and marking the missing portion of the character through the character selection control, updating the shading mask corresponding to that character based on the missing portion of the character marked by the character selection control, the shading mask already generated for that character, and the corresponding set mask color, so that the updated shading mask indicates the complete character area; when the character area indicated by the shading mask generated for any character contains redundant areas that do not belong to that character, in response to the user selecting the mask color corresponding to that character and marking the redundant areas through the erase control, updating the shading mask corresponding to that character based on the redundant areas marked by the erase control, the shading mask already generated for that character, and the corresponding set mask color, so that the updated shading mask indicates the accurate character area.

[0010] In one possible implementation, the coloring function component further includes a coloring on / off control, which is used to decide whether to display the coloring mask. The method further includes: in response to selecting to display the coloring mask through the coloring on / off control, playing the coloring video material in the player component of the video editing interface; or, in response to selecting not to display the coloring mask through the coloring on / off control, playing the original video material in the player component.

[0011] In some embodiments, the method further includes: in response to a user importing a target script, extracting character names from the target script and displaying the extracted character names from the target script through a name selection control, so that the user can select the character names of characters identified by different color overlays from the character names displayed in the name selection control.

[0012] According to another aspect of this disclosure, a video editing apparatus is provided, comprising: an interface module, configured to, in response to a user setting multiple original video clips in a video editing interface, display video frames of the multiple original video clips in a video track component of the video editing interface; and a video generation module, configured to, in response to performing a coloring function operation on any video frame of each original video clip in the video track component through a coloring function component in the video editing interface, generate and display colored video clips corresponding to each original video clip, and / or generate and display replacement video clips corresponding to each original video clip, so as to combine the multiple colored video clips and / or multiple replacement video clips corresponding to the multiple original video clips. The video footage is exported as a mashup video; wherein, the coloring function operation includes: using a coloring mask to identify the area where a person is located in the video frame, setting the name of the person identified by the coloring mask, or, further including setting at least one replacement image of the person identified by the coloring mask; each video frame of the coloring video footage is covered with a coloring mask corresponding to the person and the person's name is displayed following the coloring mask, and different people use different colored coloring masks; the person in the replacement video footage whose replacement image is set is displayed as the person's image in the replacement image; the mashup video uses the same colored coloring mask and the same person's name to identify the same person, or the same person has the same person's image.

[0013] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0014] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0015] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0016] According to various aspects of this disclosure, by using a color mask to mark the area where a person is located in any video frame of each original video clip, and setting the name of the person marked by the color mask, colored video clips are generated for each original video clip. Different colored video clips use different colored color masks and person names to mark different people. Then, after exporting multiple colored video clips into a mashup video, the same colored color mask and person name are used to mark the same person in the mashup video. This effectively lowers the barrier for viewers to identify people in the video through the colored color masks and person names, while simultaneously improving... It offers high efficiency in creating mashup videos. Furthermore, by setting at least one replacement image for a character, the generated replacement video footage can be updated to feature the user-specified character. After exporting multiple replacement video clips as a mashup video, the same character within the mashup video will have a consistent appearance. This allows viewers to more intuitively identify the same character across different segments, lowering the barrier to understanding mashup videos. Moreover, it eliminates the need for creators to manually edit the text names of each frame of the original video footage in the editor, thus effectively improving the efficiency of mashup video creation.

[0017] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0018] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0019] Figure 1 A flowchart illustrating a video editing method according to an embodiment of the present disclosure is shown.

[0020] Figure 2 A schematic diagram of a video editing interface according to an embodiment of the present disclosure is shown.

[0021] Figure 3 A schematic diagram of a coloring function component according to an embodiment of the present disclosure is shown.

[0022] Figure 4 This diagram illustrates the generation of a color mask by brush strokes according to an embodiment of the present disclosure.

[0023] Figure 5 , Figure 6 , Figure 7 and Figure 8 Schematic diagrams of some video editing interfaces according to an embodiment of the present disclosure are shown.

[0024] Figure 9This diagram illustrates a brush stroke for filling in missing areas according to an embodiment of the present disclosure.

[0025] Figure 10 A schematic diagram of erasing redundant regions is shown according to an embodiment of the present disclosure.

[0026] Figure 11 , Figure 12 and Figure 13 Schematic diagrams of some video editing interfaces according to an embodiment of the present disclosure are shown.

[0027] Figure 14 This diagram illustrates a colorized video material generation process according to an embodiment of the present disclosure.

[0028] Figure 15 A schematic diagram of a video editing interface according to an embodiment of the present disclosure is shown.

[0029] Figure 16 This diagram illustrates a situation where person region recognition fails according to an embodiment of the present disclosure.

[0030] Figure 17 , Figure 18 , Figure 19 , Figure 20 , Figure 21 Schematic diagrams of some video editing interfaces according to an embodiment of the present disclosure are shown.

[0031] Figure 22 A schematic diagram illustrating a method of replacing a character image according to an embodiment of the present disclosure is shown.

[0032] Figure 23 A schematic diagram of a video editing interface according to an embodiment of the present disclosure is shown.

[0033] Figure 24 This diagram illustrates a replacement video material generation process according to an embodiment of the present disclosure.

[0034] Figure 25 , Figure 26 , Figure 27 , Figure 28 Schematic diagrams of some coloring functional components according to an embodiment of the present disclosure are shown.

[0035] Figure 29 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0036] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0037] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0038] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0039] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0040] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0041] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0042] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0043] As mentioned above, currently, manually adding character identification text (such as character names) to each frame of footage in a video editor and aligning the character's movement is repetitive, time-consuming, and inefficient, and still presents a high understanding cost for viewers. In view of this, this disclosure proposes a video editing method, primarily addressing the recognition and understanding problem caused by the different appearances of the same character in different shots when creators are mixing video footage. Specifically, it innovatively splits character recognition capabilities into two methods, from light to heavy, for creators' convenience. The first method uses color overlays and auxiliary name stickers to lower the barrier for viewers to identify the same character in different footage, making it easier for viewers to clearly understand the movement of multiple characters, while also supporting rapid coloring of multiple characters in the same frame, improving operational efficiency. The second method allows users to upload alternative character images based on the color overlay and establish a correspondence with the selected overlay color, thereby performing character replacement (i.e., character image replacement) to achieve a high degree of consistency in dynamic character images. Creators can use this method flexibly to ensure display effects in multiple scenarios.

[0044] The video editing method of this disclosure can be deployed on various terminal devices through software or hardware modifications. The terminal devices involved in this disclosure can refer to devices with wireless and / or wired connection functions. Wireless connection means that they can connect to other devices via wireless connection methods such as Wi-Fi and Bluetooth. The terminal devices involved in this disclosure can also communicate with other devices via wired connection functions. The terminal devices involved in this disclosure can be touchscreen, non-touchscreen, or screenless. Touchscreen devices can be controlled by clicking or swiping on the display screen using fingers, styluses, etc. Non-touchscreen devices can connect to input devices such as mice, keyboards, and touch panels to control the terminal device. Screenless devices can be, for example, screenless Bluetooth speakers. For example, the terminal devices in this application can include, but are not limited to, mobile terminals, tablet computers, laptops, and desktop computers.

[0045] Figure 1 A flowchart illustrating a video editing method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes:

[0046] Step S11: In response to the user setting multiple original video clips in the video editing interface, the video track component in the video editing interface displays the video frames of the multiple original video clips.

[0047] Step S12, in response to performing a coloring function operation on any video frame in each original video material in the video track component through the coloring function component in the video editing interface, generating and displaying the coloring video material corresponding to each original video material, and / or generating and displaying the replacement video material corresponding to each original video material, so as to export multiple coloring video materials and / or multiple replacement video materials corresponding to multiple original video materials as a mashup video.

[0048] The coloring function includes: using a coloring mask to identify the area where a person is located in a video frame, setting the name of the person identified by the coloring mask, or setting at least one replacement image of the person identified by the coloring mask; each video frame of the coloring video material is covered with a coloring mask corresponding to the person and the name of the person corresponding to the person is displayed along with the coloring mask, and different people use different colored coloring masks; the person in the replacement video material whose replacement image is set is displayed as the person image in the replacement image; the same person is identified by the same colored coloring mask and the same person name in the mixed video, or the same person has the same person image.

[0049] According to the method of this disclosure, by using a color mask to mark the area where a person is located in any video frame of each original video material, and setting the name of the person marked by the color mask, colored video material is generated for each original video material. Different colored mask layers and person names are used to mark different people in the colored video material. Then, after exporting multiple colored video materials into a mashup video, the same colored mask layer and person name are used to mark the same person in the mashup video. This effectively lowers the barrier for viewers to identify people in the video through the colored mask layers and person names. This improves the efficiency of creating mashup videos. Furthermore, by setting at least one replacement image for a character, the generated replacement video footage can be updated to feature the user-specified character. After exporting multiple replacement video clips as a mashup video, the same character in the mashup video will have a consistent appearance. This allows viewers to more intuitively identify the same character across different segments, lowering the barrier to understanding mashup videos. It also eliminates the need for creators to manually edit the text names of each frame of the original video footage in the editor, thus effectively improving the efficiency of mashup video creation.

[0050] In step S11, the video editing interface can provide components for setting original video materials. For example, users can search for and select the required original video materials from a preset material library, or upload original video materials from local storage; this embodiment does not limit this. The original video material can be a short, single-shot video containing at least one character, such as a short video containing two characters. It should be understood that the characters in different original video materials can be different or the same; this embodiment does not limit this. The same original video material can contain the same character in every video frame; for example, in an original video material, each frame contains two characters, A and B. Therefore, users can use the coloring function on any frame of an original video material to set a coloring mask, character name, and replace the image for each video frame in that original video material.

[0051] If some frames in an original video clip do not contain any people, these frames can be cut out of the original video clip before executing the method of this embodiment. Alternatively, the method of this embodiment can be executed, ensuring that these frames do not contain any people and remain unchanged in the resulting colored video clip and / or replaced video clip.

[0052] If an original video clip contains different people in different video frames, such as some video frames containing person A and person B, and some video frames containing person C, the original video clip can be split into segments by editing so that each segment of the original video clip contains the same person in its frames.

[0053] For example, Figure 2 A schematic diagram of a video editing interface is shown, such as... Figure 2As shown, creators can select the original video footage they need from "AI Recommendations," "Official Footage," "My Footage," and "Modified Footage" in the media settings component (the component shown under "Materials" on the left side of the interface). "AI Recommendations" allows creators to upload scripts and, through analysis of the script content, recommends and displays original video footage, allowing creators to choose the desired footage from the intelligent recommendations. "Official Footage" displays original video footage from a preset media library, allowing creators to search for the desired footage. "My Footage" displays original video footage uploaded by the creator from local storage, allowing creators to select and use locally uploaded footage. It also displays original video footage saved or used historically from "AI Recommendations" or "Official Footage," facilitating selection. "Modified Footage" refers to original video footage currently selected and imported into the video track. It should be understood that a video clip can be reused, allowing creators to select the desired footage from these modded original video tracks. The media settings component can also include a media upload control (as shown by the "+ Upload" button in the image), allowing users to upload original video footage from their local devices. When the user's mouse focus moves over any original video footage displayed in the media preview area of ​​the media settings component, an add control will appear on that footage (as shown by the "+" in the lower right corner of the footage in the image). The user can click this add control to confirm the selection of the original video footage, which will then be imported into the video track component and displayed on its track.

[0054] In this embodiment, after the user sets multiple original video clips, the video frames of these clips can be displayed on the same track of the video track component in the video editing interface (or, multiple original video clips can be added to a specific track in the video track component). The video track component in this embodiment can refer to the implementation of video tracks in existing video editors (i.e., video editors), and this embodiment does not impose any limitations on it. Figure 2 As shown, the video track component sequentially displays video frames of multiple original video clips set by the user. This video track component supports editing the original video clips (such as splitting, merging, and adjusting the order). When the user selects any original video clip, the original video clip can be played in the player component. When the user imports multiple original video clips at once, the first original video clip can be selected by default.

[0055] It should be understood that, similar to the track component in a traditional video editor, the video track component in this embodiment can support video editing on multiple tracks, such as subtitle tracks and audio tracks. Each type of track can have multiple tracks, such as multiple subtitle tracks, multiple audio tracks, and multiple video tracks; this embodiment does not impose any limitations on this. Furthermore, the video track component in this embodiment allows control of playback time via a play pointer, and the play pointer can be dragged left and right to browse content at different time points. In this embodiment, the play pointer can also be used to select any video frame from any original video material (for example, the video frame corresponding to the time point where the play pointer is located can be considered the video frame selected by the user) to perform a colorization function operation on the selected video frame.

[0056] In practical applications, when a user selects any segment of the original video footage, or selects any frame within any segment of the original video footage, the following can be displayed: Figure 2 The illustrated coloring function component allows users to set the names of characters identified by the coloring overlay, set the overlay colors for different characters (or use the default overlay colors, as long as different characters have different overlay colors), and use the coloring overlay to identify the areas where characters are located in video frames, thereby generating colored video material. Therefore, in some embodiments, step S12 above, responding to the coloring function component in the video editing interface performing a coloring operation on any video frame in each segment of original video material in the video track component, generates and displays colored video material corresponding to each segment of original video material, including:

[0057] Step S121: In response to the user selecting any target video frame of any original video material on the video track, generate and display the shading mask corresponding to the person in the target video frame.

[0058] Step S122: In response to the user setting the name of the person identified by the shading mask and triggering the confirmation generation control, generate the shading mask corresponding to the person in each video frame of the original video material according to the shading mask corresponding to the person in the target video frame and the original video material to which the target video frame belongs.

[0059] Step S123: Based on the shading mask corresponding to the person in each video frame of the original video material and the name of the person identified by the shading mask, generate the corresponding shading video material and play the shading video material in the player component of the video editing interface.

[0060] In some embodiments, step S121, in response to the user selecting any target video frame of any segment of original video material on the video track, generating and displaying a color mask corresponding to the person in the target video frame, may include:

[0061] In response to the user selecting any target video frame from any segment of original video footage on the video track, pause the display of the target video frame in the player component;

[0062] In response to a user identifying the character area of ​​a single character on a displayed target video frame using a character selection control, a color mask corresponding to that single character is generated and displayed on the target video frame based on the initial character area identified by the character selection control and the mask color set for that single character; or, by intelligently identifying the character areas of each character in the target video frame, a color mask corresponding to each character is generated and displayed on the target video frame, with different characters in the target video frame using different colored color masks.

[0063] In some embodiments, the character selection control may be a brush drawing control, a box selection control, etc., as long as it enables the user to mark the area of ​​the character in the target video frame. This disclosure does not limit this.

[0064] For example, such as Figure 2 As shown, when a user selects any target video frame from any segment of original video footage on the video track using the play pointer, the player component can pause the display of that target video frame. At this time, the "Smart Brush," a brush smearing control, is selected by default in the coloring function component. This coloring function component can also include a thickness adjustment control, allowing the user to adjust the brush thickness during smearing. Additionally, it can include a character masking setting component (i.e., the entire component indicated by "Draw Character Masking" in the image). This character masking setting component includes color setting controls and a character name input box (i.e., the input box indicated by "Please Enter Character Name" in the image). When the user clicks... Figure 2 The color setting controls shown can display, for example... Figure 3 As shown in (a), the color list overlay allows users to select a mask color, or choose not to select one and use the default mask color. If a mask color has already been generated, that mask color cannot be selected as the mask color for other characters. Therefore, it is possible to... Figure 3As shown in (b), when the mouse focus moves to a color of a generated shading overlay, the mouse focus can display a "disable selection" icon to indicate to the user that the color cannot be selected (as shown in the red figure, which is unselectable); and the user can enter the name of the person corresponding to the overlay color of the shading overlay to be generated through the name input box, and the shading overlay and the name of the person can be associated through the overlay color to facilitate the generation of subsequent shading video materials.

[0065] In some embodiments, during the process of marking the initial character area using a brush control, when the mouse focus moves to the target video frame displayed in the player component, the mouse focus can change as follows: Figure 4 The brush smearing control shown in ① allows the user to control it to mark the initial character area on the target video frame through smearing. It should be understood that this embodiment does not limit the control method of the brush smearing control. For example, pressing the left mouse button can represent starting smearing, holding down the left mouse button and moving the mouse can represent continuous smearing, and releasing the left mouse button can represent ending smearing. By controlling the brush smearing control on the target video frame, for example, the following can be obtained: Figure 4 The smearing result shown in ② indicates that after smearing is finished (i.e., the left mouse button is released), the system can automatically generate a color mask corresponding to that individual character based on the initial character area smeared by the user and the mask color set for that individual character. The generation process can be displayed as shown in the image. Figure 4 The loading state shown in ③ can then be displayed as follows: Figure 4 ④ in the diagram shows the color mask corresponding to the character.

[0066] For example, if through the above Figure 4 The application method shown is for Figure 2 By painting over the figure on the left side of the media player component, you can generate something like... Figure 5 The shading mask shown is displayed on the target video frame already displayed in the player component. It should be understood that the initial character area painted by the user only represents a rough character area marked by the user. Based on the user-marked initial character area, image recognition and image segmentation can be used to accurately identify and segment the character range within the user-marked initial character area, obtaining an accurate character area. Then, a corresponding shading mask can be generated based on the mask color set for that character and the obtained accurate character area. In practical applications, the mask color set by the user can be the same as the brush color of the brush painting control; the brush color indicates the mask color corresponding to the character marked by the brush painting control.

[0067] When there are multiple characters in the target video frame and multiple character shading layers need to be generated, the shading function component can support painting multiple characters in the same frame. For example, after generating a character shading layer, a character shading layer setting component can be added through the character addition control in the shading function component (as indicated by "+Add Character" in the figure). For example, when the user clicks... Figure 5 After adding controls to the character, you can add features such as... Figure 6 The character masking component shown allows users to set the masking color for each character (or use the default masking color) and input the character's name. Selecting the character masking component means the current brush control uses the masking color set by that component. Users can then use the brush control to paint over the initial character area of ​​another character. After painting, the system automatically generates a color mask for that character based on the painted initial area and the masking color set for that character, and displays it on the target video frame. This results in a masking effect similar to the one shown. Figure 7 The player component shown displays a target video frame overlaid with two shading layers. The shading layers corresponding to different characters can be on different layers to facilitate adjustments and editing. It should be understood that if the target video frame contains more characters, a corresponding shading layer can be generated for each character in the same way. Shading layers are implemented by overlaying layers on top of the original video frame. The area corresponding to the character in the layer has the set mask color and preset transparency, while other areas can be colorless and transparent. When multiple shading layers correspond to the same video frame, the shading layers can be superimposed on the video frame in any predetermined order.

[0068] It should be understood that the above-described method of generating color masks through a character selection control requires the user to manually identify the initial character regions one by one to generate color masks for different characters. In fact, in some embodiments, after the user selects any target video frame in the original video material, artificial intelligence technologies such as image recognition and image segmentation known in the art can be used to directly identify the character regions of each character in the target video frame. For example, image recognition technology can be used to identify the rough character range of the characters in the target video frame (e.g., the selected character range), and then image segmentation technology can be used to accurately segment the character outlines in the identified rough character range to obtain the character regions of each character intelligently identified from the target video frame. Then, the color mask corresponding to each character can be directly generated based on the intelligently identified character regions. In this method, different default mask colors can be used for the shading layers of each character. The number of characters identified in the target video frame can be matched with the number of character mask settings components displayed in the shading layer component. For example, when a user selects any target video frame from a given original video clip, the system can automatically identify the regions containing each character in the target video frame, generate the corresponding shading layer for each character, and display it on the target video frame. This allows for direct display, for instance. Figure 8 The interface shown directly displays the shading overlays of each identified person and the corresponding person overlay settings components on the target video frame. Users can... Figure 8 The interface displays the character masking settings component where you can enter the name of each character. You can also modify the masking color using the color setting control. When the user modifies the masking color corresponding to any character, a new masking layer with the same modified masking color can be generated and the masking layer displayed in the target video frame can be updated to the modified masking layer.

[0069] In some embodiments, such as Figures 5 to 8 As shown, each character mask setting component has a delete button on one side. If a character mask setting component has a corresponding shading mask, the user can delete the corresponding character mask setting component and its corresponding shading mask by clicking the delete button. If a character mask setting component has not generated a corresponding shading mask, the user can delete only the character mask setting component by clicking the delete button.

[0070] Considering that whether the character area is generated after identifying the initial character area using a character selection control, or by directly and intelligently recognizing the character area, there is a possibility that the character area indicated by the shading layer may be missing parts or contain redundant areas that do not belong to the character, therefore, in some embodiments, after generating the shading layer for any character, the method may further include:

[0071] If the area of ​​a character indicated by the shading mask generated for any character is missing, in response to the user selecting the mask color corresponding to that character and marking the missing area of ​​the character through the character selection control, the shading mask corresponding to that character is updated according to the missing area of ​​the character marked by the character selection control, the shading mask already generated for that character, and the corresponding set mask color, so that the updated shading mask indicates the complete area of ​​the character.

[0072] If the character area indicated by the shading mask generated for any character contains redundant areas that do not belong to that character, in response to the user selecting the mask color corresponding to that character and marking the redundant area through the erase control, the shading mask corresponding to that character is updated based on the redundant area marked by the erase control, the shading mask already generated for that character, and the corresponding set mask color, so that the updated shading mask indicates the accurate character area.

[0073] For example, Figure 9 ① In the image, the shading mask indicates a missing arm area in the character's region. Users can click on the character's mask settings component (representing selecting the mask color for that character), and then use the brush control to paint over the missing arm area. After stopping painting, image segmentation technology is automatically used to segment the missing area based on the brush control, and then a shading mask corresponding to that area is generated based on the set mask color. The generation process can be displayed as shown below. Figure 9 The transition state shown in ② can then be merged with the original shading layer of the character that has already been generated, resulting in the state shown in ②. Figure 9 The shading overlay shown in ③ indicates the complete character area, and can also update the shading overlay displayed in the player component to indicate the complete character area.

[0074] For example, Figure 10 ① The character area indicated by the shading layer in the image contains redundant areas that do not belong to the character (such as the hand area in the lower right corner of the image). In this case, you can click on the area indicated by the shading layer. Figure 2 The "Erase" button shown in the image switches to the erase control. When switched to the erase control, the mouse focus on the target video frame can be changed as follows: Figure 10 The eraser style shown in ① allows users to control the erasure function. Figure 10In step ①, the already generated shading mask is erased (e.g., pressing the left mouse button starts erasing, holding down the left mouse button and moving it represents continuous erasing, and releasing the left mouse button stops erasing) to indicate redundant areas. Then, based on the redundant area identified by the erase control, the already generated shading mask for the character, and the corresponding mask color, the corresponding shading mask for that character is updated. For example, this can result in... Figure 10 ② in the diagram shows the updated color mask. It should be understood that the area marked by the erase control is considered a redundant area. The redundant area indicated by the erase control can be directly deleted from the original color mask already generated for that character to obtain the updated color mask.

[0075] In some embodiments, in the coloring function component, the eraser control may also have a corresponding thickness adjustment control to adjust the thickness of the eraser control (i.e., the "eraser"), which can refer to... Figure 2 The thickness adjustment controls are shown in the image. And, as shown... Figure 2 As shown, the coloring function component can also provide undo and redo controls (including undo and redo buttons). Clicking the undo button can undo one step of the operation (smudge or erase), and clicking the redo button can re-execute one step of the undo operation.

[0076] In some embodiments, without generating any shaded mask, confirming the generation of controls (such as...) Figure 2 The "Apply to current material" option (represented by the generated control) can be, for example, as shown below. Figure 2 The grayed-out, unclickable state; when a color mask of the person in the target video frame has already been generated, the confirmation control represented by "Apply to Current Footage" can be transformed into something like this. Figures 5 to 8The highlighted clickable state is shown. In step S122, the user can click the confirmation generation control to execute the generation process of the colored video material. That is, based on the colored mask corresponding to the person in the target video frame and the original video material to which the target video frame belongs, the colored mask corresponding to the person in each video frame of the original video material is generated, and then the replacement video material is generated. Here, the colored mask corresponding to the person in the target video frame can be understood as the generation benchmark for the colored masks corresponding to the person in other video frames. By using the person area indicated by the colored mask corresponding to the person in the target video frame, it is equivalent to distinguishing the person in the target video frame. In this way, artificial intelligence technologies such as object detection technology can be used to detect and identify the person image (including facial features, appearance, clothing, etc.) in each other video frame that is similar to the benchmark person image (for example, a detection box can be used to indicate the person area) based on the person image (including facial features, appearance, clothing, etc.) indicated by any colored mask in the target video frame. Then, image segmentation technology can be used to accurately segment the human body contour within the person area to obtain the segmented person area. Then, based on the mask color corresponding to the person in the target video frame, the colored mask corresponding to the person in other video frames can be generated.

[0077] In practical applications, when there are multiple characters in the target video frame and corresponding color masks for multiple characters are generated, the process can proceed sequentially according to the color mask of each character in the target video frame. First, based on the character image in the character area identified by a certain color mask, target detection and image segmentation are performed on each other video frame in the original video material to obtain the character area of ​​the same character in each other video frame and generate a color mask of the same color. Then, based on the character image in the character area identified by the next color mask, target detection and image segmentation are performed on each other video frame in the original video material to obtain the character area of ​​the next same character in each other video frame and generate a color mask of the same color. This process is repeated until color masks for all characters in each other video frame are generated. Alternatively, based on the character image in the character area identified by each color mask, target detection and image segmentation can be performed on each other video frame simultaneously to obtain the character area of ​​each character in each other video frame and then generate the corresponding color mask. This embodiment of the present disclosure does not limit this approach.

[0078] In step S123, after generating the shading mask for the person in each video frame of the original video material, a name tag corresponding to the person's name can be automatically added around each shading mask in each video frame based on the association between the person's name and the mask color, according to preset name identification rules (e.g., the person's name is displayed in the upper left corner of the shading mask and at a specified pixel distance from the shading mask). It should be understood that the name tag and shading mask for the same person can be displayed on the same layer. This allows the person's name in the generated shading video material to automatically follow the person's shading mask (i.e., follow the person's area), ensuring that each video frame of the shading video material is covered with the person's corresponding shading mask and displays the person's corresponding name following the shading mask.

[0079] After generating the colored video footage, it can be displayed using a player component, for example, such as... Figure 11 The player component shown in the image displays the shading video footage, which shows the shading overlays for each character and their names (i.e., "Character 1", "Character 2").

[0080] It should be understood that for each original video clip displayed in the video track component, colored video clips corresponding to each original video clip can be generated according to steps S121 to S123 above. When the user clicks on any video frame of other original video clips, if the user selects the same color as the already generated colored mask through the color setting control in the character mask setting component, the input box in the character mask setting component can automatically substitute the character name identified by the already generated colored mask. In this way, the same character can be identified in the current original video clip using a colored mask of the same color as the original video clip that has already generated colored video clips, without the user having to manually enter the character name again, improving the convenience of use. If other original video clips contain other characters that have not been colored, the user can also re-perform the colored function operation for other characters. For example, as Figure 12 and Figure 13 The images show the colored video footage corresponding to another original video clip displayed in the player component, such as... Figure 11 , Figure 12 and Figure 13 As shown, character 1 uses a red shading layer, character 2 uses a blue shading layer, and character 3 uses a green shading layer.

[0081] For example, Figure 14 The diagram illustrates a process for generating colored video footage, such as... Figure 14As shown, the process of generating replacement video footage includes: After the user sets the original video footage and selects to enable the coloring function, the user can enter the character name (person's name) and can also mark human body areas by smearing. The coordinates of the user's smeared areas can be sent to the backend. The coloring mask is calculated according to the user's smearing data (i.e., the coordinates of the user's smeared areas), and fed back to the frontend for automatic recognition and display of the mask. The mask can be further modified (such as smearing, erasing, etc.) to obtain a complete and accurate coloring mask. When the user selects to apply it to the footage (i.e., triggers the confirmation generation component), the original video footage, coloring mask, and character name can be sent to the backend. The backend can use the model to perform calculations and rendering to generate coloring video footage corresponding to the original video footage for each character. For example, the original video and the name of character 1 + mask can be used to generate the coloring video corresponding to character 1 (i.e., "Generate Video - Character 1"). Then, the coloring video corresponding to character 1 and the name of character 2 + mask can be used to generate the coloring video footage corresponding to character 1 and character 2 (i.e., "Generate Video - Character 1 + Character 2"). The original video footage or coloring video footage can be displayed back to the video track, thus obtaining a video with multiple character masks.

[0082] In some embodiments, after generating a colored video material from any original video material, the user can select any video frame on the video track using the play pointer. At this time, the video frame selected by the play pointer, along with the corresponding colored mask and name sticker, can be displayed in the player component. Alternatively, the user can also select the name of any character, and the colored mask and name sticker corresponding to that character can be displayed in the player component. This disclosure does not limit the scope of the embodiments.

[0083] In this context, the tinted video material can be understood as having a tinted overlay inserted into the original video material. Therefore, after generating the tinted video material, the video track component can still display the video frames of the corresponding original video material, and of course, it can also display the video frames of the tinted video material. To facilitate users switching between the original video material and the tinted video material, in some embodiments, the tinting function component may also include a tinting on / off control, which is used to decide whether to display the tinted overlay. The method may also include:

[0084] In response to selecting to display the shading overlay via the shading on / off control, the shading video footage is played in the player component of the video editing interface; or, in response to selecting not to display the shading overlay via the shading on / off control, the original video footage is played in the player component.

[0085] For example, the coloring on / off control can be displayed as follows: Figure 2The button control shown allows users to control whether the player component displays the shading overlay, enabling them to switch between the original video footage and the shading video footage. For example, when a user selects not to display the shading overlay using the shading overlay control, they can select from... Figure 11 The interface shown switches to Figure 15 The interface shown is... Figure 15 In the interface shown, the coloring function component has all the other function controls except for the mask opening and closing air in a grayed-out and unclickable state. The player component plays the original video footage without the color mask and name sticker.

[0086] In some embodiments, if the target video frame does not contain a person area, and the user has also marked the target video frame using a person selection control, then after image recognition fails (i.e., person area recognition fails), the following can be displayed: Figure 16 The displayed prompts the user to indicate the areas of the person they need to paint over. The prompt can be displayed in the center of the screen, or it can automatically disappear after 3 seconds, while the painted areas are automatically cleared of the paint mark.

[0087] In practical applications, when a user selects any segment of the original video footage, or selects any frame within any segment of the original video footage, the following can be displayed: Figure 17 The coloring function component shown in the video editing interface allows users to set the name of the person identified by the coloring mask, set the mask color for different people (or use the default mask color, as long as different people have different colored emphasis masks), and set a replacement image for each person to replace the person in the original video footage, thus more intuitively identifying different people in different footage. Therefore, in some embodiments, in step S12 above, in response to performing the coloring function operation on any video frame in each segment of original video footage in the video track component through the coloring function component in the video editing interface, a replacement video clip corresponding to each segment of original video footage is also generated and displayed, including:

[0088] Step S124: In response to the user selecting any target video frame of any original video material on the video track, generate and display the shading mask corresponding to the person in the target video frame;

[0089] Step S125: In response to the user having set at least one replacement image of the person identified by the shading mask and triggering the confirmation generation control, generate the shading mask corresponding to the person in each video frame of the original video material according to the shading mask corresponding to the person in the target video frame and the original video material to which the target video frame belongs.

[0090] And, in step S126, based on the color mask corresponding to the character in each video frame of the original video material and at least one replacement image set for the character, generate the replacement video material corresponding to the original video material.

[0091] Step S127: Display the video frames of the replacement video material in the video track component and play the replacement video material in the player component of the video editing interface; wherein, the replacement video material corresponding to any original video material is displayed on different tracks in the video track component.

[0092] In some embodiments, the specific implementation of step S124 can refer to the relevant content of step S121 above, and will not be repeated here.

[0093] In step S125, for example, as follows Figure 17 As shown, the character masking settings component in the shading function can also provide an image setting control indicated by "Add Image to Replace Character". After a shading mask for a character has been generated, and the image setting control in the character masking settings component for that character is clicked, the following can be displayed: Figure 18 The "Front View" and "Full Body" images shown herein represent image import controls. Users can use these controls to upload replacement images of the desired person (i.e., front view and full body images), thus enabling the setting of at least one replacement image of the person identified by the color mask. It should be understood that... Figure 18 Uploading two images is one possible implementation provided by this embodiment. In practice, images of people from more angles (such as side views, back views, etc.) can also be uploaded; this embodiment does not impose any limitations on this. Figure 17 and Figure 18 As shown, the character masking component can also set the mask color and character name. The mask color can then be used to associate the shading mask, character name, and replacement image, facilitating the subsequent generation of replacement video footage based on this association. It should be understood that users can set the character name and replacement image after generating the shading mask, during the shading mask generation process, or even before generating the shading mask; this embodiment does not impose any limitations on this.

[0094] When the user passes Figure 18 After uploading and replacing the image using the image import control shown, you can do so as follows: Figure 19The imported replacement image is displayed, and a delete button can be shown in the upper right corner of the displayed replacement image. Users can delete the imported replacement image by clicking the delete button. This embodiment of the disclosure does not limit this. After the user has set at least one replacement image of the person identified by the color mask, the replacement video material generation process in steps S125 to S126 can be executed by triggering the confirmation generation control represented by "Generate Now".

[0095] In some embodiments, the specific implementation of step S125 can refer to the relevant content of step S122 described above, and will not be repeated here. During the generation of the shading mask for each video frame, the following can be displayed on the shading function component and the video track component: Figure 20 The displayed prompts, such as "Mask Generating >>> Character Replacement" on the shading function component and "Mask Generating" on the video frames of the original source video on the video track, highlight the "Mask Generating" message, indicating that step S125 is currently being executed to generate the shading mask for each video frame. The prompts also include a "Cancel" button, which users can click to cancel the shading mask generation. After the shading mask for each video frame has been generated, the prompt can be switched to... Figure 21 The state shown, Figure 21 The prompt message in the video changes to "Character replacement generation in progress" and is highlighted, indicating that step S126 above is currently being performed to generate replacement video footage. Users can also cancel the generation of replacement video footage by clicking the "Cancel" button; this embodiment does not limit this action.

[0096] In step S126, based on the shading mask corresponding to the character in each video frame of the original video material and at least one replacement image set for that character, a replacement video material corresponding to the original video material is generated. This may include, for example, using pose recognition technology based on the character region indicated by the shading mask corresponding to each character in each video frame of the original video material to confirm the character pose data (such as the poses of multiple human contour key points and human joint key points) of each character in each video frame. This yields the action sequence of any character in the original video material (the action sequence contains the character pose data of any character in each video frame), thus achieving motion extraction (or pose tracking). Then, based on the action sequence of any character in the original video material, a 3D motion model corresponding to that character can be generated, for example, by... Figure 22 Extracting the motion of the character in ① and generating Figure 22The 3D motion model shown in ② can then be used to drive and render the character image in the replacement image, resulting in a character motion model with the character image in the replacement image. Based on this character motion model, the original character image in the original video footage can be replaced frame by frame. For example, this can result in... Figure 22 ③ shows a replacement video clip with the same actions but different character images. Alternatively, a static 3D character model with the character image in the replacement image can be generated first based on at least one replacement image image. Then, based on the extracted action sequence, the 3D character model is driven to obtain a dynamic 3D character action model. Then, based on the 3D character action model, the original character images of the characters in the original video clip are replaced frame by frame to obtain the replacement video clip. This embodiment of the present disclosure does not limit this. The original character images of different characters in the original video clip can be replaced one by one, or the original character images of different characters in the original video clip can be replaced simultaneously. This embodiment of the present disclosure does not limit this.

[0097] It should be understood that those skilled in the art can use known motion extraction techniques (e.g., including steps such as person tracking, pose estimation, and pose data post-processing) to extract the action sequence of a person based on the person region indicated by the shading mask in each video frame; and can use known model-driven techniques, such as SDGS (Software-Defined Geometry Streaming) real-time rendering technology and UE (Unreal Engine) offline rendering technology, to drive a 3D person model based on the action sequence and render a 3D person action model, etc. The embodiments disclosed herein do not limit this.

[0098] After generating replacement video footage corresponding to the original video footage, you can do the following: Figure 23 As shown, a new video track is added to the video track component to display the video frames of the replacement video clip. This ensures that the replacement video clip and the original video clip are displayed on different tracks, and the timeline of the replacement video clip on the video track is consistent with that of the original video clip. When the user selects the replacement video clip, it can also be played in the player component, such as... Figure 23 As shown, the character whose image was set to be replaced in the replacement video footage displayed in the player component has been displayed as the character image in the replacement image.

[0099] It should be understood that for each original video clip displayed in the video track component, a replacement video clip can be generated and displayed according to steps S124 to S126 above. Once a replacement video clip for a certain original video clip has been generated, and the user clicks on any frame of another original video clip, if the user selects the same color as the generated color mask using the character mask setting component in the coloring function component, the character mask setting component can automatically substitute the character name identified by the generated color mask and the corresponding replacement image. This allows the same character to be identified using a color mask of the same color as other original video clips, and the same replacement image to replace the original character image in the original video clip, improving user convenience. Furthermore, a unified character image allows viewers to more intuitively identify characters in different clips, reducing the viewer's comprehension cost.

[0100] In some embodiments, replacement video materials corresponding to the original video materials can be generated one by one, or the replacement video materials corresponding to the original video materials can be generated in batches by first generating the color mask corresponding to each video frame in multiple original video materials, and then setting the replacement image image of the person identified by each color mask. This disclosure does not limit this.

[0101] For example, Figure 24 The diagram illustrates a process for generating replacement video footage, such as... Figure 21As shown, the process of generating replacement video footage includes: After the user sets the original video footage and selects to enable the coloring function, the user can enter the character name (person's name) and upload a local image (i.e., upload the replacement image). The user can also mark human body areas by applying coloring. The coordinates of the user's applied areas can be sent to the backend. The coloring mask is calculated based on the user's applied data (i.e., the coordinates of the user's applied areas) and fed back to the frontend for automatic mask recognition and display. The mask can be further modified (e.g., applied, erased, etc.) to obtain a complete and accurate coloring mask. When the user selects to apply it to the footage (i.e., triggers the confirmation generation component), the original video footage, coloring mask, and character name can be sent to the backend. The backend can use the model to perform calculations and rendering to generate replacement video footage 1 corresponding to the original video footage 1 for each character. For example, the original video 1 and the character 1's name + mask + image can be used to generate a replacement video for the replaced character 1 (i.e., "Generate Video - Character"). 1) Then, using "Generate Video - Character 1" and the names + mask + images of Character 2, generate replacement video material 1 (i.e., "Generate Video - Character 1 + Character 2", or Video 1) where the images of both Character 1 and Character 2 have been replaced. Similarly, through model calculation and rendering, replacement video material 2 corresponding to the original video material 2 can be generated for each character. For example, you can first use the original video 2 and the names + mask + images of Character 2 to generate a replacement video where the image of Character 2 has been replaced (i.e., "Generate Video - Character 2"). Then, using "Generate Video - Character 2" and the names + mask + images of Character 3, generate replacement video material 1 where the images of both Character 2 and Character 3 have been replaced (i.e., "Generate Video - Character 2 + Character 3", or Video 2). The original video material and the generated replacement video material can be displayed back to the video track, thus obtaining the replacement video material where the character images have been replaced. At the same time, you can also obtain a video with multiple character masks (i.e., colored replacement material).

[0102] It should be understood that the replacement video footage is a new video clip compared to the original video footage. Therefore, the replacement video footage and the original video footage can be displayed on different video tracks. If a user wants to view the original video footage, they can select the original video footage from the video track; if they want to view the replacement video footage, they can select the replacement video footage from the video track. Alternatively, they can use the track hiding control provided in the video track component to hide the video track containing either the replacement video footage or the original video footage, thereby controlling the content displayed in the player component.

[0103] Considering that users may be creating montage videos according to a specified script, in order to make it easier for users to set character names, in some embodiments, the method may further include: in response to the user importing a target script, extracting the character names from the target script and displaying the character names extracted from the target script through a name selection control, so that the user can select the character names of characters identified by different color overlays from the character names displayed in the name selection control.

[0104] In practical applications, the front-end interface can provide functional controls for uploading target scripts. Users can upload target scripts from their local devices. Once the target script uploaded by the user is obtained, the names of the characters appearing in the target script can be analyzed using natural language processing techniques known in the field. Then, a name selection control can be provided on the side of the character name input box in the character masking setting component of the coloring function component, so that users can directly select the character names identified in the target script through the name selection control.

[0105] For example, Figure 25 (a) shows the state of the shading component when a character is being depicted. Figure 25 (b) shows the state of the coloring function component when there are multiple characters. "Select from Script Characters" represents a name selection control. When the user clicks "Select from Script Characters," the following can be displayed: Figure 26 The list overlay shown in (a) provides all the character names identified in the target script (such as "Bai Yu", "Si Kong Qian Luo", "Xiao Se", "Chen Hu"). After the user clicks on any character name (such as "Bai Yu"), the overlay collapses, and the selected name is entered into the input box. If the user has already entered a character name or selected a character name in the input box, the newly selected character name can overwrite the previously entered character name, resulting in the following output: Figure 26 The character selection results are shown in (b) above. The character name input box also allows users to manually enter the character name; for example, users can click the input box to display a message such as... Figure 27 The input state shown in (a) has a cursor displayed in the input box. Entering a name and pressing Enter will activate the input; and, as shown in (a), the input state has a cursor displayed in the input box. Entering a name and pressing Enter will activate the input state; and, as shown in (a), the input state has a cursor displayed in the input box. Figure 27 As shown in (b), after a character's name has been entered, the user can click the input box to modify the name, and the cursor will be anchored at the user's click position. The character name can have a character limit (e.g., 10 characters). When the entered character name reaches the character limit, it can be edited as follows: Figure 28 As shown, no further text can be entered in the input box; the input box displays an error message and the message "Character name cannot exceed 10 characters." It should be understood that the above... Figure 27 and Figure 28The related functions and operations shown can also be applied to the character name input box in the coloring function component when there is no name selection control, and this disclosure does not limit this.

[0106] In some embodiments, after generating the shading mask for each video frame in the original video material through steps S124 to S125, steps S123 and S126 can be executed simultaneously to generate shading video material and replacement video material. That is, only shading video material or replacement video material can be generated, or both can be generated simultaneously, depending on the user's actual needs. The user can choose to export multiple shading video material segments as a mashup video, or multiple replacement video material segments as a mashup video. When both shading video material and replacement video material are generated simultaneously, the player component can display the replacement video material by default. If the user wants to view the shading video material, they can select the original video material in the video track component and select to display the shading mask using the shading on / off control to play the replacement video material in the player component.

[0107] It should be understood that those skilled in the art can use known video editing techniques to splice multiple replacement video clips or multiple colored video clips into a complete mashup video. The mashup video can be the result of directly splicing multiple replacement video clips or multiple colored video clips, or it can be the result of further editing of multiple replacement video clips or multiple colored video clips. Because the mashup video uses the same colored overlay and character names to identify the same person or the same person with the same image, viewers can more intuitively understand the relationships between characters in different segments of the mashup video, and it also better expresses the creator's ideas. After the final editing is completed, the colored overlay in the mashup video can be removed to obtain the final video work.

[0108] In this embodiment, the system can intelligently apply color-coded selection areas to original video footage, automatically matching character names and movement paths, thus improving efficiency for creators. It also extracts character actions using an algorithmic model, assigning them to specific characters to better recreate their posture and expressions, while simultaneously enabling multiple characters to work in parallel. Furthermore, by applying color overlays to video characters, the system intelligently identifies them and assigns different colored overlays. Name tags automatically follow according to specified rules. After segmenting the character overlays by color, the system prioritizes generating the colored video. Users can customize single or multiple selections of color overlays in the video footage to perform character replacement operations, generating a more consistent and polished video.

[0109] In this disclosed embodiment, from character segmentation and coloring to the generation of colored video footage and then to character image replacement, a highly consistent creative montage video is formed. This allows for the intelligent generation of the current colored video footage after multiple characters are painted on the same frame, and the automatic addition of name stickers to follow the characters' movements, improving efficiency for creators. Alternatively, by uploading replacement character images and associating them with the overlay colors, batch character image replacement can be performed to generate replacement video footage, ensuring a high degree of consistency in character images and significantly lowering the recognition threshold for viewers.

[0110] Based on the video editing method provided in the above embodiments of this disclosure, this disclosure also provides a video editing apparatus, which includes:

[0111] The interface module is used to respond to the user setting multiple original video clips in the video editing interface, and to display the video frames of the multiple original video clips in the video track component of the video editing interface;

[0112] The video generation module is configured to respond to the coloring function component in the video editing interface performing a coloring function operation on any video frame in each original video material in the video track component, generating and displaying the colorized video material corresponding to each original video material, and / or generating and displaying the replacement video material corresponding to each original video material, so as to export the multiple colorized video materials and / or multiple replacement video materials corresponding to the multiple original video materials as a mashup video;

[0113] The coloring function operation includes: using a coloring mask to identify the area where a person is located in a video frame, setting the name of the person identified by the coloring mask, or setting at least one replacement image of the person identified by the coloring mask; each video frame of the coloring video material is covered with a coloring mask corresponding to the person and the name of the person corresponding to the person is displayed following the coloring mask, and different people use different colored coloring masks; in the replacement video material, the person whose replacement image is set is displayed as the person image in the replacement image; in the mixed video, the same colored coloring mask and the same person name are used to identify the same person, or the same person has the same person image.

[0114] In one possible implementation, the step of responding to perform a coloring function operation on any video frame of each original video material in the video track component through the coloring function component in the video editing interface to generate and display the colorized video material corresponding to each original video material includes: responding to the user selecting any target video frame of any original video material on the video track, generating and displaying the coloring mask corresponding to the person in the target video frame; responding to the user setting the name of the person identified by the coloring mask and triggering the confirmation generation control, generating the coloring mask corresponding to the person in each video frame of the original video material according to the coloring mask corresponding to the person in the target video frame and the original video material to which the target video frame belongs; generating the corresponding colorized video material according to the coloring mask corresponding to the person in each video frame of the original video material and the name of the person identified by the coloring mask, and playing the colorized video material in the player component of the video editing interface.

[0115] In one possible implementation, the step of responding to perform a coloring function operation on any video frame of each original video material in the video track component through the coloring function component in the video editing interface, and generating and displaying the replacement video material corresponding to each original video material, includes: responding to the user selecting any target video frame of any original video material on the video track, generating and displaying a coloring mask corresponding to the person in the target video frame; responding to the user setting at least one replacement image of the person identified by the coloring mask and triggering a confirmation generation control, generating a coloring mask corresponding to the person in each video frame of the original video material based on the coloring mask corresponding to the person in the target video frame and the original video material to which the target video frame belongs; and generating the replacement video material corresponding to the original video material based on the coloring mask corresponding to the person in each video frame of the original video material and at least one replacement image set for the person; displaying the video frame of the replacement video material in the video track component and playing the replacement video material in the player component of the video editing interface; wherein the video frame of the replacement video material corresponding to any original video material is displayed on different tracks in the video track component.

[0116] In one possible implementation, the coloring function operation further includes: setting the mask color corresponding to each character; wherein, in response to the user selecting any target video frame of any original video material on the video track, generating and displaying the color mask corresponding to the character in the target video frame includes: in response to the user selecting any target video frame of any original video material on the video track, pausing the display of the target video frame in the player component; in response to the user identifying the character area of ​​a single character on the displayed target video frame using a character selection control, generating the color mask corresponding to the single character based on the initial character area of ​​the single character identified by the character selection control and the mask color set for the single character, and displaying it on the target video frame; or, by intelligently identifying the character areas of each character present in the target video frame, generating the color mask corresponding to each character and displaying it on the target video frame, wherein different characters in the target video frame use different colored color masks.

[0117] In one possible implementation, the apparatus further includes: a supplementation module, configured to, in the case where the character area indicated by the shading mask generated for any character is missing a portion, respond to the user selecting the mask color corresponding to the character and marking the missing portion of the character through the character selection control, and update the shading mask corresponding to the character based on the missing portion of the character marked by the character selection control, the shading mask already generated for the character, and the corresponding set mask color, so that the updated shading mask indicates the complete character area; and an erasure module, configured to, in the case where the character area indicated by the shading mask generated for any character contains redundant areas that do not belong to the character, respond to the user selecting the mask color corresponding to the character and marking the redundant areas through the erasure control, and update the shading mask corresponding to the character based on the redundant areas marked by the erasure control, the shading mask already generated for the character, and the corresponding set mask color, so that the updated shading mask indicates the accurate character area.

[0118] In one possible implementation, the shading function component further includes a shading enable / disable control, which is used to decide whether to display the shading mask. The device further includes a shading enable / disable module, which is used to: play the shading video material in the player component of the video editing interface in response to selecting to display the shading mask through the shading enable / disable control; or play the original video material in the player component in response to selecting not to display the shading mask through the shading enable / disable control.

[0119] In one possible implementation, the apparatus further includes: a script analysis module, configured to extract character names from the target script in response to a user importing a target script and display the extracted character names from the target script via a name selection control, so that the user can select character names of characters identified by different color overlays from the character names displayed in the name selection control.

[0120] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0121] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0122] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0123] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0124] Figure 29 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a terminal device. (Refer to...) Figure 29 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0125] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM LinuxTM FreeBSD TM Or similar.

[0126] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0127] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0128] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0129] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0130] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0131] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0132] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0134] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A video editing method, characterized in that, include: In response to a user setting multiple original video clips in the video editing interface, the video track component in the video editing interface displays the video frames of the multiple original video clips. In response to performing a coloring function operation on any video frame in each original video material in the video track component through the coloring function component in the video editing interface, generating and displaying the coloring video material corresponding to each original video material, and / or generating and displaying the replacement video material corresponding to each original video material, so as to export the multiple coloring video materials and / or multiple replacement video materials corresponding to the multiple original video materials as a mashup video; The coloring function operation includes: using a coloring mask to identify the area where a person is located in a video frame, setting the name of the person identified by the coloring mask, or setting at least one replacement image of the person identified by the coloring mask. Each video frame of the colored video material is covered with a colored overlay corresponding to the character, and the character's name is displayed following the colored overlay. Different characters use different colored overlays. In the replacement video material, the character whose replacement image is set is displayed as the character image in the replacement image. In the mixed video, the same colored overlay and character name are used to identify the same character, or the same character has the same character image.

2. The method according to claim 1, characterized in that, The response to performing a coloring operation on any video frame in each segment of original video material in the video track component through the coloring function component in the video editing interface, generating and displaying the colorized video material corresponding to each segment of original video material, includes: In response to the user selecting any target video frame of any original video material on the video track, a color mask corresponding to the person in the target video frame is generated and displayed; In response to the user setting the name of the person identified by the shading layer and triggering the confirmation generation control, the shading layer corresponding to the person in each video frame of the original video material is generated according to the shading layer corresponding to the person in the target video frame and the original video material to which the target video frame belongs. Based on the shading overlay corresponding to the character in each video frame of the original video material and the character name identified by the shading overlay, a corresponding shading video material is generated, and the shading video material is played in the player component of the video editing interface.

3. The method according to claim 1, characterized in that, The response to performing a coloring operation on any video frame in each original video clip in the video track component through the coloring function component in the video editing interface, generating and displaying the replacement video clip corresponding to each original video clip, includes: In response to the user selecting any target video frame of any original video material on the video track, a color mask corresponding to the person in the target video frame is generated and displayed; In response to the user having set at least one replacement image of the person identified by the color mask and triggering a confirmation control, based on the color mask corresponding to the person in the target video frame and the original video material to which the target video frame belongs, a color mask corresponding to the person in each video frame of the original video material is generated; and, Based on the color mask corresponding to the character in each video frame of the original video material and at least one replacement image set for the character, generate the replacement video material corresponding to the original video material. The video frames of the replacement video material are displayed in the video track component and played in the player component of the video editing interface; wherein, the video frames of the replacement video material corresponding to any original video material are displayed on different tracks in the video track component.

4. The method according to claim 2 or 3, characterized in that, The coloring function operation also includes: setting the mask color for each character; Specifically, in response to a user selecting any target video frame from any segment of original video material on the video track, generating and displaying a color mask corresponding to a person in the target video frame includes: In response to a user selecting any target video frame from any segment of original video material on the video track, the target video frame is paused in the player component. In response to a user identifying a single character's region on a displayed target video frame using a character selection control, a color mask corresponding to that single character is generated and displayed on the target video frame based on the initial character region identified by the character selection control and the mask color set for that single character; or, By intelligently identifying the regions where each person is located in the target video frame, a color mask corresponding to each person is generated and displayed on the target video frame, with different colors of color mask used for different people in the target video frame.

5. The method according to claim 4, characterized in that, The method further includes: If a part of the character area indicated by the generated shading mask for any character is missing, in response to the user selecting the mask color corresponding to the character and marking the missing part of the character through the character selection control, the shading mask corresponding to the character is updated according to the missing part of the character marked by the character selection control, the shading mask already generated for the character, and the corresponding set mask color, so that the updated shading mask indicates the complete character area. If the character area indicated by the shading mask generated for any character contains redundant areas that do not belong to that character, in response to the user selecting the mask color corresponding to that character and marking the redundant area through the erase control, the shading mask corresponding to that character is updated based on the redundant area marked by the erase control, the shading mask already generated for that character, and the corresponding set mask color, so that the updated shading mask indicates the accurate character area.

6. The method according to claim 1, characterized in that, The shading function component also includes a shading on / off control, which is used to decide whether to display the shading mask. The method further includes: In response to selecting to display the shading overlay via the shading on / off control, the shading video material is played in the player component of the video editing interface; or, In response to selecting not to display the shading overlay via the shading on / off control, the original video footage is played in the player component.

7. The method according to claim 1, characterized in that, The method further includes: In response to a user importing a target script, the character names are extracted from the target script and displayed through a name selection control, allowing the user to select the character names identified by different color overlays from the character names displayed in the name selection control.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images