Video editing method and device, equipment, storage medium and program product

Abstract

The invention provides a video editing method and device, electronic equipment, a computer readable storage medium and a computer program product, and relates to the technical field of artificial intelligence such as computer vision, video processing and intelligent editing. According to one specific embodiment, the method comprises the steps that scene segmentation is carried out on an initial video to obtain a scene video, and the scene video corresponds to a scene in the initial video; associating the scene video to a plot unit of the initial video; extracting video frames from the initial video based on subtitles included in the scene video associated with the plot unit, and generating an edited sub-video corresponding to the plot unit; and generating an edited video of the initial video based on the edited sub-video. Therefore, the video editing and editing material providing performance and efficiency can be improved, and the video editing difficulty is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technologies such as computer vision, video processing, and intelligent editing, and especially to methods, devices, electronic devices, computer-readable storage media, and computer program products for editing videos. Background Art

[0002] With the development of society and technology, we are gradually entering the information age. In this information age, long-form video content provides a large part of internet traffic and has become an important content format in streaming platforms, social media, online education and other fields.

[0003] However, as the pace of life changes, and considering people's different pace of life, many people may not always be willing to invest a lot of time and energy in watching these "long videos." In other words, people may not always be able to spare enough time and energy to watch "long videos" like movies and TV series in their entirety.

[0004] In this context, to improve people's access to video content and enable them to use their fragmented time to conveniently and efficiently access video content, "short videos" based on "long videos" are becoming increasingly popular. Therefore, how to more efficiently and effectively generate relevant "short videos" based on "long videos" is a matter of concern and urgent need. Summary of the Invention

[0005] The embodiments of the present disclosure provide a method, apparatus, electronic device, computer-readable storage medium, and computer program product for editing a video.

[0006] In a first aspect, an embodiment of the present disclosure proposes a method for editing videos, comprising: dividing an initial video into scenes to obtain scene videos, wherein the scene videos correspond to scenes in the initial video; associating the scene videos with plot units of the initial video; extracting video frames from the initial video based on subtitles included in the scene video associated with the plot unit to generate an edited sub-video corresponding to the plot unit; and generating an edited video of the initial video based on the edited sub-video.

[0007] In a second aspect, an embodiment of the present disclosure proposes a device for editing videos, comprising: a scene video segmentation unit, configured to perform scene segmentation on an initial video to obtain a scene video, wherein the scene video corresponds to a scene in the initial video; a scene video association unit, configured to associate the scene video to a plot unit of the initial video; a clip sub-video generation unit, configured to extract video frames from the initial video based on subtitles included in the scene video associated with the plot unit, and generate a clip sub-video corresponding to the plot unit; and a clip video generation unit, configured to generate a clip video of the initial video based on the clip sub-video.

[0008] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the method for editing video as described in any implementation method in the first aspect when executing the instructions.

[0009] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the method for editing a video as described in any implementation method of the first aspect when executed.

[0010] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the method for editing a video as described in any implementation manner in the first aspect.

[0011] The methods, devices, electronic devices, computer-readable storage media, and computer program products for editing videos provided by the embodiments of the present disclosure first segment an initial video into scenes to obtain scene videos, wherein the scene videos correspond to scenes in the initial video; then, the scene videos are associated with plot units of the initial video; next, based on subtitles included in the scene videos associated with the plot units, video frames are extracted from the initial video to generate edited sub-videos corresponding to the plot units; finally, based on the edited sub-videos, an edited video of the initial video is generated.

[0012] The present disclosure can improve the performance and efficiency of video editing and editing materials, and reduce the difficulty of video editing.

[0013] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Other features, objects and advantages of the present disclosure will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 is an exemplary system architecture in which the present disclosure may be applied; Figure 2 A flowchart of a method for editing a video provided by an embodiment of the present disclosure; Figure 3 A flowchart of another method for editing videos provided by an embodiment of the present disclosure; Figure 4 A schematic diagram of an architecture for implementing a video editing process in an application scenario provided by an embodiment of the present disclosure; Figure 5 A structural block diagram of a video editing device provided by an embodiment of the present disclosure; Figure 6 A schematic structural diagram of an electronic device suitable for executing a method for editing videos provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other unless there is a conflict.

[0016] In addition, in the technical solutions involved in this disclosure, the acquisition, storage, use, processing, transportation, provision and disclosure of user personal information involved (such as videos, subtitles, etc. that may be provided by users or include users' personal information that may be involved in the subsequent disclosure) shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0017] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the method, apparatus, electronic device, and computer-readable storage medium for editing videos disclosed herein can be applied.

[0018] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0019] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed, such as video editing applications, short video generation applications, and instant messaging applications.

[0020] Terminal devices 101, 102, 103 and server 105 can be either hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules, or as a single software or software module, and are not specifically limited here. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and are not specifically limited here.

[0021] The server 105 can provide various services through various built-in applications. Taking the video editing application that can provide editing for the initial video as an example, the server 105 can achieve the following effects when running the video editing application: first, the initial video (which needs to be edited, as "material") is obtained from the terminal devices 101, 102, and 103 through the network 104; then, the server 105 performs scene segmentation on the initial video to obtain a scene video, wherein the scene video corresponds to the scene in the initial video; then, the server 105 associates the scene video to the plot unit of the initial video; then, the server 105 extracts video frames from the initial video based on the subtitles included in the scene video associated with the plot unit, and generates a clipped sub-video corresponding to the plot unit; finally, the server 105 generates a clipped video of the initial video based on the clipped sub-video.

[0022] It should be noted that, in addition to being obtained from terminal devices 101, 102, and 103 via network 104, the initial video can also be pre-stored locally on server 105 in various ways. Therefore, when server 105 detects that such data is already stored locally (for example, when it begins processing a previously reserved video editing task), it can choose to directly obtain such data locally. In this case, exemplary system architecture 100 may also not include terminal devices 101, 102, 103 and network 104.

[0023] Because scene segmentation and associating scene videos with plot units require significant computational resources and high computing power, the video editing methods provided in the subsequent embodiments of this disclosure are generally performed by a server 105 with relatively strong computing power and resources. Accordingly, the video editing apparatus is generally also located within the server 105. However, it should also be noted that, if terminal devices 101, 102, and 103 also possess sufficient computing power and resources, the terminal devices 101, 102, and 103 may also utilize a video editing application installed thereon to perform the aforementioned operations delegated to the server 105, thereby outputting the same results as the server 105. In particular, in the presence of multiple terminal devices with varying computing power, if the video editing application determines that the terminal device it is in possession of possesses relatively strong computing power and a relatively large amount of remaining computing resources, the terminal device may be instructed to perform the aforementioned operations, thereby appropriately alleviating the computational burden on the server 105. Accordingly, the video editing apparatus may also be located within the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104 .

[0024] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0025] Please refer to Figure 2 , Figure 2 A flowchart of a video editing process provided by an embodiment of the present disclosure, including process 200.

[0026] The process 200 specifically includes the following steps: Step 201: Segment the initial video into scenes to obtain scene videos; In the embodiment of the present disclosure, this step is intended to be performed by the execution subject of the method for editing the video (for example Figure 1 The server 105 shown in FIG. 1 is used to segment an initial video (e.g., a video uploaded by a user through terminal devices 101, 102, or 103 and desired to be edited) into scenes to generate scene videos. Accordingly, the resulting scene videos (i.e., a portion of the initial video) may correspond to scenes in the initial video.

[0027] For example, the execution entity can segment the initial video into scenes using a scene-consistency self-supervised representation learning solution, such as the SCRL (Scene Consistency Representation Learning) model. The video portions and video content of the initial video corresponding to or belonging to the same "scene" are then used as scene videos corresponding to that "same scene." Accordingly, the scene video can also be understood as a video segment consisting of video frames in the initial video that belong to the same "scene."

[0028] It should be noted that the initial video can be obtained by the above execution subject directly from a local storage device, or from a non-local storage device (such as Figure 1 The local storage device can be a data storage module provided within the execution entity, such as a server hard disk. In this case, the initial video can be quickly retrieved locally. The non-local storage device can also be any other electronic device configured to store data, such as a user terminal. In this case, the execution entity can obtain the required initial video by sending a retrieval command to the electronic device.

[0029] Step 202: Associating the scene video with the plot unit of the initial video; In the embodiment of the present disclosure, based on step 201, this step aims to have the aforementioned execution entity, after completing scene segmentation of the initial video and obtaining scene videos, associate each obtained scene video with a plot unit of the initial video. That is, after obtaining the scene videos, the execution entity determines the plot unit to which each scene video is associated.

[0030] In some optional implementations of this embodiment, the plot units can be obtained by the execution entity by splitting the complete plot of the initial video. For example, after obtaining the complete plot of the initial video, the execution entity can split the "complete plot" to obtain the individual plot units that constitute the "complete plot."

[0031] In some embodiments, the complete plot may be provided by the provider or producer of the original video, or may be obtained from the user who requested the video clip by the execution entity through interaction with the user.

[0032] The "complete plot" typically refers to the video and plot content included in the "initial video" in text form. For example, if the initial video is a movie, the "complete plot" could be "text information" composed of text descriptions and information about each event in the movie, organized according to the order in which the events are presented and performed in the movie.

[0033] Accordingly, the execution entity can, based on its understanding of a "complete plot," split the complete plot into multiple plot units according to unit division criteria, such as whether it includes a complete event. Accordingly, under such criteria, each split plot unit can completely include an "event."

[0034] In some optional implementations of this embodiment, after determining the scene video based on this step, the execution entity may further choose to obtain and read the subtitles included in the scene video (that is, the subtitles in the initial video that belong to the scene video part).

[0035] Then, the execution entity compares the subtitles included in the scene video with the text description information of each plot unit, determines the semantic similarity between the subtitles and each text description information, and obtains the semantic matching results between the subtitles included in the scene video and the text description information of each plot unit of the initial video.

[0036] For example, the execution entity may semantically match the subtitle text {s1, ....sn} of each scene video with the text description information {r1, ....rk} of each plot unit in chronological order to determine the semantic similarity between the two and obtain a "semantic matching result".

[0037] It should be understood that the scene videos corresponding to or associated with the same plot unit do not necessarily need to be temporally continuous, or in other words, there is no requirement for there to be no temporal gap between them, so as to avoid context loss.

[0038] Accordingly, after obtaining the semantic matching results, the execution entity can associate the scene video with the target plot unit with the highest semantic similarity in the semantic matching results, thereby completing the "association" between the scene video and the plot unit. Thus, the execution entity can achieve a "rough correspondence" between the scene video and the plot (unit) (i.e., a correspondence between the plot unit and the scene video) through semantic matching between "subtitles" and "plot."

[0039] Accordingly, after completing the above-mentioned "coarse correspondence", the subsequent execution entity can further associate and determine the corresponding "video frame" of the initial video in each plot unit through the "fine correspondence" (i.e., the correspondence between plot units and video frames) to be discussed below.

[0040] Therefore, through such a coarse and fine two-level correspondence method, not only can the correspondence quality be improved, but also excessive demand and use of computing resources in one operation can be avoided, thereby reducing the configuration requirements for computing resources.

[0041] In some embodiments, after obtaining the complete plot of the initial video, the execution entity may choose to call a large language model to split the complete plot into multiple plot units.

[0042] A Large Language Model (LLM) is an artificial intelligence model that can be used to understand and generate human language, and based on its understanding, it can perform processing operations to produce corresponding results. For example, after obtaining a complete plot, a pre-configured prompt can be used to instruct the LLM to split the complete plot into multiple plot units according to certain criteria (such as the "event" criteria mentioned above).

[0043] Typically, LLMs can be trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. LLMs are characterized by their large scale and can typically include a large number of parameters to help them learn complex patterns in language data. These models are typically based on deep learning architectures, such as transformers, which help them provide better processing performance on various NLP tasks. In an embodiment of the present disclosure, the execution entity can use a generative large language model (e.g., an LLM) as a "plot splitting model" to process and split the complete plot into plot units. This allows the "complete plot" to be split into "plot units" faster and with higher quality.

[0044] In some embodiments, the LLM can be configured to omit the "guide word" by default to further speed up its invocation. For example, after obtaining a complete plot, to split it into multiple plot units, the LLM can, based on the default configuration, automatically understand the input as a "complete plot" and the operations to be performed on it. This default configuration allows the LLM to process the "complete plot" stably and precisely, splitting it into "plot units," and speeding up its invocation.

[0045] In some embodiments, as discussed above, the complete plot may be obtained by the executing entity through the producer of the initial video or other users (for example, the other users may be users who provide evaluations and comments for the initial video). In such a case, the user can also send an acquisition instruction for the complete plot to the executing entity through the target device used by the user (for example, terminal devices 101, 102, 103) to obtain the complete plot.

[0046] Accordingly, if the execution subject receives an acquisition instruction sent by the target device for the complete plot, the execution subject may respond thereto by providing the complete plot to the target device.

[0047] Then, the user can make personalized adjustments to the complete plot according to his or her personalized needs (for example, adjustments to the content, length, etc. of the complete plot), and return updated information about the complete plot to the execution entity.

[0048] Accordingly, if the execution entity receives update information for the complete plot, it can respond to it and update the complete plot based on the update information. Then, the execution entity selects the updated complete plot as input to the LLM and calls the LLM to split the updated complete plot into multiple plot units.

[0049] Thus, through interaction with users, the execution entity can dynamically and individually meet the different users' understanding and usage needs of the plot, so that it can subsequently provide users with "edited videos" that better meet their plot needs.

[0050] When utilizing the LLM, based on its processing capabilities, the executing entity can similarly choose to delegate the step of "associating the scene video with the target plot unit with the highest semantic similarity in the semantic matching results" to the LLM. For example, the executing entity can similarly complete the step of "associating the scene video with the plot unit of the initial video" by invoking the LLM through prompts in other content forms (for example, after generating semantic matching results between the subtitles included in the scene video and the textual descriptions of each plot unit in the initial video, associating the scene video with the target plot unit with the highest semantic similarity in the semantic matching results). This improves the accuracy of determining "semantic similarity."

[0051] In some embodiments, the LLM used to complete the step of "associating the scene video to the target plot unit with the highest semantic similarity in the semantic matching results based on the semantic matching results" can be the same LLM as the LLM for the step of "splitting the complete plot" (that is, an LLM obtained based on the same initial LLM training and having both capabilities).

[0052] As a result, the execution entity can implement the above two steps by calling a complete LLM, simplifying the calling logic, and at the same time, the LLM can be based on the processing results of different steps to reduce the possibility of model hallucination caused by training differences between different LLMs due to cross-LLM.

[0053] Step 203: extracting video frames from the initial video based on the subtitles included in the scene video associated with the plot unit, and generating a clipped sub-video corresponding to the plot unit; In the disclosed embodiment, building on step 202, this step aims to extract video frames from the initial video using the subtitles of the scene video belonging to and associated with the same plot unit, after determining the association between the plot units and the scene videos. Accordingly, the execution entity can generate a sub-video clip corresponding to the plot units based on these video frames, thereby effectively completing the subsequent "fine-grained correspondence" (i.e., more specifically, "corresponding" the plot units to the video frames, achieved through the use of subtitles).

[0054] For example, the execution entity may arrange the scene videos in the order in which they appear in the initial video, and then determine a time interval based on the time when the subtitles first appear in the first scene video and the time when the subtitles last appear in the last scene video.

[0055] Then, the execution entity extracts the video frames of the initial video within the time interval to compose and generate the clip sub-video corresponding to the plot unit.

[0056] In some optional implementations of this embodiment, the executing entity may, as an alternative or alternative, choose to first determine the combination result of the subtitles of the scene videos associated with the plot unit when executing this step (that is, as discussed above, the subtitles of each scene video are arranged in sequence according to the chronological order of the appearance of the scene videos to achieve "combination" and obtain the combination result of the subtitles).

[0057] Then, based on the combination result, the execution entity more specifically selects and determines the time stamp of each subtitle included in the combination result.

[0058] Then, the execution entity extracts corresponding video frames from the original video based on the timestamps of the subtitles in the combination result, that is, only extracts the video frames including the “subtitles”.

[0059] Finally, the execution body combines the extracted video frames in the order of playback time to generate a clip sub-video corresponding to the plot unit.

[0060] Therefore, in this way, the execution entity can further "eliminate" those plots that cannot be obtained from the "subtitles", such as actions, atmosphere, etc., so that the plot and subtitles can be matched and corresponded more strictly, and obtain more accurate and fine-grained plot matching results.

[0061] Similarly, in some embodiments, the execution entity may also implement the "detailed correspondence" in this step by calling the LLM (for example, changing the prompt word, training the LLM in this capability, or configuring other LLMs with this capability, etc.), which will not be repeated here.

[0062] Step 204: Generate a clipped video of the initial video based on the clipped sub-video.

[0063] In an embodiment of the present disclosure, based on step 203 , the execution entity may use the various clip sub-videos generated in step 203 to generate a clip video of the initial video.

[0064] In this step, the "edited video" can be generated using all the edited sub-videos or using part of the edited sub-videos.

[0065] In some optional implementations of this embodiment, the execution entity may combine the clipped sub-videos corresponding to the selected plot units based on the user's selection of the plot units to generate a clipped video of the initial video.

[0066] For example, the execution entity can provide the split plot units to the user (for example, providing them to the target device mentioned above, or using a local presentation component to present the plot units to the user). Accordingly, after receiving these plot units, the user can select the required plot units based on their needs, as well as the order in which the plot units are arranged (for example, the user can indicate that the content of certain plot units occurring later should be presented first in the edited video based on different user needs), thereby generating a selection instruction for the plot units.

[0067] Then, the user may return the selection instruction to the execution entity to instruct the execution entity to generate a clipped video.

[0068] Accordingly, if the execution entity receives a selection instruction for a plot unit, it can respond to this by combining the clip sub-videos corresponding to the plot unit indicated by the selection instruction in the order indicated by the selection instruction to generate a clip video of the initial video (for example, arranging these clip sub-videos in sequence for combination).

[0069] Therefore, it is possible to provide users with personalized editing services for the initial video based on the "plot" dimension according to their different needs.

[0070] In some optional implementations of this embodiment, the user may also send a global generation instruction for the initial video to the execution entity to instruct the execution entity to use all the edited sub-videos to generate an edited video with the complete plot of the initial video, which serves as a "thumbnail version" of the initial video.

[0071] Accordingly, if the execution subject receives a global generation instruction for the initial video, it can respond to this by combining the sub-videos corresponding to the plot units indicated by the complete plot of the initial video, thereby generating a clipped video of the initial video. In this way, the user is provided with a "clipped video" that is a thumbnail of the initial video based on the plot content, allowing the user to conveniently and efficiently complete the clipping of the extraction results of the entire plot content of the initial video.

[0072] The video editing method provided by the disclosed embodiments divides an initial video into scenes to generate scene videos, wherein the scene videos correspond to scenes in the initial video; the scene videos are associated with plot units of the initial video; video frames are extracted from the initial video based on subtitles included in the scene videos associated with the plot units to generate editing sub-videos corresponding to the plot units; and a editing video of the initial video is generated based on the editing sub-videos. This improves the performance and efficiency of video editing and editing materials, and reduces the difficulty of video editing.

[0073] In some embodiments, in the process of associating scene videos and scene units in step 202, for example, the execution entity may also improve the quality of the used scene videos by eliminating scene videos that are outliers from the plot unit and may cause errors.

[0074] In some embodiments, if at least two scene videos are associated with the same plot unit (referred to as the target plot unit for ease of understanding), the execution entity may respond by determining a time reference point based on the start times of the scene videos associated with the target plot unit. For example, the time reference point may be the "average" of the start times (i.e., the sum of the start times is calculated and then divided by the number of scene videos associated with the target plot unit).

[0075] Then, if the time distance between the start time of the target scene video associated with the target plot unit and the time reference point is greater than or equal to a first distance threshold (generally, the size of the "first distance threshold" can be preset based on the size of the time distance that is considered to be possibly outliers), the execution entity can consider that the target scene video is "outliers", and then cancel the association relationship between the target video and the target plot unit to avoid interference caused by the "outlier" target scene video.

[0076] In some embodiments, when there are two or more scene videos associated with the same plot unit, the executing entity may also choose to divide the subtitles subsequently used to achieve "fine correspondence" according to the scene boundaries to avoid the situation where the subtitles (corresponding plot content) have a large span due to scene differences.

[0077] For this, please refer to Figure 3 , Figure 3 The flowchart of a process of generating a clip sub-video corresponding to a plot unit provided in an embodiment of the present disclosure includes process 300. In some embodiments, process 300 can be implemented as an alternative or substitute for step 203 above.

[0078] The process 300 specifically includes the following steps: Step 301: combining subtitles included in scene videos associated with plot units to obtain an initial combination result; Specifically, in this step, the execution subject can be similar to the discussion above, first combining the subtitles in the various scene videos that belong to and are simultaneously associated with the same plot unit to obtain an initial combination result (for example, arranging the various subtitles accordingly according to the order of the scene videos in the initial video to achieve "combination").

[0079] Step 302: If the initial combination result includes the first subtitle and the second subtitle, determine whether the time distance between the start times of the first subtitle and the second subtitle is less than a second distance threshold; Specifically, if the execution entity detects and determines that the initial combination result includes the first subtitle and the second subtitle from different scene videos (for example, the first subtitle comes from the first scene video, and the second subtitle comes from the second scene video), the execution entity can further determine whether the time distance between the start time of the first subtitle and the second subtitle is less than a second distance threshold (in some embodiments, the size of the "second distance threshold" can be determined in advance based on the standard that the distance between the first subtitle and the second subtitle is close and there will not be obvious content difference due to the span between the two).

[0080] If it is less, the execution body may continue to perform step 303 to further determine whether the first subtitles and / or the second subtitles are available with reference to the “plot unit”.

[0081] Step 303: Obtain a first time center of a first plot unit and a second time center of a second plot unit adjacent to the plot unit; Specifically, a first time center of a first plot unit and a second time center of a second plot unit adjacent to the plot unit are obtained (for example, the first plot unit may be the previous plot unit adjacent to the plot unit, and the second plot unit may be the next plot unit adjacent to the plot unit).

[0082] For the time center, taking the first time center as an example (the second time center is determined in a similar manner to the first time center and will not be repeated), the first time center can be the time point in the initial video based on the subtitles corresponding to the "key plot" of the first plot unit (that is, the time point corresponding to the subtitles of the core plot and key plot in the plot unit).

[0083] Step 304: summing the distance between the first subtitle and the first time center and the distance between the first subtitle and the second time center to obtain a first distance, and summing the distance between the second subtitle and the first time center and the distance between the second subtitle and the second time center to obtain a second distance; Specifically, the execution entity may use the time centers of the first and second subtitles to represent the first and second subtitles, and then determine the distance between the "time centers" representing the first and second subtitles and the first time center, and the distance between the "time centers" representing the first and second subtitles and the second time center, respectively. These two distances are then added together to obtain a first distance corresponding to the first subtitle and a second distance corresponding to the second subtitle.

[0084] Step 305: Compare the first distance and the second distance; If the first distance is smaller than the second distance, it can be considered that the first subtitle is closer to the plot center of the plot unit. In this case, the execution entity may choose to execute step 306 to retain the first subtitle in the initial combination result and delete the second subtitle.

[0085] Similarly, if the first distance is greater than the second distance, it can be considered that the second subtitle is closer to the plot center of the plot unit. In this case, the execution entity can choose to execute step 307, retain the second subtitle in the initial combination result, and delete the second subtitle.

[0086] Step 306: retain the first subtitle in the initial combination result and delete the second subtitle to obtain a combination result; Step 307: Delete the first subtitle from the initial combination result to obtain a combination result; Step 308: Based on the subtitles included in the combination result, video frames are extracted from the initial video to generate a clipped sub-video corresponding to the plot unit.

[0087] This step is similar to the process of generating a clipped sub-video using the "combination result" discussed above in step 203, and will not be repeated here.

[0088] In addition, in some embodiments, if the first distance and the second distance are equal, it can be considered that the correlation between the two and the "plot center" may be comparable. In this case, in order to avoid content loss, you can choose to retain the two and later generate the corresponding clip sub-video of the plot unit. The explanation will not be repeated here.

[0089] In some optional implementations of this embodiment, as discussed above, if the distance between the first subtitle and the second subtitle is already far, and there may be obvious differences in content due to the span between the two, the executor may choose to delete at least one of the two to avoid errors.

[0090] For example, the execution entity may select the subtitle that appears earlier, later, or delete both (eg, if there are other subtitles that can be used).

[0091] For example, the discussion will be based on an embodiment of deleting subtitles that start later in time. In this case, process 300 may further include step 309. For example, in step 302, if it is determined that the time distance between the start times of the first subtitle and the second subtitle is greater than or equal to a second distance threshold, the execution entity may choose to execute step 309.

[0092] Step 309: Compare the start time of the first subtitle and the start time of the second subtitle; If the start time of the first subtitle is later than that of the second subtitle, the execution entity may choose to continue executing step 310 .

[0093] If the start time of the first subtitle is earlier than that of the second subtitle, the execution entity may choose to continue executing step 311 .

[0094] Step 310: Delete the first subtitle from the initial combination result to obtain a combination result; Step 311: Delete the second subtitle from the initial combination result to obtain a combination result.

[0095] Subsequently, after obtaining the combination result, the execution entity can similarly jump to the above step 308 to extract video frames from the initial video based on the subtitles included in the combination result and generate a clip sub-video corresponding to the plot unit, which will not be repeated here.

[0096] To deepen understanding, this disclosure also provides a specific implementation solution in combination with a specific application scenario. For this, please refer to Figure 4 . Figure 4This is a schematic diagram of an architecture 400 for implementing a video editing process in an application scenario provided by an embodiment of the present disclosure. For example, the architecture 400 may be implemented by the aforementioned server 105 as an execution subject.

[0097] In architecture 400 , after receiving initial video 410 , server 105 , acting as the execution entity, may execute step S401 to segment initial video 410 into scenes, thereby obtaining corresponding scene videos 421 , 422 , ... 42N (where N is a positive integer). Each of scene videos 421 , 422 , ... 42N may correspond to a "scene" in initial video 410 .

[0098] Then, the server 105 may execute S402 to extract the “subtitles” included in each of the scene videos 421 , 422 . . . 42N.

[0099] For ease of understanding, taking the scene video 421 as an example, the server 105 can extract the subtitles 431 of the scene video 421 by executing S402.

[0100] For the purpose of easy understanding, only the processing process related to scene video 421 is similarly shown in the following. The processing processes of scene videos 422...42N can be implemented similarly to the processing process related to scene video 421, and will not be repeated here.

[0101] Furthermore, server 105 may execute S403 based on subtitles 431 to associate scene video 421 with one of plot units 441, 442, ..., 44N (where N is a positive integer. "N" here may be the same as or different from the "N" in the scene video, and this disclosure is not intended to be limiting). Plot units 441, 442, ..., 44N may be obtained by splitting the complete plot (not shown) of initial video 410.

[0102] For example, the server 105 may associate the scene video 421 with the plot unit 441 based on the “semantic matching result”.

[0103] Next, the server 105 may continue to execute S404 to extract video frames from the initial video 410 based on the subtitles 431 (for example, extracting "video frames" from the initial video 410 accordingly according to the "timestamps" corresponding to the subtitles 431) and generate a clip sub-video 451 for the plot unit 441 (for example, arranging and combining these video frames in the order of "video frames" to obtain the clip sub-video 451).

[0104] It should be understood that the video frames are extracted here only as an example based on the "subtitles" of the scene video 421, but as discussed above, if there are two or more scene videos simultaneously associated with the plot unit 441, the server 105 can simultaneously use the "subtitles" of two or more scene videos to extract video frames, which will not be repeated here.

[0105] Next, the server 105 may similarly determine the clipped sub-videos (eg, clipped sub-videos 451 , 452 . . . 45N) corresponding to the respective plot units 441 , 442 . . . 44N through the above process.

[0106] After determining the sub-video clips 451, 452, ..., 45N, server 105 may generate a video clip 460 of initial video 410 based on the sub-video clips 451, 452, ..., 45N by executing S405. For example, upon receiving a "global generation instruction" (not shown), server 105 may choose to utilize all of the sub-video clips 451, 452, ..., 45N (e.g., combining the sub-video clips 451, 452, ..., 45N in the order of the plot units 441, 442, ..., 44N to generate video clip 460).

[0107] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for editing a video. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0108] like Figure 5 As shown, the apparatus 500 for video editing of this embodiment may include: a scene video segmentation unit 501, a scene video association unit 502, a clip sub-video generation unit 503, and a clip video generation unit 504. The scene video segmentation unit 501 is configured to segment an initial video into scenes to obtain scene videos, wherein the scene videos correspond to scenes in the initial video; the scene video association unit 502 is configured to associate the scene videos with plot units of the initial video; the clip sub-video generation unit 503 is configured to extract video frames from the initial video based on subtitles included in the scene video associated with the plot unit, and generate a clip sub-video corresponding to the plot unit; and the clip video generation unit 504 is configured to generate a clip video of the initial video based on the clip sub-video.

[0109] In this embodiment, the specific processing of the scene video segmentation unit 501, the scene video association unit 502, the clip sub-video generation unit 503 and the clip video generation unit 504 and the technical effects thereof can be referred to in detail. Figure 2The relevant descriptions of steps 201-204 in the corresponding embodiment are not repeated here.

[0110] In some optional implementations of this embodiment, the clip sub-video generation unit 503 includes: a video frame extraction sub-unit, which is configured to extract corresponding video frames from the initial video based on the timestamps of each subtitle in the combination result of the subtitles of the scene video associated with the plot unit; and a video frame combination sub-unit, which is configured to combine the various video frames in the order of playback time to generate a clip sub-video corresponding to the plot unit.

[0111] In some optional implementations of this embodiment, the device 500 also includes: a time reference point determination unit, configured to determine the time reference point based on the start time of each scene video associated with the target plot unit in response to the existence of at least two scene videos associated with the target plot unit; an association relationship update unit, configured to cancel the association relationship between the target video and the target plot unit in response to the time distance between the start time of the target scene video associated with the target plot unit and the time reference point being greater than or equal to a first distance threshold.

[0112] In some optional implementations of this embodiment, the clip sub-video generation unit 503 includes: a subtitle combination sub-unit, configured to combine the subtitles included in the scene video associated with the plot unit to obtain an initial combination result; a time center determination sub-unit, configured to, in response to the initial combination result including the first subtitle and the second subtitle, and the time distance between the start time of the first subtitle and the second subtitle being less than a second distance threshold, obtain a first time center of the first plot unit and a second time center of the second plot unit adjacent to the plot unit, wherein the first subtitle and the second subtitle are from different scene videos; a time center summing sub-unit, configured to sum the first subtitle with the second subtitle. The distance between the first time center and the distance between the first subtitle and the second time center are added to obtain the first distance, and the distance between the second subtitle and the first time center and the distance between the second subtitle and the second time center are added to obtain the second distance; the first initial combination update subunit is configured to delete the second subtitle in the initial combination result in response to the first distance being less than the second distance to obtain the combination result; or to delete the first subtitle in the initial combination result in response to the first distance being greater than the second distance to obtain the combination result; the clip sub-video generation subunit is configured to extract video frames from the initial video based on the subtitles included in the combination result to generate a clip sub-video corresponding to the plot unit.

[0113] In some optional implementations of this embodiment, the clip sub-video generation unit 503 also includes: a subtitle start time comparison subunit, configured to compare the start times of the first subtitle and the second subtitle in response to the initial combination result including the first subtitle and the second subtitle, and the time distance between the start times of the first subtitle and the second subtitle is greater than or equal to a second distance threshold; a second initial combination update subunit, configured to delete the first subtitle from the initial combination result to obtain a combination result in response to the start time of the first subtitle being later than that of the second subtitle; or to delete the second subtitle from the initial combination result to obtain a combination result in response to the start time of the first subtitle being earlier than that of the second subtitle.

[0114] In some optional implementations of this embodiment, the scene video association unit 502 is further configured to associate the scene video to the target plot unit with the highest semantic similarity in the semantic matching results based on the semantic matching results between the subtitles included in the scene video and the text description information of each plot unit of the initial video.

[0115] In some optional implementations of this embodiment, the scene video association unit 502 is further configured to generate semantic matching results between the subtitles included in the scene video and the text description information of each plot unit of the initial video by calling a large language model, and associate the scene video to the target plot unit with the highest semantic similarity in the semantic matching results.

[0116] In some optional implementations of this embodiment, the device 500 also includes: a complete plot acquisition unit, configured to obtain the complete plot of the initial video; and a plot unit splitting unit, configured to call the large language model to split the complete plot into multiple plot units.

[0117] In some optional implementations of this embodiment, the device 500 also includes: a complete plot providing unit, configured to provide the complete plot to the target device in response to receiving an acquisition instruction sent by the target device for the complete plot; a complete plot updating unit, configured to update the complete plot based on the update information in response to receiving update information for the complete plot; and a plot unit splitting unit, further configured to call the large language model to split the updated complete plot into multiple plot units.

[0118] In some optional implementations of this embodiment, the clip video generation unit 504 is further configured to, in response to receiving a selection instruction for a plot unit, combine the clip sub-videos corresponding to the plot unit indicated by the selection instruction in the order indicated by the selection instruction to generate a clip video of the initial video.

[0119] In some optional implementations of this embodiment, the clip video generation unit 504 is further configured to, in response to receiving a global generation instruction for the initial video, combine the clip sub-videos corresponding to the respective plot units based on the order of the plot units indicated by the complete plot of the initial video to generate a clip video of the initial video.

[0120] This embodiment, as an apparatus embodiment corresponding to the above-described method embodiment, provides a video editing apparatus that segments an initial video into scenes to generate scene videos, wherein the scene videos correspond to scenes in the initial video; associates the scene videos with plot units of the initial video; extracts video frames from the initial video based on subtitles included in the scene videos associated with the plot units to generate edited sub-videos corresponding to the plot units; and generates an edited video of the initial video based on the edited sub-videos. This improves the performance and efficiency of video editing and editing, and reduces the difficulty of video editing.

[0121] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0122] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0123] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. Computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0124] Various components in device 600 are connected to I / O interface 605, including an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0125] The computing unit 601 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the method for editing a video. For example, in some embodiments, the method for editing a video can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method for editing a video described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the method for editing a video by any other suitable means (e.g., via firmware).

[0126] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0128] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0131] A computer system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and virtual private server (VPS) services. Servers can also be classified as distributed system servers or servers integrated with blockchain.

[0132] According to the technical solution of the disclosed embodiments, an initial video is segmented into scenes to generate scene videos corresponding to scenes in the initial video; the scene videos are then associated with plot units of the initial video; video frames are extracted from the initial video based on the subtitles included in the scene videos associated with the plot units to generate clipped sub-videos corresponding to the plot units; and a clipped video of the initial video is generated based on the clipped sub-videos. This improves the performance and efficiency of video editing and editing, and reduces the difficulty of video editing.

[0133] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions provided by this disclosure can be achieved. This is not limited herein.

[0134] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for editing a video, comprising: Performing scene segmentation on the initial video to obtain scene videos, wherein the scene videos correspond to scenes in the initial video; Associating the scene video with the plot unit of the initial video; Extracting video frames from the initial video based on subtitles included in the scene video associated with the plot unit to generate a clipped sub-video corresponding to the plot unit; Based on the clipped sub-video, a clipped video of the initial video is generated.

2. The method according to claim 1, wherein Extracting video frames from the initial video based on subtitles included in the scene video associated with the plot unit to generate a clip sub-video corresponding to the plot unit includes: Extracting corresponding video frames from the initial video based on the timestamps of the respective subtitles in the combined result of the subtitles of the scene videos associated with the plot unit; The video frames are combined in a playback time sequence to generate a clip sub-video corresponding to the plot unit.

3. The method according to claim 1, further comprising: In response to at least two of the scene videos being associated with the target plot unit, determining a time reference point based on a start time of each scene video associated with the target plot unit; In response to a time distance between a start time of a target scene video associated with the target plot unit and the time reference point being greater than or equal to a first distance threshold, the association relationship between the target video and the target plot unit is cancelled.

4. The method according to claim 1, wherein The extracting of video frames from the initial video based on the subtitles included in the scene video associated with the plot unit to generate a clipped sub-video corresponding to the plot unit includes: Combining the subtitles included in the scene videos associated with the plot unit to obtain an initial combination result; In response to the initial combination result including a first subtitle and a second subtitle, and the time distance between the start times of the first subtitle and the second subtitle being less than a second distance threshold, obtaining a first time center of a first plot unit and a second time center of a second plot unit adjacent to the plot unit, wherein the first subtitle and the second subtitle are from different scene videos; Adding the distance between the first subtitle and the first time center and the distance between the first subtitle and the second time center to obtain a first distance, and adding the distance between the second subtitle and the first time center and the distance between the second subtitle and the second time center to obtain a second distance; In response to the first distance being smaller than the second distance, deleting the second subtitle from the initial combination result to obtain a combination result; or In response to the first distance being greater than the second distance, deleting the first subtitle from the initial combination result to obtain a combination result; Based on the subtitles included in the combination result, video frames are extracted from the initial video to generate a clip sub-video corresponding to the plot unit.

5. The method according to claim 4, further comprising: In response to the initial combination result including the first subtitle and the second subtitle, and the time distance between the start times of the first subtitle and the second subtitle being greater than or equal to the second distance threshold, comparing the start times of the first subtitle and the second subtitle; In response to the start time of the first subtitle being later than the start time of the second subtitle, deleting the first subtitle from the initial combined result to obtain a combined result; or In response to the start time of the first subtitle being earlier than the start time of the second subtitle, the second subtitle is deleted from the initial combination result to obtain a combination result.

6. The method according to claim 1, wherein The step of associating the scene video with the plot unit of the initial video includes: Based on the semantic matching results between the subtitles included in the scene video and the text description information of each plot unit of the initial video, the scene video is associated with the target plot unit with the highest semantic similarity in the semantic matching results.

7. The method according to claim 6, wherein: The step of associating the scene video with a target plot unit having the highest semantic similarity in the semantic matching results based on the semantic matching results between the subtitles included in the scene video and the text description information of each plot unit of the initial video comprises: By calling a large language model to generate semantic matching results between the subtitles included in the scene video and the text description information of each plot unit of the initial video, the scene video is associated with the target plot unit with the highest semantic similarity in the semantic matching results.

8. The method according to claim 1, further comprising: Obtain the complete plot of the initial video; The large language model is called to split the complete plot into multiple plot units.

9. The method according to claim 8, further comprising: In response to receiving an acquisition instruction for the complete plot sent by a target device, providing the complete plot to the target device; In response to receiving update information for the complete plot, updating the complete plot based on the update information; as well as The calling of the large language model to split the complete plot into a plurality of plot units includes: The large language model is called to split the updated complete plot into the multiple plot units.

10. The method according to any one of claims 1 to 9, wherein The step of generating a clipped video of the initial video based on the clipped sub-video includes: In response to receiving the selection instruction of the plot unit, the clipped sub-videos corresponding to the plot unit indicated by the selection instruction are combined in the order indicated by the selection instruction to generate a clipped video of the initial video.

11. The method according to claim 10, further comprising: In response to receiving a global generation instruction for the initial video, based on the order of the plot units indicated by the complete plot of the initial video, the clip sub-videos corresponding to the respective plot units are combined accordingly to generate a clip video of the initial video.

12. A device for editing a video, comprising: A scene video segmentation unit is configured to segment the initial video into scenes to obtain scene videos, wherein the scene videos correspond to scenes in the initial video; a scene video associating unit, configured to associate the scene video with the plot unit of the initial video; a clip sub-video generating unit configured to extract video frames from the initial video based on subtitles included in the scene video associated with the plot unit, and generate a clip sub-video corresponding to the plot unit; The clip video generating unit is configured to generate a clip video of the initial video based on the clip sub-video.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for editing videos according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method for editing a video according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method for editing a video according to any one of claims 1 to 11.

Images