Video clip method, device, storage medium, and program product

Abstract

The application provides a video clipping method, device, storage medium and program product. The video clipping method comprises: acquiring a live streaming segment that meets a clipping time, wherein the live streaming segment is a segment corresponding to live streaming data generated in live streaming; clipping a head segment and / or a tail segment in the live streaming segment based on the clipping time; and generating a clipping video corresponding to the live streaming data based on the clipped head segment and / or the tail segment. The video is clipped based on the segment, without waiting for the segment to be processed into a recording file, so that the content being live streamed can be quickly clipped into a short video, and the efficiency of video clipping is improved.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video editing method, device, storage medium, and program product. Background Technology

[0002] With the rapid development of live streaming technology, more and more users are choosing to spread or obtain information through live streaming.

[0003] In some scenarios, streamers need to edit the videos generated during the live stream, such as editing the highlights and key moments into a short video to help viewers better understand the content of the live stream or to increase viewer engagement.

[0004] However, existing video editing software can only edit pre-recorded videos, resulting in a long delay between the edited short videos and the live stream, which is not timely and cannot meet user needs. Summary of the Invention

[0005] This application provides a video editing method, device, storage medium, and program product, which realizes simultaneous playback, recording, and editing, improving the timeliness of editing, and achieves short video synthesis by slicing live stream data, thereby improving editing efficiency.

[0006] Firstly, this application provides a video editing method, including:

[0007] Obtain the live stream segment whose editing time is matched, wherein the live stream segment is the segment corresponding to the live stream data generated during the live broadcast;

[0008] Based on the editing time, the first and / or last segments of the live stream are edited;

[0009] Based on the cut head and tail segments, an edited video corresponding to the live stream data is generated.

[0010] Optionally, based on the editing time, the first and / or last segments of the live stream are edited, including:

[0011] Decode the head segment and the tail segment;

[0012] Based on the start and end times in the clipping time, the decoded head and tail segments are re-encoded so that the start time of the re-encoded head segment is aligned with the start time of the clipping time, and the end time of the re-encoded tail segment is aligned with the end time of the clipping time.

[0013] Optionally, based on the start and end times in the clipping time, the decoded first and last segments are re-encoded, including:

[0014] Obtain the segmentation parameters of the live stream segment;

[0015] Based on the start and end times in the editing time and the segmentation parameters, the decoded head segment and tail segment are re-encoded to make the segmentation parameters of the re-encoded head segment and tail segment consistent with the segmentation parameters of the live stream segment.

[0016] Optionally, based on the edited first and / or last segments, an edited video corresponding to the live stream data is generated, including:

[0017] Based on the intermediate segments and the edited head and tail segments, a first index list file is generated, wherein the intermediate segments are the remaining segments after removing the head and tail segments from the hit live stream segments.

[0018] The edited video is generated based on the first index list file.

[0019] Optionally, based on the edited first and / or last segments, an edited video corresponding to the live stream data is generated, including:

[0020] Based on the intermediate segments, a second index list file is generated, wherein the intermediate segments are the remaining segments after removing the head segment and the tail segment from the hit live stream segments;

[0021] The second index list file, along with the edited first and last segments, are transcoded to obtain the edited video.

[0022] Optionally, after editing the first and / or last segments of the live stream based on the editing time, the method further includes:

[0023] If the length of the edited header segment is less than the preset length, then the edited header segment is merged with its adjacent segments to obtain a new edited header segment; and / or,

[0024] If the length of the edited tail segment is less than the preset length, then the edited tail segment is merged with the adjacent segments of the tail segment to obtain a new edited tail segment.

[0025] Optionally, the method further includes:

[0026] In response to a live streaming editing command from a user who is currently streaming, obtain the live stream segment corresponding to that user;

[0027] The editing time is determined based on the live stream segment corresponding to the user.

[0028] Optionally, the editing time is determined based on the live stream segment corresponding to the user, including:

[0029] The live stream segments corresponding to the user are input into the time prediction model, and the editing time is determined based on the time prediction model.

[0030] Secondly, this application provides another video editing method, which is applied to a cloud server and includes:

[0031] Retrieve live stream data generated during live streaming from the live streaming client, sent by the live streaming server;

[0032] The live stream data is sliced ​​to obtain multiple live stream segments;

[0033] In response to the editing instructions issued by the live streaming server, the target live stream segment whose editing time is matched in the editing instructions is obtained;

[0034] Based on the editing time, the first and / or last segments of the target live stream are edited;

[0035] Generate an edited video based on the edited first and / or last clips;

[0036] The edited video is sent to the live streaming server so that the edited video can be sent to the live streaming client through the live streaming server.

[0037] Thirdly, this application provides another video editing method, which is applied to a live streaming server and includes:

[0038] In response to the editing instructions issued by the live streaming client, the target live stream segment whose editing time is matched in the editing instructions is obtained from the cloud server;

[0039] Based on the editing time, the first and / or last segments of the target live stream are edited;

[0040] Generate an edited video based on the edited first and / or last clips;

[0041] The edited video is sent to the live streaming client.

[0042] Fourthly, this application provides another video editing method, including:

[0043] Obtain the video stream segment whose editing time matches, wherein the video stream segment is the segment corresponding to the streaming media format video;

[0044] Based on the editing time, the first and / or last segments of the video stream are edited;

[0045] Based on the edited first and last segments, a clipped video corresponding to the streaming media format video is generated.

[0046] Fifthly, this application provides a video editing apparatus, comprising:

[0047] The segment acquisition module is used to acquire the live stream segments that are matched by the editing time, wherein the live stream segments are the segments corresponding to the live stream data generated during the live broadcast;

[0048] A local editing module is used to edit the first and / or last segments of the live stream based on the editing time.

[0049] The video editing generation module is used to generate an edited video corresponding to the live stream data based on the edited first and / or last segments.

[0050] Sixthly, this application provides a video editing device, comprising:

[0051] A processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the video editing method provided in any aspect of this application.

[0052] In a seventh aspect, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the video editing method provided in any aspect of this application.

[0053] Eighthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the video editing method provided in any aspect of this application.

[0054] The video editing method, device, storage medium, and program products provided in this application, in order to improve the efficiency and timeliness of video editing based on live stream data generated during live streaming, propose a concept for synthesizing edited videos based on live stream segments obtained during live streaming. Specifically, it involves directly editing the first and / or last segments of the live stream segments whose editing time is matched, including decoding and re-encoding. Based on the edited first and / or last segments, as well as intermediate segments (if any), an edited video is generated, thereby realizing the editing of live content. This eliminates the need to wait for multiple segments to be periodically processed into a single recording file and then edited to obtain the edited video, improving the timeliness of video editing. At the same time, editing only the first and last segments, without needing to edit all segments, improves the efficiency of editing and further enhances the efficiency of synthesizing the edited video. Attached Figure Description

[0055] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0056] Figure 1 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0057] Figure 2 A flowchart illustrating a video editing method provided in an embodiment of this application;

[0058] Figure 3 For this application Figure 2 A flowchart illustrating step S203 in the illustrated embodiment;

[0059] Figure 4 For this application Figure 3 A schematic diagram of the first index list file generation process in the illustrated embodiment;

[0060] Figure 5 A flowchart illustrating another video editing method provided in an embodiment of this application;

[0061] Figure 6 For this application Figure 5 A schematic diagram illustrating the interaction process between the recording device and the cloud editing device in the illustrated embodiment;

[0062] Figure 7 For this application Figure 5 A schematic diagram of the video editing generation process in the illustrated embodiment;

[0063] Figure 8 A schematic diagram illustrating the video editing process provided in another embodiment of this application;

[0064] Figure 9 A flowchart illustrating yet another video editing method provided in an embodiment of this application;

[0065] Figure 10 For this application Figure 9 A schematic diagram of the video editing generation process in the illustrated embodiment;

[0066] Figure 11 This is a schematic diagram of the structure of a video editing device provided in an embodiment of this application;

[0067] Figure 12 This is a schematic diagram of the structure of a video editing device provided in an embodiment of this application.

[0068] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0069] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0070] First, let me explain the names involved in this application:

[0071] Live streaming refers to the process of pushing the video stream generated during a live broadcast to the network, such as to a cloud server.

[0072] Live stream data: Video stream data generated during live streaming.

[0073] Video editing: The process of non-linear editing of audio and video materials, including secondary encoding such as splitting, merging, transcoding, and adding features to the materials, in order to generate new videos with different expressive power.

[0074] Segmentation: When storing audio and video files in streaming media formats in the cloud, the files are usually divided into multiple segments, such as TS (Transport Stream, MPEG2-TS) segments and MPG (Moving Pictures Experts Group) segments.

[0075] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application, such as... Figure 1 As shown, during a live stream, the host broadcasts through a live streaming terminal. The terminal pushes the collected live stream data to a cloud server in real time. The cloud server then slices the live stream data into multiple live stream segments, such as... Figure 1 The live stream segments 1 to N are spliced, transcoded, and processed according to a fixed recording cycle (e.g., 6 seconds) and the resulting recording files are sent to the viewers' terminals so that they can watch the live stream in real time.

[0076] After the live stream ends, the host can edit the live video to generate a short video that includes highlights of the live stream, so that they can easily review the content.

[0077] In some scenarios, live streamers need to edit videos from previous live streams into short videos to promote the content of the current live stream and increase its popularity.

[0078] To achieve simultaneous streaming and editing, related technologies have implemented video editing and compositing based on recorded files generated during live streaming, such as m3u8 files. However, recorded files require a period of time to generate and process segmented files before video editing can begin, resulting in poor timeliness. Furthermore, video editing based on recorded files requires decoding and re-encoding the entire recorded file for precise editing, leading to lengthy editing times and low efficiency.

[0079] To improve the timeliness and efficiency of video editing, this application provides a video editing method that implements a video editing strategy based on live stream segmentation for the application scenario of editing while streaming. It only needs to edit the first and / or last segments that match the editing time, and the segments in the middle do not need to be decoded and re-encoded, so that video editing and compositing can be achieved, thus improving the efficiency of video editing.

[0080] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0081] Figure 2 This is a flowchart illustrating a video editing method provided in an embodiment of this application. The video editing method can be executed by an electronic device with corresponding data processing capabilities, such as a video editing device, a cloud server, or a server. The cloud server can be a live streaming server or a cloud server of a cloud service provider.

[0082] like Figure 2 As shown, this video editing method includes the following steps:

[0083] Step S201: Obtain the live stream segments whose editing time matches the cut.

[0084] Live stream segments are the segments of live stream data generated during a live broadcast, such as TS segments or mp4 segments. Live stream data is the streaming data uploaded by the live broadcast terminal during the broadcast. Editing time can be specified by the user (e.g., the broadcaster) or automatically determined, such as the highlight time segment during the live broadcast. Editing time can be a single point in time, such as a start time or an end time, indicating the start or end time of the edited video. Editing time can also be a time interval, consisting of the time interval corresponding to the start and end times. When the editing time only includes the start time, the default end time can be the current time or a preset time after the start time. When the editing time only includes the end time, the default start time can be the start time of the live recording or a preset time before the end time. The preset time can be 30 seconds, 2 minutes, 5 minutes, or other values, which can be configured by the user.

[0085] The live streaming client or center uploads the generated live stream data to the cloud server based on the streaming media transmission protocol. The cloud server then segments the live stream data, obtaining and storing multiple live stream segments. Recording is then performed on these segments at fixed intervals to obtain recording files, which are then stored and distributed. Users typically only access or view the recording files and cannot directly access the live stream segments.

[0086] In some embodiments, the cloud server storing the live stream data and the cloud server performing the editing process can be different cloud servers.

[0087] The step of obtaining the recorded file from the live stream segments can be performed by the recording device on the cloud server, and the subsequent live stream segment editing steps (including steps S201 to S203) can be performed by the live stream editing device (or video editing device).

[0088] In some embodiments, the live stream segments can be TS segments. The cloud server can concatenate multiple TS segments within a fixed period into an m3u8 file. The m3u8 file includes an index file that indicates the storage location of the file to be played (such as a TS segment). For ease of playback, the corresponding TS segments can be stored in the m3u8 file.

[0089] The live stream segment whose editing time matches the editing time can be a segment whose time overlaps with the time period corresponding to the editing time, and can be partially or completely overlapping.

[0090] Specifically, the live stream segments whose editing time is hit can be determined based on the time periods corresponding to the live stream segments.

[0091] For example, taking a 6-second duration for each live stream segment, one m3u8 file can correspond to 10, 20, or other numbers of live stream segments. Taking an edit time of 18:07:50 to 18:08:32 as an example, the first live stream segment corresponds to a time period of 18:00:00 to 18:00:06, the second live stream segment corresponds to a time period of 18:00:06 to 18:00:12, and so on. Therefore, the 79th live stream segment corresponds to a time period of 18:07:48 to 18:07:54, and the 86th live stream segment corresponds to a time period of 18:08:30 to 18:08:36. Thus, the live stream segments matching this edit time are the 79th to the 86th live stream segments.

[0092] Step S202: Based on the editing time, edit the first and / or last segments of the live stream.

[0093] The first segment is the earliest segment of the live stream that was hit, while the last segment is the latest segment of the live stream that was hit, i.e., the last segment of the live stream.

[0094] In some embodiments, by default, only the first segment of the live stream can be edited, or only the last segment can be edited. Editing segments can include cutting segments (including decoding and re-encoding) to adjust the time periods corresponding to the segments. Editing can also include splicing, transcoding, and other processing.

[0095] Specifically, the first segment can be cut based on the start time in the editing time, so that the start time of the first segment is the start time in the editing time. The last segment can be cut based on the end time in the editing time, so that the end time of the last segment is the end time in the editing time.

[0096] Other segments within the hit live stream segment are designated as intermediate segments and do not require trimming.

[0097] If the start time of the edit coincides with the start time of the first clip, then no editing is required for the first clip, such as trimming. If the end time of the edit coincides with the end time of the last clip, then no editing is required for the last clip.

[0098] Optionally, based on the editing time, the first and / or last segments of the live stream are edited, including:

[0099] The head segment and the tail segment are decoded; based on the start and end times in the clipping time, the decoded head segment and tail segment are re-encoded respectively, so that the start time of the re-encoded head segment is aligned with the start time of the clipping time, and the end time of the re-encoded tail segment is aligned with the end time of the clipping time.

[0100] Since the live stream segments are obtained by slicing the live stream data, they cannot be directly cut. Therefore, the head and tail segments need to be decoded first to obtain the decoded head and tail segments. Then, based on the editing time, the decoded head and tail segments are cut and re-encoded to obtain the edited head and tail segments. This ensures that the start time of the edited head segment and the end time of the edited tail segment are aligned with the start and end times of the editing time, respectively.

[0101] The above editing method only decodes and re-encodes the first and last segments, without needing to decode and re-encode other live stream segments that are hit, i.e., the middle segments. This greatly simplifies the editing operation, achieves precise editing, and improves editing efficiency.

[0102] Step S203: Based on the edited head and / or tail segments, generate the edited video corresponding to the live stream data.

[0103] Specifically, an edited video can be generated based on the edited first segment, the edited last segment, and the middle segments (if any).

[0104] Specifically, the edited first segment, the edited last segment, and the middle segments can be spliced ​​together to generate a recording file, and then an edited video in the required format can be generated based on the recording file.

[0105] Edited videos can be stored on cloud servers or on third-party storage providers specified by the user.

[0106] After generating the edited video, the edited video can be stored at a user-specified storage address so that the user can retrieve the synthesized edited video.

[0107] In some embodiments, when compositing and editing videos, additional encoding processes such as adding effects and transitions to the recorded files can be performed to enhance the fun and diversity of the edited videos.

[0108] The video editing method provided in this application, when editing based on live stream data generated during live streaming, proposes a concept of synthesizing edited videos based on live stream segments obtained during live streaming in order to improve the efficiency and timeliness of video editing. Specifically, it directly edits the first and / or last segments of the live stream segments whose editing time is matched, including decoding and re-encoding. Based on the cut first and / or last segments, as well as intermediate segments (if any), an edited video is generated, thereby realizing the editing of live content. It eliminates the need to wait for multiple segments to be periodically processed into a single recording file and then edited to obtain the edited video, thus improving the timeliness of video editing. At the same time, it only edits the first and last segments, eliminating the need to edit all segments, thereby improving the efficiency of editing and further improving the efficiency of synthesizing the edited video.

[0109] Optional, Figure 3 For this application Figure 2 The flowchart of step S203 in the illustrated embodiment is as follows: Figure 3 As shown, step S203 may include the following steps:

[0110] Step S301: Generate a first index list file based on the intermediate segments and the edited head and tail segments.

[0111] The middle segment is the remaining segment after removing the head and tail segments from the hit live stream segments.

[0112] The first index list file is an index file used to indicate the storage address of the corresponding fragment. The first index list file can be an m3u8 file.

[0113] Specifically, the middle segments, the edited first segment, and the edited last segment can be spliced ​​together to obtain the first index list file.

[0114] The first index list file may include a master index file and a playlist file. The master index file includes the storage addresses of each segment (including the aforementioned intermediate segments, the cut first segment, and the cut last segment), so that the corresponding segment can be read through the storage address when the edited video is played. The playlist file is used to store the playlist corresponding to the edited video, that is, each segment corresponding to the edited video.

[0115] For example, Figure 4 For this application Figure 3 The illustrated embodiment is a schematic diagram of the first index list file generation process, as shown below. Figure 4As shown, the live streaming client uploads the live stream data to the cloud, either directly or through a live streaming server. The cloud recording equipment records and segments the live stream data stored in the cloud to obtain corresponding live stream segments. When editing the live content is needed, the editing time can be sent to the cloud, such as to a cloud editing device. The start and end times of the editing time are as follows: Figure 4 As shown by the dashed line, the cloud editing device first determines the live stream segment whose editing time is matched. Then, it cuts the first segment (the first matched live stream segment) and the last segment (the last matched live stream segment) of the live stream segment whose editing time is matched. The cut first segment, the cut last segment, and the remaining matched middle segments are then spliced ​​together to generate a first index list file. If the other end, such as the live streaming client or the live streaming server, supports playing the first index list file, the first index list file can be sent to the other end as an edited video. If it is not supported, the first index list file is repackaged to generate an edited video in a format supported by the other end, and then sent to the other end, thereby completing the editing of the live content and realizing simultaneous playback and editing.

[0116] Step S302: Generate the edited video based on the first index list file.

[0117] Based on the video formats supported by the live streaming platform, the first index list file can be repackaged to obtain edited videos in the corresponding formats.

[0118] If the live streaming client supports playing m3u8 files, step S302 can be omitted, and the generated first index list file can be directly sent to the live streaming client.

[0119] Optionally, after editing the first and / or last segments of the live stream based on the editing time, the method further includes:

[0120] If the length of the edited first segment is less than the preset length, then the edited first segment is merged with its adjacent segments to obtain a new edited first segment; and / or, if the length of the edited last segment is less than the preset length, then the edited last segment is merged with its adjacent segments to obtain a new edited last segment.

[0121] The preset duration can be the duration corresponding to one or more video frames. The length of one video frame can be determined based on the video frame rate. For example, if the video frame rate is 25fps, the duration of one video frame is 0.04s (1 / 25).

[0122] In some embodiments, when the duration of the edited first or last segment is less than the duration of a single video frame, the edited first or last segment can be discarded.

[0123] However, discarding some content can cause the edited video to not meet the user's expectations, i.e., its accuracy is low. To overcome this problem and improve the accuracy of editing, when the length of the edited first or last segment is short, such as less than the length of a single frame of video, the edited first or last segment can be merged or spliced ​​with its adjacent segments to obtain a new edited first or last segment. Subsequent video editing and compositing steps are then performed based on this new edited first or last segment. For example, if the length of both the edited first and last segments is less than a preset length, the aforementioned edited video can be generated based on the new edited first and last segments.

[0124] For the edited first segment, its adjacent segment is the second live stream segment, and the edited last segment is the second to last live stream segment. The live stream segments are sorted according to their corresponding time.

[0125] In some embodiments, the editing time can be automatically determined based on the live stream content. Figure 5 This is a flowchart illustrating another video editing method provided in an embodiment of this application. Figure 2 Based on the illustrated embodiment, steps S202 and S203 are further refined, and a step for automatically determining the editing time is added before step S201. For example... Figure 5 As shown, the video editing method provided in this embodiment may include the following steps:

[0126] Step S501: In response to the live streaming editing instruction from the user who is currently live streaming, obtain the live stream segment corresponding to the user.

[0127] Users who are currently streaming can issue live editing commands through the live streaming platform, such as by clicking the editing-related buttons on the live streaming application.

[0128] The live editing instruction may or may not include editing time. This embodiment addresses the case where editing time is not included. If the live editing instruction includes editing time, step S502 is skipped, and step S503 is executed directly.

[0129] Live streaming editing commands can include user-specified editing ranges. The editing range can be a time range longer than the editing time, and the live stream segments that the editing range hits can be obtained to predict the editing time.

[0130] The live stream segments corresponding to a user can be stored in the cloud, i.e., on a cloud server, such as on a recording device stored on the cloud server. Live stream segments are files that cannot be played directly; they are obtained by slicing the live stream data corresponding to the user who is currently streaming.

[0131] On the cloud server side, such as the cloud editing device, after receiving a live editing instruction from a user who is currently live streaming, it obtains the live stream segment corresponding to that user based on the user identifier carried in the live editing instruction. For example, it obtains the live stream segment generated by that user from the beginning of this live stream to the present, or it obtains the live stream segment corresponding to that user for a fixed time period, such as the live stream segment of the first 30 minutes.

[0132] Step S502: Determine the editing time based on the live stream segment corresponding to the user.

[0133] To improve the convenience of live streaming and editing, users can configure the editing time when issuing live editing commands. The commands are issued in a triggered manner, and the cloud server automatically determines the editing time, such as the highlight period or exciting period of the live stream.

[0134] The cloud server can automatically determine the editing time based on the live stream segments corresponding to the user who is currently live streaming.

[0135] Specifically, content recognition algorithms can be used to identify live stream content based on live stream segmentation, and the editing time can be determined based on the recognition results.

[0136] In some embodiments, the editing time can be determined based on the size of the live stream segments. For example, the time segment containing the larger live stream segment can be preferentially selected as the editing time.

[0137] Furthermore, by decoding live stream segments, information such as text, audio, and video can be extracted from these segments, and editing times can be determined based on the extracted information. For example, editing times can be determined based on keywords in the text and audio information, or key features extracted from the video.

[0138] Taking live-streaming games as an example, the highlight time periods of the live stream can be determined based on the in-game voice broadcasts, the streamer's explanations, and the viewers' comments. The live stream content can then be edited using these highlight time periods to obtain the best segments of the live stream.

[0139] For example, the editing time can be selected from the period when the density of audience comments is high or the density of the host's explanation content is high.

[0140] For example, the editing time can be determined by the image features of the live stream segments corresponding to the extracted live stream segments.

[0141] Optionally, the editing time is determined based on the live stream segment corresponding to the user, including:

[0142] The live stream segments corresponding to the user are input into the time prediction model, and the editing time is determined based on the time prediction model.

[0143] The prediction model can be any AI (Artificial Intelligence) model that predicts time periods, such as a neural network model. This prediction model takes the live stream segments as input and the editing time as output.

[0144] The prediction model can first decode the live stream segments, extract the semantics, speech rate, and tone of the audio information and the visual features of the video information in the decoded live stream segments, and then make editing time predictions based on the extracted features, such as predicting the time period of interest and the highlight time period.

[0145] The predictive model can output the start and end times of the clip.

[0146] The model automatically predicts the editing time, taking into full account the audio and video features of the live stream content corresponding to the live stream segments, which improves the accuracy of the editing time prediction and thus improves the accuracy of the generated edited video.

[0147] Step S503: Obtain the live stream segments whose editing time matches the cut.

[0148] The cloud server includes recording equipment and cloud editing equipment. Figure 6 For this application Figure 5 The illustrated embodiment is a schematic diagram of the interaction process between the recording device and the cloud editing device, as shown below. Figure 6 As shown, the cloud editing device can send the user-specified editing range to the recording device. The recording device returns the live stream segments that match the editing range. Based on the content of the returned live stream segments, the cloud editing device uses AI technology to identify highlight segments, i.e., the aforementioned editing time. Thus, it determines the live stream segment that the highlight segment matches from the live stream segments that match the editing range, such as determining the storage path, segment identifier, and other information of the live stream segment that the highlight segment matches. Based on the determined information, it retrieves the live stream segment that the highlight segment matches from the recording device.

[0149] Step S504: Decode the head segment and the tail segment.

[0150] Specifically, the header and tail segments can be decoded based on their encoding methods to obtain the decoded header and tail segments.

[0151] Step S505: Obtain the segmentation parameters of the live stream segment.

[0152] Among them, the segmentation parameters are used to characterize the attributes of the segments, which may include parameters such as resolution, bit rate, frame rate, and audio / video sampling rate.

[0153] Step S506: Based on the start and end times in the editing time and the segmentation parameters, the decoded head segment and tail segment are re-encoded respectively, so that the segmentation parameters of the re-encoded head segment and tail segment are consistent with the segmentation parameters of the live stream segment.

[0154] To prevent the edited first and last segments from being spliced ​​or repackaged with other live stream segments and resulting in videos that cannot be previewed or played properly, the segmentation parameters need to be considered when cutting the first and last segments. Based on the segmentation parameters and the start time in the editing time, the decoded first segment is cut and re-encoded to obtain the cut first segment. Similarly, based on the segmentation parameters and the end time in the editing time, the decoded last segment is cut and re-encoded to obtain the cut last segment.

[0155] If the duration of the cut head or tail segment is less than the preset duration, the cut head or tail segment can be spliced ​​or merged with its adjacent live stream segments to obtain a new cut head segment or a new cut tail segment.

[0156] Step S507: Generate a second index list file based on the intermediate shards.

[0157] To speed up video editing, intermediate segments can be spliced ​​together during the cutting and other processing of the beginning and end segments to obtain a second index list file, which can be an m3u8 file.

[0158] Step S507 can be executed at any time after step S503 and before step S508, such as in parallel with step S505 or in sequence. Figure 5 Take parallel execution as an example.

[0159] Step S508: The second index list file and the edited first and last segments are repackaged to obtain the edited video.

[0160] By transcoding, edited videos in formats supported by the live streaming platform can be generated, such as mp4 and mpg formats.

[0161] Figure 7 For this application Figure 5 The illustrated embodiment is a schematic diagram of the video editing generation process. Figure 7 In the example of editing a video into MP4 format, such as... Figure 7As shown, the live stream segments ts that are hit by the editing time (the start and end times of the editing time are represented by dashed lines) are segments ts1 to tsn, where n is a positive integer greater than 2. After cutting (including decoding and re-encoding) the head segment ts1 and the tail segment tsn, and splicing the middle segments, i.e., segments ts2 to tsn-1, a second index list file in m3u8 format is obtained. Then, the head and tail segments after cutting, as well as the second index list file, are transpackaged to obtain the edited video in mp4 format.

[0162] Figure 8 A schematic diagram of the video editing process provided in another embodiment of this application, in conjunction with... Figure 7 and Figure 8 In this embodiment, for cases where the duration of both the cut head segment and the tail segment is less than the duration of a single video frame, the head segment ts1 is merged with its adjacent segment ts2 to form a new cut head segment, and the tail segment tsn is merged with its adjacent segment tsn-1 to form a new cut tail segment. Then, the new cut head segment, the new cut tail segment, and the second index list file are encapsulated to obtain the edited video.

[0163] In this embodiment, the editing time is automatically predicted based on the content of the live stream segments, thereby obtaining the live stream segments with matching editing times. The first and last segments are then decoded and re-encoded to achieve the cutting operation of the first and last segments based on the editing time, resulting in the cut first and last segments. At the same time, during re-encoding, the segmentation parameters of the live stream segments are considered, so that the cut first and last segments have consistent segmentation parameters with other live stream segments, avoiding the inability to play the synthesized edited video due to inconsistent segmentation parameters. For the intermediate segments, only the splicing operation needs to be performed during the processing of the first and last segments, without the need to decode and re-encode the intermediate segments, which greatly improves the efficiency of video cutting.

[0164] This application provides another video editing method applied to a cloud server, the method comprising:

[0165] The system acquires live stream data generated during live streaming from the live streaming client, sent by the live streaming server; slices the live stream data to obtain multiple live stream segments; responds to editing instructions issued by the live streaming server, acquires the target live stream segment whose editing time matches the editing time in the editing instructions; edits the first and / or last segments of the target live stream segment based on the editing time; generates an edited video based on the edited first and / or last segments; and sends the edited video to the live streaming server so that the edited video can be sent to the live streaming client through the live streaming server.

[0166] This application provides another video editing method, applied to a live streaming server, such as a live streaming center. The method includes:

[0167] In response to the editing instructions issued by the live streaming client, the system retrieves the target live stream segment whose editing time matches the instructions from the cloud server; based on the editing time, it edits the first and / or last segments of the target live stream segment; based on the edited first and / or last segments, it generates an edited video; and sends the edited video to the live streaming client.

[0168] The video editing method provided in this application can also be applied to the editing of videos in streaming media formats. Figure 9 This is a flowchart illustrating another video editing method provided in this application embodiment. This embodiment addresses the case of ordinary video editing, such as... Figure 9 As shown, for videos in streaming media formats, the video editing method provided in this embodiment may include the following steps:

[0169] Step S901: Obtain the video stream segments whose editing time matches the target time.

[0170] The video stream segments are segments corresponding to streaming media format videos.

[0171] Users can send the link of the streaming video format that needs to be edited to the cloud server. The cloud server will then use the link to obtain the video stream segment corresponding to the streaming video format, and select the video stream segment that matches the editing time based on the time period of the video stream segment.

[0172] Step S902: Based on the editing time, edit the first and / or last segments of the video stream.

[0173] Step S903: Based on the edited head and / or tail segments, generate the edited video corresponding to the streaming media format video.

[0174] The steps of the video editing method provided in this embodiment can be performed with reference to the steps provided in the foregoing embodiments. The only difference is that the video stream segments are changed to live stream segments and the live stream data is changed to streaming media format video. These will not be repeated here.

[0175] When the object is a streaming video format, the format of the first index list file and the second index list file mentioned above can be m3u8, dash, hls or fmp4, etc.

[0176] Figure 10 For this application Figure 9 The illustrated embodiment is a schematic diagram of the video editing and generation process, as shown below. Figure 10As shown, the video to be edited can be a streaming media format video stored in the media asset library. In the cloud, the streaming media format videos stored in the media asset library can be sliced ​​to obtain a segment list (including multiple video stream segments) corresponding to each video. When it is necessary to edit the streaming media format video in the media asset library, the cloud editing device determines the video stream segment whose editing time is matched from the segment list corresponding to the streaming media format video based on the start and end times in the editing time. Then, it performs media processing such as cutting and transcoding on the first and last segments of the video stream segment, and splices the middle segments to obtain a second index list file (which can be in formats such as m3u8, dash, hls, or fmp4). The cut first and last segments and the second index list file are spliced ​​and transcoded to generate the final cut video.

[0177] Figure 11 This is a schematic diagram of the structure of a video editing device provided in an embodiment of this application, as shown below. Figure 11 As shown, the video editing device includes: a segment acquisition module 1110, a partial editing module 1120, and an edited video generation module 1130.

[0178] The segment acquisition module 1110 is used to acquire the live stream segments whose editing time matches the cut time, wherein the live stream segments are the segments corresponding to the live stream data generated during the live broadcast; the partial editing module 1120 is used to edit the first segment and / or the last segment in the live stream segments based on the editing time; the edited video generation module 1130 is used to generate an edited video corresponding to the live stream data based on the edited first segment and / or last segment.

[0179] Optional, the partial clipping module 1120 includes:

[0180] A local decoding unit is used to decode the head segment and the tail segment; a local encoding unit is used to re-encode the decoded head segment and tail segment based on the start time and end time in the clipping time, so that the start time of the re-encoded head segment is aligned with the start time of the clipping time, and the end time of the re-encoded tail segment is aligned with the end time of the clipping time.

[0181] Optional, local coding unit, specifically used for:

[0182] Obtain the segmentation parameters of the live stream segment; based on the start and end times in the editing time and the segmentation parameters, re-encode the decoded head segment and tail segment respectively, so that the segmentation parameters of the re-encoded head segment and tail segment are consistent with the segmentation parameters of the live stream segment.

[0183] Optional, the video editing and generation module 1130 is specifically used for:

[0184] Based on the intermediate segments and the edited first and last segments, a first index list file is generated, wherein the intermediate segments are the remaining segments after removing the first and last segments from the hit live stream segments; based on the first index list file, the edited video is generated.

[0185] Optional, the video editing and generation module 1130 is specifically used for:

[0186] Based on the intermediate segments, a second index list file is generated, wherein the intermediate segments are the remaining segments after removing the head segment and the tail segment from the hit live stream segments; the second index list file and the edited head and tail segments are transcoded to obtain the edited video.

[0187] Optionally, the device further includes:

[0188] The head and tail segment update module is used to, after editing the head segment and / or tail segment of the live stream segment based on the editing time, if the duration of the edited head segment is less than a preset duration, merge the edited head segment with the adjacent segments of the head segment to obtain a new edited head segment; and / or, if the duration of the edited tail segment is less than the preset duration, merge the edited tail segment with the adjacent segments of the tail segment to obtain a new edited tail segment.

[0189] Optionally, the device further includes:

[0190] The instruction response module is used to respond to the live streaming editing instruction of a user who is currently live streaming and obtain the live stream segment corresponding to the user; the editing time determination module is used to determine the editing time based on the live stream segment corresponding to the user.

[0191] Optional, the editing time determination module is specifically used for:

[0192] The live stream segments corresponding to the user are input into the time prediction model, and the editing time is determined based on the time prediction model.

[0193] The video editing apparatus provided in this application embodiment can be used to execute the technical solutions of the video editing methods provided in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be described again in this embodiment.

[0194] Figure 12 This is a schematic diagram of the structure of a video editing device provided in an embodiment of this application, as shown below. Figure 12 As shown, the video editing device 1200 provided in this embodiment includes:

[0195] At least one processor 1210; and a memory 1220 communicatively connected to the at least one processor; wherein the memory 1220 stores computer execution instructions; the at least one processor 1210 executes the computer execution instructions stored in the memory to cause the video editing device 1200 to perform the method provided in any of the foregoing embodiments.

[0196] Optionally, the memory 1220 can be either standalone or integrated with the processor 1210.

[0197] The implementation principle and technical effects of the video editing device provided in this embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0198] This application embodiment also provides a cloud service platform, including cloud storage nodes and video editing equipment 800, wherein the cloud storage nodes are used to store data such as live stream segments and video stream segments.

[0199] This application also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the methods provided in any of the foregoing embodiments can be implemented.

[0200] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in any of the foregoing embodiments.

[0201] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0202] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application.

[0203] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor. The memory may include high-speed memory, and may also include non-volatile memory, such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disc, etc.

[0204] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0205] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside within an application-specific integrated circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or video editing device.

[0206] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0207] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0208] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods provided in the various embodiments of this application.

[0209] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0210] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A video editing method, characterized in that, include: Obtain multiple live stream segments that match the editing time, wherein the live stream segments that match the editing time are live stream segments that partially or completely overlap with the time period corresponding to the editing time, and the live stream segments are segments corresponding to the live stream data generated during the live broadcast. Based on the editing time, only the first and last segments of the multiple live stream segments are edited; Based on the edited first and last segments, generate the edited video corresponding to the live stream data; Based on the aforementioned editing time, only the first and last segments of the multiple live stream segments are edited, including: Decode the head segment and the tail segment in the plurality of live stream segments; Based on the start and end times in the clipping time, the decoded head and tail segments are re-encoded so that the start time of the re-encoded head segment is aligned with the start time of the clipping time, and the end time of the re-encoded tail segment is aligned with the end time of the clipping time.

2. The method according to claim 1, characterized in that, Based on the start and end times in the editing time, the decoded first and last segments are re-encoded, including: Obtain the segmentation parameters of the live stream segment; Based on the start and end times in the editing time and the segmentation parameters, the decoded head segment and tail segment are re-encoded to make the segmentation parameters of the re-encoded head segment and tail segment consistent with the segmentation parameters of the live stream segment.

3. The method according to claim 1, characterized in that, Based on the edited first and last segments, an edited video corresponding to the live stream data is generated, including: Based on the intermediate segments and the edited head and tail segments, a first index list file is generated, wherein the intermediate segments are the remaining segments after removing the head and tail segments from the hit live stream segments. The edited video is generated based on the first index list file.

4. The method according to claim 1, characterized in that, Based on the edited first and last segments, an edited video corresponding to the live stream data is generated, including: Based on the intermediate segments, a second index list file is generated, wherein the intermediate segments are the remaining segments after removing the head segment and the tail segment from the hit live stream segments; The second index list file, along with the edited first and last segments, are transcoded to obtain the edited video.

5. The method according to any one of claims 1-4, characterized in that, After editing only the first and last segments of the live stream based on the editing time, the method further includes: If the length of the edited header segment is less than the preset length, then the edited header segment is merged with its adjacent segments to obtain a new edited header segment; and / or, If the length of the edited tail segment is less than the preset length, then the edited tail segment is merged with the adjacent segments of the tail segment to obtain a new edited tail segment.

6. The method according to any one of claims 1-4, characterized in that, The method further includes: In response to a live streaming editing command from a user who is currently streaming, obtain the live stream segment corresponding to that user; The editing time is determined based on the live stream segment corresponding to the user.

7. The method according to claim 6, characterized in that, The editing time is determined based on the live stream segment corresponding to the user, including: The live stream segments corresponding to the user are input into the time prediction model, and the editing time is determined based on the time prediction model.

8. A video editing method, characterized in that, The method is applied to a cloud server, and the method includes: Retrieve live stream data generated during live streaming from the live streaming client, sent by the live streaming server; The live stream data is sliced ​​to obtain multiple live stream segments; In response to the editing instructions issued by the live streaming server, multiple target live stream segments whose editing time matches the editing time in the editing instructions are obtained; the live stream segments whose editing time matches the editing time are live stream segments whose time periods partially or completely overlap with the time periods corresponding to the editing time. Based on the editing time, only the first and last segments of the target live stream are edited; Generate an edited video based on the edited first and last parts; The edited video is sent to the live streaming server so that the edited video can be sent to the live streaming client through the live streaming server. Based on the aforementioned editing time, only the first and last segments of the multiple live stream segments are edited, including: Decode the head segment and the tail segment in the plurality of live stream segments; Based on the start and end times in the clipping time, the decoded head and tail segments are re-encoded so that the start time of the re-encoded head segment is aligned with the start time of the clipping time, and the end time of the re-encoded tail segment is aligned with the end time of the clipping time.

9. A video editing method, characterized in that, The method is applied to a live streaming server, and the method includes: In response to the editing instructions issued by the live streaming client, multiple target live stream segments whose editing time matches the editing time in the editing instructions are obtained from the cloud server; the live stream segments whose editing time matches the editing time are live stream segments whose time periods partially or completely overlap with the time periods corresponding to the editing time. Based on the editing time, only the first and last segments of the target live stream are edited; Generate an edited video based on the edited first and last parts; Send the edited video to the live streaming client; Based on the aforementioned editing time, only the first and last segments of the multiple live stream segments are edited, including: Decode the head segment and the tail segment in the plurality of live stream segments; Based on the start and end times in the clipping time, the decoded head and tail segments are re-encoded so that the start time of the re-encoded head segment is aligned with the start time of the clipping time, and the end time of the re-encoded tail segment is aligned with the end time of the clipping time.

10. A video editing device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-9.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images