Method, apparatus, device, and product for generating a video

Abstract

The present disclosure relates to methods, apparatuses, devices and products for generating a video. The method comprises determining beat point information of an audio associated with a first video. The method further comprises adjusting a time length of at least one video segment of a plurality of video segments included in the first video based on the beat point information. The method further comprises generating a second video based on the adjusted plurality of video segments and the audio, the second video matching the beat point information of the audio.

Description

Technical Field

[0001] This disclosure relates to the field of computers, and more specifically to methods, apparatus, devices and products for generating video. Background Technology

[0002] Music-synced videos are an innovative video format that creates a strong sense of rhythm and visual impact by synchronizing video transitions with the beat of the music, thereby enhancing the video's watchability and pacing. This video format requires that the transitions in the video content closely correspond to the beat of the music to achieve the best audiovisual effect.

[0003] Currently, users can input audio and video files and manually adjust the transition points of the video footage to align with the music's beat, according to their own creative ideas and needs. Meanwhile, to facilitate users in editing music-synced videos, many editing software programs have introduced automatic beat-syncing functions. Editing software with automatic beat-syncing can detect the music's beat and automatically adjust the video's transition points to synchronize with it. Summary of the Invention

[0004] In a first aspect of the embodiments of this disclosure, a method for generating a video is provided. The method includes determining beat point information of audio associated with a first video. The method further includes adjusting the duration of at least one video segment from a plurality of video segments included in the first video based on the beat point information. The method further includes generating a second video based on the adjusted plurality of video segments and the audio, the second video being matched with the beat point information of the audio.

[0005] A second aspect of embodiments of this disclosure provides an apparatus for generating a video. The apparatus includes a beat point information determination module configured to determine beat point information of audio associated with a first video. The apparatus also includes a video segment adjustment module configured to adjust the duration of at least one of a plurality of video segments included in the first video based on the beat point information. The apparatus further includes a video generation module configured to generate a second video based on the adjusted plurality of video segments and the audio, the second video being matched with the beat point information of the audio.

[0006] In three aspects of embodiments of this disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement a method for generating a video. The method includes determining beat point information of audio associated with a first video. The method further includes adjusting the duration of at least one video segment of a plurality of video segments included in the first video based on the beat point information. The method further includes generating a second video based on the adjusted plurality of video segments and the audio, the second video being matched with the beat point information of the audio.

[0007] In a fourth aspect of embodiments of this disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to implement a method for generating a video. The method includes determining beat point information of audio associated with a first video. The method further includes adjusting the duration of at least one of a plurality of video segments included in the first video based on the beat point information. The method further includes generating a second video based on the adjusted plurality of video segments and the audio, the second video being matched with the beat point information of the audio.

[0008] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which several embodiments of the present disclosure may be implemented is shown;

[0011] Figure 2 A flowchart of a method for generating video according to some embodiments of the present disclosure is shown;

[0012] Figure 3A A schematic diagram illustrating beat point information of audio according to some embodiments of the present disclosure is shown;

[0013] Figure 3B A schematic diagram of clipped audio according to some embodiments of the present disclosure is shown;

[0014] Figure 3CA schematic diagram illustrating inductive audio parameters according to some embodiments of the present disclosure is shown;

[0015] Figure 3D A schematic diagram of beat point data in audio according to some embodiments of the present disclosure is shown;

[0016] Figure 4A A schematic diagram illustrating the generation of video clips from edited video according to some embodiments of the present disclosure is shown;

[0017] Figure 4B A schematic diagram illustrating parameters of a video segment according to some embodiments of the present disclosure is shown;

[0018] Figure 5 A flowchart of a checkpoint algorithm according to some embodiments of the present disclosure is shown;

[0019] Figure 6A A flowchart illustrating a method for determining the expected effective duration of each video segment according to some embodiments of the present disclosure is shown;

[0020] Figure 6B The diagram illustrates the effect of determining the expected effective duration of each video segment according to some embodiments of the present disclosure;

[0021] Figure 7A A flowchart illustrating a method for adjusting the duration of a video segment according to some embodiments of the present disclosure is shown;

[0022] Figure 7B A schematic diagram illustrating the adjustment of end overlap transitions and / or start overlap transitions according to some embodiments of the present disclosure is shown;

[0023] Figure 8 The diagram shows renderings of generated videos according to some embodiments of the present disclosure;

[0024] Figure 9 A block diagram of an apparatus for generating video according to some embodiments of the present disclosure is shown; and

[0025] Figure 10 A block diagram of a device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0026] It is understood that all user-related data involved in this technical solution should be obtained and used only after authorization from the user. This means that if it is necessary to use a user's personal information in this technical solution, the user's explicit consent and authorization are required before obtaining this data; otherwise, no related data collection and use will be carried out. It should also be understood that when implementing this technical solution, relevant laws and regulations should be strictly followed in the process of data collection, use, and storage, and necessary technical measures should be taken to protect user data security and ensure the secure use of data.

[0027] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0028] For example, upon receiving a user's proactive request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0029] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0030] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0031] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0032] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0033] As mentioned above, many video editing software programs have introduced automatic beat-matching functions to facilitate users' editing of music-synced videos. While these functions simplify the creative process to some extent, they suffer from significant limitations due to their template-based nature. Advanced users find it difficult to meet personalized needs when using this feature, not only due to limitations in the number and style of templates, but also because it falls short in terms of in-depth editing. Users struggle to fine-tune the music rhythm and cannot customize beat points according to creative requirements, further restricting the diversity and depth of their work.

[0034] Furthermore, the automatic beat-matching function presents significant obstacles when used in conjunction with other editing features. When users attempt to use speed changes, overlapping transitions, or other functions after automatic beat-matching, the established rhythm is often disrupted, and the existing timeline is thrown off. Similarly, if these functions are used first, followed by automatic beat-matching, the software will clear the user's completed editing actions, such as speed changes and transitions, to ensure the beat-matching effect, forcing the user to re-edit. This not only reduces creative efficiency but also severely impacts the user's creative experience and flexibility.

[0035] To address this, this disclosure provides a method for generating videos. First, the beat point information of the audio used for video editing is determined. Based on the beat point information, the duration of at least one video segment among multiple video clips is adjusted. Finally, a music-synced video is generated based on the audio and the adjusted video clips. This method automatically aligns the video with the beat point information, effectively improving video generation efficiency. Furthermore, by adjusting only the duration of video clips, personalized music-synced videos can be generated without sacrificing other editing functions.

[0036] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of this disclosure may be implemented is shown. For example... Figure 1As shown, example environment 100 can include multiple video clips. These clips can be multiple different unedited source videos, or they can be a continuous sequence of images extracted by user 101 from source videos, with specific start and end points. User 101 can trim desired video clips from the original videos by setting editing parameters, such as start time and end time. Video clips can include content from different sources, such as movies, TV shows, documentaries, and short videos, and can be selected and combined according to user 101's creative needs. For example, in example environment 100, user 101 can cut a scene from a movie as video clip 103, and another scene from a documentary as video clip 105. Multiple video clips will be used to construct the final video, conveying a specific story or emotion through splicing, sorting, and adding special effects.

[0037] like Figure 1 As shown, multiple video clips can include video clip 103 and video clip 105. For two adjacent video clips, user 101 can set an overlapping transition 107. Overlapping transition 107 refers to a special effect used when switching between two adjacent video clips, causing the end of the previous video clip and the beginning of the next video clip to overlap for a period of time, thus achieving a smooth transition. For example, at the first frame of overlapping transition 107, the image of the previous video clip begins to change from clear to transparent, and the image of the current video clip begins to change from transparent to clear. At the last frame of overlapping transition 107, the image of the previous video clip can be completely transparent, i.e., no longer displayed, while the image of the current video clip can be fully displayed, achieving a clear state. Under the effect of overlapping transition 107, the changes of the previous video clip and the current video clip occur simultaneously. The overlapping transition has a certain duration, which can be adjusted according to user 101's preferences and creative needs. Overlapping transitions can create a variety of unique visual effects, such as fading, dissolving, and sliding, thereby enhancing the smoothness and watchability of videos.

[0038] like Figure 1As shown, the example environment 100 may also include audio 109, which may be audio selected by user 101 and associated with a video clip. For example, audio 109 may be audio that matches the rhythm of the video clip. Audio 109 and the video clip are used together to construct the final music-synced video. In some embodiments, the user may perform editing actions such as speeding up and setting transitions on the video clip based on audio 109, and generate a video based on the edited video clip and audio 109. Audio 109 may be the entire source audio or it may be extracted by user 101 from the source audio. Audio 109 may include beat point information. Beat points are the points where the first half of a beat connects to the second half of a beat in a musical rhythm, helping to determine the rhythm and cadence of the music. In some embodiments, user 101 may download audio templates from the cloud. These templates typically contain preset background music, sound effects, and template beat points 111, i.e., specific rhythm points. In some embodiments, user 101 may also record or import the entire source audio and edit it according to their needs. Additionally, users can set custom beat points 113, which means adding markers at specific locations in the audio to guide the switching of video segments or the addition of special effects, in order to achieve more accurate and coordinated audio and video synchronization.

[0039] In some embodiments, after determining the beat point information of multiple video segments and audio 109, the video segment duration adjustment module 115 can adjust the duration of the multiple video segments according to the beat point information to generate video 117. In this embodiment, the video segment duration adjustment module 115 is typically located in the client's video processing system and is configured with a beat-matching algorithm, capable of recognizing beat point information in the audio and adjusting the duration of the video segments according to the beat point information. During the generation of video 117, the beat points of audio 109 correspond to specific video editing points, including the center position of overlapping transitions, the precise start point or end point of the video segment. In this way, the rhythm of the audio can be closely integrated with the visual elements of the video, thereby presenting the desired beat point effect.

[0040] In some embodiments, the beat point information includes the start and end times of each beat interval period. Multiple beat interval periods of audio 109 can be determined based on the beat point information. A beat interval period can be the time interval between adjacent beat points. In some embodiments, the first frame and the last frame of audio 109 can also be two beat points. The time interval between the first frame of audio 109 and the first beat point can be the first beat interval period of audio 109, and the time interval between the last beat point of audio 109 and the last frame can be the last beat interval period of audio 109. In embodiments of this disclosure, the duration of one or more video segments corresponding to each beat interval period can be adjusted to align the video segments with the beat interval period. For example, for the beat interval period between beat points 117 and 115, the duration of video segment 103 can be adjusted so that the duration of video segment 103 matches the duration of the beat interval period between beat points 117 and 115.

[0041] This method automatically and accurately aligns the beat points in the video content with the audio, greatly simplifying the tedious manual alignment process in traditional video editing and significantly improving the efficiency and quality of video generation. Furthermore, this adjustment only affects the duration of the video clip and does not impact other video editing functions such as transitions, filter applications, or subtitle additions. This means users can enjoy the convenience of automatic beat matching while still freely using various editing techniques to enrich and refine their video works.

[0042] Figure 2 A flowchart of a method 200 for generating video according to some embodiments of the present disclosure is shown. Method 200 can be performed by a client, or by a combination of a client and a server. Method 200 includes blocks 202, 204, and 206.

[0043] like Figure 2 As shown, in box 202, the beat point information of the audio associated with the first video is determined. (Reference) Figure 1The first video can include multiple video segments associated with audio 109. Audio 109 and multiple video segments work together to construct the final music-synced video. The beat information of audio 109 can be determined in several ways. On one hand, the client can automatically extract beat information from audio downloaded from the cloud. These audio files are usually processed and have clearly defined rhythm points or beat points built in. The client can use built-in algorithms or tools to accurately identify and extract these beat points. On the other hand, the client also supports users 101 to set custom beat points 113. Users 101 can add custom beat points 113 at any position in audio 109 according to their creative needs, the characteristics of audio 109 content, or personal preferences. These custom beat points 113 can represent video segment transition points, special effect trigger points, or other important audio-video synchronization points. By receiving and processing user-set custom beat points 113, the client can further meet users' personalized needs in video editing.

[0044] In box 204, based on the beat point information, the duration of at least one video segment among the multiple video segments included in the first video is adjusted. (Reference) Figure 1 After determining the beat point information of multiple video segments and audio 109, the video segment duration adjustment module 115 can adjust the duration of the multiple video segments according to the beat point information to generate video 117. In this embodiment of the present disclosure, for each beat interval, the duration of one or more video segments corresponding to the targeted beat interval can be adjusted so that the video segments are aligned with the beat interval. For example, for the beat interval between beat point 117 and beat point 115, the duration of video segment 103 is adjusted so that the duration of video segment 103 is consistent with the duration of the beat interval between beat point 117 and beat point 115, the start point of video segment 103 corresponds to beat point 117, and the end point of video segment 103 corresponds to beat point 115.

[0045] In box 206, a second video is generated based on the adjusted multiple video clips and audio, and the second video is matched with the beat point information of the audio. (Reference) Figure 1 The video segment duration adjustment module 115 has adjusted the duration of the corresponding video segments within each beat interval based on the beat point information, ensuring perfect alignment between the video segments and the beat intervals. Afterwards, the client can use these adjusted video segments and audio 109 to synthesize multiple video segments and audio 109 to generate the final video 117 (also known as the second video). Video 117 not only contains the video content and audio effects required by the user but also ensures synchronization between the audio 109 and the video.

[0046] Figure 3AA schematic diagram of audio beat point information 300A according to some embodiments of the present disclosure is shown. Figure 3A As shown, audio 301 can be edited from audio materials input by the user or downloaded from the cloud, and it forms the basis for subsequent editing and creation of music-synced videos. Audio 301 can originate from segments edited by the user from the source audio or retrieved from a cloud audio library. Based on audio 301, the client can perform beat point recognition, using accent analysis methods to automatically identify the accented parts of audio 301. Furthermore, the client can employ machine learning or deep learning models to further improve the accuracy of beat point recognition.

[0047] In some embodiments, after identifying the beat points in audio 301, audio 303 containing beat point information is generated. Audio 303 not only retains the original content of audio 301 but also adds beat point information, providing users with more editing and creative possibilities. Users can further add custom beat points to audio 303 to generate audio 305. These custom beat points can be added manually by the user according to their needs or automatically generated through other methods, such as using client-provided tools or third-party plugins. Audio 305 contains the audio content and beat point information desired by the user, allowing the user to use audio 305 to generate personalized music beat-synced videos.

[0048] Figure 3B A schematic diagram of a clipped audio 300B according to some embodiments of the present disclosure is shown. Figure 3B As shown, source audio 307 can originate from direct user input, such as sound recorded through a device microphone, or audio material downloaded from the cloud. The parameters of source audio 307 include source audio duration 309, which represents the total playback time of source audio 307 and is an indicator of the amount of audio content. Based on source audio 307, the user can edit out specific segments, namely audio 313. The parameters of audio 313 can include start editing time 311 and edited audio duration 315. Start editing time 311 refers to the starting point of audio 313 on the timeline of source audio 307, determining when audio 313 begins to be extracted. The edited audio duration 315 refers to the total duration of audio 313 from the start point to the end point. When audio 313 is merged with multiple video segments 317 to generate an audio beat point video, the added time 319 is generated accordingly. Adding time 319 refers to the moment when audio 313 is first introduced and presented synchronously with the video during the music beat-matching video playback.

[0049] Figure 3CA schematic diagram of the generalized audio parameters 300C according to some embodiments of the present disclosure is shown. For example... Figure 3B As shown, based on the source audio 307, the user can cut out a specific audio segment, namely audio 313. When audio 313 is generated, its audio parameters 321 are also generated. For example... Figure 3C As shown, audio parameter 321 may include audio file path, audio file duration, audio trimming data, audio trimming start time, audio trimming duration, and the start time of adding audio to the editing track. The audio file path is the file path of the source audio 307, indicating the storage location of the audio material; the audio file duration is the source audio duration 309, reflecting the total playback time of the source audio; the audio trimming data includes the specific content extracted from the source audio by audio 313, including which parts of the audio are selected for subsequent editing and creation; the audio trimming start time is the start editing time 311 of audio 313, marking the starting position of audio 313 from the source audio 307; the audio trimming duration is the audio duration 315, representing the duration of audio 313; the start time of adding audio to the editing track is the addition time 319, determining when audio 313 begins playing in the final generated music-matched video. Audio parameter 321 not only covers the basic information of the audio file, but also records the specific data of audio trimming and its playback time in the video, providing a foundation for generating music-synced videos.

[0050] Figure 3D A schematic diagram of beat point data 300D in audio according to some embodiments of the present disclosure is shown. Figure 3D As shown, the beat point data of audio 323 is typically represented as a monotonically strictly increasing array. Each element represents the time distance of that beat point relative to the first frame of the original audio file, and the unit can be milliseconds (ms). For example, the beat point data of audio 323 can be represented as: [1000, 1800, 4100, 4750, 5350, 6850, 8850...]. In this embodiment of the disclosure, in order to ensure the display effect of the video clip, when generating the beat data, the minimum duration between two beat points can be set to 100ms, and the duration of the first beat point from the first frame of the clip is at least 100ms, and the duration of the last beat point from the last frame of the clip is at least 100ms.

[0051] Figure 4A A schematic diagram illustrating the generation of video clip 400A from edited video according to some embodiments of the present disclosure is shown. Figure 4AAs shown, the source video 401 can originate from direct user input or be video footage downloaded from the cloud. The parameters of the source video 401 include the source video duration (clip.sourceDuration) 403, which represents the total playback time of the source video 401. Based on the source video 401, the user can edit specific video clips, namely video clips (clip) 409. Video clip 409 is a part of the video 407, which includes multiple video clips. The parameters of video clip 409 can include the start clip time (clip.clipRange.startTime) 405, the duration of the video clip (clip.clipRange.duration) 411, the duration of the start overlay transition 413 (clip.preTransitionDuration) 417, the duration of the end overlay transition 415 (clip.endTransitionDuration) 419, and the effective duration (EfficientDuration) 421, also known as the second effective duration. The start editing time 405 refers to the starting point of video segment 409 on the timeline of the source video 401. The start editing time 405 determines when video segment 409 begins to be extracted. The edited video segment duration 411 refers to the total duration of video segment 409 from the start point to the end point. Video segment 409 may include a start overlapping transition 413 and / or an end overlapping transition 415, or it may not have overlapping transitions set. The number of overlapping transitions (Transition_NUM) can be selected according to actual needs.

[0052] In some embodiments, to ensure the display effect of video clips 409, the minimum duration (clipDuration_MIN) of each video clip 409 can be set to 100ms, and the minimum duration (transitionDuration_MIN) of overlapping transitions can also be set to 100ms. Of course, the duration of video clips 409 and overlapping transitions can also be selected according to actual needs, and this disclosure does not impose any restrictions on this. The trimming duration of video clips 409 should be greater than or equal to half the duration 417 of the initial overlapping transition 413 plus half the duration 419 of the ending overlapping transition 415.

[0053] In some embodiments, the user can set normal speed or curve speed for video clip 409. For normal speed, speed represents the speed multiplier, and the duration 411 of video clip 409 with duration t after normal speed is t / speed. For curve speed, an average speed multiplier (averageSpeed) can be calculated from the speed curve, so the duration 411 of video clip 409 with duration t after curve speed is t / averageSpeed. In the actual calculation of the duration after the speed change of the clip, in order to simplify the calculation, the slight difference between curve speed and average speed can be ignored, and a total speed (totalSpeed) can be used for calculation. For clips undergoing normal speed change, the total speed is equal to the speed multiplier. For clips undergoing curve speed change, the total speed is equal to the average speed multiplier. As shown in Figure 4, for video clip 409 configured with speed settings and overlapping transitions, the effective duration 421 range can be expressed as:

[0054] [clipDuration_MIN,sourceDuration / totalSpeed-TransitionDuration_MIN / 2×Transition_NUM](1)

[0055] Figure 4B A schematic diagram of parameter 400B of a video segment according to some embodiments of the present disclosure is shown. For example... Figure 4A As shown, based on the source video 401, the user can edit out a specific video clip 409. When video clip 409 is generated, its video parameters 423 are also generated. For example... Figure 4B As shown, video parameters 423 may include the source file path, source type, original source duration, normal speed adjustment factor, curve speed adjustment parameter coordinates, end overlap transition duration, start overlap transition duration, and video trimming start time. The source file path indicates the storage location of the source video 401 or source image. The source type represents whether the user-selected source is video or image. When the source is video, the effective duration 421 is as follows: Figure 4A As shown, when the source material is an image, the effective duration can be set according to actual needs. The original source material duration represents the source video duration 403, reflecting the total playback time of the source video 401. The start time for video trimming is the start editing time 405, marking the position where video segment 409 is first extracted from the source video 401.

[0056] Figure 5A flowchart of a beat-matching algorithm 500 according to some embodiments of the present disclosure is shown. In block 501, video segments and beat point information are determined. The video segments and beat point information can be determined by the client; the video segments can be user-inputted or downloaded from the cloud, and can be segments extracted from multiple different source videos. The beat point information can include beat points automatically identified by the client, beat points from templates downloaded from the cloud, and user-defined beat points. After determining the beat information, multiple beat intervals of the audio can be determined, for example, such as... Figure 3D As shown, [0.1000] can be the first beat interval of audio 323, and [1000,1800] can be the second beat interval of audio 323. It should be understood that audio with n beat points has n+1 beat intervals.

[0057] In box 503, a media pool is constructed for each beat interval period, and video clips are added to the media pool. In this embodiment of the disclosure, a media pool can be constructed for each beat interval period, and a list of media pools can be obtained for multiple beat interval periods. <materialpool>It can also build a list of source clips based on video clips. <clip>After obtaining the list of media pools and the list of media clips, you can use two pointers to traverse the list of media pools and the list of media clips from front to back, and take out video clips from the list of media clips in order and add them to the media pool corresponding to each beat interval.

[0058] In box 505, the expected effective duration (also known as the first effective duration) of each video segment is determined. To ensure that the start and end points of video segments, or overlapping transition centers, correspond to the start and end beat points of the beat interval, the sum of the longest effective times of all video segments in each media pool must be greater than or equal to the duration of the beat interval. At most, one media pool may exhaust all video segments but cannot guarantee that the sum of the longest effective times of the segments is greater than or equal to the duration of the beat interval; such a media pool will skip subsequent steps and proceed to step 513. To ensure that the sum of the longest effective times of all video segments in each media pool equals the duration of the beat interval, the duration of the video segments needs to be adjusted. In some embodiments, the expected effective duration can be determined proportionally to the duration of each video segment. The specific steps for determining the expected effective duration will be detailed in [the following section]. Figure 6A Detailed introduction.

[0059] In box 507, the duration of the video segment is adjusted based on the expected effective duration. Methods for adjusting the duration of a video segment may include cropping the video segment to shorten its duration, adding a new segment from the corresponding source video to extend its duration, or adjusting the duration of overlapping transitions. In some embodiments, during the adjustment of a video segment, priority may be given to keeping the start time of the video segment unchanged to maintain the coherence of the video narrative and the stability of the timeline; priority may be given to adjusting parts of the video segment other than overlapping transitions to ensure the smoothness and consistency of transition effects and avoid disrupting the natural transitions due to duration adjustments; and a condition of keeping the overall speed constant may be set to ensure the consistency of the video's rhythm and speed.

[0060] In box 509, determine if the total duration of the adjusted video clip is less than the duration of the audio. If the total duration of the adjusted video clip is less than the duration of the audio, execute box 511 to trim the audio. If the number of beat points exceeds the required number of video clips, the extra beat points will be discarded, and the audio will be shortened to the position of the last used beat point, i.e., the audio track end time will be adjusted to the video end time. If the total duration of the adjusted video clip is greater than or equal to the duration of the audio, execute box 513 to play the video clips not added to the media pool in the original editing effect and order. If the number of video clips exceeds the required number of beat intervals, or if there are video clips in the media pool that have been exhausted but the sum of the longest valid times of the clips cannot be guaranteed to be greater than or equal to the duration of the beat interval, the extra video clips will be arranged with their original duration, and the end overlapping transition duration will remain unchanged. The start overlapping transition duration will be aligned with the start overlapping transition of the previous video clip. Image materials can be arranged in 3-second increments by default.

[0061] Figure 6A A flowchart of a method 600A for determining the expected effective duration of each video segment according to some embodiments of the present disclosure is shown. In block 601, for each beat interval, a video segment corresponding to the beat interval is determined. As described above, after obtaining the media pool list and the media segment list, two pointers can be used to traverse the media pool list and the media segment list sequentially from front to back, taking video segments from the media segment list in order and adding them to each beat interval.

[0062] In box 603, the ratio (clip.ratio) of the duration of each video clip to the sum of the durations of all video clips is determined. In this embodiment of the disclosure, the index of the video clips in the media pool can be idx. The ratio is calculated starting from the video clip with idx of 0, and the following steps are performed according to the different ratios of each video clip. For example, if the beat interval period A corresponds to video clip A with a duration of 0.3s and video clip B with a duration of 0.6s, then the ratio of video clip A is 1 / 3 and the ratio of video clip B is 2 / 3. In box 605, it is determined whether the maximum effective duration of the video clip is less than the product of the beat time and the ratio. As mentioned above, the maximum effective duration of the video clip is the video duration plus the entropy of the overall speed change minus the shortest overlapping transition duration. When the maximum effective duration of the video clip is greater than or equal to the product of the duration of the beat interval period and the ratio, it indicates that only the duration of the video clip needs to be adjusted to meet the required timing. Then, box 607 is executed to determine the expected effective duration of the video clip as the product of the duration of the beat interval period and the ratio.

[0063] Continue to refer to Figure 6A When the maximum effective duration of a video segment is less than the product of the beat time and the ratio, it means that even extending the video segment to its maximum length is insufficient to meet the required expected effective duration. Then, execution box 609 determines the expected effective duration of each remaining video segment based on the ratio of the remaining video segments and the remaining duration of the beat interval. In this embodiment, video segments that cannot meet the expected effective duration can first be removed from the media pool. Then, the maximum effective duration of the video segment is subtracted from the duration of the beat interval to generate the remaining duration. Next, the ratio of the duration of each remaining video segment to the sum of the durations of the remaining video segments is determined. Finally, the expected effective duration of each remaining video segment is determined based on the product of the ratio and the remaining duration.

[0064] Figure 6B A diagram illustrating the determination of the expected effective duration of 600 bytes for each video segment according to some embodiments of this disclosure is shown. Figure 6B As shown, the duration of beat interval 609 is 1.6s, the original duration of video segment 611 is 0.2s, the original duration of video segment 613 is 0.4s, and the original duration of video segment 615 is 0.2s. According to... Figure 6A The method first determines the duration ratio of video segments 611, 613, and 615 to be 1:2:1. Then, based on the product of each video segment and the beat interval 609, the expected effective duration of each video segment is determined. When the maximum effective duration of each video segment is greater than or equal to the expected effective duration, the expected effective durations of video segments 611, 613, and 615 are 0.4s, 0.8s, and 0.4s, respectively. When the maximum effective duration of video segment 611 is 0.3s, the maximum effective duration of video segment 611 is subtracted from the duration of the beat interval 609 to generate a remaining duration of 1.3s. Then, the ratio of the duration of each remaining video segment to the sum of the durations of the remaining video segments is determined. The duration ratio of video segments 613 and 615 is 2:1. Finally, based on the product of the proportion and the remaining duration, the expected effective duration of each remaining video segment is determined. The expected effective duration of video segment 613 is (1.6-0.3)*2 / 3s, and the expected effective duration of video segment 615 is (1.6-0.3)*1 / 3s.

[0065] Figure 7A A flowchart of a method 700A for adjusting the duration of a video segment according to some embodiments of the present disclosure is shown. In block 701, the effective duration of the video segment and the expected effective duration are determined. According to... Figure 5 as well as Figure 6A In the steps described above, after determining the effective duration and expected effective duration of each video segment, the following duration adjustment steps can be performed for each video segment to make the effective duration of each video segment consistent with the expected effective duration. In box 703, it is determined whether the effective duration of the video segment is less than the expected effective duration. When the effective duration of the video segment is greater than the product, it indicates that the current effective duration of the video segment is sufficient to meet the required expected effective duration. Then, box 705 is executed to trim the part of the video segment other than the overlapping transition, while keeping the editing start time of the video segment unchanged, so that the effective duration of the video segment is shortened to the expected effective duration. In some embodiments, under the condition of satisfying formula (2), the duration of the video segment can be adjusted according to formula (3):

[0066] clip.clipRange.duration>=(clip.preTransitionDuration / 2+clip.expectedEfficientDuration+clip.endTransitionDuration / 2)*totalSpeed(2)clip.clipRange.duration=(clip.preTransitionDuration / 2+clip.expectedEfficientDuration+clip.endTransiti onDuration)*totalSpeed(3)

[0067] In box 707, it is determined whether the duration of the source video corresponding to the video segment is less than the product of the total duration and the overall speed change. When the effective duration of the video segment is less than the expected effective duration, it indicates that the current effective duration of the video segment is insufficient to meet the required expected effective duration. In this case, a new segment needs to be cut from the source video and added to the video segment to extend the effective duration of the video segment to the expected effective duration. The total duration represents the sum of the expected effective duration, half of the start overlapping transition duration of the video segment, and half of the end overlapping transition duration of the video segment. When the source video is greater than or equal to the product of the total duration and the overall speed change, box 709 is executed to cut a segment from the source video and add it to the video segment to extend the effective duration of the video segment. In this embodiment, it is preferable to keep the start editing time unchanged. In some embodiments, under the condition of satisfying formula (4), the duration of the video segment can be adjusted according to formula (5):

[0068] clip.sourceDuration>=(clip.preTransitionDuration / 2+clip.expectedEfficientDuration+clip.endTransition Duration / 2)*totalSpeed(4)clip.clipRange.duration=(clip.preTransitionDuration / 2+clip.expectedEfficientDuration+clip.endTransiti

[0069] onDuration / 2)*totalSpeed, clip.clipRange.startTime=clip.sourceDuration-clip.clipRange.duration(5)

[0070] In box 711, the remaining segments from the source video are added to the video segment. When the source video is less than the product of the total duration and the overall speed change, it means that even if all the clips in the source video are used to extend the video segment, the required effective duration cannot be met, and the remaining segments from the source video are added to the video segment. In some embodiments, under the condition of satisfying formula (6), the duration of the video segment can be adjusted according to formula (7):

[0071] clip.sourceDuration<(clip.preTransitionDuration / 2+clip.expectedEfficientDuration+clip.endTransition Duration / 2)*totalSpeed(6)

[0072] clip.clipRange.startTime=0,clip.clipRange.duration=clip.sourceDuration (7)

[0073] In box 713, it is determined whether the video clip has an end-overlapping transition. When using all the clips from the source video to extend the video clip still cannot meet the required expected effective duration, the duration of the overlapping transition needs to be adjusted so that the effective duration of the video clip is consistent with the expected effective duration. In this embodiment, a strategy of adjusting the end-overlapping transition first and then adjusting the beginning-overlapping transition is adopted. When there is no end-overlapping transition, 715 is executed to shorten the duration of the beginning-overlapping transition so that the effective duration of the video clip is consistent with the expected effective duration. In some embodiments, under the condition of satisfying formula (8), the duration of the beginning-overlapping transition can be adjusted according to formula (9): clip.endTransitionDuration=0,

[0074] clip.sourceDuration / totalSpeed-clip.preTransitionDuration / 2-clip.expectedEfficientDuration<0 (8)

[0075] clip.preTransitionDuration=(clip.sourceDuration / totalSpeed-clip.expectedEfficientDuration)*2 (9)

[0076] When an end-overlapping transition exists, execute box 717 to determine whether the duration of the end-overlapping transition is greater than a preset value. If the duration of the end-overlapping transition is less than the preset value, execute box 719 to extend the duration of the end-overlapping transition until it reaches the preset value, and then shorten the duration of the beginning-overlapping transition to make the effective duration of the video segment consistent with the expected effective duration. In some embodiments, under the condition of satisfying formula (10), the duration of the overlapping transition can be adjusted according to formula (11):

[0077] clip.sourceDuration / totalSpeed-clip.preTransitionDuration / 2-clip.expectedEfficientDuration<50ms

[0078] (10)clip.endTransitionDuration=100ms,

[0079] clip.preTransitionDuration=(clip.sourceDuration / totalSpeed-clip.endTransitionDuration / 2-clip.expected EfficientDuration)*2(11)

[0080] When the duration of the ending overlapping transition exceeds a preset value, execution box 721 is executed to shorten the duration of the ending overlapping transition, so that the effective duration of the video segment is consistent with the expected effective duration. In some embodiments, under the condition of satisfying formula (12), the duration of the overlapping transition can be adjusted according to formula (13):

[0081] clip.endTransitionDuration>0,

[0082] clip.sourceDuration / totalSpeed-clip.preTransitionDuration / 2-clip.expectedEfficientDuration>=50ms (12)

[0083] clip.endTransitionDuration=(clip.sourceDuration / totalSpeed-clip.preTransitionDuration / 2-clip.expected EfficientDuration)*2(13)

[0084] Figure 7B A schematic diagram illustrating the adjustment of the ending overlap transition and / or the beginning overlap transition 700B according to some embodiments of the present disclosure is shown. Figure 7B As shown, when the effective duration 737 of video segment 723 is less than the expected effective duration, and the duration 735 of video segment 723 is equal to the duration of the source video, the duration of the overlapping transition needs to be adjusted so that the effective duration 737 of video segment 723 is consistent with the expected effective duration. It should be understood that when the duration 735 of video segment 723 has reached its maximum value, i.e., the duration of the source video, and the duration 735 remains unchanged, since the duration 735 is the product of the effective duration 737 and the overall speed adjustment, the duration 733 of the ending overlapping transition 729, and the duration 727 of the starting overlapping transition 731, shortening the duration 733 and / or the duration 731 can extend the effective duration 737. In this embodiment, when the duration 733 of the ending overlapping transition 729 is greater than a preset value, the duration 733 is shortened first. When the ending overlapping transition 729 cannot be adjusted, the starting overlapping transition 727 is adjusted, while ensuring that the center time point of the starting overlapping transition 727 remains unchanged. In this way, the integrity of the video segment is ensured, while also meeting the user's requirements for video rhythm and effect.

[0085] Figure 8 The diagram illustrates the generated video 800 according to some embodiments of the present disclosure. For example... Figure 8 As shown, when the number of video clips exceeds the required number of beat intervals, firstly, the video clips that are completed in time are arranged sequentially according to the needs of the beat intervals and integrated with the audio 803. After the video clips that are completed in time are arranged, if there are any additional video clips 801, they can be placed after the video clips that are completed in time. To maintain overall smoothness and coherence, these video clips 801 can be arranged using their original duration. The start overlapping transition duration of a video clip should be aligned with the end overlapping transition duration of the previous video clip to ensure a smoother transition between video clips without abrupt jumps or breaks.

[0086] Figure 9 A block diagram of an apparatus 900 for generating video according to some embodiments of the present disclosure is shown. Figure 9 As shown, the device 900 includes a beat point information determination module 902, configured to determine the beat point information of audio associated with a first video. The device 900 also includes a video segment adjustment module 904, configured to adjust the duration of at least one of a plurality of video segments included in the first video based on the beat point information. The device 900 further includes a video generation module 906, configured to generate a second video based on the adjusted plurality of video segments and the audio, wherein the second video matches the beat point information of the audio.

[0087] Figure 10 A block diagram of a device 1000 capable of implementing various embodiments of the present disclosure is shown. For example... Figure 10 As shown, device 1000 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 1001, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 1002 or loaded from storage unit 1008 into random access memory (RAM) 1003. Various programs and data required for the operation of device 1000 can also be stored in RAM 1003. The CPU / GPU 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004. Although not shown in... Figure 10 As shown, device 1000 may also include a coprocessor.

[0088] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0089] The various methods or processes described above can be executed by CPU / GPU 1001. For example, in some embodiments, the methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by CPU / GPU 1001, one or more steps or actions in the methods or processes described above can be performed.

[0090] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0091] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0092] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0093] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0094] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0095] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0097] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0098] The following are some example implementations of this disclosure.

[0099] Example 1. A method for generating video, comprising:

[0100] Determine the beat point information of the audio associated with the first video;

[0101] Based on the beat point information, adjust the duration of at least one video segment among the multiple video segments included in the first video; and

[0102] Based on the adjusted multiple video clips and the audio, a second video is generated, which is matched with the beat point information of the audio.

[0103] Example 2. The method according to Example 1, wherein adjusting the duration of at least one video segment among a plurality of video segments included in the first video includes:

[0104] Based on the beat point information, multiple beat interval periods between adjacent beat points are determined;

[0105] Based on the multiple beat intervals, a first effective duration is determined for each video segment; and based on the first effective duration, the duration of at least one of the multiple video segments is adjusted.

[0106] Example 3. The method according to any one of Examples 1-2, wherein determining the first valid duration of each video segment includes:

[0107] For each beat interval, determine the video segment corresponding to the beat interval;

[0108] Determine the ratio of the duration of each of the plurality of video segments to the sum of the durations of the video segments; and

[0109] Based on the proportion of each video segment among the plurality of video segments and the duration of the beat interval, a first effective duration for each video segment is determined.

[0110] Example 4. According to any one of Examples 1-3, the video segment includes a first video segment and remaining video segments, the ratio of the duration of the first video segment to the sum of the durations of the remaining video segments is a first ratio, the maximum effective duration of the first video segment is equal to the duration of the first source video and the entropy of the overall speed change minus the shortest overlapping transition duration, wherein determining the first effective duration of each video segment further includes:

[0111] Determine whether the maximum effective duration of the first video segment is less than the product of the first ratio and the duration of the beat interval period;

[0112] In response to the fact that the maximum effective duration of the first video segment is less than the product of the first ratio and the duration of the beat interval, the first effective duration of the first video segment is determined to be the maximum effective duration;

[0113] Subtract the maximum effective duration from the duration of the beat interval to generate the remaining duration; and

[0114] The first effective duration of each remaining video segment is determined based on the ratio of the duration of each remaining video segment to the sum of the durations of the remaining video segments, and the remaining duration itself.

[0115] Example 5. According to any one of Examples 1-4, the effective time period of the video segment is the time period from the start point or the midpoint of the start overlapping transition to the end point or the midpoint of the end overlapping transition of the video segment, wherein the plurality of video segments includes a second video segment, and adjusting the duration of at least one of the plurality of video segments includes:

[0116] Determine whether the second valid duration corresponding to the valid time period of the second video segment is less than the first valid duration of the second video segment;

[0117] In response to the second effective duration of the second video segment being greater than the first effective duration of the second video segment, the portion of the second video segment other than overlapping transitions is trimmed, while maintaining the edit start time of the second video segment unchanged; and

[0118] In response to the second valid duration of the second video segment being equal to the first valid duration of the second video segment, the second valid duration of the second video segment remains unchanged.

[0119] Example 6. According to any one of Examples 1-5, the total duration is the sum of the first effective duration, half the start overlapping transition duration of the second video segment, and half the end overlapping transition duration of the second video segment, and the method further includes:

[0120] In response to the second effective duration of the second video segment being less than the first effective duration of the second video segment, it is determined whether the duration of the second source video corresponding to the second video segment and the entropy of the overall speed change are less than the total duration;

[0121] In response to the duration of the second source video being greater than or equal to the entropy of the combined speed change, a segment is trimmed from the second source video; and

[0122] The cropped segment is added to the second video segment.

[0123] Example 7. According to any one of Examples 1-6, the second video segment includes a start editing time and an end editing time, wherein cropping the segment from the second source video includes:

[0124] The segment after the end of the editing time is cut from the second source video to keep the start time unchanged.

[0125] Example 8. The method according to any one of Examples 1-7 further includes:

[0126] In response to the fact that the duration of the second source video and the entropy of the combined speed change are less than the total duration, the remaining segments in the second source video relative to the second video segment are added to the second video segment;

[0127] Determine if the second video segment has an overlapping transition at the end;

[0128] In response to the presence of an end-overlapping transition in the second video segment, determine whether the duration of the end-overlapping transition in the second video segment exceeds a preset value; and

[0129] In response to the end-overlap transition duration of the second video segment being greater than the preset value, the end-overlap transition duration is shortened.

[0130] Example 9. The method according to any one of Examples 1-8 further includes:

[0131] In response to the second video segment's ending overlap transition duration being less than the preset value, the ending overlap transition duration is adjusted to the preset value; and

[0132] Shorten the initial overlapping transition duration of the second video segment.

[0133] Example 10. The method according to any one of Examples 1-9 further includes:

[0134] In response to the absence of an end overlap transition in the second video segment, the duration of the beginning overlap transition in the second video segment is shortened.

[0135] Example 11. The method according to any one of Examples 1-10, wherein generating the second video includes:

[0136] In response to the fact that the sum of the durations of the adjusted video segments is greater than the duration of the audio, the video segments after the audio ends are played sequentially according to the original editing effect and the original editing order.

[0137] Example 12. The method according to any one of Examples 1-11, wherein generating the second video further includes:

[0138] In response to the audio duration being greater than the sum of the adjusted durations of the plurality of video segments, the audio is trimmed so that the audio duration matches the sum of the adjusted durations of the plurality of video segments.

[0139] Example 13. An apparatus for generating video, comprising:

[0140] The beat point information determination module is configured to determine the beat point information of the audio associated with the first video;

[0141] The video segment adjustment module is configured to adjust the duration of at least one video segment among multiple video segments included in the first video based on the beat point information; and

[0142] The video generation module is configured to generate a second video based on multiple adjusted video clips and the audio, wherein the second video matches the beat point information of the audio.

[0143] Example 14. The apparatus according to Example 13, wherein the video segment adjustment module includes:

[0144] The beat interval period determination module is configured to determine multiple beat interval periods between adjacent beat points based on the beat point information;

[0145] The first effective duration determination module is configured to determine the first effective duration of each video segment based on the multiple beat interval time periods; and

[0146] The duration adjustment module is configured to adjust the duration of at least one of the plurality of video segments based on the first effective duration.

[0147] Example 15. The apparatus according to any one of Examples 13-14, wherein the first effective duration determining module comprises:

[0148] The video segment determination module is configured to determine a video segment corresponding to each of the plurality of beat interval periods;

[0149] The ratio determination module is configured to determine the ratio of the duration of each video segment in the plurality of video segments to the sum of the durations of the video segments; and

[0150] The first effective duration determination module is configured to determine the first effective duration of each video segment based on the proportion of each video segment in the plurality of video segments and the duration of the beat interval period.

[0151] Example 16. The apparatus according to any one of Examples 13-15, wherein the video segment comprises a first video segment and remaining video segments, the ratio of the duration of the first video segment to the sum of the durations of the remaining video segments is a first ratio, and the maximum effective duration of the first video segment is equal to the duration of the first source video and the entropy of the overall speed change minus the shortest overlapping transition duration, wherein the first effective duration determination module further comprises:

[0152] The first calculation module is configured to determine whether the maximum effective duration of the first video segment is less than the product of the first ratio and the duration of the beat interval period;

[0153] The first duration determination module is configured to determine the first effective duration of the first video segment as the maximum effective duration in response to the fact that the maximum effective duration of the first video segment is less than the product of the first ratio and the duration of the beat interval period.

[0154] The remaining duration generation module is configured to subtract the maximum effective duration from the duration of the beat interval to generate the remaining duration; and

[0155] The second effective duration determination module is configured to determine the first effective duration of each remaining video segment based on the ratio of the duration of each remaining video segment to the sum of the durations of the remaining video segments and the remaining duration.

[0156] Example 17. The apparatus according to any one of Examples 13-16, wherein the effective time period of the video segment is the time period from the start point or the midpoint of the start overlapping transition to the end point or the midpoint of the end overlapping transition of the video segment, the plurality of video segments including a second video segment, wherein the duration adjustment module includes:

[0157] The second calculation module is configured to determine whether the second effective duration of the second video segment corresponding to the effective time period is less than the first effective duration of the second video segment;

[0158] The first duration adjustment module is configured to, in response to a second effective duration of the second video segment being greater than a first effective duration of the second video segment, trim the portion of the second video segment other than overlapping transitions, while keeping the edit start time of the second video segment unchanged; and

[0159] The second duration adjustment module is configured to keep the second effective duration of the second video segment unchanged in response to the second effective duration of the second video segment being equal to the first effective duration of the second video segment.

[0160] Example 18. The apparatus according to any one of Examples 13-17, wherein the total duration is the sum of the first effective duration, half the start overlap transition duration of the second video segment, and half the end overlap transition duration of the second video segment, and the apparatus further comprises:

[0161] The third calculation module is configured to determine whether the duration of the second source video corresponding to the second video segment and the entropy of the combined speed change are less than the total duration in response to the second effective duration of the second video segment being less than the first effective duration of the second video segment.

[0162] A first segment trimming module is configured to trim segments from the second source video in response to a total duration where the duration of the second source video and the entropy of the combined speed change are greater than or equal to the total duration; and

[0163] The first segment adding module is configured to add the cropped segment to the second video segment.

[0164] Example 19. The apparatus according to any one of Examples 13-18, wherein the second video segment includes a start editing time and an end editing time, and the first segment trimming module includes:

[0165] The start time preservation module is configured to cut segments from the second source video after the end edit time to keep the start edit time unchanged.

[0166] Example 20. The apparatus according to any one of Examples 13-19 further includes:

[0167] The second segment addition module is configured to add the remaining segments in the second source video relative to the second video segment to the second video segment in response to the duration of the second source video and the entropy of the comprehensive speed change being less than the total duration.

[0168] The end-overlapping transition determination module is configured to determine whether the second video segment has an end-overlapping transition;

[0169] The fourth calculation module is configured to, in response to the presence of an end-overlapping transition in the second video segment, determine whether the duration of the end-overlapping transition in the second video segment is greater than a preset value; and

[0170] The first end-overlap transition adjustment module is configured to shorten the end-overlap transition duration in response to the second video segment having an end-overlap transition duration greater than the preset value.

[0171] Example 21. The apparatus according to any one of Examples 13-20 further includes:

[0172] The second end-overlap transition adjustment module is configured to adjust the end-overlap transition duration to the preset value in response to a situation where the end-overlap transition duration of the second video segment is less than the preset value; and

[0173] The first initial overlap transition adjustment module is configured to shorten the initial overlap transition duration of the second video segment.

[0174] Example 22. The apparatus according to any one of Examples 13-21 further includes:

[0175] The second start-overlap transition adjustment module is configured to shorten the start-overlap transition duration of the second video segment in response to the absence of an end-overlap transition in the second video segment.

[0176] Example 23. The apparatus according to any one of Examples 13-22, wherein the video generation module comprises:

[0177] The video arrangement module is configured to, in response to the sum of the durations of the adjusted plurality of video segments being greater than the duration of the audio, play the video segments after the audio ends in the original editing effect and original editing order.

[0178] Example 24. The apparatus according to any one of Examples 13-23, wherein the video generation module further comprises:

[0179] An audio trimming module is configured to trim the audio in response to the audio duration being greater than the sum of the adjusted durations of the plurality of video segments, so that the audio duration matches the sum of the adjusted durations of the plurality of video segments.

[0180] Example 25. An electronic device comprising:

[0181] Processor; and

[0182] A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform actions, the actions including:

[0183] Determine the beat point information of the audio associated with the first video;

[0184] Based on the beat point information, adjust the duration of at least one video segment among the multiple video segments included in the first video; and

[0185] Based on the adjusted multiple video clips and the audio, a second video is generated, which is matched with the beat point information of the audio.

[0186] Example 26. The electronic device according to Example 25, wherein adjusting the duration of at least one video segment among a plurality of video segments included in the first video comprises:

[0187] Based on the beat point information, multiple beat interval periods between adjacent beat points are determined;

[0188] Based on the multiple beat intervals, a first effective duration is determined for each video segment; and based on the first effective duration, the duration of at least one of the multiple video segments is adjusted.

[0189] Example 27. An electronic device according to any one of Examples 25-26, wherein determining the first valid duration of each video segment includes:

[0190] For each of the multiple beat interval periods, determine the video segment corresponding to the beat interval period;

[0191] Determine the ratio of the duration of each of the plurality of video segments to the sum of the durations of the video segments; and

[0192] Based on the proportion of each video segment among the plurality of video segments and the duration of the beat interval, a first effective duration for each video segment is determined.

[0193] Example 28. An electronic device according to any one of Examples 25-27, wherein the video segment comprises a first video segment and remaining video segments, the ratio of the duration of the first video segment to the sum of the durations of the remaining video segments is a first ratio, the maximum effective duration of the first video segment is equal to the duration of the first source video and the entropy of the overall speed change minus the shortest overlapping transition duration, wherein determining the first effective duration of each video segment further includes:

[0194] Determine whether the maximum effective duration of the first video segment is less than the product of the first ratio and the duration of the beat interval period;

[0195] In response to the fact that the maximum effective duration of the first video segment is less than the product of the first ratio and the duration of the beat interval, the first effective duration of the first video segment is determined to be the maximum effective duration;

[0196] Subtract the maximum effective duration from the duration of the beat interval to generate the remaining duration; and

[0197] The first effective duration of each remaining video segment is determined based on the ratio of the duration of each remaining video segment to the sum of the durations of the remaining video segments, and the remaining duration itself.

[0198] Example 29. An electronic device according to any one of Examples 25-28, wherein the effective time period of the video segment is the time period from the start point or the midpoint of the start overlapping transition to the end point or the midpoint of the end overlapping transition of the video segment, the plurality of video segments including a second video segment, wherein adjusting the duration of at least one of the plurality of video segments includes:

[0199] Determine whether the second valid duration corresponding to the valid time period of the second video segment is less than the first valid duration of the second video segment;

[0200] In response to the second effective duration of the second video segment being greater than the first effective duration of the second video segment, the portion of the second video segment other than overlapping transitions is trimmed, while maintaining the edit start time of the second video segment unchanged; and

[0201] In response to the second valid duration of the second video segment being equal to the first valid duration of the second video segment, the second valid duration of the second video segment remains unchanged.

[0202] Example 30. An electronic device according to any one of Examples 25-29, wherein the total duration is the sum of the first effective duration, half the start overlap transition duration of the second video segment, and half the end overlap transition duration of the second video segment, and the action further includes:

[0203] In response to the second effective duration of the second video segment and the entropy of the overall speed change being less than the first effective duration of the second video segment, it is determined whether the duration of the second source video corresponding to the second video segment and the entropy of the overall speed change are less than the total duration;

[0204] In response to the second source video having a duration greater than or equal to the total duration, a segment is trimmed from the second source video; and

[0205] The cropped segment is added to the second video segment.

[0206] Example 31. An electronic device according to any one of Examples 25-30, wherein the second video segment includes a start edit time and an end edit time, wherein cropping the segment from the second source video includes:

[0207] The segment after the end of the editing time is cut from the second source video to keep the start time unchanged.

[0208] Example 32. The electronic device according to any one of Examples 25-31 further includes:

[0209] In response to the fact that the duration of the second source video and the entropy of the combined speed change are less than the total duration, the remaining segments in the second source video relative to the second video segment are added to the second video segment;

[0210] Determine if the second video segment has an overlapping transition at the end;

[0211] In response to the presence of an end-overlapping transition in the second video segment, determine whether the duration of the end-overlapping transition in the second video segment exceeds a preset value; and

[0212] In response to the end-overlap transition duration of the second video segment being greater than the preset value, the end-overlap transition duration is shortened.

[0213] Example 33. The electronic device according to any one of Examples 25-32 further includes:

[0214] In response to the second video segment's ending overlap transition duration being less than the preset value, the ending overlap transition duration is adjusted to the preset value; and

[0215] Shorten the initial overlapping transition duration of the second video segment.

[0216] Example 34. The electronic device according to any one of Examples 25-33 further includes:

[0217] In response to the absence of an end overlap transition in the second video segment, the duration of the beginning overlap transition in the second video segment is shortened.

[0218] Example 35. An electronic device according to any one of Examples 25-34, wherein generating the second video includes:

[0219] In response to the fact that the sum of the durations of the adjusted video segments is greater than the duration of the audio, the video segments after the audio ends are played sequentially according to the original editing effect and the original editing order.

[0220] Example 36. The electronic device according to any one of Examples 25-35, wherein generating the second video further includes:

[0221] In response to the audio duration being greater than the sum of the adjusted durations of the plurality of video segments, the audio is trimmed so that the audio duration matches the sum of the adjusted durations of the plurality of video segments.

[0222] Example 37. A computer-readable storage medium having stored thereon computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of Examples 1 to 12.

[0223] Example 38. A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of Examples 1 to 12.

[0224] Although this disclosure has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.< / clip> < / materialpool>

Claims

1. A method for generating video, comprising: Determine the beat point information of the audio associated with the first video; Based on the beat point information, adjust the duration of at least one video segment among the multiple video segments included in the first video; as well as Based on the adjusted multiple video clips and the audio, a second video is generated, which is matched with the beat point information of the audio.

2. The method according to claim 1, wherein adjusting the duration of at least one video segment among the plurality of video segments included in the first video comprises: Based on the beat point information, multiple beat interval periods between adjacent beat points are determined; Based on the multiple beat intervals, determine the first effective duration of each video segment; as well as Based on the first effective duration, the duration of at least one of the plurality of video segments is adjusted.

3. The method of claim 2, wherein determining the first effective duration of each video segment comprises: For each of the multiple beat interval periods, determine the video segment corresponding to the beat interval period; Determine the ratio of the duration of each video segment to the sum of the durations of the video segments; as well as Based on the proportion of each video segment in the plurality of video segments and the duration of the beat interval, a first effective duration for each video segment in the plurality of video segments is determined.

4. The method according to claim 3, wherein the video segment includes a first video segment and remaining video segments, the ratio of the duration of the first video segment to the sum of the durations of the remaining video segments is a first ratio, the maximum effective duration of the first video segment is equal to the duration of the first source video and the entropy of the overall speed change minus the shortest overlapping transition duration, wherein determining the first effective duration of each video segment further includes: Determine whether the maximum effective duration of the first video segment is less than the product of the first ratio and the duration of the beat interval period; In response to the fact that the maximum effective duration of the first video segment is less than the product of the first ratio and the duration of the beat interval, the first effective duration of the first video segment is determined to be the maximum effective duration; Subtract the maximum effective duration from the duration of the beat interval to generate the remaining duration; as well as The first effective duration of each remaining video segment is determined based on the ratio of the duration of each remaining video segment to the sum of the durations of the remaining video segments, and the remaining duration itself.

5. The method according to claim 2, wherein the effective time period of the video segment is the time period from the start point or the midpoint of the start overlapping transition to the end point or the midpoint of the end overlapping transition of the video segment, the plurality of video segments includes a second video segment, wherein adjusting the duration of at least one of the plurality of video segments includes: Determine whether the second valid duration corresponding to the valid time period of the second video segment is less than the first valid duration of the second video segment; In response to the second effective duration of the second video segment being greater than the first effective duration of the second video segment, the portion of the second video segment other than the overlapping transitions is trimmed, while keeping the editing start time of the second video segment unchanged; as well as In response to the second valid duration of the second video segment being equal to the first valid duration of the second video segment, the second valid duration of the second video segment remains unchanged.

6. The method according to claim 5, wherein the sum of the first effective duration, half the start overlap transition duration of the second video segment, and half the end overlap transition duration of the second video segment is the total duration, and the method further comprises: In response to the second effective duration of the second video segment being less than the first effective duration of the second video segment, it is determined whether the duration of the second source video corresponding to the second video segment and the entropy of the overall speed change are less than the total duration; In response to the fact that the duration of the second source video and the entropy of the integrated speed change are greater than or equal to the total duration, a segment is cut from the second source video; as well as The cropped segment is added to the second video segment.

7. The method of claim 6, wherein the second video segment includes a start editing time and an end editing time, wherein cropping the segment from the second source video includes: The segment after the end of the editing time is cut from the second source video to keep the start time unchanged.

8. The method according to claim 6, further comprising: In response to the fact that the duration of the second source video and the entropy of the combined speed change are less than the total duration, the remaining segments in the second source video relative to the second video segment are added to the second video segment; Determine if the second video segment has an overlapping transition at the end; In response to the presence of an end-overlapping transition in the second video segment, determine whether the duration of the end-overlapping transition in the second video segment exceeds a preset value; and In response to the end-overlap transition duration of the second video segment being greater than the preset value, the end-overlap transition duration is shortened.

9. The method according to claim 8, further comprising: In response to the fact that the end overlap transition duration of the second video segment is less than the preset value, the end overlap transition duration is adjusted to the preset value; as well as Shorten the initial overlapping transition duration of the second video segment.

10. The method of claim 8, further comprising: In response to the absence of an end overlap transition in the second video segment, the duration of the beginning overlap transition in the second video segment is shortened.

11. The method of claim 1, wherein generating the second video comprises: In response to the fact that the sum of the durations of the adjusted video segments is greater than the duration of the audio, the video segments after the audio ends are played sequentially according to the original editing effect and the original editing order.

12. The method of claim 11, wherein generating the second video further comprises: In response to the audio duration being greater than the sum of the adjusted durations of the plurality of video segments, the audio is trimmed so that the audio duration matches the sum of the adjusted durations of the plurality of video segments.

13. An apparatus for generating video, comprising: The beat point information determination module is configured to determine the beat point information of the audio associated with the first video; The video segment adjustment module is configured to adjust the duration of at least one video segment among multiple video segments included in the first video based on the beat point information. as well as The video generation module is configured to generate a second video based on multiple adjusted video clips and the audio, wherein the second video matches the beat point information of the audio.

14. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1-12.

15. A computer-readable storage medium having stored thereon computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of claims 1-12.

Images