Video generation method and device, computing equipment, storage medium and product

Abstract

The embodiment of the invention discloses a video generation method and device, computing equipment, a storage medium and a product, and the method comprises the steps: obtaining to-be-processed audio data and to-be-processed video data; performing modal feature extraction on the audio data and the video data to obtain audio rhythm points of the audio data and video key points of the video data; according to the audio rhythm points, aligning the video key points with the audio rhythm points, and intercepting the video data to form a plurality of video clips; and generating a target video according to the plurality of video clips and the aligned video key points and audio rhythm points. According to the method, the audio rhythm points and the video key points can be automatically identified and automatically aligned, the video with an audio-visual synchronization effect is conveniently generated, the audio data and the video data do not need to be artificially identified and matched, the video creation difficulty is effectively reduced, and the video creation efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of Internet technology, and specifically to a video generation method, apparatus, computing device, storage medium, and product. Background Art

[0002] With the development of internet technology, more and more users enjoy creating and sharing videos. Typically, users create videos using video production tools or apps. Existing video production processes often rely on manual operations, requiring manual identification of the audio rhythm and video content, and manual adjustment of the audio and video to achieve a match. However, this video creation method is not only time-consuming and labor-intensive, with low processing efficiency, but also requires users to possess a high level of professional knowledge. The high threshold for video creation makes it inconvenient for users with limited professional knowledge to create videos. Summary of the Invention

[0003] In view of the above problems, this application proposes a video generation method, device, computing equipment, storage medium and product to solve the following problems: the existing video creation method requires users to have high professional knowledge, the creation threshold is high, and it is not convenient for users with weak professional knowledge to create videos.

[0004] According to one aspect of an embodiment of the present application, a video generation method is provided, including:

[0005] Obtaining audio data and video data to be processed;

[0006] Perform modal feature extraction on audio data and video data to obtain audio rhythm points of audio data and video key points of video data;

[0007] According to the audio rhythm points, the video key points are aligned with the audio rhythm points, and the video data is intercepted to form multiple video clips;

[0008] Generate a target video based on multiple video clips and the aligned video key points and audio rhythm points.

[0009] Furthermore, aligning the video key points with the audio rhythm points according to the audio rhythm points further includes:

[0010] Based on the audio rhythm points, the video key points are aligned with the audio rhythm points in order to determine the audio rhythm points corresponding to the video key points.

[0011] Furthermore, intercepting the video data to form a plurality of video segments further includes:

[0012] For any video key point having a corresponding audio rhythm point, determining a cut position corresponding to the video key point based on the duration between the audio rhythm point corresponding to the video key point and an adjacent audio rhythm point of the audio rhythm point;

[0013] Based on the interception position, the video data corresponding to the key point of the video is intercepted to obtain the video segment corresponding to the key point of the video.

[0014] Furthermore, the method further comprises:

[0015] If the duration of any video clip does not meet the corresponding target duration, a speed change multiple is selected from the preset speed change range, and the video clip is speed-changed according to the speed change multiple;

[0016] If the duration of the video clip after speed change does not meet the corresponding target duration, the preset filler image is added to the main track.

[0017] Furthermore, generating a target video based on the plurality of video clips and the aligned video key points and audio rhythm points further includes:

[0018] Add audio data to the audio track and set the start time of the audio data;

[0019] Add multiple video clips to the master track and set the start time of each video clip based on the aligned video key points and audio rhythm points.

[0020] Furthermore, the method further comprises:

[0021] According to the video production template, add each material to the corresponding track, and set the start display time, display duration and zoom ratio of each material.

[0022] According to another aspect of an embodiment of the present application, a video generating apparatus is provided, including:

[0023] An acquisition module, adapted to acquire audio data and video data to be processed;

[0024] An extraction module, adapted to extract modal features from audio data and video data to obtain audio rhythm points of the audio data and video key points of the video data;

[0025] An alignment module adapted to align video key points with audio rhythm points according to audio rhythm points, and to intercept video data to form multiple video clips;

[0026] The generation module is suitable for generating a target video based on multiple video clips and aligned video key points and audio rhythm points.

[0027] According to another aspect of an embodiment of the present application, a computing device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0028] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned video generation method.

[0029] According to another aspect of the embodiments of the present application, a computer storage medium is provided, in which at least one executable instruction is stored. The executable instruction enables a processor to perform operations corresponding to the above-mentioned video generation method.

[0030] According to another aspect of the embodiments of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned video generation method.

[0031] According to the technical solution provided in the embodiments of the present application, modal features of audio data and video data are extracted, and the audio rhythm points of the audio data and the video key points of the video data can be automatically identified; according to the audio rhythm points, the video key points are automatically aligned with the audio rhythm points, and the video data is intercepted to form multiple video clips. Based on the multiple video clips and the aligned video key points and audio rhythm points, a video with an audio-visual synchronization effect can be easily generated, and automatic card point video generation is realized. There is no need to manually identify and match the audio data and the video data, which effectively reduces the difficulty of video creation and improves the efficiency of video creation, and provides users with weak professional knowledge with a video creation method that is easy to operate and has a good experience.

[0032] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation method of the embodiment of the present application is specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the embodiments of the present application. The same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0034] Figure 1 A schematic diagram showing a flow chart of a video generation method according to an embodiment of the present application is shown;

[0035] Figure 2A schematic flow chart of a video generation method according to another embodiment of the present application is shown;

[0036] Figure 3a Schematic diagram showing the aligned video key points and audio rhythm points and the video clips corresponding to the video key points Figure 1 ;

[0037] Figure 3b Schematic diagram showing the aligned video key points and audio rhythm points and the video clips corresponding to the video key points Figure 2 ;

[0038] Figure 3c Schematic diagram 3 showing the video key points and audio rhythm points after alignment processing and the video clips corresponding to the video key points;

[0039] Figure 3d Schematic diagram 4 showing the video key points and audio rhythm points after alignment processing and the video clips corresponding to the video key points;

[0040] Figure 4a A schematic diagram showing the composition of a video production template is shown;

[0041] Figure 4b Shows a schematic diagram of the composition of a target video Figure 1 ;

[0042] Figure 4c Shows a schematic diagram of the composition of a target video Figure 2 ;

[0043] Figure 5 A structural block diagram of a video generating device according to an embodiment of the present application is shown;

[0044] Figure 6 A schematic structural diagram of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0045] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0046] First, the terms involved in one or more embodiments of the present application are explained.

[0047] Multimodal matching: When processing audio and video, use matching between multiple modalities (such as visual and auditory modalities) to identify and synchronize feature information of different modalities.

[0048] In audio and video editing, "Cut-in" refers to precisely slicing a video based on specific beats or climaxes in the music or sound effects to achieve audio-visual synchronization. This technique is widely used in short videos and music videos to enhance the viewing experience by synchronizing the audiovisual rhythm.

[0049] Timeline: refers to the project data obtained by expanding the video in chronological order, including the main track, audio track, subtitle track, sticker track, etc.

[0050] Figure 1 A schematic diagram of a video generation method according to an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes the following steps:

[0051] Step S101: Acquire audio data and video data to be processed.

[0052] In an embodiment of the present application, multimodal matching and automatic editing can be performed on the audio data and video data to be processed, and the video key points and audio rhythm points can be automatically aligned and matched. Without the need to manually identify and match the audio data and video data, a target video with an audio-visual synchronization effect, i.e., a card point video, can be generated, which effectively reduces the difficulty of video creation.

[0053] When a user wants to create a card point video, he or she can select or upload the audio data and video data to be processed in the client corresponding to the video creation, wherein the audio data and video data are used to generate the target video, and the audio data and video data to be processed are obtained in step S101.

[0054] Step S102 : extracting modal features from the audio data and the video data to obtain audio rhythm points of the audio data and video key points of the video data.

[0055] Modal feature extraction is performed on audio data and video data to identify the feature information of audio data in the audio modality and the feature information of video data in the video modality, respectively. For example, the rhythm, spectrum, emotion, etc. of the audio data are analyzed to obtain the audio rhythm points of the audio data; the scene switching, action change, emotion, etc. of the picture content of the video data are analyzed to obtain the key frames of the video data, and then the video key points of the video data are obtained.

[0056] Step S103: aligning the video key points with the audio rhythm points according to the audio rhythm points, and intercepting the video data to form a plurality of video clips.

[0057] In an embodiment of the present application, the video key points and the audio rhythm points are aligned in order based on the audio rhythm points, and the audio rhythm points corresponding to the video key points are determined, so that the video key points and the audio rhythm points correspond one to one, thereby obtaining the aligned video key points and audio rhythm points.

[0058] After obtaining the aligned video key points and audio rhythm points, the video data can be intercepted to form multiple video segments according to the specific alignment situation, so that the start playback time of each video segment can be set according to the aligned video key points and audio rhythm points, thereby achieving matching between audio data and video data and generating a target video with audio-visual synchronization effect.

[0059] Step S104: generating a target video based on the plurality of video clips and the aligned video key points and audio rhythm points.

[0060] After obtaining the aligned video key points, audio rhythm points, and multiple video clips, the target video can be automatically generated through track laying. Specifically, the audio data and multiple video clips can be laid out on a timeline based on the aligned video key points and audio rhythm points, and then rendered to generate the target video.

[0061] According to the video generation method provided in the embodiment of the present application, modal features of audio data and video data are extracted, and the audio rhythm points of the audio data and the video key points of the video data can be automatically identified; according to the audio rhythm points, the video key points are automatically aligned with the audio rhythm points, and the video data is intercepted to form multiple video clips. Based on the multiple video clips and the aligned video key points and audio rhythm points, a video with an audio-visual synchronization effect can be easily generated, and automatic card-point video generation is realized. There is no need to manually identify and match the audio data and the video data, which effectively reduces the difficulty of video creation and improves the efficiency of video creation, and provides users with weak professional knowledge with a video creation method that is easy to operate and has a good experience.

[0062] Figure 2 A schematic diagram of a video generation method according to another embodiment of the present application is shown. Figure 2 As shown, the method includes the following steps:

[0063] Step S201: Acquire audio data and video data to be processed.

[0064] In an optional implementation, when a user wants to create a card point video, the user can upload the audio data and video data to be processed in a client corresponding to video creation.

[0065] In another optional implementation, users can select and submit audio and video data to be processed from a resource library through the corresponding video creation client. To further enhance the convenience of video creation, a resource library can be provided to users, storing and managing various types of resources, including audio and video data, text, images, stickers, etc., from which users can select the audio and video data they need for their video creation.

[0066] Step S202 , performing modal feature extraction on the audio data and the video data to obtain audio rhythm points of the audio data and video key points of the video data.

[0067] After acquiring the audio data and video data, the acquired audio data and video data may be pre-processed, for example, by demultiplexing the audio data and video data. Taking video data as an example, demultiplexing can separate the multiple streams (e.g., video stream, audio stream, etc.) contained in the video data for separate processing or editing.

[0068] The audio data is subjected to audio modality feature extraction to obtain audio rhythm points of the audio data, and the video data is subjected to video modality feature extraction to obtain video key points of the video data. The video key points can specifically be the time points corresponding to key frames in the video data.

[0069] Specifically, for audio data, beat detection algorithms (beat detection) and dynamic time warping (DTW) can be used to monitor the audio data's rhythm, and spectral analysis can be performed using Mel-Frequency Cepstral Coefficients (MFCC) and Short-Time Fourier Transform (STFT). Furthermore, emotion recognition and other processing can be performed on the audio data to obtain the audio rhythm points of the audio data. For video data, scene switching can be monitored, emotions of characters in the picture content can be extracted, and action changes can be identified to obtain key frames of the video data, and then the video key points of the video data can be obtained.

[0070] Step S203: aligning the video key points with the audio rhythm points according to the audio rhythm points.

[0071] Using the audio rhythm points as a reference, the video key points are aligned with the audio rhythm points in order, and the audio rhythm points corresponding to the video key points are determined, so that the video key points and the audio rhythm points correspond one to one. Because the number of audio rhythm points and the number of video key points may be different, for example, the number of audio rhythm points may be greater than the number of video key points, or the number of audio rhythm points may be less than the number of video key points, after the alignment process, some audio rhythm points may not have corresponding video key points, or some video key points may not have corresponding audio rhythm points.

[0072] For example, if the number of audio rhythm points is 10 and the number of video key points is 5, then the video key points and the audio rhythm points are aligned in order based on the audio rhythm points. Then, the first 5 audio rhythm points of the 10 audio rhythm points correspond one-to-one with these 5 video key points, and the last 5 audio rhythm points do not have corresponding video key points.

[0073] For example, the number of audio rhythm points is 5 and the number of video key points is 10. Then, based on the audio rhythm points, the video key points and the audio rhythm points are aligned in order. Then, these 5 audio rhythm points correspond one-to-one with the first 5 video key points among the 10 video key points, and the last 5 video key points do not have corresponding audio rhythm points.

[0074] Step S204 : for any video key point having a corresponding audio rhythm point, determine a cut position corresponding to the video key point according to the duration between the audio rhythm point corresponding to the video key point and an adjacent audio rhythm point of the audio rhythm point.

[0075] To generate a target video with audio-visual synchronization, the video data needs to be intercepted to form multiple video segments based on the specific alignment conditions. To ensure the continuity of the previous and next screen content, in this embodiment of the application, the interception position corresponding to the video key point is determined by combining the duration between the corresponding audio rhythm point and the adjacent audio rhythm point. The interception position includes: the starting interception position and the ending interception position.

[0076] The basic principle for determining the clipping position can be: within the natural boundary of the video data, for any video key point with a corresponding audio rhythm point, the video key point is called the nth video key point, and the audio rhythm point corresponding to the nth video key point is called the nth audio rhythm point. If there are adjacent audio rhythm points before and after the nth audio rhythm point, which are respectively called the n-1th audio rhythm point and the n+1th audio rhythm point, then with the nth video key point as the center, the position located in front of the nth video key point and whose time length is a preset multiple (such as 0.5 times) of the time length between the n-1th audio rhythm point and the nth audio rhythm point is used as the starting clipping position, and the position located behind the nth video key point and whose time length is a preset multiple of the time length between the nth audio rhythm point and the n+1th audio rhythm point is used as the ending clipping position. That is to say, the duration of the video clip corresponding to the nth video key point is: preset multiple * time length between the n-1th audio rhythm point and the nth audio rhythm point + preset multiple * time length between the nth audio rhythm point and the nth audio rhythm point +1 audio rhythm point; if the nth audio rhythm point has only an adjacent audio rhythm point in front of it, that is, the n-1th audio rhythm point, then the position in front of the nth video key point and with a time length that is a preset multiple of the time length between the n-1th audio rhythm point and the nth audio rhythm point will be used as the starting interception position, and the position of the nth video key point will be used as the ending interception position. In other words, the duration of the video clip corresponding to the nth video key point is: preset multiple * the time length between the n-1th audio rhythm point and the nth audio rhythm point. the duration between the nth audio rhythm point and the n+1th audio rhythm point; if the nth audio rhythm point has only an adjacent audio rhythm point behind it, that is, the n+1th audio rhythm point, then the position of the nth video key point is used as the starting interception position, and the position behind the nth video key point and with a time length that is a preset multiple of the duration between the nth audio rhythm point and the n+1th audio rhythm point is used as the ending interception position, that is, the duration of the video clip corresponding to the nth video key point is: preset multiple * the duration between the nth audio rhythm point and the n+1th audio rhythm point.

[0077] In specific applications, the original starting point and the original ending point of the video data may not be involved in the alignment, but serve as the natural boundary range of the video data; the interception position of any video key point shall not exceed the position of the adjacent video key point or the natural boundary range of the video data, but it supports overlapping the left and right interception positions corresponding to two adjacent video key points.

[0078] Considering that the video data obtained in step S201 may contain multiple component segments, it is necessary to determine its natural boundary range based on a single component segment. That is to say, in the process of determining the interception position corresponding to a certain video key point, it is necessary to determine the starting interception position and the ending interception position within the natural boundary range of the component segment corresponding to the video key point.

[0079] Due to the natural boundary range, it may be that one side of the key point of the video cannot be captured for a sufficient duration (i.e., the target duration); due to the upper and lower speed limits of the preset speed range, it may also be that even if the speed is changed, the corresponding target duration cannot be reached. Then the embodiment of the present application may also include: if the duration of any video clip does not meet the corresponding target duration, then the speed multiple is selected in the preset speed range, and the video clip is speed-changed according to the speed multiple; if the duration of the video clip after speed change does not meet the corresponding target duration, then the preset fill-in picture is added to the main track. Those skilled in the art can set the preset fill-in picture according to actual needs, and no specific limitation is made here. For example, the preset fill-in picture can be a black picture.

[0080] For example, if it is found in the process of determining the interception position that the duration between the nth video key point and the n+1th video key point is less than the preset multiple of the duration between the nth audio rhythm point and the n+1th audio rhythm point, then the termination interception position corresponding to the nth video key point can be taken as the midpoint between the nth video key point and the n+1th video key point, and the speed change processing is used to make the length of this segment equal to the preset multiple of the duration between the nth audio rhythm point and the n+1th audio rhythm point; if it is limited by the lower limit speed of the preset speed change range at this time, taking the lower limit speed as 0.1 times, even if the length of this segment is extended 10 times, it cannot be equal to the preset multiple of the duration between the nth audio rhythm point and the n+1th audio rhythm point, then a black picture is added to the final empty part in the main track as a black field placeholder.

[0081] If it is found in the process of determining the cropping position that the nth video key point and the n+1th video key point do not correspond to the same component segment in the video data, then the tail (i.e., the natural boundary) of the component segment corresponding to the nth video key point will be used as the ending cropping position corresponding to the nth video key point.

[0082] If there is at least one component segment in front of the first video key point, it needs to be processed as follows after alignment: if the original starting point of the component segment corresponding to the video key point is before the original starting point of the audio data, the position of the original starting point of the audio data is used as the starting cut position corresponding to the video key point, and is disconnected from the part of the component segment before the original starting point of the audio data; if the original starting point of the component segment corresponding to the video key point is after the original starting point of the audio data, the position of the original starting point of the component segment is used as the starting cut position corresponding to the video key point.

[0083] If the number of video key points is ≤ the number of audio rhythm points, then in the process of determining the cut position corresponding to the last video key point, the ending cut position is the smaller value between the left length and the length of the remaining component segments, which can be expressed as min(left length, remaining component segment length). After the cut, there is no need to add a preset filler picture at the end.

[0084] If the number of video key points is greater than the number of audio rhythm points, then in the process of determining the interception position corresponding to the last valid video key point, the ending interception position is the position of the video key point + min (preset multiple * (position of the next video key point - position of the current video key point), left length), and there is no need to add the preset filler picture at the end after interception.

[0085] Figure 3a Schematic diagram showing the aligned video key points and audio rhythm points and the video clips corresponding to the video key points Figure 1 ,like Figure 3a As shown, the video data contains three segments, segments 1 through 3. The number of video key points is smaller than the number of audio rhythm points. A through E represent each video key point, and X represents the original starting point of the video data. A through k represent each audio rhythm point, and x represents the original starting point of the audio data. After alignment, video key points A through E correspond one-to-one with audio rhythm points a through e, respectively. Among them, the duration between video key point A and video key point B is 3 unit time lengths, expressed as AB=3; the duration between video key point B and video key point C is 2 unit time lengths, expressed as BC=2; the duration between video key point C and video key point D is 4 unit time lengths, expressed as CD=4; the duration between video key point D and video key point E is 1 unit time length, expressed as DE=1; the duration between audio rhythm points a~k is 1 unit time length; the duration between the original starting point X of the video data and the video key point A is 2 unit time lengths, expressed as XA=2; the duration between the original starting point x of the audio data and the audio rhythm point a is 1.5 unit time lengths, expressed as xa=1.5.

[0086] When determining the interception position corresponding to the video key point, the parts of the other component segments other than the component segment corresponding to the video key point are not taken. When the preset multiple is 0.5 times, the duration of the video segment corresponding to the video key point A is expressed as [A], [A] = 0 + 0.5 = 0.5, that is, the duration of the video segment corresponding to the video key point A is 0.5 unit duration, the starting interception position corresponding to the video key point A is the position of the video key point A, and the ending interception position corresponding to the video key point A is the position behind the video key point A and has a duration of 0.5 unit duration; calculate the duration of the video segment corresponding to the video key point B, [B] = 0.5 + 0.5 = 1, that is, the duration of the video segment corresponding to the video key point B is 1 unit of time, the starting interception position corresponding to the video key point B is the position in front of the video key point B and has a duration of 0.5 units of time, and the ending interception position corresponding to the video key point B is the position behind the video key point B and has a duration of 0.5 units of time; similarly, the duration of the video segments corresponding to the video key points C to E is calculated, [C] = 0.5 + 0.5 = 1, [D] = 0.5 + 0.5 = 1, and [E] = 0.5 + 0 = 0.5.

[0087] Figure 3b Schematic diagram showing the aligned video key points and audio rhythm points and the video clips corresponding to the video key points Figure 2 ,like Figure 3bAs shown, the video data contains three segments, segments 1 through 3. The number of video key points is greater than the number of audio rhythm points. A through K represent each video key point, and X represents the original starting point of the video data. A through c represent each audio rhythm point, and x represents the original starting point of the audio data. After alignment, video key points A through C correspond one-to-one with audio rhythm points a through c, respectively. The duration between each video key point A through K is one unit of duration; the duration between audio rhythm point a and audio rhythm point b is four units of duration, represented by ab = 4; the duration between audio rhythm point b and audio rhythm point c is four units of duration, represented by bc = 4; XA = 2, and xa = 3. Then when the preset multiple is 0.5 times, the starting interception position corresponding to the video key point A is the position of the video key point A, and the ending interception position corresponding to the video key point A is the position behind the video key point A and has a duration of 0.5 unit time. However, the duration of the video clip corresponding to the video key point A obtained by intercepting in this way is 0.5 unit time, which does not meet the corresponding target duration (i.e. 0.5*ab). Then 0.25 times the speed is selected in the preset speed range, and the video clip is speed-changed according to 0.25 times the speed. The final duration of the video clip corresponding to the video key point A is expressed as [A], [A]=(0+0.5) / 0.25=2; similarly, [B]=(0.5+0.5) / 0.25=4, [C]=(0.5+0.5) / 0.25=4. At this time, there are 2 units of time (i.e. 4-2) missing, and a black picture can be added as a black field placeholder, such as Figure 3b shown.

[0088] Figure 3c Schematic diagram 3 shows the video key points and audio rhythm points after alignment and the video clips corresponding to the video key points, as shown in FIG. Figure 3c As shown, the number of video key points is greater than the number of audio rhythm points, AB = 0.2, BC = 1.8, the duration between each of the video key points C to K is 1 unit time, ab = 4, bc = 4, XA = 2, xa = 3, the lower limit of the preset speed range is 0.1 times the speed, then [A] = (0 + 0.1) / 0.1 = 1, [B] = (0.1 + 0.2) / 0.1 = 3, [C] = (0.5 + 0.5) / 0.25 = 4, at this time there are 2 (i.e. 4-2) unit time and 2 (i.e. 7-5) unit time missing, and two black pictures can be used as black field placeholders, as shown in FIG. Figure 3c shown.

[0089] Figure 3d Schematic diagram 4 shows the video key points and audio rhythm points after alignment and the video clips corresponding to the video key points, as shown in FIG. Figure 3dAs shown, the number of video key points is greater than the number of audio rhythm points, AB=1.5, BC=1.5, the duration between each video key point C~K is 1 unit of duration, ab=4, bc=4, XA=2, xa=3, then [A]=(1+0.5) / 0.25=6, [B]=(0.5+0.5) / 0.25=4, [C]=(0.5+0.5) / 0.25=4.

[0090] Step S205 , based on the interception position, the video data corresponding to the key point of the video is intercepted to obtain the video segment corresponding to the key point of the video.

[0091] Step S206 , generating a target video based on the multiple video clips and the aligned video key points and audio rhythm points.

[0092] After obtaining the aligned video key points, audio rhythm points, and multiple video clips, the target video can be automatically generated through track laying. Specifically, the audio data is added to the audio track and the start time of the audio data is set. The multiple video clips are added to the main track and the start time of each video clip is set according to the aligned video key points and audio rhythm points.

[0093] To facilitate the creation of high-quality target videos more conveniently, a video production template is provided in the embodiments of this application. Based on this template, it can automatically identify and add materials, restore position scaling, and process various boundary scenes, thereby completing intelligent editing, enriching the display content of the target video, and effectively improving the video quality and presentation effect of the target video. Based on the video production template, each material is added to the corresponding track and the start display time, display duration, and scaling ratio of each material are set.

[0094] Among them, the video production template may include specific settings for special effects, stickers, text titles, transitions, filters, subtitle styles, music and other materials. Figure 4a A schematic diagram of the composition of a video production template is shown, such as Figure 4a As shown, the video production template is a template for a video with a duration of 12 seconds.

[0095] Below Figure 4a The following describes the identification and restoration of materials using a 12-second video template. The logic for identifying and restoring special effects, stickers, and text titles is as follows: materials are categorized into opening, ending, global, and local materials based on their coverage.

[0096] Specifically, for the opening credits, according to the video production template, opening credits materials are added at the beginning of the video, and the number of opening credits materials can be multiple; the opening credits materials are located within 0 to 3 seconds in the video, and there can be a certain error (such as 0.5 seconds); the restoration method is: based on the starting point and the absolute value of the duration of the opening credits material in the video production template (hereinafter referred to as the duration), it is restored.

[0097] For the ending credits, add ending credits materials at the end of the video based on the video production template. The number of ending credits materials can be multiple; the ending credits materials are located within the last 3 seconds of the video and can have a certain error (such as 0.5 seconds); the restoration method is: restore according to the position of the end point of the ending credits material in the video production template and the duration of the video end point.

[0098] For the global, according to the video production template, the global material is flattened to the entire or most of the main track. The number of global materials can be multiple; the starting point of the global material is located within 0 to 3 seconds in the video and the end point of the global material is located within the last 3 seconds of the video. There can be a certain error (such as 0.5 seconds); the restoration method is: restore according to the distance between the start and end timestamps of the global material in the video production template and the end of the video.

[0099] For local parts, local materials are added during the video generation process based on the video production template. The number of local materials can be multiple; materials that are not at the beginning, end, or global can be regarded as local materials; the restoration method is: restore according to the starting point and duration of the local materials in the video production template.

[0100] The logic for text and sticker size and position is as follows: 1) For position, calculate the distance between the center of the material and the bottom edge of the frame, as well as the distance between the center and the left edge of the canvas, and restore them as a percentage of the distance; 2) For size, restore them as a percentage of the material width / canvas width. When the ratio needs to be converted from 16:9 to 9:16, the template information scale ratio needs to be multiplied by (9 / 16). 2 , the zoom ratio can retain the preset number of digits after the decimal point (such as 3 digits); when the ratio needs to be converted from 9:16 to 16:9, the zoom ratio of the template information needs to be multiplied by (16 / 9) 2, the zoom ratio can retain the preset number of decimal places; when the ratio needs to be converted from 16:9 to 4:3, the zoom ratio of the template information needs to be multiplied by 3 / 4, and the zoom ratio can retain the preset number of decimal places; when the ratio needs to be converted from 4:3 to 16:9, the zoom ratio of the template information needs to be multiplied by 4 / 3, and the zoom ratio can retain the preset number of decimal places; when the ratio needs to be converted from 16:9 to 1:1, the zoom ratio of the template information needs to be multiplied by 9 / 16, and the zoom ratio can retain the preset number of decimal places; when the ratio needs to be converted from 1:1 to 16:9, the zoom ratio of the template information needs to be multiplied by 16 / 9, and the zoom ratio can retain the preset number of decimal places.

[0101] For boundary scene processing, the main focus is on judging the duration of the target video to be generated and the size of the opening and ending time areas. For regional materials, when the end point of the material is greater than the end point of the video, the end point of the material can be terminated at the end point of the video; when the duration of the target video to be generated is too short, in order to avoid mutual interference between multiple materials, some processing can be reduced: 1) When the duration of the target video is less than 3s, only the global material in the video production template is used. At this time, the starting point of the global material is 0s, and its end point is the last frame of the video. The opening, ending, and local materials in the video production template are not used; 2) When 3s≤the duration of the target video is less than 6s, only the opening material and global material in the video production template are used; 3) When the duration of the target video is ≥6s, the opening material, ending material, global material, and local material in the video production template can be applied. Figure 4b and Figure 4c Schematic diagrams of the composition of a target video with a duration of 24 seconds and a target video with a duration of 6 seconds are shown respectively.

[0102] For transitions, you can add multiple transitions. The order of adding transitions will be recorded according to the order of video clips, and the transition slots of the video will be filled in sequence according to the transition order. If there is no valid video clip in the target video, no transition can be added. If the target video to be generated is not long enough and cannot be restored according to the transition duration in the video production template, the transition duration can be calculated by multiplying the duration of the shortest video clips on both sides of the target video by 2.

[0103] For filters, when there are multiple filters, only the first filter can be applied; the filter can be applied to all clips.

[0104] For subtitle styles, there is only one subtitle style, and the client needs a field to distinguish whether it is a subtitle style or a text title; the distinction is as follows: 1) When the text is checked as replaceable, it is regarded as a subtitle style. When there are multiple checks, only the subtitle style of the first checked option is selected; 2) When the text is not checked, it is regarded as a text title. The restoration method is: the first checked text is regarded as a subtitle style and will not be added to the target video; if multiple are checked, it is regarded as an invalid check and will be added to the target video as a text title; the text previously added by the user (by adding subtitles or using automatic speech recognition ASR) will be uniformly changed to the subtitle style of the template. If subtitles are not generated using ASR, they will be processed according to the fallback effect and the default subtitle style (such as white background with black stroke, etc.) will be applied.

[0105] For music, when there are multiple pieces of music, only the first piece of music can be used; the music will start playing from the music starting point at the 0th second of the video, and the music will be spread across the entire video. If the music duration is not enough, it will be played to the end of the music, and then looped from the highlight point (or music starting point) timestamp. If the duration of the target video to be generated is too short, there will be a situation where the fade-in duration + fade-out duration > the total video duration. In this case, it is necessary to determine the size of the video duration and the fade-in duration and fade-out duration: 1) If the total video duration ≤ fade-in duration, only the fade-in effect is used, and the fade-in duration is consistent with the video duration; 2) If fade-in duration < total video duration < fade-in duration + fade-out duration, the complete fade-in duration is used, and the fade-out duration uses the remaining video duration.

[0106] According to the video generation method provided by the embodiment of the present application, a multimodal content analysis method is constructed, which can automatically identify the audio rhythm points of audio data and the video key points of video data, automatically align the video key points with the audio rhythm points, realize the deep fusion of audio rhythm and vision, and obtain the effect of adaptive matching of emotional intensity; according to the duration between the audio rhythm point corresponding to the video key point and the adjacent audio rhythm point of the audio rhythm point, the interception position corresponding to the video key point is accurately determined, and combined with multi-rule constraints and boundary processing, the coherence and smoothness of the content of the previous and next pictures are effectively guaranteed, and combined with the speed change processing of the video picture, automatic editing with precise card points is realized, which improves the editing efficiency. It improves editing efficiency and accuracy, generates automatic card-point videos with coordinated visual and auditory effects and clear emotional expression, realizes automated card-point video generation, and improves the emotional expressiveness of the video and the user's viewing experience; and, based on the video production template, it can automatically identify and add materials, restore position scaling, and process various boundary scenes, etc., which greatly enriches the display content of the generated video, improves the video quality and video display effect, and effectively reduces the difficulty of video creation, so that users with weaker professional knowledge can also easily create high-quality card-point videos, effectively improving the efficiency and quality of users' video creation, and can well adapt to the growing demand for video creation.

[0107] Figure 5 FIG. 1 shows a structural block diagram of a video generating device according to an embodiment of the present application. Figure 5 As shown, the apparatus includes: an acquisition module 510 , an extraction module 520 , an alignment module 530 and a generation module 540 .

[0108] The acquisition module 510 is adapted to acquire audio data and video data to be processed.

[0109] The extraction module 520 is adapted to perform modal feature extraction on the audio data and the video data to obtain audio rhythm points of the audio data and video key points of the video data.

[0110] The alignment module 530 is adapted to align video key points with audio rhythm points according to audio rhythm points, and to intercept video data to form a plurality of video segments.

[0111] The generation module 540 is adapted to generate a target video based on a plurality of video clips and the aligned video key points and audio rhythm points.

[0112] Optionally, the alignment module 530 is further adapted to: align the video key points with the audio rhythm points in a sequential order based on the audio rhythm points, and determine the audio rhythm points corresponding to the video key points.

[0113] Optionally, the alignment module 530 is further adapted to: for any video key point having a corresponding audio rhythm point, determine the cut-off position corresponding to the video key point based on the duration between the audio rhythm point corresponding to the video key point and the adjacent audio rhythm point of the audio rhythm point; and cut off the video data corresponding to the video key point based on the cut-off position to obtain the video clip corresponding to the video key point.

[0114] Optionally, the alignment module 530 is further adapted to: if the duration of any video clip does not meet the corresponding target duration, a speed change multiple is selected in a preset speed change range, and the video clip is speed-changed according to the speed change multiple; if the duration of the video clip after speed change does not meet the corresponding target duration, a preset filler picture is added to the main track.

[0115] Optionally, the generation module 540 is further adapted to: add audio data to the audio track and set the start playback time of the audio data; add multiple video clips to the main track and set the start playback time of each video clip based on the aligned video key points and audio rhythm points.

[0116] Optionally, the generation module 540 is further adapted to: add various materials to the corresponding track according to the video production template, and set the start display time, display duration and zoom ratio of each material.

[0117] The description of each module above refers to the corresponding description in the method embodiment and will not be repeated here.

[0118] According to the video generation device provided by the embodiment of the present application, a multimodal content analysis method is constructed, which can automatically identify the audio rhythm points of audio data and the video key points of video data, and automatically align the video key points with the audio rhythm points, thereby realizing a deep fusion of audio rhythm and vision, and obtaining an effect of adaptive matching of emotional intensity; based on the duration between the audio rhythm point corresponding to the video key point and the adjacent audio rhythm point of the audio rhythm point, the interception position corresponding to the video key point is accurately determined, and combined with multi-rule constraints and boundary processing, the coherence and smoothness of the content of the previous and next pictures are effectively guaranteed, and combined with the speed change processing of the video picture, automatic editing with precise card points is realized, which improves the editing efficiency. It improves editing efficiency and accuracy, generates automatic card-point videos with coordinated visual and auditory effects and clear emotional expression, realizes automated card-point video generation, and improves the emotional expressiveness of the video and the user's viewing experience; and, based on the video production template, it can automatically identify and add materials, restore position scaling, and process various boundary scenes, etc., which greatly enriches the display content of the generated video, improves the video quality and video display effect, and effectively reduces the difficulty of video creation, so that users with weaker professional knowledge can also easily create high-quality card-point videos, effectively improving the efficiency and quality of users' video creation, and can well adapt to the growing demand for video creation.

[0119] An embodiment of the present application provides a non-volatile computer storage medium, which stores at least one executable instruction or computer program. The executable instruction or computer program can enable a processor to perform operations corresponding to the video generation method in any of the above method embodiments.

[0120] An embodiment of the present application provides a computer program product, which includes at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to perform operations corresponding to the video generation method in any of the above method embodiments.

[0121] Figure 6 A schematic structural diagram of a computing device according to an embodiment of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the computing device.

[0122] like Figure 6 As shown, the computing device may include: a processor (processor) 602 , a communications interface (Communications Interface) 604 , a memory (memory) 606 , and a communication bus 608 .

[0123] Processor 602, communication interface 604, and memory 606 communicate with each other via communication bus 608. Communication interface 604 is used to communicate with other devices, such as clients or other server network elements. Processor 602 is used to execute program 610, which may specifically perform the steps described in the aforementioned embodiment of the video generation method for a computing device.

[0124] Specifically, the program 610 may include program codes, which include computer operation instructions.

[0125] Processor 602 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0126] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0127] Program 610 can be specifically configured to cause processor 602 to execute the video generation method of any of the aforementioned method embodiments. The specific implementation of each step in program 610 can be found in the corresponding descriptions of the corresponding steps and units in the aforementioned video generation embodiments and will not be repeated here. Those skilled in the art will clearly understand that, for ease and brevity of description, the specific operating processes of the devices and modules described above can refer to the corresponding process descriptions in the aforementioned method embodiments and will not be repeated here.

[0128] The algorithm and display provided herein are not inherently relevant to any particular computer, virtual system or other device. Various general-purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the embodiments of the present application are not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the embodiments of the present application described herein, and the description of the specific languages ​​above is for the purpose of disclosing the best mode of implementation of the embodiments of the present application.

[0129] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0130] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed embodiments of the present application require more features than the features explicitly recited in each claim. More precisely, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the embodiments of the present application.

[0131] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0132] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features that are included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0133] The various component embodiments of the embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the embodiments of the present application. The embodiments of the present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for performing a part or all of the methods described herein. Such a program implementing the embodiments of the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0134] It should be noted that the above embodiments illustrate rather than limit the embodiments of the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbols placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The embodiments of the present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A video generation method, comprising: Obtaining audio data and video data to be processed; Performing modal feature extraction on the audio data and the video data to obtain audio rhythm points of the audio data and video key points of the video data; According to the audio rhythm point, aligning the video key point with the audio rhythm point, and intercepting the video data to form a plurality of video clips; Generate a target video based on multiple video clips and aligned video key points and audio rhythm points.

2. According to the method of claim 1, aligning the video key point with the audio rhythm point according to the audio rhythm point further comprises: Based on the audio rhythm point, the video key point is aligned with the audio rhythm point in a sequential order to determine the audio rhythm point corresponding to the video key point.

3. The method according to claim 1, wherein intercepting the video data to form a plurality of video segments further comprises: For any video key point having a corresponding audio rhythm point, determining a cut position corresponding to the video key point according to the duration between the audio rhythm point corresponding to the video key point and an adjacent audio rhythm point of the audio rhythm point; According to the interception position, the video data corresponding to the key point of the video is intercepted to obtain the video segment corresponding to the key point of the video.

4. The method according to any one of claims 1 to 3, further comprising: If the duration of any video segment does not meet the corresponding target duration, a speed change multiple is selected in a preset speed change range, and the video segment is speed-changed according to the speed change multiple; If the duration of the video clip after the speed change does not meet the corresponding target duration, the preset fill-in picture is added to the main track.

5. According to the method according to any one of claims 1 to 4, generating a target video based on the plurality of video clips and the aligned video key points and audio rhythm points further comprises: Add the audio data to the audio track and set the start time of playing the audio data; Add multiple video clips to the master track and set the start time of each video clip based on the aligned video key points and audio rhythm points.

6. The method according to claim 5, further comprising: According to the video production template, add each material to the corresponding track, and set the start display time, display duration and zoom ratio of each material.

7. A video generating device, comprising: An acquisition module, adapted to acquire audio data and video data to be processed; An extraction module, adapted to extract modal features from the audio data and the video data to obtain audio rhythm points of the audio data and video key points of the video data; An alignment module, adapted to align the video key points with the audio rhythm points according to the audio rhythm points, and intercept the video data to form a plurality of video clips; The generation module is suitable for generating a target video based on multiple video clips and aligned video key points and audio rhythm points.

8. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the video generation method according to any one of claims 1 to 6.

9. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to execute an operation corresponding to the video generation method according to any one of claims 1 to 6.

10. A computer program product, comprising at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the video generation method according to any one of claims 1 to 6.

Images