Music short video generation method and device

Abstract

The application provides a music short video generation method and device, the method comprising: obtaining a to-be-edited movie and target music; selecting a target shot from the to-be-edited movie based on the emotional consistency score of each shot of the to-be-edited movie and the target music, the constraint of the time length of each shot, and the constraint of the number of movie scenes; and generating a target music short video based on the target shot and the target music. The application can quickly output a music short video meeting the editing standard by calculating the emotional similarity of each shot and the whole music and optimizing the three conditions of emotional consistency, story integrity, and rhythm synchronization.

Description

Technical Field

[0001] This invention relates to the field of audio and video processing technology, and in particular to a method and apparatus for generating short videos with background music. Background Technology

[0002] In recent years, with the rapid development of social networks and mobile applications, the production of high-quality short videos and montage videos has become increasingly popular and especially important. Movies, as one of the best resources for montage, are often edited into shorter versions and accompanied by background music to create trailers, previews, and highlights of the original film.

[0003] However, existing short video editing tools typically rely on users manually selecting shots from movies and aligning them with music, which is tedious and time-consuming. With the emergence and development of machine learning technology, several methods for automatically generating montages have been proposed in recent years, such as video summarization and emotion-oriented music video generation. However, the former primarily focuses on the content of the video itself, ignoring its relevance to the input music, while the latter faces many difficulties in understanding and processing long videos. Summary of the Invention

[0004] This invention provides a method and apparatus for generating short videos with background music, which addresses the shortcomings of existing technologies where non-professional users are insensitive to the relationship between music rhythm and emotion during video editing and cannot pay attention to the integrity of the story. It can quickly generate short videos with matching music emotion and rhythm that are relatively complete and understandable.

[0005] This invention provides a method for generating short videos with background music, comprising:

[0006] Obtain the movie to be edited and the target music;

[0007] Based on the emotional consistency score of each shot in the film to be edited and the target music, the constraints on the duration of each shot and the number of film scenes, target shots are selected from the film to be edited.

[0008] Based on the target shot and the target music, generate a short video with the target background music.

[0009] According to a method for generating short videos with background music provided by the present invention, determining the emotional consistency score between each shot of the film to be edited and the target music includes:

[0010] Extract the shot features of the film to be edited and the musical features of the target music;

[0011] Based on the correlation between the shot features and the music features in any emotion classification, an emotional consistency score is obtained for each shot and the target music.

[0012] According to a method for generating short videos with background music provided by the present invention, the step of extracting the shot features of the film to be edited and the musical features of the target music includes:

[0013] The shot features of the film to be edited and the musical features of the target music are extracted using a multimodal emotion space.

[0014] The multimodal emotion space is obtained by training image samples and audio samples as a training set and using the emotion classifications corresponding to the image samples and audio samples as a label set.

[0015] According to a method for generating short videos with background music provided by the present invention, the step of selecting target shots from the film to be edited based on the emotional consistency score of each shot of the film to be edited and the target music, the constraint of the duration of each shot, and the constraint of the number of film scenes includes:

[0016] Based on the highest emotional consistency score between each shot of the film to be edited and the target music, an optimization objective function is constructed;

[0017] Based on the maximum constraint of the duration of each shot and the maximum constraint of the number of movie scenes, the constraints of the optimization objective function are determined.

[0018] Based on the constraints, the optimization objective function is solved to obtain the target shot in the movie to be edited.

[0019] According to a method for generating short videos with background music provided by the present invention, the knapsack algorithm is used to solve the optimization objective function based on the constraints to obtain the target shot in the movie to be edited.

[0020] According to the present invention, a method for generating short videos with background music is provided, wherein the method employs a knapsack algorithm to solve the optimization objective function based on the constraints to obtain the target shot in the movie to be edited, comprising:

[0021] Based on the emotional consistency score of each shot and the target music and the average emotional score of the scene in each shot, a weighted emotional score for each shot is obtained.

[0022] Based on the average duration of each measure in the target music and the continuous shot duration of each shot, the discrete shot duration of each shot is determined;

[0023] The weighted emotion score of each shot, the discrete shot duration of each shot, and the number of movie scenes are each used as individual factors in the knapsack algorithm to solve the optimization objective function, thereby obtaining the target shot in the movie to be edited.

[0024] The present invention also provides a short video generation device, comprising:

[0025] The acquisition module is used to acquire the movie to be edited and the target music.

[0026] The shot selection module is used to select target shots from the movie to be edited based on the emotional consistency score of each shot in the movie to be edited and the target music, the constraint of the duration of each shot and the constraint of the number of movie scenes;

[0027] The output module is used to generate a short video with target background music based on the target shot and the target music.

[0028] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the short video with background music generation method described above.

[0029] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the short video with background music generation method as described above.

[0030] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the short video with background music generation method described above.

[0031] The method and apparatus for generating short videos with background music provided by this invention optimize the final short video by calculating the emotional similarity between each shot and the overall music, and combining this with three conditions: emotional consistency, story integrity, and rhythm synchronization. It can achieve automatic editing, improving the generation speed and quality of short videos with background music. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0033] Figure 1 This is a flowchart illustrating the method for generating short videos with background music provided in an embodiment of the present invention;

[0034] Figure 2 This is a schematic diagram of the music-based short video generation framework provided in an embodiment of the present invention;

[0035] Figure 3 This is one of the schematic diagrams of the knapsack algorithm solution provided in the embodiments of the present invention;

[0036] Figure 4 This is the second schematic diagram of the knapsack algorithm solution provided in the embodiment of the present invention;

[0037] Figure 5 This is a schematic diagram of the structure of the short video generation device with background music provided in an embodiment of the present invention;

[0038] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0040] One of the most renowned film editors, Walter Scott Murch, summarized and emphasized the "Six Rules" of film editing, including emotion, story, rhythm, eye movement, 2D screen plane, and 3D action space, each with varying degrees of importance to the final cut. Of these six elements, emotion is the most important, with an importance factor of 51%, while story and rhythm account for 23% and 10%, respectively.

[0041] Inspired by Walter Scott Murch's "Six Rules," this invention proposes a method for generating short videos with background music. This method is a way to automatically generate montages from an input movie using user-specified music as guidance.

[0042] Specifically, this invention combines the three most important elements of film editing to construct the output montage, in order to meet the following requirements:

[0043] Emotional consistency. The shots used to synthesize the output montage are emotionally consistent with the input music.

[0044] Story integrity. Montage needs to present a relatively complete and understandable story, rather than executing a trivial splicing of shots.

[0045] Synchronized rhythm. The visual and audio content of a montage should have a synchronized rhythm.

[0046] In this embodiment of the invention, the short video with background music is a film montage.

[0047] The following is combined Figures 1-3 The method for generating short videos with background music according to embodiments of the present invention is described, such as... Figure 1 As shown, the method includes at least the following sub-steps:

[0048] Step 101: Obtain the movie to be edited and the target music;

[0049] Step 102: Based on the emotional consistency score of each shot and target music in the film to be edited, the constraints of the duration of each shot and the number of film scenes, select the target shot from the film to be edited;

[0050] Step 103: Generate a short video with target background music based on the target shot and target music.

[0051] Regarding step 101, it should be noted that this embodiment requires preprocessing of the input film to be edited and the target music. For the input target music x... m In this embodiment, the music is divided into a series of measures B = {b1, b2, ..., b...} l}. For the input movie x v In this embodiment, a scene set ε = {e1, e2, ..., e} is defined. m} and a set of shots S = {s1, s2, ..., s} n}, and use the mapping function τ(s) i ) = e j This is used to represent the relationship between scenes and shots.

[0052] Regarding step 102, it should be noted that the goal of montage generation is to select a subset R from the set of shots selected from S and associate the film shots with the music bars. That is to say, montage generation requires the following two conditions: (1) a shot indicator function. To determine which shots are selected, (2) a mapping function Φ(b k ) = s i To represent the relationship between shots and musical measures. This task is often an under-constrained problem, therefore additional constraints are needed. Constrain the feasible solution.

[0053] This invention adds an emotional perception constraint to the shot selection task by calculating an emotional consistency score to evaluate the consistency of each shot with the overall music. By setting an optimization objective, the selected subset of shots R can be used to construct a montage by maximizing the emotional consistency between audio and visual signals. The constraint on the duration of each shot corresponds to the rhythmic synchronization between the shot and the music measure, while the constraint on the number of film scenes corresponds to the total number of selected scenes. Considering these two constraints effectively ensures the story integrity and rhythmic synchronization of the edited video.

[0054] Regarding step 103, it should be noted that in this embodiment, after obtaining the portion of the target shot that is longer than the music measure to align the duration, all selected shots are spliced ​​together in chronological order, and the given target music is added to obtain the final short video with background music.

[0055] The method for generating short videos with background music according to embodiments of the present invention can generate montage videos that match the emotions and rhythm of the music and are relatively complete and understandable by editing the input movie according to the specified music, thereby improving the efficiency of video editing.

[0056] Understandably, determining the emotional consistency score between each shot of the film to be edited and the target music includes:

[0057] Extract the shot features of the film to be edited and the musical features of the target music;

[0058] Based on the correlation between shot features and music features in any emotion classification, an emotional consistency score is obtained for each shot and the target music.

[0059] It should be noted that, since the emotional consistency score in this embodiment of the invention is obtained by learning the emotional metric function between film shots and music... Therefore, the embodiments of the present invention first require constructing a network based on the AudioCLIP framework and contrastive learning. This involves embedding and aligning signals of different modes in the sentiment space, aligning the multimodal signals of music, sentiment-classified text, and images within the sentiment space. Contrastive learning and classification loss are used to optimize the joint audiovisual representation. The purpose of incorporating textual modality is to serve as an anchor point to improve classification accuracy, and the pre-trained AudioCLIP framework is used to further align the feature space to imbue it with sentiment perception.

[0060] This invention measures the correlation of different modal signals across various emotion categories by defining different emotional indices. Since the emotions of audio and visual signals can be measured together in a two-dimensional valence space, this provides a reasonable indicator for comparing the differences between the two modalities. Therefore, the emotion space is divided into six categories, each covering the emotion space in everyday communication: excitement, fear, neutrality, relaxation, sadness, and tension.

[0061] Specifically, S is the emotional consistency score, used to assess the distance between signal features of different patterns, defined as shown in Equation 1 below:

[0062]

[0063] Here, a and b represent the characteristic representations of two different modes of signals.

[0064] Understandably, this involves extracting the shot features of the film to be edited and the musical features of the target music, including:

[0065] Extracting shot features from the film to be edited and musical features from the target music using a multimodal emotional space;

[0066] The multimodal emotion space is obtained by training image and audio samples and using the corresponding emotion classifications of the image and audio samples as a label set.

[0067] It should be noted that the multimodal emotion space includes three encoders and two fully connected layers, and its training process includes:

[0068] First, encoders for three different modes (music, text, and image) were constructed. To generate a matching representation, the input is a tuple (x) consisting of music, text, and images. m ,x t ,x i ), each using a different encoder to obtain a set of latent representations (f m ,f t ,f i For each iteration of the network's training phase, three 6x6 modality constraint matrices were constructed. Taking the image-music format as an example, if a music feature and an image feature have the same sentiment indicator, they are represented as a pairwise (f...) m+ ,f i+ ), or represented as negative pair f i- and f m- This embodiment uses cross-entropy loss L. CE This is used to drive the convergence of the sentiment space between two specific patterns on the diagonal of the matrix. The specific details of the image-music contrast loss are shown in Equation 2 below:

[0069] L image-music =L CE (S (i+,m+) ,1)+L CE (S (i+,m-) ,0) Equation 2+L CE (S (i-,m+) ,0)

[0070] Here, 1 represents a complete vector of 1s, and 0 represents a zero vector. Then, the loss L for the text-image modality and the text-music modality was calculated using the same method. text-image and L text-music .

[0071] Additionally, it should be noted that, to further improve the discriminative power of emotional features, a fully connected layer was added after both the image and music encoders. and This classification of emotion categories enhances the linearity of the potential emotion space. Here, textual information serves as label C. text This influences the construction of the feature space. These classification layers use cross-entropy loss L... CE Learn to distinguish the emotional categories of images and music.

[0072] Therefore, the final classification constraint is expressed as shown in Equation 3:

[0073]

[0074] The total loss of the multimodal affective space is shown in Equation 4:

[0075] L total =L image-music +L text-image +L text-music +αL cls Formula 4

[0076] Here, α is a parameter that balances different loss terms.

[0077] The method for generating short videos with background music in this invention constructs an emotion alignment space, enabling the task of generating film montages to be achieved using artificial intelligence algorithms, thereby improving the speed of algorithm processing.

[0078] Understandably, based on the emotional consistency score of each shot and target music in the film to be edited, constraints on the duration of each shot, and constraints on the number of film scenes, target shots are selected from the film to be edited, including:

[0079] Based on the highest emotional consistency score between each shot of the film to be edited and the target music, an optimization objective function is constructed;

[0080] Based on the maximum constraint of the duration of each shot and the maximum constraint of the number of movie scenes, the constraints of the optimization objective function are determined.

[0081] The objective function is optimized based on the constraints to obtain the target shot in the movie to be edited.

[0082] It should be noted that this embodiment of the invention introduces an emotion measurement function. To evaluate the consistency of each shot with the overall music. Through Optimization goals can be set so that a selected subset of shots R can construct a montage by maximizing the emotional consistency between audio and visual signals.

[0083] When working with the film to be edited, use the trained music encoder. Extract input music x m Features, using a trained image encoder Extracting shot features: Since each frame of a single shot is similar, select the middle frame x of the shot. i This represents the content of a single shot. Then, based on the multimodal sentiment space, a set of sentiment consistency scores is calculated for each shot and the entire target music, forming a set of sentiment scores. As The original value.

[0084] The next step is to select the candidate shot that generates the highest emotional score, i.e., the highest rating. Based on the aforementioned three criteria, this embodiment of the invention uses two constraints as... To constrain the solution space, these constraints can be used to obtain the lens indicator function during the optimization process.

[0085] (1) Improve the causality of montage based on the story integrity constraints of the scene;

[0086] (2) Based on the audiovisual rhythm synchronization constraint of the shot, ensure the audiovisual harmony of the montage.

[0087] Specifically, regarding constraint (1): Since the fewer changes in characters and environment, the easier it is for the audience to understand the plot, a higher scene aggregation can provide better story integrity by limiting the number of scenes involved.

[0088] Therefore, this embodiment of the invention defines a function. To indicate whether a scene is selected, as shown in Equation 5:

[0089]

[0090] When setting constraint condition (1) simultaneously, the embodiment of the present invention will N e Indicates the maximum number of selected scenes, and uses it as... The upper bound of the sum is represented as:

[0091] Regarding constraint (2): Based on experience, it is found that when the montage shots and the music rhythm change synchronously, the audience will feel more harmonious. The music rhythm is defined as the duration of a measure. This rhythm synchronization constraint can be modeled by establishing a mapping relationship between shots and measures.

[0092] Specifically, each musical measure must correspond to one shot, and each shot must contain at least one complete musical measure. To achieve this, the duration of shots and measures is first quantified. Since the duration of musical measures varies relatively little, the average duration of consecutive measures is taken. As a unit of discrete time, it is denoted as Then, for each shot in the film, the duration of the consecutive shots is... and Divide by integer to obtain the duration of discrete shots. Furthermore, it is required that the sum of the selected discrete shot durations equals the total duration N of all discrete musical measures. b The sum is shown in Equation 6:

[0093]

[0094] Therefore, the final objective function is shown in Equation 7:

[0095]

[0096] The constraints are shown in Equation 8:

[0097]

[0098]

[0099] Understandably, the knapsack algorithm is used to solve the objective function based on constraints to obtain the target shot in the movie to be edited.

[0100] It should be noted that, in order to solve the above optimization problem, this embodiment of the invention designs an emotion-aware multidimensional knapsack solver, that is, to find the optimal solution using the knapsack algorithm. This embodiment of the invention proposes two knapsack algorithms for solving the problem.

[0101] like Figure 3 The diagram illustrates the hard-scene-constrained knapsack problem-solving method. This method represents the basic state as (i,j,k,l), indicating that exactly k scenes have been selected, the sum of the discrete durations of the shots is j, and iterates through the first i shots to find the maximum emotion score. l represents whether the scene containing the i-th shot has been selected. Consideration is given to whether the current state is on the scene boundary (q...). i ≠q i-1 This requires discussing four possible state transition paths separately. When a user queries a specific scenario, the upper limit N of the number of such queries needs to be considered. e At that time, the maximum value P of the emotion score can be quickly retrieved. Meanwhile, the Backtrack(·) method serves as a camera indicator function. By tracing the legal path in reverse, a set of possible shot indices R is returned. Then, the selected shots are matched with the music bars in chronological order to obtain the mapping function Φ(b). k ).

[0102] like Figure 4 The diagram illustrates the knapsack problem with soft scene constraints. A fixed number of scenes may prevent obtaining the highest emotional score. Therefore, the restrictions in constraint (1) are relaxed; instead of limiting the upper limit of the total number of selected scenes, the constraint is that the sum of the reciprocals of the number of shots in the scenes to which the selected scenes belong is less than N. eAs shown in Equation 9:

[0103]

[0104] Then, the soft scene number constraint score set is reconstructed as Q = {q1,q2,…,q}. n}, where each item in the set is represented as:

[0105] In this scenario, the hard-scene-constrained knapsack problem degenerates into a standard 3D knapsack solver. Assume a base state (i,j,k) that stores the maximum emotion score when traversing the i-th shot, constrained by the sum of the selected scene weight j and the duration k of the selected discrete shots. During optimization, state (i,j,k) will access all... We determine the state and select the maximum value for transition. We obtain the same result as the hard knapsack algorithm described above. We then search for the optimal solution. After the function, discard the parts of the shot that are longer than the music measure to align the duration, stitch all the selected shots together in chronological order, and then according to Φ(b k Add the given music to get the final montage video.

[0106] The embodiments of the present invention provide two deterministic optimization schemes based on a knapsack problem, which can select shots with high emotional relevance from a large number of candidate shots. The selected shots provide a reasonable and understandable summary of the original content or story, and output a high-quality film montage.

[0107] Understandably, the knapsack algorithm is used to solve the objective function based on constraints, thereby obtaining the target shots in the movie to be edited, including:

[0108] Based on the emotional consistency score of each shot and the target music, and the average emotional score of each shot scene, a weighted emotional score for each shot is obtained.

[0109] Based on the average duration of each measure in the target music and the continuous shot duration of each shot, determine the discrete shot duration of each shot;

[0110] The weighted emotion score of each shot, the discrete shot duration of each shot, and the number of movie scenes are each used as individual factors in the knapsack algorithm to solve the optimization objective function and obtain the target shot in the movie to be edited.

[0111] It should be noted that, before executing the knapsack algorithm, this embodiment of the invention needs to formulate three parameters belonging to the lens s based on the optimization objective function and constraints. i Attributes.

[0112] The first attribute is the weighted sentiment score p. i To further emphasize the importance of the scenario, for each The values ​​are adjusted by adding the average emotion score of the scene to which the shot belongs, forming a weighted emotion score set P = {p1, p2, ..., p...} n Each term in P is expressed as shown in Equation 10:

[0113]

[0114] in, It is a Boolean indicator function that returns 1 if the condition (〃) is true, and 0 otherwise.

[0115] The second attribute is discrete shot length. This is used to ensure the corresponding constraint (2). After traversing each term, the discrete shot duration set is obtained.

[0116] The third attribute is the scene number constraint score q. i Corresponding constraint (1). Define a step function based on the subscript of the scene to which the shot belongs, and use it to classify different scene categories to obtain q. i =j,e j =τ(s) i Let Q = {q1, q2, ..., q} n} represents the set of scene number constraints.

[0117] In the knapsack problem, these three attribute sets are treated as individual factors.

[0118] like Figure 2 As shown, the embodiments of the present invention employ a two-stage framework. In the first stage, as... Figure 2 (a) constructs a CLIP-based and contrastive learning-based network to align multimodal signals from music, text, and images used for training in the emotion space. In the second stage, as... Figure 2 (b) The task of synthesizing montage is set as an optimization problem, and a knapsack-based solver is used to generate the output short video with background music.

[0119] This invention divides the input film at both the scene and shot levels, ensuring the story within the montage is comprehensible by imposing restrictions on the number of selected scenes. Furthermore, quantized durations are used to align selected shots with bars of the input music to synchronize the rhythm of the visual and audio signals.

[0120] To evaluate the embodiments of the present invention, a user survey was designed to compare them with other methods and expert-edited videos. Since the proposed framework is the first to implement music-driven film montage, comparable methods and open-source code are lacking. Lin et al.'s work is most similar to this embodiment; it first recommends a matching piece of music from a fixed music database based on the user-provided video, and then selects shots under cost constraints to obtain the final montage. Although the inputs are not exactly the same, the overall system output is the same as in this embodiment; therefore, this embodiment uses it as a baseline. To further demonstrate the effectiveness of the method in this embodiment, experts were invited to edit the montage under the same conditions.

[0121] In this study, this embodiment produced montages ranging from 3 to 5 minutes in length, each accompanied by music from a user-specified 15-minute video. Finally, this embodiment produced five montages and invited 40 researchers to participate in the experiment. These participants received the same questions as the user study. They voted each montage from 1 to 5 on aspects such as emotional consistency, narrative completeness, pacing, and overall quality. Table 1 presents the results of this embodiment, which achieved a significant advantage over Lin et al. across all evaluation metrics. In particular, because this embodiment explicitly considered the influence of film editing, it achieved substantial improvements in narrative completeness and pacing, even achieving scores comparable to expert results.

[0122] Table 1

[0123]

[0124] The short video generation apparatus provided by the present invention is described below. The short video generation apparatus described below and the short video generation method described above can be referred to in correspondence.

[0125] like Figure 5 As shown, an embodiment of the present invention discloses a short video generation device, comprising:

[0126] Acquisition module 501 is used to acquire the movie to be edited and the target music;

[0127] The shot selection module 502 is used to select target shots from the movie to be edited based on the emotional consistency score of each shot and the target music, the constraint of the duration of each shot and the constraint of the number of movie scenes.

[0128] Output module 503 is used to generate a short video with target background music based on the target shot and target music.

[0129] The short video generation device of this invention optimizes the final background music short video by calculating the emotional similarity of each shot and the overall music, and combining this with three conditions: emotional consistency, story integrity, and rhythm synchronization. It can achieve automatic editing, improving the generation speed and quality of background music short videos.

[0130] Understandably, the shot selection module determines the emotional consistency score between each shot in the film to be edited and the target music, including:

[0131] Extract the shot features of the film to be edited and the musical features of the target music;

[0132] Based on the correlation between shot features and music features in any emotion classification, an emotional consistency score is obtained for each shot and the target music.

[0133] Understandably, the shot selection module extracts shot features from the film to be edited and musical features from the target music, including:

[0134] Extracting shot features from the film to be edited and musical features from the target music using a multimodal emotional space;

[0135] The multimodal emotion space is obtained by training image and audio samples and using the corresponding emotion classifications of the image and audio samples as a label set.

[0136] Understandably, the shot selection module selects target shots from the film to be edited based on the emotional consistency score between each shot and the target music, constraints on the duration of each shot, and constraints on the number of film scenes. These include:

[0137] Based on the highest emotional consistency score between each shot of the film to be edited and the target music, an optimization objective function is constructed;

[0138] Based on the maximum constraint of the duration of each shot and the maximum constraint of the number of movie scenes, the constraints of the optimization objective function are determined.

[0139] The objective function is optimized based on the constraints to obtain the target shot in the movie to be edited.

[0140] Understandably, the shot selection module uses a knapsack algorithm to solve the objective function based on constraints, thereby obtaining the target shot in the movie to be edited.

[0141] Understandably, the shot selection module uses a knapsack algorithm to solve for and optimize the objective function based on constraints, thereby obtaining the target shots in the film to be edited, including:

[0142] Based on the emotional consistency score of each shot and the target music, and the average emotional score of each shot scene, a weighted emotional score for each shot is obtained.

[0143] Based on the average duration of each measure in the target music and the continuous shot duration of each shot, determine the discrete shot duration of each shot;

[0144] The weighted emotion score of each shot, the discrete shot duration of each shot, and the number of movie scenes are each used as individual factors in the knapsack algorithm to solve the optimization objective function and obtain the target shot in the movie to be edited.

[0145] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a method for generating short videos with background music, the method including:

[0146] Obtain the movie to be edited and the target music;

[0147] Based on the emotional consistency score of each shot and target music in the film to be edited, the constraints of the duration of each shot and the number of film scenes, target shots are selected from the film to be edited.

[0148] Generate a short video with target background music based on the target shot and target music.

[0149] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0150] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the music-based short video generation method provided by the above methods, the method including:

[0151] Obtain the movie to be edited and the target music;

[0152] Based on the emotional consistency score of each shot and target music in the film to be edited, the constraints of the duration of each shot and the number of film scenes, target shots are selected from the film to be edited.

[0153] Generate a short video with target background music based on the target shot and target music.

[0154] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the music-based short video generation method provided by the methods described above, the method comprising:

[0155] Obtain the movie to be edited and the target music;

[0156] Based on the emotional consistency score of each shot and target music in the film to be edited, the constraints of the duration of each shot and the number of film scenes, target shots are selected from the film to be edited.

[0157] Generate a short video with target background music based on the target shot and target music.

[0158] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0159] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating short videos with background music, characterized in that, include: Obtain the movie to be edited and the target music; Based on the emotional consistency score of each shot in the film to be edited and the target music, the constraints on the duration of each shot and the number of film scenes, target shots are selected from the film to be edited. Based on the target shot and the target music, generate a short video with the target background music; The step of selecting target shots from the film to be edited, based on the emotional consistency score of each shot in the film to be edited and the target music, constraints on the duration of each shot, and constraints on the number of film scenes, includes: Based on the highest emotional consistency score between each shot of the film to be edited and the target music, an optimization objective function is constructed; Based on the maximum constraint of the duration of each shot and the maximum constraint of the number of movie scenes, the constraints of the optimization objective function are determined. The knapsack algorithm is used to solve the optimization objective function based on the constraints to obtain the target shot in the movie to be edited. The step of using the knapsack algorithm to solve the optimization objective function based on the constraints to obtain the target shot in the movie to be edited includes: Based on the emotional consistency score of each shot and the target music and the average emotional score of the scene in each shot, a weighted emotional score for each shot is obtained. Based on the average duration of each measure in the target music and the continuous shot duration of each shot, the discrete shot duration of each shot is determined; The weighted emotion score of each shot, the discrete shot duration of each shot, and the number of movie scenes are each used as individual factors in the knapsack algorithm to solve the optimization objective function, thereby obtaining the target shot in the movie to be edited.

2. The method for generating short videos with background music according to claim 1, characterized in that, Determine the emotional consistency score between each shot of the film to be edited and the target music, including: Extract the shot features of the film to be edited and the musical features of the target music; Based on the correlation between the shot features and the music features in any emotion classification, an emotional consistency score is obtained for each shot and the target music.

3. The method for generating short videos with background music according to claim 2, characterized in that, The extraction of shot features from the film to be edited and musical features from the target music includes: The shot features of the film to be edited and the musical features of the target music are extracted using a multimodal emotion space. The multimodal emotion space is obtained by training image samples and audio samples as a training set and using the emotion classifications corresponding to the image samples and audio samples as a label set.

4. A short video generation device, characterized in that, include: The acquisition module is used to acquire the movie to be edited and the target music. The shot selection module is used to select target shots from the movie to be edited based on the emotional consistency score of each shot in the movie to be edited and the target music, the constraint of the duration of each shot and the constraint of the number of movie scenes; The output module is used to generate a short video with target background music based on the target shot and the target music. The shot selection module is used to select target shots from the film to be edited based on the emotional consistency score of each shot in the film to be edited and the target music, the constraint of the duration of each shot, and the constraint of the number of film scenes, including: Based on the highest emotional consistency score between each shot of the film to be edited and the target music, an optimization objective function is constructed; Based on the maximum constraint of the duration of each shot and the maximum constraint of the number of movie scenes, the constraints of the optimization objective function are determined. The knapsack algorithm is used to solve the optimization objective function based on the constraints to obtain the target shot in the movie to be edited. The step of using the knapsack algorithm to solve the optimization objective function based on the constraints to obtain the target shot in the movie to be edited includes: Based on the emotional consistency score of each shot and the target music and the average emotional score of the scene in each shot, a weighted emotional score for each shot is obtained. Based on the average duration of each measure in the target music and the continuous shot duration of each shot, the discrete shot duration of each shot is determined; The weighted emotion score of each shot, the discrete shot duration of each shot, and the number of movie scenes are each used as individual factors in the knapsack algorithm to solve the optimization objective function, thereby obtaining the target shot in the movie to be edited.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the short video with background music generation method as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for generating short videos with background music as described in any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for generating short videos with background music as described in any one of claims 1 to 3.

Images