Lens assembling method based on video auxiliary editing

Abstract

The invention discloses a shot assembling method based on video auxiliary editing, and relates to the technical field of media asset content creation, and the method comprises the steps: firstly segmenting a video material through employing a transition recognition technology, and obtaining a candidate shot group; secondly, labeling the scene and the motion grammar of the candidate lens group by using a scene recognition model and a motion grammar recognition model to obtain a labeled candidate lens group, and then representing a slice sequence by using a one-hot matrix based on the labeled candidate lens group; and finally, calculating an energy value of the fragment sequence based on an extensible scene motion grammar combination energy model, gradually adjusting and optimizing the fragment sequence by adopting Langevin dynamics, and obtaining an optimal lens assembly sequence to form a fragment. The method and the device can assist video editing of news manuscripts, improve the production efficiency of news videos, improve the timeliness of news release, and improve the video editing efficiency of the news manuscripts. And meanwhile, the quality of the film can be better controlled, the film is ensured to accord with certain audio-visual language knowledge, and the watching experience of audiences is improved.

Description

Technical Field

[0001] The present application relates to the technical field of media content creation, and in particular to a shot assembly method based on video-assisted editing. Background Art

[0002] With the advent of the new media era, the forms of news reporting have gradually diversified, among which video has become an irreversible trend. On the one hand, through video and short video news reports, traditional media can broaden its dissemination channels and enhance social influence; on the other hand, this form of news content can more quickly and intuitively meet the public's information needs, improving the efficiency and effectiveness of information dissemination.

[0003] However, the creation process of traditional news videos is cumbersome and time-consuming. After the video material is shot, it needs to be carefully screened and matched with the text, and finally the material is integrated and edited in professional editing software. In this process, editors must not only pay attention to the quality of the material, but also ensure that the video content is highly consistent with the text description. At the same time, they also need to strictly follow the basic principles of audio-visual language, such as reasonably arranging the scene conversion of front and back shots, the change of dynamic elements, and color matching. This traditional manual post-production method not only places extremely high demands on the professional skills of editors, but also may affect the immediacy and freshness of the news due to its time-consuming and labor-intensive nature.

[0004] Therefore, how to simplify the production process of news videos, improve production efficiency, and at the same time ensure the professionalism and attractiveness of video content has become one of the technical problems that need to be solved urgently. Summary of the invention

[0005] The purpose of this application is to provide a shot assembly method based on video-assisted editing in order to overcome the existing technical defects, integrate advanced technologies in vision and deep learning into the post-production stage of video production, build a deep learning model based on relevant knowledge of audio-visual language, automatically select materials and assemble them, shape the editing style, ensure the rationality of editing, and improve the quality of video editing and production.

[0006] The purpose of this application is achieved through the following technical solutions:

[0007] In a first aspect, the present application proposes a shot assembly method based on video-assisted editing, the method comprising:

[0008] Use transition recognition technology to segment the video material to obtain candidate shot groups;

[0009] Using a scene recognition model and a motion grammar recognition model respectively to mark the scene and motion grammar of the candidate shot group to obtain a marked candidate shot group;

[0010] Based on the labeled candidate shot groups, a one-hot matrix is ​​used to represent the film sequence;

[0011] The energy value of the film sequence is calculated based on the scalable shot-motion grammar combined energy model, and the film sequence is gradually optimized using Langevin dynamics to obtain the optimal shot combination sequence to form a film.

[0012] In a possible implementation, the formula used for gradually optimizing the slice sequence using Langevin dynamics is: Where X t is the slice sequence after the tth optimization, X t+1 is the film sequence after the t+1th optimization, Z is the scene and motion grammar label value of the candidate sequence, E c (X t , Z) is the energy value of the slice sequence, is the partial derivative of the energy model with respect to the slice sequence, is the random noise after the t-th tuning and before the t+1-th tuning, and γ is a hyperparameter representing the update rate.

[0013] In a possible implementation, the scalable scene-by-scene motion grammar combined energy model E c For: E c =αE view +βE move , E view represents the scene connection energy model, E move represents the motion grammar connection energy model, α represents the weight of scene connection, and β represents the weight of motion grammar connection.

[0014] In a possible implementation, the steps of constructing the scene connection energy model are:

[0015] Input the scene labels of the candidate shot group into the defined scene label vectorization layer, obtain the scene vectorization result and convert it into a scene matrix;

[0016] Calculate the scene features of the film sequence based on the scene matrix and the film sequence where f RELU () is the activation function, X is the sequence of slices, V view is the scene matrix;

[0017] The scene characteristics of the sequence Add the sequential feature vectors to obtain the scene feature results;

[0018] The scene feature results are input into the context information fusion layer composed of the Transformer-based encoding layer, and the global scene feature is obtained by averaging.

[0019] The global scene features are input into the scene connection energy calculation module to obtain the scene connection energy value.

[0020] In a possible implementation manner, the scene connection energy calculation module includes a first scene connection energy calculation layer f v-1 (y), activation function layer f RELU (y) and the second scene connection energy calculation layer f v-2 (y);

[0021] The first scene connection energy calculation layer f v-1 (y) is: f v-1 (y) = W ve1 y+b ve1 ,in is the weight of the first scene connection energy calculation layer, is the bias parameter of the first scene connection energy calculation layer, and y is the input value;

[0022] The second scene connection energy calculation layer f v-2 (y) is: f v-2 (y) = W ve2 y+b ve2 ,in is the weight of the second scene connection energy calculation layer, b ve1 ∈R is the bias parameter of the second scene connection energy calculation layer.

[0023] In a possible implementation, the loss function Loss of the scene connection energy model is: where p D is the data distribution, x view+ From the data distribution p D The slice sequence obtained by sampling in view- Represents the learned data distribution p θ The sampled sequence is represents the expected value of the scene connection energy value of the film sequence obtained by sampling from the data distribution pD, Represents the learned data distribution p θ The expected value of the scene connection energy value of the sampled sequence, Respectively represent the data distribution p D The expected value of the square of the scene connection energy value of the sequence obtained by sampling and the data distribution p learned from θ The expected value of the square of the scene connection energy value of the sampled sequence.

[0024] In a possible implementation, the steps for constructing the motion grammar cohesion energy model are:

[0025] Input the motion grammar label of the candidate shot into the defined motion grammar label vectorization layer, obtain the motion grammar vectorization result and convert it into a motion grammar matrix;

[0026] The motion grammar features of the piece sequence are obtained based on the motion grammar matrix and the piece sequence calculation where f RELU () is the activation function, V move is the motion grammar matrix, X is the piece sequence;

[0027] The motion grammar features of the sequence Add the sequential feature vectors to get the motion grammar feature results;

[0028] The motion grammar feature results are input into the context information fusion layer composed of the Transformer-based encoding layer, and the global motion grammar connection features are obtained by averaging.

[0029] The global motion grammar connection features are input into the motion grammar connection energy calculation module to obtain the motion grammar connection energy value.

[0030] In a possible implementation, the motion grammar connection energy calculation module includes a first motion grammar connection energy calculation layer f m-1 (y), activation function layer f RELU (y) and the second motion grammar connection energy calculation layer f m-2 (y);

[0031] The first motion grammar connection energy calculation layer f m-1 (y):f m-1 (y) = W me1 y+b me1 ,in is the weight of the first movement grammar connection energy layer, is the bias parameter of the first motion grammar connection energy layer, and y is the input value.

[0032] The second motion grammar connects the energy calculation layer f m-2 (y): fm- 2(y)=W me2y +b me2 ,in is the weight of the second motion grammar connection energy calculation layer, b me2 ∈R is the bias parameter of the second motion syntax connection energy calculation layer.

[0033] In a possible implementation, the loss function Loss of the motion grammar connection energy model is: where p D is the data distribution, x move + is the distribution from the data p D The slice sequence obtained by sampling in move- Represents the learned data distribution p θ The sampled sequence is Represents the data distribution p D The expected value of the energy value of the motion syntax connection of the piece sequence obtained by sampling, represents the expected value of the energy value of the motion grammar connection of the piece sequence sampled from the learned data distribution pθ, Respectively represent the data distribution p D The expected value of the square of the energy value of the motion grammar connection of the piece sequence obtained by sampling and the data distribution p learned from θ The expected value of the square of the motion grammar connection energy of the sampled sequence.

[0034] In a possible implementation, the method further includes:

[0035] Construct a shot-by-shot composition scoring table / shot-by-shot movement grammar composition scoring table based on editing style;

[0036] Set the number of times each scene tag / movement grammar tag can appear;

[0037] Set the length of the segment sequence and randomly generate the scene label sequence / motion grammar label sequence;

[0038] Calculate the total score of the cohesion of the scene label sequence / movement grammar label sequence according to the scoring table;

[0039] Set the threshold, divide the sequence data that meets the conditions into positive and negative samples, and combine the positive and negative samples in pairs to obtain training data.

[0040] The training data is used to train the scene connection energy model and the motion grammar connection energy model.

[0041] The above-mentioned main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application; and in the present application, (non-conflicting options) options and other options can also be freely combined. After understanding the scheme of the present application, those skilled in the art can understand that there are multiple combinations based on the prior art and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0042] The present application discloses a shot assembly method based on video-assisted editing, which relates to the technical field of media content creation. First, the video material is segmented by using transition recognition technology to obtain candidate shot groups. Secondly, the scene recognition model and the motion grammar recognition model are used to mark the scene and motion grammar of the candidate shot group to obtain the marked candidate shot group. Then, based on the marked candidate shot group, a one-hot matrix is ​​used to represent the film sequence. Finally, the energy value of the film sequence is calculated based on an extensible scene motion grammar combined energy model. The Langevin dynamics is used to gradually optimize the film sequence to obtain the best shot assembly sequence to form a film. The present application can assist in the video editing of news manuscripts, improve the production efficiency of news videos, and improve the timeliness of news releases. At the same time, it can better control the quality of the film, ensure that the film meets certain audio-visual language knowledge, and improve the audience's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A schematic flow chart of a shot assembly method based on video-assisted editing proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0045] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0046] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of this application.

[0047] In the prior art, traditional news video creation is both complex and time-consuming. After shooting relevant video materials and selecting suitable candidate materials in combination with news copy, editors will organize the candidate materials in relevant editing software to form news videos. When editing video materials, editors need to consider a variety of factors. In addition to the quality of the picture itself, it also involves the fit between the video material picture and the text. More importantly, editors must always consider the constraints of audio-visual language knowledge, such as the changes in the field of view of the front and back shots when the video materials are edited, the changes in motion grammar, and the changes in color. This traditional, purely manual video production post-production process not only requires high professionalism from editors, but is also very time-consuming and labor-intensive, and it is easy to lose the timeliness of news.

[0048] Therefore, in order to solve the above-mentioned technical problems, the embodiment of the present application proposes a shot assembly method based on video-assisted editing, which integrates advanced technologies in vision and deep learning into the post-production stage of video production, and builds a deep learning model in combination with relevant knowledge of audio-visual language. It automatically selects materials and assembles them, which can improve the efficiency and quality of the post-production stage of news video production and ensure that the generated news video meets certain audio-visual language knowledge constraints. It is described in detail below.

[0049] Please refer to Figure 1 , Figure 1 A schematic diagram of a process flow of a shot assembly method based on video-assisted editing proposed in an embodiment of the present application is shown, and the method includes:

[0050] Step S1: Use transition recognition technology to segment video material to obtain candidate shot groups.

[0051] Given a video material related to a news article, a transition recognition technology (e.g., based on pixel difference, feature matching, or machine learning) is used to accurately identify the transition points in the video and segment the video material into multiple independent shot segments. Each segment is called a candidate shot, i.e., {shot0, shot2, ..., shot L-1}, where shot0 represents the first candidate shot, shot L-1 represents the Lth candidate shot, and L represents the total number of candidate shots.

[0052] Step S2: using the scene recognition model and the motion grammar recognition model to label the scene and motion grammar of the candidate shot group respectively to obtain a labeled candidate shot group.

[0053] The scene recognition model is used to identify the scene type in the video shot, where the scene type can include long shot, full shot, medium shot, close shot and close-up. The scene recognition model can determine the scene type by analyzing factors such as the size of the object in the shot and the proportion of the background. The motion grammar recognition model is used to identify the motion mode in the video shot, where the motion mode can include stable, up, down, left and right. The motion grammar recognition model can determine the motion mode by analyzing the motion trajectory, optical flow and other information in the shot.

[0054] During the labeling process, the input is a group of candidate shots, and then each candidate shot is input into the scene recognition model, which outputs the scene label of each shot (such as long shot, full shot, medium shot, close-up, close-up). Then, each candidate shot is input into the motion grammar recognition model, which outputs the motion label of each shot (such as stable, up, down, left, right), and finally the labeled candidate shot group is obtained.

[0055] Step S3: Based on the labeled candidate shot groups, a one-hot matrix is ​​used to represent the film sequence.

[0056] Randomly select a few shots from the candidate shots to construct the initial value of the film sequence X, where X is represented by a one-hot matrix. Assume that the final film requires a total of M shots, that is, X∈R M*L In the matrix X, X i，j =1 means that the i+1th shot in the film sequence is the j+1th candidate shot. At the initial moment, the candidate shots can be randomly selected. For example, when the film requires 3 shots and the 3rd candidate shot, the 2nd candidate shot and the Lth candidate shot are selected at the initial moment, namely shot2, shot1 and shot L-1 When X∈R 3*L , X 0，2 =1,X 1，1 =1,X 2，L-1 =1, that is, the initial value of X 0 for:

[0057] Step S4: Calculate the energy value of the film sequence based on the scalable scene motion grammar combined energy model, and gradually optimize the film sequence using Langevin dynamics to obtain the best shot assembly sequence to form a film.

[0058] Scalable scene motion grammar combined energy model E c For: E c =αE view +βE move , E view represents the scene connection energy model, E moverepresents the energy model of motion grammar connection, α represents the weight of scene connection, β represents the weight of motion grammar connection, and using this energy model E c Able to find the best film sequence.

[0059] The formula used to gradually optimize the slice sequence using Langevin dynamics is: Where X t is the slice sequence after the tth optimization, X t+1 is the film sequence after the t+1th optimization, Z is the scene and motion grammar label value of the candidate sequence, E c (X t , Z) is the energy value of the slice sequence, is the partial derivative of the energy model with respect to the slice sequence, is the random noise after the t-th tuning and before the t+1-th tuning, and γ is a hyperparameter representing the update rate.

[0060] When t = 0, there is X t =X 0 , that is, the value of the slice sequence X before optimization, at this time After each optimization, it is necessary to ensure that each element in X has a value between 0 and 1. After T optimizations, the best shot sequence Video = {Video0, Video1, ..., Video i ,..,Video M-1}, where Video i Indicates the i+1th shot in the film, Video i =shot j ,in What we want is that after the tuning process is completed, X T The column j where the largest element in the i-th row is located, shot j represents the j+1th shot among the candidate shots.

[0061] Combining energy models in scalable scene-by-scene motion grammars c In the , the construction steps of the scene connection energy model are:

[0062] Input the scene labels of the candidate shot group into the defined scene label vectorization layer, obtain the scene vectorization result and convert it into a scene matrix;

[0063] Calculate the scene features of the film sequence based on the scene matrix and the film sequence where f RELU () is the activation function, X is the sequence of slices, V view is the scene matrix;

[0064] The scene characteristics of the sequence Add the sequential feature vectors to obtain the scene feature results;

[0065] The scene feature results are input into the context information fusion layer composed of the Transformer-based encoding layer, and the global scene feature is obtained by averaging.

[0066] The global scene features are input into the scene connection energy calculation module to obtain the scene connection energy value.

[0067] First, define the scene label vectorization layer and input the scene label of the candidate shot. To the scene label vectorization layer, output the scene vectorization results and combine them into a scene matrix in, represents the scene label of the i+1th candidate shot, The corresponding vectorized result is D view is the vector dimension, and the scene matrix is:

[0068] Then calculate the scene features of the film sequence. First input the film sequence X and the scene vectorization result V view , use the activation function to calculate the scene features of the film sequence where f RELU () is the activation function, and its construction formula is Where δ is the scaling parameter and y is the input value. represents the scene feature of the i+1th shot in the film sequence, Indicates the scene feature of the Mth shot in the film sequence, the scene feature of the film sequence for:

[0069] Then add the sequential feature vector to the scene features of the sequence: where t i Represents the sequential feature vector of the i-th position of the shot in the film sequence, where t i is defined as follows: where t i，2j , t i，2j+1 They are the 2j-th and 2j+1-th components of the sequential eigenvector at position i respectively.

[0070] Then the scene feature results are sent to the context information fusion layer composed of 3 layers of Transformer-based encoding layers, and the results are averaged to obtain the global scene feature.

[0071] Construct a scene connection energy calculation module, the scene connection energy calculation module includes a first scene connection energy calculation layer f v-1 (y), activation function layer F RELU (y) and the second scene connection energy calculation layer f v-2 (y);

[0072] The first scene connection energy calculation layer f v-1 (y) is: f v-1 (y) = W ve1 y+b ve1 ,in is the weight of the first scene connection energy calculation layer, is the bias parameter of the first scene connection energy calculation layer, and y is the input value;

[0073] The second scene connection energy calculation layer f v-2 (y) is: f v-2 (y) = W ve2 y+b ve2 ,in is the weight of the second scene connection energy calculation layer, b ve1 ∈R is the bias parameter of the second scene connection energy calculation layer.

[0074] Finally, the global scene features are input into the scene connection energy calculation module to obtain the scene connection energy value: That is, firstly, the global scene feature is sent to the first scene connection energy layer f v-1 (y), the calculation result is sent to the activation function layer f RELU (y) Get the activation value, and then send it to the second scene connection energy calculation layer f v-2 (y) Obtain the scene connection energy value.

[0075] The loss function Loss of the scene connection energy model is: where p D is the data distribution, x view + is the distribution from the data p D The slice sequence obtained by sampling in view - represents the learned data distribution p θ A sequence of slices sampled in . Represents the data distribution p D The expected value of the scene connection energy value of the film sequence obtained by sampling, Represents the learned data distribution p θ The expected value of the scene connection energy value of the sampled film sequence in . In order to prevent overfitting, an L2 regularization constraint is added, that is, Respectively represent the data distribution p DThe expected value of the square of the scene connection energy value of the sequence obtained by sampling is obtained from the learned data distribution p θ The expected value of the square of the scene connection energy value of the sampled sequence.

[0076] More specifically, x view+ represents the sequence with higher score in scene connection (positive sample), x view- It represents the sequence of inputs (negative samples) with low scores in scene connection after optimization. Note that the scalable scene motion grammar combined energy model E in the optimization formula here c =αE view +βE move In this case, α=1, β=0.

[0077] Combining energy models in scalable scene-by-scene motion grammars c In the paper, the steps of constructing the cohesion energy model of movement grammar are:

[0078] Input the motion grammar label of the candidate shot into the defined motion grammar label vectorization layer, obtain the motion grammar vectorization result and convert it into a motion grammar matrix;

[0079] The motion grammar features of the piece sequence are obtained based on the motion grammar matrix and the piece sequence calculation where f RELU () is the activation function, V move is the motion grammar matrix, X is the piece sequence;

[0080] The motion grammar features of the sequence Add the sequential feature vectors to get the motion grammar feature results;

[0081] The motion grammar feature results are input into the context information fusion layer composed of the Transformer-based encoding layer, and the global motion grammar connection features are obtained by averaging.

[0082] The global motion grammar connection features are input into the motion grammar connection energy calculation module to obtain the motion grammar connection energy value. First, the motion grammar label vectorization layer is defined. Input into the motion grammar label vectorization layer, obtain the motion grammar vectorization result and convert it into a motion grammar matrix in, represents the motion grammar label of the i+1th candidate shot, The corresponding vectorized result is D move is the vector dimension, the motion grammar matrix V move for:

[0083] Then, the motion grammar features of the piece sequence are obtained based on the motion grammar matrix and the piece sequence. where f RELU () is the activation function, and the construction formula of the activation function is Where δ is the scaling parameter and y is the input value. represents the motion grammar features of the i+1th shot in the film sequence, Represents the motion grammar features of the Mth shot in the sequence

[0084] Then add the sequential feature vectors for the motion grammar features of the sequence: where t i Represents the sequential feature vector of the i-th position of the shot in the film sequence, where t i is defined as: t i，2j , t i，2j+1 They are the 2j-th and 2j+1-th components of the sequential eigenvector at position i respectively.

[0085] The motion grammar feature results are then sent to the context information fusion layer consisting of three layers of Transformer-based encoding layers, and the results are averaged to obtain the global motion grammar connection features.

[0086] Construct a motion grammar connection energy calculation module, the motion grammar connection energy calculation module includes a first motion grammar connection energy calculation layer f m-1 (y), activation function layer f RELU (y) and the second motion grammar connection energy calculation layer f m-2 (y);

[0087] The first motion grammar connection energy calculation layer f m-1 (y):f m-1 (y) = W me1 y+b me1 ,in is the weight of the first movement grammar connection energy layer, is the bias parameter of the first motion grammar connection energy layer, and y is the input value.

[0088] The second motion grammar connects the energy calculation layer f m-2 (y):f m-2 (y) = W me2 y+b me2 ,in is the weight of the second motion grammar connection energy calculation layer, bme2 ∈R is the bias parameter of the second motion syntax connection energy calculation layer.

[0089] Finally, the global motion grammar connection features are sent to the motion grammar connection energy calculation module to calculate the energy value: That is, firstly, the global motion grammar cohesion feature is sent to the first motion grammar cohesion energy layer f m-1 (y), the calculation result is sent to the activation function layer f RELU (y) Get the activation value, and then send it to the second motion grammar connection energy calculation layer f m-2 (y) Obtain the motion grammar connection energy value.

[0090] The loss function Loss of the motion grammar connection energy model is: where p D is the data distribution, x move+ From the data distribution p D The slice sequence obtained by sampling in move- Represents the learned data distribution p θ A sequence of slices sampled in . Represents the data distribution p D The expected value of the energy value of the motion syntax connection of the piece sequence obtained by sampling, Represents the learned data distribution p θ The expected value of the energy value of the motion syntax connection of the sampled piece sequence. In order to prevent overfitting, an L2 regularization constraint is added, that is, Respectively represent the data distribution p D The expected value of the square of the energy value of the motion grammar connection of the piece sequence obtained by sampling is obtained from the learned data distribution p θ The expected value of the square of the motion grammar connection energy of the sampled sequence.

[0091] x move+ represents the sequence with higher score in motion grammar cohesion (positive sample), x move- It represents the sequence of inputs (negative samples) with low scores in motion grammar cohesion after optimization. Note that the scalable scene-specific motion grammar combination energy model E in the optimization formula is c =αE view +βE move In this case, α=0, β=1.

[0092] The lens assembly method also includes:

[0093] Construct a shot-by-shot composition scoring table / shot-by-shot movement grammar composition scoring table based on editing style;

[0094] Set the number of times each scene tag / movement grammar tag can appear;

[0095] Set the length of the segment sequence and randomly generate the scene label sequence / motion grammar label sequence;

[0096] Calculate the total score of the cohesion of the scene label sequence / movement grammar label sequence according to the scoring table;

[0097] Set the threshold, divide the sequence data that meets the conditions into positive and negative samples, and combine the positive and negative samples in pairs to obtain training data.

[0098] The training data is used to train the scene connection energy model and the motion grammar connection energy model.

[0099] The above is the construction method of positive and negative samples and the training model using positive and negative samples. First, create a table with rows and columns representing different scene labels or motion grammar labels. The value in the cell represents the composition / connection score between the two labels. According to the specific editing style, define the compatibility and fluency scoring criteria between different scene labels or motion grammar labels. Then set a maximum number of occurrences for each scene label or motion grammar label to ensure that the distribution of each label in the generated sequence is consistent with the actual situation and avoid excessive concentration or missing of a certain type of label. Then set a fixed sequence length, such as 10 shots, and then randomly generate a series of scene labels or motion grammar labels within this range to ensure that the total number of these labels does not exceed their respective limited number of times. At this time, a sequence that meets the actual video length is generated.

[0100] The scoring table is used to calculate the connection score between each pair of adjacent labels, and all scores are added up to get the total score of the entire sequence to evaluate whether the generated sequence conforms to a good editing style. A threshold is set, and sequences above this threshold are marked as positive samples (good editing), and sequences below this threshold are marked as negative samples (bad editing) to distinguish high-quality and low-quality shot combinations or motion grammar combinations, forming positive and negative samples that can be used for training. Finally, the positive samples and negative samples are paired to form a training data set. Each pair of samples can be used to train the model to identify high-quality and low-quality editing styles, providing the model with enough comparison samples to learn editing styles of different qualities.

[0101] Using the training dataset constructed above, the scene connection energy model and the motion grammar connection energy model are trained to optimize the model parameters and improve their prediction accuracy. Through a large number of positive and negative sample training, the model can accurately predict the quality of new sequences, thereby assisting video editing decisions.

[0102] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0103] First, the automated method of selecting and stitching shots significantly reduces the time and effort required for manual editing, making the news video production process more efficient and increasing the overall production speed.

[0104] Second, the deep learning model built with audio-visual language knowledge can better control the quality of the finished film. The model learns and applies the rules of audio-visual language to ensure that the generated video meets professional editing standards.

[0105] Third, by naturally incorporating audiovisual language knowledge into the model, we ensure that the generated videos have a high level of professionalism both visually and aurally, improving the audience’s viewing experience.

[0106] Fourth, the combined energy model adopted in this application is highly scalable and can add new audio-visual language constraints such as hue, color saturation, etc. according to different needs and scenarios, so as to better adapt to different editing styles and requirements.

[0107] Fifth, the model used in this application can automatically select the most suitable shots from a large amount of materials for assembly, reducing the time of manual screening and improving work efficiency.

[0108] In summary, this application significantly improves the production efficiency and quality of news videos by combining automation technology and deep learning models, while ensuring that the finished film conforms to professional audio-visual language knowledge and improves the audience's viewing experience. In addition, the scalability and flexibility of this method make it have a wide range of application potentials and can play a role in a variety of video production scenarios.

[0109] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A shot assembly method based on video-assisted editing, characterized in that: The method comprises: Use transition recognition technology to segment the video material to obtain candidate shot groups; Using a scene recognition model and a motion grammar recognition model respectively to mark the scene and motion grammar of the candidate shot group to obtain a marked candidate shot group; Based on the labeled candidate shot groups, a one-hot matrix is ​​used to represent the film sequence; The energy value of the film sequence is calculated based on the scalable shot-motion grammar combined energy model, and the film sequence is gradually optimized using Langevin dynamics to obtain the optimal shot combination sequence to form a film.

2. The lens assembly method according to claim 1, wherein: The formula used to gradually optimize the slice sequence using Langevin dynamics is: Where X t is the slice sequence after the tth optimization, X t+1 is the film sequence after the t+1th optimization, Z is the scene and motion grammar label value of the candidate sequence, E c (X t ,Z) is the energy value of the piece sequence, is the partial derivative of the energy model with respect to the slice sequence, is the random noise after the t-th tuning and before the t+1-th tuning, and γ is a hyperparameter representing the update rate.

3. The lens assembly method according to claim 1, wherein: Scalable scene motion grammar combined energy model E c For: E c =αE view +βE move , E view represents the scene connection energy model, E move represents the motion grammar connection energy model, α represents the weight of scene connection, and β represents the weight of motion grammar connection.

4. The lens assembly method according to claim 3, wherein: The steps to construct the scene connection energy model are as follows: Input the scene labels of the candidate shot group into the defined scene label vectorization layer, obtain the scene vectorization result and convert it into a scene matrix; Calculate the scene features of the film sequence based on the scene matrix and the film sequence where f RELU () is the activation function, X is the sequence of slices, V view is the scene matrix; The scene characteristics of the sequence Add the sequential feature vectors to obtain the scene feature results; The scene feature results are input into the context information fusion layer composed of the Transformer-based encoding layer, and the global scene feature is obtained by averaging. The global scene features are input into the scene connection energy calculation module to obtain the scene connection energy value.

5. The lens assembly method according to claim 4, wherein: The scene connection energy calculation module includes a first scene connection energy calculation layer f v-1 (y), activation function layer f RELU (y) and the second scene connection energy calculation layer f v-2 (y); The first scene connection energy calculation layer f v-1 (y) is: f v-1 (y) = W ve1 y+b ve1 ,in is the weight of the first scene connection energy calculation layer, is the bias parameter of the first scene connection energy calculation layer, and y is the input value; The second scene connection energy calculation layer f v-2 (y) is: f v-2 (y) = W ve2 y+b ve2 ,in is the weight of the second scene connection energy calculation layer, b ve1 ∈R is the bias parameter of the second scene connection energy calculation layer.

6. The lens assembly method according to claim 3 or 4, characterized in that: The loss function Loss of the scene connection energy model is: where p D is the data distribution, x view+ From the data distribution p D The slice sequence obtained by sampling in view- Represents the learned data distribution p θ The sampled sequence is Represents the data distribution p D The expected value of the scene connection energy value of the film sequence obtained by sampling, Represents the learned data distribution p θ The expected value of the scene connection energy value of the sampled sequence, Respectively represent the data distribution p D The expected value of the square of the scene connection energy value of the sequence obtained by sampling and the data distribution p learned from θ The expected value of the square of the scene connection energy value of the sampled sequence.

7. The lens assembly method according to claim 3, wherein: The steps to construct the movement grammar cohesion energy model are: Input the motion grammar label of the candidate shot into the defined motion grammar label vectorization layer, obtain the motion grammar vectorization result and convert it into a motion grammar matrix; The motion grammar features of the piece sequence are obtained based on the motion grammar matrix and the piece sequence calculation where f RELU () is the activation function, V move is the motion grammar matrix, X is the piece sequence; The motion grammar features of the sequence Add the sequential feature vectors to get the motion grammar feature results; The motion grammar feature results are input into the context information fusion layer composed of the Transformer-based encoding layer, and the global motion grammar connection features are obtained by averaging. The global motion grammar connection features are input into the motion grammar connection energy calculation module to obtain the motion grammar connection energy value.

8. The lens assembly method according to claim 7, wherein: The motion grammar connection energy calculation module includes a first motion grammar connection energy calculation layer f m*1 (y), activation function layer f RELU (y) and the second motion grammar connection energy calculation layer f m-2 (y); The first motion grammar connection energy calculation layer f m-1 (y):f m-1 (y) = W me1 y+b me1 ,in is the weight of the first movement grammar connection energy layer, is the bias parameter of the first motion grammar connection energy layer, and y is the input value; The second motion grammar connects the energy calculation layer f m-2 (y):f m-2 (y) = W me2 y+b me2 ,in is the weight of the second motion grammar connection energy calculation layer, b me2 ∈R is the bias parameter of the second motion syntax connection energy calculation layer.

9. The lens assembly method according to claim 3 or 7, characterized in that: The loss function Loss of the motion grammar connection energy model is: where p D is the data distribution, x move+ From the data distribution p D The slice sequence obtained by sampling in move- Represents the learned data distribution p θ The sampled sequence is Represents the data distribution p D The expected value of the energy value of the motion syntax connection of the piece sequence obtained by sampling, Represents the learned data distribution p θ The expected value of the energy value of the motion syntax connection of the sampled sequence, Respectively represent the data distribution p D The expected value of the square of the energy value of the motion grammar connection of the piece sequence obtained by sampling and the data distribution p learned from θ The expected value of the square of the motion grammar connection energy of the sampled sequence.

10. The lens assembly method according to claim 3, wherein: The method further comprises: Construct a shot-by-shot composition scoring table / shot-by-shot movement grammar composition scoring table based on editing style; Set the number of times each scene tag / movement grammar tag can appear; Set the length of the segment sequence and randomly generate the scene label sequence / motion grammar label sequence; Calculate the total score of the cohesion of the scene label sequence / movement grammar label sequence according to the scoring table; Set the threshold, divide the sequence data that meets the conditions into positive and negative samples, and combine the positive and negative samples in pairs to obtain training data. The training data is used to train the scene connection energy model and the motion grammar connection energy model.

Images