Video synthesis method and device

Abstract

The invention provides a video synthesis method and device, and belongs to the technical field of video processing, and the method comprises the steps: inputting a video watched by a user and an expression video when the user watches the video into a high-quality video clip recognition model, and outputting and obtaining the position information of a high-quality video clip in the video watched by the user; determining a consistency score of the high-quality video clip based on the position information of the high-quality video clip; if the difference of the consistency scores of the two high-quality video clips is within a consistency threshold value, performing consistency marking on the two high-quality video clips; then, for each high-quality video clip, based on the position information of the high-quality video clip, carrying out sub-lens softening processing on the high-quality video clip to obtain an optimized video clip; determining an optimal transition point for every two optimized video clips with the consistent marks; and combining the two optimized video clips with the consistency marks at the optimal transition point to obtain a target video. And the generated target video accurately reflects the interest points of the user.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video synthesis method and device. Background Art

[0002] In recent years, user-generated content (UGC) based on popular IP video materials has been widely used around the world. Especially on video platforms, the demand for users to edit and share the highlights of their videos to create new content has increased significantly. However, the current video material generation method has the following major problems:

[0003] 1. Low efficiency of manual production: The existing manual editing method is inefficient and it is difficult to quickly produce high-quality secondary creative materials on a large scale.

[0004] 2. Insufficient automation quality: Although traditional AI-based content recognition technology can achieve automatic editing, it has low accuracy in identifying the emotions and key moments of video content, and it is difficult to meet users' demand for high-quality and accurate materials. Summary of the invention

[0005] In view of this, the present application provides a video synthesis method and device, which generates a target video by combining user emotions, so that the generated target video accurately reflects the user's interests.

[0006] The first aspect of the present application provides a video synthesis method, comprising:

[0007] The video watched by the user and the expression video of the user when watching the video are input into the high-quality video segment recognition model, and the location information of the high-quality video segment in the video watched by the user is output; wherein the high-quality video segment recognition model is obtained by training a deep learning model according to a training sample set; the training sample set includes at least one training sample; the training sample includes the original video of the training sample, the expression video of the user when watching the original video of the training sample, and the location information of the real high-quality segment in the original video of the training sample;

[0008] For each high-quality video segment, determining a consistency score of the high-quality video segment based on the position information of the high-quality video segment;

[0009] If the difference in consistency scores between two high-quality video clips is within a consistency threshold, the two high-quality video clips are marked as consistent;

[0010] For each high-quality video segment, based on the position information of the high-quality video segment, the high-quality video segment is subjected to a split-frame softening process to obtain an optimized video segment;

[0011] For every two optimized video clips with consistent marks, determine the best transition point;

[0012] The two optimized video segments with consistent marks are merged at the optimal transition point to obtain a target video.

[0013] Optionally, the video synthesis method further includes:

[0014] For each high-quality video clip, extract all user interaction data based on the location information of the high-quality video clip;

[0015] Determine the emotional high points in the high-quality video clips based on the facial expression videos of all users when they watch the high-quality video clips and all user interaction data;

[0016] For each emotional high point in the high-quality video clip, according to the target emotional type of the emotional high point, a post-effect corresponding to the target emotional type is inserted at the position of the emotional high point.

[0017] Optionally, the method for constructing the high-quality video clip recognition model includes:

[0018] Constructing a training sample set; wherein the training sample set includes at least one training sample; the training sample includes an original video of the training sample, a video of a user's facial expression when watching the original video of the training sample, and real position information of high-quality clips in the original video of the training sample;

[0019] The original video of the training sample and the expression video of the user when watching the original video of the training sample are input into the deep learning model, and the predicted position information of the high-quality clips in the original video of the training sample is output; wherein the deep learning model determines the expression features of the user in each frame in the expression video of the user when watching the original video of the training sample; determines the emotion type of the user according to the expression features of the user; determines the weight of the expression features according to the emotion type of the user; determines the user expression score of the frame according to the score of the expression features and the weight of the expression features; determines the predicted position information of the high-quality clips in the original video of the training sample according to the user expression scores of all frames;

[0020] The error between the predicted position information of the high-quality clips in the original video of the training sample and the actual position information of the high-quality clips in the original video of the training sample is used to adjust the parameters in the deep learning model until the error between the predicted position information of the high-quality clips in the original video of the training sample and the actual position information of the high-quality clips in the original video of the training sample meets the preset convergence condition, and the deep learning model is used as a high-quality video clip recognition model.

[0021] Optionally, for each high-quality video segment, determining the consistency score of the high-quality video segment based on the position information of the high-quality video segment includes:

[0022] For each high-quality video clip, based on the position information of the high-quality video clip, extract multi-dimensional feature information of the video from the video watched by the user;

[0023] For the feature information of each dimension, calculating the feature similarity of the dimension;

[0024] The consistency score of the high-quality video clip is determined according to the feature similarities of all the dimensions.

[0025] Optionally, for each high-quality video segment, based on the position information of the high-quality video segment, performing a split-frame softening process on the high-quality video segment to obtain an optimized video segment includes:

[0026] For each high-quality video segment, extract a frame set according to the position information of the high-quality video segment; wherein the frame set is divided into a start frame set, an end frame set and an overall frame set;

[0027] For each frame set, feature processing is performed on the frame set to obtain a target feature vector of the frame set;

[0028] Determine the similarity of the starting storyboard according to the target feature vector of the starting frame set and the target feature vector of the entire frame set;

[0029] Determine the ending storyboard similarity according to the target feature vector of the ending frame set and the target feature vector of the entire frame set;

[0030] Determining a coherence evaluation result of the high-quality video segment according to the start storyboard similarity and the end storyboard similarity;

[0031] An optimized video segment is generated according to the coherence evaluation result of the high-quality video segment.

[0032] Optionally, for each frame set, performing feature processing on the frame set to obtain a target feature vector of the frame set includes:

[0033] For each frame image in each frame set, extract the color histogram of the frame image to obtain a first feature vector;

[0034] Input the image of the frame into the feature extraction model, and output a second feature vector;

[0035] Performing feature fusion on the first feature vector and the second feature vector to obtain a fusion vector of the frame;

[0036] Performing feature aggregation on the fusion vectors of all frames in the frame set to obtain a feature vector of the frame set;

[0037] The feature vectors of all frame sets are normalized to obtain the target feature vector of the frame set.

[0038] Optionally, determining the best transition point for every two optimized video segments having consistent marks includes:

[0039] For every two optimized video segments with consistent marks, determine the order of the two optimized video segments with consistent marks according to the position information of the optimized video segments;

[0040] Extracting an end frame set of the first optimized video segment according to the position information of the first optimized video segment;

[0041] Extracting a start frame set of the second optimized video segment according to the position information of the second optimized video segment;

[0042] The most similar frame pair between the end frame set of the first optimized video segment and the start frame set of the second optimized video segment is used as the best transition point.

[0043] Optionally, determining the emotional high point in the high-quality video clip according to the facial expression videos of all users when watching the high-quality video clip and all user interaction data includes:

[0044] Determine the average emotion intensity of users for each frame of the high-quality video clip based on the expression videos of all users when watching the high-quality video clip;

[0045] Determine the total user interaction emotion intensity of each frame of the high-quality video clip based on all user interaction data;

[0046] For each frame of the high-quality video clip, determining a user comprehensive emotion score for the frame according to the average emotion intensity of the user for the frame and the total interactive emotion intensity of the user for the frame;

[0047] If the user comprehensive emotion score of the frame is greater than the emotion high point threshold, the time point of the frame is used as the emotion high point in the high-quality video clip.

[0048] Optionally, for each emotional high point in the high-quality video clip, according to the target emotional type of the emotional high point, inserting a post-effect corresponding to the target emotional type at the emotional high point position includes:

[0049] For each emotional high point in the high-quality video clip, determining whether there is text data in the user interaction data at the emotional high point position;

[0050] If it is determined that there is text data in the user interaction data at the emotional high point, determining the emotional type of the emotional high point according to the text data;

[0051] If it is determined that there is no text data in the user interaction data at the emotional high point, determining that the emotional type of the emotional high point is a general emotional type;

[0052] Based on user interaction data, determine the total interactive emotion intensity of emotional high points;

[0053] Determining a number of a target emotion type according to the emotion type of the emotion high point and the total interactive emotion intensity of the emotion high point;

[0054] Determining a later effect corresponding to the target emotion type according to the number of the target emotion type;

[0055] Insert the post-effect corresponding to the target emotion type at the emotional high point.

[0056] A second aspect of the present application provides a video synthesis device, including:

[0057] A high-quality video segment determination unit is used to input the video watched by the user and the expression video of the user when watching the video into a high-quality video segment recognition model, and output the location information of the high-quality video segment in the video watched by the user; wherein the high-quality video segment recognition model is obtained by training a deep learning model according to a training sample set; the training sample set includes at least one training sample; the training sample includes an original video of the training sample, an expression video of the user when watching the original video of the training sample, and the location information of the real high-quality segment in the original video of the training sample;

[0058] A consistency processing unit, configured to determine, for each high-quality video segment, a consistency score of the high-quality video segment based on position information of the high-quality video segment;

[0059] a consistency marking unit, configured to mark the two high-quality video clips as consistent if the difference in consistency scores between the two high-quality video clips is within a consistency threshold;

[0060] A frame softening unit is used to perform frame softening processing on each high-quality video segment based on the position information of the high-quality video segment to obtain an optimized video segment;

[0061] An optimal transition point determination unit, used for determining an optimal transition point for each two optimized video segments having consistent marks;

[0062] The merging unit is used to merge the two optimized video segments with consistent marks at the optimal transition point to obtain a target video.

[0063] A third aspect of the present application provides an electronic device, including:

[0064] one or more processors;

[0065] a storage device having one or more programs stored thereon;

[0066] When the one or more programs are executed by the one or more processors, the one or more processors implement the video synthesis method as described in the first aspect.

[0067] A fourth aspect of the present application provides a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the video synthesis method as described in the first aspect is implemented.

[0068] As can be seen from the above scheme, the present application provides a video synthesis method and device. First, the video watched by the user and the expression video of the user when watching the video are input into the high-quality video segment recognition model, and the position information of the high-quality video segment in the video watched by the user is output; then, for each high-quality video segment, based on the position information of the high-quality video segment, the consistency score of the high-quality video segment is determined; if the difference in the consistency score of two high-quality video segments is within the consistency threshold, the two high-quality video segments are marked as consistent; then, for each high-quality video segment, based on the position information of the high-quality video segment, the high-quality video segment is subjected to split-screen softening processing to obtain an optimized video segment; finally, for each two optimized video segments with consistency marks, the best transition point is determined; at the best transition point, the two optimized video segments with consistency marks are merged to obtain a target video. The present invention generates a target video by combining user emotions, so that the generated target video accurately reflects the user's interest points. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0070] Figure 1 A specific flow chart of a video synthesis method provided in an embodiment of the present application;

[0071] Figure 2A schematic diagram of a video synthesis device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0073] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0074] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0075] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0076] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0077] The present application embodiment provides a video synthesis method, such as Figure 1 As shown, the specific steps include:

[0078] S101, input the video watched by the user and the expression video of the user when watching the video into a high-quality video segment recognition model, and output the location information of the high-quality video segment in the video watched by the user.

[0079] Among them, the high-quality video clip recognition model is obtained by training a deep learning model based on a training sample set; the training sample set includes at least one training sample; the training sample includes the original video of the training sample, the expression video of the user when watching the original video of the training sample, and the location information of the real high-quality clips in the original video of the training sample.

[0080] In the specific implementation process of this application, the facial expression video of the user when watching the video can be directly collected based on the camera carried by the device itself (for example: the front camera of a mobile phone, the camera of a laptop computer, etc.), which is not limited here. Of course, when collecting the user's facial expression video, the user's authorization should be obtained.

[0081] Among them, the facial expression video of the user when watching the video corresponds one-to-one with the progress time point of the video watched by the user.

[0082] It should be noted that the location information of the high-quality video clips in the video watched by the user is the location information of the high-quality video clips in the video watched by the user. The location information can be but is not limited to timestamp information, that is, the start and end timestamps of the high-quality video clips in the video watched by the user, which is not limited here.

[0083] Optionally, in another embodiment of the present application, an implementation of a method for constructing a high-quality video clip recognition model includes steps S201 to S205:

[0084] S201: Construct a training sample set.

[0085] The training sample set includes at least one training sample; the training sample includes the original video of the training sample, the expression video of the user when watching the original video of the training sample, and the real position information of the high-quality clips in the original video of the training sample.

[0086] In the implementation process of this application, the method of obtaining training samples includes but is not limited to: the original video of the training sample can be selected in advance by the staff, and multiple volunteers are recruited to conduct experiments, and facial expression videos of the volunteers when watching the original video of the training sample are collected, and the volunteers determine the true information of the high-quality clips in the original video of the training sample, that is, the true location information of the high-quality clips in the original video of the training sample is known.

[0087] S202: Input the original video of the training sample and the expression video of the user when watching the original video of the training sample into the deep learning model, and output the predicted position information of the high-quality clips in the original video of the training sample.

[0088] Among them, the deep learning model determines the expression features of the user in each frame of the expression video when the user watches the original video of the training sample; determines the user's emotion type based on the user's expression features; determines the weight of the expression features based on the user's emotion type; determines the user's expression score for the frame based on the score of the expression features and the weight of the expression features; and determines the predicted position information of high-quality clips in the original video of the training sample based on the user's expression scores of all frames.

[0089] In the specific implementation process of this application, expression features include but are not limited to pupil features, squinting features, crow's feet features (orbicularis oculi muscle is at work), mouth corner features (zygomatic major muscle is at work) and expression symmetry, which are not limited here.

[0090] It is understandable that when a person is excited, the pupils will dilate, when normal, the pupils will be in a neutral state, when negative, the pupils will shrink, and when laughing happily, the corners of the mouth will rise.

[0091] In the specific implementation process of the present application, a corresponding weight may be set for each expression feature, as shown in Table 1, which is an example of the score of the expression feature and the weight of the expression feature.

[0092] Table 1

[0093] Then, according to the score of the expression feature and the weight of the expression feature, the user expression score of the frame can be calculated using the following formula.

[0094] , ;in, For the The weight of the facial features, For the The score of the facial features.

[0095] In the specific implementation process of the present application, an implementation method of determining the predicted position information of the high-quality segments in the original video of the training sample according to the user expression scores of all frames includes:

[0096] Based on the user expression scores of all frames, a user expression score sequence is generated. Based on the user expression score sequence, a user expression score trend is generated. The user expression score trend is matched with the preset "true" expression score trend to determine the video clip containing the user's "true" expression. The position information of the video clip containing the user's "true" expression is used as the predicted position information of the high-quality clip in the original video of the training sample.

[0097] It should be noted that the duration of the user expression score sequence and the trend of the "true" expression score are set by technicians, experts, etc. through multiple experiments and are not limited here.

[0098] S203: Determine whether the error between the predicted position information of the high-quality segment in the original video of the training sample and the actual position information of the high-quality segment in the original video of the training sample meets a preset convergence condition.

[0099] The preset convergence condition is preset and changed by a technician or authorized staff and is not limited here.

[0100] It should be noted that, in the specific implementation process of the present application, it is not limited to using the preset convergence conditions to train the model, and the model can also be trained using a preset number of iterations, which is also not limited here.

[0101] Specifically, if it is determined that the error between the predicted position information of the high-quality clips in the original video of the training sample and the actual position information of the high-quality clips in the original video of the training sample meets the preset convergence condition, step S204 is executed; if it is determined that the error between the predicted position information of the high-quality clips in the original video of the training sample and the actual position information of the high-quality clips in the original video of the training sample does not meet the preset convergence condition, step S205 is executed.

[0102] S204: Using the deep learning model as a high-quality video clip recognition model.

[0103] S205. Adjust the parameters in the deep learning model using the error between the predicted position information of the high-quality segments in the original video of the training sample and the actual position information of the high-quality segments in the original video of the training sample.

[0104] S102: For each high-quality video segment, determine a consistency score of the high-quality video segment based on the position information of the high-quality video segment.

[0105] Optionally, in another embodiment of the present application, an implementation of step S102 includes steps S301 to S303:

[0106] S301. For each high-quality video segment, based on the position information of the high-quality video segment, extract multi-dimensional feature information of the video from the video watched by the user.

[0107] Among them, the multi-dimensional feature information includes but is not limited to themes, characters, scenes, storyboards, etc., which are not limited here.

[0108] S302: Calculate the feature similarity of each dimension of feature information.

[0109] In the specific implementation process of this application, high-quality video clips The similarity of the topic The method of calculating is to calculate the topic vector of the topic label of each high-quality video clip through text similarity (such as cosine similarity), and then calculate the topic similarity based on the topic vector , assuming and They are the topic vectors of two high-quality video clips:

[0110] ;

[0111] In the specific implementation process of this application, high-quality video clips Similarity of characters The calculation method can be based on the facial features of the main characters in the high-quality video clips, and the character feature vector can be extracted using a deep learning model. and Then, the character similarity is calculated based on the character feature vector :

[0112] ;

[0113] In the specific implementation process of this application, high-quality video clips The similarity of the scene The calculation method can be to extract the scene feature vector through the color histogram or deep learning feature, and then calculate the scene similarity based on the scene feature vector. Assume and They are the scene feature vectors of two high-quality video clips:

[0114] ;

[0115] In the specific implementation process of this application, high-quality video clips Similarity of storyboards The calculation method can be based on the similarity of the frame sequence of the storyboard, using frame features (such as color histogram and deep learning features) to calculate the storyboard feature vector, and then calculate the storyboard similarity based on the storyboard feature vector. Assume and They are the storyboard feature vectors of two high-quality video clips:

[0116] ;

[0117] S303: Determine the consistency score of the high-quality video clips according to the feature similarities of all dimensions.

[0118] Continuing with the above example, we can use the following formula to calculate the high-quality video clips Consistency score :

[0119] ;in, , , , is the weight coefficient of each feature, which can be adjusted according to different application scenarios to optimize the consistency measure and satisfy .

[0120] S103: If the difference in consistency scores between the two high-quality video clips is within a consistency threshold, the two high-quality video clips are marked as consistent.

[0121] The consistency threshold is preset and changed by technicians, experts, etc. and is not limited here.

[0122] Specifically, when high-quality video clips Consistency score , with high-quality video clips Consistency score The difference is at the consistency threshold within (i.e. ), then the high-quality video clips and high-quality video clips Classify them into segments with the same content and mark them as consistent.

[0123] S104: for each high-quality video segment, based on the position information of the high-quality video segment, perform a split-screen softening process on the high-quality video segment to obtain an optimized video segment.

[0124] Specifically, by performing frame softening processing on the high-quality video clips based on their position information, adjusting the start and end positions of the high-quality video clips, and obtaining optimized video clips, the problem of sudden starts and abrupt ends of video clips can be avoided.

[0125] Optionally, in another embodiment of the present application, an implementation of step S104 includes steps S401 to S406:

[0126] S401 . For each high-quality video segment, extract a frame set according to position information of the high-quality video segment.

[0127] The frame set is divided into a start frame set, an end frame set and a whole frame set.

[0128] Specifically, the starting frame set is from high-quality video Beginning (superscript position 1) extraction Frames, forming a start frame set , from high-quality video The end of (Superscript 1) Extract Frames, forming an end frame set , from high-quality video uniformly sampled throughout the period Frames, forming an overall frame set It should be noted that , , The length setting based on the specific high-quality video clips is confirmed in advance by technicians, experts, etc. through multiple experimental adjustments and is not limited here.

[0129] S402 : For each frame set, perform feature processing on the frame set to obtain a target feature vector of the frame set.

[0130] Optionally, in another embodiment of the present application, an implementation of step S402 includes steps S501 to S505:

[0131] S501 . For each frame image in each frame set, extract a color histogram of the frame image to obtain a first feature vector.

[0132] First, convert the image from RGB color mode to HSV color space, where H stands for hue, S stands for saturation, and V stands for value.

[0133] For each image, the histograms of the three channels H, S, and V are calculated separately, and the appropriate number of bins is selected (such as 16 bins for each channel). Then each histogram is normalized so that its sum is 1. Finally, the histograms of the three channels are spliced ​​into a feature vector, namely the first eigenvector , .in, , , They are the normalized H channel histogram, the normalized S channel histogram, and the normalized V channel histogram.

[0134] S502: Input the image of the frame into a feature extraction model, and output a second feature vector.

[0135] The feature extraction model is a pre-trained neural network (such as ResNet50, VGG16, etc.). The second eigenvector can be used to indicate the high-level semantic features of the image.

[0136] In the specific implementation process of the present application, before the image of the frame is input into the feature extraction model, the image can also be preprocessed: the image size is adjusted (such as 224×224) and normalized, which is not limited here.

[0137] Specifically, the image of the frame is input into the feature extraction model, and the output of the fully connected layer or the global average pooling layer is extracted as the second feature vector , ,in, represents the neural network model for extracting features, is the input image.

[0138] S503: Fusing the first feature vector and the second feature vector to obtain a fusion vector of the frame.

[0139] Specifically, the first feature vector and the second feature vector are concatenated to obtain a fusion vector of the frame.

[0140] In the actual application process of the present application, weights can also be set for the first feature vector and the second feature vector according to actual needs, so the fusion vector of the frame The calculation method can be as follows:

[0141] ,in, is the weight of the first eigenvector, is the weight of the second eigenvector.

[0142] S504 , performing feature aggregation on the fusion vectors of all frames in the frame set to obtain a feature vector of the frame set.

[0143] Specifically, an average feature vector is formed for the frame set as the feature vector of the frame set.

[0144] For the start frame set, the feature vector of the start frame set The calculation method is as follows:

[0145] ;in, The first frame in the start frame set frame;

[0146] For the end frame set, the feature vector of the end frame set The calculation method is as follows:

[0147] ;in, The first frame in the end frame set frame;

[0148] For the entire frame set, the feature vector of the entire frame set The calculation method is as follows:

[0149] ;in, is the first frame.

[0150] S505 : Normalize the feature vectors of all frame sets to obtain a target feature vector of the frame set.

[0151] Specifically, Normalize to get the target feature vector of the starting frame set ,Will Normalize to get the target feature vector of the end frame set ,Will Normalize to get the target feature vector of the whole frame set , normalized in the form of For example, .

[0152] S403 , determining the similarity of the starting storyboard according to the target feature vector of the starting frame set and the target feature vector of the entire frame set.

[0153] In the specific implementation process of the present application, cosine similarity can be used to determine the similarity of two target feature vectors to obtain the starting storyboard similarity.

[0154] The calculation method of cosine similarity is:

[0155] ; where A and B are two eigenvectors, • represents the dot product, ||A|| represents the Euclidean norm of vector A, and ||B|| represents the Euclidean norm of vector B.

[0156] So, let’s start with storyboard similarity. The following formula can be used for calculation:

[0157] , Sim is the above cosine similarity calculation method.

[0158] S404, determining the ending storyboard similarity according to the target feature vector of the ending frame set and the target feature vector of the entire frame set.

[0159] In the specific implementation process of the present application, cosine similarity can be used to determine the similarity of two target feature vectors to obtain the ending storyboard similarity.

[0160] Continuing from the above example, let’s start with storyboard similarity The following formula can be used for calculation:

[0161] , Sim is the above-mentioned cosine similarity calculation method.

[0162] S405. Determine the coherence evaluation result of the high-quality video segment according to the start-shot similarity and the end-shot similarity.

[0163] Specifically, the coherence evaluation result of the high-quality video segment can be determined according to the preset coherence evaluation rules.

[0164] The coherence evaluation rules are, for example:

[0165] If ≥ T and ≥ T, it is considered that the start shot and the end of the high-quality video segment are coherent with the overall content.

[0166] If < T and ≥ T, it is considered that the start shot of the high-quality video segment is not coherent with the overall content.

[0167] If ≥ T and < T, it is considered that the end shot of the high-quality video segment is not coherent with the overall content.

[0168] If < T and < T, it is considered that both the start and end shots of the high-quality video segment are not coherent with the overall content.

[0169] Among them, T is the similarity threshold, for example, T = 0.75. It should be noted that T is determined by technicians, experts, etc. through multiple experimental adjustments, and is not limited here.

[0170] S406. Generate an optimized video segment according to the coherence evaluation result of the high-quality video segment.

[0171] Continuing the above example, find the adjustment center frame in the set of start frames determined to be incoherent. The adjustment center frame is usually the center frame of the frame set, that is , and the frame set with a length of a before and after centered on the center frame (including) is used as the new start frame set to replace the original start frame set, and then the new is calculated. In the same way, the new of the end frame set is calculated, and then the coherence evaluation is performed. If it is determined to be incoherent with the similarity threshold T, then a increases by a preset step length, and this process is repeated until , are both within the similarity threshold T range, it is considered that the start and end shots have met the content coherence requirements.

[0172] Among them, a starts from 1, and a is not greater than the length from the first frame to the center frame of the start frame set, that is .

[0173] In the specific implementation process of the present application, there may be multiple central frames for the start frame set and the end frame set. In this case, reasonable positions of the start frame set and the end frame set can be determined based on the multiple central frames, and then the progress position range of the final start frame set and the end frame set can be determined through comprehensive evaluation.

[0174] In the specific implementation process of the present application, if the coherence evaluation result is coherent, the high-quality video segment is directly used as the optimized video segment. If the coherence evaluation result is incoherent, the high-quality video segment is cut according to the progress timestamps of the start frame set and the end frame set after the above adjustment to obtain the optimized video segment.

[0175] S105 : Determine an optimal transition point for every two optimized video segments having consistent marks.

[0176] Specifically, for optimized video segments with consistent content, content analysis is performed on the end of each optimized video segment and the beginning of the next optimized video segment to find the best transition point so that they can be merged based on the transition point later.

[0177] Optionally, in another embodiment of the present application, an implementation of step S105 includes steps S601 to S604:

[0178] S601: For every two optimized video segments with consistent marks, determine the sequence of the two optimized video segments with consistent marks according to position information of the optimized video segments.

[0179] The location information of the optimized video segments may be start and end timestamps, and the sequence of the two optimized video segments with consistent marks may be determined according to the timestamps.

[0180] Then, similar to the smoothing process in step S104, for optimizing the video clips , taken from the end Frame, remember , for optimizing video clips , from the beginning Frame, remember .

[0181] S602: Extract an end frame set of the first optimized video segment according to the position information of the first optimized video segment.

[0182] Specifically, similar to the smoothing process of the storyboard in step S104, for optimizing the video clips (First optimized video), taken from the end Frame, get the end frame set of the first optimized video segment, record .

[0183] S603: Extract a start frame set of the second optimized video segment according to the position information of the second optimized video segment.

[0184] Specifically, similar to the smoothing process of the storyboard in step S104, for optimizing the video clips (Second optimized video), taken from the beginning Frames, get the starting frame set of the second optimized video segment, record .

[0185] S604: Use the most similar frame pair in the end frame set of the first optimized video segment and the start frame set of the second optimized video segment as the best transition point.

[0186] Specifically, similar to the smoothing processing of the storyboard in step S104, the end frame set of the first optimized video segment is For each frame in Combination characteristics of frames (fusion of color and deep learning features), and the set of starting frames for the second optimized video segment For each frame in Frame combination features (Integration of color and deep learning features), and calculate the similarity between each two combined feature vectors. In the specific implementation process of this application, the following formula can be used to calculate the similarity between each two combined feature vectors:

[0187] ;

[0188] And use the following calculation formula to find the frame pair with the maximum similarity:

[0189] ;in, and Optimize video clips for the first The end frame set and the second optimized video segment The frame index of the frame pair with the maximum similarity in the starting frame set is used, and the frame pair with the maximum similarity is used as the optimal transition point.

[0190] S106: Merge the two optimized video clips with consistent marks at the best transition point to obtain a target video.

[0191] Continuing with the above example, if the first optimized video segment The end frame set and the second optimized video segment If the similarity of the frame pair with the maximum similarity in the starting frame set is greater than the transition point threshold, then the frame index of the frame pair with the maximum similarity is directly and , the first optimized video and the second optimized video are spliced. The transition point threshold is preset and changed by technicians, experts, etc., and is not limited here.

[0192] If the first optimized video clip The end frame set and the second optimized video segment If the similarity of the frame pair with the maximum similarity in the start frame set is not greater than the transition point threshold, the number of frames T for the transition to continue is set, where T is preset and changed by technicians, experts, etc. and is not limited here.

[0193] Then the ending transition frame sequence will be formed:

[0194] ;

[0195] Start the transition frame sequence:

[0196] ;

[0197] It should be noted that it is necessary to ensure and There are enough frames for transition. If not, adjust and or T, no limitation is given here.

[0198] Specifically, at the merge ( and ) for the first optimized video segment The ending transition frame and the second optimized video clip The starting transition frames are spliced ​​using common video processing transition effects such as overlay, fade in and fade out to obtain the target video.

[0199] To further enhance the user experience, in another embodiment of the present application, an implementation of the video synthesis method further includes steps S701 to S703:

[0200] S701. For each high-quality video segment, extract all user interaction data based on the location information of the high-quality video segment.

[0201] Among them, user interaction data includes but is not limited to text data (such as barrage, comments, etc.), like data (which is binary data, indicating whether there is a like behavior), progress drag data (which is numerical data, indicating the dragging behavior of the progress bar), etc., and is not limited here.

[0202] S702: Determine emotional high points in the high-quality video clips based on the facial expression videos of all users when they watch the high-quality video clips and all user interaction data.

[0203] Optionally, in another embodiment of the present application, an implementation of step S702 includes steps S801 to S804:

[0204] S801. Determine the average emotion intensity of users in each frame of the high-quality video clip based on the facial expression videos of all users when watching the high-quality video clip.

[0205] Specifically, referring to the method of obtaining the user expression score sequence in step S202, firstly, according to the expression video of the user when watching the high-quality video clip, according to the user expression scores of all frames, a user expression score sequence is generated, then the user expression score sequence is set. Users in progress time The "true" expression score of , then all users at time Average emotional intensity for:

[0206] ;in, The total number of users who viewed the video.

[0207] It is understandable that in order to reduce the impact of noise, in the specific implementation process of the present application, each user expression score sequence can also be smoothed using methods such as sliding average and Gaussian filtering, which is not limited here.

[0208] S802: Determine the total user interaction emotion intensity of each frame of the high-quality video clip based on all user interaction data.

[0209] If the user's interactive data includes: text data (such as barrage, comments, etc.), like data (which is binary data, indicating whether there is a like behavior), progress drag data (which is numerical data, indicating the dragging behavior of the progress bar).

[0210] 1. To calculate the emotional intensity of the total text data, you first need to determine the emotional type of the text data, then determine the emotional intensity of the text data based on the emotional type of the text data, and then calculate the emotional intensity of the total text data :

[0211] ;

[0212] in, For the The sentiment intensity score corresponding to the text data.

[0213] First, use the trained natural language model to train each text data Classify emotions and get emotion types , ,in, is a trained natural language model.

[0214] Statistics progress time The number of occurrences of each sentiment type in all text data ;

[0215] in, ;in, is the Kronecker function.

[0216] ;

[0217] Finally, in the progress time , select the emotion type with the highest number of occurrences as the user emotion type:

[0218] ;

[0219] In the specific implementation process of this application, after determining the user's emotion type, the emotion is divided into two dimensions, namely, valence (positive and negative) and arousal (intensity), through an emotion dimension model, such as the valence-arousal model. Table 2 shows an example of a mapping table from emotion type to basic emotion dimensions provided by the present invention:

[0220] Table 2

[0221] The valence V represents the positivity of the emotion, ranging from −1 (extremely negative) to +1 (extremely positive). The arousal A represents the intensity of the emotion, ranging from −1 (extremely low arousal) to +1 (extremely high arousal).

[0222] In the specific implementation process of this application, the emotional intensity of the text data can be calculated using the following calculation formula:

[0223] ;in, Indicates the valence value corresponding to the emotion type, Indicates the arousal value corresponding to the emotion type.

[0224] Taking laughter as an example, the emotion type number in Table 2 is replaced with the specific data value, that is, valence is +1, arousal is +1, that is, ;

[0225] Taking boredom as an example, the emotion type number in Table 2 is replaced with the specific data value, that is, the valence is -0.8 and the arousal is -0.4, that is, .

[0226] It should be noted that since "boredom" is a negative emotion with low arousal, its arousal is negative, resulting in a positive emotion intensity score, but the value is small.

[0227] As shown in Table 3, it is an example of a complete emotion intensity score table provided by the present invention.

[0228] Table 3

[0229] It should be noted that for negative emotions with low arousal such as "boredom" and "sadness", since both V and A are negative, the emotional intensity score is This reflects that even if the emotion is negative, the impact on the intensity of the interaction emotion may be positive, but the value is low, due to low arousal.

[0230] 2. Calculate the emotional intensity of the likes data. Specifically, the following calculation formula can be used to calculate the emotional intensity of the likes data. :

[0231] ;in, Progress time Number of likes. The sentiment intensity weight of the like (e.g. +1).

[0232] 3. Calculate the total progress drag emotional intensity: First of all, it should be noted that when calculating the emotional intensity of drag data, two situations need to be considered: (1) From the progress time Drag away, (2) drag to the progress time .

[0233] (1) From the progress time Drag away: First define the number of drag aways Then, based on the number of drags, the emotional intensity of the drag data is calculated. ,in, is the intensity weight of the emotion of dragging away the video. If it is less than 0, it means negative emotion (i.e., the video is dragged away because the user does not want to watch the video as a whole).

[0234] (2) Drag to the progress time :First define the number of drags , frequency threshold Because a dragging behavior cannot be judged as a user's behavior with a clear purpose, for example, the user may simply want to leave the original progress point, this application adds a frequency parameter to improve data accuracy.

[0235] when ≥ When dragging the emotional intensity of the data ,in, is the drag emotion intensity weight, which is greater than 0, indicating positive emotion, otherwise .

[0236] So, the overall progress drags down the emotional intensity. The calculation method can be as follows:

[0237] .

[0238] Then the total interactive emotion intensity of the frame is .

[0239] In the specific implementation process of this application, it is also possible to , , Set the corresponding weight parameters. Generally speaking, the weight of text emotion intensity is greater than that of likes and drags, so as to ensure that the total interactive emotion intensity value matches the emotion type and avoids generating unreasonable emotion high points such as "'boredom' emotion with extremely high emotion intensity value". The weight can be adjusted as needed and is not limited here.

[0240] It is understandable that during the application of the present application, there may be situations such as no text data, no progress dragging data, no like data, no text data and no like data, no like data and no dragging data. Taking the absence of like data as an example, the emotional intensity of the like data can be recorded as 0, and no limitation is made here.

[0241] S803 . For each frame of the high-quality video clip, determine a comprehensive user emotion score for the frame according to the average user emotion intensity of the frame and the total user interaction emotion intensity of the frame.

[0242] Continuing with the above example, the user comprehensive emotion score of the frame is determined based on the average emotion intensity of the user in the frame and the total interaction emotion intensity of the user in the frame. The specific calculation method can be calculated using the following formula:

[0243] ;in, and is the weight coefficient, which can be adjusted as needed and is not limited here.

[0244] S804: If the user comprehensive emotion score of the frame is greater than the emotion high point threshold, the time point of the frame is used as the emotion high point in the high-quality video clip.

[0245] Among them, the emotional high point threshold is set and changed in advance by technicians, experts, etc. and is not limited here.

[0246] It is understandable that in the same high-quality video clip, there may be multiple emotional high points, that is, there are multiple frames where the user's comprehensive emotional score is greater than the emotional high point threshold.

[0247] S703 , for each emotional high point in the high-quality video clip, according to the target emotional type of the emotional high point, insert a post-effect corresponding to the target emotional type at the emotional high point position.

[0248] Optionally, in another embodiment of the present application, an implementation of step S703 includes steps S901 to S907:

[0249] S901: For each emotional high point in the high-quality video clip, determine whether there is text data in the user interaction data at the emotional high point position.

[0250] Specifically, if it is determined that there is text data in the user interaction data at the emotional high point position, step S902 is executed; if it is determined that there is no text data in the user interaction data at the emotional high point position, step S903 is executed.

[0251] S902: Determine the emotion type of the emotional high point according to the text data.

[0252] Specifically, the implementation method of determining the emotion type of the emotional peak according to the text data can refer to the implementation method in step S802, which will not be repeated here.

[0253] S903: Determine that the emotion type of the emotion high point is a general emotion type.

[0254] S904: Determine the total interactive emotion intensity of the emotional high point according to the user interaction data.

[0255] Specifically, the implementation method of determining the emotion type of the emotional peak according to the text data can refer to the implementation method in step S802, which will not be repeated here.

[0256] S905 . Determine the number of the target emotion type according to the emotion type of the emotion high point and the total interactive emotion intensity of the emotion high point.

[0257] S906: Determine the post-effect corresponding to the target emotion type according to the number of the target emotion type.

[0258] In the specific implementation process of this application, the preset emotion type-total interactive emotion intensity-later effect number can be used to determine the target emotion type and the later effect number corresponding to the target emotion type according to the emotion type of the emotional high point and the total interactive emotion intensity of the emotional high point. As shown in Table 4, an example of the mapping relationship between the preset emotion type-total interactive emotion intensity-later effect number is shown:

[0259] Table 4

[0260] In the specific implementation process of the present application, the emotion type of the emotional high point is the emotion level 1 type in Table 4, and then the target emotion type (emotion level 2 type in Table 4) is determined according to the total interactive emotion intensity of the emotional high point, and then the later effect number corresponding to the number of the target emotion type (emotion level 2 type in Table 4) is determined.

[0261] S907, inserting the post-effect corresponding to the target emotion type at the emotional high point.

[0262] Specifically, the emotional high point can be determined according to the mapping relationship in Table 4 above. A three-dimensional vector:

[0263] ;in, Number the emotion type. Number the post-effects. The time point of the emotional peak in the video.

[0264] Understandably, Can point to a collection of post effects If it points to a post-effects collection , then prioritize the post-effects (they can be pre-sorted by staff, experts, etc., or they can be sorted according to the number of likes, user feedback, etc., which is not limited here), and select the most appropriate effect as the final inserted post-effect.

[0265] Specifically, the method for determining the time period for inserting the post-effect includes, but is not limited to, determining the duration of the post-effect according to the type of post-effect and the nature of the emotional point. , the inserted time period is:

[0266] , is the time point of the emotional peak in the video, is the starting time point of the inserted time period, The starting time point of the inserted time period. To avoid conflicts, perform post-effect superposition detection in the same time period to ensure that multiple post-effect superpositions do not occur in the same time period. If there is a conflict, adjust the insertion time period according to the post-effect priority.

[0267] Specifically, the application method of the post-effect includes but is not limited to: audio effect insertion, mixing the audio effect with the sound track of the original video during the insertion time period. Visual effect insertion, superimposing animation, pictures, text, special effects and other visual effects on the video frame during the insertion time period. As needed, setting the transparency α of the visual effect in the range of 0-1. Determining the position coordinates (x, y) of the visual effect in the video frame.

[0268] In the specific implementation process of the present application, all post-effects are inserted after the emotional high point to generate a new video stream, and the new video can be encoded according to the target format to output the final video file.

[0269] In the specific implementation process of this application, for overlapping emotional high points, the insertion time of the post-effects is adjusted according to the priority of the emotion type to avoid conflicts and ensure that the visual effect of the video is natural and smooth.

[0270] This application provides intelligent post-production effects to be inserted into the video, so that the final generated video not only accurately reflects the user's interests, but also enhances the viewing effect at key emotional nodes.

[0271] From the above scheme, it can be seen that the present application provides a video synthesis method. First, the video watched by the user and the expression video of the user when watching the video are input into the high-quality video segment recognition model, and the position information of the high-quality video segment in the video watched by the user is output; then, for each high-quality video segment, based on the position information of the high-quality video segment, the consistency score of the high-quality video segment is determined; if the difference in the consistency score of two high-quality video segments is within the consistency threshold, the two high-quality video segments are marked for consistency; then, for each high-quality video segment, based on the position information of the high-quality video segment, the high-quality video segment is subjected to split-screen softening processing to obtain an optimized video segment; finally, for each two optimized video segments with consistency marks, the best transition point is determined; at the best transition point, the two optimized video segments with consistency marks are merged to obtain a target video. The present invention generates a target video by combining user emotions, so that the generated target video accurately reflects the user's interest points.

[0272] The present application embodiment provides a video synthesis device, such as Figure 2 As shown, specifically including:

[0273] The high-quality video segment determination unit 201 is used to input the video watched by the user and the expression video of the user when watching the video into the high-quality video segment recognition model, and output the location information of the high-quality video segment in the video watched by the user.

[0274] Among them, the high-quality video clip recognition model is obtained by training a deep learning model based on a training sample set; the training sample set includes at least one training sample; the training sample includes the original video of the training sample, the expression video of the user when watching the original video of the training sample, and the location information of the real high-quality clips in the original video of the training sample.

[0275] Optionally, in another embodiment of the present application, an implementation of a construction unit of a high-quality video segment recognition model includes:

[0276] The training sample set construction unit is used to construct the training sample set.

[0277] The training sample set includes at least one training sample; the training sample includes the original video of the training sample, the expression video of the user when watching the original video of the training sample, and the real position information of the high-quality clips in the original video of the training sample.

[0278] The analysis unit is used to input the original video of the training sample and the expression video of the user when watching the original video of the training sample into the deep learning model, and output the predicted position information of the high-quality clips in the original video of the training sample.

[0279] Among them, the deep learning model determines the expression features of the user in each frame of the expression video when the user watches the original video of the training sample; determines the user's emotion type based on the user's expression features; determines the weight of the expression features based on the user's emotion type; determines the user's expression score for the frame based on the score of the expression features and the weight of the expression features; and determines the predicted position information of high-quality clips in the original video of the training sample based on the user's expression scores of all frames.

[0280] An adjustment unit is used to adjust the parameters in the deep learning model by using the error between the predicted position information of the high-quality clips in the original video of the training sample and the actual position information of the high-quality clips in the original video of the training sample, until the error between the predicted position information of the high-quality clips in the original video of the training sample and the actual position information of the high-quality clips in the original video of the training sample meets a preset convergence condition, and the deep learning model is used as a high-quality video clip recognition model.

[0281] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0282] The consistency processing unit 202 is configured to determine a consistency score of each high-quality video segment based on the position information of the high-quality video segment.

[0283] Optionally, in another embodiment of the present application, an implementation of the consistency processing unit 202 includes:

[0284] The multi-dimensional feature information extraction unit is used to extract multi-dimensional feature information of the video from the video watched by the user based on the position information of each high-quality video segment.

[0285] The feature similarity calculation unit is used to calculate the feature similarity of each dimension based on the feature information of each dimension.

[0286] The consistency score calculation unit is used to determine the consistency score of the high-quality video clips according to the feature similarity of all dimensions.

[0287] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0288] The consistency marking unit 203 is configured to mark the two high-quality video clips as being consistent if the difference in consistency scores between the two high-quality video clips is within a consistency threshold.

[0289] The frame softening unit 204 is used to perform frame softening processing on each high-quality video segment based on the position information of the high-quality video segment to obtain an optimized video segment.

[0290] Optionally, in another embodiment of the present application, an implementation of the split mirror softening unit 204 includes:

[0291] The frame set extraction unit is used to extract a frame set for each high-quality video segment according to the position information of the high-quality video segment.

[0292] The frame set is divided into a start frame set, an end frame set and a whole frame set.

[0293] The feature processing unit is used to perform feature processing on each frame set to obtain a target feature vector of the frame set.

[0294] The first determining unit is used to determine the similarity of the starting storyboard according to the target feature vector of the starting frame set and the target feature vector of the entire frame set.

[0295] The second determining unit is used to determine the ending storyboard similarity according to the target feature vector of the ending frame set and the target feature vector of the entire frame set.

[0296] The continuity evaluation unit is used to determine the continuity evaluation result of the high-quality video clip according to the similarity of the starting storyboard and the similarity of the ending storyboard.

[0297] The optimized video generation unit is used to generate optimized video segments according to the coherence evaluation results of the high-quality video segments.

[0298] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0299] Optionally, in another embodiment of the present application, an implementation of the feature processing unit includes:

[0300] The first feature processing subunit is used to extract the color histogram of each frame image in each frame set to obtain a first feature vector.

[0301] The second feature processing subunit is used to input the image of the frame into the feature extraction model and output a second feature vector.

[0302] The feature fusion unit is used to perform feature fusion on the first feature vector and the second feature vector to obtain a fusion vector of the frame.

[0303] The feature aggregation unit is used to perform feature aggregation on the fusion vectors of all frames in the frame set to obtain a feature vector of the frame set.

[0304] The feature normalization unit is used to normalize the feature vectors of all frame sets to obtain the target feature vectors of the frame sets.

[0305] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0306] The optimal transition point determination unit 205 is configured to determine the optimal transition point for each two optimized video segments having consistent marks.

[0307] Optionally, in another embodiment of the present application, an implementation of the optimal transition point determination unit 205 includes:

[0308] The sequence determination unit is used to determine the sequence of each two optimized video segments with consistent marks according to the position information of the optimized video segments.

[0309] The second extraction unit is used to extract the end frame set of the first optimized video segment according to the position information of the first optimized video segment.

[0310] The third extraction unit is used to extract a start frame set of the second optimized video segment according to the position information of the second optimized video segment.

[0311] The optimal transition point determination subunit uses the most similar frame pair between the end frame set of the first optimized video segment and the start frame set of the second optimized video segment as the optimal transition point.

[0312] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0313] The merging unit 206 is configured to merge the two optimized video segments with consistent marks at the best transition point to obtain a target video.

[0314] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 1 As shown, no further description is given here.

[0315] Optionally, in another embodiment of the present application, an implementation of the video synthesis device further includes:

[0316] The first extraction unit is used to extract all user interaction data for each high-quality video segment based on the position information of the high-quality video segment.

[0317] The emotional high point determination unit is used to determine the emotional high point in the high-quality video clip based on the expression videos of all users when watching the high-quality video clip and all user interaction data.

[0318] The insertion unit is used to insert a post-effect corresponding to a target emotion type at each emotion high point in the high-quality video clip according to the target emotion type of the emotion high point.

[0319] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0320] Optionally, in another embodiment of the present application, an implementation of the emotional high point determination unit includes:

[0321] The user average emotion intensity determination unit is used to determine the user average emotion intensity of each frame of the high-quality video segment according to the expression videos of all users when watching the high-quality video segment.

[0322] The user total interaction emotion intensity determination unit is used to determine the user total interaction emotion intensity of each frame of the high-quality video clip according to all user interaction data.

[0323] The user comprehensive emotion score determination unit is used to determine the user comprehensive emotion score of each frame of the high-quality video clip according to the user average emotion intensity of the frame and the user total interactive emotion intensity of the frame.

[0324] The emotion high point determination subunit is used to regard the time point of the frame as the emotion high point in the high-quality video clip if the user's comprehensive emotion score of the frame is greater than the emotion high point threshold.

[0325] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0326] Optionally, in another embodiment of the present application, an implementation of the insertion unit includes:

[0327] The judgment unit is used to judge whether there is text data in the user interaction data at the emotional high point position for each emotional high point in the high-quality video clip.

[0328] The emotion type determination unit is used to determine the emotion type of the emotion high point according to the text data if the determination unit determines that there is text data in the user interaction data at the emotion high point position.

[0329] The emotion type determination unit is further configured to determine that the emotion type of the emotion high point is a general emotion type if the determination unit determines that there is no text data in the user interaction data at the emotion high point.

[0330] The total interactive emotion intensity determination unit is used to determine the total interactive emotion intensity of the emotional high point according to the user interaction data.

[0331] The target emotion type determination unit is used to determine the number of the target emotion type according to the emotion type of the emotion high point and the total interactive emotion intensity of the emotion high point.

[0332] The post-effect determination unit is used to determine the post-effect corresponding to the target emotion type according to the number of the target emotion type.

[0333] The insertion subunit is used to insert the post-effect corresponding to the target emotion type at the emotional high point.

[0334] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0335] As can be seen from the above scheme, the present application provides a video synthesis device. First, the high-quality video segment determination unit 201 inputs the video watched by the user and the expression video of the user when watching the video into the high-quality video segment recognition model, and outputs the position information of the high-quality video segment in the video watched by the user; then, the consistency processing unit 202 determines the consistency score of the high-quality video segment based on the position information of the high-quality video segment for each high-quality video segment; if the difference in the consistency score of the two high-quality video segments is within the consistency threshold, the consistency marking unit 203 marks the two high-quality video segments as consistency; then, the split-screen softening unit 204 performs split-screen softening processing on the high-quality video segment based on the position information of the high-quality video segment for each high-quality video segment to obtain an optimized video segment; finally, the best transition point determination unit 205 determines the best transition point for each two optimized video segments with consistency marks; the merging unit 206 merges the two optimized video segments with consistency marks at the best transition point to obtain the target video. The present invention generates a target video by combining user emotions, so that the generated target video accurately reflects the user's interest points.

[0336] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0337] Another embodiment of the present application provides an electronic device, including:

[0338] One or more processors.

[0339] A storage device having one or more programs stored thereon.

[0340] When the one or more programs are executed by the one or more processors, the one or more processors implement the video synthesis method as described in the above embodiments.

[0341] Another embodiment of the present application provides a computer storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the video synthesis method as described in the above embodiment is implemented.

[0342] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0343] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0344] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0345] Another embodiment of the present application provides a computer program product, which, when executed, is used to perform the above-mentioned video synthesis method.

[0346] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0347] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in this application is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing this application.

[0348] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0349] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above application concept. For example, the above features are replaced with (but not limited to) technical features with similar functions applied in the present application.

Claims

1. A video synthesis method, characterized in that: include: Inputting the video watched by the user and the expression video of the user when watching the video into the high-quality video segment recognition model, and outputting the location information of the high-quality video segment in the video watched by the user; For each high-quality video segment, determining a consistency score of the high-quality video segment based on the position information of the high-quality video segment; If the difference in consistency scores between two high-quality video clips is within a consistency threshold, the two high-quality video clips are marked as consistent; For each high-quality video segment, based on the position information of the high-quality video segment, the high-quality video segment is subjected to a split-frame softening process to obtain an optimized video segment; For every two optimized video clips with consistent marks, determine the best transition point; The two optimized video segments with consistent marks are merged at the optimal transition point to obtain a target video.

2. The video synthesis method according to claim 1, characterized in that: Also includes: For each high-quality video clip, extract all user interaction data based on the location information of the high-quality video clip; Determine the emotional high points in the high-quality video clips based on the facial expression videos of all users when they watch the high-quality video clips and all user interaction data; For each emotional high point in the high-quality video clip, according to the target emotional type of the emotional high point, a post-effect corresponding to the target emotional type is inserted at the position of the emotional high point.

3. The video synthesis method according to claim 1, characterized in that: The method for constructing the high-quality video clip recognition model comprises: Constructing a training sample set; wherein the training sample set includes at least one training sample; the training sample includes an original video of the training sample, a video of a user's facial expression when watching the original video of the training sample, and real position information of high-quality clips in the original video of the training sample; The original video of the training sample and the expression video of the user when watching the original video of the training sample are input into the deep learning model, and the predicted position information of the high-quality clips in the original video of the training sample is output; wherein the deep learning model determines the expression features of the user in each frame in the expression video of the user when watching the original video of the training sample; determines the emotion type of the user according to the expression features of the user; determines the weight of the expression features according to the emotion type of the user; determines the user expression score of the frame according to the score of the expression features and the weight of the expression features; determines the predicted position information of the high-quality clips in the original video of the training sample according to the user expression scores of all frames; The error between the predicted position information of the high-quality clips in the original video of the training sample and the actual position information of the high-quality clips in the original video of the training sample is used to adjust the parameters in the deep learning model until the error between the predicted position information of the high-quality clips in the original video of the training sample and the actual position information of the high-quality clips in the original video of the training sample meets the preset convergence condition, and the deep learning model is used as a high-quality video clip recognition model.

4. The video synthesis method according to claim 1, characterized in that: The step of determining, for each high-quality video segment, a consistency score of the high-quality video segment based on the position information of the high-quality video segment includes: For each high-quality video clip, based on the position information of the high-quality video clip, extract multi-dimensional feature information of the video from the video watched by the user; For the feature information of each dimension, calculating the feature similarity of the dimension; The consistency score of the high-quality video clip is determined according to the feature similarities of all the dimensions.

5. The video synthesis method according to claim 1, characterized in that: The step of performing a split-screen softening process on each high-quality video segment based on the position information of the high-quality video segment to obtain an optimized video segment includes: For each high-quality video segment, extract a frame set according to the position information of the high-quality video segment; wherein the frame set is divided into a start frame set, an end frame set and an overall frame set; For each frame set, feature processing is performed on the frame set to obtain a target feature vector of the frame set; Determine the similarity of the starting storyboard according to the target feature vector of the starting frame set and the target feature vector of the entire frame set; Determine the ending storyboard similarity according to the target feature vector of the ending frame set and the target feature vector of the entire frame set; Determining a coherence evaluation result of the high-quality video segment according to the start storyboard similarity and the end storyboard similarity; An optimized video segment is generated according to the coherence evaluation result of the high-quality video segment.

6. The video synthesis method according to claim 5, characterized in that: The step of performing feature processing on each frame set to obtain a target feature vector of the frame set includes: For each frame image in each frame set, extract the color histogram of the frame image to obtain a first feature vector; Input the image of the frame into the feature extraction model, and output a second feature vector; Performing feature fusion on the first feature vector and the second feature vector to obtain a fusion vector of the frame; Performing feature aggregation on the fusion vectors of all frames in the frame set to obtain a feature vector of the frame set; The feature vectors of all frame sets are normalized to obtain the target feature vector of the frame set.

7. The video synthesis method according to claim 1, characterized in that: The step of determining the best transition point for each two optimized video segments having consistent marks includes: For every two optimized video segments with consistent marks, determine the order of the two optimized video segments with consistent marks according to the position information of the optimized video segments; Extracting an end frame set of the first optimized video segment according to the position information of the first optimized video segment; Extracting a start frame set of the second optimized video segment according to the position information of the second optimized video segment; The most similar frame pair between the end frame set of the first optimized video segment and the start frame set of the second optimized video segment is used as the best transition point.

8. The video synthesis method according to claim 2, characterized in that: Determining the emotional high point in the high-quality video clip according to the facial expression videos of all users when watching the high-quality video clip and all user interaction data includes: Determine the average emotion intensity of users for each frame of the high-quality video clip based on the expression videos of all users when watching the high-quality video clip; Determine the total user interaction emotion intensity of each frame of the high-quality video clip based on all user interaction data; For each frame of the high-quality video clip, determining a user comprehensive emotion score for the frame according to the average emotion intensity of the user for the frame and the total interactive emotion intensity of the user for the frame; If the user comprehensive emotion score of the frame is greater than the emotion high point threshold, the time point of the frame is used as the emotion high point in the high-quality video clip.

9. The video synthesis method according to claim 8, characterized in that: For each emotional high point in the high-quality video clip, inserting a post-effect corresponding to the target emotional type at the emotional high point position according to the target emotional type of the emotional high point includes: For each emotional high point in the high-quality video clip, determining whether there is text data in the user interaction data at the emotional high point position; If it is determined that there is text data in the user interaction data at the emotional high point, determining the emotional type of the emotional high point according to the text data; If it is determined that there is no text data in the user interaction data at the emotional high point, determining that the emotional type of the emotional high point is a general emotional type; Based on user interaction data, determine the total interactive emotion intensity of emotional high points; Determining a number of a target emotion type according to the emotion type of the emotion high point and the total interactive emotion intensity of the emotion high point; Determining a later effect corresponding to the target emotion type according to the number of the target emotion type; Insert the post-effect corresponding to the target emotion type at the emotional high point.

10. A video synthesis device, characterized in that: include: A high-quality video segment determination unit, used to input the video watched by the user and the expression video of the user when watching the video into the high-quality video segment recognition model, and output the location information of the high-quality video segment in the video watched by the user; A consistency processing unit, configured to determine, for each high-quality video segment, a consistency score of the high-quality video segment based on position information of the high-quality video segment; a consistency marking unit, configured to mark the two high-quality video clips as consistent if the difference in consistency scores between the two high-quality video clips is within a consistency threshold; A frame softening unit is used to perform frame softening processing on each high-quality video segment based on the position information of the high-quality video segment to obtain an optimized video segment; An optimal transition point determination unit, used for determining an optimal transition point for each two optimized video segments having consistent marks; The merging unit is used to merge the two optimized video segments with consistent marks at the optimal transition point to obtain a target video.

Images