Auxiliary video material automatic generation method, system and equipment

Abstract

The invention is suitable for the technical field of videos, and provides an auxiliary video material automatic generation method, which comprises the following steps: analyzing a main video, and extracting the structural features of the main video, the structural features comprising a visual semantic vector, a video atmosphere code and motion trail data; inputting the visual semantic vector and the video atmosphere code into a condition editor to generate a condition embedding sequence; inputting the motion trail data into a motion editor to generate a structure guide graph sequence; inputting the condition embedding sequence and the structure guide graph sequence into a video generation model, and generating a plurality of candidate auxiliary materials in combination with the random noise latent image; and for each candidate auxiliary material, performing consistency evaluation in combination with the main video, and determining the auxiliary video material. According to the invention, the visual semantic vector and the video atmosphere code of the main video are extracted and utilized as the generation conditions, so that the accurate guidance of the generated content is realized, and the problems of information loss and ambiguity of the traditional text cue word are overcome.

Description

Technical Field

[0001] This application belongs to the field of video technology, and in particular relates to a method, system and device for automatically generating auxiliary video materials. Background Technology

[0002] In the video production process, the proper combination of main video and auxiliary video materials is an important means to improve video quality, which is conducive to enhancing the creative expression of video production and generating high-quality video content. The generation of auxiliary video materials in the current technology usually relies on keyword matching generation methods in existing material libraries, which lacks in-depth understanding of video content and creative insertion, resulting in limited material selection and insufficient creative expression during the video production process.

[0003] With the development of artificial intelligence, a method for automatically generating auxiliary video materials using AI large models has emerged. Specifically, this involves analyzing the content of the main video to generate text descriptions, generating text prompts based on the text descriptions, and then inputting the text prompts into the AI ​​large model to automatically generate auxiliary video materials.

[0004] However, the conversion from video to text in the aforementioned auxiliary video materials generated by large AI models results in a certain degree of loss of video information. In particular, detailed visual textures, complex motion patterns, and abstract emotional atmospheres are difficult to describe accurately with text. Secondly, the text prompts themselves are ambiguous, and the same prompts may generate content with completely different styles, thus making it difficult to fundamentally guarantee the consistency between the generated auxiliary video materials and the main video. Summary of the Invention

[0005] This application provides a method, system, and device for automatically generating auxiliary video materials, which can solve one of the problems in the prior art mentioned above.

[0006] In a first aspect, embodiments of this application provide a method for automatically generating auxiliary video materials, including: The main video is analyzed to extract its structured features, which include visual semantic vectors, video atmosphere encoding, and motion trajectory data. Input the visual semantic vector and the video atmosphere encoding into the conditional editor to generate a conditional embedding sequence; The motion trajectory data is input into the motion editor to generate a sequence of structured guidance diagrams; The conditional embedding sequence and the structure guidance map sequence are input into the video generation model, and combined with random noise latent images, multiple candidate auxiliary materials are generated. For each of the candidate auxiliary materials, a consistency evaluation is performed in conjunction with the main video to determine the auxiliary video materials.

[0007] Furthermore, the analysis of the main video extracts its structured features, which include visual semantic vectors, video atmosphere encoding, and motion trajectory data, including: Based on shot boundary detection and content change saliency analysis, keyframe sequences are extracted from the main video; For each keyframe in the keyframe sequence, the core object is identified by the instance segmentation model, and the visual semantic vector of each core object is extracted based on the pre-trained visual encoder. For the keyframe sequence, global image statistical features are calculated using a style feature extraction network, and video atmosphere coding is generated. In consecutive frames of the main video, a target tracking algorithm is used to track a specified object across frames, obtain its motion coordinate sequence, and construct the motion trajectory data of the specified object.

[0008] Furthermore, the extraction of keyframe sequences from the main video based on shot boundary detection and content change saliency analysis includes: A shot boundary detection algorithm is used to detect shot transition points in the main video, and a shot unit is corresponding to two adjacent shot transition points. For each of the aforementioned camera units, a content change significance analysis is performed to generate a content change curve; Based on the local extreme points of the content change curve and in combination with the duration of the shot unit, determine the number of keyframes to be extracted and the candidate keyframes within the shot unit. Cluster the candidate keyframes of all shot units, remove candidate keyframes with redundant visual content, and generate a keyframe sequence.

[0009] Furthermore, the step of calculating global image statistical features for the keyframe sequence through a style feature extraction network and generating video atmosphere coding includes: The keyframe sequence is input into a pre-trained style feature extraction network to calculate the Gram matrix of each keyframe, and the Gram matrix is ​​used as the style statistical feature of the corresponding keyframe. Average pooling is performed on the style statistical features of each keyframe to generate a global style descriptor; The global style descriptor is compressed and mapped to a video atmosphere code, which is a predefined latent space vector.

[0010] Furthermore, the step of tracking a specified object across frames in consecutive frames of the main video using a target tracking algorithm to obtain its motion coordinate sequence and construct the motion trajectory data of the specified object includes: The specified object is determined based on the core object of each keyframe, or through user interaction. A multi-target tracking algorithm is used to continuously predict and update the bounding box and unique identifier of the specified object in the video frame sequence of the main video; Record the sequence of center point coordinates of each specified object in the main video to form the motion trajectory data.

[0011] Furthermore, the step of inputting the visual semantic vector and the video atmosphere encoding into a conditional editor to generate a conditional embedding sequence includes: The visual semantic vector and the video atmosphere encoding are standardized and serialized. The processed visual semantic vector and the video atmosphere code are mapped to the visual semantic adapter and the atmosphere code adapter, respectively, to generate a conditional embedding sequence.

[0012] Furthermore, the step of inputting the motion trajectory data into the motion editor to generate a sequence of structured guidance diagrams includes: Based on the center point coordinates of each time step in the motion trajectory data, a Gaussian heat map centered on the center point coordinates is generated in the two-dimensional coordinate system of the corresponding frame. The Gaussian heatmaps corresponding to each time step in the motion trajectory data are stacked to form a sequence of structure guidance maps. The structure guidance map sequence is convolved and fused with the latent feature map of the video generation model to guide the composition and motion of the generated content in a spatially constrained manner.

[0013] Furthermore, the step of performing a consistency evaluation on each of the candidate auxiliary materials in conjunction with the main video to determine the auxiliary video materials includes: Calculate the temporal coherence score and spatial structure score of the candidate auxiliary materials; A multimodal model is used to extract the candidate auxiliary materials and the main video respectively, and semantic similarity scores are calculated. Based on the temporal coherence score, spatial structure score, and semantic similarity score, a comprehensive quality score is generated for each candidate auxiliary material using a weighted decision model, and the auxiliary video material is determined based on the comprehensive quality score.

[0014] Secondly, embodiments of this application provide an auxiliary video material automatic generation system, including: The first processing module is used to analyze the main video and extract its structured features, which include visual semantic vectors, video atmosphere encoding, and motion trajectory data. The second processing module is used to input the visual semantic vector and the video atmosphere encoding into the condition editor to generate a conditional embedding sequence. The third processing module is used to input the motion trajectory data into the motion editor and generate a sequence of structure guidance diagrams. The fourth processing module is used to input the conditional embedding sequence and the structure guidance map sequence into the video generation model, and combine them with random noise latent images to generate multiple candidate auxiliary materials. The fifth processing module is used to perform a consistency evaluation on each of the candidate auxiliary materials in conjunction with the main video, and to determine the auxiliary video materials.

[0015] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method for automatically generating auxiliary video materials.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium, including a computer program stored in the computer-readable storage medium, which, when executed by a processor, implements the aforementioned method for automatically generating auxiliary video materials.

[0017] The beneficial effects of the embodiments in this application compared with the prior art are: This application discloses an automatic generation method for auxiliary video materials. By extracting and utilizing the visual semantic vector and video atmosphere encoding of the main video as generation conditions, it achieves precise guidance for the generated content, overcoming the information loss and ambiguity problems of traditional text prompts. At the same time, by using motion trajectory data to drive the control network to constrain the underlying constraints of the generation process, it ensures that the final generated auxiliary video materials are physically coordinated with the main video in terms of motion patterns. In addition, during the structured feature extraction process, the extraction of key frames reduces the computational load of subsequent deep analysis. Furthermore, based on the same set of control signals, multiple candidate auxiliary materials are generated in parallel by combining random noise latent images, and combined with multi-dimensional consistency quality assessment, the generation efficiency and quality of auxiliary video materials are improved. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an automatic video material generation method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an auxiliary video material automatic generation system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0021] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0022] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0023] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0024] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0025] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0026] Please see Figure 1 As shown, the present invention is an auxiliary method for automatically generating video materials, comprising the following steps: S100. Analyze the main video and extract the structured features of the main video. The structured features include visual semantic vectors, video atmosphere encoding, and motion trajectory data. This application achieves precise guidance of generated content by extracting and utilizing the visual semantic vector and video atmosphere encoding of the main video as generation conditions, overcoming the information loss and ambiguity problems of traditional text prompts. At the same time, by driving the control network through motion trajectory data to constrain the underlying generation process, it ensures that the final generated auxiliary video material is physically coordinated with the main video in terms of motion mode.

[0027] In some embodiments, step S100 above includes: Based on shot boundary detection and content change saliency analysis, keyframe sequences are extracted from the main video; For each keyframe in the keyframe sequence, the core object is identified by the instance segmentation model, and the visual semantic vector of each core object is extracted based on the pre-trained visual encoder. For the keyframe sequence, global image statistical features are calculated using a style feature extraction network, and video atmosphere coding is generated. In consecutive frames of the main video, a target tracking algorithm is used to track a specified object across frames, obtain its motion coordinate sequence, and construct the motion trajectory data of the specified object.

[0028] In this embodiment, a shot boundary detection algorithm is used to detect the cut points of all shot transitions in the main video. For each shot, a content change saliency analysis is performed. When the content change exceeds a preset threshold, the frame is extracted as a key frame to ensure that significant content evolution can be captured even in long shots. Through the above method, a key frame sequence that can represent the visual content evolution of the main video is generated for subsequent analysis of the main video.

[0029] In some embodiments, the extraction of keyframe sequences from the main video based on shot boundary detection and content change saliency analysis includes: A shot boundary detection algorithm is used to detect shot transition points in the main video, and a shot unit is corresponding to two adjacent shot transition points. For each of the aforementioned camera units, a content change significance analysis is performed to generate a content change curve; Based on the local extreme points of the content change curve and in combination with the duration of the shot unit, determine the number of keyframes to be extracted and the candidate keyframes within the shot unit. Cluster the candidate keyframes of all shot units, remove candidate keyframes with redundant visual content, and generate a keyframe sequence.

[0030] In this embodiment, based on the color histogram, edge direction histogram, and motion vector of the main video, and combined with a preset threshold, the lens cut and gradient boundaries are detected, and the main video is divided into continuous lens units. Specifically, the chi-square distance of the color histogram and the cosine distance of the edge direction histogram between consecutive frames in the main video are calculated in parallel, and the motion intensity is calculated using an optical flow algorithm to determine the motion intensity difference. After normalizing the above-mentioned color histogram chi-square distance, edge direction histogram cosine distance, and motion intensity difference, they are fused to generate the inter-frame difference degree, and an inter-frame difference degree curve is constructed. Based on the inter-frame difference degree curve, the mean and standard deviation of the inter-frame difference degree are calculated, and the dynamic difference degree threshold is calculated based on the mean and standard deviation. Specifically, the difference degree threshold = mean + k * standard deviation, where k is an adjustable parameter. When the difference degree of a certain inter-frame exceeds the difference degree threshold, it is determined that there is a lens boundary at this point, specifically a lens switching point. Then, there is a lens unit between two adjacent lens switching points.

[0031] Furthermore, in this embodiment, for each shot unit, the visual saliency region difference between its consecutive frames is calculated, and combined with the inter-frame feature difference, a content change curve is generated. Based on the local extreme points of the content change curve and combined with the shot unit duration, the number of keyframes to be extracted and candidate keyframes within the shot unit are determined. For all candidate keyframes of shot units, their clustering in the depth feature space is calculated to remove candidate keyframes with visual content redundancy, and finally a keyframe sequence that can represent the global content without redundant information is generated.

[0032] Specifically, the deep learning-based visual saliency model processes each frame of the image in the shot unit to generate a visual saliency heatmap, identifying the key areas of focus for the human eye. The intersection joint value of the visual saliency heatmaps between consecutive frames is calculated, such as calculating the intersection-union ratio of the key areas of two consecutive frames to determine the intersection joint value. If the intersection-union ratio between two frames is low, it indicates that the focus has shifted significantly. Furthermore, based on a pre-trained CNN model, global features of the frames are extracted, and the cosine distance between consecutive frames is calculated to capture the changes in global semantics between consecutive frames. The intersection joint value and the cosine distance are then weighted and fused to generate a content change curve. Understandably, this content change curve produces obvious peaks at points of change such as when a person turns their head or when a new object enters the frame.

[0033] It is worth noting that visual saliency models are pre-trained models based on large-scale image datasets, capable of annotating key regions of images, such as BASNet or U. 2 After training, algorithms such as -Net can segment the regions that the human eye is most likely to focus on from image frames, such as faces, moving objects, and high-contrast regions.

[0034] More specifically, a minimum number of keyframes N_min based on the shot duration is set. In one embodiment, N_min = max(1, floor(shot duration (seconds) / 3)), indicating that in this setting, the number of keyframes is calculated at a density of one frame every 3 seconds, and there is at least one keyframe. Furthermore, on the content change curve, all local maxima points are selected as initial candidate points. If the number of initial candidate points is less than N_min, candidate points are supplemented from the non-extreme segments of the content change curve in descending order of change until the N_min requirement is met. If the number of initial candidate points is too large, a minimum change threshold is set for filtering, such as setting the minimum change threshold to 1.5 times the average of the content change curve. This is used to initially determine candidate keyframes, which avoids information loss due to too few extracted frames in shot units with excessively long shot durations, and also avoids a surge in subsequent computational workload due to too many extracted frames in fast-paced shot units.

[0035] Furthermore, a convolutional neural network pre-trained on a large-scale image dataset is used to extract the depth feature vectors of all candidate keyframes. Cluster analysis is performed on all depth feature vectors to group frames with similar visual content into the same cluster. Then, the frame closest to the cluster center is selected from each cluster as the representative of the visual content to determine the final keyframe sequence.

[0036] In one embodiment, for each frame in the keyframe sequence, the core object is identified by an instance segmentation model, and the visual semantic vector of each core object is extracted by a pre-trained visual encoder. Specifically, the keyframe is processed using a Transformer-based instance segmentation model to obtain pixel-level segmentation masks and object categories. The segmentation mask is a binary image used to label the object instance to which each pixel belongs, i.e., to distinguish different objects. The object category is used to identify the category to which each segmented object belongs, such as "person" or "vehicle". For each identified object, the object region is cropped from the original frame based on its segmentation mask. Then, a visual encoder pre-trained on a large-scale vision-language dataset is used to perform forward propagation on each cropped object region to extract its high-dimensional feature vector as a visual semantic vector. Understandably, this visual semantic vector encodes the visual features of the object (color, shape, texture, etc.) and its association with linguistic concepts, such as a red fire truck.

[0037] In some embodiments, the step of calculating global image statistical features for the keyframe sequence using a style feature extraction network and generating video atmosphere coding includes: The keyframe sequence is input into a pre-trained style feature extraction network to calculate the Gram matrix of each keyframe, and the Gram matrix is ​​used as the style statistical feature of the corresponding keyframe. Average pooling is performed on the style statistical features of each keyframe to generate a global style descriptor; The global style descriptor is compressed and mapped to a video atmosphere code, which is a predefined latent space vector.

[0038] In this embodiment, the keyframe sequence is input into a pre-trained style feature extraction network, which extracts visual texture features, local structural features, and global semantic features from its shallow, middle, and deep convolutional layers. Specifically, the style feature extraction network uses a pre-trained CNN such as VGG or ResNet as its backbone network and is trained on a large-scale image dataset to enable it to capture multi-level features of the image. The shallow layer captures color and basic texture; the middle layer captures complex texture and local patterns; and the deep layer captures high-level, content-related style information. A feature layer is selected from the shallow, middle, and deep layers respectively, and the Gram matrix of the corresponding feature map is calculated. The Gram matrix is ​​specifically the inner product of the feature maps of that layer. Then, the calculated multiple Gram matrices are flattened and concatenated to form the style statistical features of the frame.

[0039] Specifically, in one embodiment, the feature map F of the selected feature layer is reshaped to C×(H*W) in size C×H×W. That is, the feature map of each channel is flattened into a vector, and all channels are concatenated into a matrix with C rows and H*W columns, where C represents the number of channels, H represents the height, and W represents the width. Then, the Gram matrix G = F·F^T is calculated, where F is a C×(H*W) matrix and F^T is its transpose ((h*w)×C). Therefore, the Gram matrix is ​​a C×C matrix. Each element G_{i,j} of the Gram matrix represents the correlation between feature channels i and j. Further, the Gram matrices of the shallow, middle, and deep layers are flattened and converted into vectors. These vectors are concatenated in hierarchical order to form a comprehensive style statistical feature. Thus, this style statistical feature integrates multi-level style features, enabling it to simultaneously obtain the local details and global pattern features of the corresponding keyframe.

[0040] In this embodiment, for all keyframes within a shot, their corresponding style statistical features are arranged in the time dimension. An attention-weighted average pooling method is used to perform a weighted average on the temporally arranged feature vectors, thereby generating a global style descriptor.

[0041] Furthermore, Mel frequency cepstral coefficient features are extracted from the corresponding audio segments of the main video, and a recurrent neural network model is used for emotion classification to output an audio emotion vector. For the facial regions in the keyframe sequence, facial expression features are extracted using a facial expression recognition model and aggregated into a visual emotion vector. The audio emotion vector and the visual emotion vector are fused into a unified emotion feature vector, and then concatenated and fused with a global style descriptor through a fully connected layer. Finally, the fused features are input into an encoder network, and a nonlinear activation function is used to gradually compress the high-dimensional fused features into a latent space of a predetermined dimension. This allows the complex intrinsic relationship between style and emotion to be learned, redundant information to be removed, and video atmosphere coding to be generated, ensuring that the final generated auxiliary video material is highly consistent with the main video in terms of aesthetics and emotion.

[0042] In some embodiments, the step of tracking a specified object across frames in consecutive frames of the main video using a target tracking algorithm to obtain its motion coordinate sequence and construct the motion trajectory data of the specified object includes: The specified object is determined based on the core object of each keyframe, or through user interaction. A multi-target tracking algorithm is used to continuously predict and update the bounding box and unique identifier of the specified object in the video frame sequence of the main video; Record the sequence of center point coordinates of each specified object in the main video to form the motion trajectory data.

[0043] In this embodiment, a focus list is preset. In one embodiment, the focus list contains semantic information of focus categories such as "person", "hand", "vehicle", and "animal". The results of core object identification are matched in the focus list to determine the specified objects worth tracking, or through an interactive specification method, such as providing a graphical interface for users to pause at the corresponding keyframe and select one or more specified objects to be tracked by box selection. Furthermore, an initial bounding box and a unique identifier are created for each object.

[0044] Furthermore, within consecutive frames of the video, a multi-object tracking algorithm continuously associates, predicts, and updates the position and state of each reference object. For example, in one embodiment, a deep learning-based object detector is used to detect each frame of the main video, obtaining candidate bounding boxes for each specified object in the current frame. For tracked objects existing in the previous frame, Kalman filtering is used to predict their expected position in the current frame based on their motion state, marking them as predicted bounding boxes. The apparent feature cosine similarity and spatial location IoU overlap between the predicted bounding box of each tracked object and the candidate bounding boxes in the current frame are calculated, and the apparent feature cosine similarity and spatial location IoU overlap are then calculated. The overlapping data is weighted and fused to construct a cost matrix. The Hungarian algorithm is used for optimal data association. Each tracked target is assigned a detection box in the current frame or marked as "not detected" to achieve cross-frame identity matching. For tracked targets that are successfully matched, the target state is updated with the new detection box. For tracked targets that are not matched, their trajectory prediction is maintained during their brief disappearance. For tracked targets that are continuously unmatched, it is determined that they have left the frame and tracking is terminated. It is worth noting that the appearance features are deep feature vectors that are robust to changes in viewpoint and illumination, extracted by a deep learning network pre-trained on a person re-identification dataset.

[0045] Furthermore, the sequence of center point coordinates of each successfully tracked specified object in the main video is recorded during its lifecycle, and this sequence of center point coordinates is associated with the category semantic information of the specified object to construct motion trajectory data with semantic labels. Specifically, for each successfully tracked reference object, the center point coordinates (x, y) of its bounding box in each frame during the entire occurrence period are recorded to form an initial motion path. A smoothing filtering algorithm is applied to this initial motion path to eliminate detection jitter and generate smoother, more physically reasonable motion trajectory data. This motion trajectory data is then bound to the visual semantic vector of the core object to form motion trajectory data with semantic information.

[0046] S200: Input the visual semantic vector and the video atmosphere encoding into the condition editor to generate a conditional embedding sequence; In some embodiments, step S200 above includes: The visual semantic vector, the video atmosphere encoding, and the motion trajectory data are standardized and serialized. The processed visual semantic vector and the video atmosphere code are mapped to the visual semantic adapter and the atmosphere code adapter, respectively, to generate a conditional embedding sequence.

[0047] In this embodiment, the visual semantic vector and video atmosphere encoding are standardized and serialized to adapt to the input requirements of the video generation model. Specifically, the visual semantic vector and video atmosphere encoding are Z-score standardized to adjust their numerical range to a range that is compatible with the weights of the video generation model, ensuring the stability of training and generation. Furthermore, for static video atmosphere encoding, it is copied in the time dimension to form a sequence with the same number of video frames as the target to be generated. For visual semantic embedding, if there are multiple objects or object state changes in a shot, a corresponding semantic condition sequence needs to be constructed and aligned with the video frames.

[0048] Furthermore, the visual semantic vector and video atmosphere encoding are mapped to vectors with the same dimension as the conditional latent space of the video generation model, generating a conditional embedding sequence. Specifically, the conditional editor includes at least two independent multilayer perceptrons: a visual semantic adapter and an atmosphere encoding adapter. The visual semantic adapter projects the visual semantic vector to a vector with the same dimension as the conditional latent space of the video generation model, i.e., the visual semantic conditional vector. Similarly, the atmosphere encoding adapter projects the video atmosphere encoding to a vector with the same dimension as the conditional latent space of the video generation model, i.e., the video atmosphere conditional vector. Then, the visual semantic conditional vector and the video atmosphere conditional vector are weighted and / or concatenated to form a conditional embedding sequence, which serves as a guiding signal for the cross-attention layer during the denoising process of the video generation model. This guides the generated content to align with the specified semantics and style, ensuring that the final generated candidate auxiliary materials are consistent with the main video in terms of content theme and visual style.

[0049] S300. Input the motion trajectory data into the motion editor to generate a sequence of structure guidance diagrams; In some embodiments, step S300 above includes: Based on the center point coordinates of each time step in the motion trajectory data, a Gaussian heat map centered on the center point coordinates is generated in the two-dimensional coordinate system of the corresponding frame. The Gaussian heatmaps corresponding to each time step in the motion trajectory data are stacked to form a sequence of structure guidance maps. The structure guidance map sequence is convolved and fused with the latent feature map of the video generation model to guide the composition and motion of the generated content in a spatially constrained manner.

[0050] In this embodiment, the motion trajectory data of the specified object is used as the motion guide for the candidate auxiliary material to ensure the coordination between the candidate auxiliary material and the main video in terms of composition and motion mode generation, and to ensure the dynamic consistency when the candidate auxiliary material and the main video are mixed and edited.

[0051] Specifically, motion trajectory data is transformed into a temporally aligned sequence of structured guide maps. For example, based on smoothed motion trajectory data, a Gaussian heatmap is created for each frame. A highlighted area, such as a solid circle, is drawn at the center point coordinates of the tracked object, while the remaining areas are dark. Thus, the Gaussian heatmaps corresponding to each time step constitute the sequence of structured guide maps. Then, the sequence of structured guide maps is passed through a parallel control network module. At each time step, its features are convolved and fused with the latent feature map of the corresponding layer of the video generation model. This guides the composition and motion of the generated content in a spatially constrained manner. Specifically, during the denoising process of the video generation model, the control network module extracts spatial control features from the structured guide map for each time step and convolves and fuses them with the latent feature map generated by the corresponding layer of the video generation model. This ensures that the content generated by the video generation model is spatially constrained to a "reasonable composition that conforms to the motion trajectory".

[0052] It is worth noting that the control network module is a convolutional neural network initialized with zero convolutions and running in parallel with the main UNet structure of the video generation model. In the early stages of training, it does not affect the generation capability of the video generation model. During training, it gradually learns to inject spatial control features from the structure guidance graph into the generation process. When the video generation model generates data, it inputs the latent image of random noise into the UNet of the video generation model and receives the semantic and style features of the conditional embedding sequence. Inside the UNet, a series of convolutional layers and attention layers are used to extract, transform, and fuse the features of the input random noise. In this process, the intermediate result of each layer output is a latent feature map. It can be understood that the latent feature map evolves from shallow to deep, gradually from capturing coarse motion trends to generating fine textures and details.

[0053] S400. Input the conditional embedding sequence and the structure guidance map sequence into the video generation model, and combine them with random noise latent images to generate multiple candidate auxiliary materials; In this embodiment, the video generation model uses random noise latent images, conditional embedding sequences, and structural guidance map sequences as inputs. During the iterative denoising process, dual guidance is implemented. The conditional embedding sequence uses the cross-attention mechanism in UNet to globally influence the high-level semantics and overall atmosphere of the generated content. The structural guidance map sequence uses a parallel control network module to convolve and fuse its spatial control features with the latent feature maps of the corresponding layers of UNet at each time step, guiding the composition and movement of objects to follow the trajectory of the main video. At the same time, different random noise latent images are injected into the same set of control conditions to generate multiple candidate auxiliary materials with differences in detail, providing a selection basis for subsequent evaluation and optimization steps.

[0054] Specifically, this application maps visual semantic vectors and video atmosphere encodings to vectors with the same dimension as the conditional latent space of the video generation model, generating a conditional embedding sequence. Then, through a motion trajectory layer, the motion trajectory data is transformed into a series of temporally aligned structure guide maps. The conditional embedding sequence and the structure guide map sequence, along with random noise latent images, are then input into the video generation model. Through its denoising process, candidate auxiliary materials that are consistent with the main video in terms of semantics, style, and motion are generated.

[0055] S500. For each of the candidate auxiliary materials, a consistency evaluation is performed in conjunction with the main video to determine the auxiliary video materials.

[0056] In this application, during the structured feature extraction process, the extraction of keyframes reduces the computational load of subsequent depth analysis. At the same time, based on the same set of control signals, multiple candidate auxiliary materials are generated in parallel by combining random noise latent images. Combined with multi-dimensional consistency quality assessment, the generation efficiency and generation quality of auxiliary video materials are improved.

[0057] In some embodiments, step S500 above includes: Calculate the temporal coherence score and spatial structure score of the candidate auxiliary materials; A multimodal model is used to extract the candidate auxiliary materials and the main video respectively, and semantic similarity scores are calculated. Based on the temporal coherence score, spatial structure score, and semantic similarity score, a comprehensive quality score is generated for each candidate auxiliary material using a weighted decision model, and the auxiliary video material is determined based on the comprehensive quality score.

[0058] In this embodiment, for each candidate auxiliary material, its temporal coherence score and spatial structure score are calculated to quantify its visual quality. Specifically, for the temporal coherence score, the optical flow field between consecutive frames of the candidate auxiliary material is calculated, and the smoothness and consistency of the optical flow are analyzed to detect whether there are unreasonable flickering, jitter, or object abrupt changes. The temporal coherence score is positively correlated with the smoothness of the optical flow. For the spatial structure score, a deep learning-based image quality assessment model is used to score the sharpness and naturalness of each frame of the candidate auxiliary material, and the mean and variance of the entire candidate auxiliary material are calculated, where the mean represents the average quality and the variance represents the quality fluctuation.

[0059] Specifically, the optical flow algorithm is used to calculate the optical flow field F from frame t (I_t) to frame (I_t+1) in the candidate auxiliary material. t→t+1 , of which F t→t+1 =(u t→t+1 ,v t→t+1 ), u t→t+1 (x,y) represents the horizontal optical flow vector, v t→t+1 (x,y) represents the vertical optical flow vector, and (x,y) represents the pixel coordinates.

[0060] More specifically, for smooth and consistent video, the forward optical flow F t→t+1 With reverse optical flow F t+1→t They should be inverse processes, i.e., F t→t+1 ≈-F t+1→t Furthermore, the optical flow vectors of adjacent pixels should be similar, therefore the forward optical flow F is calculated. t→t+1 With reverse optical flow F t+1→t If the optical flow is exactly the same, the reverse optical flow should be able to move the pixel back to its original position. Specifically, the optical flow difference value... Where H and W represent the frame height and frame width, respectively, and ||·|| represents the L2 norm. Therefore, the average loss of all frames in the candidate auxiliary material is calculated, specifically as follows: , where T represents the total number of frames of the candidate auxiliary material.

[0061] In some embodiments, the loss is converted into a fraction St by a negative exponential function. The smaller the loss, the higher the fraction. Specifically, St = exp(-β × L_flow(t)), where β represents a parameter that controls the rate of fraction decay. When L_flow(t) → 0, it means that the two frames are completely consistent, so St → 1. When L_flow(t) → ∞, it means that the two frames are completely inconsistent, so St → 0.

[0062] Therefore, the temporal coherence score of candidate auxiliary materials T represents the total number of frames of the candidate auxiliary material.

[0063] More specifically, for the spatial structure score, an image quality assessment model pre-trained based on a subjective rating dataset is used to score each frame of the candidate auxiliary material. This image quality assessment model can evaluate the sharpness, naturalness, noise level, etc. of each frame. The spatial structure score S_s=μ / (1+σ) is calculated by calculating the average score μ and standard deviation σ of all frame scores.

[0064] More specifically, deep features of the main video and candidate auxiliary materials are extracted through a pre-trained multimodal model, and the similarity score between the two in the shared semantic space is calculated. Specifically, a model pre-trained on large-scale video-text pair data is used to encode the main video and candidate auxiliary materials into fixed-dimensional feature vectors, and the cosine similarity between the two feature vectors in the high-dimensional semantic space is calculated as the semantic similarity score S_c.

[0065] Furthermore, for each candidate auxiliary material, its temporal coherence score S_t, spatial structure score S_s, and semantic similarity score S_c are recorded. A pre-trained weighted decision model is used to calculate the comprehensive quality score of each candidate material, specifically Q=w1*S_t+w2*S_s+w3*S_c, where w1, w2, and w3 are weight coefficients. Finally, the candidate auxiliary material with the highest comprehensive quality score Q is selected as the final output and chosen as the auxiliary video material.

[0066] Please see Figure 2 As shown, the present invention also provides an auxiliary video material automatic generation system, the system comprising: First processing module 201: used to analyze the main video and extract the structured features of the main video, the structured features including visual semantic vectors, video atmosphere encoding and motion trajectory data; The second processing module 202 is used to input the visual semantic vector and the video atmosphere encoding into the condition editor to generate a conditional embedding sequence; The third processing module 203 is used to input the motion trajectory data into the motion editor to generate a sequence of structure guidance diagrams; The fourth processing module 204 is used to input the conditional embedding sequence and the structure guidance map sequence into the video generation model, and combine them with random noise latent images to generate multiple candidate auxiliary materials; The fifth processing module 205 is used to perform a consistency evaluation on each of the candidate auxiliary materials in conjunction with the main video, and to determine the auxiliary video materials.

[0067] It is understandable that, such as Figure 1The content shown in the embodiments of the automatic generation method for auxiliary video materials is applicable to the embodiments of this automatic generation system for auxiliary video materials. The specific functions implemented in the embodiments of this automatic generation system for auxiliary video materials are the same as those shown in the figure. Figure 1 The method for automatically generating auxiliary video materials shown is the same as the embodiment described above, and the beneficial effects achieved are the same as those described above. Figure 1 The beneficial effects achieved by the illustrated method embodiment for automatically generating auxiliary video materials are also the same.

[0068] It should be noted that the information interaction and execution process between the above systems are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0069] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0070] Please see Figure 3 As shown, this embodiment of the invention also provides a computer device 3, including: a memory 302 and a processor 301, and a computer program 303 stored in the memory 302. When the computer program 303 is executed on the processor 301, it implements the automatic generation method of auxiliary video material as described in any of the above methods.

[0071] The computer device 3 may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will understand that... Figure 3 The computer device 3 is merely an example and does not constitute a limitation on the computer device 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0072] The processor 301 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0073] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 302 may be an external storage device of the computer device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 3. Furthermore, the memory 302 may include both internal and external storage units of the computer device 3. The memory 302 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 302 can also be used to temporarily store data that has been output or will be output.

[0074] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method for automatically generating auxiliary video materials as described in any of the above methods.

[0075] In this embodiment, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / computer device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0076] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for automatically generating auxiliary video materials, characterized in that, include: The main video is analyzed to extract its structured features, which include visual semantic vectors, video atmosphere encoding, and motion trajectory data. Input the visual semantic vector and the video atmosphere encoding into the conditional editor to generate a conditional embedding sequence; The motion trajectory data is input into the motion editor to generate a sequence of structured guidance diagrams; The conditional embedding sequence and the structure guidance map sequence are input into the video generation model, and combined with random noise latent images, multiple candidate auxiliary materials are generated. For each of the candidate auxiliary materials, a consistency evaluation is performed in conjunction with the main video to determine the auxiliary video materials.

2. The method as described in claim 1, characterized in that, The analysis of the main video extracts its structured features, which include visual semantic vectors, video atmosphere encoding, and motion trajectory data, including: Based on shot boundary detection and content change saliency analysis, keyframe sequences are extracted from the main video; For each keyframe in the keyframe sequence, the core object is identified by the instance segmentation model, and the visual semantic vector of each core object is extracted based on the pre-trained visual encoder. For the keyframe sequence, global image statistical features are calculated using a style feature extraction network, and video atmosphere coding is generated. In consecutive frames of the main video, a target tracking algorithm is used to track a specified object across frames, obtain its motion coordinate sequence, and construct the motion trajectory data of the specified object.

3. The method as described in claim 2, characterized in that, The extraction of keyframe sequences from the main video based on shot boundary detection and content change saliency analysis includes: A shot boundary detection algorithm is used to detect shot transition points in the main video, and a shot unit is corresponding to two adjacent shot transition points. For each of the aforementioned camera units, a content change significance analysis is performed to generate a content change curve; Based on the local extreme points of the content change curve and in combination with the duration of the shot unit, determine the number of keyframes to be extracted and the candidate keyframes within the shot unit. Cluster the candidate keyframes of all shot units, remove candidate keyframes with redundant visual content, and generate a keyframe sequence.

4. The method as described in claim 2, characterized in that, The step of calculating global image statistical features for the keyframe sequence using a style feature extraction network and generating video atmosphere coding includes: The keyframe sequence is input into a pre-trained style feature extraction network to calculate the Gram matrix of each keyframe, and the Gram matrix is ​​used as the style statistical feature of the corresponding keyframe. Average pooling is performed on the style statistical features of each keyframe to generate a global style descriptor; The global style descriptor is compressed and mapped to a video atmosphere code, which is a predefined latent space vector.

5. The method as described in claim 2, characterized in that, The step of tracking a specified object across frames in consecutive frames of the main video using a target tracking algorithm to obtain its motion coordinate sequence and construct the motion trajectory data of the specified object includes: The specified object is determined based on the core object of each keyframe, or through user interaction. A multi-target tracking algorithm is used to continuously predict and update the bounding box and unique identifier of the specified object in the video frame sequence of the main video; Record the sequence of center point coordinates of each specified object in the main video to form the motion trajectory data.

6. The method as described in claim 1, characterized in that, The step of inputting the visual semantic vector and the video atmosphere encoding into a conditional editor to generate a conditional embedding sequence includes: The visual semantic vector and the video atmosphere encoding are standardized and serialized. The processed visual semantic vector and the video atmosphere code are mapped to the visual semantic adapter and the atmosphere code adapter, respectively, to generate a conditional embedding sequence.

7. The method as described in claim 1, characterized in that, The step of inputting the motion trajectory data into the motion editor to generate a sequence of structured guided graphs includes: Based on the center point coordinates of each time step in the motion trajectory data, a Gaussian heat map centered on the center point coordinates is generated in the two-dimensional coordinate system of the corresponding frame. The Gaussian heatmaps corresponding to each time step in the motion trajectory data are stacked to form a sequence of structure guidance maps. The structure guidance map sequence is convolved and fused with the latent feature map of the video generation model to guide the composition and motion of the generated content in a spatially constrained manner.

8. The method as described in claim 1, characterized in that, The step of performing a consistency evaluation on each of the candidate auxiliary materials in conjunction with the main video to determine the auxiliary video materials includes: Calculate the temporal coherence score and spatial structure score of the candidate auxiliary materials; A multimodal model is used to extract the candidate auxiliary materials and the main video respectively, and semantic similarity scores are calculated. Based on the temporal coherence score, spatial structure score, and semantic similarity score, a comprehensive quality score is generated for each candidate auxiliary material using a weighted decision model, and the auxiliary video material is determined based on the comprehensive quality score.

9. An automatic video material generation system, characterized in that, include: The first processing module is used to analyze the main video and extract its structured features, which include visual semantic vectors, video atmosphere encoding, and motion trajectory data. The second processing module is used to input the visual semantic vector and the video atmosphere encoding into the condition editor to generate a conditional embedding sequence. The third processing module is used to input the motion trajectory data into the motion editor and generate a sequence of structure guidance diagrams. The fourth processing module is used to input the conditional embedding sequence and the structure guidance map sequence into the video generation model, and combine them with random noise latent images to generate multiple candidate auxiliary materials. The fifth processing module is used to perform a consistency evaluation on each of the candidate auxiliary materials in conjunction with the main video, and to determine the auxiliary video materials.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

Images