Short video production method and system based on artificial intelligence

Abstract

The application discloses a short video production method and system based on artificial intelligence, relates to the technical field of multimedia processing, and comprises the following steps: inputting a shot script into a multi-modal generation engine, generating a visual stream by adopting a lower triangular spatio-temporal attention matrix, and generating a text stream and an audio stream; calculating the neural conduction delay amount between the visual stream and the audio stream based on an audio-visual perception delay prediction model, and generating a corrected audio stream through time axis forward compensation; calculating a cross-modal attention weight matrix according to the node correlation strength of a dynamic causal diagram, projecting the text stream and the corrected audio stream to a joint feature space, and generating a multi-modal fusion feature; inputting the multi-modal fusion feature into a format adaptation engine, embedding a neural implicit watermark after spatio-temporal consistency verification, and encoding the neural implicit watermark into a short video. The application improves the semantic consistency and coordination between text, audio and visual content, thereby realizing high-quality and highly intelligent short video automatic generation as a whole.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multimedia processing, and in particular to a short video production method and system based on artificial intelligence. BACKGROUND

[0002] In recent years, breakthroughs in generative artificial intelligence technology have driven the rapid development of short video automatic generation technology. Deep learning architectures represented by generative adversarial networks (GAN) and diffusion models (Diffusion Model) have been able to achieve high-quality synthesis from text descriptions to video clips through large-scale multi-modal data training. The generated results have significantly improved in terms of resolution, color fidelity and other objective indicators. In the field of multi-modal alignment, cross-modal embedding models represented by CLIP (Contrastive Language-Image Pretraining) and VQGAN-CLIP have preliminarily solved the consistency matching problem of text semantics and visual content through joint semantic space mapping mechanisms.

[0003] However, there are still key bottlenecks in practical applications: first, existing models lack explicit modeling capabilities for the causal logic implied in user requirements, leading to temporal contradictions or logical discontinuities in generated content. Second, multi-modal temporal synchronization mechanisms are still limited to static rules or single-modal dominant alignment strategies, failing to effectively quantify the neural conduction characteristics of audiovisual perception differences. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a short video production method based on artificial intelligence to solve the problems of logical temporal discontinuity and multi-modal perception mismatch in short video generation.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides an artificial intelligence-based short video production method, which comprises: collecting user original demand instructions; extracting a causal factor set through a causal discovery algorithm; constructing a dynamic causal graph containing node correlation strength and generating a split script based on the topological structure of the dynamic causal graph; inputting the split script into a multi-modal generation engine, generating a visual stream using a lower triangular spatiotemporal attention matrix, and generating a text stream and an audio stream; calculating the neural conduction delay between the visual stream and the audio stream based on an audiovisual perception delay prediction model, and generating a corrected audio stream through time axis forward compensation; calculating a cross-modal attention weight matrix according to the node correlation strength of the dynamic causal graph, and projecting the text stream and the corrected audio stream to a joint feature space to generate a multi-modal fusion feature; inputting the multi-modal fusion feature into a format adaptation engine, performing spatiotemporal consistency verification, embedding a neural implicit watermark, and encoding into a short video.

[0008] As a preferred scheme of the artificial intelligence-based short video production method of the present application, the steps of extracting the causal factor set through the causal discovery algorithm are as follows,

[0009] The user original demand instructions are subjected to morphological analysis by a word segmentation tool to generate an initial vocabulary set, and a candidate causal phrase set is extracted from the initial vocabulary set.

[0010] The causal factor set is identified from the causal phrase set through a partial directed acyclic graph (PC) algorithm.

[0011] As a preferred scheme of the artificial intelligence-based short video production method of the present application, the steps of constructing the dynamic causal graph containing the node correlation strength and generating the split script based on the topological structure of the dynamic causal graph are as follows,

[0012] Based on the causal factor set, the causal effect strength is estimated through a LiNGAM linear non-Gaussian acyclic model to construct the dynamic causal graph containing the node correlation strength.

[0013] The topological sorting of the dynamic causal graph is performed through a Kahn algorithm to generate an ordered sequence of causal events, and the split time length is generated through linear interpolation according to the causal edge weight of the dynamic causal graph.

[0014] The causal factor triplets corresponding to the causal events are extracted based on the ordered sequence, the causal factor triplets are mapped to generate split descriptions, and the split descriptions are combined with the split time length to form the split script.

[0015] As a preferred scheme of the artificial intelligence-based short video production method of the present application, the steps of inputting the split script into the multi-modal generation engine, generating the visual stream using the lower triangular spatiotemporal attention matrix, and generating the text stream and the audio stream are as follows,

[0016] The shot script is input into a multi-modal generation engine to generate a multi-modal generation instruction sequence;

[0017] A lower triangular spatio-temporal attention matrix is constructed, and a matrix forced visual flow is generated through matrix multiplication and matrix encoding;

[0018] Based on the matrix forced visual flow, a visual flow is generated through visual frame rendering;

[0019] Based on the visual flow, an audio flow is generated through audio-visual synchronization, and a text flow is extracted from the causal factor triplets.

[0020] As a preferred scheme of the short video production method based on artificial intelligence, the neural conduction delay between the visual flow and the audio flow is calculated based on the audio-visual perception delay prediction model, and a corrected audio flow is generated through time axis forward compensation, and the specific steps are as follows,

[0021] Through the visual encoder and the audio encoder, the visual feature vector and the audio feature vector are extracted from the visual flow and the audio flow respectively;

[0022] An audio-visual perception delay prediction model is constructed, and the visual feature vector and the audio feature vector are input into the audio-visual perception delay prediction model to generate a neural conduction delay;

[0023] According to the neural conduction delay, the time axis forward compensation is performed on the audio flow to generate a corrected audio flow.

[0024] As a preferred scheme of the short video production method based on artificial intelligence, the neural conduction delay between the visual flow and the audio flow is calculated based on the audio-visual perception delay prediction model, and a corrected audio flow is generated through time axis forward compensation, and the specific steps are as follows,

[0025] The causal effect strength of the nodes is extracted from the dynamic causal graph, a node correlation strength matrix is constructed, and a text flow feature vector and an audio flow feature vector are extracted from the text flow and the corrected audio flow respectively;

[0026] Based on the text flow feature vector, the audio flow feature vector and the node correlation strength matrix, a cross-modal attention weight matrix is generated through Hadamard product operation;

[0027] Based on the cross-modal attention weight matrix, the text flow and the corrected audio flow are projected into a joint feature space through linear transformation, and a multi-modal fusion feature is generated through full connection layer compression.

[0028] As a preferred scheme of the short video production method based on artificial intelligence, the multi-modal fusion feature is input into a format adaptation engine, embedded with a neural implicit watermark after spatio-temporal consistency verification, and encoded into a short video, and the specific steps are as follows,

[0029] The multi-modal fusion feature input format adaptation engine calculates a motion vector field between adjacent feature vectors, and generates a spatio-temporally consistent feature sequence through spatio-temporal consistency verification;

[0030] The spatio-temporally consistent feature sequence is embedded with a neural implicit watermark through a frequency domain watermark encoder, and a short video is generated through dynamic adjustment of the encoding parameters.

[0031] In a second aspect, the present application provides an artificial intelligence-based short video production system, comprising a cause-and-effect storyboard module for collecting user original demand instructions, extracting a cause-and-effect factor set through a cause-and-effect discovery algorithm, constructing a dynamic cause-and-effect graph containing node correlation strength, and generating a storyboard script based on the topological structure of the dynamic cause-and-effect graph; a multi-modal generation module for inputting the storyboard script into a multi-modal generation engine, generating a visual stream using a lower triangular spatio-temporal attention matrix, and generating a text stream and an audio stream; a delay correction module for calculating the neural conduction delay amount between the visual stream and the audio stream based on a visual-audio perception delay prediction model, and generating a corrected audio stream through time axis forward compensation; a feature fusion module for calculating a cross-modal attention weight matrix according to the node correlation strength of the dynamic cause-and-effect graph, and projecting the text stream and the corrected audio stream into a joint feature space to generate multi-modal fusion features; and a video synthesis module for inputting the multi-modal fusion features into a format adaptation engine, embedding a neural implicit watermark after spatio-temporal consistency verification, and encoding the neural implicit watermark into a short video.

[0032] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and wherein the computer program, when executed by the processor, implements any step of the artificial intelligence-based short video production method according to the first aspect of the present application.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any step of the artificial intelligence-based short video production method according to the first aspect of the present application.

[0034] The present application has the following beneficial effects: the dynamic cause-and-effect graph modeling realizes the logical generation of short video storyboards, so that the video content has a clear narrative structure and reasonable time rhythm; at the same time, the cross-modal attention mechanism is combined to guide multi-modal information fusion using the node correlation strength in the cause-and-effect graph, which improves the semantic consistency and coordination between text, audio and visual content, thereby realizing high-quality and intelligent short video automatic generation as a whole. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0036] Fig. 1 Flowchart for the short video production method based on artificial intelligence.

[0037] Fig. 2 Flowchart for generating a structured script.

[0038] Fig. 3 Flowchart for generating a corrected audio stream.

[0039] Fig. 4 Flowchart for generating a short video. DETAILED DESCRIPTION

[0040] In order to make the above objectives, features and advantages of the present application more apparent and comprehensible, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0041] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be practiced without other different ways from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0042] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0043] Reference Figs. 1-4 For one embodiment of the present application, the embodiment provides a short video production method based on artificial intelligence, comprising the following steps:

[0044] S1: Collecting user original demand instructions, extracting a set of causal factors through a causal discovery algorithm, constructing a dynamic causal graph containing node association strength, and generating a script based on the topological structure of the dynamic causal graph;

[0045] S1.1: The user original demand instruction is analyzed by a word segmentation tool, segmented into a word sequence and removed stop words, and an initial vocabulary set is generated;

[0046] Specifically, input text data of a user's original demand instruction is received, all letters in the input text data are converted into lowercase form, and punctuation marks are replaced with spaces; the processed input text data is input into a word segmentation tool, and a dictionary matching strategy is combined to identify word boundaries according to the positions of spaces and hyphens, so as to segment the input text data into a word sequence; a set of stop words is loaded from a stop word list in a natural language processing tool NLTK library, the set of stop words including non-substantive articles, prepositions, pronouns and auxiliary verbs; each word in the word sequence is traversed, and words existing in the set of stop words are deleted, and words not existing in the set of stop words are retained in an intermediate vocabulary set; each word in the intermediate vocabulary set is subjected to morphological reduction processing, the word is converted into a base form by querying a preset morphological reduction dictionary, and a standardized word is generated; the standardized words are arranged in the original order, adjacent same words that appear repeatedly are removed, and an initial vocabulary set is generated;

[0047] It should be noted that the preset morphological reduction dictionary is determined based on the lexical form mapping relationship of WordNet 3.0, using the Penn Treebank part-of-speech tagging rule, combining the VerbNet of WordNet, and applying the Oxford-s / -es plural transformation rule.

[0048] S1.2: Based on the initial vocabulary set, a pre-trained semantic role labeling model is used to label the predicate-argument structure in the word sequence, and a candidate causal phrase set is extracted;

[0049] Further, the pre-trained semantic role labeling model adopts a BERT-Base model based on a Transformer architecture, the input layer maps the standardized words into word vectors through wordpiece tokenization processing to form a word vector sequence; the word vector sequence is input into the encoder layer of the pre-trained semantic role labeling model to output a label sequence; the label sequence is traversed, and when consecutive B-I labels are detected, a predicate-argument pair is extracted to generate a structure set, the predicate in the predicate-argument pair is a standardized word corresponding to a verb word vector, and the argument is a standardized word corresponding to a noun word vector; the predicate-argument pairs that meet the causal relationship (matching is performed using the cause synonym set in WordNet, and the PropBank semantic role definition is met) are screened from the structure set; the screened predicate-argument pairs are converted into phrase form to generate a candidate causal phrase set containing causal factors.

[0050] S1.3: Filtering the candidate causal phrase set based on the causal probability matrix, and retaining statistically significant causal factor pairs;

[0051] It should be noted that based on the co-occurrence frequency of the subject and the object in the candidate causal phrase set, the original frequency matrix is formed, the original frequency matrix is direction normalized, and the conditional probability of each subject generating the object is calculated, the original frequency matrix after direction normalization is converted into a conditional probability matrix through the conditional probability, the conditional probability matrix is maximum-minimum normalized to generate a causal probability matrix, the row and column indexes of the causal probability matrix correspond to the causal factors in the candidate causal phrase set, further, the Pearson conditional mutual information formula is used to calculate the causal strength, and the expression is:

[0052]

[0053] Wherein, M ij is the causal strength of the causal factor pair in the i-th row and the j-th column of the causal probability matrix, c i is the causal factor in the i-th row of the causal probability matrix, c j is the causal factor in the j-th column of the causal probability matrix, P(c i ,c j ,Pa(c j )) is the probability of the simultaneous occurrence of the causal factor c i , the causal factor c j and the parent node set Pa(c j ), Pa(c j ) is the parent node set of the causal factor c j , P(c i ,c j |Pa(c j )) is the joint conditional probability of the simultaneous occurrence of the causal factor c j to the causal factor c j under the condition that the parent node set Pa(c i ) of the causal factor c j is given, P(c i |Pa(c j )) is the conditional probability of the independent occurrence of the causal factor c j under the condition that the parent node set Pa(c i ) is given, P(c j |Pa(c j )) is the conditional probability of the independent occurrence of the causal factor c j under the condition that the parent node set Pa(c j ) is given.

[0054] The co-occurrence frequency of the candidate causal phrase pair is counted to generate a frequency table containing four elements:

[0055] The number of times of the simultaneous occurrence of the causal factor c i and the causal factor c j , the causal factor ci occurs but causal factor c j does not occur, causal factor c j occurs but causal factor c i does not occur, causal factor c i and causal factor c j both do not occur.

[0056] The statistical significance value of the causal factor pair is obtained based on the hypergeometric distribution of the frequency table by using the Fisher exact test method, and the statistical significance threshold value is set according to the significance level (the first type error rate threshold value set according to IEEE 1855-2016) (for example, the statistical significance threshold value is 0.05), and the causal factor pair with a statistical significance value greater than the statistical significance threshold value is reserved as a statistically significant causal factor pair;

[0057] The causal strength of all causal factor pairs is extracted from the causal probability matrix according to the historical causal data, and the causal factor pair with a causal strength quantile distribution higher than the quantile threshold value is reserved, and the statistically significant causal factor pair and the causal factor pair higher than the quantile threshold value are reserved to generate a filtered candidate causal phrase set.

[0058] It should be noted that the quantile threshold value is determined by Monte Carlo cross-validation: the historical causal data is randomly divided into 80% training set and 20% validation set, the causal strength of all causal factor pairs is sorted on the training set, the causal strength quantile distribution is obtained by linear interpolation method, and the quantile point that makes the causal relationship retention rate of the validation set maximum is selected as the quantile threshold value.

[0059] S1.4: Apply the partial directed acyclic graph PC algorithm to construct the causal graph skeleton and identify the causal factor set;

[0060] Further, each causal factor in the filtered candidate causal phrase set is taken as a causal factor node, a complete undirected edge is drawn between the causal factor nodes, a fully connected undirected graph is constructed, any two adjacent causal factor node pairs are selected from the fully connected undirected graph, the causal strength is obtained based on the four-element frequency table, the undirected edge between the two causal factor nodes is deleted according to the causal strength, and after iteration and reduction of redundant edges, an undirected skeleton graph is generated; identify the V-shaped structure in the undirected skeleton graph and combine it with the remaining undirected edges after directional propagation to generate a causal graph skeleton, extract all candidate causal phrases corresponding to the causal factor nodes from the causal graph skeleton, and form a causal factor set.

[0061] S1.5: Estimate the causal effect strength by a linear non-Gaussian model and construct a weighted dynamic causal graph;

[0062] Specifically, each causal factor in the causal factor set is standardized. Independent component analysis is applied to the standardized causal factors to obtain a mixture matrix. The mixture matrix is ​​then transposed into a connection matrix, where the elements represent the intensity of the direct causal effect between causal factors. The directional relationship between causal factors is determined based on the sign of the non-zero elements in the connection matrix (positive for positive causality and negative for negative causality). The elements of the connection matrix are used as the causal effect intensity to generate a causal weight matrix. The causal weight matrix is ​​then normalized. Directed edges are drawn based on the causal direction, with the causal factor nodes as nodes. The weights of the directed edges are set to the normalized causal effect intensity to generate a weighted dynamic causal graph.

[0063] S1.6: Perform topological sorting on the weighted dynamic causal graph using the Kahn algorithm to generate an ordered sequence of causal events;

[0064] Furthermore, traverse all nodes in the weighted dynamic causal graph, count the number of predecessor nodes for each node and record it as the in-degree value, generate an initial in-degree table, scan the initial in-degree table, add nodes with an in-degree value of zero to the queue to be processed, take a node from the queue to be processed and append the node to the end of the ordered sequence, traverse all direct successor nodes of the node, decrement the in-degree value of each successor node by one, when the in-degree value of a successor node is reduced to zero, add this successor node to the ordered sequence, and repeat the loop until the queue to be processed is empty, generating an ordered sequence of causal events.

[0065] S1.7: Based on the ordered sequence, obtain the causal edge weights, and map the sum of the causal edge weights to the storyboard duration through linear interpolation to generate a storyboard duration set;

[0066] It should be noted that, by traversing the ordered sequence of causal events, selecting adjacent causal event pairs in turn, finding the directed edges between each pair of adjacent causal events in the weighted dynamic causal graph, recording the corresponding weights, generating a set of causal edge weights, summing all the weights in the causal edge weight set to obtain the total weight sum, and mapping the sum of causal edge weights to the storyboard duration through linear interpolation to generate a set of storyboard durations.

[0067] S1.8: Based on an ordered sequence, a storyboard description is generated through causal factor triples and combined with a set of storyboard durations to form a structured storyboard script;

[0068] Specifically, traverse the ordered sequence of causal events, extract the causal factor triplets corresponding to each causal event, including the subject, action, and object; map the standardized names of the subject and object in the causal factor triplet to the domain term library, define the natural language generation template, fill the causal factor triplet into the template slot, generate the storyboard description of the natural language description, and match the storyboard description with the corresponding storyboard duration in the storyboard duration set according to the order of the sequence, generate the key-value pair set, the key is the storyboard description text, and the value is the storyboard duration; arrange the key-value pair set according to the ordered sequence to generate the structured storyboard script, each storyboard entry in the storyboard script contains a storyboard description and a storyboard duration, and the format is a JSON array;

[0069] S2: input the storyboard script into the multi-modal generation engine, generate the visual stream using the lower triangular spatio-temporal attention matrix, and generate the text stream and audio stream;

[0070] S2.1: input the storyboard script into the multi-modal generation engine, parse the scene description and timing data of the storyboard script, and generate a multi-modal generation instruction sequence;

[0071] Input the storyboard script into the multi-modal generation engine, and based on the causal factor triplet, generate a multi-modal generation instruction sequence including visual elements, text content, and audio markers through multi-modal instruction element mapping;

[0072] Visual elements: based on the verb and object in the causal factor triplet, match the graphic / animation template from the domain term library;

[0073] Text content: input the causal factor triplet into the causal constraint text generation model to generate concise narration text and subtitle content;

[0074] It should be noted that based on the GPT-2 architecture, the storyboard script and the causal factor triplet are encoded as input through structured causal control symbols, a causal awareness adaptation layer is added, including independent control symbol embedding, constraint attention head, and gated FFN unit; adopt a finite state machine constraint decoding strategy, combine regular expression rules to dynamically filter the candidate word table obtained by the word segmentation tool, and apply probability bias to causal related words; fine-tune on the FilmScript-12K domain dataset through a multi-task loss function to ensure that the generated text and the logic of the causal graph are strictly consistent, and finally obtain the output causal constraint text generation model that generates concise narration text and subtitle content.

[0075] Audio markers: according to the verb type, match the audio clip from the sound effect library.

[0076] S2.2: construct a lower triangular spatio-temporal attention matrix to generate a matrix forced visual stream;

[0077] The array length of the split shot script is parsed to obtain the number of split shots, and an attention matrix with a dimension of the number of split shots is constructed. The elements above the main diagonal of the attention matrix are forced to be zero to generate a lower triangular matrix. For each non-zero element position in the lower triangular matrix, the time interval is obtained through the split shot duration, and the weight is calculated by applying the natural exponential decay function to generate a lower triangular spatio-temporal attention matrix. The visual element sequence in the multi-modal generation instruction sequence is serialized into a visual element vector, and the aggregated visual features are generated by matrix multiplication. The matrix forced visual flow is encoded into a matrix forced visual flow.

[0078] S2.3: Based on the matrix forced visual flow, a visual flow is generated by visual frame rendering, a text flow is generated by a causal constraint text generation model, and an audio flow is generated by audio-visual synchronization;

[0079] Based on the aggregated visual features, a graphics rendering engine is called to generate images frame by frame. The duration of each frame of image is set according to the split shot duration of the split shot script to generate a visual flow.

[0080] The causal factor triplets are input into the causal constraint text generation model to generate a text flow including narration text and subtitle content. The text display start and end times are set according to the split shot duration of the split shot script.

[0081] The split shot description text of the split shot script is matched with the audio clips in the sound effect library. The split shot start time is generated by accumulating according to the split shot duration. The audio trigger time point is set as the split shot start time. The narration voice (generated by the TTS engine from the narration text in the text flow) is superimposed with the sound effect clip according to the time axis to generate an audio flow.

[0082] S3: Based on the audio-visual perception delay prediction model, the neural conduction delay between the visual flow and the audio flow is calculated, and the modified audio flow is generated by time axis forward compensation;

[0083] S3.1: The visual feature vector and the audio feature vector are extracted from the visual flow and the audio flow respectively by the visual encoder and the audio encoder.

[0084] It should be noted that each frame of image in the visual flow is adjusted to a fixed resolution and pixel normalization processing is performed. The visual encoder ResNet-50 is input, and the 2048-dimensional visual feature vector is extracted by the convolution layer and the global average pooling layer after removing the fully connected layer. The visual feature vector sequence is generated according to the split shot timing. The audio flow is segmented into audio clips aligned with the split shot timing, resampled to a standard sampling rate, and converted to single-channel waveform data. The audio encoder VGGish model is input, and the 128-dimensional audio feature vector is extracted. The audio feature vector sequence is generated according to the split shot timing.

[0085] S3.2: input the visual feature vector and the audio feature vector into the audio-visual perception delay prediction model to generate a neural conduction delay amount;

[0086] Further, the visual feature vector and the audio feature vector are aligned according to the shot timing, spliced into a multi-modal feature pair through a time sliding window, and input into the audio-visual perception delay prediction model. A bidirectional long short-term memory network layer is used to capture the time sequence dependency relationship, and a neural conduction delay amount is obtained through a cross attention mechanism.

[0087] It should be noted that the construction process of the audio-visual perception delay prediction model is as follows: the visual features (frame-level CNN features extracted by ResNet-50) and the audio features (128-dimensional Mel spectrum embedding extracted by VGGish) are aligned to the same time axis through dynamic time warping (DTW), and then the synchronized multi-modal feature pairs are cut off in a sliding window (window length 1 second, step 0.5 second); the audio-visual perception delay prediction model architecture adopts a double-flow Bi-LSTM network, the visual features and the audio features are input into independent Bi-LSTM layers (hidden layer 256 dimensions) to extract the time sequence context representation, and then the cross attention mechanism is used to calculate the audio-visual feature correlation weight to generate an attention weighted joint representation; finally, the neural conduction delay amount is output through a fully connected layer (including ReLU activation), and a Huber loss function with a time sequence smoothing constraint is used for training.

[0088] S3.3: performing time axis forward shift compensation on the audio stream according to the neural conduction delay amount to generate a corrected audio stream;

[0089] Specifically, based on the neural conduction delay amount, the start timestamp and the end timestamp of the audio segment are both shifted forward by the neural conduction delay amount, and the audio waveform cutting and splicing technology is used to eliminate the silent gap caused by the time axis forward shift, to generate the audio segment after time axis correction; the corrected audio segment is recombined into continuous waveform data according to the shot timing to generate a corrected audio stream strictly synchronized with the visual stream.

[0090] S4: calculating a cross-modal attention weight matrix according to the node correlation strength of the dynamic causal graph, and projecting the text stream and the corrected audio stream into a joint feature space to generate a multi-modal fusion feature;

[0091] S4.1: extracting a node correlation strength matrix from the dynamic causal graph, and extracting a text stream feature vector and an audio stream feature vector from the extracted text stream and the corrected audio stream, respectively;

[0092] It should be noted that, by traversing all causal factor node pairs in the dynamic causal graph, reading the directed edge weights (i.e., the causal effect strength), and generating a node association strength matrix, the narration text in the text stream is input into the encoder layer of the semantic role labeling model BERT-Base to extract the 768-dimensional text stream feature vector of each scene description text; and the audio segments of the corrected audio stream are input into the Embedding layer of the VGGish model to extract the 128-dimensional audio stream feature vector.

[0093] S4.2: Calculate the cross-modal attention weight matrix based on the text stream feature vector, audio stream feature vector, and node association strength matrix;

[0094] Furthermore, the text stream feature vectors and audio stream feature vectors are aligned according to the scene sequence, and the cosine similarity of each pair of text-audio feature vectors is calculated by dot product operation to generate a cross-modal similarity matrix; the cross-modal similarity matrix and the node association strength matrix are subjected to Hadamard product operation; the Hadamard product result is subjected to Softmax normalization processing row by row to generate a cross-modal attention weight matrix.

[0095] S4.3: Based on the cross-modal attention weight matrix, the text stream and the corrected audio stream are projected onto the joint feature space through linear transformation to generate a joint feature vector;

[0096] Specifically, the text stream feature vector and the modified audio stream feature vector are aligned according to the scene sequence. A weighted summation operation is performed on the text stream feature vector and the modified audio stream feature vector through a cross-modal attention weight matrix to generate a cross-modal fusion feature vector. The cross-modal fusion feature vector is input into a fully connected layer (dimension 2048+128→1024), and the dimension is reduced to a joint feature space through a linear transformation to generate a joint feature vector with a dimension of 1024.

[0097] S4.4: Perform fully connected layer compression on the joint feature vector to generate multimodal fusion features.

[0098] It should be noted that the joint feature vector with a dimension of 1024 is input into the fully connected layer, and a linear transformation is performed through matrix multiplication and bias term addition to compress the joint feature vector to 512 dimensions. The ReLU activation function is applied to the compressed feature vector to eliminate negative values ​​and enhance nonlinear expressive power, and output 512-dimensional multimodal fusion features.

[0099] S5: Input the multimodal fusion features into the format adaptation engine, and after spatiotemporal consistency verification, embed a neural implicit watermark and encode it into a short video.

[0100] S5.1: Multimodal fusion feature input optical flow estimation model, calculates the motion vector field between adjacent feature vectors, and generates a spatiotemporally consistent feature sequence through spatiotemporal consistency verification;

[0101] Furthermore, the multimodal fusion features are input into the optical flow estimation model FlowNet2.0. The displacement between adjacent feature vectors is matched frame by frame through convolutional layers and correlation calculation layers to generate a vector field describing the direction of feature motion. Bidirectional optical flow consistency verification is performed on the vector field to calculate the cyclic consistency error between forward and reverse optical flow. Abnormal displacement is corrected by interpolation between adjacent frames (abnormal displacement is identified by an adaptive statistical model that combines the cyclic consistency error distribution with the variance of local motion direction and the temporal interpolation residual to identify abnormal motion vector regions that deviate from the spatiotemporal continuity constraint). The verified vector field is then superimposed on the multimodal fusion features in chronological order to generate a spatiotemporally consistent feature sequence.

[0102] S5.2: The spatiotemporally consistent feature sequence is embedded with a neural implicit watermark through a frequency domain watermark encoder, and short videos are generated by dynamically adjusting the encoding parameters;

[0103] The spatiotemporally consistent feature sequence is input into the frequency domain watermark encoder. The frequency domain watermark encoder maps the spatiotemporally consistent feature sequence to the frequency domain space through discrete cosine transform. A perturbation signal based on neural implicit watermarking is superimposed on the low-frequency components. The perturbation intensity is dynamically adjusted according to the watermark robustness requirements of the target platform. The frequency domain signal with embedded watermark is inversely transformed to the time domain to generate a spatiotemporally feature sequence with watermark. The target platform's standard video encoder (such as H.264 encoder) is called to render the spatiotemporal feature sequence into a video frame sequence. The sequence is compressed and encapsulated according to the segment length and resolution requirements to generate a short video that conforms to the platform's bitrate and encapsulation format.

[0104] This embodiment also provides an artificial intelligence-based short video production system, including: a causal storyboard module, used to collect original user demand instructions, extract a set of causal factors through a causal discovery algorithm, construct a dynamic causal graph containing node association strength, and generate a storyboard script based on the topological structure of the dynamic causal graph; a multimodal generation module, used to input the storyboard script into a multimodal generation engine, generate a visual stream using a lower triangular spatiotemporal attention matrix, and generate a text stream and an audio stream; a delay correction module, used to calculate the neural transmission delay between the visual stream and the audio stream based on an audiovisual perception delay prediction model, and generate a corrected audio stream by time axis forward compensation; a feature fusion module, used to calculate a cross-modal attention weight matrix based on the node association strength of the dynamic causal graph, and project the text stream and the corrected audio stream onto a joint feature space to generate multimodal fusion features; and a video synthesis module, used to input the multimodal fusion features into a format adaptation engine, and after spatiotemporal consistency verification, embed a neural implicit watermark and encode it into a short video.

[0105] This embodiment also provides a computer device applicable to the short video production method based on artificial intelligence, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the short video production method based on artificial intelligence as proposed in the above embodiment.

[0106] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0107] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the artificial intelligence-based short video production method proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0108] In summary, this invention achieves the logical generation of short video storyboards through dynamic causal graph modeling, giving the video content a clear narrative structure and a reasonable time rhythm; at the same time, it combines a cross-modal attention mechanism, using the node association strength in the causal graph to guide the fusion of multimodal information, improving the semantic consistency and coordination between text, audio and visual content, thereby achieving high-quality and highly intelligent automatic generation of short videos as a whole.

[0109] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A short video production method based on artificial intelligence, characterized in that: include, Collect users' original demand instructions, extract the set of causal factors through causal discovery algorithms, construct a dynamic causal graph containing the strength of node associations, and generate storyboard scripts based on the topological structure of the dynamic causal graph. The storyboard script is input into the multimodal generation engine, and a lower triangular spatiotemporal attention matrix is ​​used to generate a visual stream, as well as a text stream and an audio stream. The neural conduction delay between the visual stream and the audio stream is calculated based on the audiovisual perception delay prediction model, and a corrected audio stream is generated by time axis forward compensation. The cross-modal attention weight matrix is ​​calculated based on the node association strength of the dynamic causal graph, and the text stream and the corrected audio stream are projected into the joint feature space to generate multimodal fusion features. The specific steps are as follows. The causal effect strength of nodes is extracted from the dynamic causal graph, a node association strength matrix is ​​constructed, and text stream feature vectors and audio stream feature vectors are extracted from the text stream and the modified audio stream, respectively. Based on text stream feature vectors, audio stream feature vectors, and node association strength matrices, a cross-modal attention weight matrix is ​​generated through Hadamard product operations. Based on the cross-modal attention weight matrix, the text stream and the corrected audio stream are projected into the joint feature space through linear transformation, and multimodal fusion features are generated by compression through a fully connected layer; The multimodal fusion features are input into the format adaptation engine, and after spatiotemporal consistency verification, a neural implicit watermark is embedded and encoded into a short video.

2. The short video production method based on artificial intelligence as described in claim 1, characterized in that: The specific steps for extracting the set of causal factors using the causal discovery algorithm are as follows. The user's original demand command is lexically analyzed using a word segmentation tool to generate an initial vocabulary set, and a candidate causal phrase set is extracted from the initial vocabulary set. Using a partially directed acyclic graph PC algorithm, we identify the set of causal factors from the set of causal phrases.

3. The short video production method based on artificial intelligence as described in claim 2, characterized in that: The specific steps for constructing a dynamic causal graph that includes the node association strength, and generating a storyboard script based on the topology of the dynamic causal graph, are as follows: Based on the set of causal factors, the intensity of causal effects is estimated by the LiNGAM linear non-Gaussian acyclic model, and a dynamic causal graph containing the node association strength is constructed. The dynamic causal graph is topologically sorted using the Kahn algorithm to generate an ordered sequence of causal events, and the segment duration is generated by linear interpolation based on the causal edge weights of the dynamic causal graph. Based on the ordered sequence, extract the causal factor triples corresponding to the causal events, map the causal factor triples to generate storyboard descriptions, and combine them with the storyboard duration to form a storyboard script.

4. The short video production method based on artificial intelligence as described in claim 3, characterized in that: The process involves inputting the storyboard script into a multimodal generation engine, using a lower triangular spatiotemporal attention matrix to generate a visual stream, and then generating a text stream and an audio stream. The specific steps are as follows: Input the storyboard script into the multimodal generation engine to generate a multimodal generation instruction sequence; Construct a lower triangular spatiotemporal attention matrix, and generate a matrix-forced visual flow through matrix multiplication and matrix encoding; Based on matrix-forced visual flow, a visual flow is generated through visual frame rendering. Based on visual streams, an audio stream is generated through audio-visual synchronization, and a text stream is extracted from causal factor triples.

5. The short video production method based on artificial intelligence as described in claim 4, characterized in that: The method involves calculating the neural conduction delay between the visual and audio streams based on an audiovisual perception delay prediction model, and then compensating for this delay by shifting the time axis forward to generate a corrected audio stream. The specific steps are as follows: Visual feature vectors and audio feature vectors are extracted from the visual stream and audio stream, respectively, using a visual encoder and an audio encoder. A visual and auditory perception delay prediction model is constructed, and visual feature vectors and audio feature vectors are input into the model to generate neural conduction delay. The audio stream is compensated for by shifting the time axis forward based on the neural conduction delay, resulting in a corrected audio stream.

6. The short video production method based on artificial intelligence as described in claim 1, characterized in that: The process involves inputting multimodal fusion features into a format adaptation engine, performing spatiotemporal consistency verification, embedding a neural implicit watermark, and encoding it into a short video. The specific steps are as follows: A multimodal fusion feature input format adaptation engine calculates the motion vector field between adjacent feature vectors and generates a spatiotemporally consistent feature sequence through spatiotemporal consistency verification; Spatiotemporally consistent feature sequences are embedded with neural implicit watermarks through a frequency domain watermark encoder, and short videos are generated by dynamically adjusting the encoding parameters.

7. An artificial intelligence-based short video production system, based on the artificial intelligence-based short video production method according to any one of claims 1 to 6, characterized in that: This includes a causal storyboard module, a multimodal generation module, a delay correction module, a feature fusion module, and a video synthesis module. The causal storyboard module is used to collect the user's original requirement instructions, extract the set of causal factors through the causal discovery algorithm, construct a dynamic causal graph containing the strength of node associations, and generate storyboard scripts based on the topological structure of the dynamic causal graph. The multimodal generation module is used to input the storyboard script into the multimodal generation engine, generate a visual stream using a lower triangular spatiotemporal attention matrix, and generate a text stream and an audio stream. The delay correction module is used to calculate the neural conduction delay between the visual stream and the audio stream based on the audiovisual perception delay prediction model, and to generate a corrected audio stream by shifting the time axis forward to compensate for it. The feature fusion module is used to calculate the cross-modal attention weight matrix based on the node association strength of the dynamic causal graph, and project the text stream and the corrected audio stream onto the joint feature space to generate multimodal fusion features; The video synthesis module is used to input multimodal fusion features into the format adaptation engine, and after spatiotemporal consistency verification, embed neural implicit watermarks and encode them into short videos.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the artificial intelligence-based short video production method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the artificial intelligence-based short video production method according to any one of claims 1 to 6.

Images