Short video production method and system based on artificial intelligence

Abstract

The invention discloses a short video production method and system based on artificial intelligence, and relates to the technical field of multimedia processing, and the method comprises the steps: inputting a split script into a multi-modal generation engine, employing a lower triangle space-time attention matrix to generate a visual stream, and generating a text stream and an audio stream; calculating a neural conduction delay amount between the visual stream and the audio stream based on an audio-visual perception delay prediction model, and generating a corrected audio stream through time axis forward compensation; calculating a cross-modal attention weight matrix according to the node association strength of the dynamic causal graph, and projecting the text stream and the corrected audio stream to a joint feature space to generate a multi-modal fusion feature; and inputting the multi-modal fusion features into a format adaptation engine, and after time-space consistency verification, embedding a neural implicit watermark and encoding the neural implicit watermark into a short video. According to the invention, the semantic consistency and coordination among the text, the audio and the visual content are improved, so that high-quality and high-intelligence automatic generation of the short video is realized on the whole.

Description

Technical Field

[0001] The present invention relates to the field of multimedia processing technology, and in particular to a short video production method and system based on artificial intelligence. Background Art

[0002] In recent years, breakthroughs in generative artificial intelligence (AI) have driven the rapid development of automated short video generation technology. Deep learning architectures, such as generative adversarial networks (GANs) and diffusion models, have been able to achieve high-quality synthesis from text descriptions to video clips through large-scale multimodal data training. The generated results have significantly improved objective metrics such as resolution and color fidelity. In the field of multimodal alignment, cross-modal embedding models, such as CLIP (Contrastive Language-Image Pretraining) and VQGAN-CLIP, have initially addressed the problem of consistent matching of textual semantics and visual content through a joint semantic space mapping mechanism.

[0003] Key bottlenecks remain when it comes to practical applications: 1. Existing models lack the ability to explicitly model the causal logic implicit in user needs, leading to temporal inconsistencies or logical gaps within the generated models. 2. Multimodal temporal synchronization mechanisms are still limited to static rules or alignment strategies dominated by a single modality, failing to effectively quantify the neural conduction characteristics of audiovisual perception differences. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an artificial intelligence-based short video production method to solve the problems of logical timing faults and multimodal perception mismatch in short video generation.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a short video production method based on artificial intelligence, which includes collecting user original demand instructions, extracting a set of causal factors through a causal discovery algorithm, constructing a dynamic causal graph including node correlation strength, and generating a storyboard script based on the topological structure of the dynamic causal graph; inputting the storyboard script into a multimodal generation engine, using a lower triangular spatiotemporal attention matrix to generate a visual stream, and generating a text stream and an audio stream; calculating the neural conduction delay between the visual stream and the audio stream based on an audio-visual perception delay prediction model, and generating a corrected audio stream by shifting the time axis forward to compensate; calculating the cross-modal attention weight matrix according to the node correlation strength of the dynamic causal graph, and projecting the text stream and the corrected audio stream into a joint feature space to generate multimodal fusion features; inputting the multimodal fusion features into a format adaptation engine, and after spatiotemporal consistency verification, embedding a neural implicit watermark and encoding it into a short video.

[0008] As a preferred solution of the short video production method based on artificial intelligence of the present invention, wherein: the causal factor set is extracted by the causal discovery algorithm, and the specific steps are as follows:

[0009] Perform lexical analysis on the user's original demand instructions through a word segmentation tool to generate an initial vocabulary set, and extract a set of candidate causal phrases from the initial vocabulary set;

[0010] The causal factor set is identified from the causal phrase set using the partial directed acyclic graph (PC) algorithm.

[0011] As a preferred solution of the short video production method based on artificial intelligence of the present invention, the steps of constructing a dynamic causal graph including node association strength and generating a storyboard based on the topological structure of the dynamic causal graph are as follows:

[0012] Based on the causal factor set, the LiNGAM linear non-Gaussian acyclic model is used to estimate the causal effect strength and construct a dynamic causal graph including the node association strength.

[0013] The dynamic causal graph is topologically sorted using the Kahn algorithm to generate an ordered sequence of causal events. Based on the causal edge weights of the dynamic causal graph, the duration of the storyboard is generated through linear interpolation.

[0014] The causal factor triples corresponding to the causal events are extracted based on the ordered sequence, the causal factor triples are mapped to generate storyboard descriptions, and then combined with the storyboard duration to form a storyboard script.

[0015] As a preferred solution of the short video production method based on artificial intelligence described in the present invention, wherein: the storyboard script is input into the multimodal generation engine, the lower triangular spatiotemporal attention matrix is ​​used to generate the visual stream, and the text stream and audio stream are generated. The specific steps are as follows:

[0016] Input the storyboard script into the multimodal generation engine to generate a multimodal generation instruction sequence;

[0017] Construct a lower triangular spatiotemporal attention matrix and generate matrix-enforced visual flow through matrix multiplication and matrix encoding;

[0018] Based on matrix-enforced visual flow, visual flow is generated through visual frame rendering;

[0019] Based on the visual stream, an audio stream is generated through audio-visual synchronization, and a text stream is extracted from the causal factor triples.

[0020] As a preferred embodiment of the short video production method based on artificial intelligence of the present invention, the method calculates the neural conduction delay between the visual stream and the audio stream based on the audio-visual perception delay prediction model, and generates a corrected audio stream by compensating for it by moving the time axis forward. The specific steps are as follows:

[0021] Extracting visual feature vectors and audio feature vectors from the visual stream and audio stream respectively through a visual encoder and an audio encoder;

[0022] Constructing an audiovisual perception delay prediction model, and inputting the visual feature vector and the audio feature vector into the audiovisual perception delay prediction model to generate a neural conduction delay;

[0023] The audio stream is compensated for time axis advance according to the amount of nerve conduction delay to generate a corrected audio stream.

[0024] As a preferred solution of the short video production method based on artificial intelligence described in the present invention, wherein: the cross-modal attention weight matrix is ​​calculated according to the node association strength of the dynamic causal graph, and the text stream and the modified audio stream are projected into the joint feature space to generate multimodal fusion features. The specific steps are as follows:

[0025] Extracting the causal effect strength of nodes from the dynamic causal graph, constructing a node correlation strength matrix, and extracting text stream feature vectors and audio stream feature vectors from the text stream and the modified audio stream respectively;

[0026] Based on the text stream feature vector, audio stream feature vector and node association strength matrix, a cross-modal attention weight matrix is ​​generated through Hadamard product operation;

[0027] Based on the cross-modal attention weight matrix, the text stream and the modified audio stream are projected into the joint feature space through linear transformation, and multimodal fusion features are generated through compression through the fully connected layer.

[0028] As a preferred solution of the short video production method based on artificial intelligence described in the present invention, wherein: the multimodal fusion feature is input into the format adaptation engine, after the spatiotemporal consistency check, the neural implicit watermark is embedded and encoded into a short video. The specific steps are as follows:

[0029] Multimodal fusion feature input format adaptation engine calculates the motion vector field between adjacent feature vectors and generates a spatiotemporally consistent feature sequence through spatiotemporal consistency verification;

[0030] The time-space consistent feature sequence is embedded into the neural implicit watermark through the frequency domain watermark encoder, and the encoding parameters are dynamically adjusted to generate a short video.

[0031] In a second aspect, the present invention provides an artificial intelligence-based short video production system, including a causal storyboard module for collecting user original demand instructions, extracting a causal factor set through a causal discovery algorithm, constructing a dynamic causal graph including node association strength, and generating a storyboard script based on the topological structure of the dynamic causal graph; a multimodal generation module for inputting the storyboard script into a multimodal generation engine, using a lower triangular spatiotemporal attention matrix to generate a visual stream, and generating a text stream and an audio stream; a delay correction module for calculating the neural conduction delay between the visual stream and the audio stream based on an audio-visual perception delay prediction model, and generating a corrected audio stream by forward-shifting the time axis; a feature fusion module for calculating a cross-modal attention weight matrix based on the node association strength of the dynamic causal graph, and projecting the text stream and the corrected audio stream into a joint feature space to generate multimodal fusion features; a video synthesis module for inputting the multimodal fusion features into a format adaptation engine, embedding a neural implicit watermark after spatiotemporal consistency verification, and encoding them into a short video.

[0032] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the artificial intelligence-based short video production method as described in the first aspect of the present invention is implemented.

[0033] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the artificial intelligence-based short video production method as described in the first aspect of the present invention.

[0034] The beneficial effects of the present invention are: through dynamic causal graph modeling, the logical generation of short video storyboards is realized, so that the video content has a clear narrative structure and a reasonable time rhythm; at the same time, combined with the cross-modal attention mechanism, the node association strength in the causal graph is used to guide the fusion of multimodal information, thereby improving the semantic consistency and coordination between text, audio and visual content, thereby realizing high-quality and highly intelligent automatic generation of short videos as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 Flowchart of the short video production method based on artificial intelligence.

[0037] Figure 2 A flowchart for generating a structured storyboard.

[0038] Figure 3 Flowchart for generating a modified audio stream.

[0039] Figure 4 Flowchart for generating short videos. DETAILED DESCRIPTION

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0041] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0042] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0043] Reference Figures 1 to 4 , is an embodiment of the present invention, which provides a short video production method based on artificial intelligence, comprising the following steps:

[0044] S1: Collect the user's original demand instructions, extract the causal factor set through the causal discovery algorithm, build a dynamic causal graph including the node association strength, and generate a storyboard based on the topological structure of the dynamic causal graph;

[0045] S1.1: Perform lexical analysis on the user's original demand instructions using a word segmentation tool, split them into word sequences, remove stop words, and generate an initial vocabulary set;

[0046] Specifically, the input text data of the user's original demand instruction is received, all letters in the input text data are converted to lowercase, and punctuation marks are replaced with spaces; the processed input text data is input into the word segmentation tool, and combined with the dictionary matching strategy, word boundaries are identified according to the position of spaces and hyphens, and the input text data is divided into word sequences; a stop word set is loaded from the stop word list in the natural language processing tool NLTK library, and the stop word set includes meaningless articles, prepositions, pronouns and auxiliary verbs; each word in the word sequence is traversed, and words that exist in the stop word set are deleted. Words that do not exist in the stop word set are retained in the intermediate vocabulary set; word form restoration is performed on each word in the intermediate vocabulary set, and the words are converted into basic forms by querying the preset word form restoration dictionary to generate standardized words; the standardized words are arranged in the original order, and repeated adjacent identical words are removed to generate an initial vocabulary set;

[0047] It should be noted that the preset morphological restoration dictionary is based on the lexical morphological mapping relationship of WordNet 3.0, adopts the Penn Treebank part-of-speech tagging rules, combined with the verb frame set of WordNet (VerbNet), and applies the "Oxford-s / -es" plural inflection rule.

[0048] S1.2: Based on the initial vocabulary set, use the pretrained semantic role labeling model to annotate the predicate-argument structure in the word sequence and extract a set of candidate causal phrases;

[0049] Furthermore, the pre-trained semantic role labeling model adopts the BERT-Base model based on the Transformer architecture. The input layer maps standardized words into word vectors through word fragment tokenization to form a word vector sequence; the word vector sequence is input into the encoder layer of the pre-trained semantic role labeling model, and a label sequence is output; the label sequence is traversed, and when continuous BI labels are detected, the predicate-argument pairs are extracted to generate a structure set, in which the predicate in the predicate-argument pair is the standardized word corresponding to the verb word vector, and the argument is the standardized word corresponding to the noun word vector; the predicate-argument pairs that satisfy the causal relationship (matched using the cause synonym set in WordNet and in line with the PropBank semantic role definition) are screened from the structure set; the screened predicate-argument pairs are converted into phrase form to generate a set of candidate causal phrases containing causal factors.

[0050] S1.3: Filter the candidate causal phrase set based on the causal probability matrix and retain statistically significant causal factor pairs;

[0051] It should be noted that based on the co-occurrence frequency of the subject and the object in the candidate causal phrase set, the original frequency matrix is ​​formed, the original frequency matrix is ​​directionally normalized, and the conditional probability of each subject generating the object is calculated. The direction-normalized original frequency matrix is ​​converted into a conditional probability matrix through the conditional probability. The conditional probability matrix is ​​maximum-minimum normalized to generate the causal probability moment. The row and column indexes of the causal probability matrix correspond to the causal factors in the candidate causal phrase set. Furthermore, the Pearson conditional mutual information formula is used to calculate the causal strength, which is expressed as follows:

[0052]

[0053] Among them, M ij is the causal strength of the causal factor pair in the i-th row and j-th column of the causal probability matrix, c i is the causal factor of the i-th row in the causal probability matrix, c j is the causal factor of the jth column in the causal probability matrix, P(c i ,c j ,Pa(c j )) is the causal factor c i , causal factor c j and parent node set Pa(c j ) occur simultaneously, Pa(c j ) is the causal factor c j The parent node set, P(c i ,c j |Pa(c j )) is the given causal factor c j The parent node set Pa(c j ) under the condition that the causal factor c i Causal factor c j The joint conditional probability of simultaneous occurrence, P(c i |Pa(c j )) is the given parent node set Pa(c j ) under the condition that the causal factor c i The conditional probability of a single occurrence, P(c j |Pa(c j )) is the given parent node set Pa(c j ) under the condition that the causal factor c j Conditional probability of a single occurrence;

[0054] Count the co-occurrence frequencies of candidate causal phrase pairs and generate a frequency table containing four elements:

[0055] Causal factor c i and causal factor c j The number of simultaneous occurrences, causal factor ci Appear but causal factor c j Number of non-appearances, causal factor c j Appear but causal factor c i Number of non-appearances, causal factor c i and causal factor c j The number of times it did not appear.

[0056] Fisher's exact test method is used to obtain the statistical significance value of the causal factor pair based on the hypergeometric distribution of the frequency table. The statistical significance threshold is set according to the significance level (the Type I error rate threshold set by IEEE 1855-2016) (for example, the statistical significance threshold is 0.05). The causal factor pairs with statistical significance values ​​greater than the statistical significance threshold are retained as statistically significant causal factor pairs;

[0057] A quantile threshold is set according to historical causal data, and the causal strength of all causal factor pairs is extracted from the causal probability matrix. The causal factor pairs whose causal strength quantile distribution is higher than the quantile threshold are retained. The statistically significant causal factor pairs and the causal factor pairs above the quantile threshold are retained to generate a filtered set of candidate causal phrases.

[0058] It should be noted that the quantile threshold is determined through Monte Carlo cross-validation: the historical causal data are randomly divided into 80% training set and 20% validation set, the causal strength of all causal factor pairs are sorted in the training set, the quantile distribution of causal strength is obtained by linear interpolation, and the quantile that maximizes the causal relationship retention rate in the validation set is selected as the quantile threshold.

[0059] S1.4: Apply the partial directed acyclic graph (PC) algorithm to construct the causal graph skeleton and identify the set of causal factors;

[0060] Furthermore, each causal factor in the filtered candidate causal phrase set is taken as a causal factor node, and completely undirected edges are drawn between the causal factor nodes to construct a fully connected undirected graph. Any two adjacent causal factor node pairs are selected from the fully connected undirected graph, and the causal strength is obtained based on the frequency table of four elements. The undirected edges between the two causal factor nodes are deleted according to the causal strength, and the redundant edges are iteratively eliminated to generate an undirected skeleton graph. The V-shaped structure is identified in the undirected skeleton graph and combined with the remaining undirected edges after directed propagation to generate a causal graph skeleton. The candidate causal phrases corresponding to all causal factor nodes are extracted from the causal graph skeleton to form a causal factor set.

[0061] S1.5: Estimate the strength of causal effects using linear non-Gaussian models and construct weighted dynamic causal graphs;

[0062] Specifically, each causal factor in the causal factor set is standardized, and the independent component analysis algorithm is applied to the standardized causal factors to obtain a mixing matrix. The mixing matrix is ​​transposed into a connection matrix. The elements in the connection matrix represent the direct causal effect intensity between the causal factors. According to the positive or negative signs of the non-zero elements of the connection matrix (positive represents positive causality, and negative represents negative causality), the directional relationship between the causal factors is determined. The elements of the connection matrix are used as the causal effect intensity to generate a causal weight matrix. The causal weight matrix is ​​normalized, and directed edges are drawn according to the causal direction with the causal factor nodes as nodes. The weights of the directed edges are set to the normalized causal effect intensity to generate a weighted dynamic causal graph.

[0063] S1.6: Perform topological sorting on the weighted dynamic causal graph using the Kahn algorithm to generate an ordered sequence of causal events.

[0064] Furthermore, all nodes in the weighted dynamic causal graph are traversed, the number of predecessor nodes of each node is counted and recorded as the in-degree value, an initial in-degree table is generated, the initial in-degree table is scanned, nodes with an in-degree value of zero are added to the pending queue, a node is taken out from the pending queue, the node is appended to the end of the ordered sequence, all direct successor nodes of the node are traversed, the in-degree value of each successor node is reduced by one, when the in-degree value of a successor node is reduced to zero, the successor node is added to the ordered sequence, and the cycle is repeated until the pending queue is empty, thereby generating an ordered sequence of causal events.

[0065] S1.7: Based on the ordered sequence, obtain the causal edge weights and map the sum of the causal edge weights to the shot duration through linear interpolation to generate a set of shot durations.

[0066] It should be noted that the ordered sequence of causal events is traversed, and adjacent pairs of causal events are selected in turn. The directed edges between each pair of adjacent causal events are found in the weighted dynamic causal graph, and the corresponding weights are recorded to generate a causal edge weight set. All weights in the causal edge weight set are accumulated to obtain the total weight sum. The sum of the causal edge weights is mapped to the shot duration through linear interpolation to generate a shot duration set.

[0067] S1.8: Generate storyboard descriptions based on ordered sequences using causal factor triplets, and combine them with the storyboard duration set to form a structured storyboard script;

[0068] Specifically, the ordered sequence of causal events is traversed, and the causal factor triples corresponding to each causal event, including the subject, action, and object, are extracted; the standardized names of the subject and object in the causal factor triples are mapped to the domain terminology library, a natural language generation template is defined, and the causal factor triples are filled into the template slots to generate a natural language description of the storyboard description; in the order of the ordered sequence, the storyboard descriptions are matched one by one with the corresponding storyboard durations in the storyboard duration set to generate a key-value pair set, where the key is the storyboard description text and the value is the storyboard duration; the key-value pair set is arranged in an ordered sequence to generate a structured storyboard script, where each storyboard entry in the storyboard script contains a storyboard description and a storyboard duration in the form of a JSON array;

[0069] S2: Input the storyboard into the multimodal generation engine, use the lower triangular spatiotemporal attention matrix to generate the visual stream, and generate the text stream and audio stream;

[0070] S2.1: Input the storyboard into the multimodal generation engine, parse the scene description and timing data of the storyboard, and generate a multimodal generation instruction sequence;

[0071] The storyboard is input into the multimodal generation engine, and based on the causal factor triples, a multimodal generation instruction sequence including visual elements, text content and audio tags is generated through multimodal instruction element mapping;

[0072] Visual elements: Based on the verbs and objects in the causal factor triples, graphic / animation templates are matched from the domain terminology library;

[0073] Text content: Input the causal factor triples into the causal constraint text generation model to generate concise narration text and subtitle content;

[0074] It should be noted that based on the GPT-2 architecture, the storyboard script and the causal factor triplet are encoded as input through structured causal control symbols, and a causal-aware adaptation layer is added, including independent control symbol embedding, constrained attention head and gated FFN unit; a finite state machine constrained decoding strategy is adopted, combined with regular expression rules to dynamically filter the candidate word list obtained by the word segmentation tool, and a probabilistic bias is imposed on causally related words; fine-tuning is performed on the FilmScript-12K domain dataset through a multi-task loss function to ensure that the generated text is strictly consistent with the logic of the causal graph, and finally a causal constraint text generation model that outputs concise narration text and subtitle content is obtained.

[0075] Audio Tagging: Match audio clips from a sound library based on verb type.

[0076] S2.2: Construct the lower triangular spatiotemporal attention matrix to generate matrix-enforced visual flow;

[0077] Parse the array length of the storyboard script to obtain the number of storyboards, and construct an attention matrix with the dimension of the number of storyboards. Force the elements on the main diagonal and above of the attention matrix to zero to generate a lower triangular matrix; for each non-zero element position in the lower triangular matrix, obtain the time interval through the storyboard duration, and apply the natural exponential decay function to calculate the weight to generate a lower triangular spatiotemporal attention matrix; serialize the visual elements in the multimodal generation instruction sequence into visual element vectors, and generate aggregated visual features through matrix multiplication, and encode the aggregated visual feature matrix into a matrix-forced visual flow.

[0078] S2.3: Based on matrix-enforced visual flow, generate visual flow through visual frame rendering, generate text flow through causal constraint text generation model, and generate audio flow through audio and video synchronization;

[0079] Based on the aggregated visual features, the graphics rendering engine is called to generate images frame by frame. The duration of each frame is set according to the storyboard duration of the storyboard script to generate a visual flow.

[0080] Input the causal factor triples into the causal constraint text generation model to generate a text stream including narration text and subtitle content, and set the start and end time of text display according to the storyboard duration of the storyboard script;

[0081] Match the storyboard description text of the storyboard script with the audio clips in the sound effect library, generate the storyboard start time based on the cumulative storyboard duration, set the audio trigger time point to the storyboard start time, and superimpose the narration voice (generated by the narration text in the text stream through the TTS engine) and the sound effect clips along the timeline to generate an audio stream.

[0082] S3: Calculate the neural conduction delay between the visual stream and the audio stream based on the audio-visual perception delay prediction model, and generate a corrected audio stream by compensating for it by shifting the time axis forward.

[0083] S3.1: extract visual feature vectors and audio feature vectors from the visual stream and audio stream respectively through the visual encoder and audio encoder;

[0084] It should be noted that each frame image in the visual stream is adjusted to a fixed resolution and pixel normalization is performed, and then input into the visual encoder ResNet-50. After removing the fully connected layer, the convolution layer and the global average pooling layer are used to extract the 2048-dimensional visual feature vector, and the visual feature vector sequence is generated by arranging them in the shot timing. The audio stream is divided into audio clips aligned with the shot timing, resampled to the standard sampling rate and converted into mono waveform data, and input into the audio encoder VGGish model to extract the 128-dimensional audio feature vector, and the audio feature vector sequence is generated by arranging them in the shot timing.

[0085] S3.2: Input the visual feature vector and the audio feature vector into the audiovisual perception delay prediction model to generate a neural conduction delay;

[0086] Furthermore, the visual feature vector and the audio feature vector are aligned according to the shot timing and spliced ​​into multimodal feature pairs through a time sliding window; the multimodal feature pairs are input into the audio-visual perception delay prediction model, and a bidirectional long short-term memory network layer is used to capture the temporal dependency, and the neural conduction delay is obtained through a cross-attention mechanism.

[0087] It should be noted that the construction process of the audiovisual perception delay prediction model is as follows: the visual features (frame-level CNN features extracted by ResNet-50) and the audio features (128-dimensional Mel spectrum embedding extracted by VGGish) are aligned to the same time axis through dynamic time warping (DTW), and then the synchronized multimodal feature pairs are intercepted with a sliding window (window length 1 second, step length 0.5 second); the audiovisual perception delay prediction model architecture adopts a dual-stream Bi-LSTM network, and the visual features and audio features are respectively input into independent Bi-LSTM layers (hidden layer 256 dimensions) to extract temporal context representation, and then the audiovisual feature correlation weights are calculated through the cross-attention mechanism to generate an attention-weighted joint representation; finally, the neural conduction delay amount is output through the fully connected layer (including ReLU activation), and the Huber loss function with temporal smoothing constraint is used for training.

[0088] S3.3: Perform time axis forward compensation on the audio stream according to the amount of nerve conduction delay to generate a corrected audio stream;

[0089] Specifically, based on the amount of nerve conduction delay, the start timestamp and end timestamp of the audio clip are both shifted forward by the amount of nerve conduction delay, and the silent gaps caused by the forward shift of the time axis are eliminated through audio waveform cropping and splicing technology to generate an audio clip with a corrected time axis; the corrected audio clip is recombined into continuous waveform data according to the storyboard timing to generate a corrected audio stream that is strictly synchronized with the visual stream.

[0090] S4: Calculate the cross-modal attention weight matrix based on the node correlation strength of the dynamic causal graph, and project the text stream and the modified audio stream into the joint feature space to generate multimodal fusion features;

[0091] S4.1: Extract the node correlation strength matrix from the dynamic causal graph, and extract the text stream feature vector and audio stream feature vector from the extracted text stream and the corrected audio stream respectively;

[0092] It should be noted that all causal factor node pairs in the dynamic causal graph are traversed, the directed edge weights (i.e., causal effect strength) are read, and a node association strength matrix is ​​generated; the narration text in the text stream is input into the encoder layer of the semantic role labeling model BERT-Base, and the 768-dimensional text stream feature vector of each storyboard description text is extracted; the audio clips of the corrected audio stream are input into the Embedding layer of the VGGish model, and the 128-dimensional audio stream feature vector is extracted.

[0093] S4.2: Calculate the cross-modal attention weight matrix based on the text stream feature vector, audio stream feature vector, and node association strength matrix;

[0094] Furthermore, the text stream feature vector and the audio stream feature vector are aligned according to the frame time sequence, and the cosine similarity of each pair of text-audio feature vectors is calculated through the dot product operation to generate a cross-modal similarity matrix; the cross-modal similarity matrix is ​​subjected to a Hadamard product operation with the node association strength matrix; the Hadamard product operation results are subjected to Softmax normalization processing row by row to generate a cross-modal attention weight matrix.

[0095] S4.3: Based on the cross-modal attention weight matrix, the text stream and the modified audio stream are projected into the joint feature space through linear transformation to generate a joint feature vector;

[0096] Specifically, the text stream feature vector and the corrected audio stream feature vector are aligned according to the frame sequence, and a weighted sum operation is performed on the text stream feature vector and the corrected audio stream feature vector through the cross-modal attention weight matrix to generate a cross-modal fusion feature vector; the cross-modal fusion feature vector is input into the fully connected layer (dimension is 2048+128→1024), and the dimension is reduced to the joint feature space through linear transformation to generate a joint feature vector with a dimension of 1024.

[0097] S4.4: Perform fully connected layer compression on the joint feature vector to generate multimodal fusion features.

[0098] It should be noted that the joint feature vector with a dimension of 1024 is input into the fully connected layer, and a linear transformation is performed through matrix multiplication and bias addition to compress the joint feature vector to 512 dimensions; the ReLU activation function is applied to the compressed feature vector to eliminate negative values ​​and enhance nonlinear expression capabilities, and output 512-dimensional multimodal fusion features.

[0099] S5: The multimodal fusion features are input into the format adaptation engine. After spatiotemporal consistency verification, the neural implicit watermark is embedded and encoded into a short video.

[0100] S5.1: Multimodal fusion features are input into the optical flow estimation model to calculate the motion vector field between adjacent feature vectors and generate a spatiotemporally consistent feature sequence through spatiotemporal consistency check;

[0101] Furthermore, the multimodal fusion features are input into the optical flow estimation model FlowNet2.0, and the displacements between adjacent feature vectors are matched frame by frame through the convolution layer and the correlation calculation layer to generate a vector field describing the feature motion direction; a bidirectional optical flow consistency check is performed on the vector field, the cyclic consistency error of the forward optical flow and the reverse optical flow is calculated, and the abnormal displacement is corrected by interpolation of adjacent frames (the abnormal displacement is distributed through the cyclic consistency error, combined with the adaptive statistical model of the local motion direction variance and the time domain interpolation residual to identify the abnormal motion vector area that deviates from the spatiotemporal continuity constraint); the verified vector field is superimposed on the multimodal fusion features in chronological order to generate a spatiotemporally consistent feature sequence.

[0102] S5.2: The time-space consistent feature sequence is embedded into a neural implicit watermark through a frequency domain watermark encoder, and the encoding parameters are dynamically adjusted to generate a short video;

[0103] The time-space consistent feature sequence is input into the frequency domain watermark encoder. The frequency domain watermark encoder maps the time-space consistent feature sequence to the frequency domain space through discrete cosine transform, and superimposes the disturbance signal based on neural implicit watermark in the low-frequency component. The disturbance intensity is dynamically adjusted according to the watermark robustness requirements of the target platform; the frequency domain signal embedded with the watermark is inversely transformed into the time domain to generate a time-space feature sequence containing the watermark; the standard video encoder of the target platform (such as the H.264 encoder) is called to render the time-space feature sequence into a video frame sequence, which is compressed and packaged according to the frame length and resolution requirements to generate a short video that meets the platform bit rate and packaging format.

[0104] This embodiment also provides an artificial intelligence-based short video production system, including: a causal storyboard module, which is used to collect user original demand instructions, extract a set of causal factors through a causal discovery algorithm, construct a dynamic causal graph including node association strength, and generate a storyboard script based on the topological structure of the dynamic causal graph; a multimodal generation module, which is used to input the storyboard script into a multimodal generation engine, use a lower triangular spatiotemporal attention matrix to generate a visual stream, and generate a text stream and an audio stream; a delay correction module, which is used to calculate the neural conduction delay between the visual stream and the audio stream based on an audio-visual perception delay prediction model, and generate a corrected audio stream by shifting the time axis forward to compensate; a feature fusion module, which is used to calculate the cross-modal attention weight matrix based on the node association strength of the dynamic causal graph, and project the text stream and the corrected audio stream into a joint feature space to generate multimodal fusion features; a video synthesis module, which is used to input the multimodal fusion features into a format adaptation engine, embed a neural implicit watermark after spatiotemporal consistency verification, and encode them into a short video.

[0105] This embodiment also provides a computer device suitable for the case of an artificial intelligence-based short video production method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the artificial intelligence-based short video production method proposed in the above embodiment.

[0106] The computer device may be a terminal, comprising a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse.

[0107] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the short video production method based on artificial intelligence as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.

[0108] In summary, the present invention achieves the logical generation of short video storyboards through: dynamic causal graph modeling, so that the video content has a clear narrative structure and a reasonable time rhythm; at the same time, it combines the cross-modal attention mechanism and uses the node association strength in the causal graph to guide multimodal information fusion, thereby improving the semantic consistency and coordination between text, audio and visual content, thereby achieving high-quality and highly intelligent automatic generation of short videos as a whole.

[0109] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A short video production method based on artificial intelligence, characterized by: include, Collect the user's original demand instructions, extract the causal factor set through the causal discovery algorithm, build a dynamic causal graph including the node correlation strength, and generate a storyboard based on the topological structure of the dynamic causal graph; The storyboard is fed into a multimodal generation engine, which uses a lower triangular spatiotemporal attention matrix to generate a visual stream, as well as a text stream and an audio stream. The neural conduction delay between the visual stream and the audio stream is calculated based on the audio-visual perception delay prediction model, and the modified audio stream is generated by compensating for it by shifting the time axis forward. The cross-modal attention weight matrix is ​​calculated based on the node correlation strength of the dynamic causal graph, and the text stream and the modified audio stream are projected into the joint feature space to generate multimodal fusion features; The multimodal fusion features are input into the format adaptation engine, and after spatiotemporal consistency verification, they are embedded with neural implicit watermarks and encoded into short videos.

2. The artificial intelligence-based short video production method according to claim 1, wherein: The causal factor set is extracted by the causal discovery algorithm. The specific steps are as follows: Perform lexical analysis on the user's original demand instructions through a word segmentation tool to generate an initial vocabulary set, and extract a set of candidate causal phrases from the initial vocabulary set; The causal factor set is identified from the causal phrase set using the partial directed acyclic graph (PC) algorithm.

3. The artificial intelligence-based short video production method according to claim 2, characterized in that: The specific steps of constructing a dynamic causal graph including node association strength and generating a storyboard based on the topological structure of the dynamic causal graph are as follows: Based on the causal factor set, the LiNGAM linear non-Gaussian acyclic model is used to estimate the causal effect strength and construct a dynamic causal graph including the node association strength. The dynamic causal graph is topologically sorted using the Kahn algorithm to generate an ordered sequence of causal events. Based on the causal edge weights of the dynamic causal graph, the duration of the storyboard is generated through linear interpolation. The causal factor triples corresponding to the causal events are extracted based on the ordered sequence, the causal factor triples are mapped to generate storyboard descriptions, and then combined with the storyboard duration to form a storyboard script.

4. The artificial intelligence-based short video production method according to claim 3, wherein: The storyboard script is input into the multimodal generation engine, and the lower triangular spatiotemporal attention matrix is ​​used to generate the visual stream, and the text stream and audio stream are generated. The specific steps are as follows: Input the storyboard script into the multimodal generation engine to generate a multimodal generation instruction sequence; Construct a lower triangular spatiotemporal attention matrix and generate matrix-enforced visual flow through matrix multiplication and matrix encoding; Based on matrix-enforced visual flow, visual flow is generated through visual frame rendering; Based on the visual stream, an audio stream is generated through audio-visual synchronization, and a text stream is extracted from the causal factor triples.

5. The artificial intelligence-based short video production method according to claim 4, characterized in that: The neural conduction delay between the visual stream and the audio stream is calculated based on the audiovisual perception delay prediction model, and the modified audio stream is generated by forward shifting the time axis for compensation. The specific steps are as follows: Extracting visual feature vectors and audio feature vectors from the visual stream and audio stream respectively through a visual encoder and an audio encoder; Constructing an audiovisual perception delay prediction model, and inputting the visual feature vector and the audio feature vector into the audiovisual perception delay prediction model to generate a neural conduction delay; The audio stream is compensated for time axis advance according to the amount of nerve conduction delay to generate a corrected audio stream.

6. The artificial intelligence-based short video production method according to claim 5, characterized in that: The cross-modal attention weight matrix is ​​calculated based on the node association strength of the dynamic causal graph, and the text stream and the modified audio stream are projected into the joint feature space to generate multimodal fusion features. The specific steps are as follows: Extracting the causal effect strength of nodes from the dynamic causal graph, constructing a node correlation strength matrix, and extracting text stream feature vectors and audio stream feature vectors from the text stream and the modified audio stream respectively; Based on the text stream feature vector, audio stream feature vector and node association strength matrix, a cross-modal attention weight matrix is ​​generated through Hadamard product operation; Based on the cross-modal attention weight matrix, the text stream and the modified audio stream are projected into the joint feature space through linear transformation, and multimodal fusion features are generated through compression through the fully connected layer.

7. The artificial intelligence-based short video production method according to claim 6, characterized in that: The multimodal fusion features are input into the format adaptation engine, and after the spatiotemporal consistency check, the neural implicit watermark is embedded and encoded into a short video. The specific steps are as follows: Multimodal fusion feature input format adaptation engine calculates the motion vector field between adjacent feature vectors and generates a spatiotemporally consistent feature sequence through spatiotemporal consistency verification; The time-space consistent feature sequence is embedded into the neural implicit watermark through the frequency domain watermark encoder, and the encoding parameters are dynamically adjusted to generate a short video.

8. An artificial intelligence-based short video production system, based on the artificial intelligence-based short video production method according to any one of claims 1 to 7, characterized in that: Including causal storyboard module, multimodal generation module, delay correction module, feature fusion module and video synthesis module, The causal storyboard module is used to collect the user's original demand instructions, extract the causal factor set through the causal discovery algorithm, construct a dynamic causal graph including the node association strength, and generate the storyboard script based on the topological structure of the dynamic causal graph; The multimodal generation module is used to input the storyboard into the multimodal generation engine, generate the visual stream using the lower triangular spatiotemporal attention matrix, and generate the text stream and audio stream; A delay correction module is used to calculate the neural conduction delay between the visual stream and the audio stream based on the audio-visual perception delay prediction model, and to generate a corrected audio stream by compensating for it by shifting the time axis forward; The feature fusion module is used to calculate the cross-modal attention weight matrix based on the node correlation strength of the dynamic causal graph, and project the text stream and the modified audio stream into the joint feature space to generate multimodal fusion features; The video synthesis module is used to input multimodal fusion features into the format adaptation engine. After spatiotemporal consistency verification, the neural implicit watermark is embedded and encoded into a short video.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the artificial intelligence-based short video production method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the artificial intelligence-based short video production method according to any one of claims 1 to 7 are implemented.

Images