Video plot generation and scene synthesis method and system based on natural language processing

Abstract

The present invention provides a method and system for video plot generation and scene synthesis based on natural language processing, which relates to the field of video generation technology, including: receiving script text to construct a multi-layer scene graph, extracting keywords to calculate semantic relevance, performing feature decomposition and reconstruction to obtain a scene synthesis vector, generating video scenes based on distance measurement, extracting spatiotemporal features using a feature pyramid, performing adaptive feature fusion and applying a self-attention mechanism to segment the video, inserting transition effects, performing style migration, and outputting a finished video.

Description

Technical Field

[0001] The present invention relates to the field of video generation technology, and in particular to a method and system for video plot generation and scene synthesis based on natural language processing. Background Art

[0002] With the rapid development of artificial intelligence and computer vision technologies, technology that converts text content into video scenes based on natural language processing has gradually become a research hotspot. By parsing text descriptions, extracting semantic features, and utilizing corresponding visual generation models, it can achieve automatic conversion from script text to video content. Traditional video production processes usually require a lot of manpower and material resources for shooting, editing, and post-processing. However, video generation technology based on natural language processing provides an efficient and automated solution that can significantly reduce video production costs and improve production efficiency.

[0003] However, existing technologies still lack in-depth understanding of scene semantics and have limited ability to grasp complex plots and multi-level narrative structures. This leads to significant differences in semantic expression between the generated video scenes and the original script text, making it impossible to accurately reflect the narrative logic and emotional expression of the script, and the reconstruction and representation of visual features are not precise enough. There is a lack of effective feature fusion mechanisms, making it difficult to generate video content with natural visual effects and a unified style. In particular, the technology performs poorly in handling scene transitions and visual coherence, and lacks a systematic approach to video sequence combination and style processing. It is impossible to achieve intelligent segmentation and transition effect generation based on content semantics. At the same time, the style transfer process lacks consideration for the global consistency of the video, making it difficult to ensure the artistic expression and professional quality of the final video work.

[0004] Therefore, a solution is urgently needed to solve the problems existing in the prior art. Summary of the Invention

[0005] The embodiments of the present invention provide a method and system for video plot generation and scene synthesis based on natural language processing, which can at least solve some of the problems existing in the prior art.

[0006] A first aspect of an embodiment of the present invention provides a method for video plot generation and scene synthesis based on natural language processing, comprising:

[0007] Receive script text input by the user and extract scene semantic features, and construct the scene semantic features into a multi-layer scene graph;

[0008] Extracting a keyword sequence based on the multi-layer scene graph, calculating the semantic relevance of the keyword sequence to obtain a feature vector matrix, performing semantic feature decomposition and visual feature reconstruction on the feature vector matrix to obtain a scene description vector and a scene structure vector, alternately updating feature representations based on the scene description vector and the scene structure vector to obtain a scene synthesis vector, calculating the similarity between the scene synthesis vector and a reference sample in a video sample library to obtain a distance metric value, and when the distance metric value is less than a preset threshold, generating a target video scene based on the current scene synthesis vector;

[0009] Based on a pre-constructed feature pyramid and the target video scene, spatiotemporal features are extracted at each layer in combination with the inter-layer correlation relationship in the multi-layer scene graph to obtain a feature graph sequence, adaptive feature fusion is performed on the feature graph sequence to obtain a scene feature representation and calculate an inter-frame similarity matrix, a self-attention mechanism is applied to the similarity matrix to obtain a scene key frame index and segment the video content, and a transition effect based on the depth map is inserted between adjacent segments to obtain a combined video sequence;

[0010] Perform style transfer on the combined video sequence and output a finished video.

[0011] In an optional embodiment,

[0012] Receiving the script text input by the user and extracting scene semantic features, and constructing the scene semantic features into a multi-layer scene graph includes:

[0013] Receive a script text input by a user, perform semantic segmentation on the script text to obtain a word sequence, and encode the word sequence based on a pre-trained language model to obtain initial semantic features;

[0014] Inputting the initial semantic features into a multi-head attention network, computing feature representations of different attention heads in parallel, and performing hierarchical clustering on the feature representations to obtain scene semantic features, wherein each attention head focuses on semantic information of character interactions, scene attributes, and plot development respectively;

[0015] A multi-layer scene graph is constructed based on the scene semantic features, and the neighborhood relationship of the nodes in the graph is calculated using the graph attention network. Nodes with similar semantic features are dynamically aggregated into a character relationship layer, a scene attribute layer, and a plot development layer, and the correlation relationship between the layers is determined.

[0016] In an optional embodiment,

[0017] Extracting a keyword sequence based on the multi-layer scene graph, calculating the semantic relevance of the keyword sequence to obtain a feature vector matrix, performing semantic feature decomposition and visual feature reconstruction on the feature vector matrix to obtain a scene description vector and a scene structure vector, and alternately updating feature representation according to the scene description vector and the scene structure vector to obtain a scene synthesis vector includes:

[0018] Extract node features from the multi-layer scene graph, calculate the information entropy of each level node and its adjacent nodes to obtain node weights, perform hierarchical clustering based on the node weights, and select cluster centers to form a keyword sequence;

[0019] Calculating semantic relevance for word pairs in the keyword sequence to obtain a relevance matrix, wherein the semantic relevance is obtained by multiplying the cosine distance of the word vectors by the edge weights between the nodes in the graph, and subjecting the relevance matrix to weight mapping and nonlinear transformation to obtain a feature vector matrix;

[0020] Performing semantic feature decomposition on the feature vector matrix to obtain a scene description vector, and performing visual feature reconstruction to obtain a scene structure vector;

[0021] The scene description vector and the scene structure vector are alternately iteratively optimized to obtain candidate feature representations, a joint optimization objective of semantic consistency loss, visual reconstruction loss, and temporal continuity loss is calculated based on the candidate feature representations, the candidate feature representations are updated by minimizing the joint optimization objective until convergence, and the converged feature representation is determined as the scene synthesis vector.

[0022] In an optional embodiment,

[0023] Calculating the similarity between the scene synthesis vector and a reference sample in a video sample library to obtain a distance metric value, and when the distance metric value is less than a preset threshold, generating a target video scene based on the current scene synthesis vector includes:

[0024] Constructing a scene semantic gradient map, calculating information entropy based on normalized eigenvalues ​​of scene elements in the scene semantic gradient map, obtaining importance distribution of scene elements, dividing the scene into a key area and a transition area based on the information entropy and the importance distribution, extracting a first scene feature vector corresponding to the key area and a second scene feature vector corresponding to the transition area, and calculating the semantic association strength between the key area and the transition area based on the Euclidean distance between the first scene feature vector and the second scene feature vector;

[0025] Extracting the scene synthesis vector and reference samples from the video sample library, performing a cosine distance metric on the key region, performing a Mahalanobis distance metric based on a feature covariance matrix on the transition region, weighting the cosine distance metric values ​​of the key region and the Mahalanobis distance metric values ​​of the transition region by a preset region weight coefficient, weighting the semantic association strength in combination with a pre-acquired semantic association coefficient, and combining the weighted values ​​to obtain the distance metric value;

[0026] Calculate the distance gradient of the distance metric value, superimpose the product of the distance gradient and the update step size on the original area boundary to obtain an updated area boundary, and determine whether the distance metric value is less than a preset threshold based on the updated area boundary. If so, generate the target video scene based on the scene synthesis vector.

[0027] In an optional embodiment,

[0028] Based on the pre-built feature pyramid and the target video scene, the spatiotemporal features are extracted at each layer in combination with the inter-layer correlation relationship in the multi-layer scene graph to obtain a feature graph sequence. Adaptive feature fusion is performed on the feature graph sequence to obtain a scene feature representation and calculate an inter-frame similarity matrix. A self-attention mechanism is applied to the similarity matrix to obtain a scene key frame index and segment the video content. A transition effect based on the depth map is inserted between adjacent segments to obtain a combined video sequence including:

[0029] Based on the target video scene, a feature map of each layer in a pre-constructed feature pyramid is obtained. In combination with the inter-layer correlation relationship in the multi-layer scene graph, spatiotemporal features are extracted from the feature map by multi-scale convolution. Convolution operation is performed on the feature map corresponding to each layer using convolution kernel parameters and a bias term is superimposed to obtain the feature map sequence.

[0030] Performing a matrix product operation on the query matrix of the current layer of the feature pyramid and the key matrix of the adjacent layer, normalizing the calculation result and multiplying it with the value matrix of the adjacent layer to obtain feature association data, and fusing the feature association data with the feature graph sequence to obtain a feature sequence with enhanced association;

[0031] performing average pooling and multi-layer perceptron processing on the association-enhanced feature sequence to obtain channel attention weights, performing weighted summation of the channel attention weights and the association-enhanced feature sequence to obtain the scene feature representation, calculating the inter-frame similarity matrix based on the scene feature representation, and applying a self-attention mechanism to extract the scene key frame index;

[0032] The video content is segmented according to the scene key frame index to obtain video segments, a transition effect is generated based on a time-varying interpolation coefficient calculated by a cosine function and a depth map of adjacent video segments, and the video segments and the transition effect are combined to obtain the combined video sequence.

[0033] In an optional embodiment,

[0034] Generating a transition effect based on a time-varying interpolation coefficient calculated based on a cosine function and a depth map of adjacent video segments, and combining the video segments with the transition effect to obtain the combined video sequence includes:

[0035] For the transition period between adjacent video segments, the time-varying interpolation coefficient is calculated based on the cosine function using the ratio of the current time point to the transition duration;

[0036] Collecting scene information of the adjacent video segments, performing position encoding on the spatial position and viewing direction of any point in the current scene using a multi-layer perceptron to obtain encoding features, and constructing a neural radiation field representation including spatial density features and color features based on the encoding features;

[0037] Obtaining depth maps of the adjacent video segments, calculating a depth consistency loss between a spatial density integral of a camera ray and the depth map, optimizing the neural radiance field representation based on the depth consistency loss, applying the time-varying interpolation coefficients to the optimized neural radiance field representation for weighted fusion, performing volume rendering on the fused features, and generating an intermediate view sequence by calculating an integral of cumulative transmittance and spatial density and color features;

[0038] The gradient difference of the depth map is calculated to obtain a depth consistency weight, the depth consistency weight is applied to the intermediate view sequence and the depth map weighted by the time-varying interpolation coefficient to perform adaptive fusion to obtain a transition effect, and the adjacent video segments and the transition effect are combined in time sequence to obtain the combined video sequence.

[0039] In an optional embodiment,

[0040] Performing style transfer on the combined video sequence, and outputting a finished video including:

[0041] Acquire the combined video sequence, and extract the spatiotemporal features and feature graph sequence corresponding to the combined video sequence;

[0042] Mapping and matching the spatiotemporal features and feature map sequence corresponding to the combined video sequence with reference to the feature space of the target style template to obtain fused features after style transfer;

[0043] The fused features after style transfer are reconstructed and decoded to obtain a video product with unified style.

[0044] A second aspect of an embodiment of the present invention provides a video plot generation and scene synthesis system based on natural language processing, comprising:

[0045] The first unit is configured to receive a script text input by a user and extract scene semantic features, and construct the scene semantic features into a multi-layer scene graph;

[0046] The second unit is configured to extract a keyword sequence based on the multi-layer scene graph, calculate the semantic relevance of the keyword sequence to obtain a feature vector matrix, perform semantic feature decomposition and visual feature reconstruction on the feature vector matrix to obtain a scene description vector and a scene structure vector, alternately update feature representation according to the scene description vector and the scene structure vector to obtain a scene synthesis vector, calculate the similarity between the scene synthesis vector and a reference sample in a video sample library to obtain a distance metric value, and when the distance metric value is less than a preset threshold, generate a target video scene based on the current scene synthesis vector;

[0047] The third unit is configured to extract spatiotemporal features at each layer based on a pre-constructed feature pyramid and the target video scene, in combination with the inter-layer correlation relationship in the multi-layer scene graph, to obtain a feature graph sequence, perform adaptive feature fusion on the feature graph sequence, obtain a scene feature representation, and calculate an inter-frame similarity matrix, apply a self-attention mechanism to the similarity matrix to obtain a scene key frame index and segment the video content, insert a transition effect based on the depth map between adjacent segments, and obtain a combined video sequence;

[0048] The fourth unit is configured to perform style migration on the combined video sequence and output a finished video.

[0049] According to a third aspect of an embodiment of the present invention, an electronic device is provided, including:

[0050] A processor and a memory for storing processor-executable instructions, wherein the processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0051] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0052] In the present invention, by constructing a multi-layer scene graph to extract the semantic features of the script, the accurate conversion from natural language to visual content is achieved, which effectively solves the problems of semantic information loss and inconsistent visual expression in traditional methods. Through the similarity matching mechanism of scene synthesis vectors and reference samples, a video scene that is highly consistent with the original script can be generated, while retaining the creator's intention to express, improving the narrative coherence and visual expressiveness of the video content. The processing method based on feature pyramid and adaptive feature fusion, combined with depth map transition effects and style transfer technology, realizes high-quality video sequence generation, significantly improves video editing efficiency and product quality, and provides intelligent and automated technical support for film and television creation. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of the process of video plot generation and scene synthesis method based on natural language processing according to an embodiment of the present invention;

[0054] Figure 2 Build a system architecture diagram for script text analysis and multi-layer scene graph;

[0055] Figure 3 Schematic diagram of the correlation between feature pyramid layers and adaptive feature fusion;

[0056] Figure 4 A schematic diagram showing the performance comparison of video transition effects based on neural radiation fields. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0058] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0059] Figure 1 FIG. 1 is a flow chart of a method for video plot generation and scene synthesis based on natural language processing according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0060] Receive script text input by the user and extract scene semantic features, and construct the scene semantic features into a multi-layer scene graph;

[0061] Extracting a keyword sequence based on the multi-layer scene graph, calculating the semantic relevance of the keyword sequence to obtain a feature vector matrix, performing semantic feature decomposition and visual feature reconstruction on the feature vector matrix to obtain a scene description vector and a scene structure vector, alternately updating feature representations based on the scene description vector and the scene structure vector to obtain a scene synthesis vector, calculating the similarity between the scene synthesis vector and a reference sample in a video sample library to obtain a distance metric value, and when the distance metric value is less than a preset threshold, generating a target video scene based on the current scene synthesis vector;

[0062] Based on a pre-constructed feature pyramid and the target video scene, spatiotemporal features are extracted at each layer in combination with the inter-layer correlation relationship in the multi-layer scene graph to obtain a feature graph sequence, adaptive feature fusion is performed on the feature graph sequence to obtain a scene feature representation and calculate an inter-frame similarity matrix, a self-attention mechanism is applied to the similarity matrix to obtain a scene key frame index and segment the video content, and a transition effect based on the depth map is inserted between adjacent segments to obtain a combined video sequence;

[0063] Perform style transfer on the combined video sequence and output a finished video.

[0064] In an optional embodiment,

[0065] Receiving the script text input by the user and extracting scene semantic features, and constructing the scene semantic features into a multi-layer scene graph includes:

[0066] Receive a script text input by a user, perform semantic segmentation on the script text to obtain a word sequence, and encode the word sequence based on a pre-trained language model to obtain initial semantic features;

[0067] Inputting the initial semantic features into a multi-head attention network, computing feature representations of different attention heads in parallel, and performing hierarchical clustering on the feature representations to obtain scene semantic features, wherein each attention head focuses on semantic information of character interactions, scene attributes, and plot development respectively;

[0068] A multi-layer scene graph is constructed based on the scene semantic features, and the neighborhood relationship of the nodes in the graph is calculated using the graph attention network. Nodes with similar semantic features are dynamically aggregated into a character relationship layer, a scene attribute layer, and a plot development layer, and the correlation relationship between the layers is determined.

[0069] Figure 2 A schematic diagram of the system architecture for script text analysis and multi-layer scene graph construction, such as Figure 2As shown, the script text input by the user is received, and semantic word segmentation is performed on the received script text to obtain a word sequence. During the semantic word segmentation process, a word segmentation model based on a bidirectional long short-term memory network is adopted. By encoding the characters in the text, the word boundaries are identified, thereby generating a word sequence. For example, for the input script fragment "On a rainy night, Li Ming pushed open the door of the coffee shop and looked around for the friend who had an appointment to meet", it is segmented into ["On a rainy night", ",", "Li Ming", "pushed open", "the coffee shop", "of", "the door", ",", "looked around", "looked for", "had an appointment", "to meet", "of", "the friend"].

[0070] After obtaining the word sequence, the initial semantic features are encoded based on a pre-trained language model. A pre-trained language model with a 12-layer Transformer encoder structure is adopted, and the model has been pre-trained on a large-scale text corpus. For each input word, it is converted into a 512-dimensional word embedding vector, and then the feature extraction is performed through the multi-layer encoder of the language model to obtain the initial semantic feature vector representing the semantics of the word. Each word corresponds to a 768-dimensional vector. Taking the above word segmentation result as an example, an initial semantic feature set containing 16 768-dimensional vectors will be generated.

[0071] The initial semantic features are input into the multi-head attention network, and the feature representations of different attention heads are calculated in parallel. The multi-head attention network contains 8 attention heads, among which 3 attention heads specifically focus on character interaction information, 3 attention heads focus on scene attribute information, and 2 attention heads focus on plot development information. Each attention head adopts the self-attention mechanism to capture the correlation between words by calculating the similarity between the query vector, key vector and value vector. Taking the attention head that focuses on character interaction as an example, higher attention weights are given to the character names and the words describing the character's behavior. In the example, "Li Ming", "pushed open", "looked around", "looked for", "friend" obtain higher weights in the character interaction attention head, forming a feature representation focused on character interaction.

[0072] When performing hierarchical clustering on feature representations to obtain scene semantic features, a bottom-up hierarchical clustering algorithm is used to calculate the cosine similarity between each word feature representation and construct a similarity matrix. Initially, each word feature is treated as a separate cluster. The system gradually merges clusters with the highest similarity until a preset number of clusters is reached. For character interaction features, the similarity threshold is set to 0.75, forming semantic clusters such as "character-action-object"; for scene attribute features, the similarity threshold is set to 0.7, forming semantic clusters such as "scene-description-state"; and for plot development features, the similarity threshold is set to 0.8, forming semantic clusters representing plot changes. In this example, "Li Ming - Pushing the Door Open" and "Li Ming - Looking for Friends" are generated as character interaction semantic clusters, "Rainy Night - Café" is generated as a scene attribute semantic cluster, and "Pushing the Door Open - Looking Around - Looking for Friends" is generated as a plot development semantic cluster.

[0073] When constructing a multi-layer scene graph based on scene semantic features, a graph attention network is used to calculate the neighborhood relationships of nodes in the graph. Each semantic cluster is treated as a node in the graph, and node features are derived by taking the weighted average of the features of all words in the cluster. The graph attention network consists of two graph convolutional layers, each with 256 hidden units. The graph attention mechanism calculates the strength of the relationship between each node and its neighboring nodes. This relationship strength is determined by the similarity of the node features. For each pair of nodes, an attention coefficient is calculated. The attention coefficient is calculated by taking the inner product of the linearly transformed node features and normalizing it using the SoftMax function.

[0074] Nodes with similar semantic features are dynamically aggregated into the character relationship layer, scene attribute layer, and plot development layer, using a hierarchical division based on the node representations learned by the graph attention network. For the character relationship layer, nodes containing character names and focusing on character interactions are aggregated into this layer; for the scene attribute layer, nodes describing scene elements such as environment, time, and space are aggregated into this layer; and for the plot development layer, nodes describing the development of events and plot changes are aggregated into this layer. For example, the nodes "Li Ming - Pushing the Door Open" and "Li Ming - Looking for Friends" are aggregated into the character relationship layer, the node "Rainy Night - Café" is aggregated into the scene attribute layer, and the node "Pushing the Door Open - Looking Around - Looking for Friends" is aggregated into the plot development layer.

[0075] To determine inter-layer associations, the cross-attention scores between nodes in different layers are calculated. This score is calculated by taking the dot product between node features, and a threshold is used to filter out weak associations. Node pairs with a cross-attention score greater than 0.6 are connected to each other, forming a complete multi-layer scene graph. In this constructed multi-layer scene graph, the "Rainy Night - Cafe" node in the scene attribute layer is associated with the "Li Ming - Pushing the Door" node in the character relationship layer, and with the "Pushing the Door - Looking Around - Finding Friends" node in the plot development layer, forming a multi-layer semantic representation that describes the complete scene.

[0076] In this embodiment, by semantically processing the script text and constructing a multi-layer scene graph, a deep semantic understanding and structured expression of the script content is achieved, and key semantic information such as character interactions, scene attributes and plot development in the script is automatically identified and distinguished. The originally flat text content is converted into a hierarchical semantic structure, which not only retains the core semantic content of the original script, but also clarifies the relationship between each element through the graph structure, providing a richer and more accurate semantic basis for subsequent video scene generation, thereby enabling the generation of video scenes that are highly matched with the script content, and improving the accuracy and content consistency of the conversion from text to video.

[0077] In an optional embodiment,

[0078] Extracting a keyword sequence based on the multi-layer scene graph, calculating the semantic relevance of the keyword sequence to obtain a feature vector matrix, performing semantic feature decomposition and visual feature reconstruction on the feature vector matrix to obtain a scene description vector and a scene structure vector, and alternately updating feature representation according to the scene description vector and the scene structure vector to obtain a scene synthesis vector includes:

[0079] Extract node features from the multi-layer scene graph, calculate the information entropy of each level node and its adjacent nodes to obtain node weights, perform hierarchical clustering based on the node weights, and select cluster centers to form a keyword sequence;

[0080] Calculating semantic relevance for word pairs in the keyword sequence to obtain a relevance matrix, wherein the semantic relevance is obtained by multiplying the cosine distance of the word vectors by the edge weights between the nodes in the graph, and subjecting the relevance matrix to weight mapping and nonlinear transformation to obtain a feature vector matrix;

[0081] Performing semantic feature decomposition on the feature vector matrix to obtain a scene description vector, and performing visual feature reconstruction to obtain a scene structure vector;

[0082] The scene description vector and the scene structure vector are alternately iteratively optimized to obtain candidate feature representations, a joint optimization objective of semantic consistency loss, visual reconstruction loss, and temporal continuity loss is calculated based on the candidate feature representations, the candidate feature representations are updated by minimizing the joint optimization objective until convergence, and the converged feature representation is determined as the scene synthesis vector.

[0083] A multi-layer scene graph is constructed, which contains object nodes, relationship nodes, and their hierarchical structure. Keyword sequences are extracted from this graph and their semantic relevance is calculated. Finally, a scene synthesis vector is generated for subsequent scene generation tasks.

[0084] Information entropy is calculated for each node to determine its importance. For node vi in ​​the graph, the amount of information transferred between it and its adjacent node vj is calculated. The feature representation fi of node vi is obtained, including node type, attributes, and contextual information. For object nodes, features include object category, location, and size; for relationship nodes, features include relationship type and associated object information. For the scenario "a person sitting in a chair reading a book" as an example, the features of the node "person" include the category "person", the location coordinates (x1, y1), and the attribute "adult male".

[0085] For node vi and all of its adjacent nodes vj, calculate the information entropy Hi of node vi, taking into account the connection strength wij between nodes, representing the degree of association between the two nodes. For example, the connection strength between "person" and "sitting" might be 0.85, while the connection strength between "person" and "reading" might be 0.78. Based on the calculated information entropy Hi, assign a weight wi to each node. A higher weight indicates a more important node in scene understanding. In this example scenario, the weights of the nodes "person," "chair," "book," "sitting," and "reading" might be 0.92, 0.87, 0.83, 0.79, and 0.76, respectively.

[0086] Hierarchical clustering is performed based on node weights. The clustering process uses a bottom-up approach, initially treating each node as a separate cluster and gradually merging clusters whose similarity exceeds a pre-set similarity threshold. The similarity calculation considers the cosine similarity of node feature vectors and the distances between nodes in the graph. When clustering reaches a pre-set stopping criterion (such as the number of clusters or a threshold for intra-cluster variance), the node with the highest weight is selected from each cluster as a representative to form a keyword sequence. In the above example, the resulting keyword sequence might be ["person", "chair", "book", "reading"].

[0087] After obtaining the keyword sequence, calculate the semantic relatedness between word pairs in the sequence. For each pair of words \(w_i\) and \(w_j\) in the sequence, obtain their pre-trained word vectors \(v_i\) and \(v_j\), and calculate the cosine similarity of the word vectors \(\text{sim}(v_i, v_j)\). At the same time, look up the edge weight \(\text{edge}(w_i, w_j)\) between the corresponding nodes in the graph. The semantic relatedness is obtained by multiplying these two factors: \(\text{rel}(w_i, w_j)=\text{sim}(v_i, v_j)\times\text{edge}(w_i, w_j)\). In actual calculation, for example, the cosine similarity of the word vectors of "person" and "read" is 0.65, and the edge weight in the graph is 0.78, then their semantic relatedness is 0.507.

[0088] After calculating all word pairs, construct an \(n\times n\) relatedness matrix \(R\), where \(n\) is the length of the keyword sequence. To enhance the expressive power of the matrix, apply a weight mapping function to adjust \(R\), and perform non-linear transformations such as ReLU to obtain the feature vector matrix \(F\). For a sequence of 4 keywords, \(F\) may be a \(4\times128\) matrix, and each row represents the feature representation of a keyword.

[0089] Perform semantic feature decomposition on the feature vector matrix \(F\) to extract the core semantic information, and retain the main semantic information and filter out noise through dimensionality reduction techniques. Project \(F\) into the semantic space to obtain a scene description vector of dimension \(d\). For the example scene, the scene description vector is a 128-dimensional vector that captures the core semantics of "a person is reading a book on a chair".

[0090] Perform visual feature reconstruction, map \(F\) to the visual representation space, consider the spatial relationships and visual attributes between objects, and generate a scene structure vector. The scene structure vector encodes the spatial layout and visual relationships of the objects in the scene, such as structural information like "the person is above the chair" and "the book is in front of the person".

[0091] Take the scene description vector and the scene structure vector as the initial inputs, and update the feature representation through an alternating iterative optimization method. In each iteration, the system updates the scene structure vector based on the current scene description vector, and then updates the scene description vector based on the updated scene structure vector, forming a mutually promoting optimization process. The optimization objectives include three parts: the semantic consistency loss ensures that the feature representation is consistent with the original semantics; the visual reconstruction loss ensures that the features can correctly reconstruct the visual scene; the temporal continuity loss ensures a smooth transition in time when processing continuous scenes.

[0092] Taking the above scene as an example, the initial scene description vector may focus on the semantic representation of the "reading" behavior, while the initial scene structure vector may not fully capture the spatial relationship between the "person" and the "chair". Through iterative optimization, the final scene description vector will contain richer semantic information, and the scene structure vector will more accurately express the spatial structure, and the two together form a comprehensive scene understanding.

[0093] When the joint optimization objective falls below a preset threshold (e.g., 0.001) or reaches the maximum number of iterations (e.g., 50), the optimization is considered converged, and the final feature representation is determined as the scene synthesis vector. The scene synthesis vector integrates the semantic content and structural information of the scene, providing a comprehensive scene representation for subsequent scene generation or understanding tasks.

[0094] In this embodiment, node weights are calculated based on information entropy and hierarchical clustering is performed, so that the extracted keyword sequence can accurately reflect the core semantics of the multi-layer scene graph. Semantic relevance is calculated by combining the word vector cosine distance with the graph structure edge weight, which fully integrates the information of the semantic space and the graph structure space, enhances the expressive power of the relevance matrix, and performs semantic feature decomposition and visual feature reconstruction on the feature vector matrix respectively to obtain complementary scene description vectors and scene structure vectors, solving the problem that traditional methods cannot take into account both semantic expression and visual structure at the same time.

[0095] Existing technologies typically use simple keyword matching or single feature extraction methods, which are difficult to accurately capture the complex semantic relationships and visual structure information in the script. This results in the generated scenes not matching the semantics of the original script and lacking visual expressiveness.

[0096] This embodiment significantly improves the semantic expression ability and visual structured representation quality of scene synthesis vectors, provides a more accurate feature basis for subsequent video scene generation, and effectively solves technical problems in traditional methods such as insufficient semantic-visual conversion, single feature expression, and one-sided optimization targets.

[0097] In an optional embodiment,

[0098] Calculating the similarity between the scene synthesis vector and a reference sample in a video sample library to obtain a distance metric value, and when the distance metric value is less than a preset threshold, generating a target video scene based on the current scene synthesis vector includes:

[0099] Constructing a scene semantic gradient map, calculating information entropy based on normalized eigenvalues ​​of scene elements in the scene semantic gradient map, obtaining importance distribution of scene elements, dividing the scene into a key area and a transition area based on the information entropy and the importance distribution, extracting a first scene feature vector corresponding to the key area and a second scene feature vector corresponding to the transition area, and calculating the semantic association strength between the key area and the transition area based on the Euclidean distance between the first scene feature vector and the second scene feature vector;

[0100] Extracting the scene synthesis vector and reference samples from the video sample library, performing a cosine distance metric on the key region, performing a Mahalanobis distance metric based on a feature covariance matrix on the transition region, weighting the cosine distance metric values ​​of the key region and the Mahalanobis distance metric values ​​of the transition region by a preset region weight coefficient, weighting the semantic association strength in combination with a pre-acquired semantic association coefficient, and combining the weighted values ​​to obtain the distance metric value;

[0101] Calculate the distance gradient of the distance metric value, superimpose the product of the distance gradient and the update step size on the original area boundary to obtain an updated area boundary, and determine whether the distance metric value is less than a preset threshold based on the updated area boundary. If so, generate the target video scene based on the scene synthesis vector.

[0102] The feature distribution of each pixel in the original scene is analyzed, and the scene image is filtered using a convolution kernel to extract the edge features of scene elements. For a pixel (x, y), its gradient intensity is obtained by taking the square root of the sum of the squares of the horizontal and vertical gradients. The gradient direction is determined by the inverse tangent of the horizontal and vertical gradients. For example, for a scene containing people, buildings, and natural environments, a 5×5 Gaussian convolution kernel can be used to obtain the edge gradient distribution of each element, thereby constructing a complete scene semantic gradient map.

[0103] Information entropy and importance distribution are calculated based on the scene semantic gradient map. The eigenvalues ​​of each scene element are normalized so that all eigenvalues ​​are distributed within the interval [0, 1]. The scene information entropy is calculated based on the normalized eigenvalues. Assuming there are n elements in a scene, and the normalized eigenvalue of each element i is pi, the scene information entropy is calculated by multiplying the logarithmic product of the normalized eigenvalues ​​of each element. In practice, for a video frame containing eight major scene elements, the normalized eigenvalues ​​are [0.15, 0.22, 0.08, 0.12, 0.18, 0.09, 0.06, 0.10], resulting in a calculated information entropy of approximately 2.87. Information entropy analysis determines the importance distribution of scene elements. Elements with higher importance values ​​occupy a more prominent position in visual perception.

[0104] The scene is divided into key regions and transition regions based on information entropy and importance distribution. An importance threshold of 0.15 is set. Regions containing features above this threshold are labeled key regions, such as faces and key action areas. Regions below this threshold are labeled transition regions, such as background and out-of-focus objects. In practice, a 1920×1080 resolution video frame might have a central area of ​​800×600 pixels designated as the key region, with the remainder as the transition region.

[0105] Extract the first scene feature vector for the key region and the second scene feature vector for the transition region. Use a pre-trained deep neural network model to extract features for each of these regions. High-resolution processing is used for the key region, extracting a feature vector with a dimension of 512. Standard-resolution processing is used for the transition region, extracting a feature vector with a dimension of 256. For example, the first five dimensions of the feature vector for the key region might be [0.78, 0.45, 0.23, 0.67, 0.39], while the first five dimensions of the feature vector for the transition region might be [0.32, 0.56, 0.41, 0.29, 0.52].

[0106] The semantic association strength between the key region and the transition region is calculated based on the Euclidean distance between the first and second scene feature vectors. The Euclidean distance is calculated by taking the square root of the sum of the squared differences between the elements at corresponding positions in the two feature vectors. To handle feature vectors of different dimensions, the vectors with the shorter dimension are extended to the longer dimension by zero padding. In actual calculations, the Euclidean distance value obtained for the feature vectors of the two regions may be 8.74. This distance value is then mapped to the interval [0, 1], resulting in a semantic association strength of 0.64.

[0107] Different distance metrics are applied to key regions and transition regions. The scene synthesis vectors are extracted from the video sample library and compared with reference samples. The cosine distance metric is applied to the key regions, calculating the cosine of the angle between the feature vectors. For example, the cosine distance between the feature vectors of the key regions and the reference sample might be 0.13. The Mahalanobis distance metric based on the feature covariance matrix is ​​applied to the transition regions, taking into account the correlation between features. A feature covariance matrix of 256×256 is constructed, and the calculated Mahalanobis distance value might be 3.28.

[0108] A weighted combination is performed using regional weight coefficients. The preset key region weight coefficient is 0.7, and the transition region weight coefficient is 0.3. The distance metrics of the two regions are weighted, and the previously acquired semantic association coefficient is 0.5. The semantic association strength of 0.64 is weighted to 0.32. The weighted key region distance metric, transition region distance metric, and semantic association strength are combined to obtain a final distance metric of 1.46.

[0109] Calculate the distance gradient of the distance metric and update the region boundary. During the iteration, calculate the difference between the current distance metric and the distance metric from the previous iteration to obtain the distance gradient. Assuming the distance metric from the previous iteration was 1.52 and the current one is 1.46, the distance gradient is -0.06. Set the update step size to 5 pixels and superimpose the distance gradient and the update step size -0.3 pixels onto the original region boundary. For example, if the original key region boundary is the rectangle [400, 300, 1200, 900], the updated boundary will be [400.3, 300.3, 1199.7, 899.7].

[0110] Based on the updated region boundaries, the distance metric is determined to be less than a preset threshold of 1.5. The current distance metric, 1.46, is less than the threshold, so the target video scene is generated based on the scene synthesis vector. During the generation process, a deep generative model is used, using the scene synthesis vector as a conditional input to generate the target video scene with a resolution of 1920×1080 and a frame rate of 30fps. The scene content meets the original semantic requirements and has high-quality visual effects.

[0111] In this embodiment, by constructing a scene semantic gradient map and dividing the scene based on information entropy, refined processing and accurate matching of the video scene are achieved. By dividing the scene into key areas and transition areas, and adopting different distance measurement methods for different areas, the core semantic information and secondary transition information in the scene can be captured more accurately, and the semantic consistency between the key areas and the transition areas is fully considered to ensure that the generated video scene maintains the unity of the overall style in visual performance and can accurately express the semantic transition between different areas. Through the calculation of distance gradients and dynamic updating of area boundaries, the area division can be adaptively adjusted, so that the final generated video scene is more in line with the semantic structure of the original script.

[0112] In an optional embodiment,

[0113] Based on the pre-built feature pyramid and the target video scene, the spatiotemporal features are extracted at each layer in combination with the inter-layer correlation relationship in the multi-layer scene graph to obtain a feature graph sequence. Adaptive feature fusion is performed on the feature graph sequence to obtain a scene feature representation and calculate an inter-frame similarity matrix. A self-attention mechanism is applied to the similarity matrix to obtain a scene key frame index and segment the video content. A transition effect based on the depth map is inserted between adjacent segments to obtain a combined video sequence including:

[0114] Based on the target video scene, a feature map of each layer in a pre-constructed feature pyramid is obtained. In combination with the inter-layer correlation relationship in the multi-layer scene graph, spatiotemporal features are extracted from the feature map by multi-scale convolution. Convolution operation is performed on the feature map corresponding to each layer using convolution kernel parameters and a bias term is superimposed to obtain the feature map sequence.

[0115] Performing a matrix product operation on the query matrix of the current layer of the feature pyramid and the key matrix of the adjacent layer, normalizing the calculation result and multiplying it with the value matrix of the adjacent layer to obtain feature association data, and fusing the feature association data with the feature graph sequence to obtain a feature sequence with enhanced association;

[0116] performing average pooling and multi-layer perceptron processing on the association-enhanced feature sequence to obtain channel attention weights, performing weighted summation of the channel attention weights and the association-enhanced feature sequence to obtain the scene feature representation, calculating the inter-frame similarity matrix based on the scene feature representation, and applying a self-attention mechanism to extract the scene key frame index;

[0117] The video content is segmented according to the scene key frame index to obtain video segments, a transition effect is generated based on a time-varying interpolation coefficient calculated by a cosine function and a depth map of adjacent video segments, and the video segments and the transition effect are combined to obtain the combined video sequence.

[0118] Figure 3 It is a schematic diagram of the correlation between feature pyramid layers and adaptive feature fusion, as shown in Figure 3 As shown in the figure, the target video scene and the pre-built feature pyramid are obtained. The feature pyramid contains multiple scale levels. For example, it can be set to a three-layer structure, corresponding to high, medium, and low resolution features. In practical applications, for a video scene with a resolution of 1920×1080, the first layer can maintain the original resolution, the second layer can be downsampled to 960×540, and the third layer can be further downsampled to 480×270.

[0119] For the feature extraction process, multi-scale convolution operations are used to extract spatiotemporal features for each layer of the feature pyramid. Taking the first layer as an example, convolution kernels with sizes of 3×3, 5×5, and 7×7 are used for feature extraction, and the convolution kernel parameters are set to {W 11 , W 12 , W 13}, the corresponding bias term is {b 11 , b 12 , b 13 For the input feature map F1, the output feature F'1=W is obtained through convolution operation 11 *F1+W 12 *F1+W 13 *F1+b 11+b 12 +b 13 , where "*" represents a convolution operation. Similarly, similar operations are performed on the second and third layers, yielding F'2 and F'3, respectively. In practical implementations, the number of convolution kernels can be set to 64, the number of feature channels to 128, and the activation function to ReLU.

[0120] Combined with the inter-layer correlation relationship in the multi-layer scene graph to enhance feature representation, the query matrix of the current layer of the feature pyramid is associated with the key matrix of the adjacent layer. Taking the second layer as an example, the query matrix Q2 is generated from F'2, and the key matrices K1, K3 and value matrices V1, V3 are generated from the first layer F'1 and the third layer F'3 respectively. The matrix product of Q2 and K1 is calculated to obtain the attention score S 12 , S 12 Divide by 8 (the square root of the feature dimension) for normalization, and then multiply by V1 to obtain the feature association data A 12 . Similarly, we can calculate A 23 . 12 With A 23 Weighted fusion into F'2, the associated enhanced feature sequence E2=F'2+0.5×A is obtained 12 +0.5×A 23 The same approach is used for other levels.

[0121] To achieve adaptive feature fusion, a channel attention mechanism is implemented on the feature sequence of the associated enhancement. Taking E2 as an example, global average pooling is performed on E2 to obtain the feature vector G2, which is then processed through a two-layer fully connected network: G'2=W 22 (ReLU(W 21 (G2))), where W 21 and W 22 α2 is the fully connected layer parameter. G'2 is normalized to the range of 0-1 using the Sigmoid function to obtain the channel attention weight. Multiplying α2 by E2 along the channel dimension yields the weighted feature E"2. Similarly, the weighted features at all levels are combined according to the weight coefficients: FS = 0.5 × E"1 + 0.3 × E"2 + 0.2 × E"3 to obtain the scene feature representation.

[0122] Calculate the inter-frame similarity matrix based on scene feature representation. For any two frames i and j in the video, extract the corresponding feature vectors FS_i and FS_j, and calculate the cosine similarity: Sim(i, j) = (FS_i·FS_j) / (||FS_i|| × ||FS_j||), to obtain the inter-frame similarity matrix M. For example, for a video clip containing 100 frames, a 100×100 similarity matrix is ​​generated, where M[50,51] = 0.92, indicating that frames 50 and 51 have high similarity.

[0123] A self-attention mechanism is applied to the similarity matrix to extract keyframes. The average similarity between each frame and all other frames is calculated. Frames with significant changes in similarity are selected as scene transition points. The first-order derivative of the similarity is calculated. When the absolute value of the derivative exceeds a preset threshold (e.g., 0.15), the frame is marked as a keyframe. For example, in a 30-second video, the system might mark scene transitions at 3, 12, and 25 seconds, thereby dividing the video into four segments.

[0124] Generate a depth map-based transition effect between adjacent video segments. Extract depth maps D1 and D2 from the end and start frames of adjacent segments, and normalize the depth value range to 0-1. Based on the time-varying interpolation coefficient λ(t)=0.5-0.5×cos(πt) (t varies from 0 to 1), calculate the depth map of the transition frame: D(t)=(1-λ(t))×D1+λ(t)×D2. Use the depth map to control pixel blending to produce a natural transition effect. In practical applications, for transitions between two scenes, the transition duration can be set to 1 second (corresponding to 30 frames), and the depth blending weight is calculated frame by frame and applied to the original pixels.

[0125] The video segments and the generated transition effects are combined in time order to obtain a combined video sequence with coherent transition effects.

[0126] In this embodiment, high-quality video scene segmentation and natural and smooth transition effects are achieved through the feature fusion mechanism of multi-scale feature extraction and association enhancement. By combining the inter-layer association relationship of the feature pyramid and the multi-layer scene graph, the spatiotemporal characteristics of the video scene can be captured at different scales, and the detailed information and overall structure of the scene can be retained. The query-key-value matrix operation mechanism between the feature pyramid layers enhances the information exchange between features of different scales, effectively solving the information island problem caused by independent processing of features of different scales in traditional methods. Based on the enhanced scene feature representation, the inter-frame similarity can be accurately calculated and the self-attention mechanism can be used to intelligently identify key frames, thereby achieving natural segmentation of the video content and significantly improving the visual quality and narrative fluency of the generated video.

[0127] In an optional embodiment,

[0128] Generating a transition effect based on a time-varying interpolation coefficient calculated based on a cosine function and a depth map of adjacent video segments, and combining the video segments with the transition effect to obtain the combined video sequence includes:

[0129] For the transition period between adjacent video segments, the time-varying interpolation coefficient is calculated based on the cosine function using the ratio of the current time point to the transition duration;

[0130] Collecting scene information of the adjacent video segments, performing position encoding on the spatial position and viewing direction of any point in the current scene using a multi-layer perceptron to obtain encoding features, and constructing a neural radiation field representation including spatial density features and color features based on the encoding features;

[0131] Obtaining depth maps of the adjacent video segments, calculating a depth consistency loss between a spatial density integral of a camera ray and the depth map, optimizing the neural radiance field representation based on the depth consistency loss, applying the time-varying interpolation coefficients to the optimized neural radiance field representation for weighted fusion, performing volume rendering on the fused features, and generating an intermediate view sequence by calculating an integral of cumulative transmittance and spatial density and color features;

[0132] The gradient difference of the depth map is calculated to obtain a depth consistency weight, the depth consistency weight is applied to the intermediate view sequence and the depth map weighted by the time-varying interpolation coefficient to perform adaptive fusion to obtain a transition effect, and the adjacent video segments and the transition effect are combined in time sequence to obtain the combined video sequence.

[0133] To generate transition effects between adjacent video segments, every time point within the transition period is processed. Assuming the transition duration is T seconds and the current time point is t seconds, the system calculates the time-varying interpolation coefficient α, where the α value is calculated based on the cosine function. Specifically, α = 0.5*(1-cos(π*t / T)). For example, when t = 0, α = 0; when t = T / 2, α = 0.5; and when t = T, α = 1. This smoothly changing interpolation coefficient ensures a gradual effect during the transition process and avoids visual discomfort caused by sudden changes.

[0134] To obtain scene information from adjacent video segments, image data and camera parameters are collected from these segments. For any 3D point P (x, y, z) and viewing direction D (θ, φ) in the scene, a multilayer perceptron (MLP) is used to perform position encoding, generating a high-dimensional feature vector. The MLP consists of four fully connected layers, each with 256 neurons, and uses the Reinforced Luminance (ReLU) activation function. The input layer receives a 6-dimensional vector of spatial position (x, y, z) and viewing direction (θ, φ), which is expanded into 96-dimensional features through position encoding. The output layer generates a neural radiance field representation consisting of spatial density features and RGB color features. For example, for a point P (1.5, 2.3, 0.8) and viewing direction D (0.7, 1.2) in the scene, after position encoding and MLP processing, the density feature σ = 0.85 and the color feature c = (0.7, 0.6, 0.5) are obtained.

[0135] The depth maps of adjacent video segments are obtained through existing depth estimation methods. In order to optimize the neural radiance field representation, the depth consistency loss between the spatial density integral of the camera ray and the depth map is calculated. Assuming that the depth value of a point in the depth map is d_gt=5.2 meters, and the expected depth value calculated by the neural radiance field is d_pred=5.5 meters, the depth consistency loss can be expressed as |d_gt-d_pred|=0.3 meters. The Adam optimizer is used in the training process, with an initial learning rate of 0.001, and 2000 iterations of optimization are performed. The learning rate is reduced to 0.5 times the original every 500 iterations until the depth consistency loss converges to below 0.05 meters. The optimized neural radiance field can more accurately represent the geometric structure of the scene.

[0136] For the generation of intermediate views during the transition process, the time-varying interpolation coefficient α is applied to the neural radiance field representations of two adjacent video segments for weighted fusion. For example, for the same point in space, if the density feature σ1=0.8 and the color feature c1=(0.7, 0.6, 0.5) in the first video segment, and the density feature σ2=0.6 and the color feature c2=(0.5, 0.4, 0.6) in the second video segment, when α=0.3, the fused density feature σ=0.3*0.6+(1-0.3)*0.8=0.74, and the color feature c=0.3*(0.5, 0.4, 0.6)+(1-0.3)*(0.7, 0.6, 0.5)=(0.64, 0.54, 0.53).

[0137] Using volume rendering, 128 points are sampled along camera rays, and the integrated cumulative transmittance, spatial density, and color features are calculated to generate intermediate views. For a video frame with a resolution of 1920×1080, the sampling interval for each ray is 0.05 meters, the near plane distance is 0.1 meters, and the far plane distance is 10 meters. The intermediate view sequence generated by volume rendering is smooth during transitions, but may contain artifacts caused by depth inconsistency.

[0138] To address depth inconsistency, the depth map gradient difference is calculated to obtain a depth consistency weight. The depth gradient is calculated using the Sobel operator. For example, if the depth values ​​of adjacent pixels are 5.2 meters and 5.3 meters, the gradient value is 0.1 meters. For the depth maps of two video segments, a large gradient difference at corresponding locations indicates depth inconsistency in that area. The system sets a threshold of 0.2 meters. When the gradient difference is greater than the threshold, the depth consistency weight w is set to 0.8; otherwise, it is set to 0.2.

[0139] The depth consistency weight w is applied to the intermediate view sequence and the depth map weighted by the time-varying interpolation coefficient for adaptive fusion to obtain the transition effect. For example, for pixel (800, 600), if the color value of the intermediate view is (0.64, 0.54, 0.53), the depth consistency weight w = 0.8, the time-varying interpolation coefficient α = 0.3, the color value of the first video segment is (0.7, 0.6, 0.5), and the color value of the second video segment is (0.5, 0.4, 0.6), then the fused color value is 0.8*(0.64, 0.54, 0.53)+(1-0.8)*(0.3*(0.5, 0.4, 0.6)+(1-0.3)*(0.7, 0.6, 0.5))=(0.628, 0.532, 0.534).

[0140] After the transition effects are generated, the adjacent video segments and transition effects are combined in time sequence to obtain the final combined video sequence. For example, if there are two video segments with a duration of 5 seconds each, and the transition duration is 1 second, the final combined video sequence will consist of the first 4.5 seconds of the first video segment, the 1-second transition effect, and the last 4.5 seconds of the second video segment, for a total length of 10 seconds.

[0141] In this embodiment, high-quality, natural and smooth video transition effects are achieved through neural radiation field technology and depth consistency optimization. The time-varying interpolation coefficient calculated based on the cosine function provides a smooth time transition characteristic, avoiding the mechanical feeling that may be brought about by traditional linear interpolation, and making the transition process present a more natural acceleration-deceleration rhythm. The scene representation method based on neural radiation field breaks through the limitations of traditional pixel-based or optical flow-based transition technology, and can accurately model the geometric structure and appearance characteristics of the scene in three-dimensional space. By performing time-varying weighted fusion on the optimized neural radiation field representation, a smooth transition between different perspectives and scenes is achieved, avoiding the geometric distortion and flickering artifacts common in traditional transition effects.

[0142] Figure 4 A performance comparison diagram of video transitions based on neural radiance fields shows that the neural radiance field method achieved a visual quality score of 8.7, significantly higher than both optical flow transitions (6.3) and traditional fades (4.5). The neural radiance field method performed particularly well in terms of geometric consistency, achieving a score of 9.0, demonstrating that the technology can more accurately preserve the 3D geometry of the scene, avoiding the geometric distortion common in traditional methods.

[0143] Temporal smoothness is the metric where the gap between the methods is smallest, but the Neural Radiance Field method still leads with a score of 8.5, while the Optical Flow Field and traditional cross-fading methods score 7.1 and 7.3, respectively. This indicates that all methods perform well in temporal coherence, but the Neural Radiance Field's cosine function time-varying interpolation provides a more natural transition effect.

[0144] The user experience score reflects the audience's subjective feelings. The neural radiation field method achieved the highest score of 9.1, far exceeding the baseline level (5.0 points), indicating that this transition technology can significantly improve the viewing experience.

[0145] Overall, the transition technology based on neural radiation field has achieved significant breakthroughs in visual quality, geometric consistency and user experience through advanced algorithms such as deep consistency optimization and adaptive fusion, providing a new technical path for the production of high-quality video content.

[0146] In an optional embodiment,

[0147] Performing style transfer on the combined video sequence, and outputting a finished video including:

[0148] Acquire the combined video sequence, and extract the spatiotemporal features and feature graph sequence corresponding to the combined video sequence;

[0149] Mapping and matching the spatiotemporal features and feature map sequence corresponding to the combined video sequence with reference to the feature space of the target style template to obtain fused features after style transfer;

[0150] The fused features after style transfer are reconstructed and decoded to obtain a video product with unified style.

[0151] Obtain a combined video sequence. A combined video sequence can be composed of multiple video clips shot in different scenes, with different cameras, or at different times. For example, consider a user with three videos: the first is a 5-second clip of a person walking outdoors in sunlight, with a resolution of 1920×1080; the second is a close-up of a person shot indoors under lighting, with a duration of 3 seconds and a resolution of 1280×720; and the third is a 4-second shot of a landscape shot at dusk, with a resolution of 3840×2160. These three videos differ significantly in tone, lighting, and atmosphere, requiring a unified style.

[0152] After obtaining the combined video sequence, a deep convolutional neural network is used to analyze the video sequence to extract spatiotemporal features and feature map sequences. Spatiotemporal features refer to feature representations in the temporal dimension between video frames and the spatial dimension within each frame, while feature map sequences are a collection of multi-level visual features extracted for each frame in the video. A pre-trained video feature extraction network is used, which contains multiple convolutional layers, pooling layers, and non-linear activation functions. For each input frame, low-level features such as edges and textures are extracted through shallow convolutions, mid-level features such as object parts are extracted through mid-level convolutions, and high-level features at the semantic level are extracted through deep convolutions. The extraction of spatiotemporal features is achieved through 3D convolutions or a combination of 2D convolutions and long short-term memory networks to capture the temporal correlation between video frames.

[0153] Three videos were processed separately. For the first outdoor video, the extracted features showed high brightness, high contrast, and warm tones; for the second indoor video, the extracted features showed medium brightness, low contrast, and cool tones; and for the third evening video, the extracted features showed low brightness, medium contrast, and a purple-red hue. The extracted features were encoded into multidimensional feature vectors and feature maps. The dimensionality depends on the neural network architecture used, typically ranging from 512 to 2048.

[0154] Select a target style template as a reference. The target style template can be a user-specified video clip, an image, or a predefined style descriptor. For example, suppose the user selects a movie clip as the target style template. The clip has a typical film color style, characterized by medium brightness, high contrast, and warm tones without loss of detail. The target style template is analyzed using the aforementioned feature extraction network to obtain its feature space representation.

[0155] In the mapping and matching phase, a style transfer algorithm is used to map the spatiotemporal features and feature map sequence of the combined video sequence into the feature space of the target style template. While preserving the original video content, its visual style is adjusted to match the target template. Adaptive instance normalization is used to achieve style transfer by adjusting the mean and variance of the feature map. For each video clip's feature map, its mean μ and standard deviation σ are calculated. The original feature map is then normalized and rescaled based on the mean μ_target and standard deviation σ_target of the target style feature map to obtain the style-transferred feature map.

[0156] In order to maintain the temporal coherence of the video sequence, temporal consistency constraints are applied to the features between adjacent frames, the differences in features between adjacent frames are calculated, and a temporal smoothing term is introduced to make the style transfer process smoother in the temporal dimension and avoid flickering or incoherence.

[0157] For example, the high-brightness features of the first outdoor video are adjusted to medium brightness, but the high contrast is maintained; the cool tones of the second indoor video are adjusted to warm tones, while the contrast is improved; the low brightness of the third evening video is appropriately increased, and the purple-red tones are adjusted to warm tones that are more in line with the film style.

[0158] After feature map matching, the style-transferred fused features are obtained. These fused features incorporate both the content information of the original video and the stylistic features of the target style template. A generative network consisting of multiple deconvolutional layers, upsampling layers, and residual connections is used to input these fused features. Through layer-by-layer decoding and upsampling, video frames with the target style are reconstructed. To ensure reconstruction quality, a multi-scale feature fusion mechanism is introduced during the decoding process, combining features at different levels to preserve more detailed information.

[0159] After processing, the three originally distinct videos were unified into a cinematic sequence with a medium-brightness, high-contrast, and warm tonal palette, while maintaining the integrity of the original videos' content and narrative. The final output video had a uniform resolution of 1920×1080, a total length of 12 seconds, a frame rate of 30 fps, and was encoded in H.264 format, resulting in a file size of approximately 35 MB.

[0160] In this embodiment, through style transfer and fusion technology, the style of the finished video is unified, which significantly improves the artistic expression and visual consistency of the video. By extracting spatiotemporal features and multi-level feature maps of the combined video sequence, the structural information and temporal consistency of the video content are retained, providing a complete feature basis for subsequent style transfer. The mapping and matching method based on the target style template is used as a reference, so that the style transfer process can not only accurately capture the feature distribution of the target style, but also maintain the content integrity of the original video, avoiding the common content distortion and style imbalance problems in traditional style transfer methods. While ensuring the style effect, it can also maintain high-quality temporal consistency, avoiding problems such as style flickering and temporal instability, and greatly improving the overall quality and artistic expression of the finished video.

[0161] A second aspect of an embodiment of the present invention provides a video plot generation and scene synthesis system based on natural language processing, comprising:

[0162] The first unit is configured to receive a script text input by a user and extract scene semantic features, and construct the scene semantic features into a multi-layer scene graph;

[0163] The second unit is configured to extract a keyword sequence based on the multi-layer scene graph, calculate the semantic relevance of the keyword sequence to obtain a feature vector matrix, perform semantic feature decomposition and visual feature reconstruction on the feature vector matrix to obtain a scene description vector and a scene structure vector, alternately update feature representation according to the scene description vector and the scene structure vector to obtain a scene synthesis vector, calculate the similarity between the scene synthesis vector and a reference sample in a video sample library to obtain a distance metric value, and when the distance metric value is less than a preset threshold, generate a target video scene based on the current scene synthesis vector;

[0164] The third unit is configured to extract spatiotemporal features at each layer based on a pre-constructed feature pyramid and the target video scene, in combination with the inter-layer correlation relationship in the multi-layer scene graph, to obtain a feature graph sequence, perform adaptive feature fusion on the feature graph sequence, obtain a scene feature representation, and calculate an inter-frame similarity matrix, apply a self-attention mechanism to the similarity matrix to obtain a scene key frame index and segment the video content, insert a transition effect based on the depth map between adjacent segments, and obtain a combined video sequence;

[0165] The fourth unit is configured to perform style migration on the combined video sequence and output a finished video.

[0166] According to a third aspect of an embodiment of the present invention, an electronic device is provided, including:

[0167] A processor and a memory for storing processor-executable instructions, wherein the processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0168] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0169] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video plot generation and scene synthesis method based on natural language processing, characterized in that: include: Receive script text input by the user and extract scene semantic features, and construct the scene semantic features into a multi-layer scene graph; Extracting a keyword sequence based on the multi-layer scene graph, calculating the semantic relevance of the keyword sequence to obtain a feature vector matrix, performing semantic feature decomposition and visual feature reconstruction on the feature vector matrix to obtain a scene description vector and a scene structure vector, alternately updating feature representations based on the scene description vector and the scene structure vector to obtain a scene synthesis vector, calculating the similarity between the scene synthesis vector and a reference sample in a video sample library to obtain a distance metric value, and when the distance metric value is less than a preset threshold, generating a target video scene based on the current scene synthesis vector; Based on a pre-constructed feature pyramid and the target video scene, spatiotemporal features are extracted at each layer in combination with the inter-layer correlation relationship in the multi-layer scene graph to obtain a feature graph sequence, adaptive feature fusion is performed on the feature graph sequence to obtain a scene feature representation and calculate an inter-frame similarity matrix, a self-attention mechanism is applied to the similarity matrix to obtain a scene key frame index and segment the video content, and a transition effect based on the depth map is inserted between adjacent segments to obtain a combined video sequence; Perform style transfer on the combined video sequence and output a finished video.

2. The method according to claim 1, characterized in that Receiving the script text input by the user and extracting scene semantic features, and constructing the scene semantic features into a multi-layer scene graph includes: Receive a script text input by a user, perform semantic segmentation on the script text to obtain a word sequence, and encode the word sequence based on a pre-trained language model to obtain initial semantic features; Inputting the initial semantic features into a multi-head attention network, computing feature representations of different attention heads in parallel, and performing hierarchical clustering on the feature representations to obtain scene semantic features, wherein each attention head focuses on semantic information of character interactions, scene attributes, and plot development respectively; A multi-layer scene graph is constructed based on the scene semantic features, and the neighborhood relationship of the nodes in the graph is calculated using the graph attention network. Nodes with similar semantic features are dynamically aggregated into a character relationship layer, a scene attribute layer, and a plot development layer, and the correlation relationship between the layers is determined.

3. The method according to claim 1, characterized in that Extracting a keyword sequence based on the multi-layer scene graph, calculating the semantic relevance of the keyword sequence to obtain a feature vector matrix, performing semantic feature decomposition and visual feature reconstruction on the feature vector matrix to obtain a scene description vector and a scene structure vector, and alternately updating feature representation according to the scene description vector and the scene structure vector to obtain a scene synthesis vector includes: Extract node features from the multi-layer scene graph, calculate the information entropy of each level node and its adjacent nodes to obtain node weights, perform hierarchical clustering based on the node weights, and select cluster centers to form a keyword sequence; Calculating semantic relevance for word pairs in the keyword sequence to obtain a relevance matrix, wherein the semantic relevance is obtained by multiplying the cosine distance of the word vectors by the edge weights between the nodes in the graph, and subjecting the relevance matrix to weight mapping and nonlinear transformation to obtain a feature vector matrix; Performing semantic feature decomposition on the feature vector matrix to obtain a scene description vector, and performing visual feature reconstruction to obtain a scene structure vector; The scene description vector and the scene structure vector are alternately iteratively optimized to obtain candidate feature representations, a joint optimization objective of semantic consistency loss, visual reconstruction loss, and temporal continuity loss is calculated based on the candidate feature representations, the candidate feature representations are updated by minimizing the joint optimization objective until convergence, and the converged feature representation is determined as the scene synthesis vector.

4. The method according to claim 1, wherein Calculating the similarity between the scene synthesis vector and a reference sample in a video sample library to obtain a distance metric value, and when the distance metric value is less than a preset threshold, generating a target video scene based on the current scene synthesis vector includes: Constructing a scene semantic gradient map, calculating information entropy based on normalized eigenvalues ​​of scene elements in the scene semantic gradient map, obtaining importance distribution of scene elements, dividing the scene into a key area and a transition area based on the information entropy and the importance distribution, extracting a first scene feature vector corresponding to the key area and a second scene feature vector corresponding to the transition area, and calculating the semantic association strength between the key area and the transition area based on the Euclidean distance between the first scene feature vector and the second scene feature vector; Extracting the scene synthesis vector and reference samples from the video sample library, performing a cosine distance metric on the key region, performing a Mahalanobis distance metric based on a feature covariance matrix on the transition region, weighting the cosine distance metric values ​​of the key region and the Mahalanobis distance metric values ​​of the transition region by a preset region weight coefficient, weighting the semantic association strength in combination with a pre-acquired semantic association coefficient, and combining the weighted values ​​to obtain the distance metric value; Calculate the distance gradient of the distance metric value, superimpose the product of the distance gradient and the update step size on the original area boundary to obtain an updated area boundary, and determine whether the distance metric value is less than a preset threshold based on the updated area boundary. If so, generate the target video scene based on the scene synthesis vector.

5. The method according to claim 1, wherein Based on the pre-built feature pyramid and the target video scene, the spatiotemporal features are extracted at each layer in combination with the inter-layer correlation relationship in the multi-layer scene graph to obtain a feature graph sequence. Adaptive feature fusion is performed on the feature graph sequence to obtain a scene feature representation and calculate an inter-frame similarity matrix. A self-attention mechanism is applied to the similarity matrix to obtain a scene key frame index and segment the video content. A transition effect based on the depth map is inserted between adjacent segments to obtain a combined video sequence including: Based on the target video scene, a feature map of each layer in a pre-constructed feature pyramid is obtained. In combination with the inter-layer correlation relationship in the multi-layer scene graph, spatiotemporal features are extracted from the feature map by multi-scale convolution. Convolution operation is performed on the feature map corresponding to each layer using convolution kernel parameters and a bias term is superimposed to obtain the feature map sequence. Performing a matrix product operation on the query matrix of the current layer of the feature pyramid and the key matrix of the adjacent layer, normalizing the calculation result and multiplying it with the value matrix of the adjacent layer to obtain feature association data, and fusing the feature association data with the feature graph sequence to obtain a feature sequence with enhanced association; performing average pooling and multi-layer perceptron processing on the association-enhanced feature sequence to obtain channel attention weights, performing weighted summation of the channel attention weights and the association-enhanced feature sequence to obtain the scene feature representation, calculating the inter-frame similarity matrix based on the scene feature representation, and applying a self-attention mechanism to extract the scene key frame index; The video content is segmented according to the scene key frame index to obtain video segments, a transition effect is generated based on a time-varying interpolation coefficient calculated by a cosine function and a depth map of adjacent video segments, and the video segments and the transition effect are combined to obtain the combined video sequence.

6. The method according to claim 5, characterized in that Generating a transition effect based on a time-varying interpolation coefficient calculated by a cosine function and a depth map of adjacent video segments, and combining the video segments with the transition effect to obtain the combined video sequence includes: For the transition period between adjacent video segments, the time-varying interpolation coefficient is calculated based on the cosine function using the ratio of the current time point to the transition duration; Collecting scene information of the adjacent video segments, performing position encoding on the spatial position and viewing direction of any point in the current scene using a multi-layer perceptron to obtain encoding features, and constructing a neural radiation field representation including spatial density features and color features based on the encoding features; Obtaining depth maps of the adjacent video segments, calculating a depth consistency loss between a spatial density integral of a camera ray and the depth map, optimizing the neural radiance field representation based on the depth consistency loss, applying the time-varying interpolation coefficients to the optimized neural radiance field representation for weighted fusion, performing volume rendering on the fused features, and generating an intermediate view sequence by calculating an integral of cumulative transmittance and spatial density and color features; The gradient difference of the depth map is calculated to obtain a depth consistency weight, the depth consistency weight is applied to the intermediate view sequence and the depth map weighted by the time-varying interpolation coefficient to perform adaptive fusion to obtain a transition effect, and the adjacent video segments and the transition effect are combined in time sequence to obtain the combined video sequence.

7. The method according to claim 1, characterized in that Performing style transfer on the combined video sequence, and outputting a finished video including: Acquire the combined video sequence, and extract the spatiotemporal features and feature graph sequence corresponding to the combined video sequence; Mapping and matching the spatiotemporal features and feature map sequence corresponding to the combined video sequence with reference to the feature space of the target style template to obtain fused features after style transfer; The fused features after style transfer are reconstructed and decoded to obtain a video product with unified style.

8. A video plot generation and scene synthesis system based on natural language processing, used to implement the method according to any one of claims 1 to 7, characterized in that: include: The first unit is configured to receive a script text input by a user and extract scene semantic features, and construct the scene semantic features into a multi-layer scene graph; The second unit is configured to extract a keyword sequence based on the multi-layer scene graph, calculate the semantic relevance of the keyword sequence to obtain a feature vector matrix, perform semantic feature decomposition and visual feature reconstruction on the feature vector matrix to obtain a scene description vector and a scene structure vector, alternately update feature representation according to the scene description vector and the scene structure vector to obtain a scene synthesis vector, calculate the similarity between the scene synthesis vector and a reference sample in a video sample library to obtain a distance metric value, and when the distance metric value is less than a preset threshold, generate a target video scene based on the current scene synthesis vector; The third unit is configured to extract spatiotemporal features at each layer based on a pre-constructed feature pyramid and the target video scene, in combination with the inter-layer correlation relationship in the multi-layer scene graph, to obtain a feature graph sequence, perform adaptive feature fusion on the feature graph sequence, obtain a scene feature representation, and calculate an inter-frame similarity matrix, apply a self-attention mechanism to the similarity matrix to obtain a scene key frame index and segment the video content, insert a transition effect based on the depth map between adjacent segments, and obtain a combined video sequence; The fourth unit is configured to perform style migration on the combined video sequence and output a finished video.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images