A text-driven episodic video screen composition method and system

Abstract

The application discloses a kind of based on text driving's episode video picture synthesis method and system, it is related to video processing and natural language understanding technical field, the present application is by parsing the dependency syntax structure of script text, constructs the hierarchical feature representation with entity as center, and combines timestamp information accurately captures the attribute and behavior dynamic change of entity in the evolution process of script, to realize the implicit expression of semantic consistency and timing logic of multi-entity space layout in video generation stage;Meanwhile, based on the importance density of semantic change intensity and area calculation, adaptively filter key local area and carry out super-resolution enhancement, effectively improve the visual definition of detail area in generated video, avoid the calculation redundancy brought by global uniform enhancement;In addition, through attention mechanism, real-time semantic segmentation is realized in generation process, significantly improve the boundary naturalness of video picture and overall visual quality.

Description

Technical Field

[0001] This invention relates to the fields of video processing and natural language understanding, and in particular to a text-driven method and system for synthesizing video frames from TV series. Background Technology

[0002] Text-to-video generation is a hot research topic at the intersection of computer vision and natural language processing, aiming to automatically generate dynamic video content that semantically matches natural language descriptions. In recent years, with the rapid development of deep learning technology, especially the widespread application of Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and Diffusion Models, video generation technology has achieved significant breakthroughs. Early methods were mainly based on retrieval and template concatenation, combining segments similar to the text from existing video libraries. However, the generated content was limited by the library capacity and lacked flexibility. Subsequently, generative model-based methods gradually became mainstream, such as using Conditional Generative Adversarial Networks (GANs) to generate video frames based on text features, or using temporal generative networks to model frame sequences. However, existing technologies still face many challenges when processing complex text descriptions, especially the insufficient ability to fine-tune the modeling of entities, attributes, behaviors, and their dynamic changes in the text. This results in the generated videos failing to meet the semantic consistency, detail richness, and visual quality of local regions for practical applications.

[0003] Existing text-to-video generation methods typically encode the entire text into a global feature vector, using this as input to the generative network. This approach ignores the independence between entities within the text and the evolution of the attributes and behaviors of the same entity at different points in time. This leads to chaotic spatial layouts of multiple objects in the generated video, discrepancies between object attributes and text descriptions, and difficulty in capturing the temporal dynamics inherent in script-like texts. For example, existing methods often fail to capture the attribute change in a single video frame when describing a girl dancing in a red dress, and then changing into a blue dress to continue dancing, let alone focus on depicting the changed areas (such as the dress).

[0004] Furthermore, different semantic regions have varying importance during video generation. Existing methods typically employ a global generation strategy with uniform resolution, lacking adaptive enhancement mechanisms for key local regions (such as facial expressions and detailed textures of objects). This results in videos with acceptable overall clarity but blurry details, failing to meet the high-quality requirements of film and television production, virtual reality, and other fields. For semantic region segmentation and localization, existing technologies largely rely on post-processing segmentation models, decoupling them from the generation process. This limits segmentation accuracy to the quality of the generated image and prevents the use of intermediate features from the generation process for joint optimization. Summary of the Invention

[0005] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section, as well as in the abstract and title of the present application, to avoid obscuring the purpose of this section, the abstract and title of the invention. Such simplifications or omissions shall not be used to limit the scope of the present invention.

[0006] In view of the aforementioned existing problems, the present invention is proposed.

[0007] Therefore, the technical problem solved by this invention is: how to accurately extract entity-centric dynamic semantic information from script text, and accordingly perform adaptive enhancement of key local areas in the generated video to improve the semantic consistency and visual detail quality of the video.

[0008] To address the aforementioned technical problems, this invention provides the following technical solution: a text-driven method for compositing video frames of a TV series, comprising: parsing the script text, extracting hierarchical features centered on entities through dependency parsing, obtaining the attribute and behavior representations of each entity at each time point, and aggregating them into a comprehensive feature vector; inputting the comprehensive feature vector of each entity into a video synthesis network to generate an initial video frame containing multiple entities, and using an attention mechanism to segment the visual semantic region corresponding to each entity from the initial video frame; calculating the importance weight of the visual semantic region based on the intensity of attribute and behavior changes of each entity in the time series, selecting visual semantic regions higher than a preset importance threshold as key local regions, and performing super-resolution enhancement; re-embedding the enhanced key local regions into the corresponding positions of the upsampled initial video frame, and outputting the final high-resolution video frame of the TV series after boundary fusion.

[0009] As a preferred embodiment of the present invention, the extraction of entity-centric hierarchical features includes: inputting the script text into a pre-trained dependency parser, outputting a dependency tree, extracting all noun phrases from the dependency tree as entity layer candidates, extracting words modifying the noun phrases as attribute layer candidates, and extracting verbs with subject-predicate or verb-object relationships with the noun phrases as behavior layer candidates; the representation of the attributes and behaviors of each entity at each time point includes: based on the dependency arcs in the dependency tree, for each entity layer candidate, collecting all attribute layer candidates that directly modify the entity as an attribute list, collecting all behavior layer candidates with subject-predicate or verb-object relationships with the entity as a behavior list; generating a tuple of entity-attribute list-behavior list for the entity in the current sentence, and recording the position of the entity in the text as a timestamp; if the entity has no attributes or no behaviors in the sentence, the corresponding list is empty.

[0010] In a preferred embodiment of the present invention, the aggregation into a comprehensive feature vector includes: inputting each tuple into a referential resolution model, identifying different tuples pointing to the same entity across sentences, and merging the entity identifiers in these tuples into a single identifier to form a tuple list for each entity, wherein the tuples in the list are sorted by timestamp; for each entity, traversing all tuples, and for each tuple, averaging the attribute feature vectors in the attribute list to obtain the attribute representation of the tuple; similarly, averaging the behavior feature vectors in the behavior list to obtain the behavior representation; if the attribute list of a tuple is empty, the attribute representation is set to a zero vector; if the behavior list is empty, the behavior representation is set to a zero vector; averaging the attribute representations of all tuples to obtain the comprehensive attribute vector of the entity; averaging the behavior representations of all tuples to obtain the comprehensive behavior vector of the entity; concatenating the comprehensive attribute vector and the comprehensive behavior vector to obtain the comprehensive feature vector of the entity.

[0011] In a preferred embodiment of the present invention, the generation of the initial video frame includes: using the comprehensive feature vector of each entity as the semantic representation of the entity in the global scene; inputting the comprehensive feature vector of each entity into an independent conditional batch normalization layer to generate affine transformation parameters for the corresponding entity; fusing the affine transformation parameters of all entities as the overall modulation parameters; injecting the overall modulation parameters into a video synthesis network; and the video synthesis network, based on the overall modulation parameters, decoding layer by layer starting from a random noise tensor to output an initial video frame containing multiple objects with a resolution of H×W, where H is the height of the initial video frame and W is the width of the initial video frame.

[0012] As a preferred embodiment of the present invention, the step of segmenting the visual semantic region corresponding to each entity from the initial video frame includes: during the decoding process, calculating the attention response using the comprehensive feature vector of each entity and the intermediate layer feature map, and simultaneously introducing a learnable background vector to calculate the background attention response; performing softmax normalization on the responses of all entities and the background to obtain a probability map of each pixel belonging to each entity or background, wherein the probability map of each entity is the attention heatmap of the entity, and the probability map of the background is denoted as the background heatmap; for each pixel, taking the category with the highest probability as the label of the pixel; if the label is background, then ignoring it; if the label is an entity, then marking the pixel as the mask pixel of the entity to obtain the binary segmentation mask of each entity; for each entity, if the mask is not empty, then: calculating the number of pixels of the entity mask as the actual area of ​​the region; calculating the minimum bounding rectangle of the mask, and recording the coordinates of the upper left corner of the minimum bounding rectangle. and width and height The image block within the minimum bounding rectangle is cropped from the initial video frame as a local image block of the entity; the corresponding mask sub-image is cropped from the binary mask of the entity as a local mask of the entity.

[0013] In a preferred embodiment of the present invention, the calculation of the importance weight of the visual semantic region includes: for each entity, obtaining a corresponding time-sorted list of tuples; for each tuple, taking the attribute representation as the attribute feature at the corresponding time point; similarly taking the behavior representation as the behavior feature at the corresponding time point; calculating the Euclidean distance between the attribute features of adjacent time points to obtain a series of attribute change values, and taking the average of the attribute change values ​​as the attribute change intensity of the entity; similarly calculating the Euclidean distance between the behavior features of adjacent time points, and taking the average as the behavior change intensity; averaging the attribute change intensity and the behavior change intensity to obtain the comprehensive semantic change intensity value of the entity; if the entity has only one time point, the change intensity is set to 0; dividing the comprehensive semantic change intensity value of each entity by the actual area of ​​the corresponding visual semantic region image patch to obtain the importance density value of the corresponding region, and normalizing the importance density value as the importance weight of the visual semantic region.

[0014] In a preferred embodiment of the present invention, the importance weight of each visual semantic region image block is compared with a preset weight threshold, and the region image block with the importance weight greater than the weight threshold is selected as the key local region; the key local region is input into a pre-trained super-resolution reconstruction network, and a high-resolution key local region image block is output; non-key regions are not super-resolved.

[0015] As a preferred embodiment of the present invention, the output of the high-resolution final episode video image includes: upsampling the initial video frame by a factor of s to obtain a high-resolution base frame; upsampling the local mask of each key local region by a factor of s to obtain an enlarged local mask; and multiplying the original minimum bounding rectangle coordinates by a factor of s to obtain the target minimum bounding rectangle in the high-resolution base frame.

[0016] As a preferred embodiment of the present invention, the output of the high-resolution final episode video frame further includes: for each key local region, placing the high-resolution image block at the target minimum bounding rectangle position of the high-resolution base frame, and performing blending using the magnified local mask: for each pixel within the target minimum bounding rectangle position, if the corresponding magnified local mask value is 1, then the pixel value is weighted and fused with the base frame pixel at the corresponding position; if the mask value is 0, then the base frame pixel is retained unchanged; during fusion, to smooth the boundary, the distance field of the magnified local mask is calculated, and a fusion weight map is generated accordingly; the fused pixels are updated to the corresponding positions of the high-resolution base frame; and the high-resolution complete video frame after the weighted fusion processing is output as the final episode video frame.

[0017] On the other hand, the present invention also provides a text-driven video composition system for TV series, including: a script feature extraction module, which parses the script text, extracts hierarchical features centered on entities through dependency parsing, obtains the attribute and behavior representations of each entity at each time point, and aggregates them into a comprehensive feature vector; The video generation and segmentation module inputs the comprehensive feature vector of each entity into the video synthesis network to generate an initial video frame containing multiple entities, and uses an attention mechanism to segment the visual semantic region corresponding to each entity from the initial video frame. The importance super-resolution module calculates the importance weight of visual semantic regions based on the intensity of attribute and behavior changes of each entity in the time series, selects visual semantic regions with an importance threshold higher than the preset threshold as key local regions, and performs super-resolution enhancement on them. The embedded fusion module re-embeds the enhanced key local areas into the corresponding positions of the upsampled initial video frames, and outputs high-resolution final episode video images after boundary fusion.

[0018] The beneficial effects of this invention are as follows: By parsing the dependency syntax structure of the script text, this invention constructs a hierarchical feature representation centered on entities, and combines timestamp information to accurately capture the dynamic changes in the attributes and behaviors of entities during the script's evolution. This achieves semantic consistency in the spatial layout of multiple entities and implicit expression of temporal logic during the video generation stage. Simultaneously, based on the semantic change intensity and the importance density of the region area calculation, key local regions are adaptively selected and super-resolution enhancement is performed, effectively improving the visual clarity of detailed regions in the generated video and avoiding computational redundancy caused by global uniform enhancement. Furthermore, real-time semantic segmentation is achieved during the generation process through an attention mechanism, and a distance-field-based weighted fusion strategy is used to seamlessly embed the enhanced regions into the background, significantly improving the naturalness of the video's boundaries and the overall visual quality.

[0019] This invention can significantly improve the expressiveness of key details while ensuring the overall composition of the generated video is reasonable, thus meeting the needs of film and television production, virtual reality and other fields for the generation of high-definition video content. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating a text-driven video composition method for TV series, as shown in this invention.

[0021] Figure 2 This is a structural diagram of a text-driven video compositing system for TV series, as shown in this invention. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0023] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort should fall within the scope of protection of this invention.

[0024] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0025] According to an embodiment of the present invention, in combination Figure 1 The flowchart shown illustrates a text-driven method for compositing video frames from TV series, comprising: S1: Parse the script text, extract hierarchical features centered on entities through dependency parsing, obtain the attribute and behavior representations of each entity at each time point, and aggregate them into a comprehensive feature vector.

[0026] S1.1: Input the script text into the pre-trained dependency parser, output the dependency relation tree, extract all noun phrases from the dependency relation tree as entity layer candidates, extract words that modify the noun phrases as attribute layer candidates, and extract verbs that have a subject-predicate or verb-object relationship with the noun phrases as behavior layer candidates.

[0027] The dependency parser employs a deep model based on graph neural networks (such as the Biaffine parser) to output a dependency tree of the text. The dependency tree uses words as nodes and dependency arcs as edges, annotating the grammatical relationships between words. Based on this tree structure, the following extraction operations are performed: Traverse all nodes and identify all noun phrases (NPs) as candidates for the entity layer.

[0028] For each entity layer candidate, noun phrases can be identified through rules such as phrase structure analysis or compound noun modification in dependency relations. For example, nodes with noun parts of speech and core words and their modifiers can be combined into noun phrases.

[0029] For each attribute-level candidate, find the modification relation pointing to the entity in the dependency arc of each entity-level candidate, including adjective modification, noun modification, qualifier modification, etc., and extract the corresponding modifier as attribute-level candidate, which is usually an adjective or a noun.

[0030] For each behavior-level candidate, find the relationship in the dependency arc where the entity is the subject or object, and extract the corresponding predicate verb as the behavior-level candidate. If the entity is the object, extract its corresponding verb; if it is the subject, extract the verb in the same way.

[0031] The above operations yield a candidate list of entities, attributes, and behaviors for each sentence. This process ensures accurate mapping of grammatical relations, laying the foundation for subsequent structured representation.

[0032] S1.2: Generate entity-attribute-behavior tuples with timestamps.

[0033] Based on the dependency arcs in the dependency tree obtained in S1.1, for each entity layer candidate, perform the following pairing operation: Collect all candidate attributes that directly modify the entity to form an attribute list. Each of them For the corresponding words, For the attribute list The number of attributes in the middle.

[0034] Collect all candidate behaviors that have a subject-verb or verb-object relationship with the entity to form a behavior list. Each of them For the corresponding verb, For the list of behaviors The number of rows.

[0035] Generate a triple for this entity in the current sentence. ,in, This is the entity's own data, and it records the entity's position in the text as a timestamp, specifically including sentence index and word order index (e.g., the position of the entity's core words). If the entity has no attributes or behaviors in the current sentence, the corresponding list is empty. .

[0036] By iterating through all sentences, a set of triples with timestamps is obtained. Each triple represents the semantic role of an entity at a specific point in time.

[0037] S1.3: Input each tuple into the referential resolution model, identify different tuples pointing to the same entity across sentences, and merge the entity identifiers in these tuples into the same identifier to form a list of tuples for each entity. The tuples in the list are sorted by timestamp.

[0038] In this embodiment of the invention, the referential resolution model adopts a Transformer-based architecture (such as SpanBERT), which can identify different expressions in text that refer to the same entity. The specific operation steps include: For each triple, extract its entity-level words and their context, input them into the reference resolution model, and output the reference chain of each entity (i.e. all mentions that point to the same entity). Merge the entity identifiers in triples belonging to the same referential chain into a single unique identifier, and form a list of tuples for each entity. , of which each A triplet corresponding to a single point in time; The list of tuples for each entity is sorted in ascending order by timestamp (sentence index and word order) to ensure chronological order.

[0039] By resolving referential inconsistencies, the unification of entities across sentences is achieved, enabling the association of the attributes and behaviors of the same entity in different sentences.

[0040] It should be noted that referential resolution solves the problem that the same entity may appear in the script with different names or pronouns, ensuring the integrity and consistency of entity features and avoiding the splitting of the same entity into multiple independent objects in subsequent processing.

[0041] S1.4: For each entity, iterate through all tuples, and for each tuple, extract the attribute list. Each word in the tuple is converted into a feature vector. The conversion uses a pre-trained word embedding model (such as GloVe or BERT) to obtain the vector representation of each word. Then, the average of all vectors in the attribute list is taken to obtain the attribute representation of the tuple. Similarly, for the list of behaviors... The average of each verb in the expression is used to obtain the behavior representation. If the attribute list is empty, then (Zero vector); if the list of rows is empty, then .

[0042] Furthermore, for all tuples of this entity, the attribute representation and behavior representation are averaged separately to obtain the comprehensive attribute vector: Combined behavioral vector: in, The number of different time points in the text where an entity appears is concatenated to obtain the entity's comprehensive feature vector. .

[0043] At the same time, the original attributes of each tuple are represented and behavioral expression It is stored in association with a timestamp for subsequent calculation of the intensity of changes in attributes and behaviors.

[0044] Furthermore, the comprehensive feature vector of each entity is... The unique identifier is stored as a structured feature pair, along with a complete list of tuples for that entity.

[0045] It should be noted that during feature vectorization, if a contextual model such as BERT is used, the contextual representation of the corresponding words is extracted to enhance semantic richness. The introduction of zero vectors ensures mathematical consistency in the case of an empty list.

[0046] S2: Input the comprehensive feature vector of each entity into the video synthesis network to generate an initial video frame containing multiple entities, and use the attention mechanism to segment the visual semantic region corresponding to each entity from the initial video frame.

[0047] S2.1: Use the comprehensive feature vector of each entity as the semantic representation of the entity in the global scene.

[0048] The comprehensive feature vector of each entity Each independent Conditional Batch Normalization (CBN) layer is input to generate affine transformation parameters for the corresponding entity. The affine transformation parameters of all entities are then fused together to form the overall modulation parameters.

[0049] The calculation formula for the conditional batch normalization layer is as follows: in, and These represent the scaling factor and translation factor, respectively, with dimensions consistent with the number of channels in the feature map to be modulated in the video synthesis network. MLP stands for Multilayer Perceptron, used to map entity features to an affine parameter space.

[0050] Furthermore, the affine transformation parameters of all entities are fused to form the overall modulation parameters. The fusion method can be element-wise addition or channel-wise concatenation, depending on the network architecture design. In a preferred embodiment, element-wise addition is used: The overall modulation parameters are then injected into all residual modules of the video synthesis network, replacing the parameters of the standard batch normalization layer, thereby achieving semantic control over the generation process.

[0051] The video synthesis network employs an architecture based on Generative Adversarial Networks (GANs) or Diffusion Models, such as StyleGAN or Stable Diffusion. The network starts with a random noise tensor and decodes it layer by layer through multiple upsampled residual blocks. In each residual block, the input feature map first passes through a convolutional layer, then is modulated through a conditional batch normalization layer, where... and The feature map undergoes an affine transformation, followed by a non-linear activation function. After multiple levels of decoding, the network outputs a resolution of [resolution value missing]. initial video frame This frame is a static image, and its content is a visual representation of the attributes and behaviors of all entities in the script at different points in time. That is, the information of the time dimension is implied through a single picture, which provides a basis for subsequent local enhancement based on the intensity of time changes.

[0052] It should be noted that this invention decouples the comprehensive features of each entity into independent modulation parameters through a conditional batch normalization layer, and then fuses them into overall parameters. This ensures that each entity contributes independently to the generation process, while avoiding the complex coupling of multiple conditional inputs, enabling the network to generate complex scenes containing multiple coordinating objects.

[0053] The initial video frame is a single static image, which integrates the entity states at all points in time in the text description. It is a visual presentation of the aggregated attributes and behaviors of each entity at different points in time, thus implying time-dimensional information in the static image and providing a basis for subsequent local enhancement based on the intensity of time changes.

[0054] S2.2: During the decoding process, the attention response is calculated using the comprehensive feature vector of each entity and the intermediate layer feature map. At the same time, a learnable background vector is introduced to calculate the background attention response. The responses of all entities and the background are normalized by softmax to obtain the probability map of each pixel belonging to each entity or background. The probability map of each entity is the attention heatmap of that entity, and the probability map of the background is denoted as the background heatmap.

[0055] Specifically, feature maps from a specific layer in the decoder (e.g., before the last upsampling layer) are selected. ,in, The feature map space size, For each entity identifier, this represents the number of channels. Calculate the eigenvector Similarity to the feature vectors at each spatial location in the feature map: in, For position The feature vectors are used, and the dot product represents the cosine similarity. Simultaneously, a learnable background vector is introduced. Calculate the background attention response The formula is the same as above.

[0056] Stacking all entity responses and background responses in a spatial dimension yields the response tensor. ,in The value represents the number of entities, with 1 added to represent the background. Then, softmax normalization is applied to each spatial location to obtain the probability map: in, Indicates that a pixel belongs to an entity. The probability, These probability maps, which are called attention heatmaps, represent the degree to which each entity belongs to the generated image pixels.

[0057] It should be noted that by introducing a background vector, each pixel can be classified as not only as an entity but also as background, thus avoiding the forced assignment of non-entity regions to the entity with the highest probability, thereby improving the accuracy of segmentation.

[0058] S2.3: For each pixel, take the category (entity or background) with the highest probability as the label of the pixel; if the label is background, ignore it; if the label is an entity, mark the pixel as the mask pixel of the entity, and obtain the binary segmentation mask (global size) of each entity.

[0059] Specifically, based on the obtained probability map, a hard label is assigned to each pixel. For each spatial location... The category with the highest probability is taken as the label for that pixel. If the label is a background (i.e.) If the label is an entity, then the pixel is ignored; if the label is an entity, then the pixel is marked as an entity. The mask pixels. This yields the global binary segmentation mask for each entity. Pixels with a value of 1 belong to the entity.

[0060] For each entity, if the mask is not empty (i.e., there is at least one pixel), then perform the following: (1) Count the number of pixels with a value of 1 in the mask, denoted as . , which represents the actual area.

[0061] (2) Traverse all pixels with a value of 1 in the mask and obtain the minimum row coordinate. Maximum row coordinate Minimum column coordinates Maximum column coordinate The coordinates of the top-left corner of the bounding box are Width and height are: , This yields the smallest bounding rectangle (bounding box).

[0062] (3) From the initial video frame The bounding box region is cropped out to obtain a local image patch of the entity. .

[0063] (4) From the global mask Cut out the same bounding box region from the middle to obtain the local mask. It is consistent with the size of the local image patch.

[0064] Each entity ultimately corresponds to a unique set of local image patches, local masks, and bounding box coordinates. and actual area .

[0065] As can be seen, pixel-level hard label assignment achieves a precise correspondence between entities and image regions, with each entity generating only a unique visual region, avoiding the problems of overlapping or splitting multiple regions. Furthermore, recording the actual area rather than the bounding box area provides a precise pixel-level measurement for subsequent importance density calculations.

[0066] S3: Based on the intensity of attribute and behavior change of each entity in the time series, calculate the importance weight of the visual semantic region, select the visual semantic region with an importance higher than the preset threshold as the key local region, and perform super-resolution enhancement.

[0067] S3.1: Calculation of the intensity of comprehensive semantic change based on temporal variation.

[0068] For each entity, obtain the corresponding list of tuples sorted by time. For each tuple, the attribute representation is taken as the attribute feature at the corresponding time point; similarly, the behavior representation is taken as the behavior feature at the corresponding time point.

[0069] Furthermore, to quantify the dynamic changes of entities during the script's evolution, the following calculations are performed: (1) Calculate the Euclidean distance between attribute features at adjacent time points. Iterate through all adjacent tuples to obtain a set of attribute change values, and take the arithmetic mean of these values ​​as the attribute change intensity of the entity. , Similarly, calculate the Euclidean distance between behavioral characteristics at adjacent time points. The average value is taken as the intensity of behavioral change. .

[0070] The overall semantic change intensity value of the entity is obtained by averaging the attribute change intensity and the behavior change intensity (or by weighting as needed; this embodiment uses an equal-weighted average). If an entity has only one point in time, then there are no adjacent changes. This indicates that the entity has not undergone semantic changes in the script.

[0071] Through the above calculations, each entity obtains a scalar value. This reflects the overall dynamism of attributes and behaviors. The larger the value, the more drastic the changes of the entity during the script's progression, and the more detail may need to be preserved in the visual presentation.

[0072] S3.2: Calculate the intensity value of the comprehensive semantic change for each entity. The actual area of ​​the corresponding visual semantic region image patch Divide to obtain the importance density value of the corresponding region. Normalize the importance density value (e.g., divide by the maximum density of all regions) and use it as the importance weight of the visual semantic region.

[0073] The calculation formula is as follows: in, This is a very small positive number used to avoid division by zero errors (when the area of ​​the region is zero, the entity does not actually exist, but S2.3 has guaranteed that only non-empty masks are output, so this is mainly for numerical stability considerations). This density value reflects the intensity of semantic change of the entity per unit pixel; that is, the stronger the change and the smaller the area of ​​the entity, the higher its density, indicating that the local area may contain richer details.

[0074] Furthermore, the normalization method employs maximum-minimum normalization, mapping the density values ​​to... Interval. In a preferred embodiment, the interval is determined by dividing by the maximum density value among all entities: in, This is the maximum value among all entity density values. If the density of all entities is zero (e.g., the intensity of change of all entities is zero), then all weights are set to zero. This yields the importance weight of the visual semantic region for each entity. .

[0075] S3.3: The importance weight of each image block in the visual semantic region is compared with a preset weight threshold. (This can be adjusted according to the actual application scenario; in this embodiment, it is set to 0.5) A comparison is made, if... , Then the local image patch corresponding to the entity and its associated information (local mask) The bounding box coordinates and actual area are marked as critical local regions; otherwise, they are considered non-critical regions and will not be enhanced in subsequent steps.

[0076] The selected key local regions form a set, which will serve as input for super-resolution enhancement.

[0077] S3.4: Input the key local region into the pre-trained super-resolution reconstruction network and output a high-resolution key local region image patch; the magnification factor of the super-resolution reconstruction network is a fixed value s.

[0078] The super-resolution reconstruction network adopts a deep learning-based architecture, such as Enhanced Super-Resolution Generative Adversarial Network (ESRGAN) or Deep Residual Channel Attention Network (RCAN), with a fixed magnification factor of s (e.g., 2 or 4). The network recovers high-frequency details in the image by learning a mapping from low resolution to high resolution.

[0079] For input low-resolution image patches Super-resolution networks output high-resolution image patches. .

[0080] In a preferred embodiment, the network employs subpixel convolutional layers for upsampling, supplemented by perceptual loss and adversarial loss, to generate visually realistic textures.

[0081] Non-critical areas are left unprocessed, maintaining their original resolution.

[0082] It should be noted that using a super-resolution network with a fixed multiplier avoids the implementation complexity caused by dynamic multipliers, and the pre-trained network can be used directly without retraining for each region.

[0083] S4: The enhanced key local regions are re-embedded into the corresponding positions of the upsampled initial video frames, and after boundary fusion, the high-resolution final episode video is output.

[0084] S4.1: Set the initial video frames The overall upsampling is performed by a factor of 's' to match the resolution of key local regions after super-resolution enhancement. First, the initial video frames are upsampled by a factor of 's' (the same as the upsampling factor in the super-resolution network in S3.4). The upsampling method employs bicubic interpolation or subpixel convolution to obtain high-quality, high-resolution base frames. .

[0085] This base frame serves as the background canvas for subsequent embedding operations, preserving the overall structure of the original generated image and the original details of the unenhanced areas.

[0086] S4.2: Upsample the local mask of each key local region by a factor of s to obtain the magnified local mask, and multiply the original minimum bounding rectangle coordinates by a factor of s to obtain the target minimum bounding rectangle in the high-resolution base frame; the target minimum bounding rectangle is the region to be embedded.

[0087] Specifically, the local mask Upsampled by a factor of s, the magnified local mask is obtained. The upsampling method uses nearest-neighbor interpolation to maintain the binary properties of the mask and avoid introducing intermediate values ​​that could lead to blurred boundaries.

[0088] Multiplying the original bounding box coordinates by a factor of 's' yields the target bounding box in the high-resolution base frame: top-left corner coordinates. Width and height .

[0089] Through the above processing, high-resolution image patches of each key local region are obtained. (size ), corresponding precise mask and target location Everything is ready.

[0090] S4.3: To naturally integrate high-resolution image patches into the background and avoid harsh boundaries or visual abruptness caused by direct overlay, a distance-field-based weighted fusion strategy is adopted. The specific steps are as follows: For each critical local region, its magnified local mask is used. As input, calculate the distance from each pixel to the mask boundary. The distance is defined as the Euclidean distance from the pixel to the nearest mask boundary (i.e., the boundary between values ​​of 1 and 0). For pixels inside the mask, the distance is positive, indicating how far away from the boundary; for pixels outside the mask, the distance is negative (or set to 0, but since fusion only applies to internal pixels, only the distances of internal pixels need to be calculated). In a preferred embodiment, a fast traversal algorithm or a distance transformation algorithm is used to calculate the distance field. Each element Represents pixels Distance to the mask boundary.

[0091] Generate fusion weight map based on distance field This is used to control the blending ratio of high-resolution image patches to the base frame. The design principle of the weights is: fully utilize the high-resolution image patches within the region (away from the boundary); near the boundary, gradually transition to the base frame to eliminate seams. Specifically, the weight function is defined as follows: in, The preset transition band width (e.g., 5 pixels). When hour, This indicates that high-resolution image patches are used entirely; otherwise, the weights decrease linearly to 0 (at the boundaries). hour This ensures a smooth transition at the boundary.

[0092] For each pixel position within the target bounding box ,in, , , ,like If the pixel belongs to the current entity, then weighted blending is performed: like 0, then (This remains unchanged because the pixel does not belong to the current entity region; it may belong to the background or other entities and should be preserved.) By traversing all key local regions and updating them sequentially, the final fused high-resolution base frame is obtained. .

[0093] It should be noted that multiple key local regions may overlap (e.g., regions of different entities are adjacent or partially overlap in space). During sequential processing, the region processed later will cover the region processed earlier, but due to mask constraints, each pixel belongs to only one entity (guaranteed by the segmentation in S2.3), thus preventing conflicts. If entities in the actual scene are occluded, overlapping parts should be processed in depth order; however, the spatial layout is implicitly included in the initial frame generation of this scheme, and the segmentation masks are mutually exclusive, so the order does not affect the result.

[0094] Finally, the high-resolution base frame after fusing all key local regions will be... This is the final video output for the series. The image retains the overall composition of the initial video frames while enhancing the detail and clarity of key local areas, resulting in smoother boundary transitions and improved visual effects.

[0095] As can be seen, this invention ensures that the resolution of the background and non-critical areas is consistent with that of the enhanced area by upsampling the initial frame as a whole, thus avoiding the resolution mismatch problem. The weighted fusion based on the distance field achieves boundary smoothing, eliminates splicing marks, and makes the enhanced area seamlessly connected with the surrounding environment. Moreover, the linear transition strategy is simple and efficient, and the transition band width is adjustable to adapt to different scene requirements.

[0096] The final output high-resolution video has higher clarity in key details, significantly improving the overall visual quality and meeting the high requirements of film and television production for image detail.

[0097] The present invention also provides a text-driven video composition system for TV series, including: a script feature extraction module, which parses the script text, extracts hierarchical features centered on entities through dependency parsing, obtains the attribute and behavior representations of each entity at each time point, and aggregates them into a comprehensive feature vector; The video generation and segmentation module inputs the comprehensive feature vector of each entity into the video synthesis network to generate an initial video frame containing multiple entities, and uses an attention mechanism to segment the visual semantic region corresponding to each entity from the initial video frame. The importance super-resolution module calculates the importance weight of visual semantic regions based on the intensity of attribute and behavior changes of each entity in the time series, selects visual semantic regions with an importance threshold higher than the preset threshold as key local regions, and performs super-resolution enhancement on them. The embedded fusion module re-embeds the enhanced key local areas into the corresponding positions of the upsampled initial video frames, and outputs high-resolution final episode video images after boundary fusion.

[0098] The system also includes one or more processors and memory.

[0099] The memory is used to store operable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, including the flow of a text-driven video frame compositing method for TV series according to the foregoing embodiments, particularly... Figure 1 The flowchart of the method is shown.

[0100] Other aspects disclosed in the embodiments of the present invention also propose a computer-readable medium for storing software including instructions executable by one or more computers, which, upon execution, cause the one or more computers to perform operations including the flow of a text-driven video frame synthesis method for TV series according to the foregoing embodiments, particularly... Figure 1 The flowchart of the method is shown.

[0101] It should be recognized that embodiments of the present invention may be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium.

[0102] The method can be implemented using standard programming techniques, including a non-transitory computer-readable storage medium configured with a computer program in the computer program, wherein the storage medium is configured such that the computer operates in a specific and predefined manner.

[0103] Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system; however, if required, the program can be implemented in assembly or machine language.

[0104] In any case, the language can be either compiled or interpreted.

[0105] Furthermore, for this purpose, the program can run on programmed application-specific integrated circuits.

[0106] The processes described herein (or variations and / or combinations thereof) can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program includes a plurality of instructions executable by one or more processors.

[0107] Furthermore, the method can be implemented in any suitable computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices.

[0108] Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether portable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein.

[0109] Furthermore, machine-readable code, or parts thereof, can be transmitted via wired or wireless networks.

[0110] When such media includes instructions or programs that combine with a microprocessor or other data processor to implement the steps described above, the invention described herein includes these and other different types of non-transitory computer-readable storage media.

[0111] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A text-driven method for compositing video frames from TV series, characterized in that: include: The script text is analyzed, and hierarchical features centered on entities are extracted through dependency parsing. The attributes and behaviors of each entity at each time point are obtained and aggregated into a comprehensive feature vector. The comprehensive feature vector of each entity is input into the video synthesis network to generate an initial video frame containing multiple entities, and the visual semantic region corresponding to each entity is segmented from the initial video frame using an attention mechanism. Based on the intensity of attribute and behavior change of each entity in the time series, the importance weight of the visual semantic region is calculated, and the visual semantic regions with an importance threshold higher than the preset threshold are selected as key local regions and super-resolution enhancement is performed. The enhanced key local regions are re-embedded to their corresponding positions in the upsampled initial video frames, and after boundary fusion, a high-resolution final episode video is output.

2. The text-driven video frame synthesis method for TV series as described in claim 1, characterized in that: The extraction of entity-centric hierarchical features includes: The script text is input into a pre-trained dependency parser, which outputs a dependency tree. All noun phrases are extracted from the dependency tree as entity layer candidates, words that modify the noun phrases are extracted as attribute layer candidates, and verbs that have a subject-predicate or verb-object relationship with the noun phrases are extracted as behavior layer candidates. The attributes and behaviors of each entity at each point in time include: Based on the dependency arcs in the dependency tree, for each entity layer candidate, all attribute layer candidates that directly modify the entity are collected as an attribute list, and all behavior layer candidates that have a subject-predicate or verb-object relationship with the entity are collected as a behavior list. Generate a tuple of entity-attribute list-behavior list for the entity in the current sentence, and record the position of the entity in the text as a timestamp; If an entity has no attributes or behaviors in a sentence, the corresponding list will be empty.

3. The text-driven video frame synthesis method for TV series as described in claim 2, characterized in that: The aggregation into a comprehensive feature vector includes: Each tuple is input into the referential resolution model to identify different tuples pointing to the same entity across sentences, and the entity identifiers in these tuples are merged into the same identifier to form a list of tuples for each entity, with the tuples in the list sorted by timestamp. For each entity, iterate through all tuples. For each tuple, average the attribute feature vectors in the attribute list to obtain the attribute representation of the tuple. Similarly, average the behavior feature vectors in the behavior list to obtain the behavior representation. If the attribute list of a tuple is empty, the attribute representation is set to a zero vector; if the behavior list is empty, the behavior representation is set to a zero vector. The comprehensive attribute vector of the entity is obtained by averaging the attribute representations of all tuples. The average of the behavioral representations of all tuples is used to obtain the comprehensive behavioral vector of the entity. The comprehensive attribute vector and the comprehensive behavior vector are concatenated to obtain the comprehensive feature vector of the entity.

4. The text-driven video frame synthesis method for TV series as described in claim 3, characterized in that: The generation of the initial video frame includes: The comprehensive feature vector of each entity is used as the semantic representation of the entity in the global scene; The comprehensive feature vector of each entity is input into an independent conditional batch normalization layer to generate the affine transformation parameters of the corresponding entity. The affine transformation parameters of all entities are then fused together as the overall modulation parameters. The overall modulation parameters are injected into the video synthesis network. Based on the overall modulation parameters, the video synthesis network decodes layer by layer, starting from the random noise tensor, and outputs an initial video frame containing multiple objects with a resolution of H×W, where H is the height of the initial video frame and W is the width of the initial video frame.

5. The text-driven video frame compositing method for TV series as described in claim 4, characterized in that: The step of segmenting the visual semantic region corresponding to each entity from the initial video frame includes: During the decoding process, the attention response is calculated using the comprehensive feature vector of each entity and the intermediate layer feature map. At the same time, a learnable background vector is introduced to calculate the background attention response. The responses of all entities and the background are normalized using softmax to obtain the probability map of each pixel belonging to each entity or background. The probability map of each entity is called the attention heatmap of that entity, and the probability map of the background is called the background heatmap. For each pixel, the category with the highest probability is taken as the label of the pixel; if the label is background, it is ignored; if the label is an entity, the pixel is marked as the mask pixel of the entity, and a binary segmentation mask for each entity is obtained. For each entity, if the mask is not empty, then: Calculate the number of pixels in the entity mask as the actual area; Calculate the minimum bounding rectangle of the mask and record the coordinates of its top-left corner. and width and height ; The image block within the minimum bounding rectangle is cropped from the initial video frame and used as a local image block of the entity; The corresponding mask sub-image is cropped from the binary mask of the entity and used as the local mask of the entity.

6. The text-driven video frame compositing method for TV series as described in claim 5, characterized in that: The importance weights of the computational visual semantic regions include: For each entity, obtain the corresponding list of tuples sorted by time; For each tuple, the attribute representation is taken as the attribute feature at the corresponding time point; Similarly, the behavioral representation is taken as the behavioral feature at the corresponding time point; Calculate the Euclidean distance between attribute features at adjacent time points to obtain a series of attribute change values, and take the average value of the attribute change values ​​as the attribute change intensity of the entity. Similarly, the Euclidean distance between behavioral characteristics at adjacent time points is calculated, and the average value is taken as the intensity of behavioral change. The average of the attribute change intensity and the behavior change intensity is used to obtain the comprehensive semantic change intensity value of the entity. If an entity has only one point in time, then the intensity of change is set to 0; The importance density value of the corresponding region is obtained by dividing the comprehensive semantic change intensity value of each entity by the actual area of ​​the corresponding visual semantic region image block. The importance density value is then normalized and used as the importance weight of the visual semantic region.

7. The text-driven video frame synthesis method for TV series as described in claim 6, characterized in that: The importance weight of each visual semantic region image block is compared with a preset weight threshold, and the region image block with the importance weight greater than the weight threshold is selected as the key local region; The key local regions are input into a pre-trained super-resolution reconstruction network, which outputs high-resolution key local region image patches; non-key regions are not super-reconstructed.

8. The text-driven video frame synthesis method for TV series as described in claim 7, characterized in that: The output high-resolution final episode video footage includes: The initial video frames are upsampled by a factor of s to obtain a high-resolution base frame; The local mask of each key local region is upsampled by a factor of s to obtain the magnified local mask. The original minimum bounding rectangle coordinates are also multiplied by a factor of s to obtain the minimum bounding rectangle of the target in the high-resolution base frame.

9. The text-driven video frame synthesis method for TV series as described in claim 8, characterized in that: The output of the high-resolution final episode video also includes: For each key local region, a high-resolution image patch is placed at the position of the minimum bounding rectangle of the target in the high-resolution base frame, and then blended using a magnified local mask: For each pixel within the minimum bounding rectangle of the target, if the corresponding local mask value after magnification is 1, then the pixel value is weighted and fused with the base frame pixel at the corresponding position; if the mask value is 0, then the base frame pixel is left unchanged. During fusion, to smooth the boundaries, the distance field of the magnified local mask is calculated, and a fusion weight map is generated accordingly. Update the merged pixels to the corresponding positions in the high-resolution base frame; The high-resolution complete video frames after the weighted fusion process are output as the final episode video images.

10. A text-driven video frame synthesis system for TV dramas, based on the text-driven video frame synthesis method for TV dramas according to any one of claims 1 to 9, characterized in that: Also includes: The script feature extraction module parses the script text and extracts hierarchical features centered on entities through dependency parsing. It obtains the attribute and behavior representations of each entity at each time point and aggregates them into a comprehensive feature vector. The video generation and segmentation module inputs the comprehensive feature vector of each entity into the video synthesis network to generate an initial video frame containing multiple entities, and uses an attention mechanism to segment the visual semantic region corresponding to each entity from the initial video frame. The importance super-resolution module calculates the importance weight of visual semantic regions based on the intensity of attribute and behavior changes of each entity in the time series, selects visual semantic regions with an importance threshold higher than the preset threshold as key local regions, and performs super-resolution enhancement on them. The embedded fusion module re-embeds the enhanced key local areas into the corresponding positions of the upsampled initial video frames, and outputs high-resolution final episode video images after boundary fusion.

Images