Intelligent synthesis method of film and television scene based on generative multi-modal script semantic mapping

Abstract

The application discloses a kind of based on generative multimodal script semantic mapping film and television scene intelligent synthesis method.The method is first to film script text is carried out semantic unit segmentation and dependency structure analysis, and establishes script semantic mapping relationship;Adopt generative multimodal semantic alignment method and carry out cross-modal feature mapping training, to generate initial scene layout chart;Initial scene layout chart is carried out space-time consistency optimization, according to time sequence and space depth information constraint role action sequence and camera visual angle Synchronization relationship;Scene layout chart sequence is refined reconstruction, generates rendering result sequence;According to rendering result sequence and script semantic mapping relationship determines shot switching rhythm and focal length change trajectory, automatically generates the playable film and television scene video of natural plot logic, visual style uniformity.The application realizes from script text to video content generative semantic adaptive synthesis, improves the automation and intelligent level of film and television production.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital image processing, and in particular to a method for intelligent synthesis of film and television scenes based on generative multi-modal script semantic mapping. BACKGROUND

[0002] With the development of the integration of artificial intelligence and film and television production technology, research on generating visual scenes based on text content has gradually become a hot topic. In the prior art, a text-to-image or text-to-video generation model is usually used, taking script text as input, and generating corresponding picture content through a deep neural network. Some methods use a pre-trained language model to extract semantic features, and then combine a diffusion model or a generative adversarial network (GAN) to generate static pictures. Other methods use time series modeling to generate short dynamic video clips to assist storyboard design or scene preview. Such technology has been preliminarily applied in the fields of advertising, animation pre-production, and virtual shooting, and can reduce the workload of manual drawing and shot building to a certain extent.

[0003] However, the prior art has the problems of shallow semantic understanding, unstable modal correspondence, and insufficient temporal and spatial continuity of shots when processing film script text. Due to the complex semantic structure of the text and the cross-paragraph development of the plot logic, existing methods have difficulty in accurately identifying character relationships, spatial positions, and event sequences, resulting in unnatural discontinuities in the generated scenes in terms of character actions, lighting changes, and shot transitions. At the same time, the lack of global consistency in feature alignment between different modalities causes the generated results to deviate from the original script intent in terms of visual style, composition, and narrative rhythm, limiting its usability in real film production.

[0004] In view of the above problems, an automated synthesis method is needed that can start from the deep semantics of the script text, establish cross-modal correspondence, and maintain global consistency of shots. SUMMARY

[0005] The present application provides a method for intelligent synthesis of film and television scenes based on generative multi-modal script semantic mapping to improve the automation and artistic expression of film and television scene synthesis.

[0006] The present application provides a method for intelligent synthesis of film and television scenes based on generative multi-modal script semantic mapping, comprising:

[0007] Performing semantic unit segmentation and dependency structure analysis on the obtained film script text, and establishing script semantic mapping relationships based on character entities, plot events, and spatial position relationships;

[0008] The generative multi-modal semantic alignment method is used for cross-modal feature mapping training of the script semantic mapping relationship, so that the semantic units of the film script text and the visual style, composition relationship and lighting elements are in one-to-one correspondence, and an initial scene layout diagram matched with the script semantic mapping relationship is generated.

[0009] The initial scene layout diagram is subjected to spatio-temporal consistency optimization processing, the time sequence and spatial depth information in the script semantic mapping relationship are used to establish a synchronization constraint relationship between the role action sequence and the camera view angle, the continuity of the lighting direction, the human posture and the background continuity between adjacent shots is maintained through a dynamic feature propagation mechanism, and a scene layout diagram sequence subjected to spatio-temporal consistency optimization processing is obtained.

[0010] The generative self-supervised rendering method is used for rendering reconstruction of the scene layout diagram sequence, the semantic boundary maintaining constraint of the script semantic mapping relationship and the cross-frame pixel self-supervised signal are used for fine generation of the generated image, and a rendering result sequence with real lighting reflection characteristics and texture consistency is generated.

[0011] According to the rendering result sequence, the event rhythm nodes in the script semantic mapping relationship are used to determine the shot switching rhythm and the focal length change trajectory, a semantic-driven shot motion control method is used to plan the camera motion path, and a playable film scene video consistent with the script semantic mapping relationship and with natural plot logic is generated.

[0012] The beneficial effects of the technical solutions provided in the application include:

[0013] (1) By establishing the script semantic mapping relationship, the role entities, plot events and spatial position relationship in the film script text are structured and expressed, so that the script text information can be completely retained in the visual generation process, the high-precision correspondence of the text semantics to the visual semantics is realized, and the semantic understanding depth and the plot coherence of the film generation are significantly improved. (2) The generative multi-modal semantic alignment method is used for joint mapping of the text semantics and the visual feature space, so that the generated scene is consistent with the script semantics in terms of lighting, composition and style elements, the problems of semantic deviation and style distortion in the traditional text generated video are effectively avoided, and the global consistency of the cross-modal synthesis is improved. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a flowchart of a film scene intelligent synthesis method based on generative multi-modal script semantic mapping provided by the first embodiment of the application. DETAILED DESCRIPTION

[0015] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0016] The first embodiment of this application provides a method for intelligent synthesis of film and television scenes based on generative multimodal script semantic mapping. Please refer to... Figure 1 This figure is a schematic diagram of the first embodiment of this application. The following is in conjunction with... Figure 1 The first embodiment of this application provides a detailed description of a method for intelligent synthesis of film and television scenes based on generative multimodal script semantic mapping.

[0017] Step S101: Perform semantic unit segmentation and dependency structure analysis on the acquired film and television script text, and establish script semantic mapping relationship based on character entities, plot events and spatial location relationships.

[0018] Step S101 first requires obtaining a complete film / television script. Film / television scripts typically include dialogue, scene descriptions, character action descriptions, and plot development instructions. They are generally written in natural language, such as TXT, PDF, or industry-standard script formats. To enable the computer to understand this text, it is necessary to break it down into fine-grained semantic units. A "semantic unit" refers to the smallest component with relatively independent meaning in natural language processing; this component can be a word, phrase, or fixed-structure short sentence. For example, "He walked into the room" can be broken down into three semantic units: "he," "walked into," and "room." This processing can be accomplished using word segmentation algorithms. Common methods suitable for Chinese include the forward maximum matching algorithm based on dictionary matching, the Hidden Markov Model (HMM) word segmentation method based on statistical models, and the BiLSTM-CRF word segmentation model based on deep learning. Those skilled in the art can choose an appropriate model as needed, as long as it ensures that the semantic units remain independent and identifiable after segmentation.

[0019] After obtaining the semantic units, dependency structure analysis is needed to examine the relationships between them. Dependency structure analysis is a method in natural language processing used to identify the internal grammatical structure of sentences. Its core task is to determine the dependency relationships between words, such as subject-verb, verb-object, and modifier relationships. Common implementations of dependency structure analysis include the Arc-Standard algorithm based on transformational dependency grammar, neural network-based dependency parsing models, and dependency recognition models trained using Transformers structures. For example, for the sentence "The girl put down her backpack and sat on the sofa," dependency structure analysis can identify "girl" as the agent, "put down" as the action, "backpack" as the patient, "sit" as the action, and "sofa" as the location, thus structuring the sentence semantics into a computable dependency tree structure. In this step, the dependency analysis model needs to be used to convert each sentence in the script text into a corresponding dependency tree, thus providing a foundation for the subsequent construction of logical relationships.

[0020] After semantic unit segmentation and dependency structure analysis, it is necessary to identify character entities, plot events, and spatial relationships in the script text. Character entities refer to identified people or animals in the script, such as "Zhang Wei," "nurse," or "tiger." Technically, a Named Entity Recognition (NER) model is generally used. This model, through training, can recognize words in the text that represent categories such as "names," "occupations," "locations," and "time." Plot events refer to actions or state changes in the script that have a temporal progression, such as "opening the door," "escaping the scene," or "losing consciousness," and are usually composed of verbs and their extended structures. Those skilled in the art can use dependency analysis results combined with an event trigger word recognition model to determine which semantic units belong to plot events. Spatial relationships include the spatial distribution of characters, things, or events in the script scene, such as "indoors," "by the window," or "located on the right," and need to be identified through prepositional phrases or location entities in the dependency tree.

[0021] After extracting character entities, plot events, and spatial relationships, it is necessary to establish script semantic mapping relationships. Script semantic mapping relationships are a structured representation used to describe the connections between semantic elements within a script. Essentially, it is a graph structure, typically implemented as a graph. Each node in the graph represents a semantic element (such as a character, event, or scene location), and each edge represents a relationship between semantic elements, such as "Character A performs event B," "Event B occurs at location C," and "Character A is located at location C." For example, if the script contains the sentence "Li Ming picks up a water glass in the living room," the script semantic mapping relationship will include nodes such as "Li Ming" (character entity), "picks up" (plot event), "water glass" (prop entity), and "living room" (location entity), and establish three types of relationship edges: "Li Ming → picks up," "picks up → water glass," and "Li Ming → living room." To ensure that the script semantic mapping relationship can support subsequent cross-modal feature mapping processes, it is necessary to guarantee that this graph structure accurately reflects the temporal order, spatial structure, and character behavioral logic within the script's semantics. Therefore, when constructing relation edges, it is necessary not only to record the relationship type between semantic units, but also to label information such as the order of events, positional hierarchy, and action association strength. For example, the logical order of events can be represented by numbers, such as E1, E2, and E3, where E1 occurs before E2; spatial hierarchy can be represented by a tree structure, such as "indoor → living room → sofa area"; action association strength can be calculated based on the number of co-occurrences or syntactic distance. For example, if the character "Zhang Wei" repeatedly performs the action "entering the room" in four different scenes, the association strength can be set to high.

[0022] In constructing semantic mapping relationships within a script, if semantic ambiguity is encountered, a secondary determination is required using a context window. For example, in the text "He walked towards her," "he" and "she" might refer to different characters mentioned in the previous sentence. Those skilled in the art can use a referential resolution model to address this issue. This model determines the referential meaning through context comparison. For instance, if the preceding text states "Li Ming picked up his coat. He walked towards her," then "he" refers to "Li Ming," and "she" refers to the female character in the previous sentence. This referential relationship also needs to be incorporated into the script semantic mapping relationship to ensure that the subsequently generated visual scene correctly reflects the relationships between the characters.

[0023] By completing these steps, a well-structured, logically complete script semantic mapping relationship can be obtained, which can be directly processed by the computational model. The ultimate goal of this step is to transform the natural language form of film and television script text into a computable semantic structure, so that subsequent processes such as cross-modal alignment, scene layout generation, spatiotemporal consistency optimization, and rendering control have a reliable semantic foundation.

[0024] Step S102: Use a generative multimodal semantic alignment method to train cross-modal feature mapping on the script semantic mapping relationship, so that the semantic units of the film and television script text are associated with visual style, composition relationship and lighting elements, and generate an initial scene layout map that matches the script semantic mapping relationship.

[0025] In step S102, it is first necessary to clarify the meaning of the generative multimodal semantic alignment method. The generative multimodal semantic alignment method refers to a generative learning method that simultaneously processes textual and visual modal data. This method jointly models the semantic representation of the film script text and the target visual representation, learning the correspondence between the two in the same latent feature space. This allows the semantic units contained in the script semantic mapping relationship to be mapped into visual layout elements that can be used to generate the screen. Here, the textual modality mainly refers to various semantic units in the script semantic mapping relationship obtained in step S101, such as character entities, plot events, spatial location descriptions, etc.; the visual modality can be pre-collected shot reference images, storyboard sketches, or style reference images. These visual data are used to characterize visual style, compositional relationships, and lighting elements. The generative multimodal semantic alignment method uses a jointly trained deep network to map "who," "where," "what," and "what is the atmosphere of the scene" in the script into a continuous vector space, allowing this semantic information to directly drive the subsequent screen layout generation.

[0026] In practical implementation, two feature encoding networks can be constructed separately for the script semantic mapping relationship and the visual reference data. For the script semantic mapping relationship, a graph neural network can be used to encode the graph structure, mapping each character entity node, plot event node, and spatial location node to a semantic embedding vector with fixed dimensions. A semantic embedding vector is a set of real numbers representing the meaning of a semantic unit. For example, a character entity can be encoded as a vector of length 128, with each dimension corresponding to some abstract semantic feature, such as personality, emotion, or importance. These dimensions do not need to be explicitly defined by humans but are automatically learned through training. For the visual modality, a convolutional neural network or a visual Transformer can be used to extract features from existing shot reference images, encoding information such as overall composition, subject position, lighting direction, and color style into a 128-dimensional feature vector of the same length. To achieve cross-modal feature mapping training, a shared latent feature space needs to be designed, allowing direct comparison between the semantic embedding vector and the visual feature vector within this space.

[0027] The process of cross-modal feature mapping training can be understood as continuously adjusting the parameters of the two encoding networks to make the semantically corresponding text description and visual image as close as possible in the latent feature space, while semantically unrelated text and images are kept apart in this space. For example, suppose there is a script segment described as "On a city street at night, the protagonist stands under a dim street lamp," and a corresponding reference image showing a night street, street lamps, and a person standing under a lamp. After feature encoding, we can obtain a 128-dimensional vector on the text side and a 128-dimensional vector on the image side. The similarity between these two vectors is defined as cosine similarity, which is calculated by multiplying the two vectors by the same dimension, summing the results, and then dividing each sum by the product of the lengths of the two vectors. The closer the directions of the two vectors are, the closer the similarity value is to one; when the directions of the two vectors are very different, the similarity value is close to zero or negative. During training, by increasing the similarity of semantically corresponding samples and decreasing the similarity of non-corresponding samples, the model gradually learns to establish a stable mapping relationship between semantics such as "night," "street," "streetlight," and "protagonist" in the script and the overall dark tone, street structure, light source position, and character outline in the corresponding scene.

[0028] When generating the initial scene layout, it's necessary to further define visual style, compositional relationships, and lighting elements. Visual style can be understood as the overall artistic characteristics of the image, such as whether it's realistic, cartoonish, or cinematic, and also includes whether the color tone is cool or warm, the contrast is high or low, and whether the lens has noticeable graininess. These can be represented by encoding the statistical and high-level features of the entire image. Compositional relationships mainly refer to the relative positions, orientations, and center of gravity of the various subjects in the image. For example, the main character might be located in the left third of the image, background buildings occupy the upper half, and foreground objects are located in the lower edge area. In implementation, the image can be divided into a fixed-size two-dimensional grid, for example, dividing the image width into ten columns and the height into ten rows. Each grid cell represents a small area of ​​the image. By recording the grid position of each character or key object, a discrete representation of the compositional relationships is formed. Lighting elements include the approximate direction of the light source, light intensity, and color temperature, such as strong cool light projected from the upper right corner of the image, or warm backlighting coming horizontally from behind the image. This information can be obtained by explicitly encoding the light source direction vector and the light intensity scalar in the visual features.

[0029] After the generative multimodal semantic alignment method completes cross-modal feature mapping training, for a new script fragment, it only needs to input the script semantic mapping relationship into the text-side encoding network to obtain the corresponding semantic embedding vector. Then, based on the position of this vector in the joint feature space, the generative network is used to predict the corresponding visual layout parameters. The generative network can be a conditional generative adversarial network, a diffusive generative network, or other generative structures. Its input is the semantic embedding vector, and its output is a data structure containing layout information. This data structure is the internal representation of the initial scene layout map. Specifically, it can be designed as a two-dimensional matrix, where each row represents a scene element to be placed, such as a character, key prop, or main background object. Each column records the attributes of the element, including its normalized coordinate position in the image, its depth level (foreground, midground, background), its corresponding illumination intensity coefficient, and its associated visual style label. For example, for a simple scene containing a main character and a street lamp, the initial scene layout diagram can contain two rows of records: one row is "Main Character", with a horizontal position of 0.3, a vertical position of 0.6, a depth level of mid-ground, a light intensity of 0.8, and a style tag of "Realistic"; the other row is "Street Lamp", with a horizontal position of 0.2, a vertical position of 0.7, a depth level of background, a light intensity of 1, and a style tag of "Warm Light Source", etc.

[0030] To establish a one-to-one correspondence between semantic units in film and television scripts and visual style, compositional relationships, and lighting elements, sample pairs need to be constructed during the training phase. Each sample pair contains a sub-graph representing the semantic mapping relationship of the script and a matching real visual layout annotation. Real visual layout annotations can be generated manually, by having experienced storyboard artists or art designers delineate the positions of characters and main objects on reference frames and annotate their depth levels and lighting features; alternatively, they can be obtained automatically through analysis of existing film and television clips. For example, object detection networks can be used to find the positions of characters and objects, depth estimation networks can be used to infer the scene's depth distribution, and lighting estimation networks can be used to predict the direction and intensity of lighting. During training, the network parameters are gradually adjusted by comparing the differences between the layout parameters output by the generated network and the real layout annotations, enabling each type of semantic unit to form a stable visual correspondence pattern. For example, when the semantic pattern "the protagonist stands by the window" appears repeatedly in the script, the model will gradually learn to map this semantic pattern to a set of layout and lighting features such as "the character is close to one side of the frame, there is a window structure behind them, and there is bright lighting from outside the window," thus achieving a reliable semantic-to-visual correspondence.

[0031] The generation of the initial scene layout is not merely for determining object positions, but also for providing structured input for subsequent spatiotemporal consistency optimization. Therefore, time tags and scene identifiers can be attached when generating the initial scene layout. Time tags indicate the relative position of the layout within the overall script timeline, such as the act or scene number; scene identifiers distinguish different spatial locations, such as indoors, outdoors, streets, and room numbers. These tags can be directly inherited from the script's semantic mapping relationships; for example, paragraph structures and built-in scene numbers in the script can be used to generate these tags. In subsequent steps, spatiotemporal consistency optimization can determine which layouts should be temporally consecutive based on time tags and which layouts belong to the same physical space based on scene identifiers, thus applying spatial and temporal constraints across shots.

[0032] Through the above process, step S102 achieves cross-modal feature mapping training for script semantic mapping relationships and generation of the initial scene layout map, enabling the semantic units of the film and television script text to have clear corresponding positions, style attributes, and lighting features in visual space. This step not only provides an operable structural foundation for subsequent spatiotemporal consistency optimization, but also ensures that the mapping from semantics to vision is automatically established through a trainable generative multimodal semantic alignment method.

[0033] Below is a specific example of a deep network structure that maintains complete consistency with the logic described above. This deep network can be designed as a whole to consist of three main parts.

[0034] In the first part, the deep network includes a text-side feature encoding network for processing script semantic mapping relationships. This text-side feature encoding network takes the script semantic mapping relationship constructed in step S101 as input. The input format can be a graph structure containing several nodes and relational edges. Each node corresponds to a character entity, plot event, or spatial location description, and each edge represents a semantic connection between nodes, such as a character performing an action or an event occurring at a certain location. The text-side feature encoding network can be implemented using a graph neural network, such as a graph convolutional network or a graph attention network. Its core process includes neighborhood aggregation of the initial feature vector of each node, compressing the contextual semantics of the node into a fixed-dimensional vector representation through multi-layer information propagation. For example, if the initial feature of a node is a two-dimensional one-hot encoded vector, the features of adjacent nodes can be aggregated in the first layer, and weight updates and relation type encoding can be added in the second layer, ultimately outputting a semantic embedding vector of length 128 dimensions. For a script paragraph containing several characters and events, the text-side feature encoding network will output multiple semantic embedding vectors, each corresponding to a semantic unit.

[0035] In the second part, the deep network includes a visual side feature encoding network for processing visual reference data. This network takes a reference image, shot example, or style sample as input, which can be a three-channel color image or an image sequence. The network can be implemented using a convolutional neural network or a visual Transformer, extracting high-level semantic features of the image through several layers of convolution, pooling, or self-attention mechanisms, including overall visual style, color distribution, subject composition patterns, and lighting direction. The network ultimately encodes the input image into a 128-dimensional visual feature vector with the same dimensions as the text-side semantic embedding. The output of the visual side feature encoding network is used for cross-modal contrastive learning during the training phase, enabling the model to learn the correspondence between textual semantics and visual features.

[0036] In the third part, the deep network includes a joint latent space mapping network for performing cross-modal alignment and layout generation. This network takes the semantic embedding vectors output by the text-side feature encoding network as its primary input and uses the visual feature vectors output by the visual-side feature encoding network as supervision or contrast targets during the training phase. The joint latent space mapping network can employ a multilayer perceptron structure or a conditional generative network structure, and its main function is to map the semantic embedding vectors into a continuous space that can describe the scene layout. The output format of this network can be a structured vector containing layout parameters, such as a fixed-length vector group, where each component represents the horizontal position, vertical position, depth level, illumination intensity, and visual style label of a certain type of character or object in the image. For example, if layout information needs to be generated for the main character, the output could be a five-dimensional vector, where the first and second dimensions represent the normalized horizontal and vertical coordinates, the third dimension represents the depth level number, the fourth dimension represents the illumination coefficient, and the fifth dimension represents the style classification number. The Joint Latent Space Mapping Network achieves cross-modal mapping from semantics to visual layout by minimizing the difference between text-side features and visual-side features in the latent space, and minimizing the error between the generated layout and the real layout annotation.

[0037] Through the collaborative work of these three parts, the deep network can realize the generative multimodal semantic alignment method described in step S102, automatically predict scene layout parameters from script semantic mapping relationships, and generate an initial scene layout map that matches the script semantic mapping relationships.

[0038] In this embodiment, when using a generative multimodal semantic alignment method to train cross-modal feature mapping on script semantic mapping relationships, it is necessary to first refine the semantic elements of the character entities, plot events, and spatial relationships in the script semantic mapping relationships. Semantic element refinement refers to expanding each semantic unit in the script semantic mapping relationship to include not only the literal meaning of the text but also semantic attributes that guide visual generation, scene significance, and visual presentation intent. Semantic attributes typically describe the category, emotional tendency, action type, or event nature of the semantic unit, such as "character anger," "character walking," or "nighttime indoors." Scene significance describes the function of the semantic unit in the narrative, such as "foreshadowing before conflict" or "guiding key turning points." Visual presentation intent expresses the stylistic tendency that the semantic unit should present in the visual image, such as "high-contrast lighting," "soft warm tones," or "low-angle shooting." Through the above expansion, a set of semantic elements is formed, enabling it to fully express the narrative function and stylistic tendency that the semantic unit in the script semantic mapping relationship should present in visualization. In practice, semantic element sets can be generated by manually annotating the script text, extracting rules, or analyzing pre-trained semantic models.

[0039] After obtaining the semantic element set, a visual reference element set needs to be acquired for each semantic element set. The visual reference element set refers to a set of visual features that can express the character's appearance style, scene composition, and lighting direction characteristics. In this invention, the visual reference element set is not arbitrarily selected, but rather extracted from pre-established visual style reference information, composition feature reference information, and lighting feature reference information based on the character's identity, plot structure, and spatial environment in the script's semantic mapping relationship. For example, if the character's semantic attribute in the semantic element set is "calm middle-aged male detective," the plot description includes "nighttime interrogation scene," and the spatial relationship is "small indoor space," then the visual reference element set can include information such as "low color temperature light source," "side single light source illumination," "centered composition," and "compact background object arrangement." A visual reference library can be established using film and television datasets, photographic composition guidelines, or art reference material libraries, and categorized according to character identity tags, plot type tags, and spatial type tags to extract the corresponding visual reference element sets.

[0040] After obtaining the semantic element set and the visual reference element set, generative semantic correspondence inference needs to be performed between them. The purpose of generative semantic correspondence inference is to establish a one-to-one correspondence between textual semantic content and visual expression. The inference process requires analyzing the semantic description content, narrative function, and visual presentation intention layer by layer, and comparing and integrating them with the style information, composition information, and lighting information in the visual reference element set. For example, when the semantic element set indicates that the event is "oppressive" and the character is in a state of "psychological tension," while the visual reference element set contains visual features such as "low-angle upward view" and "strong light source in one direction," the inference process can map this emotional tension into visual generation instructions such as "increased contrast," "deepened shadows," and "adopting a strongly biased lighting direction." Ultimately, a cross-modal semantic correspondence result containing visual style information, compositional relationship information, and lighting direction information can be obtained, making the result fully reflect the visual characteristics of the script's semantic mapping relationship.

[0041] After generating the cross-modal semantic correspondence results, to ensure that the final visual result accurately expresses the textual semantics, multiple rounds of consistency checks need to be performed based on the cross-modal semantic correspondence results. The purpose of these multiple rounds of consistency checks is to detect whether the narrative logic contained in the textual semantics is consistent with the style features, composition, and lighting conditions contained in the visual reference element set. If inconsistencies are found, the style information, composition information, and lighting feature information in the cross-modal semantic correspondence results need to be readjusted. For example, if the textual semantics describe "a character running on a bright morning street," but the inferred lighting direction shows long shadows characteristic of evening, the consistency check process needs to correct the lighting direction to the typical short shadow direction of morning. If the textual semantics indicate that the composition should present an open space, but the inferred result includes dense background objects, the composition needs to be readjusted to conform to the narrative logic. Through multiple rounds of bidirectional checks, the consistency of the cross-modal semantic correspondence results among narrative logic, visual style, and lighting conditions can be ensured.

[0042] After obtaining the cross-modal semantic correspondence results that have passed consistency verification, an initial scene layout diagram needs to be generated based on these results. The initial scene layout diagram needs to include character positions, character poses, scene composition framework, lighting direction, and the arrangement of major objects in the scene, arranged according to the structural order of the script semantic mapping relationship. For example, if the script semantic mapping relationship describes "the character enters the room and sits down on the right side of the table," then the initial scene layout diagram needs to specify the character's spatial route from the doorway to the right side of the table, the character's pose changes, the table's position in the composition, and the direction from which indoor lighting should illuminate. Those skilled in the art can generate a complete initial scene layout diagram by converting the composition relationships in the cross-modal semantic correspondence results into spatial arrangement instructions, applying visual style information to the overall color tone setting of the image, and using lighting direction information to determine the positions of light sources for subsequent rendering.

[0043] This embodiment achieves a rigorous mapping between text semantics and visual elements by continuously refining semantics, establishing cross-modal correspondences, performing consistency checks, and generating visually executable layouts. This enables the invention to accurately reflect narrative intent and form standardized visual output during film and television synthesis.

[0044] Furthermore, the step of performing generative semantic correspondence inference between the semantic element set and the visual reference element set involves inferring the expression of each semantic element set in terms of visual style, composition, and lighting direction through layer-by-layer correlation analysis of semantic description content, narrative function, and visual presentation intention. This generates cross-modal semantic correspondence results containing visual style information, compositional relationship information, and lighting direction information, ensuring that the cross-modal semantic correspondence results fully reflect the visual characteristics of the script's semantic mapping relationship. Specifically, this includes the following implementation steps:

[0045] First, the process of extending the narrative context of semantic descriptions within a semantic element set requires clarifying what constitutes semantic description, what constitutes narrative context extension, and how to connect the two. Semantic description refers to the textual description of a specific character entity, plot event, or spatial relationship within the semantic element set. For example, the sentence "The protagonist walks quickly down a dimly lit corridor and looks back" is a typical semantic description. Narrative context extension goes beyond the direct literal meaning of the statement. It involves analyzing the statement's position within the entire script, the development of surrounding events, and the relationships between relevant characters to infer the implied atmosphere, characters' psychological states, and narrative pace. The atmosphere can include categories such as tension, oppression, relaxation, and warmth; characters' psychological states can include descriptions of fear, hesitation, anger, and anticipation; and the narrative pace can be understood as whether the plot progresses rapidly, smoothly, or deliberately slowly within that segment. For example, if the script has repeatedly mentioned "unknown pursuers" and "continuous unusual sounds," and the current semantic description is "the protagonist walking quickly in a dimly lit corridor and looking back," then the narrative context can be expanded to infer that the scene's atmosphere is tense and oppressive, the characters' psychological state includes vigilance and unease, and the narrative pace is accelerated. The system can assign qualitative levels to this atmosphere intensity, psychological tension level, and pacing, such as labeling the atmosphere as "high tension," the psychological state as "high vigilance," and the pacing as "accelerated progression." This expanded information is collectively referred to as semantic contextual information, which is one of the fundamental inputs for subsequent visual inference processes.

[0046] After obtaining semantic contextual information, it is necessary to perform narrative association filtering on the visual style, compositional relationships, and lighting direction information within the visual reference element set. The visual reference element set is a pre-established set of visual features. Visual style information can include cool tones, warm tones, high contrast, low saturation, realism, and dreamlike styles, etc. Compositional relationships can include centered composition, diagonal composition, foreground prominence, and background extension, etc. Lighting direction information can include top lighting, side lighting, backlighting, and shadow lighting, etc. Narrative association filtering means excluding visual elements that do not match the current situation from the visual reference element set based on the intensity of the atmosphere, the characters' psychological tension, and the changing characteristics of the narrative rhythm reflected in the semantic contextual information, while retaining those visual candidates that accurately represent the semantic contextual information. Continuing with the example of "walking quickly down a dimly lit corridor," if the semantic context indicates a highly tense atmosphere, characters on high alert, and an accelerated narrative pace, the system will tend to retain cool tones, high contrast, localized highlights, and large areas of darkness when filtering visual style information, rather than choosing bright, warm, or soft color tendencies. When filtering compositional information, it will retain compositions that emphasize the characters' sense of confinement and oppression, such as using a narrow, deep corridor perspective, positioning the characters to one side of the frame, and leaving ample shadow space on the other side, rather than choosing expansive panoramic or symmetrical compositions. Regarding lighting direction information, it will prioritize backlighting from the end of the corridor, top lighting from above, or sidelight filtering through a door crack to reinforce the atmosphere of uncertainty and insecurity. The visual information obtained through narrative association filtering constitutes the candidate visual expression information, which has already achieved a preliminary match with the current narrative context at the semantic level.

[0047] After obtaining candidate visual expression information, it is necessary to rank the candidate visual expression information according to the importance of different visual features to the narrative expression. Ranking the expression weights refers to determining the priority order of each visual feature in the final image generation based on its contribution to conveying semantic contextual information. For those skilled in the art, this can be understood as a process of ranking importance. For example, in the tense corridor scene mentioned above, in order to express a "highly tense atmosphere" and a "highly vigilant psychological state," the contrast between light and shadow created by the lighting in the image often has a more direct impact on the viewer's perception than subtle changes in the position of objects in the composition. Therefore, lighting-related visual features can be given higher expression weights, while color style and composition structure can be given slightly lower weights. In practical implementation, a qualitative ranking method can be used, such as marking the direction and intensity of lighting as "high priority," the overall color tone as "medium priority," and the layout of secondary background elements as "low priority." The visual information after expression weight ranking will serve as input for visual fusion processing. In subsequent processing, high-weight features will have stronger control when compositing the image, while low-weight features will only be used for auxiliary modification without compromising the overall semantic expression.

[0048] After ranking the expression weights, it is necessary to perform a layer-by-layer fusion process based on the visual information after the expression weight ranking, according to the order of influence of visual style information on compositional information and the order of influence of compositional information on lighting direction information. Layer-by-layer fusion is not an abstract concept, but refers to superimposing different types of visual features onto a unified scene expression in a certain order, ensuring that the final image is harmonious in color, composition, and lighting. Specifically, the overall style tendency of the image can be determined first by the visual style information. For example, the entire corridor scene can be determined to use a cool blue-green tone and high contrast, thus establishing the basic color atmosphere of the image. On this basis, the positions of the characters, camera angles, and the distribution of the main objects in the scene can be determined according to the compositional information. For example, the protagonist can be placed on the left side of the image, the depth of the corridor can be regarded as the direction of depth, and the local light source can be placed in the right rear of the image. Finally, the lighting direction information can be used to determine where the light enters from, which areas become bright areas, which positions form shadows, and the areas where the viewer's eyes are likely to focus first. For example, a design could be created with a flickering overhead light deep in the corridor, illuminating the subject from a slightly rearward position. This would cause the subject's face to be partially obscured by shadow, while highlights would appear on the ground in front of their feet, thus creating clear tonal gradations and contrasts within the image. The result of this layer-by-layer fusion process is a preliminary cross-modal visual expression, which already maintains consistency with the aforementioned semantic contextual information in terms of style, composition, and lighting.

[0049] After obtaining preliminary cross-modal visual expression results, it is necessary to conduct a narrative consistency retrospective analysis on these results based on the character relationships, event logic, and spatial order within the script's semantic mapping. The purpose of narrative consistency retrospective analysis is to examine whether the visual expression of the current scene is consistent with the preceding and following scenes and the character's long-term image from the perspective of the overall plot structure. For example, if the script's semantic mapping clearly states that the protagonist appears as "resolute and calm" from the beginning of the story, but the current scene's visual expression uses an overly exaggerated, uncontrolled overhead camera angle, it may be inconsistent with the character's consistent image. Similarly, if the previous scene was a brightly lit outdoor scene during the day, and the current scene is almost immediately following the previous scene, yet uses an extremely dark environment with no natural transition, it may also be inconsistent with the event logic. Spatial order is used to confirm whether the current scene is consistent with the spatial relationships described earlier. For example, if the story has stated that the character continuously moves through the living room, corridor, and stairs indoors, if the direction of the lighting in the current scene's corridor is completely opposite to the direction of the lighting at the stairwell mentioned earlier, it may affect the audience's understanding of spatial continuity. During the backtracking analysis, if contradictions are found between the initial cross-modal visual expression results and the script semantic mapping relationship in terms of character relationships, event logic, or spatial order, necessary fine-tuning of visual style information, compositional information, and lighting direction information is required. For example, the lighting intensity can be appropriately reduced to maintain consistency with the lighting environment of the previous night scene, or the composition can be adjusted to connect the character placement positions with the movement paths in the previous scene. Through the above backtracking and correction, the cross-modal visual expression results achieve consistency among narrative logic, spatial expression, and visual presentation. The final cross-modal semantic correspondence results can then serve as a reliable basis for subsequent initial scene layout generation and spatiotemporal consistency optimization.

[0050] Step S103: Perform spatiotemporal consistency optimization on the initial scene layout diagram. Based on the temporal order and spatial depth information in the script semantic mapping relationship, establish a synchronization constraint relationship between the character action sequence and the camera perspective. Maintain the consistency of lighting direction, character posture and background continuity between adjacent shots through a dynamic feature propagation mechanism to obtain the scene layout diagram sequence after spatiotemporal consistency optimization.

[0051] In step S103, the goal is to perform spatiotemporal consistency optimization on the initial scene layout generated in step S102, ensuring consistency in lighting direction, character poses, background structure, and action sequence between consecutive shots, thus forming a sequence of layouts with natural transitions. To accomplish this, it's necessary to first define what an initial scene layout is. An initial scene layout is a structured layout description containing the two-dimensional position coordinates, depth level, lighting coefficients, and style tags for each character or key scene element. This information is typically stored in vector or matrix form. A typical layout can be represented as multiple rows of data, each containing the character name, horizontal position, vertical position, depth level, and lighting intensity. For example, if a shot includes a character named "protagonist," that character might have a horizontal position of 0.3, a vertical position of 0.6, a mid-range depth level, and a lighting intensity of 0.8. These initial layouts typically only reflect the spatial information of a single frame and lack dynamic connections between shots, thus requiring spatiotemporal consistency optimization.

[0052] To ensure that the layout between consecutive shots conforms to human visual perception, a sequence of character actions needs to be constructed first based on the temporal order information in the script's semantic mapping. This semantic mapping, established in step S101, includes the sequential relationships of events; for example, "the protagonist gets up" occurs before "the protagonist walks to the window." If the text does not explicitly state the temporal order of certain events, the implied order can be determined through dependency trees and context. For instance, in the sentence "He then looked out the window," "then" indicates the order of the actions. The action sequence can be represented as a time-ordered list, with each element containing the action type and the character involved. For example, "E1: The protagonist gets up" and "E2: The protagonist walks to the window." Subsequently, this action sequence needs to be mapped to the initial scene layout diagram to ensure that the posture changes presented in the layout diagram are consistent with the semantics of the text.

[0053] Camera perspective refers to parameters such as the virtual camera's position, orientation, focal length, and field of view in three-dimensional space. In this step, it's not necessary to render specific images; rather, it's sufficient to determine how these parameters change over time. For example, if a script requires "the camera to follow the protagonist toward the window," the camera's lateral position should gradually shift with the protagonist's position, while the focal length can remain stable. If the script describes "the camera gradually zooms in on the protagonist's face," the focal length gradually decreases over time to create a visual push-in effect. Smooth changes can be achieved by interpolating the camera parameters, for example, using linear interpolation to ensure that the camera's position parameters change at a uniform rate during the transition from the first shot to the second.

[0054] After initializing the synchronization constraints, a dynamic feature propagation mechanism needs to be executed to maintain the continuity of lighting direction, character poses, and background structure between consecutive shots. The dynamic feature propagation mechanism refers to applying key features from the previous layout image to the next layout image through certain propagation, smoothing, or transformation rules. This ensures that the next layout image inherits the visual features of the previous shot while making necessary changes based on the action sequence. For example, if the lighting direction in the previous shot came from the upper right corner, the light source in the current shot should remain in a similar direction; otherwise, the image will appear abruptly inconsistent. Lighting direction can be represented using vectors, such as a three-dimensional vector (x, y, z). If the direction in the previous shot was (0.5, 0.8, 0.2), the direction in the current shot can be maintained by making small random perturbations on this vector, for example, within ±0.05, changing it to (0.48, 0.79, 0.22), thus maintaining a roughly consistent lighting direction. Similarly, character poses can be represented by the positions of key body points, such as the coordinates of the head, shoulders, hands, and feet. If the protagonist is raising his hand in the previous shot, and the action sequence indicates that he will continue to raise his hand in the next instant, then the coordinates of the key points should be moved appropriately along the direction of the arm raising, rather than suddenly jumping to a completely different position.

[0055] The background structure also needs to remain stable in a series of shots. It can be represented as a layered structure, for example, divided into foreground, midground, and background. The background typically contains elements that don't change rapidly with movement, such as rooms, streets, and buildings, and their positions can be adjusted using perspective transformations. For example, when the camera moves from one position to another, objects in the background will shift accordingly according to perspective rules. This can be achieved using homogeneous coordinates and a 3D transformation matrix. For instance, if a background object's 3D coordinates are (x, y, z), and the camera position moves by an offset (dx, dy, dz), then the object's new coordinates can be represented as (x−dx, y−dy, z−dz). In this way, the background can maintain a realistic movement effect in a series of shots.

[0056] During the optimization process, if unreasonable jumps are found between certain layout images, such as sudden and drastic changes in character positions, discontinuous lighting directions, or background structures that do not conform to perspective rules, these jumps can be corrected using smoothing constraints. Smoothing constraints refer to weighted interpolation of intermediate frames based on the values ​​of the previous and next frames, making the transitions more natural. For example, if a character's horizontal position is 0.2 in the previous frame, 0.4 in the next frame, and the initial value in the current frame is 0.6, a weighted average formula, such as (0.2×0.25 + 0.4×0.25 + 0.6×0.5), can be used to correct it, making its new horizontal position 0.45, thus making the displacement changes more reasonable.

[0057] Through the aforementioned dynamic feature propagation, smoothing constraints, and camera synchronization control, a sequence of scene layout maps optimized for spatiotemporal consistency can be obtained. This sequence not only retains the initial layout information generated in step S102, but also ensures clear logical coherence between consecutive shots in both time and space, thus providing structured, continuous, and highly reliable input data for subsequent rendering reconstruction.

[0058] In this embodiment, in the initial scene layout diagram, based on the temporal order of each plot event in the script semantic mapping relationship and the foreground, midground, and background annotation results of the characters in spatial depth, a time index and spatial depth marker are added to each initial scene layout diagram. The time index can be understood as an identifier used to indicate the specific position of the shot on the plot timeline. For example, it can be numbered sequentially from scene 1 to scene N according to the order of events in the script, or it can be further subdivided within the same scene into "shot 1", "shot 2", etc., to indicate the chronological order of the scenes. The spatial depth marker is used to indicate the distance relationship of characters or important objects in the same scene layout diagram in terms of shot size. For example, objects closer to the virtual camera are marked as foreground, objects at a medium distance are marked as midground, and objects far from the camera and mainly serving as background environment are marked as background. For specific implementation methods, the layout can be divided based on the depth coordinates of each character or object relative to the virtual camera or their relative size in the frame. For example, characters occupying a certain proportion of the frame and located near the bottom are designated as foreground; medium-sized objects in the center of the frame are designated as midground; and smaller buildings or scenery near the top of the frame or the vanishing point are designated as background. In this way, a sequence of scene layout diagrams arranged chronologically, with each diagram marked with spatial depth, can be formed, providing a clear foundation for subsequent spatiotemporal consistency analysis.

[0059] After obtaining a sequence of scene layout maps with time indexes and spatial depth markers, it is necessary to organize the movement of each character between adjacent scene layout maps based on the continuous changes in character position and posture within the sequence. This forms a character action sequence describing the character's movement path and posture transitions over time. A character action sequence refers to a trajectory that, for the same character, is linked together according to the time index, connecting its position, orientation, body posture, and other elements in each scene layout map into a trajectory that evolves over time. For example, in a series of consecutive scene layout maps, a character is located on the left-foreground side of the first map, facing right; in the second map, they move to the center of the map, leaning slightly forward; and in the third map, they have moved to the right-side position and turned around. In these three maps, the character forms a movement path from left to right and then slightly to the right front, with their posture transitioning from standing naturally to leaning forward and then turning around. This information can be recorded as the direction of position change, the magnitude of displacement, and the type of posture change. Building upon this, it's necessary to extract the changes in camera viewpoint over time based on the camera's framing position, orientation, and spatial depth markers in the scene layout diagram. This results in a camera viewpoint change sequence that corresponds one-to-one with the character's action sequence. The camera viewpoint change sequence describes how the virtual camera's observation position and orientation change relative to the previous time index at each time index. For example, does the camera remain stationary and only change its focus, or does it translate with the character, or rotate around the character by a certain angle? By constructing a one-to-one correspondence between the character's action sequence and the camera viewpoint change sequence, a data foundation can be provided for establishing synchronization constraints later.

[0060] After constructing the character action sequence and camera view change sequence, synchronization constraints need to be established for the scene layout diagram between each pair of adjacent time indices. Synchronization constraints can be understood as a set of allowable ranges that limit how the character moves in time, how the camera adjusts in time, and how the foreground, midground, and background change within the frame. This ensures that the changes in the frame conform to the character's behavioral logic while maintaining a continuous and natural viewing experience. These synchronization constraints include at least the expected position range of the character at the current time index, the allowable deviation range of the character's posture changes, the allowable offset range of the camera view relative to the previous time index, and the allowable displacement range of the foreground, midground, and background within the frame calculated based on spatial depth markers. The expected position range refers to the approximate reasonable area of ​​the character's position in the current frame, considering the character's movement trend and position in the previous frame. For example, if the character took a step to the right in the previous frame, the character's position in the current frame should be within a certain distance to the right of the previous frame's position, rather than suddenly jumping to the other side of the frame. An allowable offset distance can be empirically set; for example, the character's displacement between two adjacent frames should not exceed a certain proportion of the screen width to prevent a sense of jumpiness. The allowable deviation range for posture changes limits illogical abrupt changes in character posture, such as instantly changing from standing upright to crouching or falling without a transition. Regarding camera perspective, smooth camera movement can be maintained by setting the maximum movement distance and maximum rotation angle of the camera relative to the previous frame in each frame. The allowable displacement range of foreground, midground, and background within the frame can be determined by spatial depth markers and temporal indexes. For example, distant buildings in the background are typically allowed only slight translation between adjacent frames to maintain background stability, while foreground objects can be allowed more significant displacement to simulate parallax changes during real camera movement. These constraints combine to form a set of spatiotemporal constraint parameters for constraining character actions, camera perspective, and scene hierarchy.

[0061] After obtaining the set of spatiotemporal constraint parameters, dynamic feature propagation processing needs to be performed between adjacent scene layouts based on this parameter set. Dynamic feature propagation processing refers to using the lighting direction information, character pose information, and background object spatial distribution information that already satisfy the spatiotemporal constraints at the previous time index as reference information, and passing this reference information along the time index to the next frame or multiple frames. During this process, lighting directions, character poses, and background object positions that do not satisfy the spatiotemporal constraint parameter set are smoothly adjusted. Specifically, when performing dynamic feature propagation, the system can first check whether the lighting direction in the current frame's scene layout deviates too much from the previous frame. For example, if the previous frame's lighting came from the upper right of the screen, while the current frame's lighting suddenly changes to coming from the lower left, this change usually creates an unnatural feeling for the viewer. In this case, the system can adjust the lighting direction of the current frame to a relatively close position based on the previous frame's lighting direction, such as gradually rotating from the upper right to directly above, rather than suddenly turning to the left, thus creating a gradual transition effect between multiple frames. Regarding character poses, if the previous frame shows a character raising their hand, but the current frame shows the character's hands hanging down and facing the opposite direction, the current frame can be adjusted to reflect the latter half of the hand-raising motion by referencing the previous frame's pose. For example, the arm could be raised further or slightly paused at shoulder height to ensure continuity in the pose change. The spatial distribution of background objects can also be adjusted by referencing the previous frame's layout. For instance, if a door at the end of a corridor was centered in the previous frame, and its position shifts slightly to the left in the current frame due to a slight camera pan, this is within the normal range of movement. However, if the door disappears or moves completely to the edge of the frame in the current frame, the background layout needs to be corrected using dynamic feature propagation to maintain reasonable spatial continuity. In this way, a sequence of intermediate scene layouts adjusted by dynamic feature propagation can be obtained, achieving a smooth transition in local timing regarding lighting direction, character pose, and background continuity.

[0062] After obtaining the intermediate scene layout sequence, a higher-level global consistency backtracking analysis is needed, combining the cross-shot character relationships and event logic within the script semantic mapping relationship. This global consistency backtracking analysis differs from the aforementioned local smoothing based on adjacent frames; it examines the overall consistency of multiple consecutive shots in terms of temporal order, spatial depth, character movement trajectories, and camera perspective changes from the perspective of the entire plot segment. For example, if the script semantic mapping relationship indicates that the plot describes "a character entering the hall from the courtyard, then walking towards the stairs and slowly climbing up," then in the intermediate scene layout sequence processed by the spatiotemporal constraint parameter set, the courtyard scene should appear at the first time index, followed by the hall interior scene, and then the staircase scene. If a spatial jump inconsistent with the script logic occurs in a certain intermediate frame—for example, a character suddenly appears at the top of the stairs without any intermediate climbing process—the global consistency backtracking analysis will mark this as an anomaly inconsistent with the overall spatial layout. The character position and camera perspective in that frame need to be readjusted to create a reasonable spatial transition with the preceding and following shots. Similarly, regarding lighting direction, if the plot sets the time as evening when characters move from outdoors to indoors, the overall lighting in the scene should gradually shift from strong natural light to artificial indoor lighting as the characters move from the courtyard into the hall. If a lighting scene appears in the middle frame that is completely inconsistent with the preceding and following frames, such as a sudden appearance of strong midday light, the lighting information of that frame needs to be corrected during the backtracking process to ensure consistency with the preceding and following frames and the event time setting. Regarding character postures and the relative positions between characters, if two characters are facing each other and talking in the previous shots, but suddenly stand back-to-back with their positions completely reversed in a middle shot, this also needs to be corrected through global backtracking to ensure that the change in behavior conforms to the logic of the event development in the script. Through such global consistency backtracking analysis, the lighting direction, character postures, and background structure are readjusted as necessary, thereby creating a unified spatiotemporal continuity in the entire scene layout sequence across time sequence, spatial depth, character movement trajectories, and camera perspective changes, ultimately obtaining a scene layout sequence optimized for spatiotemporal consistency.

[0063] Step S104: The scene layout map sequence is rendered and reconstructed using a generative self-supervised rendering method. The generated image is refined based on the semantic boundary preservation constraint of the script semantic mapping relationship and the cross-frame pixel self-supervised signal, generating a rendering result sequence with realistic lighting reflection characteristics and texture consistency.

[0064] In step S104, the scene layout sequence obtained in step S103 after spatiotemporal consistency optimization needs to be transformed into a realistic image sequence. This process is accomplished using a generative self-supervised rendering method. A scene layout sequence refers to a set of structured layout descriptions generated for multiple consecutive time points or multiple camera shots. Each layout corresponds to an expected frame and includes information such as the position, size, depth level, and lighting intensity of characters or objects within the frame. This information is typically encoded using two-dimensional coordinates, depth markers, and lighting coefficients. For example, a decimal between zero and one can represent the normalized position of a character, integers can be used to mark the foreground, midground, and background, and numerical values ​​between zero and one can represent the intensity of illumination. The goal of the generative self-supervised rendering method is to automatically generate high-quality images that match the layout, without relying on manual frame-by-frame annotation, by utilizing the layout information itself and the temporal continuity between adjacent frames.

[0065] Generative self-supervised rendering can be understood as an image generation process with layout as the conditional input. "Generative" means that image pixels are directly synthesized through a trained generative network, rather than simply calling fixed textures or preset templates; "self-supervised" means that during training, the inherent constraints of the data itself, such as temporal consistency, spatial structure, and semantic boundaries, are mainly used as supervision signals, without relying on externally labeled target images. Specifically, in implementation, a generative network can be designed with a layout map as input. This network receives scene layout information for each frame and converts parameters such as character positions, object sizes, lighting intensity, and style tags into a preliminary rendered image. The preliminary rendered image refers to an image that, under the premise that the geometry and lighting are generally correct, already possesses basic image content and subject outlines, but the details, textures, edge sharpness, and reflection features are not yet fully refined. At this stage, each element in the layout map can be mapped to a corresponding local image region. For example, the character's position in the image is determined based on the character's two-dimensional position and size, the depth of field effect is determined based on the depth level, and the background area is filled by distorting or scaling pre-learned style texture blocks.

[0066] After generating the initial rendered image, semantic boundary preservation constraints need to be introduced to ensure that the boundaries between different semantic regions in the image are clear and consistent with the object range defined in the script's semantic mapping relationship. Semantic boundaries refer to the contour lines that separate different categories of regions in an image, such as the boundary line between a person's outline and the background wall, or the dividing line between a tabletop and an object. Technically, a semantic mask can be derived from the layout map. This mask divides the image into multiple semantic regions, each corresponding to a semantic label, such as a person, table, wall, or window. Then, after the generative network outputs the image, the intensity change areas in the semantic mask and the generated image are compared. The accuracy of edge generation is determined by calculating whether the color changes near the semantic boundaries closely match the mask boundaries. If a significant offset is found between the edge region of the person's outline in the generated image and the boundary of the person region in the semantic mask, the feature weights of the corresponding region in the generative network can be adjusted so that the person's edges are drawn more closely along the semantic mask boundaries in the next iteration, thus preserving the semantic boundaries.

[0067] Cross-frame pixel self-supervised signals refer to using the inherent temporal continuity between adjacent frames to constrain the rendering process, ensuring that the pixel representation of the same object does not exhibit unreasonable jumps between adjacent frames. To construct such a self-supervised signal, the motion trajectory of objects between consecutive frames needs to be inferred from the scene layout map sequence obtained in step S103. For example, if the lateral position of a character in the layout map changes little from frame one to frame two, while the depth level and illumination coefficients remain essentially the same, it can be inferred that the character's pose changes little between the two frames, and the pixel color and texture of the corresponding character area in the image should be highly similar. In a concrete implementation, a cross-frame correspondence can be established for each semantic region. For example, the position and size of the character in the previous frame can be remembered, and the position and size of the character in the current frame can be predicted based on the layout map, thus establishing a pixel correspondence between the two frames. Then, the pixel color differences at these corresponding positions are compared. If the difference is small, it indicates that the rendering result conforms to temporal continuity; if the difference is large, such large differences are penalized during training, causing the generative network to reduce such unnatural jumps in subsequent iterations.

[0068] Let's take a simple example to illustrate how to constrain based on cross-frame pixel self-supervised signals. Suppose that in frame one, the average color intensity of the area corresponding to the protagonist's left shoulder can be represented by a three-dimensional vector, corresponding to the red, green, and blue channels, with components of 150, 130, and 120 respectively. In the layout image, the protagonist's position in frame two is only slightly moved to the right, while the lighting intensity remains similar. Therefore, the average color intensity of the area corresponding to the protagonist's left shoulder in frame two should not differ significantly from that in frame one; for example, the variation in each channel should not exceed 20. If, in the generated image, the color intensity vector for this area in frame two becomes 200, 100, and 80, then the difference compared to frame one is too large, indicating that the generation process introduced an unreasonable abrupt change. During training, this difference can be used as a loss signal to feed back to the generation network, prompting the network to adjust the colors to be closer to the level of frame one in subsequent generation, for example, adjusting them to 165, 125, and 110. This allows for slight differences due to subtle changes in the shot while avoiding drastic and unnatural variations.

[0069] Within the framework of self-supervised rendering, illumination consistency can be used to provide additional self-supervised signals. Illumination consistency refers to the principle that, within the same scene, unless the script's semantic mapping explicitly describes a change in the light source, the overall illumination direction and brightness should remain stable within a certain range. Based on the illumination coefficients and light source direction descriptions in the layout diagram, a global illumination vector can be pre-estimated for each frame. After image generation, the position and brightness level of the main highlighted areas in each frame are detected to determine whether the actual generated illumination matches the estimated illumination. If a highlighted area in a frame suddenly appears in a position inconsistent with the light source direction—for example, if the script describes "sunlight shining in from the window," but the bright area in the generated image is concentrated on the wall opposite the window—this inconsistency can be used as a penalty signal during training. This causes the network to be more inclined to increase brightness in areas consistent with the semantic light source position during subsequent generation, thus achieving physical plausibility in the illumination direction.

[0070] By combining semantic boundary preservation constraints and cross-frame pixel self-supervised signals, generative self-supervised rendering methods continuously refine the initially rendered image, making object outlines clearer, textures richer, and lighting effects more consistent with real-world reflection characteristics. Refinement generation not only increases resolution but also includes rendering more details on faces, clothing wrinkles, ground reflections, and highly reflective materials such as metal or glass. For example, in an indoor scene, a metal kettle might be placed on a table. In the initial rendering, the kettle's surface is just a rough gray area. After self-supervised refinement, the metal surface can reflect highlights from windows and blurred reflections of other objects in the room, making it look more realistic and believable.

[0071] By repeatedly rendering, reconstructing, and refining each frame of the scene layout sequence, a continuous sequence of rendered results can be obtained. This sequence refers to an image sequence that has undergone self-supervised rendering and refinement. Each image possesses a scene structure matching the layout and lighting and style features consistent with the script's semantic mapping. This sequence not only exhibits realistic lighting and reflection characteristics and texture consistency frame by frame but also maintains natural temporal continuity and semantic consistency between frames, providing a reliable visual foundation for subsequent steps involving camera movement control and editing rhythm planning based on the rendered result sequence.

[0072] The following example, using the passage "Daiyu's First Visit to the Jia Mansion" from *Dream of the Red Chamber*, demonstrates how a generative self-supervised rendering method generates a realistic sequence of rendered images from a layout graph: Assume the script's semantic mapping is described as: "Daiyu enters through the mansion gate and walks towards the main hall. The midday sun shines vertically down from above the courtyard, making the bluestone ground reflect light clearly. The accompanying maidservant is slightly behind." Step S103 has already generated a layout graph sequence for three frames, where the positions of the characters and scenes, as well as the direction and intensity of the lighting, all have explicit values.

[0073] For example, in the first frame layout image, Daiyu's 2D position is (0.32, 0.60), the depth level is mid-ground, and the lighting coefficient is 0.85; the position of the nanny is (0.28, 0.63), and the depth level is slightly behind mid-ground; the bluestone ground of the courtyard serves as the background, with a lighting coefficient of 0.9; the light source direction is (0.02, −0.98), indicating directly above and downwards. In the second and third frame layout images, Daiyu's horizontal position shifts slightly to the right (the change does not exceed 0.03), the nanny maintains the same relative position, and the lighting direction remains stable. At the beginning of the rendering stage, the generator network receives the first frame layout image and maps each semantic region into preliminary image blocks. For example, Daiyu's figure is initially rendered as the outline of a blue-green long dress, the bluestone ground is rendered as a roughly gray-green texture block, and the sunlight-irradiated area generates slightly bright rectangular light spots, but the details are not yet fully refined. The image at this stage can be understood as a "sketch," without complex textures, lighting reflections, and material properties.

[0074] Subsequently, based on semantic boundary preservation constraints, the generative network performs fine-tuning of the image contours. For example, in the initial image, the edge of Daiyu's robe hem might overlap with the background bluestone ground. By comparing with the semantic mask, the system finds that the color gradient changes of the figure's contour are inconsistent with the semantic boundary. Therefore, the rendering network enhances the edge features in this area, making the edge of the figure's hem present clearer refracted lines, and adds a faint highlight line to the edge of the fabric according to the direction of the light, so that it matches the "midday sunlight" in the script's semantics.

[0075] In the cross-frame pixel self-supervised stage, the system infers the movement trajectories of characters and objects in consecutive frames based on the layout map. For example, from frame one to frame two, Daiyu's position changes very little, so the color of the chest area of ​​her robe should remain basically consistent. Assuming the average color of this area in frame one is (175, 155, 150), the average color of this area in the generated image of frame two becomes (200, 120, 140), which is too large a difference. The system feeds this unreasonable difference back to the network through self-supervised loss, so that the network adjusts the color to be closer to that of frame one in the next rendering, for example, correcting it to (178, 152, 149), which allows for slight changes in lighting but avoids abrupt changes.

[0076] For the bluestone ground, since the illumination coefficient remains constant across the three frames in the layout diagram, the self-supervised mechanism requires that the reflective intensity of this area remain stable over time. For example, if the average brightness of the highlight area on the ground in frame one is 240 (represented as 0~255), it should not suddenly drop to 180 in frame two. Otherwise, the system will apply a smoothing constraint to the brightness trend, correcting the brightness of frame two to approximately 235 to ensure the continuity of the highlights.

[0077] During the material rendering refinement stage, the system automatically adds feature details based on semantic boundaries and self-supervised signals. For example, the bluestone ground will naturally exhibit slightly damp reflective patterns, the embroidery texture on Daiyu's sleeves will maintain a consistent position across consecutive frames, and the folds of the nanny's clothing will subtly change without abrupt shifts during walking. Based on the top-to-bottom lighting direction in the layout diagram, the character's head and shoulders will have enhanced highlights, while the lower body clothing will have noticeable shadows. After processing using this method, the three final rendered images present the following effect: the texture of Daiyu's clothing remains consistent across the three frames, the reflective brightness of the bluestone ground transitions naturally, the semantic boundaries between the character and the background are clear and stable, sunlight from above creates soft shadows that change slightly with the character's movement, and the overall image appears visually natural, conforming to the lighting style of traditional film and television "midday courtyard" scenes.

[0078] In this embodiment, within the scene layout sequence, the semantic annotations of character outlines, scene object ranges, and structural boundaries are analyzed based on the script semantic mapping relationship. First, it's necessary to clarify the meaning of the script semantic mapping relationship and semantic annotation. The script semantic mapping relationship refers to mapping semantic information about characters, scenes, objects, and events in the film / television script onto corresponding visual structures, obtaining the character's position in the frame, the approximate range of important objects in the scene, and the semantic meaning of various structural boundaries in the frame. For example, if the script states, "The female lead stands by the window, behind her is an old wooden cabinet, and outside the window is a rainy night street," the script semantic mapping relationship will include the character outline area corresponding to the "female lead" character entity, the scene object ranges corresponding to "window," "wooden cabinet," and "street," and the structural boundary positions between indoors and outdoors. Semantic annotation can be obtained through the earlier semantic analysis and multimodal alignment training phase, that is, in each scene layout diagram, the outline of the area where the character is located, the outline of the area where the main object is located, and the positions of structural boundaries such as walls, window frames, and door frames are marked. This step requires parsing these semantic annotations. The parsing process can be understood as converting the semantic annotations into region partitioning information usable in the rendering stage. Each scene layout image is divided into several semantic regions, such as character regions, background environment regions, foreground object regions, and structural boundary regions. After parsing, semantic boundary indication information that distinguishes different semantic regions is generated on each scene layout image. This semantic boundary indication information can be represented pixel-by-pixel or region-by-region. For example, pixels inside a character's outline are labeled as character regions, pixels belonging to indoor objects are labeled as environment regions, and pixels belonging to the street outside the window are labeled as external scene regions. All the labeled information combined constitutes the semantic boundary indication information for that scene layout image. In this way, different semantic regions can be accurately distinguished during the rendering stage, allowing for differentiated processing of different semantic regions in subsequent lighting and texture refinement processes.

[0079] After obtaining semantic boundary indication information, to construct cross-frame constraints, pixel correspondence analysis needs to be performed on adjacent scene layout images based on their time indices. The time index represents the chronological order of each scene layout image within the entire shot sequence. For example, frames in the same shot sequence can be numbered sequentially as 1, 2, 3, etc., or explicit timestamps can be used, such as 0.5 seconds, 1.0 seconds, 1.5 seconds, etc. The goal of pixel correspondence analysis is to identify the range of pixels belonging to the same semantic region and temporally equivalent to the same object between two adjacent frames. Examples include pixels belonging to the same character's body surface, the same building facade surface, or the same furniture surface. In practice, changes in character position, lighting direction, and background structure position can be used to aid in this judgment. For example, when a character moves from the left to the center of a scene layout sequence, although the character's outline area shifts slightly in position between adjacent frames, its shape, size, and relationship with other objects remain largely consistent. In this case, pixels within the character's area in the previous frame can be correlated with pixels in the same location and similar shape within the character's area in the next frame. For relatively static structures such as background walls and the ground, spatial depth markers and projection positions can be used to determine which pixels in different frames belong to the same background structure. Changes in lighting direction can be used to explain changes in the position of bright and dark areas. For example, when the light source direction changes slightly, the position of bright spots on the same object's surface will shift slightly, but they can still be considered pixels on the same object based on structural position and surface continuity. After pixel correspondence analysis, the range of pixels belonging to the same semantic region in adjacent frames can be labeled, that is, the set of pixels that can be considered the same character or object in time can be marked, forming cross-frame pixel correspondence reference information. This cross-frame pixel correspondence reference information provides the basic data for subsequent generation of cross-frame pixel self-supervised signals, enabling the system to automatically extract constraint information from inter-frame changes without manual annotation.

[0080] Based on the cross-frame pixel correspondence reference information, it is necessary to organize the brightness variation range, color variation range, and texture detail variation range of corresponding pixels in adjacent frames to generate cross-frame pixel self-supervised signals to maintain semantic consistency between adjacent frames. Brightness can be understood as the lightness or darkness of each pixel; for example, the part of the color value reflecting light intensity can be extracted separately as a brightness index. Color variation range can be understood as the difference in hue, saturation, or color channels. Texture detail variation range can be measured by high-frequency changes in local areas, such as the number of edges and the complexity of fine textures. In practice, the brightness difference, color difference, and local texture contrast difference of each pair of corresponding pixels in adjacent frames can be statistically analyzed, and then it can be judged whether these changes are within a reasonable range within a certain threshold. For example, if a character's coat shows only slight color and brightness changes between two frames, it can be considered that the pixel should maintain a high degree of consistency in color and texture; if the changes are large, it may indicate changes in lighting or inconsistent material representation, which needs to be balanced in subsequent rendering refinement. By analyzing the changes in all corresponding pixels, a self-supervised signal can be formed. This signal does not rely on manual labels but rather on the inherent consistency patterns of adjacent frames, expressing the reasonable range of brightness, color, and texture variations that the same semantic object should exhibit in adjacent frames. The cross-frame pixel self-supervised signal can be represented using a data description structure, such as defining allowed brightness variation ranges, color variation ranges, and texture variation degrees for each corresponding pixel pair or each semantic region. These constraints are then used as input conditions for image thinning processing to guide subsequent lighting thinning and texture thinning processes.

[0081] After generating the cross-frame pixel self-supervised signal, illumination refinement processing needs to be performed on each image based on the semantic boundary indication information and the cross-frame pixel self-supervised signal. The core of illumination refinement processing lies in combining semantic region segmentation and inter-frame consistency constraints to make more precise adjustments to the lighting direction, lighting intensity distribution, and shadow area layout. The semantic boundary indication information is used to limit the cross-region propagation range of lighting changes. That is, when adjusting a semantic region, such as the highlight position of a character's face, it will not unintentionally affect the lighting of the background wall or other unrelated areas; conversely, when adjusting the lighting gradient of a background area, it will not destroy the lighting structure inside the character's outline. The cross-frame pixel self-supervised signal provides a temporal continuity constraint for illumination refinement. When the surface pixels of the same object or the same character in adjacent frames already have a certain brightness distribution and lighting direction in the previous frame, the current frame needs to refer to this trend when adjusting the lighting to make the change in lighting direction present a smooth transition in time. For example, when the script semantic mapping indicates that there is light coming in from the left window and the character is slowly moving forward, the lighting refinement should cause the highlight area on the character's face to shift slightly with the character's position, rather than suddenly jumping to a completely unrelated position. By combining semantic boundary indication information with cross-frame pixel self-supervised signals, an intermediate rendered image containing lighting refinement content can be generated for each image. The intermediate rendered image already meets the semantic and temporal consistency requirements in terms of lighting direction, contrast, and shadow transition.

[0082] After obtaining the intermediate rendered image, texture refinement processing needs to be performed on the semantic regions based on the semantic descriptions of material attributes in the script's semantic mapping relationship. The semantic descriptions of material attributes can come from the textual descriptions of scenes and characters in the script, or from the association results between the previously established visual style reference library and the script's semantic mapping relationship, such as "old wooden floors," "patched cement walls," "a silk dress with a subtle sheen," and "handrails with metallic reflections." These descriptions can be abstracted into material type information, such as wood, metal, fabric, stone, and skin. The specific process of texture refinement processing involves adding texture details corresponding to the material attributes to the intermediate rendered image within the semantic regions determined by the semantic boundary indication information. For example, in the character region, if the script's semantic mapping relationship indicates that the character is wearing a rough fabric coat, then subtle fiber textures and slightly uneven color distribution can be added to the area of ​​the character's coat; in the environment region, if the ground material is a wet cobblestone path, then subtle water stain textures and specular reflection features can be added; in the object region, if the furniture is made of polished wood, then wood grain textures and specular highlights that change with lighting can be added. During texture refinement, cross-frame pixel self-supervised signals must still be referenced. This ensures the continuity and stability of textures between adjacent frames, preventing sudden disappearances or random changes in texture structure over time. For example, in an indoor dialogue scene, the roughness, crack distribution, or decorative patterns of the background wall should not change significantly between adjacent frames. Therefore, the constraints on texture variation range in the cross-frame pixel self-supervised signals can be used to ensure smooth continuity of texture details over time. By adding texture details corresponding to material properties to the character area, environment area, and object area, and adjusting texture changes with reference to the cross-frame pixel self-supervised signals, a rendering result sequence that maintains continuity in both texture representation and temporal sequence can be obtained. This rendering result sequence has completed self-supervised refinement and reconstruction at the levels of lighting distribution and texture detail, and can be directly used as the basis for subsequent shot compositing and playback.

[0083] Furthermore, based on the intermediate rendered image, according to the semantic description of material attributes in the script semantic mapping relationship, texture refinement processing is performed on the semantic region. By adding texture details corresponding to the material attributes to the character region, environment region, and object region respectively, and adjusting the texture changes with reference to the cross-frame pixel self-supervised signal, the texture details are kept coherent between adjacent frames, resulting in a rendering result sequence, including the following implementation steps:

[0084] During the texture refinement process of the intermediate rendered image, it is necessary to first label each semantic region with material attributes based on the semantic description of material attributes in the script semantic mapping relationship. The so-called "material attribute semantic description" refers to the textual information about the material of character clothing, the surface composition of the environment, and the material type of objects obtained through semantic analysis in the film and television script text. Examples include "linen coat," "metal railings," and "wet stone floor." These words have been mapped to specific material types after the preceding semantic parsing steps. This step analyzes the material information related to character areas, environment areas, and object areas in the script semantic mapping relationship, mapping "linen" to a fabric material, "metal railings" to a metal material, and "wet stone floor" to a stone material. These material attributes are then written into the corresponding semantic regions, giving each semantic region a clear material attribute label. Those skilled in the art should understand that material attribute labels are not simple tags, but rather a set of parameters including the roughness of the material surface, the clarity of the texture direction, whether the material has reflective characteristics, and the texture density of the material.

[0085] After obtaining the material attribute tags, suitable texture samples need to be selected from a pre-established texture sample library based on these tags. The "texture sample library" is a set of pre-collected or generated standard texture units. The library should be categorized according to material type, with each material category containing multiple texture samples. For example, fabric materials should provide photos or generated samples of different fabric textures; metal materials should include samples of different features such as brushed metal, sprayed metal, and oxidized metal; and stone materials should include texture samples of rough stone slabs, rock surfaces, and brick walls. When selecting texture samples, it is necessary to determine, based on the material attribute tag data, whether the texture scale of the texture sample matches the size ratio of the character or object in the image, whether the texture direction is consistent with the main direction in the semantic region, and whether the texture density matches the material's performance in a real environment. The texture configuration set obtained by combining the above information constitutes the "texture candidate configuration." The texture candidate configuration is a constraint on the textures that may be used in a semantic region, including the texture type, texture direction, texture scale, and texture density.

[0086] After obtaining the texture candidate configuration, it is necessary to adjust the texture candidate configuration in the time dimension based on the cross-frame pixel self-supervised signal. The so-called "cross-frame pixel self-supervised signal" is the cross-frame consistency reference information calculated in the previous steps based on the range of brightness change, color change, and texture detail change in adjacent frames. In order to maintain the consistency of texture between adjacent frames, the texture candidate configuration of each semantic region needs to be combined with the cross-frame pixel self-supervised signal. By comparing whether the brightness difference of the same semantic region in adjacent frames exceeds the allowable range, whether the color shift is consistent, and whether the texture detail change is within a reasonable range, the texture intensity, texture direction, and texture scale allowed for use in the current frame of that region can be obtained. For example, if the texture direction of the character's clothes in the previous frame is from the upper left to the lower right, and the cross-frame pixel self-supervised signal allows the texture direction offset to be within five degrees in the current frame, then the texture direction in the current frame needs to be limited to this range. As another example, if the texture scale of the stone slab in the previous frame is four millimeters of texture detail, and the character's camera zooms in slightly in the current frame, then the allowable range of texture scale change derived from the cross-frame pixel self-supervised signal can be limited to between five and eight millimeters. The control information generated in this step is called "texture application control information".

[0087] After obtaining the texture application control information, the intermediate rendered image needs to be texture overlayed according to this information. During texture overlay, the extent of each semantic region in the image should first be identified based on the semantic boundary indicator information. Then, within the corresponding region, texture details are overlaid according to the texture direction, scale, and density determined by the texture application control information. When overlaying textures, it is crucial to avoid textures crossing boundaries into other semantic regions; therefore, the boundaries defined by the semantic boundary indicator information must be strictly adhered to. For example, a character's skin texture can only be applied to the character area, and a transition zone needs to be set near the semantic boundary to avoid abrupt texture breaks between the character area and the clothing area. The image sequence after texture overlay constitutes the "initial rendered image sequence," which already includes texture details corresponding to the material attribute markers.

[0088] After generating the initial rendered image sequence, the texture performance in adjacent frames needs to be checked based on the cross-frame pixel self-supervision signal to ensure the continuity of texture details across frames. During the check, if the system detects that the texture intensity, texture direction, or texture scale of a certain semantic region exceeds the allowable range of the cross-frame pixel self-supervision signal, the texture of that region needs to be adjusted according to the texture application control information to bring the amount of texture change back within the allowable range. For example, if the texture of a metal object region in the previous frame is a fine brushed texture, but in the current frame, a slight camera movement causes a shift in texture direction, and if the shift angle exceeds the maximum shift angle allowed by the cross-frame pixel self-supervision signal, the direction of the texture in that region needs to be readjusted back to the allowable range. The adjusted initial rendered image sequence constitutes the final rendered result sequence, ensuring the smoothness and stability of the texture in the time dimension.

[0089] Step S105: Based on the rendering result sequence, determine the camera switching rhythm and focal length change trajectory according to the event rhythm nodes in the script semantic mapping relationship, use the semantic-driven camera motion control method to plan the camera motion path, and generate a playable film and television scene video that is consistent with the script semantic mapping relationship and has natural plot logic.

[0090] In step S105, the rendered result sequence obtained in step S104 needs to be organized into a natural, coherent, and playable video scene that conforms to the rules of cinematic language. To achieve this, the camera transition rhythm of the entire video needs to be determined first based on the event rhythm nodes in the script semantic mapping relationship. Event rhythm nodes refer to key points with structural significance in the plot semantics, such as changes in character actions, emotional shifts, scene changes, or dramatic moments that need to be emphasized. These nodes can be obtained by analyzing action verbs, spatial transition markers, and character emotional descriptions in the script text. For example, phrases such as "the protagonist suddenly stops," "the camera shifts to outside the courtyard," and "the two begin to argue" can all serve as event rhythm nodes.

[0091] After identifying the event rhythm nodes, it's necessary to establish the time interval relationships between these nodes. This interval can be inferred based on the speed of event progression and semantic importance in the script. If the script describes a fast-paced action, such as "running rapidly towards the gate," the camera dwell time should be shortened; if the plot is more leisurely, such as "strolling slowly in the courtyard," the camera dwell time should be extended. To be more specific, the rhythm between events can be expressed using a "speed" label, such as "fast," "medium," or "slow." The system will automatically determine the duration the camera should linger between two events based on the label; for example, the dwell time between fast events is shorter, and the dwell time between slow events is longer, thus creating a natural camera transition rhythm.

[0092] After determining the camera transition rhythm, it's necessary to further plan the focal length change trajectory. The focal length change trajectory refers to the changes in shot size presented during the narrative, including the visual effects corresponding to wide shots, medium shots, and close-ups. Focal length changes should be automatically determined based on the character's behavior, emotional intensity, and the focus of the shot within the script's semantic mapping. For example, if the script indicates "the protagonist looks up and gazes into the distance," the camera should gradually move closer to the character's face, allowing the audience to see facial details; if the script indicates "showing the entire courtyard layout," the camera should slowly move back, allowing more background to appear in the frame. When calculating focal length changes, a smooth, gradual transition can be used to avoid sudden jumps. For example, if it's necessary to gradually zoom from a distant shot to a close-up of the character, the system will gradually change the focal length over a period of time, rather than changing it instantly, giving the audience a sense of smooth camera movement.

[0093] After determining the focal length change, the camera movement path needs to be planned. The camera movement path refers to the camera's trajectory in virtual space, including factors such as direction, speed, framing angle, and camera height. The camera movement path must be consistent with the script's semantics; that is, it needs to be automatically generated based on the character's actions and the scene's spatial structure in the script. For example, if the script's semantics describe "the camera follows the protagonist into the main hall," the camera's trajectory should be consistent with the character's direction of movement and maintain a stable forward / backward distance to create a natural tracking effect. If the script describes "the scene switches to a bird's-eye view of the courtyard," the camera path should gradually rise and move backward appropriately to fully display the courtyard's structure. When planning the camera path, it's crucial to ensure smooth camera movement without abrupt jumps. Therefore, a gradual approach is usually used, such as breaking the camera movement into several small steps, each a continuation of the previous step, to maintain a continuous trajectory.

[0094] Emotional expression must also be considered in camera path planning. If the characters in the script are tense or the plot is intense, the camera movement can be appropriately accelerated, and even slight swaying can be added to the path to simulate the texture of handheld shooting, making the image more immersive. If the plot is calm and soothing, the camera movement speed should be slowed down and kept stable to avoid strong senses of movement. This ensures that the visual language is consistent with the emotional tone of the plot.

[0095] After completing the shot transition rhythm, focal length change trajectory, and camera movement path, the rendered images generated in step S104 need to be organized in chronological order, and shot transitions applied at event rhythm nodes. For example, when switching between two shots, direct switching, fade-in / fade-out, or transition effects can be used. Direct switching is suitable for fast-paced scenes, fade-in / fade-out for gentle scenes, and transition effects for scenes with significant changes, such as entering a flashback scene. The specific switching method can be automatically selected based on the script's mood type. For example, if the script describes "the scene suddenly changes," a direct switch should be used; if the script describes "time slowly passing," a fade-in / fade-out method should be used.

[0096] When compositing image sequences, the rendered images can be arranged into consecutive frames according to a standard video frame rate (e.g., 24 or 30 frames per second) to ensure smooth playback. The camera position, focal length, and angle of each frame are automatically inserted by the aforementioned trajectory control module, ensuring that there are no stutters, jumps, or unnatural changes between video frames.

[0097] For example, taking the aforementioned "Daiyu's First Visit to the Jia Mansion" scene, the script semantics describe Daiyu slowly entering the courtyard from the mansion gate, then looking up at the main hall. The system identifies the event rhythm nodes as "entering the courtyard" and "looking up at the main hall" based on the script. During the "entering the courtyard" phase, the camera moves slowly forward in a stable tracking manner, maintaining a medium shot focus, showing Daiyu's figure and a partial view of the courtyard environment. Before the "looking up at the main hall" node, the camera begins to slow down and gradually stabilizes, then slowly zooms in, smoothly transitioning from a medium shot to a close-up, allowing the audience to see the character's subtle expressions. When the script semantic mapping relationship indicates "scene switch to courtyard panorama," the camera quickly rises and moves back, presenting the entire Jia Mansion courtyard. The entire camera movement, switching, and focus changes are all automatically completed by semantic drive, ensuring that the final video conforms to both the plot content and the expressive methods of traditional film and television cinematography.

[0098] A second embodiment of this application provides an electronic device, the electronic device comprising:

[0099] processor;

[0100] The memory is used to store a program, which, when read and executed by the processor, executes the intelligent synthesis method for film and television scenes based on generative multimodal script semantic mapping provided in the first embodiment of this application.

[0101] The third embodiment of this application provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it executes a film and television scene intelligent synthesis method based on generative multimodal script semantic mapping provided in the first embodiment of this application.

[0102] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A method for intelligent synthesis of film and television scenes based on generative multimodal script semantic mapping, characterized in that, include: Semantic unit segmentation and dependency structure analysis are performed on the acquired film and television script texts, and script semantic mapping relationships are established based on character entities, plot events and spatial location relationships; A generative multimodal semantic alignment method is used to train cross-modal feature mapping on the semantic mapping relationship of the script, so that the semantic units of the film and television script text are associated with visual style, composition relationship and lighting elements, and an initial scene layout map matching the semantic mapping relationship of the script is generated. The initial scene layout is optimized for spatiotemporal consistency. Based on the temporal order and spatial depth information in the script semantic mapping relationship, a synchronization constraint relationship between the character action sequence and the camera view is established. The consistency of lighting direction, character posture and background continuity between adjacent shots is maintained through a dynamic feature propagation mechanism, and a scene layout sequence after spatiotemporal consistency optimization is obtained. A generative self-supervised rendering method is used to reconstruct the scene layout map sequence. Based on the semantic boundary preservation constraint of script semantic mapping relationship and cross-frame pixel self-supervised signal, the generated image is refined to generate a rendering result sequence with realistic lighting reflection characteristics and texture consistency. Based on the sequence of rendering results, the camera switching rhythm and focal length change trajectory are determined according to the event rhythm nodes in the script semantic mapping relationship. A semantically driven camera motion control method is used to plan the camera motion path and generate playable film and television scene videos that are consistent with the script semantic mapping relationship and have natural plot logic.

2. The intelligent synthesis method for film and television scenes based on generative multimodal script semantic mapping according to claim 1, characterized in that, The method employs a generative multimodal semantic alignment approach to train cross-modal feature mapping on script semantic mapping relationships. This establishes a one-to-one correspondence between semantic units of the film and television script text and visual style, compositional relationships, and lighting elements, and generates an initial scene layout diagram that matches the script semantic mapping relationships. This includes: The semantic elements of each character entity, plot event and spatial position relationship in the script semantic mapping relationship are refined and expanded into a set of semantic elements that include semantic attributes, scene function and meaning and visual presentation intention. This enables each set of semantic elements to fully express the narrative function and style tendency that the semantic unit in the script semantic mapping relationship should present when visualized, and the set of semantic elements is used as the input semantic information for cross-modal mapping training. Obtain a set of visual reference elements corresponding to the set of semantic elements. The set of visual reference elements is extracted from pre-established visual style reference information, composition feature reference information and lighting feature reference information based on the character identity, plot structure and spatial environment in the script semantic mapping relationship. This ensures that each set of semantic elements corresponds to a set of visual reference elements and that the set of visual reference elements can express the character's appearance style, scene composition method and lighting direction features. Generative semantic correspondence inference is performed between the semantic element set and the visual reference element set. Through layer-by-layer correlation analysis of semantic description content, narrative function and visual presentation intention, the expression mode of each semantic element set in visual style, composition mode and lighting direction is inferred, and cross-modal semantic correspondence results containing visual style information, composition relationship information and lighting direction information are generated, so that the cross-modal semantic correspondence results fully reflect the visualization characteristics of script semantic mapping relationship. Based on the cross-modal semantic correspondence results, multiple rounds of consistency checks are performed. The narrative logic required by the text semantics is bidirectionally checked against the style features, spatial composition, and lighting conditions contained in the visual reference element set. When deviations are found in the narrative logic, visual style, or lighting direction, the style information, composition information, and lighting feature information in the cross-modal semantic correspondence results are readjusted to ensure that the cross-modal semantic correspondence results maintain consistency among the narrative logic, visual style, and lighting conditions. An initial scene layout diagram is generated based on the cross-modal semantic correspondence results after consistency verification. The character positions, character poses, scene composition framework, lighting direction, and the arrangement of the main objects in the scene are arranged according to the structural order of the script semantic mapping relationship, so that the initial scene layout diagram can fully express the visual structure required by the script semantic mapping relationship.

3. The intelligent synthesis method for film and television scenes based on generative multimodal script semantic mapping according to claim 2, characterized in that, The process involves performing generative semantic correspondence inference between the semantic element set and the visual reference element set. Through layer-by-layer correlation analysis of semantic description content, narrative function, and visual presentation intention, it infers the expression of each semantic element set in terms of visual style, composition, and lighting direction. This generates cross-modal semantic correspondence results containing visual style information, compositional relationship information, and lighting direction information, ensuring that the cross-modal semantic correspondence results fully reflect the visual characteristics of the script's semantic mapping relationship. This includes: The semantic description content in the semantic element set is processed by narrative context expansion, which extracts the scene atmosphere, characters' psychological state and narrative speed implied in the semantic description content, forming semantic context information that can express the narrative atmosphere, characters' psychology and narrative rhythm. Based on semantic context information, the visual style information, compositional relationship information and lighting direction information in the visual reference element set are filtered for narrative association, so that the visual information after narrative association filtering can reflect the intensity of atmosphere, the psychological tension of characters and the characteristics of narrative rhythm changes expressed in the semantic context information, and the visual information after narrative association filtering is used as candidate information for visual expression. Based on the importance of different visual features in the visual expression candidate information to the narrative expression, the visual expression candidate information is ranked by expression weight, so that the visual style, composition relationship and lighting direction with the highest expression weight express the core narrative intention in the semantic context information, and the visual information ranked by expression weight is used as the input for visual fusion processing. Based on the visual information sorted by expression weight, the visual information is then fused layer by layer according to the order of influence of visual style information on compositional relationship information and the order of influence of compositional relationship information on lighting direction information. This process allows visual style information to determine the overall style of the image, compositional relationship information to determine the spatial layout of the image, and lighting direction information to determine the lighting level and visual focus position of the image, thus forming a preliminary cross-modal visual expression result. The preliminary cross-modal visual expression results are analyzed for narrative consistency by retrospectively analyzing the role relationships, event logic, and spatial order in the script semantic mapping relationship. This ensures that the cross-modal visual expression results are consistent across narrative logic, spatial expression, and visual presentation, forming the final cross-modal semantic correspondence result.

4. The intelligent synthesis method for film and television scenes based on generative multimodal script semantic mapping according to claim 1, characterized in that, The initial scene layout map undergoes spatiotemporal consistency optimization. Based on the temporal order and spatial depth information in the script semantic mapping relationship, a synchronization constraint relationship between the character action sequence and the camera perspective is established. A dynamic feature propagation mechanism is used to maintain consistency in lighting direction, character posture, and background continuity between adjacent shots, resulting in a scene layout map sequence optimized for spatiotemporal consistency, including: In the initial scene layout diagram, based on the time order of each plot event in the script semantic mapping relationship and the foreground, midground and background annotation results of the characters in spatial depth, a time index and spatial depth mark are added to each initial scene layout diagram to form a sequence of scene layout diagrams arranged in chronological order and marked with spatial depth. In the scene layout sequence with time index and spatial depth marker, based on the continuous changes in character position and posture, the translation direction, displacement amplitude and posture changes of each character between adjacent scene layouts are organized to form a character action sequence describing the character's movement path and posture transition in the time dimension. At the same time, based on the camera's viewfinder position, view direction and spatial depth marker in the scene layout, the changes in the camera's viewpoint in the time dimension are extracted to form a camera viewpoint change sequence that corresponds one-to-one with the character action sequence. Based on the character action sequence and the camera view change sequence, synchronization constraints are established for the scene layout diagram between each adjacent time index. The synchronization constraints include at least the expected position range of the character at the current time index, the allowable deviation range of the character's posture change, the allowable offset range of the camera view relative to the previous time index, and the allowable displacement range of the foreground, midground, and background in the picture calculated based on the spatial depth marker, thereby forming a set of spatiotemporal constraint parameters for constraining character actions, camera view, and scene hierarchy. Based on the set of spatiotemporal constraint parameters, dynamic feature propagation processing is performed between adjacent scene layout diagrams. The lighting direction information, character posture information, and background object spatial distribution information that have satisfied the spatiotemporal constraints under the previous time index are used as reference information. The reference information is passed along the time index to the next frame or multiple frames. During the transmission process, the lighting direction, character posture, and background object position that do not satisfy the set of spatiotemporal constraint parameters are smoothly adjusted so that the lighting direction changes continuously in time, the character posture transitions naturally in time, and the background objects maintain reasonable spatial continuity in time, resulting in an intermediate scene layout diagram sequence after dynamic feature propagation adjustment. Based on the intermediate scene layout sequence, and combined with the cross-shot character relationships and event logic in the script semantic mapping relationship, a global consistency backtracking analysis is performed on the entire intermediate scene layout sequence. During the backtracking analysis, the lighting direction, character posture, or background structure that are inconsistent with the overall narrative logic or overall spatial layout in the local dynamic feature propagation processing are readjusted again, so that the entire scene layout sequence forms a unified spatiotemporal continuity in terms of time sequence, spatial depth, character action trajectory, and camera perspective changes, thereby obtaining a scene layout sequence after spatiotemporal consistency optimization processing.

5. The intelligent synthesis method for film and television scenes based on generative multimodal script semantic mapping according to claim 1, characterized in that, The method employs a generative self-supervised rendering approach to reconstruct the scene layout sequence. Based on semantic boundary preservation constraints of script semantic mapping relationships and cross-frame pixel self-supervised signals, the generated images are refined to produce a sequence of rendering results with realistic lighting reflection features and texture consistency. This includes: In the scene layout diagram sequence, the semantic annotations of character outlines, scene object ranges and structural boundaries are parsed according to the script semantic mapping relationship, and semantic boundary indication information that can distinguish different semantic regions is generated for each scene layout diagram. After obtaining semantic boundary indication information, the pixel correspondence of adjacent scene layout maps is analyzed according to their time index. By comparing changes in character position, changes in lighting direction, and background structure position, the range of pixels belonging to the same semantic region in adjacent frames is calibrated to form cross-frame pixel correspondence reference information. The cross-frame pixel correspondence reference information is used as the input basis for cross-frame pixel self-supervised signal. Based on the cross-frame pixel correspondence reference information, the brightness change range, color change range, and texture detail change range of corresponding pixels in adjacent frames are sorted out to generate cross-frame pixel self-supervised signals to maintain semantic consistency between adjacent frames, and the cross-frame pixel self-supervised signals are used as input conditions for image thinning processing. Based on semantic boundary indication information and cross-frame pixel self-supervised signals, illumination thinning is performed on each image. During the illumination thinning process, semantic boundary indication information is used to limit the cross-regional propagation range of illumination changes, and the illumination direction changes are smoothly adjusted according to the cross-frame pixel self-supervised signals to generate an intermediate rendered image containing illumination thinning content. Based on the intermediate rendered image, according to the semantic description of material properties in the script semantic mapping relationship, texture refinement processing is performed on the semantic region. By adding texture details corresponding to the material properties to the character region, environment region and object region respectively, and adjusting the texture changes with reference to the cross-frame pixel self-supervised signal, the texture details are kept coherent between adjacent frames, and the rendering result sequence is obtained.

6. The intelligent synthesis method for film and television scenes based on generative multimodal script semantic mapping according to claim 5, characterized in that, Based on the intermediate rendered image, and according to the semantic description of material attributes in the script semantic mapping relationship, texture refinement processing is performed on the semantic region. This involves adding texture details corresponding to the material attributes to the character region, environment region, and object region, respectively, and adjusting texture changes with reference to cross-frame pixel self-supervised signals. This ensures that texture details remain consistent between adjacent frames, resulting in a rendering result sequence, including: Based on the semantic description of material attributes in the script semantic mapping relationship, material attribute labels are applied to the character area, environment area and object area in the intermediate rendered image, so that each semantic area has a corresponding material attribute label. Based on the material attribute markers, texture samples matching the material attribute markers are selected from a pre-established texture sample library for each semantic region, and texture candidate configurations are formed according to the texture scale, texture direction and texture density of the texture samples. Based on the cross-frame pixel self-supervised signal, the texture candidate configuration is adjusted in the time dimension. According to the brightness change range, color change range, and texture detail change range of adjacent frames, the texture intensity, texture direction, and texture scale that can be used in each semantic region in the current frame are determined to form texture application control information. Based on the texture application control information, texture details consistent with the material attribute markings are superimposed on the character area, environment area and object area in the intermediate rendering image, and the spatial range of texture details within each semantic area is limited according to the semantic boundary indication information, forming an initial rendering image sequence containing texture refinement content. Based on the cross-frame pixel self-supervised signal, the initial rendered image sequence is checked for cross-frame texture consistency. For semantic regions where the detected texture intensity, texture direction, or texture scale exceeds the allowable range of variation, texture refinement is performed according to the texture application control information. The initial rendered image sequence is then updated and output as the rendered result sequence.

Images