Scientific and technology movie and television interactive virtual scene generation method based on fractal generation large model

Abstract

The invention relates to a technology movie and television interactive virtual scene generation method based on a fractal generation large model, and belongs to the field of virtual reality content generation. The method comprises the following steps: firstly, extracting space occupation and motion states of actors, structured semantic information of a script and a real-time creation instruction of a director through parallel processing; the feature vectors are mapped to a unified semantic space and fused, comprehensive state representation is generated, and then a virtual scene structure blueprint is constructed; and finally, taking the generated blueprint as a spatialization condition and a global constraint, driving a fractal autoregression generation unit to perform recursive iteration, and synthesizing scene content details conforming to blueprint design on a pixel level from coarse to fine. According to the method, the advantages of the vertical diffusion model in structure planning and the potential of the fractal autoregression model in detail generation are combined, virtual scene synthesis which is synchronous with script logic, director intention and actor performance and is coherent in content and rich in details is achieved, and the automation and intelligence level of virtual film content generation is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of virtual reality content generation and intelligent visual rendering, and relates to a science and technology film and television interactive virtual scene generation method based on a fractal generated large model. BACKGROUND

[0002] With the rapid development of virtual reality and artificial intelligence technologies, virtual scene generation as an important content creation method has also seen increasing demand. In recent years, virtual scene generation technology has been developing in the direction of intelligence, automation, strong realism, and enhanced interactivity. Although the overall trend is positive, there is still a significant gap between virtual scene generation methods and technology levels and the requirements of modernization of film and television production.

[0003] One of the representative problems is that high-quality virtual scene generation still highly depends on manual modeling and manual adjustment. In addition, in professional film and television production, the labor required for scene construction and adjustment accounts for more than 50% of the total production process, and is one of the tasks with the longest time consumption, the largest workload, and the highest creation difficulty. Therefore, the demand for innovative virtual scene generation methods in the film and television industry is increasingly urgent, and the implementation of automated and intelligent scene generation has become an inevitable choice for the sustainable development of content production.

[0004] In recent years, with the progress of large language models and computer vision technologies, virtual scene generation methods based on deep learning have made great progress. Through deep learning technologies such as generative adversarial networks (GAN) and diffusion models, realistic image content can be generated, laying a foundation for intelligent scene construction. For scene generation control, text condition guidance, reference image style transfer, and Layout guidance are currently used to obtain scene frameworks. However, film and television scene content is complex and involves the unified processing of director intent, script semantics, and actor performance. Traditional single-input guided scene generation methods often face problems such as poor semantic consistency, low performance stability, and insufficient visual fidelity. At the same time, existing methods lack effective capture and fusion mechanisms for real-time performance characteristics of actors, making it difficult to achieve performance-driven dynamic scene generation and unable to meet the high requirements of film and television creation for scene and actor performance synchronization.

[0005] In addition, traditional scene generation methods usually generate the entire scene as a whole, lack a hierarchical generation strategy, and are difficult to balance the generation quality of global structure and local details. Some emerging methods attempt to use a progressive generation approach from coarse to fine, but most lack a unified theoretical framework and fail to fully utilize the inherent fractal characteristics of scenes for efficient generation. Therefore, there is an urgent need to explore a virtual scene generation method that can organically integrate multi-source heterogeneous inputs, hierarchically process scene structure and details, and support actor-driven virtual scene generation. SUMMARY

[0006] Therefore, the present application aims to provide a technology and film interactive virtual scene generation method based on fractal generation large model, which can generate high-fidelity and interactive virtual scenes in virtual production according to scripts, real-time intentions of directors and real-time performances of actors. The method uses real-time video streams of actors, film script texts and creative intentions of directors as inputs, provides solutions for virtual production through feature extraction technology, multi-modal fusion technology and generation model, so that it can realize real-time synchronization, content coherence and detailed dynamic virtual scene synthesis, thereby improving the automation and intelligence level of virtual production and the final visual effect quality.

[0007] To achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0008] A technology and film interactive virtual scene generation method based on fractal generation large model, characterized in that the method comprises:

[0009] Firstly, multi-source information extraction is performed, including real-time video streams of actors, film script texts and creative intentions of directors; dynamic feature vectors are extracted from the real-time video streams of actors through a dynamic feature capture method based on mask segmentation; the film script texts are converted into semantic graph feature vectors through a semantic knowledge graph construction algorithm; director intention graph feature vectors are extracted from the creative intentions of directors through a semantic intention analysis method;

[0010] Then, the dynamic feature vectors, semantic graph feature vectors and director intention graph feature vectors are mapped to a unified semantic embedding space, cross-modal semantic alignment and fusion are performed, and a unified fusion state representation is generated; the unified fusion state representation is taken as a conditional input of a vertical diffusion model, a global scene framework representation is generated according to script semantics and director intentions, and an interactive area and constraint are calibrated in the global scene framework representation in combination with actor dynamic features, and finally a virtual scene structure blueprint is output;

[0011] Finally, the virtual scene structure blueprint is taken as a spatialization condition and global constraint, and virtual content synthesis is performed through a fractal self-recurrent generation unit of a recursive hierarchical structure.

[0012] Further, the dynamic feature capture method based on mask segmentation extracts dynamic feature vectors from the real-time video streams of actors, which comprises: performing continuous frame extraction on the real-time video streams of actors and separating foreground and background by using a deep learning model to generate an accurate actor mask, and then capturing motion change features between continuous frames through time series analysis, and applying a feature vectorization algorithm to convert performance dynamics into a dynamic feature vector F, which is a function of the mask feature M t and the motion change gradient The weighted fusion of the two results is obtained as follows:

[0013]

[0014] wherein, α t and β t are the mask feature weight and the action change gradient weight at time t respectively, and T is the total length of the time sequence;

[0015] The method for converting the script text into a semantic graph feature vector comprises the following steps: performing word segmentation, syntax analysis and entity recognition on the script text to extract core semantic elements; using semantic role labeling technology to identify action subjects, receptors and environment descriptions in the script to form a preliminary semantic relationship network; applying a knowledge reasoning enhancement algorithm to supplement implicit semantic associations based on a pre-trained language model and a domain knowledge base to generate a script semantic knowledge graph; and finally encoding the knowledge graph through a graph neural network to generate a script semantic graph feature vector S:

[0016]

[0017] wherein, v i represents the feature vector of the i-th node in the graph, N is the total number of graph nodes, w i is the semantic importance weight of the corresponding node, R represents the transformation matrix of the node feature in the semantic graph relationship space, b is a global bias feature vector, and σ(·) is a nonlinear activation function;

[0018] The method for extracting a director intention graph feature vector from the director creation intention comprises the following steps: performing natural language processing on the director's voice or text instructions to distinguish different categories of creation requirements; applying a multi-layer attention mechanism to assign different weights to different categories of creation requirements; converting the processing result into a structured director intention feature vector through a semantic vectorization algorithm; and finally constructing a director intention graph to represent the hierarchical and associated relationships between different creation intentions through a graph structure to generate a director intention graph feature vector D:

[0019] D = γ · tanh (β c · I c + β s · I s ) (3)

[0020] wherein, I c represents a scene content intention vector, I s represents a scene style intention vector, β c and β s represent weight coefficients; γ is a feature scaling coefficient, and tanh(·) is a hyperbolic tangent activation function.

[0021] Further, the dynamic feature vector, semantic graph feature vector and director intention graph feature vector are mapped to a unified semantic embedding space for cross-modal semantic alignment and fusion to generate a unified fusion state representation:

[0022] H = W f σ(M1F + M2S + M3Dθ

[0023] where M1, M2 and M3 are transformation matrices of visual, text and director intention features respectively, W f is a fusion mapping matrix, and σ(·) is a nonlinear activation function.

[0024] Further, the unified fusion state representation H is taken as a conditional input of the vertical diffusion model to guide the denoising process of the vertical diffusion model; wherein the unified fusion state representation H is semantically deconstructed and decomposed into a script narrative framework feature vector H s and a director artistic intention feature vector H d :

[0025]

[0026] where W s i and are weight matrices of the i-th narrative feature extraction layer and the j-th director intention extraction layer respectively, k and m represent the total number of narrative feature extraction layers and director intention extraction layers respectively, α i and β j are feature weight coefficients, and MLP(·) represents a multi-layer perception network; based on the extracted H s and H d , the denoising process of the vertical diffusion model is guided, and the vertical diffusion model generates a global scene framework representation in the denoising iteration process, and the denoising process of the vertical diffusion model is represented as:

[0027]

[0028] where x t represents a noise state vector at time t, x t-1 represents a noise state vector at time t-1; α t and are predefined noise scheduling parameters, ∈ θ represents a noise prediction network with parameter θ, σ is a noise scale, and z is a standard normal distribution sampling vector.

[0029] Further, in combination with the global scene framework representation, and the actor dynamic mask feature and the accurate motion vector information in the unified fusion state representation H, a virtual scene structure blueprint is constructed: by mapping the real-time spatial occupancy information of the actor to the global scene framework representation, and dynamically labeling the corresponding region in the global scene framework representation with local constraints and interaction attributes according to the actor's posture, motion and potential interaction intention, the virtual scene structure blueprint is generated, as shown in the following formula:

[0030]

[0031] In the formula, B1 is a spatial layout blueprint matrix, B2 is an interaction relationship blueprint matrix, and B is a virtual scene structure blueprint matrix; K is a global scene framework representation, F a is an actor dynamic mask feature matrix, represents the spatial gradient of the actor dynamic mask feature matrix, V is an actor accurate motion vector matrix, α is an actor dynamic mask feature weight coefficient, γ is a transposed mapping fusion coefficient, δ is a gradient response coefficient, ω1 is a spatial layout weight coefficient, ω2 is an interaction relationship weight coefficient, and ω3 is a multi-mode collaborative coefficient, represents matrix direct sum operation, and ⊙ represents element point multiplication.

[0032] Further, taking the virtual scene structure blueprint as a spatialization condition and global constraint, virtual content synthesis is performed through a recursive hierarchical fractal self-regressive generation unit; in the fractal self-regressive generation unit, a top-level generation unit first generates a scene coarse-grained representation according to the macro layout information, key object settings and actor spatial constraints provided by the virtual scene structure blueprint, and then expands in a top-down and recursive nested manner, and each level of the generation unit receives the output of the last level and the guide information of the corresponding spatial region extracted from the virtual scene structure blueprint as a condition to generate fine-grained content details, and finally outputs a virtual scene rendering picture.

[0033] Further, the top-level generation unit first generates a scene coarse-grained representation according to the macro layout information, key object settings and actor spatial constraints provided by the virtual scene structure blueprint, which includes: first converting the virtual scene structure blueprint into a comprehensive guide matrix:

[0034] G = λ1·G1+ λ o ·G o + λ c ·G c

[0035] In the formula, G is a comprehensive guide matrix, G1, G o and G care a layout guide matrix, an object position matrix and a space constraint matrix extracted from a virtual scene structure blueprint matrix B, respectively, λ o and λ c are weight coefficients;

[0036] The top layer generation unit receives a comprehensive guide matrix G as a conditional input, and constructs a coarse-grained representation of the scene:

[0037] X = T(G, θ t )·L(B)

[0038] In the formula, X is a coarse-grained representation of the scene, T is a top layer generation unit, θ t is a parameter set of T, and L(·) is a blueprint adaptation function for extracting structure features related to the current level from a virtual scene structure blueprint matrix B.

[0039] Then, based on the coarse-grained representation X of the scene and the virtual scene structure blueprint matrix B, a regional sub-task sequence is constructed to decompose the overall generation task into multiple local regional detail generation tasks; the subsequent levels further generate detailed content through iteration:

[0040] X q+1 = g q+1 (X q ,C q+1 ,M q+1 ; Θ q+1 ), q = 0, 1,..., Q-1

[0041] In the formula, Q is a preset number of iterations, X q+1 is the generated content of the q+1th layer generation unit, g q+1 (·) is the q+1th layer generation unit, C q+1 is the refinement condition of the q+1th layer generation unit, Θ q+1 is a parameter set of the q+1th layer generation unit, and M q+1 is the generation task of the q+1th layer generation unit.

[0042] According to the output X Q of the last layer generation unit, virtual content synthesis is performed, the detail condition C f is extracted from the virtual scene structure blueprint matrix B through the blueprint adaptation function; and X Q and the detail condition C f are input into the conversion function G final (·) to calculate and output a rendering representation P:

[0043] P = G final (X Q ,C f ; Ψ)

[0044] Where P represents the final output rendering representation, that is, the virtual scene rendering screen, and Ψ is the conversion function G final (·) parameter set.

[0045] The beneficial effects of the present invention are:

[0046] 1) This invention uses a two-stage generation architecture, combining the advantages of the vertical diffusion model in structural planning with the potential of the fractal autoregressive generation model in detail generation. It achieves the synthesis of dynamic virtual scenes that are synchronized with the script logic, director's intention and actor performance in real time, with coherent content and rich details. This significantly improves the automation and intelligence level of content generation in virtual production and the final visual effect quality.

[0047] 2) The fractal autoregressive generation method proposed in this invention, which is guided by a virtual scene structure blueprint, uses the blueprint generated in the first stage as a precise, spatialized conditional guide and global constraint, driving the fractal autoregressive generation unit to perform recursive iterations, accurately synthesizing scene content details that conform to the blueprint design from coarse to fine, at the pixel level, greatly improving the consistency and visual quality of the generated content.

[0048] 3) The present invention realizes the effective fusion and precise control of multi-source information. Through parallel processing modules, the dynamic characteristics of actors, the semantic information of the script and the creative intention of the director are extracted respectively, and then aligned and integrated through multi-source information shared space mapping, ensuring that the generated virtual scene can accurately respond to the spatial constraints of the actors' performances, conform to the narrative logic of the script and the artistic expression requirements of the director, providing an efficient and flexible virtual scene generation solution for scientific and technological film and television production.

[0049] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description.

[0050] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0052] Figure 1A flowchart of a method for generating a sci-tech film interactive virtual scene based on a fractal generated large model is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0053] The advantages and effects of the present application can be easily understood by those skilled in the art from the disclosure of the embodiments. The present application can also be implemented or applied in different specific embodiments, and various modifications or changes can be made to the details based on different views and applications without departing from the spirit of the present application. It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0054] As Figure 1 shown, a method for generating a sci-tech film interactive virtual scene based on a fractal generated large model is provided for an embodiment of the present application, and the method is as follows:

[0055] S1, extracting multi-source information, including actor real-time video stream, film script text and director creation intention. Among them, the actor real-time video stream extracts the action features in the performance process through a dynamic feature capture method based on mask segmentation, and converts them into mask feature vectors describing the performance details; the film script text is structured into a script semantic graph feature vector by a relationship enhanced semantic knowledge graph construction algorithm, realizing explicit modeling of semantic association; the director creation intention extracts the semantic features in the director creation intention through a language command analysis method based on semantic intention analysis, and converts them into a director intention graph feature vector, providing a complete data basis for subsequent semantic control and feature fusion of scene generation.

[0056] Specifically, for the actor real-time video stream, first, continuous frames are extracted and a deep learning model is used to realize foreground and background separation, generating an accurate actor mask; then, the action change features between continuous frames are captured through time series analysis, and a feature vectorization algorithm is applied to convert the performance dynamics into a descriptive feature vector, i.e. a dynamic feature vector F. The dynamic feature vector F can be obtained by weighted fusion of the mask feature M t and the action change gradient .

[0057]

[0058] Among them, α t and β t are the mask feature weight and the action change gradient weight at time t, respectively, and T is the total length of the time series.

[0059] For film and television script text, a relationship-enhanced semantic knowledge graph construction method is used to extract structured semantic information. First, the script text is segmented, syntactically parsed, and entity recognized to extract core semantic elements such as scene description, character dialogue, and plot development; then, semantic role labeling technology is used to identify action subjects, recipients, and environment descriptions in the script to form a preliminary semantic relationship network; then, a knowledge reasoning enhancement algorithm is applied to supplement the implicit semantic associations based on pre-trained language models and domain knowledge bases to generate a complete script semantic knowledge graph; finally, the overall knowledge graph is encoded by a graph neural network to generate a script semantic graph feature vector S:

[0060]

[0061] where v i represents the feature vector of the i-th node in the graph, N is the total number of graph nodes, w i is the semantic importance weight of the corresponding node, R represents the transformation matrix of the node features in the semantic graph relationship space, which is used to capture the structured semantic association between nodes, b is the global bias feature vector, and σ(·) is a non-linear activation function used to enhance the fitting ability of feature expression. The script semantic graph feature vector S encodes multi-dimensional semantic information of plot logic, scene description, and character relationships, providing an overall semantic framework and content constraints for subsequent scene generation.

[0062] For director's creative intent, a language command processing method based on semantic intent analysis is used. First, the director's voice or text instructions are processed by natural language processing, including semantic segmentation, keyword extraction, and intent recognition, to distinguish between different categories of creative requirements such as scene construction, character behavior, and shot language; then, a multi-layer attention mechanism is applied to assign different weights to different categories of creative requirements to highlight core creative elements and artistic expression needs; then, a semantic vectorization algorithm is used to convert the processing results into structured director intent feature vectors and establish mapping relationships with visual expression, spatial layout, and narrative rhythm; finally, a director intent graph is constructed to represent the hierarchical and associative relationships between different creative intents, generating a director intent graph feature vector D:

[0063] D=γ·tanh(β c ·I c +β s ·I s ) (3)

[0064] where I c represents the scene content intent vector, I s represents the scene style intent vector, β c and β sare the weight coefficients of scene content and scene style respectively. γ is the feature scaling coefficient, and tanh(·) is the hyperbolic tangent activation function to improve the nonlinear expression ability of feature combination. The director intention graph feature vector D realizes the guidance of the core creative intention in the virtual scene generation process.

[0065] S2, first, based on the alignment fusion method of multi-source information sharing space mapping, the dynamic feature vector F, the script semantic graph feature vector S and the director intention graph feature vector D are projected into a unified high-dimensional semantic embedding space to complete the semantic alignment and deep fusion of cross-modal feature vectors, and a unified fusion state representation containing comprehensive context and real-time dynamic information is generated; then, a virtual scene structure blueprint building method based on a sci-fi vertical diffusion model is used, which uses a diffusion model that has been deeply modified and fine-tuned by sci-fi field data and rules as the core, and takes the unified fusion state representation as the mandatory condition input. According to the script semantics and director intention in the unified fusion state representation, the sci-fi vertical diffusion model drives the generation of scene macro structure, spatial layout, key object setting and overall light and shadow atmosphere containing global narrative logic and visual style requirements, and combines the real-time space and form data provided by the dynamic feature vector F to accurately mark the key areas affected by the actors in the scene structure blueprint and impose structural constraints for defining interactive spaces or setting reflection and occlusion properties; finally, a structured "virtual scene structure blueprint" is output as a key intermediate representation that provides multi-level accurate guidance information for the next stage, ensuring that the macro setting is followed and can accurately respond to the real-time presence of actors.

[0066] S21, based on the alignment fusion method of multi-source information sharing space mapping, the dynamic feature vector F, the script semantic graph feature vector S and the director intention graph feature vector D are projected into a unified high-dimensional semantic embedding space to obtain a unified fusion state representation H:

[0067] H = W f · σ(M1F + M2S + M3D) (4)

[0068] where M1, M2 and M3 are transformation matrices of visual, text and director intention features respectively, W f is a fusion mapping matrix for unified mapping to a shared semantic space, and σ(·) is a nonlinear activation function. The final unified fusion state representation H integrates the projection and semantic complementarity of the three types of heterogeneous information in the unified semantic space, providing a shared feature basis for subsequent layered scene generation.

[0069] S22, generating a high-quality virtual scene structure blueprint based on the sci-fi vertical diffusion model. This blueprint is a multi-level structured representation that defines each component of the scene from macro to micro: the top layer contains the overall layout and spatial structure of the scene; the middle layer defines the position, scale, and mutual relationship of key objects; and the detail layer contains elements such as material properties, lighting parameters, and environmental atmosphere. In this process, the semantic conditions from the script mainly affect the overall narrative structure and atmosphere of the scene, while the dynamic conditions from the actors precisely define the actor's placement area, interaction space, and possible motion trajectory in the scene, ensuring that the generated scene structure can adapt to the actor's real-time performance.

[0070] 1) First, a vertical diffusion model for the sci-fi film field is constructed. This model is pre-trained on a large amount of sci-fi film material and then fine-tuned on a specific domain dataset to form a precise understanding and generation ability of sci-fi scene structure elements. Unlike general diffusion models, this vertical diffusion model is optimized for the visual language, aesthetic style, and narrative structure specific to the sci-fi film field, allowing it to more accurately understand and generate scene elements that conform to the aesthetic and structural characteristics of sci-fi, fantasy, and other types of films.

[0071] 2) Then, the generated unified fusion state representation H is input as a condition to extract the script global setting and director real-time intention information, which serves as the core semantic basis for generating the global scene framework representation and guiding the denoising process of the vertical diffusion model. The unified fusion state representation H is semantically decomposed into a script narrative framework feature vector H s and a director artistic intention feature vector H d :

[0072]

[0073] where W s i and are the weight matrices of the i-th narrative feature extraction layer and the j-th director intention extraction layer, k and m represent the total number of narrative feature extraction layers and director intention extraction layers, and α i and β j are the corresponding feature weight coefficients, and MLP(·) represents a multi-layer perception network.

[0074] In the conditional guidance mechanism, the unified fusion state representation H is deeply coupled with the noise prediction network of the vertical diffusion model through an attention mechanism. The noise prediction network calculates the optimal denoising direction at each iteration based on the current noise state, time step, and unified fusion state representation H. Specifically, H s and H dBy embedding into the denoising network through cross-attention mechanism, adopting DDPM scheduling strategy to control the denoising intensity, and using the classifier to guide the enhancement condition, the multi-scale fusion is carried out according to the different resolution level condition information. Finally, the global scene framework representation is generated by the vertical diffusion model. This process adopts the diffusion generation framework with conditional control, and the core denoising step is:

[0075]

[0076] where x t represents the noise state vector at time t, x t-1 represents the noise state vector at time t-1. α t and are predefined noise scheduling parameters, ∈ θ is a noise prediction network with parameter θ, σ is the noise scale, and z is a standard normal distribution sampling vector.

[0077] Through this top-level generation method centered on the script and director's intention, the global scene framework representation can be output, including the macro three-dimensional space layout, the main geometric outline, the positioning and form draft of large static structure, and the division of key narrative function area as well as the dominant visual style and lighting keynote. This framework representation provides clear high-level guidance for the subsequent fractal autoregressive generation process, ensuring that the final rendering result can accurately reflect the creator's intention and the visual language specific to science and television.

[0078] 3) The vertical diffusion model gradually constructs the virtual scene structure blueprint in the denoising iteration process. In this process, the semantic conditions from the script mainly affect the overall narrative structure and atmosphere of the scene, while the dynamic conditions from the actors precisely define the actor's positioning area, interaction space and possible motion trajectory in the scene, ensuring that the generated scene structure can adapt to the real-time performance of the actors. Specifically as follows:

[0079] Receive the global scene framework representation and integrate the actor dynamic mask features and accurate motion vector information in the unified fusion state representation H to complete the final construction of the structured virtual scene structure blueprint containing all necessary guidance information. By accurately mapping the real-time spatial occupation information of the actors to the global scene framework representation, and dynamically labeling the corresponding regions in the framework with local constraints and interaction attributes according to the actor's posture, motion and potential interaction intention, such as contact surface feedback type, real-time reflection area setting, associated object state update prompt, accompanying special effect trigger area, etc. These real-time dynamic constraints centered on the actors are integrated into the global scene framework representation, and finally an information complete, structure clear and encoding all details from macro setting to micro real-time interaction virtual scene structure blueprint is output, which is used to directly drive the fractal content generation.

[0080]

[0081] wherein B1 is a spatial layout blueprint matrix, B2 is an interaction relationship blueprint matrix, and B is a final integrated virtual scene structure blueprint matrix. K is a global scene framework representation, K T is a transpose matrix thereof, F a is an actor dynamic mask feature matrix, represents a spatial gradient of the actor dynamic mask feature matrix, V is an actor accurate motion vector matrix, V T is a transpose matrix thereof, a is an actor dynamic mask feature weight coefficient, γ is a transpose mapping fusion coefficient, δ is a gradient response coefficient, ω1 is a spatial layout weight coefficient, ω2 is an interaction relationship weight coefficient, and ω3 is a multi-mode coordination coefficient, represents a matrix direct sum operation, and ⊙ represents element-wise point multiplication. tr(·) represents a trace operation of a matrix.

[0082] S3, fractal autoregressive method based on structure blueprint guidance to generate virtual scene content: taking the virtual scene structure blueprint as the core of the spatialization condition input and global constraint, the virtual content is synthesized through the fractal autoregressive generation unit of the recursive hierarchical structure. Among them, the fractal autoregressive generation unit is organized into a recursive hierarchical structure. When the generation process is started, the top-level generation unit generates the preliminary and coarse-grained content representation of the scene according to the macro layout information, key object setting and actor spatial constraint provided by the virtual scene structure blueprint; then, this process is expanded in a top-down and recursive nested manner. Each level of the generation unit receives the output of the last level and the more detailed guidance information of the corresponding spatial region extracted from the virtual scene structure blueprint as the condition, and generates more detailed content details. Through this recursive iteration from coarse to fine and guided by the structure blueprint throughout the process, the accurate synthesis of the content is finally completed at the pixel level, and the high-resolution and high-detail virtual scene rendering picture consistent with the script setting, director's intention and actor dynamic interaction logic is output.

[0083] S31, the virtual scene structure blueprint is parsed to extract macro layout information, key object setting and actor spatial constraint, etc., and on this basis, the top-level unit of the fractal autoregressive generation system is activated.

[0084] Specifically, first, the feature decoupling and blueprint mapping method is used to convert the virtual scene structure blueprint into a comprehensive guidance matrix:

[0085] G = λ1·G1+ λ o ·G o + λ c ·G c (8)

[0086] In the formula, G is a comprehensive guidance matrix, G1, G oand G c are layout guide matrix, object position matrix and space constraint matrix extracted from virtual scene structure blueprint matrix B respectively, λ1, λ o and λ c are corresponding weight coefficients respectively.

[0087] Subsequently, the top-level generation unit receives the integrated guide matrix G as a conditional input, and initiates the macroscopic content generation process. This process uses a recursive autoregressive approach to gradually build a coarse-grained representation of the scene:

[0088] X = T(G, θ t ) · L(B) (9)

[0089] where X is the generated coarse-grained representation of the scene, T represents the top-level generation unit, θ t is the parameter set of T, L(·) is a blueprint adaptation function used to extract structural features related to the current level from B. The generated coarse-grained representation X contains basic spatial structures, main object outlines and light and shadow atmosphere that conform to the blueprint planning, but the details are not rich.

[0090] In addition, based on X and the virtual scene structure blueprint matrix B, a regional sub-task sequence is constructed, and the overall generation task is decomposed into multiple local region detail generation tasks, providing more accurate guide information for subsequent levels. This top-down task decomposition strategy ensures that the final generation result not only follows the macroscopic setting of the blueprint, but also realizes rich visual performance in details.

[0091] S32, the generation unit of the subsequent level receives the coarse-grained representation output by the previous level and the corresponding generation task M (the generation task M is one of the local region detail generation tasks obtained by decomposing the overall generation task), first calls the blueprint adaptation function to extract the refinement conditions C of the current level from the virtual scene structure blueprint, and then drives the one-time refinement of the current level generation unit g(·) to obtain a higher resolution content representation. Taking the subsequent level generation unit closest to the top-level generation unit as an example, the higher resolution content representation generated by it is as follows:

[0092] X' = g(X, C, M; Θ) (10)

[0093] where Θ represents the parameter set of the fractal autoregressive generation unit g(·) of the current level, which defines the detail filling rules.

[0094] The subsequent level further enriches the details according to the following iteration rule:

[0095] X q+1 =g q+1 (X q ,C q+1 ,M q+1 ;Θq+1 ), q = 0, 1,..., Q - 1 (11)

[0096] where Q is the preset number of iterations, i.e., the number of layers of the fractal auto-regressive generation unit, X q+1 is the generated content of the q+1th layer generation unit, g q+1 (·) is the q+1th layer generation unit, C q+1 is the refinement condition of the q+1th layer generation unit, Θ q+1 is the parameter set of the q+1th layer generation unit, M q+1 is the generation task of the q+1th layer generation unit. Thereafter, X Q is divided into a local representation divided according to a spatial region or a functional module, and then is transmitted together with corresponding detailed guidance information containing more detailed spatial constraints, material properties, lighting parameters, etc.

[0097] S33, this step performs the final synthesis stage of the fractal auto-regressive recursive process, aiming to generate a final rendering representation that meets all micro constraints. After receiving the refined content representation X Q , first, the detailed conditions C f containing pixel-level details or accurate rendering properties corresponding to the current task are extracted from the virtual scene structure blueprint B through a blueprint adaptation function. final Subsequently, the conversion function G Q (·) of the final synthesis stage is used, which receives X f and takes C final as the accurate condition input, directly calculates and outputs the final rendering representation P:

[0098] P = G Q (X f , C final ; Ψ) (12)

[0099] where P is the final output rendering representation, i.e., the virtual scene rendering picture, and Ψ is the parameter set of the conversion function G final (·).

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the purpose and scope of the technical solutions, which should be covered in the scope of the claims of the present application.

Claims

1. A method for generating interactive virtual scenes for science and technology films based on a large fractal generation model, characterized in that: The method includes: First, multi-source information extraction is performed, including real-time video streams of actors, film and television script text, and director's creative intent; dynamic feature vectors are extracted from the real-time video streams of actors using a dynamic feature capture method based on mask segmentation; the film and television script text is converted into a semantic graph feature vector using a semantic knowledge graph construction algorithm; and the director's creative intent graph feature vector is extracted from the director's creative intent using a semantic intent parsing method; Then, the dynamic feature vectors, semantic graph feature vectors, and director's intention graph feature vectors are mapped to a unified semantic embedding space, and cross-modal semantic alignment and fusion are performed to generate a unified fusion state representation. The unified fusion state representation is used as the conditional input of the vertical diffusion model, and a global scene framework representation is generated based on the script semantics and director's intention. The interaction areas and constraints in the global scene framework representation are calibrated in combination with the actor's dynamic features, and finally a virtual scene structure blueprint is output. Finally, taking the virtual scene structure blueprint as the spatial condition and global constraint, virtual content synthesis is performed through a fractal autoregressive generation unit with a recursive hierarchical structure.

2. The method according to claim 1, characterized in that The method for extracting dynamic feature vectors from the actor's real-time video stream by using a dynamic feature capture method based on mask segmentation includes: extracting continuous frames from the actor's real-time video stream and using a deep learning model to separate foreground and background, generating an accurate actor mask, then capturing the action change features between continuous frames through time series analysis, applying a feature vectorization algorithm to convert the performance dynamics into a dynamic feature vector F, and the dynamic feature vector F is obtained by using the mask feature M on the time series. t and action change gradient The weighted fusion of Where, α t and β t are the mask feature weight and action change gradient weight at time t, respectively, and T is the total length of the time series; The method of converting the film and television script text into a semantic graph feature vector by using a semantic knowledge graph construction algorithm includes: performing word segmentation, syntax analysis, and entity recognition on the film and television script text to extract core semantic elements; using semantic role labeling technology to identify the action subject, receptor, and environment description in the script to form a preliminary semantic relationship network; applying a knowledge reasoning enhancement algorithm to supplement implicit semantic associations based on a pre-trained language model and a domain knowledge base to generate a script semantic knowledge graph; and finally encoding the knowledge graph through a graph neural network to generate a script semantic graph feature vector S: Where, v i represents the feature vector of the i-th node in the graph, N is the total number of graph nodes, w i is the semantic importance weight of the corresponding node, R represents the transformation matrix of the node feature in the semantic graph relationship space, b is the global bias feature vector, and σ(·) is the nonlinear activation function; The method of extracting the director's intention graph feature vector from the director's creative intention by using the semantic intention analysis method includes: performing natural language processing on the director's voice or text instructions to distinguish different categories of creative requirements; applying a multi-layer attention mechanism to assign differentiated weights to different categories of creative requirements; converting the processing results into a structured director's intention feature vector by using a semantic vectorization algorithm; and finally constructing a director's intention graph, representing the hierarchy and association relationship between different creative intentions through a graph structure, to generate a director's intention graph feature vector D: D=γ·tanh(β c ·I c +β s ·I s ) (3) Where, I c Represents the scene content intention vector, I s represents the scene style intention vector, β c and β s represents the weight coefficient; γ is the feature scaling coefficient, and tanh(·) is the hyperbolic tangent activation function.

3. The method according to claim 2, characterized in that The dynamic feature vector, semantic graph feature vector, and director intention graph feature vector are mapped to a unified semantic embedding space, and cross-modal semantic alignment and fusion are performed to generate a unified fusion state representation: H=W f ·σ(M1F+M2S+M3D) Where M1, M2 and M3 are the transformation matrices of visual, text and director intention features respectively, and W f is the fusion mapping matrix, and σ(·) is the nonlinear activation function.

4. The method according to claim 3, characterized in that The unified fusion state representation H is used as the conditional input of the vertical diffusion model to guide the denoising process of the vertical diffusion model; wherein the unified fusion state representation H is semantically deconstructed and decomposed into the script narrative framework feature vector H s and the director's artistic intention feature vector H d : Where W s i and are the weight matrices of the i-th narrative feature extraction layer and the j-th director intention extraction layer, k and m represent the total number of narrative feature extraction layers and director intention extraction layers, respectively. i and β j is the feature weight coefficient, MLP(·) represents the multi-layer perception network; based on the extracted H s and H d The denoising process of the vertical diffusion model is guided. During the denoising iteration process, the vertical diffusion model generates a global scene frame representation. The denoising process of the vertical diffusion model is expressed as: Where x t represents the noise state vector at time t, x t-1 represents the noise state vector at time t-1; α t and is the predefined noise scheduling parameter, ∈ θ represents a noise prediction network with parameter θ, σ is the noise scale, and z is a standard normal distribution sampling vector.

5. The method according to claim 4, characterized in that Combining the global scene framework representation with the actor's dynamic mask features and precise motion vector information in the unified fusion state representation H, a virtual scene structure blueprint is constructed: the actor's real-time spatial occupancy information is mapped onto the global scene framework representation, and local constraints and interaction attributes are dynamically annotated on the corresponding areas in the global scene framework representation based on the actor's posture, movement, and potential interaction intention. This virtual scene structure blueprint is generated as shown in the following formula: Where B1 is the spatial layout blueprint matrix, B2 is the interaction relationship blueprint matrix, B is the virtual scene structure blueprint matrix; K is the global scene framework representation, F a is the actor dynamic mask feature matrix, represents the spatial gradient of the actor's dynamic mask feature matrix, V is the actor's precise motion vector matrix, α is the actor's dynamic mask feature weight coefficient, γ is the transposed mapping fusion coefficient, δ is the gradient response coefficient, ω1 is the spatial layout weight coefficient, ω2 is the interaction relationship weight coefficient, and ω3 is the multi-modal synergy coefficient. represents the matrix direct sum operation, ⊙ represents the dot product between elements, and tr(·) represents the matrix trace operation.

6. The method according to claim 5, characterized in that Taking the virtual scene structure blueprint as the spatial condition and global constraint, virtual content synthesis is performed through a recursive hierarchical fractal autoregressive generation unit; in the fractal autoregressive generation unit, the top-level generation unit first generates a coarse-grained scene representation based on the macro layout information, key object settings and actor space constraints provided by the virtual scene structure blueprint, and then expands in a top-down, recursive nested manner. The generation unit at each level receives the output of the previous level and the guidance information of the corresponding spatial area extracted from the virtual scene structure blueprint as conditions, generates fine-grained content details, and finally outputs a virtual scene rendering picture.

7. The method according to claim 6, characterized in that The top-level generation unit first generates a scene coarse-grained representation based on the macro layout information, key object settings and actor space constraints provided by the virtual scene structure blueprint, including: first converting the virtual scene structure blueprint into a comprehensive guidance matrix: G=λ1·G1+λ o ·G o +λ c ·G c Where G is the comprehensive guidance matrix, G1, G o and G c are the layout guidance matrix, object position matrix and space constraint matrix extracted from the virtual scene structure blueprint matrix B, λ1, λ o and λ c is the weight coefficient; The top-level generation unit receives the comprehensive guidance matrix G as a conditional input and constructs a coarse-grained representation of the scene: X=T(G,θ t )·L(B) Where X is the coarse-grained representation of the scene, T is the top-level generation unit, and θ t is the parameter set of T, and L(·) is the blueprint adaptation function, which is used to extract the structural features related to the current level from the virtual scene structure blueprint matrix B. Then, based on the coarse-grained representation X of the scene and the virtual scene structure blueprint matrix B, a regional subtask sequence is constructed to decompose the overall generation task into multiple local region detail generation tasks. Subsequent levels further generate detailed content through iteration: X q+1 =g q+1 (X q ,C q+1 ,M q+1 ;Θ q+1 ),q=0,1,...,Q-1 Where Q is the preset number of iterations, X q+1 is the generated content of the q+1th layer generation unit, g q+1 (·) is the q+1th layer generation unit, C q+1 is the refinement condition for the q+1th layer generating unit, Θ q+1 is the parameter set of the q+1 layer generation unit, M q+1 Generate the task of generating units for the q+1th layer; According to the output X of the last layer generation unit Q Perform virtual content synthesis and extract detail conditions C from the virtual scene structure blueprint matrix B through the blueprint adaptation function f ; X Q and detail condition C f Input conversion function G final (·), calculate and output the rendering representation P: P6 G final (X Q ,C f (Ψ) Where P represents the final output rendering representation, that is, the virtual scene rendering screen, and Ψ is the conversion function G final (·) parameter set.

Images