An XR virtual walkthrough content generation method based on latent space time interpolation and semantic asset recommendation

Abstract

The application discloses an XR virtual preview content generation method based on latent space timing interpolation and semantic asset recommendation, and belongs to the technical field of XR virtual preview, latent space animation generation and three-dimensional asset retrieval. The method obtains a text demand, a reference image, a key frame, a three-dimensional asset and a scene constraint, obtains text, image and asset semantic vectors through CLIP multimodal semantic alignment, generates or refines special effect key frames through Latent Diffusion, and performs linear interpolation, spherical interpolation or diffusion guided interpolation on adjacent key frame latent vectors; meanwhile, asset recommendation results are generated by fusing CLIP similarity, LFD shape similarity, scene relationship, historical reuse stability and artificial feedback. The application can improve the continuity of special effect generation, the accuracy of asset recommendation and the scheme reuse efficiency in XR preview.

Description

Technical Field

[0001] This invention belongs to the technical fields of XR virtual pre-show, latent space animation generation, 3D asset retrieval, semantic recommendation, and digital asset reuse. Specifically, it relates to a method for generating XR virtual pre-show content based on latent space temporal interpolation and semantic asset recommendation. In particular, it relates to an engineering solution that combines a 2D encoder, latent space vector interpolation, 3D generator and discriminator, LFD light field descriptor, CNN / DNN classification, and knowledge graph recommendation with XR virtual filming, live streaming, virtual-real fusion, or virtual pre-show scenarios. Background Technology

[0002] With the development of digital technology and new digital products, image processing, computer graphics, and real-time interactive technologies are widely used in film and television special effects, live streaming of galas, e-commerce live streaming, and virtual press conferences. Real-time rendering and fusion of virtual and real scenes requires that the real scene and the virtual scene maintain consistency in spatial location, temporal synchronization, lighting style, and output frame rate.

[0003] In practical applications, the system is based on digital display technology and develops an AR live streaming platform system based on binocular vision. It focuses on common technologies such as real-time rendering and fusion of virtual and real images, real-time camera tracking, real-time foreground keying under green screen background, dynamic rendering, motion capture, automatic scene setting for virtual pre-show, virtual and real interaction, real-time character generation, and construction of a 3D model asset library, forming a complete solution for real-time live streaming and virtual pre-show.

[0004] XR virtual pre-visualization requires the rapid generation of dynamic special effects such as smoke, flames, and water flow, and the retrieval of suitable assets from a large number of 3D models. Traditional manual production and retrieval methods are time-consuming, and direct latent space linear interpolation may cause abrupt changes and blurring in animation motion, making it difficult to support rapid pre-visualization of live streaming or film and television content.

[0005] Existing systems often only address a single aspect, such as camera tracking, keying, or rendering, lacking a complete technical chain connecting real-world shooting, algorithmic processing, virtual scenes, and application cases. Therefore, it is necessary to refine systems, methods, and implementation examples based on the actual engineering needs of XR virtual filming.

[0006] In this invention, XR stands for Extended Reality; AR stands for Augmented Reality; CLIP stands for Contrastive Language-Image Pre-training; LFD stands for Light Field Descriptor; CNN stands for Convolutional Neural Network; DNN stands for Deep Neural Network; and MLP stands for Multi-Layer Perceptron. Summary of the Invention

[0007] The purpose of this invention is to provide an XR virtual pre-show content generation method based on latent space temporal interpolation and semantic asset recommendation, which enables director text, reference images, smoke or transition effects, 3D assets and scene constraints to be quickly generated, retrieved, combined and reused during the pre-show stage.

[0008] To achieve the above objectives, this invention encodes text requirements, reference images, keyframes, volumetric effects data, and a 3D asset library into a unified multimodal representation, and employs CLIP semantic alignment, Latent Diffusion latent space diffusion generation, latent space temporal interpolation, and asset sorting feedback to form a pre-visualization content generation chain.

[0009] The CLIP multimodal semantic alignment module is used to calculate the similarity between text requirements, reference images, and asset thumbnails, enabling the system to understand synonyms, visual styles, and scene relationships, reducing reliance on keyword matching.

[0010] The Latent Diffusion module is used to generate or refine keyframes for special effects in latent space. Compared to direct pixel or voxel generation, latent space diffusion is better suited for rapid previewing, allowing the overall structure of smoke, lighting effects, transitions, and virtual backgrounds to be expressed in a lower-dimensional representation.

[0011] The latent space temporal interpolation module is used to generate a continuous preview sequence between adjacent keyframes. The system can use linear interpolation, spherical interpolation, or diffusion-guided interpolation, and suppresses abrupt changes in smoke, flame, or volumetric effects between keyframes through temporal smoothing loss.

[0012] The asset ranking feedback module generates asset recommendation scores based on CLIP similarity, LFD shape similarity, CNN / DNN category features, scene relationships, historical reuse stability, and human selection results. Assets that are adopted multiple times have their ranking weight increased, while assets that are rejected or replaced have their ranking weight decreased.

[0013] The input interface of this invention includes director's text requirements, reference images, keyframes, 3D model assets, asset tags, scene relationships, and human feedback; the output interface includes candidate special effects sequences, asset recommendation lists, generation confidence scores, temporal smoothing indicators, human review tasks, and pre-rehearsal archive records.

[0014] Through the above technical solutions, this invention solves the problems of insufficient semantics in keyword retrieval, time-consuming manual special effects production, easy mutation of latent space interpolation, and lack of feedback learning in asset reuse in XR pre-shows, and forms a specific multimodal algorithm link for virtual pre-show content generation.

[0015] The knowledge graph recommendation module is a semantic constraint module set up in this invention. The system organizes 3D assets, asset categories, material styles, applicable scenarios, lens uses, historical adoption records, and reasons for manual rejection into graph nodes and relational edges, so that asset recommendations can not only be based on keyword or thumbnail similarity, but also consider the scene relationships between the press conference background, product display stands in the live broadcast room, smoke transitions, main visual elements, and auxiliary props.

[0016] The CLIP semantic alignment module maps director text, reference images, and asset thumbnails to the same semantic space, solving the cross-modal matching problem between natural language descriptions and visual materials; the Latent Diffusion module generates or refines smoke, lighting effects, transitions, or background keyframes in the latent space; and the latent space temporal interpolation module generates continuous pre-play sequences between keyframes and suppresses abrupt changes through temporal smoothing constraints.

[0017] The LFD shape retrieval module addresses the shortcomings of CLIP in representing 3D geometric shapes. The system extracts light field descriptors from 3D models from multiple perspectives and combines them with CLIP similarity, CNN / DNN category features, knowledge graph relationship scores, historical reuse stability, and human feedback to form a recommendation ranking. Output includes candidate assets, recommendation reasons, similarity breakdowns, substitute assets, and low-confidence verification tags. Attached Figure Description

[0018] Figure 1 This is the overall structure diagram of the virtual pre-show content generation system of the present invention.

[0019] Figure 2 This invention relates to CLIP semantic alignment and latent space diffusion generation graph.

[0020] Figure 3 This invention relates to latent spatial temporal interpolation and 3D special effects generation.

[0021] Figure 4 This invention relates to a knowledge graph asset database and a multimodal retrieval graph.

[0022] Figure 5This is a flowchart of the asset recommendation and pre-simulation combination output process of the present invention.

[0023] Figure 6 This is a flowchart of the seven types of smoke sequence interpolation generation in Embodiment 1 of the present invention.

[0024] Figure 7 This is a flowchart of the three-dimensional model asset recommendation process according to Embodiment 2 of the present invention.

[0025] Figure 8 This is a flowchart of the XR virtual pre-show archiving and reuse process in Embodiment 3 of the present invention.

[0026] Figure 9 This is a comparison chart of the smoke effect interpolation generation effect of the present invention. Detailed Implementation

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the following embodiments are used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection; the present invention can be implemented by making similar substitutions to the encoder structure, diffusion steps, asset features or recommendation weights without departing from the technical concept of combining CLIP multimodal semantic alignment, LatentDiffusion latent space generation, latent space temporal interpolation and 3D asset recommendation.

[0028] Combination Figures 1 to 3 This implementation incorporates pre-visualization text requirements, reference images, keyframes, volume data, 3D models, asset thumbnails, shape features, scene relationships, and director confirmation results into a unified content generation chain. Figure 2 The relationship between CLIP text-image semantic alignment, conditional vector generation, and Latent Diffusion stepwise denoising is given; Figure 3 The relationships between keyframe latent vectors, linear or spherical interpolation, diffusion-guided interpolation, 3D decoding, temporal smoothing loss, and MAE kernel are given to suppress abrupt changes in smoke, lighting effects, or volumetric effects between keyframes.

[0029] The reason for introducing CLIP is that XR preview requirements are often expressed using a mix of natural language, reference images, and asset libraries. Simple keyword searches cannot understand semantics such as "background of a technology launch event," "smoke transition," and "futuristic live stream room." CLIP can align textual and visual semantics. The reason for introducing Latent Diffusion is that latent space diffusion can generate or correct keyframes in a lower-dimensional space, which is more suitable for rapid previewing than direct pixel or voxel generation.

[0030] In this embodiment, the CLIP multimodal semantic alignment, Latent Diffusion, and asset ranking joint generation algorithm is not run as an isolated software module, but is embedded in the acquisition, modeling, prediction, and verification chain of text requirements, reference images, and 3D asset recommendation generation scenarios in XR virtual pre-visualization. The system first encodes scene labels, asset states, generation conditions, and manual verification results into computable states, then the algorithm module outputs predicted values, confidence levels, constraint residuals, or candidate solutions, and finally the pre-visualization rules transform the algorithm results into verifiable generation, recommendation, or archiving actions.

[0031] The reason for choosing the CLIP multimodal semantic alignment, Latent Diffusion, and asset ranking joint generation algorithm is that XR previews typically include director text, reference images, smoke or transition effects, a 3D asset library, and scene constraints simultaneously. Single keyword retrieval is insufficient to express cross-modal semantics. If only static labels or ordinary similarity calculations are used, the system struggles to understand style, spatial relationships, and historical selection preferences, and is prone to recommending semantically similar but scene-mismatched materials.

[0032] Furthermore, CLIP can map text, images, and asset previews to a unified semantic space, LatentDiffusion can generate or correct keyframes in the latent space, and the ranking model can combine semantic similarity, shape similarity, and historical reuse feedback. Therefore, this invention does not directly equate the algorithm output with the final processing result during the training and inference phases. Instead, it sets data fitting terms, temporal stability terms, scene constraint terms, and manual verification terms, enabling the model to achieve high predictive ability while maintaining interpretability of the pre-production boundary.

[0033] In terms of algorithm configuration, this implementation scenario requires that the generated content can be quickly reviewed by directors or art directors. The training objectives need to simultaneously cover textual semantics, visual references, temporal continuity, and asset recommendation feedback. Training records retain sample sources, prompt words, loss components, thresholds, and human confirmation opinions. After changes in project style, asset library, or special effects type, technical personnel can use this information to determine whether to update CLIP alignment, diffusion model, recommendation weights, or simply perform human review.

[0034] Before samples are entered into the database, the system labels text requirements, reference images, asset categories, special effects types, keyframes, and manual selection results. Text embeddings, image embeddings, latent vectors, asset labels, and user feedback are normalized at the project scale, and rejection or modification records are used as high-confidence supervision signals in training.

[0035] Model training is conducted in three steps: general multimodal pre-training, project asset library calibration, and feedback correction during the preview period. Pre-training learns the basic semantic relationships between text and images, project calibration incorporates differences in director style, asset naming, and 3D model categories, and the preview period only updates recommendation weights, prompt word templates, and a small number of adaptation parameters to avoid disrupting the consistency of the existing asset library.

[0036] CLIP text and image encoding can be represented as:

[0037]

[0038] In the formula, For text demand vectors, For text encoders, For text requirements, As a reference image vector, For asset image vectors, For image encoders, For reference image, This refers to a thumbnail or rendered image of an asset.

[0039] Multimodal similarity is:

[0040]

[0041] In the formula, For text requirements With assets Multimodal similarity, and Weights for text and images, , and For normalized semantic vectors, superscript This indicates transpose or inner product.

[0042] The keyframe latent vector is encoded as follows:

[0043]

[0044] In the formula, and These are latent vectors of adjacent keyframes or volume data. For latent space encoders, and For adjacent keyframes or volume data.

[0045] The Latent Diffusion noise reduction steps are as follows:

[0046]

[0047] In the formula, and For adjacent latent variables in the diffusion process, and The diffusion coefficient is... For noise prediction networks, For text, reference images, or scene conditions, For noise scale, It is Gaussian noise.

[0048] The condition vector can be represented as:

[0049]

[0050] In the formula, To generate the condition vector, The conditional mapping matrix, For text vectors, For image vectors, For scene constraint vectors, This indicates vector concatenation.

[0051] Latent space temporal interpolation is:

[0052]

[0053] In the formula, For interpolation latent vectors, These are the interpolation coefficients. and For latent vectors of adjacent keyframes, .

[0054] For latent vectors with large amplitude differences, spherical interpolation can be used:

[0055]

[0056] In the formula, For spherical interpolation latent vectors, These are the interpolation coefficients. for and The included angle, It is a sine function. and These are adjacent latent vectors.

[0057] The generated result and the time series smoothing loss are:

[0058]

[0059]

[0060] In the formula, For the generated intermediate frames or volume data, For generator, For latent vectors, For conditional vectors, For time-series smoothing loss, For time intervals.

[0061] The overall asset recommendation score is:

[0062]

[0063] In the formula, For demand With assets Recommended score, to These are the weighting coefficients. For multimodal similarity, For shape similarity, Score the scene relationship. For historical reuse stability, To prevent artificial refusal of punishment.

[0064] When uniformly evaluating the generated schemes and recommendation results, the confidence level of the simulated scheme can be calculated using the following formula:

[0065]

[0066] In the formula, To generate or recommend confidence levels, For the Sigmoid function, to These are the weighting coefficients. For recommended score, For time-series smoothing loss, To rebuild the losses, This represents the number of times a person can manually reject an offer.

[0067] Combination Figure 4 and Figure 5 The system organizes 3D assets, asset categories, material styles, applicable scenarios, lens uses, and historical adoption records into a knowledge graph, and uses CLIP semantic similarity, LFD shape similarity, CNN / DNN category features, and human feedback together for asset ranking. Figure 5 This demonstrates the closed loop between candidate assets, special effects sequences, pre-visualization combinations, director confirmation, alternative assets, and archived reuse. Combined with... Figures 6 to 8 The examples illustrate the generation of seven types of smoke sequences, the recommendation of three-dimensional model assets, and the archiving and reuse of pre-shows.

[0068] Although this invention introduces CLIP and Latent Diffusion, 3D asset recommendation still requires the disclosure of traditional shape retrieval and scene relationship scoring. LFD shape descriptors are used to supplement the semantic deficiencies of text and thumbnails.

[0069]

[0070] In the formula, For assets Light field shape descriptor, to Features for contour, depth, or projection views from multiple perspectives.

[0071]

[0072] In the formula, To determine the shape similarity between two assets. and For the assets to be compared, This is a shape distance scale parameter. This formula is used to avoid ranking assets with similar text but unsuitable shapes for camera requirements.

[0073] For the rehearsal scene, there are also relationships between assets such as main visual, background, transition, and props. The system calculates the degree of relationship matching through the scene relationship diagram.

[0074]

[0075] In the formula, For demand With assets Scene relationship score, For relational types, For relation weights, For the set of edges in the asset knowledge graph, This is an indicator function that takes a value of 1 if the condition is met and 0 otherwise. This formula is used to illustrate that the recommendation ranking is not simply the output of keywords or a deep model.

[0076] During the training phase, the system primarily focuses on keyframes for special effects, continuous pre-visualization sequences, and 3D asset recommendation ranking, but does not use a single mean squared error as the overall optimization objective. This is because a single error term only indicates the distance between the model output and historical labels; it cannot constrain whether the prediction process conforms to the pre-visualization production logic, nor can it prevent the model from making unstable judgments on boundary conditions, low-confidence samples, or stage-switching samples.

[0077] The loss function is set around CLIP contrastive learning, diffusion noise prediction, temporal smoothing, and asset ranking feedback. Contrastive learning constrains the text to be consistent with the visual reference, noise prediction ensures the quality of generation, temporal smoothing suppresses abrupt changes in volumetric effects such as smoke, and ranking feedback makes the recommended assets closer to human selection.

[0078] CLIP contrastive learning loss is:

[0079]

[0080] In the formula, CLIP contrastive learning loss, For the sample size, and The sample number. For similarity function, For text embedding, and Embed for images or assets, This is the temperature coefficient.

[0081] The predicted loss for diffuse noise is:

[0082]

[0083] In the formula, To predict the loss for diffused noise, For expectation operator, As a latent variable, For the diffusion step, This is real noise. For conditional vectors, This is a noise prediction network.

[0084] The asset ranking loss is:

[0085]

[0086] In the formula, Losses due to asset ranking. For demand, and These are positive sample assets and negative sample assets, respectively. For interval parameters, A score is given for asset recommendations.

[0087] The total loss is:

[0088]

[0089] In the formula, For the total loss, To compare learning loss, For the spread of loss, For time-series smoothing loss, For ranking loss, For manual feedback of losses, to These are the weighting coefficients.

[0090] The weights of each loss component are adaptively configured according to the pre-playing stage and human feedback. The semantic alignment and diversity weights are increased in the concept exploration stage, the diffusion reconstruction and temporal smoothing weights are increased in the keyframe refinement stage, and the ranking feedback weights are increased in the asset reuse stage. Human-confirmed samples are used to calibrate the recommendation scores, and human-rejected samples are used to form negative feedback.

[0091] When a certain loss component continues to rise, the system first cross-checks it against the original input, the generation log, and the manual review record. If the data fit component rises while the time-stability component remains normal, it usually indicates a bias in the label or local input; if the time-stability component rises while the data fit component remains normal, it indicates that the model may have learned a random correlation; if the confidence boundary component rises, the conservative generation strategy and the manual review process are triggered first.

[0092] CLIP contrastive learning loss is used to ensure that textual requirements, reference images, and asset previews are comparable within the same semantic space. For XR previews, directors might describe text such as "cool-toned smoke, behind the stage, slowly spreading," which is difficult to match with ordinary keywords, but semantic embedding can capture style and spatial intent.

[0093] Latent Diffusion noise prediction loss is used to generate keyframes or correct effect frames in latent space. Latent space generation is better suited for fast previewing than pixel space generation because it can express the overall structure of smoke, lighting effects, and transitions in a lower dimension, which is then reconstructed by the decoder.

[0094] The asset ranking feedback loss enables the system to learn the selection preferences of directors and art directors. If a certain type of asset is accepted multiple times, its semantic and shape weights are increased; if a certain type of asset is rejected, the system lowers its ranking or requires human confirmation, thereby forming reusable pre-visualization knowledge.

[0095] To ensure the CLIP multimodal semantic alignment, Latent Diffusion, and asset ranking joint generation algorithm can stably adapt to the text requirements, reference image, and 3D asset recommendation generation scenarios in XR virtual pre-shows, the system does not treat all historical data equally at the sample level. Instead, it stratifies samples into five categories: normal operation, rehearsal switching, asset loading perturbation, manual intervention, and verification / confirmation. Normal operation samples are used to learn basic mapping relationships; concept exploration, keyframe refinement, and asset reuse samples are used to learn boundary transition rules at different production stages; manual rejection and replacement samples are used to identify recommendation biases; manual intervention samples are used to constrain the system from outputting actions that are difficult to execute; and verification / confirmation samples are used to calibrate the model's confidence level.

[0096] Regarding the sample time window setting, the system determines the window length based on the business latency of state changes in the generated scene, according to the text requirements, reference images, and 3D assets recommended in the XR virtual pre-simulation. For variables with fast responses, short windows are used to capture instantaneous changes; for variables with lag effects, long windows are used to preserve historical impacts. Overlapping windows are allowed, but the same anomalous event will not be repeatedly counted as multiple independent high-value samples to avoid exaggerating the weight of a small number of anomalous events during the training phase.

[0097] During label construction, the system records the final results of special effects keyframes, continuous pre-play sequences, and 3D asset recommendation ranking, while simultaneously retaining intermediate explanation labels. These intermediate explanation labels include key input sources, trigger thresholds, human-judgmented reasons, executed actions, and verification conclusions. After model training is complete, the system can simultaneously output prediction results and their main supporting evidence, facilitating technical personnel to verify the consistency between the model's conclusions and the pre-play logic.

[0098] The loss weights are adjusted according to the production task. In the concept exploration phase, the weights for semantic alignment and diversity are increased; in the keyframe refinement phase, the weights for diffusion reconstruction and temporal smoothing are increased; and in the asset reuse phase, the weights for ranking feedback are increased. This processing enables the model to serve the pre-visualization process, avoiding the creation of isolated images.

[0099] Model validation does not employ a single random partitioning method, but rather grouped validation by time, stage, and scenario. If random validation performs well but cross-stage validation results decline, it indicates that the model may have learned accidental correlations; if cross-scenario validation results decline, it indicates that the model needs on-site fine-tuning; if the error on manually reviewed samples is large, the loss weights and sample labels are corrected first, and then the model size is evaluated.

[0100] A cold start period is set up before the system is deployed on-site. During the cold start period, the CLIP multimodal semantic alignment, LatentDiffusion, and asset ranking joint generation algorithm only outputs prediction results, risk breakdowns, and recommendation basis, without triggering automatic execution. Only after the prediction results, loss breakdowns, and manual review results continuously meet the stability conditions will the system allow the output of semi-automatic or automatic pre-simulation recommendations. This setting ensures that the algorithm results are first verified on-site and manually confirmed before entering the semi-automatic or automatic generation stage.

[0101] During runtime, the system performs drift detection on the input distribution. If the mean, variance, missing rate, or scene label proportion of the input variables significantly deviates from the training distribution, the system increases the confidence penalty and marks the corresponding samples as samples to be reviewed. If the drift occurs only on a few variables, the system prioritizes adjusting the input scaling parameters; if the drift occurs on multiple core variables, it triggers small-sample fine-tuning or manual recalibration.

[0102] The model output records text similarity, image similarity, generation confidence, temporal smoothing value, and the reason for asset ranking. Technical personnel can use this information to determine whether deviations are due to prompts, reference images, latent space interpolation, or insufficient matching of the asset library.

[0103] Low-confidence results are not directly used as the final solution, but instead go through a branch of regeneration, re-search, or manual review. The system can adjust prompts, tighten asset categories, reduce generation randomness, reuse similar solutions, or prompt manual selection of key assets.

[0104] Regarding the utilization of anomalous samples, the system distinguishes between real pre-simulation anomalies and collected anomalies. Real anomalies retain their risk labels when entering the training set, while collected anomalies retain their quality labels. This prevents the model from mislearning blurred images, missing logs, or human errors as project patterns, and also prevents it from losing its ability to identify real risks by deleting all anomalous samples.

[0105] For parameter updates, the system employs versioned model management. Each model version saves the training data range, main loss weights, threshold configuration, manual review pass rate, and deployment time. If a new version experiences output jumps or unstable interpretations during field operation, the system can revert to the previous version and retain the failed samples from the new version for subsequent analysis.

[0106] The project execution is divided into three levels: prompts, suggestions, and pre-project outputs. The prompts level explains the semantic or asset risks, the suggestions level provides candidate content and reasons for ranking, and the pre-project outputs level is only open when both the generated confidence level and manual confirmation meet the requirements.

[0107] In the flowchart corresponding to the attached figure, the data input node is used for sample construction and quality labeling; the algorithm node is used to execute the CLIP multimodal semantic alignment, Latent Diffusion, and asset ranking joint generation algorithm and calculate the loss component; the judgment node is used to check the threshold, confidence level, and pre-simulation boundary; the output node is used to generate the results or perform a review task; and the archiving node is used to save the model evidence chain. Therefore, each flowchart in the attached figure corresponds to the steps of on-site data processing, model calculation, manual review, and archiving.

[0108] Manual review records include adoption, replacement, rejection, and explanations of reasons. Adoption records highlight similar prompts, replacement records correct asset sorting, rejection records create negative samples, and explanations of reasons are added to the project asset archive.

[0109] Regarding the boundaries of patent implementation, the CLIP multimodal semantic alignment, Latent Diffusion, and asset ranking joint generation algorithm can be deployed on local servers, edge computing nodes, or cloud training platforms. However, on-site inference should retain a conservative strategy for situations involving network outages or high latency. When the cloud is unavailable, the system uses the most recently approved model version and local thresholds for execution; once local data is restored, the training logs and approval results are synchronized.

[0110] In terms of security, the system will not automatically relax the pre-simulation quality boundaries due to short-term improvements in model predictions. All limits are derived from constraints on image quality, temporal continuity, asset availability, or manually set limits, and the algorithm can only find better solutions within these limits. If the model's suggestion exceeds the limits, the system marks the suggestion as unexecutable and outputs the loss component that generated the suggestion for manual evaluation to determine if there is an input or model anomaly.

[0111] During on-site implementation, the generated content from the pre-rehearsal first enters the candidate pool of the director or art director, rather than being directly used as final shot material. The system records prompts, reference images, asset categories, generation seeds, latent space interpolation methods, and manual selection results; when generating similar scenes again for the same project, the confirmed combination of prompts and asset categories are used first to reduce repeated trial and error.

[0112] Example 1: Latent Spatial Temporal Interpolation Generation of Seven Types of Smoke Sequences

[0113] Combination Figure 6 This embodiment uses test data containing random sampling sequences of seven smoke shapes as the object. Figure 6 Taking smoke keyframes as input, the data is processed through latent vector encoding, interpolation, and 3D decoding before entering the MAE verification and judgment. The Yes branch outputs a continuous smoke sequence and forms a test record, while the No branch supplements keyframes or adjusts the interpolation method before entering the sample write-back and archiving process.

[0114] The system first encodes keyframes or volume data into latent space vectors, and then uses Latent Diffusion to generate, complete, or refine keyframes under text or reference map conditions. Interpolation is then performed between adjacent keyframes. To avoid sudden changes in smoke morphology between adjacent frames, the system incorporates temporal smoothing loss during training or generation to ensure that the generated sequence remains continuous in shape, density, and motion direction.

[0115] The 3D decoding node restores the interpolated latent vectors to volumetric effects or renders usable smoke states. The MAE contrast node measures the difference between the generated result and the reference frame. When the difference exceeds a threshold, the system prompts to add keyframes, adjust the interpolation interval, or retrain the generator.

[0116] The animation output node feeds a continuous smoke sequence into the XR preview scene. Directors or artists can quickly compare different smoke patterns during the preview stage without having to create complete effects from scratch each time.

[0117] Before keyframe encoding, the system can perform scale normalization, density clipping, and center alignment on the smoke data, making different shape sequences comparable in the latent space. This processing helps reduce shape drift that occurs during interpolation.

[0118] Temporal smoothing loss not only constrains differences between adjacent frames but also constrains second-order changes, making the smoke diffusion, curling, and dissipation processes more continuous. For rapidly changing smoke, the system can increase keyframe density instead of forcibly smoothing it.

[0119] When outputting animation, the system retains keyframes, interpolation coefficients, and generation parameters. If the director or art director wants to adjust the smoke rhythm, they can reset the interpolation coefficients without having to regenerate all the footage.

[0120] During the data preparation phase, the system parses the required text and reference images. If missing, conflicting, or low-confidence inputs are found in this step, the system retains quality markers in the feature vectors and incorporates them into subsequent confidence assessments. In the model computation phase, the system obtains semantic vectors from CLIP and sets diffusion conditions. The engineering constraint phase focuses on generating keyframes using LatentDiffusion and performing temporal interpolation for verification. In the archiving phase, the system archives random seeds, conditional vectors, and manually verified results.

[0121] Through the above process Figure 6Each process node corresponds to a specific implementation action: input nodes correspond to data preparation and quality verification, algorithm nodes correspond to model calculation and constraint verification, and output nodes correspond to disposal suggestions and archiving review. Therefore, this embodiment can illustrate the complete implementation path of the CLIP multimodal semantic alignment, Latent Diffusion, and asset sorting joint generation algorithm in the generation of smoke effects time series, and make the algorithm steps correspond to the on-site implementation actions.

[0122] Figure 6 The abnormal branches in the algorithm are handled according to their source: for input anomalies, re-samples, retests, or weight reduction are performed first; for model anomalies, parameters are frozen and a backup model is called; for boundary condition anomalies, a conservative solution is output; and when review opinions are inconsistent, dispute records and final handling results are retained. This approach ensures that the algorithm module retains an auditable operating trajectory even under on-site disturbances.

[0123] Example 2: Semantic Recommendation of a 3D Model Asset Library

[0124] Combination Figure 7 This embodiment takes an asset database retrieval scenario consisting of no fewer than 50 three-dimensional models as its object. Figure 7 First, read the 3D asset database, extract CLIP semantic features, LFD shape features and category features, and then integrate knowledge graph relationships for sorting. When the sorting meets the requirements, the Yes branch outputs recommended assets and records the reason for replacement. When the sorting does not meet the requirements, the No branch allows manual adjustment of the sorting or correction of the retrieval weights.

[0125] The system builds a multi-dimensional index for each 3D model, including text labels, asset thumbnails, CLIP semantic vectors, LFD shape features, category features, scene relationships, and historical usage records. LFD features are used to express the model's shape, CLIP features are used to express textual and visual semantics, CNN or DNN features are used to express categories, and text labels are used to respond to the natural language retrieval needs of directors or art directors.

[0126] When making recommendations, the system considers semantic similarity, shape similarity, and scene relationship scores. If a user inputs requirements such as "technological background of a press conference," "smoke transition," or "product showcase in a live stream," the system outputs candidate assets, similarity scores, and reasons for recommendation, facilitating manual verification.

[0127] Figure 7 This invention demonstrates that the asset recommendation is not simply keyword matching, but rather incorporates shape, semantics, and scene usage into the scoring, enabling reusable materials to be quickly found during the XR preview stage.

[0128] When assets are added to the database, the system saves a thumbnail, bounding box, category label, material label, and applicable scenarios for each model. During retrieval, textual semantics and shape similarity are both used for sorting to avoid matching based solely on filename.

[0129] LFD features are used to describe the 3D shape contour, CNN or DNN classification results are used to describe the visual category, and scene relationships are used to describe the compatibility between assets. The weights of these three elements can be adjusted according to the type of pre-test task.

[0130] The recommendations not only provide the asset name, but also the matching reason and similarity score. Users can quickly determine whether an asset is suitable as a main visual element, background element, transition effect, or auxiliary prop based on the recommendation reason.

[0131] During the data preparation phase, the system extracts asset thumbnails, LFD shape features, and text labels; during the model calculation phase, the system calculates CLIP similarity and shape similarity; during the engineering constraint phase, the system focuses on reordering assets according to scene relationships and historical feedback; during the archiving phase, the system records the reasons for recommendation, alternative assets, and manual selections.

[0132] Combination Figure 7 Example 2 focuses on recording the comparison of asset selection before and after recommendation. The system saves CLIP similarity, LFD shape similarity, category features, asset ranking, reasons for manual replacement, and final asset portfolio to distinguish differences caused by insufficient semantic matching, shape inconsistency, and manual style selection.

[0133] Figure 7 The abnormal branches in the algorithm are handled according to their source: for input anomalies, re-samples, retests, or weight reduction are performed first; for model anomalies, parameters are frozen and a backup model is called; for boundary condition anomalies, a conservative solution is output; and when review opinions are inconsistent, dispute records and final handling results are retained. This approach ensures that the algorithm module retains an auditable operating trajectory even under on-site disturbances.

[0134] Example 3: Archiving and Reusing XR Virtual Preview Results

[0135] Combination Figure 8 This embodiment focuses on live broadcast previews, film set previews, or virtual press conference set previews. Figure 8 Starting from the input of the pre-rehearsal project, generate a solution and asset combination and obtain the director's confirmation result; the judgment node is used to confirm whether the solution can be reused. The Yes branch outputs the recommended template and project file, and the No branch enters the manual review, records the reason for rejection and writes back the reuse sample.

[0136] Once the pre-visualization plan is manually approved, the system saves keyframes, latent vectors, asset IDs, parameter configurations, recommended scores, and output versions. If a similar scene is required in a subsequent project, the archived records can be read and adjusted according to the resolution, camera movement, and style requirements of the new scene.

[0137] The reuse record is used to track which assets or effects are used in different projects, and whether they need to be re-rendered, downgraded, or replaced after use. This record, in turn, affects the asset recommendation weight, giving frequently used and stable assets a higher ranking in similar applications.

[0138] This invention is applicable to scenarios such as XR virtual pre-visualization, film and television set special effects planning, live broadcast set design, and virtual press conference content preparation. It can be implemented in conjunction with 3D asset libraries, rendering engines, and project archiving systems.

[0139] Pre-visualization requirements can come from director's storyboards, art concept art, or text descriptions. The system breaks down the requirements into scenes, styles, special effects, main assets, and output formats, and then calls the generation and recommendation modules respectively.

[0140] The director's confirmation node allows manual replacement, locking, or deletion of candidate assets. Confirmed selections are archived, and assets that are not adopted but have high similarity can be saved as alternatives for subsequent version iterations.

[0141] Reusing records enables the system to identify frequently used and consistently effective footage. For footage that has been replaced multiple times or failed to pass review, the system lowers its recommendation weight and prompts the user to re-annotate or update the model file.

[0142] During the data preparation phase, the system reads historical pre-simulation requirements and the final adopted assets; during the model calculation phase, the system compares the current requirements with historical semantic vectors; during the engineering constraint phase, the focus is on reusing similar solutions or triggering regeneration; during the archiving phase, the system feeds the new solutions back into the asset sorting model.

[0143] Combination Figure 8 In Example 3, the pre-rehearsal project, generated plan, asset portfolio, director confirmation results, and reuse conclusions are archived together. Reusable plans output recommended templates and form project files along the Yes branch, while non-reusable plans enter manual review along the No branch, recording the reasons for rejection, alternative assets, and corrected prompts, which serve as samples for subsequent reuse.

[0144] Figure 8 The abnormal branches in the algorithm are handled according to their source: for input anomalies, re-samples, retests, or weight reduction are performed first; for model anomalies, parameters are frozen and a backup model is called; for boundary condition anomalies, a conservative solution is output; and when review opinions are inconsistent, dispute records and final handling results are retained. This approach ensures that the algorithm module retains an auditable operating trajectory even under on-site disturbances.

[0145] Upon completion of this embodiment, the system binds the project number, prompt words, reference images, latent space vectors, generated model version, asset sorting model version, manual selection results, and reuse records into the same archive unit. Subsequently, when similar directorial intentions, special effects types, or asset styles appear, the system can call upon this archive unit as a priori for pre-visualization generation.

[0146] Combination Figure 9 Smoke, cloud, or particle volume effects in XR previews typically require generating continuous animation from a small number of keyframes. Figure 9 The visual comparison between the generated sequence and the reference sequence illustrates the constraints imposed by the latent space temporal interpolation module on volumetric effect boundaries, motion continuity, and morphological changes. In this invention, the system first obtains keyframe latent vectors from the encoder, then generates intermediate latent vectors using linear interpolation, spherical interpolation, or diffusion-guided interpolation, and outputs a continuous smoke sequence through a 3D decoder; when Figure 9 When boundary jumps, abrupt changes in shape, or an increase in mean absolute error occur in the comparison results shown, the system enters the supplementary keyframe, adjustment of interpolation weights, or manual review branch.

Claims

1. A method for generating XR virtual pre-show content based on latent space temporal interpolation and semantic asset recommendation, characterized in that, Includes the following steps: S1. Obtain the text requirements from the XR virtual pre-visualization task. Reference image Adjacent keyframes or volume data and 3D assets Asset thumbnails or renderings and scene constraint vectors The text requirements, reference images, keyframes, 3D asset labels, and scene relationships are normalized and quality-marked. S2. Using the CLIP multimodal semantic alignment model, textual requirements, reference images, and asset thumbnails are mapped to a unified semantic space to obtain textual requirement vectors. Reference image vector and asset image vectors ,in: ； And calculate text requirements With assets Multimodal similarity: ； in, For text encoders, For image encoders, and These are text similarity weights and image similarity weights, respectively, indicated by superscript. Indicates transpose or inner product; S3. Generate the condition vector for the Latent Diffusion model based on the text demand vector, reference image vector, and scene constraint vector. ,in: ； in, The conditional mapping matrix, This represents vector concatenation; and generates or refines smoke, lighting effects, background, transitions, or pre-show keyframes in the latent space according to the following denoising relationship: ； in, and For adjacent latent variables in the diffusion process, and The diffusion coefficient is... For noise prediction networks, For noise scale, It is Gaussian noise; S4. Encode adjacent keyframes or volume data into latent vectors: ； in, This is a latent space encoder; and latent space temporal interpolation is performed between the latent vectors, when linear interpolation is used: ； When using spherical interpolation: ； in, For interpolation latent vectors, These are the interpolation coefficients. for and The included angle; S5, interpolation latent vector With condition vector Input generator Output intermediate frames or volume data And calculate the time-series smoothing loss. : ； ； in, The temporal smoothing loss is used to suppress abrupt changes in smoke, flames, lighting effects, or volumetric effects between adjacent keyframes, where the time interval is specified. S6. Extract CLIP semantic features, LFD shape features, CNN / DNN visual category features, text labels, and scene relationship features from 3D assets, and calculate the asset recommendation score by integrating multimodal similarity, shape similarity, scene relationship score, historical reuse stability, and manual rejection penalty: ； in, to These are the weighting coefficients. For shape similarity, Score the scene relationship. For historical reuse stability, For the purpose of artificially refusing punishment; S7. Calculate the confidence level of the simulation scheme based on the asset recommendation score, time-series smoothing loss, reconstruction loss, and number of manual rejections: ； in, For the Sigmoid function, to The confidence weighting coefficient is... To reconstruct the loss; when the confidence level of the pre-play scheme meets the threshold requirement, output the special effects keyframes, continuous pre-play sequences, asset recommendation results, matching reasons, similarity sub-items, and archived records; when the confidence level of the pre-play scheme does not meet the threshold requirement or the number of manual rejections exceeds the threshold, trigger regeneration, re-retrieval, or manual review.

2. The XR virtual pre-show content generation method based on latent space temporal interpolation and semantic asset recommendation according to claim 1, characterized in that, The CLIP multimodal semantic alignment model encodes director text requirements, reference images, asset thumbnails, and asset tags, and is used to calculate text-image, text-asset, and image-asset similarities, enabling the system to perform cross-modal matching based on natural language descriptions, visual references, and 3D asset preview images.

3. The XR virtual pre-show content generation method based on latent space temporal interpolation and semantic asset recommendation according to claim 1, characterized in that, The Latent Diffusion model uses at least one of the following as generation conditions: text conditions, reference image conditions, and scene conditions. It generates or corrects smoke, lighting effects, background, transitions, or pre-show keyframes in the latent space and sends the generation results to the latent space temporal interpolation step.

4. The XR virtual pre-show content generation method based on latent space temporal interpolation and semantic asset recommendation according to claim 1, characterized in that, The latent space temporal interpolation includes at least one of linear interpolation, spherical interpolation, and diffusion-guided interpolation; when the mean absolute error or temporal smoothing loss of the interpolation result exceeds a preset threshold, the system supplements keyframes, adjusts interpolation coefficients, or switches interpolation methods.

5. The XR virtual pre-show content generation method based on latent space temporal interpolation and semantic asset recommendation according to claim 1, characterized in that, The LFD shape feature is constructed from contour, depth, or projected view features from multiple perspectives, and its expression is: ； The shape similarity is calculated based on the LFD shape characteristics of the two assets: ； in, and For the assets to be compared, This is the shape distance scale parameter.

6. The XR virtual pre-show content generation method based on latent space temporal interpolation and semantic asset recommendation according to claim 1, characterized in that, The scene relationship score is calculated based on a knowledge graph constructed from 3D assets, asset categories, material styles, applicable scenarios, lens uses, historical adoption records, and reasons for manual rejection. The scene relationship score is as follows: ； in, For relational types, For relation weights, For the set of edges in the asset knowledge graph, This is an indicator function that takes a value of 1 if the condition is true and 0 otherwise.

7. The XR virtual pre-show content generation method based on latent space temporal interpolation and semantic asset recommendation according to claim 1, characterized in that, The semantic recommendation ranking is configured with different recommendation weights based on live broadcast previews, film and television set previews, virtual press conference sets, or e-commerce scenarios. Increase the weight of semantic alignment and diversity in the concept exploration phase, increase the weight of diffusion reconstruction and temporal smoothing in the keyframe refinement phase, and increase the weight of sorting feedback in the asset reuse phase.

8. The XR virtual pre-show content generation method based on latent space temporal interpolation and semantic asset recommendation according to claim 1, characterized in that, The pre-rehearsal archive includes saving text requirements, reference images, keyframes, latent vectors, diffusion seeds, conditional vectors, asset numbers, recommendation scores, generation model versions, asset ranking model versions, director confirmation results, alternative assets, and rejection reasons. The pre-rehearsal archive is used as prior samples for generation and recommendation when similar pre-rehearsal requirements occur in the future.

9. The XR virtual pre-show content generation method based on latent space temporal interpolation and semantic asset recommendation according to claim 1, characterized in that, The asset recommendation results include candidate assets, matching reasons, CLIP similarity sub-items, LFD shape similarity sub-items, scene relationship scores, alternative assets, and low-confidence review markers, which can be used by directors or art directors to adopt, replace, reject, or review and confirm the recommendation results.

10. An XR virtual pre-show content generation system, characterized in that, The system is used to execute the XR virtual preview content generation method based on latent space temporal interpolation and semantic asset recommendation as described in any one of claims 1 to 9. The system includes a preview requirement parsing module, a CLIP multimodal semantic alignment module, a Latent Diffusion latent space diffusion generation module, a latent space temporal interpolation module, a 3D asset feature retrieval module, a semantic recommendation ranking module, a confidence judgment module, and a preview archiving and reuse module. The preview requirement parsing module is used to parse text requirements, reference images, keyframes, 3D assets, and scene constraints. The CLIP multimodal semantic alignment module is used to obtain multimodal semantic vectors. The Latent Diffusion latent space diffusion generation module is used to generate or refine special effects keyframes. The latent space temporal interpolation module is used to generate continuous preview sequences. The 3D asset feature retrieval module and the semantic recommendation ranking module are used to retrieve and recommend 3D assets. The confidence judgment module is used to trigger output, regeneration, re-retrieval, or manual review. The preview archiving and reuse module is used to output preview results and form archive records.

Images