Immersive content generation model fast adaptation method and device, equipment and medium

Abstract

The application provides a method and device for quickly adapting an immersive content generation model, equipment and a medium. The method comprises: acquiring multi-field immersive data, constructing meta-task data sets containing time sequence segments, and performing meta-training on a basic generation model using the meta-task data sets; acquiring a few-shot labeled data of a target scene, inputting the few-shot labeled data into the basic generation model after meta-training, and weighting the parameter update gradient according to the motion intensity of the target scene time sequence sample in the basic generation model integrated with a lightweight fine-tuning module to obtain a preliminary adaptation model; based on the prior knowledge of the target scene, the candidate samples generated by the preliminary adaptation model are constrained and screened to obtain pseudo samples with consistent semantics, and a target scene expansion data set is constructed based on the pseudo samples; and the preliminary adaptation model is optimized by layered migration using the target scene expansion data set to obtain an immersive content generation model adapted to the target scene.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and virtual reality technology, specifically to a method, apparatus, device, and medium for rapid adaptation of an immersive content generation model. Background Technology

[0002] In the field of immersive content generation, model adaptation for new scenarios typically employs a combination of pre-training and fine-tuning. However, existing technologies still have shortcomings when facing limited sample conditions: on the one hand, traditional fine-tuning strategies rely heavily on labeled data, making it difficult to converge fully in scenarios with scarce samples. Furthermore, full-parameter updates can easily lead to overfitting of the model to a limited number of samples, affecting the general feature representations obtained during pre-training. On the other hand, existing generative models often fail to adequately model the dynamic motion patterns and multimodal interaction relationships in immersive content, resulting in problems such as temporal jumps and spatial structure distortions in the generated results, making it difficult to meet the high fidelity and low latency requirements of applications such as virtual reality. In addition, conventional data augmentation methods lack domain-specific prior knowledge constraints, making it difficult to generate high-quality pseudo-data with consistent semantics and structure under extremely limited sample conditions, further limiting the model's adaptation accuracy and generalization ability. Summary of the Invention

[0003] The purpose of this application is to provide a method, apparatus, device and medium for rapid adaptation of immersive content generation models, so as to solve the problems of low adaptation efficiency, high risk of overfitting, poor temporal stability of generated content and insufficient consistency of target scenes in existing immersive content generation models in target scenes with few samples.

[0004] Firstly, a method for rapid adaptation of an immersive content generation model is provided, including:

[0005] Acquire immersive data from multiple domains, construct a meta-task dataset containing time-series fragments, and use the meta-task dataset to perform meta-training on the basic generative model;

[0006] Acquire few sample labeled data of the target scene, input the few sample labeled data into the basic generative model after meta-training, and in the basic generative model with integrated lightweight fine-tuning module, weight the parameter update gradient according to the motion intensity of the temporal samples of the target scene to obtain a preliminary adaptation model.

[0007] Based on prior knowledge of the target scene, the candidate samples generated by the preliminary adaptation model are constrained and screened to obtain semantically consistent pseudo-samples, and the target scene augmented dataset is constructed based on the pseudo-samples.

[0008] The preliminary adaptation model is then subjected to hierarchical transfer optimization using the target scene-expanded dataset to obtain an immersive content generation model adapted to the target scene.

[0009] Optionally, constructing the meta-task dataset containing time-series segments includes:

[0010] Data from multiple different domains was sampled from the filtered multi-domain immersive data;

[0011] The sampled data is segmented according to a preset time window to obtain multiple time-series segments;

[0012] Within each sampling domain, a portion of the time series segments are divided into a support set, and another portion of the time series segments are divided into a query set;

[0013] The support set and the query set are combined into a meta-task to construct the meta-task dataset.

[0014] Optionally, performing meta-training on the base generative model using the meta-task dataset includes:

[0015] Input the support set of the meta-task into the basic generative model, calculate the in-task loss, and update the initial parameters of the basic generative model in-task based on the in-task loss to obtain the task-specific parameters.

[0016] The task-specific parameters are applied to the query set of the meta-task to calculate the meta-loss;

[0017] The general parameters of the basic generative model are updated based on the meta-loss until the meta-validation result meets the preset convergence condition.

[0018] Optionally, the lightweight fine-tuning module includes an adaptive feature fusion layer disposed between the feature extraction layer and the temporal coding layer of the basic generative model, and a temporal-aware adjustment layer disposed before the output layer of the basic generative model; the adaptive feature fusion layer is used to fuse spatial features and target scene-specific features, and the temporal-aware adjustment layer is used to enhance the temporal dynamic features of the target scene.

[0019] Optionally, when fine-tuning the time-aware parameters of the basic generative model, the underlying parameters of the basic generative model are kept frozen, and the top-level parameters of the basic generative model, the parameters of the adaptive feature fusion layer, and the parameters of the time-aware adjustment layer are updated.

[0020] Optionally, the step of weighting the parameter update gradient based on the motion intensity of the time-series samples of the target scene includes:

[0021] Motion intensity is calculated based on the pixel differences between adjacent frames in the temporal samples of the target scene.

[0022] The mask weights are determined based on the motion intensity to construct a dynamic mask matrix, wherein the smaller the pixel difference, the larger the corresponding mask weight;

[0023] The dynamic mask matrix is ​​multiplied element-wise with the original gradient vector during the parameter update process to obtain the weighted gradient vector;

[0024] The trainable parameters are updated based on the weighted gradient vector.

[0025] Optionally, the constraint screening of candidate samples generated by the preliminary adaptation model based on prior knowledge of the target scene includes:

[0026] Transform the scene structure constraints, physical motion laws and / or lighting constraints of the target scene into quantifiable constraints.

[0027] Candidate samples are generated using the preliminary adaptation model;

[0028] The candidate samples are matched and verified against the quantifiable constraints.

[0029] Candidate samples that satisfy the quantifiable constraints are identified as pseudo samples.

[0030] Optionally, the step of performing hierarchical transfer optimization on the preliminary adaptation model using the target scene-expanded dataset includes:

[0031] The target scene augmented dataset is input into the preliminary adaptation model, and general features and target features are extracted;

[0032] Apply cross-domain contrastive loss constraints to the general features;

[0033] Apply target scene semantic loss constraints to the target features;

[0034] The preliminary adaptation model is jointly optimized based on the cross-domain contrast loss and the target scene semantic loss.

[0035] Optionally, during the hierarchical transfer optimization process, a progressively decreasing learning rate is applied to the network layers used to extract general features, while a progressively increasing learning rate is applied to the network layers used to extract target features.

[0036] Optionally, before obtaining an immersive content generation model adapted to the target scenario, the method further includes:

[0037] Perform spatiotemporal consistency and / or detail fidelity verification on the generated model after hierarchical transfer optimization, and if the verification passes, determine the generated model after hierarchical transfer optimization as an immersive content generation model adapted to the target scenario.

[0038] Secondly, a device for rapid adaptation of an immersive content generation model is provided, including:

[0039] The meta-training module is used to acquire multi-domain immersive data, construct a meta-task dataset containing time-series fragments, and perform meta-training on the basic generative model using the meta-task dataset.

[0040] The fine-tuning module is used to obtain few sample labeled data of the target scene, input the few sample labeled data into the basic generative model after meta-training, and in the basic generative model integrated with the lightweight fine-tuning module, the parameter update gradient is weighted according to the motion intensity of the temporal samples of the target scene to obtain a preliminary adaptation model.

[0041] An expansion module is used to constrain and filter candidate samples generated by the initial adaptation model based on prior knowledge of the target scene, obtain semantically consistent pseudo samples, and construct an expanded dataset of the target scene based on the pseudo samples.

[0042] The optimization module is used to perform hierarchical transfer optimization on the preliminary adaptation model using the target scene expanded dataset to obtain an immersive content generation model adapted to the target scene.

[0043] Thirdly, a communication device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0044] Fourthly, a readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0045] Fifthly, a computer program product is provided, comprising a computer program or instructions that, when executed, implement the method described in the first aspect.

[0046] In this embodiment, multi-domain immersive data is first acquired and a meta-task dataset containing temporal segments is constructed. Then, the meta-task dataset is used to perform meta-training on the basic generative model. Since the meta-task dataset contains immersive data and its temporal segments from different domains, the basic generative model can learn transferable spatial and temporal features between different immersive scenes during the meta-training stage, thereby providing a better parameter foundation for few-sample adaptation of the target scene.

[0047] After meta-training, a small number of labeled data samples of the target scene are input into the meta-trained base generative model. Within this base generative model, which integrates a lightweight fine-tuning module, the parameter update gradients are weighted according to the motion intensity of the time-series samples of the target scene. This process ensures that the parameter update process matches the dynamic changes of the time-series samples of the target scene, reducing interference from drastic motion segments or abnormally changing segments on model parameter updates. Furthermore, it mitigates the risk of overfitting caused by full-scale parameter fine-tuning under limited sample conditions, thus obtaining a preliminary adapted model with initial adaptability to the target scene.

[0048] After obtaining the initial adaptation model, the candidate samples generated by the initial adaptation model are constrained and filtered based on prior knowledge of the target scene. Candidate samples that satisfy the prior knowledge of the target scene are identified as semantically consistent pseudo-samples. Since these pseudo-samples have been filtered by prior knowledge such as scene structure constraints, physical motion laws, and / or lighting constraints, they have a higher consistency with the target scene. Therefore, they can improve the effectiveness of target scene augmentation datasets when the number of real labeled samples is limited.

[0049] Furthermore, by expanding the dataset with the target scene, hierarchical transfer optimization is performed on the initially adapted model. This allows the model to maintain the general transferability developed during the meta-training phase while further enhancing the learning of target scene-specific features. By processing general and target features hierarchically and combining corresponding loss constraints and learning rate update strategies, the optimization conflict between maintaining general features and adapting to the target scene can be reduced, thereby improving the model's stability in adapting to the target scene.

[0050] Therefore, this application can improve the adaptation efficiency of the immersive content generation model, reduce the risk of overfitting, and improve the temporal stability, spatial rationality and consistency with the target scene when there are few labeled samples in the target scene. This is achieved through the synergistic processing of meta-training, gradient weighted fine-tuning based on motion intensity, pseudo-sample expansion with prior constraints, and hierarchical transfer optimization. Attached Figure Description

[0051] Figure 1 This is one of the flowcharts of a rapid adaptation method for an immersive content generation model provided in the embodiments of this application;

[0052] Figure 2 This is the second flowchart of a rapid adaptation method for an immersive content generation model provided in the embodiments of this application;

[0053] Figure 3 This is a structural diagram of a rapid adaptation device for an immersive content generation model provided in an embodiment of this application;

[0054] Figure 4 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application. Detailed Implementation

[0055] In the embodiments of this application, the terms "first," "second," and "third" are used to distinguish identical or similar items with essentially the same function and purpose. Those skilled in the art will understand that the terms "first," "second," and "third" do not limit the quantity or execution order, nor are they necessarily required to be different. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0056] It should be understood that in the embodiments of this application, "at least one" refers to one or more; "multiple" refers to two or more. Furthermore, the word "equal to" in this application can be used in conjunction with "greater than" or "less than". When "equal to" and "greater than" are used together, the technical solution of "greater than" is adopted; when "equal to" and "less than" are used together, the technical solution of "less than" is adopted.

[0057] In the embodiments of this application, the terms "of," "corresponding (relevant)," "corresponding," "associated (related)," and "mapped" may sometimes be used interchangeably. It should be noted that when no distinction is emphasized, the concepts or meanings expressed are consistent.

[0058] like Figure 1 As shown in the figure, this embodiment provides a method for rapid adaptation of an immersive content generation model, which includes steps 11 to 14.

[0059] Step 11: Obtain multi-domain immersive data, construct a meta-task dataset containing time-series fragments, and use the meta-task dataset to perform meta-training on the basic generative model.

[0060] In one implementation, multi-domain immersive data refers to data originating from multiple immersive content application domains. The multi-domain immersive data may include at least one of three-dimensional geometric scene data, volumetric video data, and virtual reality interactive content data. Specifically, three-dimensional geometric scene data can be used to characterize scene spatial structure, object outlines, and texture information; volumetric video data can be used to characterize the continuous motion state of objects in three-dimensional space; and virtual reality interactive content data can be used to characterize user interaction behavior, scene response relationships, and changes in interaction timing.

[0061] In one implementation, before constructing the meta-task dataset, multi-domain immersive data can be filtered. This filtering may include data integrity filtering, data diversity filtering, and data correlation filtering. Specifically, data integrity filtering removes data with missing frames, texture corruption, or pose tracking failures; data diversity filtering ensures that the retained data covers different scene complexities and varying dynamic change intensities; and data correlation filtering retains samples with common semantic features or common physical constraints. Through these filtering methods, the impact of abnormal data on the subsequent meta-training process can be reduced.

[0062] In one implementation, the meta-task dataset refers to a dataset organized according to a meta-learning training method, and the meta-task dataset contains time-series segments. Constructing a meta-task dataset containing time-series segments may include: sampling data from multiple different domains from filtered multi-domain immersive data; segmenting the sampled data according to a preset time window to obtain multiple time-series segments; within each sampled domain, dividing a portion of the time-series segments into a support set and another portion into a query set; and combining the support set and the query set into a meta-task to construct the meta-task dataset.

[0063] Specifically, data from different domains can originate from different scene types, object categories, interaction methods, or dynamic change patterns. The preset time window can be a continuous image sequence with a preset number of frames, or a video clip or interactive behavior segment within a preset duration. The support set is used to perform in-task learning, and the query set is used to calculate the meta-loss and evaluate the in-task update performance. Each meta-task can correspond to an immersive content sub-scene, an object category, an interaction behavior type, or a dynamic change pattern.

[0064] In one implementation, the base generative model refers to a generative model used to generate immersive content. The base generative model can be used to generate 3D scenes, volumetric videos, virtual reality interactive content, or other immersive content. Optionally, the base generative model may include a spatial feature extraction network and a temporal feature encoding network, wherein the spatial feature extraction network is used to extract scene structure, object contours, and texture features, and the temporal feature encoding network is used to extract dynamic relationship changes between adjacent frames or consecutive temporal segments.

[0065] In one implementation, the base generative model can employ a model structure combining a convolutional neural network and a transformer network, where the convolutional neural network is used to extract spatial features, and the transformer network is used to extract temporal dependency features. The base generative model may also include an output layer, which outputs the immersive content generation result based on the spatial and temporal dependency features.

[0066] In one implementation, meta-training of a basic generative model using a meta-task dataset includes: inputting the support set of the meta-task into the basic generative model, calculating the in-task loss, and updating the initial parameters of the basic generative model in-task based on the in-task loss to obtain task-specific parameters; applying the task-specific parameters to the query set of the meta-task and calculating the meta-loss; and updating the general parameters of the basic generative model based on the meta-loss until the meta-validation result meets the preset convergence condition.

[0067] Specifically, in-task losses can include at least one of reconstruction loss, semantic loss, pose loss, texture loss, or temporal prediction loss. Task-specific parameters refer to temporary parameters obtained after the base generative model completes in-task updates to the support set of a specific meta-task. These task-specific parameters are used to calculate the meta-loss on the query set of the corresponding meta-task. The meta-loss is used to evaluate the generalization ability of the base generative model on the query set of the same meta-task after updates to a small number of support set samples.

[0068] Optionally, meta-training can employ a model-independent meta-learning algorithm. When using a model-independent meta-learning algorithm, the meta-training process includes inner-layer updates and outer-layer updates. Inner-layer updates calculate the in-task loss based on the support set of each meta-task and obtain task-specific parameters; outer-layer updates calculate the meta-loss based on the query set of each meta-task and update the general parameters of the underlying generated model according to the meta-loss. Optionally, outer-layer updates can employ a second-order gradient update method, or a first-order approximate update method depending on computational resources and training efficiency requirements.

[0069] In one implementation, the meta-validation result refers to the result obtained by evaluating the transfer performance of the base generative model on the meta-validation task. Preset convergence conditions may include at least one of the following: the meta-validation loss is lower than a preset loss threshold, the change in meta-validation loss is lower than a preset change threshold over several consecutive training rounds, the meta-validation metric reaches a preset metric threshold, or the number of training rounds reaches a preset maximum.

[0070] In this way, the basic generative model can learn shared spatial structure features, temporal dynamic features, and semantic association features across different immersive scenes in multiple meta-tasks, and obtain a general initial parameter state suitable for rapid updates with few samples. This reduces the dependence on a large number of labeled samples in the target scene adaptation stage and improves the convergence efficiency of subsequent fine-tuning of the target scene with few samples.

[0071] Step 12: Obtain few sample labeled data of the target scene, input the few sample labeled data into the basic generative model after meta-training, and in the basic generative model with integrated lightweight fine-tuning module, weight the parameter update gradient according to the motion intensity of the time series samples of the target scene to obtain a preliminary adaptation model.

[0072] In one implementation, the target scenario refers to a specific immersive content application scenario to be adapted, such as a virtual exhibition hall, a digital twin operation and maintenance scenario, an industrial simulation training scenario, or a virtual reality interaction scenario. Few-sample labeled data refers to a small number of data samples in the target scenario that contain labeled information. The labeled information may include at least one of semantic labels, posture parameters, scene structure annotations, motion state annotations, or interaction state annotations. Optionally, the few-sample labeled data may include 3 to 10 labeled samples, or a number of labeled samples below a preset sample threshold.

[0073] In one implementation, the few-sample labeled data can be preprocessed before being input into the meta-trained base generative model. This preprocessing may include data normalization, temporal alignment, and semantic enhancement. Data normalization ensures that the target scene data is scaled in line with the data from the meta-training stage; temporal alignment matches the temporal length or position of dynamic samples to the model input requirements; and semantic enhancement converts semantic labels into semantic embedding information that can be input into the model.

[0074] In one implementation, the lightweight fine-tuning module refers to an additional network structure integrated into the base generative model for target scene adaptation. The lightweight fine-tuning module includes an adaptive feature fusion layer positioned between the feature extraction layer and the temporal coding layer of the base generative model, and a temporal-aware adjustment layer positioned before the output layer of the base generative model. The adaptive feature fusion layer is used to fuse spatial features and target scene-specific features, while the temporal-aware adjustment layer is used to enhance the temporal dynamic features of the target scene.

[0075] Specifically, the feature extraction layer refers to the network layer in the basic generative model used to extract spatial features, texture features, contour features, or local structural features of the input data. The temporal coding layer refers to the network layer in the basic generative model used to extract temporal dependencies between consecutive frames, consecutive volumetric video segments, or consecutive interactive segments. The output layer refers to the network layer in the basic generative model used to output the generated result or intermediate generated representation.

[0076] In one implementation, the adaptive feature fusion layer may employ at least one of a linear mapping layer, an attention fusion layer, a gated fusion layer, or a lightweight adaptation layer. The target scene-specific features may include specific object layouts, unique texture styles, specific lighting and shadow distributions, semantic label distributions, or interaction behavior patterns within the target scene.

[0077] In one implementation, the temporal-aware adjustment layer can employ at least one of gated recurrent units, long short-term memory networks, temporal attention modules, or one-dimensional temporal convolutional modules. By setting the temporal-aware adjustment layer, the basic generative model can better represent the motion trends and dynamic changes between consecutive frames in the target scene.

[0078] Optionally, before updating the parameters of the lightweight fine-tuning module, the weight parameters of the adaptive feature fusion layer and the temporal-aware adjustment layer can be orthogonally initialized to reduce the correlation between parameters and improve the stability of the fine-tuning process.

[0079] In one implementation, when adapting the base generative model to the target scene, a strategy of freezing the lower-level parameters and fine-tuning the upper-level parameters can be adopted. That is, the lower-level parameters used to extract general features in the base generative model are kept unchanged or basically unchanged, while only the parameters of the lightweight fine-tuning module and some of the upper-level parameters in the base generative model are updated, in order to reduce the risk of overfitting under few-sample conditions.

[0080] In one implementation, the target scene temporal sample refers to sample data with temporal continuity in the target scene, such as consecutive image frames, consecutive volumetric video frames, consecutive 3D point cloud sequences, or consecutive interactive behavior sequences. The motion intensity of the target scene temporal sample can be determined based on pixel differences, optical flow changes, pose changes, keypoint displacements, or 3D coordinate changes between adjacent frames.

[0081] In one implementation, the parameter update gradient is weighted according to the motion intensity of the temporal samples of the target scene, including: calculating the motion intensity based on the pixel differences between adjacent frames in the temporal samples of the target scene; determining the mask weights based on the motion intensity to construct a dynamic mask matrix, where the smaller the pixel difference, the larger the corresponding mask weight; multiplying the dynamic mask matrix element-wise with the original gradient vector in the parameter update process to obtain a weighted gradient vector; and updating the trainable parameters based on the weighted gradient vector.

[0082] Specifically, higher motion intensity indicates more pronounced object movement, viewpoint changes, lighting changes, or interaction state changes within the corresponding time segment; lower motion intensity indicates smoother temporal changes within the corresponding time segment. A dynamic mask matrix is ​​a weight matrix composed of multiple mask weights used to adjust the gradient of parameter updates. The dynamic mask matrix can be set to correspond to the parameters to be updated, feature channels, time segments, or sample batches.

[0083] In one implementation, the original gradient vector refers to the gradient updated by backpropagation of the loss function before dynamic masking weighting. The weighted gradient vector is the gradient vector obtained by element-wise multiplying the dynamic mask matrix and the original gradient vector. Trainable parameters may include top-level parameters, adaptive feature fusion layer parameters, temporal-aware adjustment layer parameters, or other lightweight fine-tuning module parameters.

[0084] In one implementation, the preliminary adaptation model refers to the intermediate model obtained after the base generative model, which has undergone meta-training, has completed parameter adaptation on the target scene's few-sample labeled data. This preliminary adaptation model retains the general parameters obtained by the base generative model during the meta-training stage and further includes lightweight fine-tuning module parameters updated based on the target scene's few-sample labeled data, as well as some top-level parameters.

[0085] By using the above method, the spatial features, semantic features, and temporal change features of the target scene can be introduced into the model adaptation process without updating all parameters of the basic generative model. This makes the parameter update process match the dynamic change of the temporal samples of the target scene, reduces the interference of violent motion segments or abnormal temporal segments on the model parameter update, thereby improving the temporal stability of the initial adaptation model and reducing the risk of overfitting in the small sample fine-tuning process.

[0086] Step 13: Based on prior knowledge of the target scene, the candidate samples generated by the preliminary adaptation model are constrained and screened to obtain semantically consistent pseudo-samples, and the target scene extended dataset is constructed based on the pseudo-samples.

[0087] In one implementation, prior knowledge of the target scene refers to constraint information in the target scene that is predetermined or can be obtained from business rules, physical laws, or spatial layout rules. The prior knowledge of the target scene may include at least one of scene structure constraints, physical motion laws, lighting constraints, object position constraints, interaction logic constraints, or semantic relationship constraints.

[0088] In one implementation, scene structure constraints are used to define the spatial, topological, or layout relationships between objects in a target scene. For example, in a virtual exhibition hall scene, exhibits should be located within display stands or cases, and passageways should not be blocked by fixed objects; in an industrial simulation scene, equipment components should meet preset assembly relationships. Physical motion laws are used to define the speed, direction, continuity, or collision relationships of objects. Lighting and shadow constraints are used to define brightness relationships, lighting direction, shadow distribution, or reflection relationships.

[0089] In one implementation, quantifiable constraints refer to screening conditions that can be represented by numerical thresholds, logical rules, spatial coordinate relationships, or evaluation functions. Transforming prior knowledge of the target scene into quantifiable constraints means converting abstract scene rules into computable and judgeable screening criteria. For example, "objects must not float" can be transformed into "the distance between the bottom of the object and the supporting plane must be less than a preset distance threshold"; "motion should be continuous" can be transformed into "the displacement of key points in adjacent frames must be less than a preset displacement threshold"; and "lighting should be reasonable" can be transformed into "the brightness difference of the target area meets a preset brightness range".

[0090] In one implementation, candidate samples are generated using a preliminary adaptation model. Candidate samples refer to immersive content samples generated by the preliminary adaptation model based on a small number of sample-annotated data of the target scene, semantic conditions, pose conditions, scene structure conditions, or random noise. Candidate samples may include 3D scene samples, volumetric video samples, virtual reality interactive samples, or dynamic sequence samples.

[0091] In one implementation, candidate samples are matched and verified against quantifiable constraints, and candidate samples that satisfy the quantifiable constraints are identified as pseudo-samples. Matching verification refers to the process of determining whether a candidate sample satisfies the quantifiable constraints. Semantically consistent pseudo-samples refer to generated samples that, after constraint screening, match the target scene in terms of semantic content, scene structure, motion patterns, or interaction logic.

[0092] In one implementation, the target scene augmentation dataset refers to a dataset composed of real, few-sample labeled data of the target scene and selected pseudo-samples. Optionally, before constructing the target scene augmentation dataset, the selected pseudo-samples can be style-consistent to ensure that the texture style, color distribution, lighting, or resolution of the pseudo-samples are consistent with those of the real samples in the target scene. Optionally, the ratio between real samples and pseudo-samples can be controlled to reduce the risk of weakening the features of real samples due to an excessive number of pseudo-samples.

[0093] By using the above methods, the scale of data available for training can be expanded when the number of real labeled samples in the target scene is limited. Furthermore, by using prior knowledge constraints, the number of generated samples that do not conform to the structural rules, physical laws, or semantic logic of the target scene can be reduced from entering the subsequent optimization process, thereby improving the effectiveness and reliability of expanding the target scene dataset.

[0094] Step 14: Use the target scene to expand the dataset and perform hierarchical transfer optimization on the initial adaptation model to obtain an immersive content generation model adapted to the target scene.

[0095] In one implementation, hierarchical transfer optimization refers to further optimizing the parameters of the initially adapted model using a target scene augmented dataset, while constraining general features and target features separately during the optimization process. Unlike the few-sample parameter adaptation in step 12, hierarchical transfer optimization is performed on a target scene augmented dataset composed of real few-sample labeled data and semantically consistent pseudo-samples, in order to further enhance the model's adaptability to the target scene.

[0096] In one implementation, general features refer to features shared across different immersive scenes, such as at least one of the following: basic object outlines, general spatial structures, basic texture patterns, common motion patterns, or basic semantic relationships. The network layer used to extract general features is typically a feature extraction network layer located near the input or an intermediate layer. Target features refer to features unique to or more strongly relevant to the target scene, such as at least one of the following: target scene-specific texture style, specific lighting effects, specific object layouts, specific interaction logic, specific semantic label distribution, or specific motion patterns. The network layer used to extract target features is typically a network layer located near the output, a lightweight fine-tuning module, or a target scene adaptation layer.

[0097] In one implementation, hierarchical transfer optimization of the preliminary adaptation model is performed using a target scene augmented dataset, including: inputting the target scene augmented dataset into the preliminary adaptation model and extracting general features and target features; applying cross-domain contrast loss constraints to the general features; applying target scene semantic loss constraints to the target features; and jointly optimizing the preliminary adaptation model based on the cross-domain contrast loss and the target scene semantic loss.

[0098] In one implementation, cross-domain contrastive loss refers to a loss used to constrain the discriminability and consistency of general features across different domains or different samples. Imposing cross-domain contrastive loss constraints on general features can keep the general features extracted by the current model similar to those obtained during the meta-training stage, and reduce the damage to general transferability during hierarchical transfer optimization.

[0099] In one implementation, the target scene semantic loss refers to the loss used to ensure that the generated content maintains consistency with the target scene's semantic labels, object categories, scene structure annotations, or interaction state annotations. Imposing the target scene semantic loss constraint on the target features can encourage the generative model to learn the specific semantic distribution and scene representation methods within the target scene.

[0100] In one implementation, joint optimization refers to constructing a total loss function based on cross-domain contrastive loss and target scene semantic loss, and updating the parameters of the initial adaptation model according to the total loss function. Optionally, the total loss function can be a weighted sum of cross-domain contrastive loss and target scene semantic loss. Optionally, reconstruction loss, texture loss, temporal consistency loss, or regularization loss can also be further introduced to improve the stability of the generated results.

[0101] In one implementation, during the hierarchical transfer optimization process, a progressively decreasing learning rate can be applied to the network layer used for extracting general features, while a progressively increasing learning rate can be applied to the network layer used for extracting target features. A progressively decreasing learning rate means that as the number of training epochs increases, the learning rate of the network layer used for extracting general features gradually decreases, thereby reducing the magnitude of parameter updates in the general feature layer. A progressively increasing learning rate means that as the number of training epochs increases, the learning rate of the network layer used for extracting target features gradually increases, or adaptively increases within a preset range, thereby enhancing the target feature layer's ability to learn the target scene's specific textures, lighting, layout, and semantic distributions.

[0102] In one implementation, the immersive content generation model adapted to the target scene refers to the final model obtained after the initial adaptation model has been hierarchically transferred and optimized using an expanded dataset of the target scene. This model is used to generate immersive content that matches the spatial structure, temporal changes, and semantic constraints of the target scene based on the input data of the target scene.

[0103] By employing the above methods, general features and target features can be constrained separately during the hierarchical transfer optimization process, reducing the optimization conflict between preserving general capabilities and learning target scene-specific features. On the one hand, cross-domain contrastive loss and a gradually decreasing learning rate help maintain the general transfer capabilities acquired during the meta-training stage; on the other hand, target scene semantic loss and a gradually increasing learning rate help strengthen the learning of target scene-specific features. Therefore, the adaptability stability of the generative model in the target scene and the fidelity of the generated content can be improved.

[0104] Optionally, after obtaining an immersive content generation model adapted to the target scene, spatiotemporal consistency and / or detail fidelity verification can be performed on the immersive content generation model. Spatiotemporal consistency verification is used to determine whether the immersive content output by the model meets preset requirements in the temporal and spatial dimensions. Temporal consistency verification can be achieved by calculating the differences in optical flow, pixel changes, or pose changes between adjacent frames in the generated dynamic scene sequence. Spatial consistency verification can be achieved by judging the 3D structural integrity of the generated scene, the rationality of the spatial position of objects, and the topological relationships between objects. Detail fidelity verification is used to determine whether the generated content closely approximates the real sample of the target scene in terms of texture, structure, lighting, and semantic details.

[0105] In one implementation, peak signal-to-noise ratio (PSNR) and structural similarity index can be used as objective evaluation indicators of detail fidelity. When the PSNR and structural similarity index respectively meet their corresponding preset thresholds, the generated content can be determined to meet the detail fidelity requirements. Optionally, manual evaluation can also be combined to score the immersiveness, semantic accuracy, and dynamic fluency of the generated scene.

[0106] In one implementation, if validation fails, the process of generating pseudo-samples and augmenting data for the target scene can be returned to, the constraints corresponding to the prior knowledge of the target scene can be adjusted, the augmented dataset for the target scene can be reconstructed, and hierarchical transfer optimization can be performed again. This process can further improve the usability of the final model in the target scene.

[0107] From the processing stage perspective, step 11 can form the meta-training stage, step 12 can form the initial adaptation stage for the target scene, step 13 can form the target scene data expansion stage, and step 14 can form the target scene hierarchical transfer optimization stage. Through the above steps 11 to 14, this embodiment first uses multi-domain immersive data to construct a meta-task dataset containing time-series fragments and performs meta-training; then, it uses a small number of labeled data of the target scene and weights the parameter update gradient according to the motion intensity of the time-series samples of the target scene to obtain an initial adaptation model; subsequently, it filters semantically consistent pseudo-samples based on prior knowledge of the target scene and constructs an expanded dataset for the target scene; further, it uses the expanded dataset of the target scene to perform hierarchical transfer optimization on the initial adaptation model to obtain an immersive content generation model adapted to the target scene.

[0108] Therefore, this embodiment can improve the low-sample adaptation efficiency of the immersive content generation model, reduce the risk of overfitting, and enhance the temporal consistency, spatial rationality, and semantic consistency between the generated content and the target scene when the number of labeled samples in the target scene is small. This is achieved through the collaborative processing of meta-training of the meta-task dataset, gradient weighting based on motion intensity, prior knowledge constraint screening of the target scene, and hierarchical transfer optimization.

[0109] like Figure 2 As shown, a specific implementation process of the method in this application may include a cross-domain meta-task pre-training stage, a time-aware parameter fine-tuning stage, and a domain constraint adaptation stage. Specifically, the cross-domain meta-task pre-training stage can be used to achieve step 11, which involves acquiring multi-domain immersive data, constructing a meta-task dataset, and performing meta-training; the time-aware parameter fine-tuning stage can be used to achieve step 12, which involves acquiring target scene few-sample labeled data, integrating a lightweight fine-tuning module, constructing a dynamic mask matrix, and obtaining a preliminary adapted model; the domain constraint adaptation stage can be used to achieve steps 13 and 14, which involve pseudo-sample generation, constructing a target scene expanded dataset, hierarchical transfer optimization, and spatiotemporal consistency and / or detail fidelity verification.

[0110] like Figure 2As shown, to address the challenge of adapting immersive content generation models in scenarios with few samples, this paper proposes a three-core approach: cross-domain meta-task pre-training, temporal-aware parameter fine-tuning, and domain-constraint adaptation. This approach enables the model to quickly and accurately adapt to the target scene. First, in the cross-domain meta-task pre-training stage, immersive data from multiple domains (including 3D scenes, volumetric videos, and virtual reality (VR) interactive content) are collected to construct a meta-task dataset. The basic generation model is trained to learn cross-scene general feature representations and rapid transfer capabilities, laying the foundation for subsequent adaptation. Second, in the temporal-aware parameter fine-tuning stage, a lightweight fine-tuning module is designed for a very small number of labeled samples from the target scene. An orthogonal initialization strategy is used to optimize the adaptation layer parameters, and a dynamic masking mechanism is combined to adjust the parameter update intensity, balancing adaptation efficiency and generation stability. Finally, in the domain-constraint adaptation stage, semantically consistent pseudo-samples are generated to augment the data based on prior knowledge of the target scene. Hierarchical transfer learning is used to achieve accurate matching between general knowledge and target features. Adaptation is completed after verification of spatiotemporal consistency and detail fidelity. The overall flowchart is shown below. Figure 2 As shown. The specific implementation steps are as follows:

[0111] 1) Cross-domain meta-task pre-training phase.

[0112] In the cross-domain meta-task pre-training stage, multi-domain immersive data is collected and a meta-task dataset adapted to the training requirements of meta-learning is constructed. Meta-training is then performed on the basic generative model, enabling it to fully acquire general feature representation capabilities across different immersive scenarios and rapid transfer capabilities with few samples. This lays a solid foundation of meta-learning capabilities for accurate optimization in the subsequent time-aware parameter fine-tuning stage (target scene adaptation) and domain constraint adaptation stage, ensuring the efficiency and stability of the subsequent adaptation process. The specific implementation steps of this stage are as follows:

[0113] Step (1): Multi-domain immersive data collection and screening.

[0114] First, a multi-dimensional, cross-domain data acquisition process is performed, covering 3D scene data, volumetric video data, and VR interactive content data. High-precision 3D scanners, multi-camera simultaneous shooting equipment, and VR tracking devices are used to ensure the spatiotemporal continuity and accuracy of the data.

[0115] The data filtering process is then performed, with the following filtering criteria: ① Data integrity: Incomplete data with missing frames, texture damage, or pose tracking failures are removed; ② Data diversity: Ensure that data from each domain covers different scene complexities and dynamic change intensities (such as slowly moving natural landscapes and fast-interactive VR game scenes); ③ Data correlation: Retain samples with common semantic features from data from each domain (such as the patterns of light and shadow changes in different scenes and the physical constraints of object motion). After filtering, an initial cross-domain dataset is formed.

[0116] Step (2): Construction of the meta-task dataset.

[0117] Based on the selected cross-domain dataset, meta-task partitioning and dataset construction are performed. The core of this step is to construct meta-tasks through a combination of "multi-domain sampling, few-sample partitioning, and temporal augmentation," ensuring that the meta-tasks accurately match the two-layer learning mechanism of meta-learning: "in-task learning - meta-parameter update." The specific implementation is as follows: ① Meta-task definition: Each meta-task corresponds to an "immersive content generation task for a single sub-scene," such as "2-bedroom, 1-living room 3D furniture generation," "human running motion volumetric video generation," and "virtual car showroom roaming interactive generation," all of which are independent meta-tasks. ② Multi-domain sampling: Randomly sample N different domains (N≥5) from the cross-domain dataset to ensure that the meta-task covers sufficient cross-domain feature differences, enabling the model to learn more generalized features during meta-training, avoiding overfitting to a single domain, and improving the universality of meta-learning transfer; ③ Few-shot partitioning: Within each sampling domain, randomly select K labeled samples as the support set (few samples, used for rapid model adaptation, K=3~10) and M samples as the query set (test samples, used to verify the adaptation effect) according to the proportion of few-shot scenarios. The number of samples in the query set is the support set. The support set is 2 to 3 times larger, and the model's few-sample adaptation ability is trained through the "support set learning - query set verification" mode; ④ Temporal segment integration: For dynamic immersive data such as volumetric videos and VR interactive content, it is divided into temporal segments according to fixed time windows (such as 0.5s / window). Each temporal segment is included as an independent sample in the meta-task to ensure that the meta-task data contains spatiotemporal dimension information, adapts to the dynamic generation requirements of immersive content, and at the same time makes the general capabilities learned by meta-learning cover the temporal dimension, avoiding the problem of missing temporal features in subsequent adaptation.

[0118] The final constructed meta-task dataset contains no fewer than 1,000 meta-tasks, which are divided into a meta-training set, a meta-validation set, and a meta-test set in an 8:1:1 ratio for subsequent meta-training and performance validation of the model.

[0119] Step (3): Construction of basic generative model and meta-training.

[0120] First, we construct the basic generative model architecture: This solution adopts a generator architecture based on the fusion of Transformer and Convolutional Neural Network (CNN). It balances spatial feature extraction and long-term temporal dependency capture, providing a suitable model carrier for meta-learning training and ensuring the model can effectively learn cross-scene general features. Specific architecture design: ① Feature extraction layer: A CNN module (using the first 10 layers of ResNet50) is used to extract spatial features of the scene (such as scene texture and object contours). The kernel sizes of the CNN module are 7×7 and 3×3, with strides of 2 and 1 respectively. Batch normalization is applied. The Normalization (BN) layer suppresses overfitting, ensuring the stability of feature extraction and laying the foundation for general feature transfer in meta-learning; ② Temporal encoding layer: A Transformer module (6 encoder layers, 8 multi-head attention heads, and 512 hidden layers) is introduced, inputting the spatial feature sequence extracted by CNN into the Transformer to accurately capture the motion trajectory of dynamic objects, the temporal logic of VR interaction, and other long temporal dependencies, adapting to the dynamic generation requirements of immersive content, while enabling the model to learn general transfer capabilities in the temporal dimension through meta-learning; ③ Output layer: A structure combining transposed convolution and temporal convolution is adopted. The core function is to restore the feature map size and enhance the temporal continuity of the output content, ultimately outputting immersive content generation results with the same scale as the input data, ensuring adaptation to subsequent meta-learning fine-tuning requirements. The mathematical formula for transposed convolution (used to restore the feature map size) is:

[0121] , (1)

[0122] in, The temporal characteristics output by the Transformer. Here is the weight matrix of the transposed convolution kernel (kernel size is 4×4), and s=2 is the stride of the transposed convolution. For the transposed convolution bias term, This is the transpose convolution operation. The mathematical formula for temporal convolution (used to enhance temporal continuity) is:

[0123] , (2)

[0124] in, This is the temporal convolution kernel weight matrix (the kernel size is 3×3); For temporal convolution bias terms, This is a one-dimensional temporal convolution operation. Generate the final immersive content output.

[0125] Then, meta-training is performed: Model-Agnostic Meta-Learning (MAML) is used as the core training framework, enabling the model to acquire rapid transfer capabilities through meta-learning. The specific training process is performed as follows: ① Parameter initialization: Initialize the model's general parameters. The learning rate was set to 0.001 for the meta-learning model and 0.01 for the in-task learning model. Parameters were initialized using a Xavier normal distribution to ensure balanced gradients across layers. ② In-task learning: A batch of meta-tasks (batch size = 32) was randomly selected from the meta-training set. For each meta-task, the support set was input into the model, and the in-task loss function was calculated (a combination of L1 loss and perceptual loss was used, with L1 loss weighting 0.3 and perceptual loss weighting 0.7; L1 loss was used to constrain pixel-level errors by calculating the mean pixel difference between generated and real samples; perceptual loss was calculated using the Euclidean distance from the output features of the 5th convolutional layer of the pre-trained VGG16 network to constrain feature-level differences). The gradient was calculated based on the loss function, and the model parameters were updated using gradient descent to obtain task-specific parameters. ③ Meta-loss calculation: update the parameters The query set applied to this meta-task is used to calculate the mean generation loss of the query set as the meta-loss; ④ Meta-parameter update: Based on the backpropagation of the meta-loss, the Adam optimizer is used to update the general parameters θ of the model. The optimization objective is to minimize the mean meta-loss of all meta-tasks, so that the model can be quickly fine-tuned and adapted in different meta-tasks with a small number of samples; ⑤ Convergence judgment: Repeat the above steps ② to ④ until the generation accuracy (semantic consistency, detail fidelity) on the meta-validation set is stably converged (accuracy fluctuation does not exceed 2% for 10 consecutive epochs), then stop training and obtain the basic generative model with cross-domain general feature representation capability.

[0126] 2) Timing-aware parameter fine-tuning stage.

[0127] This stage targets a very small number of labeled samples for the target scene (sample size <= 10). Through a lightweight fine-tuning module and optimization strategies, it achieves rapid adaptation of the base model to the target scene, balancing adaptation efficiency and generation stability. The specific steps are as follows:

[0128] Step (1): Preprocessing of small sample data for the target scenario.

[0129] First, a small number of labeled data samples of the target scene are obtained, and data preprocessing actions are performed: ① Data standardization: The pixel values ​​and pose parameters of the target scene data are normalized to the [0,1] interval to maintain consistency with the data scale in the pre-training stage; ② Temporal alignment: For dynamic samples, the dynamic time warping algorithm is used to align the temporal segments of the target scene with the temporal lengths of the pre-training data to ensure that the temporal feature dimensions match; ③ Semantic enhancement: The semantic labels of the labeled samples are One-Hot encoded to generate semantic embedding vectors, which serve as additional inputs for model fine-tuning.

[0130] Step (2): Lightweight fine-tuning module design and integration.

[0131] To avoid overfitting and inefficiency caused by full-parameter fine-tuning, a lightweight adaptation layer is designed as the fine-tuning module. The specific design and integration scheme are as follows: ① Adaptive layer structure design: a. Adaptive feature fusion layer: Two adaptive feature fusion layers are inserted between the CNN feature extraction layer and the Transformer temporal coding layer of the basic generative model. Each layer uses a 1×1 convolutional kernel to achieve channel dimension adaptation (the number of convolutional kernels is consistent with the number of feature channels in the basic model). The convolutional kernel weights are initialized to a random normal distribution. The spatial features extracted by the CNN are fused with the specific features of the target scene through 1×1 convolution; b. Temporal-aware adjustment layer: A temporal-aware adjustment layer is added before the model output layer, introducing a gated recurrent unit (Gated Recurrent Unit). The GRU (Generative Unit) captures the temporal dynamic features of the target scene. The input dimension of the GRU is the top-level feature dimension of the basic model (512), the hidden layer dimension is set to 512, and the output dimension is consistent with the output feature dimension of the basic model. The feature weights at different temporal positions are adjusted through the gating mechanism of the GRU to enhance the temporal dynamic adaptation of the target scene. ② Model integration and parameter freezing strategy: The designed fine-tuning module is integrated with the basic generative model, and the strategy of "bottom freezing-top fine-tuning" is adopted: the bottom parameters of the basic model are fixed (the first 80% of the convolutional layers and Transformer layers, i.e. the first 8 layers of ResNet50 and the first 5 layers of Transformer), and only the parameters of the fine-tuning module and a small number of parameters of the top layer of the basic model (the last 20% of the feature fusion layer parameters, i.e. the 9th-10th layers of ResNet50 and the 6th layer of Transformer) are trained. According to statistics, the total number of fine-tuning parameters does not exceed 10% of the basic model, which greatly improves the adaptation efficiency and reduces the risk of overfitting.

[0132] Step (3): Orthogonal initialization and dynamic mask parameter optimization.

[0133] First, the fine-tuning module parameter initialization action is performed, and an orthogonal initialization strategy is adopted: the weight parameters of the adaptive feature fusion layer and the temporal awareness adjustment layer are initialized through orthogonal matrices. The initialization process is as follows: generate a random matrix that follows a standard normal distribution, perform singular value decomposition on the matrix, and take the orthogonal matrix as the parameter initialization matrix to ensure that the column vectors of the parameter matrix are pairwise orthogonal, reduce the redundant correlation between parameters, reduce the risk of gradient vanishing, and accelerate model convergence.

[0134] Then, a dynamic mask parameter update operation is performed, specifically: ① Constructing a temporal dynamic mask matrix: A mask matrix is ​​generated based on the motion intensity of temporal samples of the target scene. The motion intensity is calculated through the pixel differences between adjacent frames, and the specific formula is as follows:

[0135] , (3)

[0136] in, Let the mask weights be the mask weights corresponding to the i-th frame and the j-th frame. , These are the pixel value matrices for the i-th and j-th frames, respectively, where σ is an adjustment coefficient (set to 0.1). The smaller the pixel difference, the better. The larger the value, the higher the mask weight. ② Gradient-weighted update: During parameter update, the dynamic mask matrix is ​​element-wise multiplied with the original gradient vector to obtain the weighted gradient vector. The core formula is:

[0137] , (4)

[0138] in, For the weighted gradient vector, M is the original gradient vector, and M is the dynamic mask matrix. This weighting operation gives higher update weights to parameters corresponding to time segments with smooth motion (small pixel differences, large mask weights), while reducing the update magnitude of parameters corresponding to time segments with drastic motion (large pixel differences, small mask weights). This avoids interference from feature fluctuations in dynamic segments on stable model learning, while ensuring sufficient learning of static basic features. ③ Optimizer and training parameter settings: A mini-batch gradient descent optimizer is used to perform parameter updates. The specific training parameter configuration is: batch size = 4, initial learning rate set to 0.0005, momentum coefficient set to 0.9, and training iterations ranging from 50 to 100. During training, the validation loss is calculated after each iteration (using optical flow field difference loss to calculate the temporal consistency loss of the target sample generation). When the validation loss drops to a preset threshold (below 0.05) or the number of iterations reaches the upper limit, the fine-tuning process is stopped, resulting in a preliminary generative model adapted to the target scene.

[0139] 3) Domain constraint adaptation stage.

[0140] In this stage, the model adaptation accuracy is optimized through pseudo-sample augmentation and hierarchical transfer learning. The adaptation effect is ensured through multi-dimensional verification, and the final adaptation is completed.

[0141] Step (1): Target scene pseudo-sample generation and data augmentation.

[0142] Based on prior knowledge of the target scene, pseudo-sample generation and data augmentation are performed. The specific implementation method is as follows: The core of this step is to introduce prior constraints of the target scene to generate semantically consistent pseudo-samples, avoid pseudo-samples from deviating from the core features of the target scene, and ensure the effectiveness of data augmentation. The specific implementation process is as follows: ① Prior knowledge modeling: The prior knowledge of the target scene (such as scene structure constraints and physical motion laws) is transformed into quantifiable constraints. Taking the "science fiction exhibition hall" scene as an example, the constraints include: the layout of the display cases must be symmetrical along the central axis of the exhibition hall (position constraint), the speed of the characters' movement does not exceed 5m / s (motion constraint), and the brightness of the holographic projection must be 30% higher than the ambient brightness (light and shadow constraint); ② Candidate sample screening: The candidate samples of the target scene generated by the model after preliminary adaptation are semantically matched and verified with the above quantified constraints. Samples with chaotic layout, movement that violates physical laws, and light and shadow that do not meet the requirements are eliminated, and candidate samples with consistent semantics are selected; ③ Style enhancement: A style transfer network (based on the CycleGAN architecture, which integrates the prior constraints of the target scene into the generator loss function) is used to enhance the style of the selected candidate samples. Through style enhancement, the texture style and lighting effects of the pseudo samples are made consistent with the real samples of the target scene; ④ Ratio control: The number of pseudo samples is strictly controlled to ensure that the ratio of real samples to pseudo samples in the expanded dataset is 1:5 to 1:8, so as to avoid the deviation of real features due to too many pseudo samples, and finally form the target scene expanded dataset.

[0143] Step (2): Hierarchical transfer learning optimization.

[0144] To achieve accurate matching between general knowledge and target features and avoid interference from general knowledge, the following hierarchical transfer learning is implemented: ① Feature layering and input: The target scene extended dataset is processed hierarchically into a "general feature layer - target feature layer" and input into the model. The bottom layer inputs general scene features, including basic object outlines and general motion laws. These features correspond to the common features extracted from the meta-learning pre-training of the basic generative model's bottom module, ensuring stable transfer of general meta-learning capabilities to the target scene. The top layer inputs target scene-specific features, including the holographic projection effect of the science fiction exhibition hall and specific interactive logic. These features correspond to the unique attributes carried by the target scene-specific samples in the extended dataset, accurately anchoring the target scene adaptation requirements. This hierarchical input achieves the orderly transfer of the two types of features, laying the foundation for subsequent accurate fusion. ② Hierarchical loss function design: A differentiated hierarchical loss function is designed to adapt to the meta-learning transfer optimization requirements, achieving precise constraints on the two types of features. The underlying layer employs cross-domain contrastive loss, the core purpose of which is to ensure consistency between the currently extracted general features and the general knowledge learned during the meta-learning pre-training stage, thereby preventing the degradation of general capabilities. The loss function formula is as follows:

[0145] , (5)

[0146] in The instantaneous general features extracted by the underlying feature extraction module (the CNN module built in step 3 of stage 1). The general features that are solidified after the convergence of the 3-ary training in step 1. Cosine similarity is used to quantify the degree of fit between two types of features. The top layer employs target scene semantic loss, calculated based on the semantic label differences between real and generated samples, using the cross-entropy loss function, with the formula:

[0147] , (6)

[0148] in For real semantic tags, To generate semantic prediction probabilities for samples, the learning effect of target scene-specific features can be enhanced. ③ Dynamic learning rate fine-tuning: Based on the above hierarchical loss function, the model is fine-tuned a second time, using a dynamic learning rate strategy to guide the direction of knowledge fusion. The initial learning rate of the general feature layer is set to 0.0005, decreasing by 0.0001 every 10 iterations, eventually reaching 0.0001. By weakening the parameter update intensity of this layer, the general transferability of meta-learning is preserved to the maximum extent. The initial learning rate of the target feature layer is also set to 0.0005, increasing by 0.0001 every 10 iterations, eventually reaching 0.001. By strengthening the parameter update intensity of this layer, the model is guided to focus on learning target scene-specific features, achieving a two-way balance of "preservation of general capabilities - enhancement of target features". The number of iterations for the second fine-tuning is controlled between 30 and 50, until the top-level semantic loss converges (the loss fluctuation does not exceed 1% in 5 consecutive iterations), completing the deep fusion of general knowledge of meta-learning and target scene features.

[0149] Step (3): Verification of spatiotemporal consistency and detail fidelity.

[0150] First, perform spatiotemporal consistency verification: ① Temporal consistency verification: For the dynamic scene sequence generated by the model, calculate the difference in optical flow field between adjacent frames, set a difference threshold (optical flow vector amplitude ≤ 0.1), and count the proportion of frames with excessive differences. If the proportion is less than 5%, the temporal consistency requirement is met; ② Spatial consistency verification: Calculate the 3D structural integrity of the generated scene (missing area ≤ 3%) and the rationality of the spatial position of objects (based on prior knowledge of the target scene, such as the display case must be located on the ground and cannot be suspended), to ensure that there are no obvious deviations in the spatial structure.

[0151] The following steps are performed to verify the fidelity of details: ① Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) are used as quantitative indicators. A PSNR value ≥ 35dB and an SSIM value ≥ 0.9 indicate that the fidelity of details meets the standard. ② Manual evaluation is introduced, with 3-5 professionals scoring the immersion, semantic accuracy, and dynamic fluency of the generated scene (out of 10 points). An average score ≥ 8.5 is considered passing. If the verification passes, the model's adaptation to the target scene is complete. If the verification fails, the process returns to the target scene pseudo-sample generation and data augmentation steps in the domain constraint adaptation stage, adjusts the prior constraints of pseudo-sample generation, and re-executes the augmentation and optimization process until the verification passes.

[0152] See Figure 3 This application provides a rapid adaptation device for an immersive content generation model. The device 300 includes: a meta-training module 301, a fine-tuning module 302, an expansion module 303, and an optimization module 304.

[0153] The meta-training module 301 is used to acquire multi-domain immersive data, construct a meta-task dataset containing time-series segments, and perform meta-training on the basic generative model using the meta-task dataset.

[0154] The fine-tuning module 302 is used to obtain few sample labeled data of the target scene, input the few sample labeled data into the basic generative model after meta-training, and in the basic generative model integrated with the lightweight fine-tuning module, the parameter update gradient is weighted according to the motion intensity of the temporal samples of the target scene to obtain a preliminary adaptation model.

[0155] The expansion module 303 is used to perform constraint screening on the candidate samples generated by the preliminary adaptation model based on prior knowledge of the target scene, to obtain semantically consistent pseudo samples, and to construct an expanded dataset of the target scene based on the pseudo samples.

[0156] The optimization module 304 is used to perform hierarchical migration optimization on the preliminary adaptation model using the target scene expanded dataset to obtain an immersive content generation model adapted to the target scene.

[0157] In one implementation, the meta-training module 301 is further configured to:

[0158] Data from multiple different domains was sampled from the filtered multi-domain immersive data;

[0159] The sampled data is segmented according to a preset time window to obtain multiple time-series segments;

[0160] Within each sampling domain, a portion of the time series segments are divided into a support set, and another portion of the time series segments are divided into a query set;

[0161] The support set and the query set are combined into a meta-task to construct the meta-task dataset.

[0162] In one implementation, the meta-training module 301 is further configured to:

[0163] Input the support set of the meta-task into the basic generative model, calculate the in-task loss, and update the initial parameters of the basic generative model in-task based on the in-task loss to obtain the task-specific parameters.

[0164] The task-specific parameters are applied to the query set of the meta-task to calculate the meta-loss;

[0165] The general parameters of the basic generative model are updated based on the meta-loss until the meta-validation result meets the preset convergence condition.

[0166] In one embodiment, the fine-tuning module 302 is further configured to:

[0167] An adaptive feature fusion layer is set between the feature extraction layer and the temporal coding layer of the basic generative model to fuse spatial features and target scene-specific features;

[0168] A temporal-aware adjustment layer is set before the output layer of the basic generative model to enhance the temporal dynamic features of the target scene;

[0169] The target scene's few-sample labeled data is input into the basic generative model that integrates the adaptive feature fusion layer and the temporal awareness adjustment layer;

[0170] While keeping the underlying parameters frozen, the top-level parameters and the parameters of the lightweight fine-tuning module are updated to complete the parameter fine-tuning under the target scenario.

[0171] In one embodiment, the lightweight fine-tuning module includes an adaptive feature fusion layer disposed between the feature extraction layer and the temporal coding layer of the basic generative model, and a temporal-aware adjustment layer disposed before the output layer of the basic generative model; the adaptive feature fusion layer is used to fuse spatial features and target scene-specific features, and the temporal-aware adjustment layer is used to enhance the temporal dynamic features of the target scene.

[0172] In one embodiment, the fine-tuning module 302 is further configured to:

[0173] Motion intensity is calculated based on the pixel differences between adjacent frames in the temporal samples of the target scene.

[0174] The mask weights are determined based on the motion intensity to construct a dynamic mask matrix, wherein the smaller the pixel difference, the larger the corresponding mask weight;

[0175] The dynamic mask matrix is ​​multiplied element-wise with the original gradient vector during the parameter update process to obtain the weighted gradient vector;

[0176] The trainable parameters are updated based on the weighted gradient vector.

[0177] In one implementation, updating trainable parameters based on the weighted gradient vector may include: increasing the update weights of parameters corresponding to time segments with smooth motion and decreasing the update magnitude of parameters corresponding to time segments with intense motion. This reduces the interference of intense motion segments or abnormal time segments on the parameter update process, improving the temporal stability of the initial adaptation model in the target scene.

[0178] In one embodiment, the expansion module 303 is further configured to:

[0179] Transform at least one of the scene structure constraints, physical motion laws, and lighting constraints of the target scene into quantifiable constraints for sample selection.

[0180] Candidate samples are generated using the preliminary adaptation model;

[0181] The candidate samples are matched and verified with the quantifiable constraints, and candidate samples that satisfy the prior knowledge of the target scenario are selected as pseudo samples.

[0182] The target scene augmented dataset is constructed based on the pseudo-samples obtained through screening.

[0183] In one implementation, the prior knowledge of the target scene includes at least one of scene structure constraints, physical motion laws, lighting constraints, object position constraints, interaction logic constraints, or semantic relationship constraints. The quantifiable constraints can be represented by numerical thresholds, logical rules, spatial coordinate relationships, or evaluation functions.

[0184] In one implementation, the expansion module 303 is further configured to perform style consistency processing on the selected pseudo samples before constructing the target scene expansion dataset, so that the texture style, color distribution, lighting and shadow performance or resolution of the pseudo samples are consistent with the real samples of the target scene.

[0185] In one implementation, the optimization module 304 is further configured to:

[0186] The target scene augmented dataset is input into the preliminary adaptation model, and general features and target features are extracted from the target scene augmented dataset;

[0187] A cross-domain contrastive loss constraint is applied to the general features to maintain the general transferability obtained during the meta-training phase;

[0188] Apply target scene semantic loss constraints to the target features to enhance the learning of target scene-specific features;

[0189] The preliminary adaptation model is jointly optimized based on the cross-domain contrast loss and the target scene semantic loss. During the optimization process, the learning rate of the network layer used to extract general features is gradually reduced, while the learning rate of the network layer used to extract target features is gradually increased.

[0190] In one implementation, the general features refer to features shared across different immersive scenes, including at least one of the following: basic object outlines, general spatial structures, basic texture patterns, common motion patterns, or basic semantic relationships. The target features refer to features unique to or more strongly related to the target scene, including at least one of the following: target scene-specific texture styles, specific lighting effects, specific object layouts, specific interaction logic, specific semantic tag distribution, or specific motion patterns.

[0191] In one implementation, the optimization module 304 is further configured to construct a total loss function based on cross-domain contrastive loss and target scene semantic loss, and update the parameters of the preliminary adaptation model according to the total loss function. Optionally, the total loss function can be a weighted sum of cross-domain contrastive loss and target scene semantic loss.

[0192] In one embodiment, the optimization module 304 is further configured to perform at least one of spatiotemporal consistency verification and detail fidelity verification on the immersive content generation model after obtaining the immersive content generation model adapted to the target scene; the spatiotemporal consistency verification is used to determine whether the immersive content output by the model meets the preset requirements in the time dimension and the spatial dimension; the detail fidelity verification is used to determine whether the generated content is close to the real sample of the target scene in terms of texture, structure, lighting and semantic details.

[0193] Based on the same inventive concept, the principle and beneficial effects of the device 300 provided in the embodiments of this application in solving the problem are the same as those in this application. Figure 1 The principles and beneficial effects of solving the problem in the illustrated embodiments are similar. For the same reason, please refer to the principles and beneficial effects of the aforementioned method embodiments. For the sake of brevity, they will not be repeated here.

[0194] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application. The communication device 400 can be a network device, or a device compatible with a network device, such as a processor, chip, or chip module; or it can be a terminal device, or a device compatible with a terminal device, such as a processor, chip, or chip module. The communication device 400 may include a processor 401. Optionally, the communication device 400 may further include a memory 402 and a computer program or instructions stored on the memory 402. Figure 4 (Not shown in the diagram). The processor 401 and memory 402 are interconnected. Optionally, the communication device 400 may further include a transceiver 403. The processor 401, memory 402, and transceiver 403 can be connected via a bus 404 or other means. The bus is in... Figure 4 The connections between other components are shown in bold lines only and are not intended to be limiting. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0195] The coupling in this embodiment is an indirect coupling or communication connection between devices, units, or modules, which can be electrical, mechanical, or other forms, used for information exchange between devices, units, or modules. This embodiment does not limit the specific connection medium between the processor 401, memory 402, and transceiver 403. Memory 402 may include read-only memory and random access memory, and provides instructions and data to processor 401. A portion of memory 402 may also include non-volatile random access memory.

[0196] Processor 401 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor; optionally, processor 401 can also be any conventional processor.

[0197] Transceiver 403 is used to receive or send data.

[0198] In one implementation, memory 402 is used to store computer programs or instructions; processor 401 is used to call the computer programs or instructions stored in memory 402.

[0199] In the embodiments of this application, the methods provided in the embodiments of this application can be implemented by running a computer program (including program code or instructions) capable of performing the steps involved in the above-described methods on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a CPU, random access memory (RAM), and read-only memory (ROM). The computer program or instructions can be recorded on, for example, a computer-readable recording medium, loaded into the aforementioned computing device through the computer-readable recording medium, and executed therein.

[0200] Based on the same inventive concept, the communication device 400 provided in the embodiments of this application solves the problem in the same way and with the same beneficial effects as this application. Figure 1 The principles and beneficial effects of solving the problem in the illustrated embodiments are similar. For the same reason, please refer to the principles and beneficial effects of the aforementioned method embodiments. For the sake of brevity, they will not be repeated here.

[0201] It should be noted that, for the sake of simplicity, the above embodiments are all described as a series of actions. Those skilled in the art should understand that this application is not limited to the described order of actions, as some steps in the embodiments of this application can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions, steps, modules, or units involved are not necessarily essential to the embodiments of this application.

[0202] In the above embodiments, the descriptions of each embodiment in this application have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0203] The steps of the methods or algorithms described in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in RAM, flash memory, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, read-only optical discs (CD-ROMs), or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Additionally, the ASIC can reside in a network device or terminal device. Alternatively, the processor and storage medium can exist as discrete components in the network device or terminal device.

[0204] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in the embodiments of this application can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0205] Regarding the modules / units included in the various devices and products described in the above embodiments, they can be software modules / units, hardware modules / units, or a combination of both. For example, for various devices and products applied to or integrated into a chip, all of their modules / units can be implemented using hardware methods such as circuits, or at least some modules / units can be implemented using software programs that run on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits; for various devices and products applied to or integrated into a chip module, all of their modules / units can be implemented using hardware methods such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware methods such as circuits. The implementation is achieved through a software program that runs on the processor integrated within the chip module. The remaining modules / units (if any) can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into terminal equipment, each of their modules / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components within the terminal equipment. Alternatively, at least some modules / units can be implemented through a software program that runs on the processor integrated within the terminal equipment, while the remaining modules / units (if any) can be implemented using hardware methods such as circuits.

[0206] The above detailed embodiments further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above are merely specific embodiments of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A method for rapid adaptation of an immersive content generation model, characterized in that, include: Acquire immersive data from multiple domains, construct a meta-task dataset containing time-series fragments, and use the meta-task dataset to perform meta-training on the basic generative model; Acquire few sample labeled data of the target scene, input the few sample labeled data into the basic generative model after meta-training, and in the basic generative model with integrated lightweight fine-tuning module, weight the parameter update gradient according to the motion intensity of the temporal samples of the target scene to obtain a preliminary adaptation model. Based on prior knowledge of the target scene, the candidate samples generated by the preliminary adaptation model are constrained and screened to obtain semantically consistent pseudo-samples, and the target scene augmented dataset is constructed based on the pseudo-samples. The preliminary adaptation model is then subjected to hierarchical transfer optimization using the target scene-expanded dataset to obtain an immersive content generation model adapted to the target scene.

2. The method according to claim 1, characterized in that, The construction of the meta-task dataset containing time-series segments includes: Data from multiple different domains was sampled from the filtered multi-domain immersive data; The sampled data is segmented according to a preset time window to obtain multiple time-series segments; Within each sampling domain, a portion of the time series segments is divided into a support set, and another portion of the time series segments is divided into a query set; The support set and the query set are combined into a meta-task to construct the meta-task dataset.

3. The method according to claim 2, characterized in that, The step of performing meta-training on the basic generative model using the meta-task dataset includes: Input the support set of the meta-task into the basic generative model, calculate the in-task loss, and update the initial parameters of the basic generative model in-task based on the in-task loss to obtain the task-specific parameters. The task-specific parameters are applied to the query set of the meta-task to calculate the meta-loss; The general parameters of the basic generative model are updated based on the meta-loss until the meta-validation result meets the preset convergence condition.

4. The method according to claim 1, characterized in that, The lightweight fine-tuning module includes an adaptive feature fusion layer set between the feature extraction layer and the temporal coding layer of the basic generative model, and a temporal-aware adjustment layer set before the output layer of the basic generative model; the adaptive feature fusion layer is used to fuse spatial features and target scene-specific features, and the temporal-aware adjustment layer is used to enhance the temporal dynamic features of the target scene.

5. The method according to claim 1, characterized in that, The step of weighting the parameter update gradient based on the motion intensity of the time-series samples of the target scene includes: Motion intensity is calculated based on the pixel differences between adjacent frames in the temporal samples of the target scene. The mask weights are determined based on the motion intensity to construct a dynamic mask matrix, wherein the smaller the pixel difference, the larger the corresponding mask weight; The dynamic mask matrix is ​​multiplied element-wise with the original gradient vector during the parameter update process to obtain the weighted gradient vector; The trainable parameters are updated based on the weighted gradient vector.

6. The method according to claim 1, characterized in that, The step of performing hierarchical transfer optimization on the preliminary adaptation model using the target scene-expanded dataset includes: The target scene augmented dataset is input into the preliminary adaptation model, and general features and target features are extracted; Apply cross-domain contrastive loss constraints to the general features; Apply target scene semantic loss constraints to the target features; The preliminary adaptation model is jointly optimized based on the cross-domain contrast loss and the target scene semantic loss.

7. A rapid adaptation device for an immersive content generation model, characterized in that, include: The meta-training module is used to acquire multi-domain immersive data, construct a meta-task dataset containing time-series fragments, and perform meta-training on the basic generative model using the meta-task dataset. The fine-tuning module is used to obtain few sample labeled data of the target scene, input the few sample labeled data into the basic generative model after meta-training, and in the basic generative model integrated with the lightweight fine-tuning module, the parameter update gradient is weighted according to the motion intensity of the temporal samples of the target scene to obtain a preliminary adaptation model. An expansion module is used to constrain and filter candidate samples generated by the initial adaptation model based on prior knowledge of the target scene, obtain semantically consistent pseudo samples, and construct an expanded dataset of the target scene based on the pseudo samples. The optimization module is used to perform hierarchical transfer optimization on the preliminary adaptation model using the target scene expanded dataset to obtain an immersive content generation model adapted to the target scene.

8. A communication device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.

9. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed, they implement the method as described in any one of claims 1 to 6.

Images