Scene-oriented neural representation and model training adaptive sampling methods and systems

Abstract

The application discloses a scene-oriented neural representation and model training adaptive sampling method and system, comprising: establishing a scene parameterized representation, establishing a scene latent space based on scene coding of the scene parameterized representation, and obtaining a scene latent representation; generating a candidate scene based on the scene latent representation and adaptively sampling; generating sample data based on the sampled scene, training a neural model, and then performing feedback evaluation to obtain a feedback signal of the scene; and optimizing the scene latent space through contrast learning based on the similarity between different scene feedback signals for candidate scene generation and adaptive sampling in the next round. The method and system can solve the problems of limited scene change dimension, lack of effective measurement of sampling space, insufficient utilization of downstream model feedback, and difficulty in forming a continuous closed-loop optimization.

Description

Technical Field

[0001] This invention belongs to the technical field of computer graphics, machine learning and intelligent data generation, and specifically relates to a scene-oriented neural representation and model training adaptive sampling method and system. Background Technology

[0002] In recent years, neural models that rely on scene-generated training data have been widely applied to tasks such as neural global illumination, neural denoising, neural rendering, inverse rendering, and related visual reconstruction. The training of these models heavily depends on large-scale, high-coverage training data with a sufficiently diverse difficulty distribution. For tasks that generate training samples from 3D scenes, the training effect depends not only on the model structure itself but also significantly on whether the scene sampling strategy can consistently provide the most valuable training data for the current model.

[0003] Existing approaches typically fall into the following categories: one is to randomly or uniformly sample scene parameters; another is to supplement with several "difficult" scenarios designed based on human experience; and the third is to perform heuristic search or active sampling within a predefined parameter space for specific graphics tasks. These approaches can achieve certain results when the parameter dimensionality is low and the scene structure is relatively simple. However, in training real-world variable scenes, there are often multiple discrete and continuous factors simultaneously, such as object identity, object quantity, object geometry and material, pose and scale, camera parameters, lighting parameters, and background structure. The scene space exhibits characteristics such as high dimensionality, combinatorial explosion, and long-tailed distribution.

[0004] One of the closest prior art applications to this application is the non-patent document "Active Exploration for Neural Global Illumination of Variable Scenes." This document proposes an active exploration approach for neural global illumination training, which improves training efficiency and model performance by searching for more valuable training states in variable scenes. This approach illustrates that not all scene samples are equally important for model improvement during training; identifying and prioritizing more valuable scene regions can potentially reduce ineffective rendering budget.

[0005] However, the aforementioned existing technologies still have significant limitations. First, these methods are typically based on a single scene template or a fixed set of objects, with variations primarily manifested in a few continuous parameters, such as local geometric positions, a few material parameters, or changes in perspective. They struggle to support more general scene editing, such as object replacement, object addition or removal, changes in background structure, and complex lighting combinations. In other words, their "variable scenes" are often still limited to low-dimensional variable changes, rather than being geared towards truly high-dimensional composite scene spaces.

[0006] Secondly, most existing solutions directly search within a manually defined scene parameter space. While this parameter space is interpretable, it often fails to naturally characterize the "training similarity" between different scenes for downstream neural models. For example, two scenes with significantly different geometric parameters may produce similar loss gradients for downstream models; conversely, two scenes with similar parameters may have vastly different training values. Without a learnable latent representation space for scenes, it becomes difficult to organize sampling and transfer within a more suitable metric space.

[0007] Furthermore, existing technologies typically utilize feedback signals in a limited way, often relying only on instantaneous errors, rendering residuals, or a few task metrics. They struggle to comprehensively utilize feedback signals that reflect "informative" information, such as training loss, uncertainty, gradient direction, gradient norm, teacher-student output distribution bias, sample coverage, and sample diversity. This results in insufficient ability of sampling strategies to identify truly high-value samples, thus affecting the utilization rate of training samples.

[0008] Furthermore, in many existing methods, sampling strategies and scene representation learning are disconnected: on the one hand, the way scenes are represented is not continuously reshaped with feedback from downstream models; on the other hand, newly discovered high-value samples are difficult to influence the scene representation space itself. Therefore, these methods often lack a closed-loop mechanism of "scene representation learning—sample generation—downstream training—feedback update," and are prone to problems such as decreased exploration efficiency, insufficient coverage, and inadequate adaptation to long-tailed and difficult scenes after long-term iterations.

[0009] Therefore, a new technical solution is urgently needed: one that can construct a decodeable, measurable, and sampleable latent scene space for high-dimensional scenes containing variable objects; and combine this with various feedback signals generated during the training process of a neural model that generates training data based on the downstream scene to perform adaptive sampling and closed-loop updates of the scene, thereby generating high-training-value samples more effectively within a limited data generation and rendering budget. Factors such as variable cameras, variable lighting, and variable backgrounds can be incorporated into a unified framework according to specific task requirements. Summary of the Invention

[0010] In view of the above, the purpose of this invention is to address the problems existing in the training data generation schemes for neural models that generate training data based on scene-dependent scenarios, such as limited dimensionality of scene changes, lack of effective measurement of sampling space, insufficient utilization of downstream model feedback, and difficulty in forming continuous closed-loop optimization. The invention proposes a scene-oriented neural representation and model training adaptive sampling method and system.

[0011] To achieve the above-mentioned objectives, an embodiment provides a scene-oriented neural representation and model training adaptive sampling method, comprising the following steps: Establish a scene parameterized representation, perform scene encoding based on the scene parameterized representation to build a scene latent space and obtain the scene latent representation; Candidate scenes are generated based on the latent representation of the scene and adaptively sampled; After generating sample data based on the sampled scene and training a neural model, feedback evaluation is performed to obtain the feedback signal of the scene. Based on the similarity between feedback signals from different scenarios, comparative learning is used to optimize the latent space of the scenarios for the next round of candidate scenario generation and adaptive sampling.

[0012] Preferably, a scene parameterized representation is established, including: The 3D scene of the sample data to be generated is represented as a scene state. Each scene state includes at least multiple object states. Each object state includes at least the object body and object parameter information. The object body includes at least one of the following: object identity information, object geometry, and object encoding. The object parameter information includes at least one of the following: material properties, translation parameters, rotation parameters, scale parameters, and lighting parameters. The background is included as a special object in the object state set.

[0013] Preferably, scene encoding based on scene parameterization to establish a scene latent space and obtain a scene latent representation includes: The object ontology in the scene is encoded to obtain object features; based on multiple object features and object parameter information, a set encoding method that is insensitive to object order is used to establish the scene latent space and obtain the scene-level latent representation.

[0014] Preferably, generating candidate scenes based on the latent representation of the scene and adaptively sampling includes: The current scene state is obtained by decoding the scene latent representation. Based on the current scene state, a candidate scene scheme is executed: local perturbation or jump of the current scene latent representation is performed in the scene latent space, object replacement is performed in the current scene state, and / or camera parameter perturbation is performed to generate candidate scenes. After generating sample data based on candidate scenes and training a neural model, feedback evaluation is performed to obtain feedback signals for the candidate scenes. Adaptive sampling and filtering of candidate scenes are performed based on feedback signals from candidate scenes.

[0015] Preferably, the current scene state is obtained by decoding the latent representation of the scene, including: query-based decoding to generate the scene state for a given query target, or set-based joint decoding combined with the encoding process to automatically generate the current scene state.

[0016] Preferably, the feedback signal of the scene is obtained by generating sample data based on the scene and training a neural model, followed by feedback evaluation, including: The scene is rendered using multi-camera parameters to generate sample data. The model is trained using the sample data and feedback evaluation is performed to obtain the scene feedback signal. The feedback signal includes at least the training loss, rendering error, uncertainty estimation, gradient direction, gradient norm, teacher model-student model output distribution bias, sample novelty, or sample diversity index.

[0017] Preferably, adaptive sampling and filtering of candidate scenes based on feedback signals includes: Informational evaluation is performed based on the feedback signals of candidate scenarios to obtain informational scores, which consist of normalized loss, gradient norm, and gradient direction correlation statistics, or may be further incorporated into the differences in output between the teacher model and the student model, sample novelty, and diversity constraints. Adaptive sampling and screening of candidate scenes based on informative scores includes: using weighted sampling, sorting, thresholding, or Markov chain Monte Carlo acceptance / rejection mechanisms to retain high-training-value scenes from multiple candidate scenes based on the improvement of informative scores of candidate scenes, their diversity relationship with existing samples, and proposal probabilities.

[0018] Preferably, the latent space of a scene is optimized for the next round of candidate scene generation and adaptive sampling by comparative learning based on the similarity between feedback signals from different scenes, including: Contrastive learning loss is constructed based on the similarity between feedback signals from different scenarios. Based on the contrastive learning loss, scenarios with similar training value are constrained to be close to each other in the scenario latent space, while scenarios with large differences in training value are kept far apart in the scenario latent space, thereby optimizing the scenario latent space.

[0019] To achieve the above-mentioned objectives, the embodiments also provide a scene-oriented neural representation and model training adaptive sampling system, including: The scene representation module is used to establish a parameterized representation of the scene, perform scene encoding based on the parameterized representation to establish the latent space of the scene and obtain the latent representation of the scene; An adaptive sampling module is used to generate candidate scenes based on the latent representation of the scene and perform adaptive sampling; a sample generation and training module is used to generate sample data based on the sampled scenes, train a neural model, and then perform feedback evaluation to obtain the feedback signal of the scene. The spatial optimization module optimizes the latent space of a scene by comparative learning based on the similarity between feedback signals from different scenes, which is then used for the next round of candidate scene generation and adaptive sampling.

[0020] To achieve the above-mentioned objectives, the embodiments also provide a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-mentioned scene-oriented neural representation and model training adaptive sampling method.

[0021] To achieve the above-mentioned objectives, the embodiments also provide a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the above-mentioned scene-oriented neural representation and model training adaptive sampling method.

[0022] Compared with the prior art, the beneficial effects of the present invention include at least the following: (1) This invention is no longer limited to a single scene template and a small number of continuous variable changes, but can uniformly handle discrete / continuous mixed factors such as object identity, object quantity, geometric material, pose scale, etc., and is applicable to more general high-dimensional variable scenes; for control factors such as camera and lighting, they can also be gradually incorporated into the unified framework according to the specific task requirements.

[0023] (2) By constructing a decodeable and measurable scene potential space, the present invention enables adaptive sampling to no longer be directly limited by the manual parameter space, thereby enabling more effective organization of search and migration in a space that is “more consistent with the training value of downstream models” and improving scene exploration efficiency.

[0024] (3) This invention integrates feedback such as loss, uncertainty, gradient characteristics, and teacher-student distribution differences into the scene information assessment, so that the sampling strategy can more accurately identify high-value, long-tail and difficult samples, and reduce invalid data generation and invalid rendering.

[0025] (4) The present invention establishes a closed-loop update mechanism between scene representation learning, sample generation and downstream model training, so that the potential space of the scene can be continuously reshaped with the downstream training state, thereby improving the sample utilization rate, the ability to cover difficult samples and the training targeting in the later stage of model iteration.

[0026] (5) This invention has strong task versatility and can be used for data generation and training optimization of neural models that rely on scene-generated training data in tasks such as neural global illumination, neural denoising, neural rendering, and inverse rendering. It has clear engineering application value. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of the scene-oriented neural representation and model training adaptive sampling method provided in the embodiment; Figure 2 This is a schematic diagram of scene parameterization and scene coding provided in the embodiment; Figure 3 This is a schematic diagram of adaptive sampling provided in the embodiment; Figure 4 This is a schematic diagram of the potential space optimization shaping provided in the embodiment; Figure 5 This is a schematic diagram of the structure of the scene-oriented neural representation and model training adaptive sampling system provided in the embodiment. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0030] The inventive concept of this invention is as follows: This invention provides a scene-oriented neural representation and adaptive sampling scheme for model training, aiming to construct a latent scene representation capable of representing key information such as object identity, object geometry and material properties, pose, and scale. This latent scene representation can support similarity measurement and sampling search between scenes, and can also be decoded back into an executable scene configuration for generating actual training data. Furthermore, it aims to utilize the scene informative indicators formed by the loss, uncertainty, gradient features, and their combinations generated during the training process of the neural model that generates training data dependent on the scene, to shape the latent scene space, and thereby drive candidate sample generation and adaptive sampling. This allows for the priority generation of the most valuable training samples for the model under limited rendering and training budgets, improving sample utilization, coverage of difficult samples, and model generalization performance.

[0031] like Figure 1 As shown in the embodiment, an adaptive sampling method for scene-oriented neural representation and model training includes the following steps: S1. Establish a scene parameterized representation, perform scene encoding based on the scene parameterized representation to establish the scene latent space and obtain the scene latent representation.

[0032] In the embodiments, such as Figure 2As shown, the 3D scene to be generated as sample data is represented as a scene state. Each scene state includes a set of object states with at least N object states. The background can be included as a special object in the scene object set. Each object state includes at least the object itself and object parameter information. The object itself includes at least one of object identity information, object geometry, and object encoding. The object parameter information includes at least one of material properties, translation parameters, rotation parameters, scale parameters, and lighting parameters. The lighting parameters are used to characterize the object's luminescence properties, including at least one of luminescence map, luminescence intensity, and luminescence color. For non-luminous objects, the lighting parameters take a zero value or a default value. The lighting parameters can be added to the object state as an object parameter to distinguish whether the object has lighting, and can also be treated as an independent control factor or extended variable.

[0033] The 3D scene representation also includes camera parameters related to rendering or data generation, where camera parameters can be used as independent control variables in subsequent candidate scene generation.

[0034] In this embodiment, objects are encoded based on scene parameterization representation to establish a scene latent space and obtain a scene latent representation. Specifically, firstly, the object ontology in the scene is encoded to obtain object features. The object encoding can be derived from object geometry, point cloud, mesh, voxel, material texture, semantic tag, or a combination thereof; object features can also be provided directly by pre-computed embedding or existing encoder.

[0035] Then, based on multiple object features and their parameter information, a scene-level latent representation is obtained using an ensemble encoding method that is insensitive to object order. For example, the scene encoding network is made insensitive to the input order of objects during encoding to adapt to a variable number of object sets. The encoded scene latent representation can be a single latent variable or a scene memory representation composed of multiple latent slots or multiple labels, in order to balance global scene semantics and object-level operability. Camera parameters and lighting parameters may not be directly incorporated into the scene latent representation, or they can be included as additional labels or independent branches in the unified representation.

[0036] S2 generates candidate scenes based on the latent representation of the scene and performs adaptive sampling.

[0037] In this embodiment, based on a scene decoding mechanism, scene parameters are recovered from the scene-level latent representation to obtain an executable current scene state configuration for training sample generation. The decoding mechanism can employ query-based decoding, that is, recovering the scene state corresponding to a given query target, i.e., the object itself and object parameter information, based on object queries or slot queries. This approach facilitates compatibility with existing explicit parameter decoding pipelines.

[0038] The decoding mechanism can also employ ensemble joint decoding to automatically and directly output the scene state, i.e., the object identity and object parameter set. This ensemble joint decoding refers to joint encoding and decoding of the scene parameterized representation and the current latent representation of the scene. In this case, the encoding and decoding networks need to be jointly trained so that the decoding network can automatically recover the scene state. This method supports more flexible scene editing. Each decoding operation recovers the scene state of one scene.

[0039] like Figure 3 As shown, after obtaining the current scene state, a candidate scene scheme is executed: local perturbation or jumps are performed on the latent representation of the current scene in the scene latent space, object replacement is performed in the current scene state, and / or camera parameter perturbation is performed to generate candidate scenes. Of course, explicit object pose perturbation, material parameter perturbation, scale perturbation, or other control variable perturbation can also be introduced, i.e., fine-tuning of object translation, rotation, scale, or material, to enhance local exploration capabilities. Object replacement can be achieved through object codebooks, nearest neighbor retrieval, or constrained semantic retrieval to expand the exploration range while maintaining scene plausibility; the candidate scenes are obtained after decoding following the local perturbation or jump of the current scene latent representation.

[0040] After obtaining candidate scenes, the candidate scenes are rendered using multi-camera parameters to generate multiple sample data. The sample data is then used to train downstream neural models, which include neural global illumination models, neural denoising models, neural rendering models, inverse rendering models, differentiable rendering models, and other neural network models that require scene-based training data generation.

[0041] If the downstream task is neural denoising, noisy images, reference images, and auxiliary geometric buffers (G-buffers) at different sampling rates can be generated; if the downstream task is neural global illumination, corresponding radiance, visibility, or indirect illumination-related training data can be generated.

[0042] While training the neural model, feedback evaluation is also performed to obtain feedback signals for candidate scenes. These feedback signals include at least training loss, rendering error, uncertainty estimation, gradient direction, gradient norm, teacher-student model output distribution bias, sample novelty, or sample diversity index.

[0043] In this embodiment, adaptive sampling and filtering of candidate scenes are also performed based on the feedback signals of the candidate scenes. Specifically, this includes: to quantify the value of a scene for model training, firstly, an informative evaluation is performed based on the feedback signals of the candidate scenes, and an informative score is constructed for each candidate scene. The informative score consists of multiple feedback items, specifically including normalized loss. Gradient norm gradient direction The relevant statistical composition may be further incorporated into the output differences of the teacher-student model. Information on sample novelty and diversity constraints This allows for a more comprehensive portrayal of the training value of candidate scenarios.

[0044] Specifically, the scene informative score can be represented as a weighted combination of multiple feedback items. For scene S, the feedback item vector can be calculated first. : Each feedback item can be normalized separately, and then the informative score can be obtained using the following formula. : in, For the k-th feedback term after normalization, The weight of the k-th feedback item can be dynamically set based on task experience, validation set performance, or the training phase. In practice, it is not required that all feedback items be used simultaneously; one or more of them can be selected.

[0045] Then, based on the informative score, the candidate scenarios are adaptively sampled and screened. Specifically, based on the informative score of the candidate scenario, its relationship with the diversity of existing samples, and the proposal probability, weighted sampling, sorting and screening, threshold screening, or Markov chain Monte Carlo acceptance and rejection mechanism can be used to retain high training value scenarios from multiple candidate scenarios.

[0046] The implementation process of the Markov Chain Monte Carlo Acceptance / Rejection Mechanism is as follows: Let the current scenario be S, the candidate scenario be S′, the forward proposal probability be q(S′|S), and the reverse proposal probability be q(S|S′). Then, the acceptance probability α can be calculated based on the difference between the informative score U(S′) of the candidate scenario S′ and the informative score U(S) of the current scenario S, as well as the forward / reverse proposal probabilities, to balance local exploration, global jump, sample quality, and sample distribution coverage. When a candidate scenario is accepted, it enters the scenario pool and is used for subsequent data generation; when a candidate scenario is not accepted, the current scenario is retained or a new proposal is made.

[0047] S3 generates sample data based on the sampled scene, trains a neural model, and then performs feedback evaluation to obtain the scene's feedback signal.

[0048] In this embodiment, for the scenes stored in the scene pool, sample data is generated and a neural model is trained in the same way as in step S2, and then feedback evaluation is performed to obtain the scene feedback signal. These scene feedback signals are used for shaping and optimizing the potential space of the scene.

[0049] S4 optimizes the latent space of the scene based on the similarity between feedback signals from different scenes through comparative learning for the next round of candidate scene generation and adaptive sampling.

[0050] In the embodiments, such as Figure 4 As shown, a contrastive learning loss is constructed based on the similarity between feedback signals from different scenarios. Based on this loss, the scene encoding network in the encoding process is updated. This constrains scenes with similar training value to be close to each other in the scene latent space, while scenes with significant differences in training value are kept far apart, thus optimizing the scene latent space. This allows the scene latent space to reflect similarity relationships that are more meaningful for downstream model training.

[0051] The optimized scene latent space is used for the next round of candidate scene generation and adaptive sampling, that is, jumping to the scene encoding in step S1 to establish the scene latent space and obtain the scene latent representation. After multiple rounds of iteration, the scene regions in the sampled scene that have the most positive impact on the current model can be gradually parameterized, thus forming a closed loop of "scene representation learning - candidate scene generation - training data rendering - downstream model update - feedback-driven resampling".

[0052] In another implementation, the scene latent space can also be used in conjunction with a scene memory pool or a sample pool. The system stores the scene latent representations, informative scores, and feedback statistics related to downstream models for historical scenes; during subsequent sampling, anchor scenes can be selected from the memory pool based on latent similarity, informative scores, and diversity constraints to improve the revisit efficiency of high-value scenes and reduce duplicate sampling.

[0053] In addition to the above-described implementation methods, the object encoding method, scene encoding network, feedback item combination method, candidate proposal method, and selection criteria can still be adjusted based on subsequent experimental results. The core of this invention lies not in a single network structure, but in the overall closed loop of "constructing a sampleable scene latent space and using downstream model feedback to drive the adaptive generation of training data".

[0054] like Figure 5As shown in the embodiment, an adaptive sampling system for scene-oriented neural representation and model training is also provided, including a scene representation module, an adaptive sampling module, a sample generation and training module, and a space optimization module. The scene representation module is used to establish a parameterized representation of the scene, encode the scene based on the parameterized representation, establish a latent space for the scene, and obtain a latent representation of the scene. The adaptive sampling module is used to generate candidate scenes based on the latent representation of the scene and adaptively sample them. The sample generation and training module is used to generate sample data based on the sampled scenes, train a neural model, and then perform feedback evaluation to obtain feedback signals for the scene. The space optimization module is used to optimize the latent space of the scene through comparative learning based on the similarity between feedback signals from different scenes for the next round of candidate scene generation and adaptive sampling.

[0055] It should be noted that the scene-oriented neural representation and model training adaptive sampling system provided in the above embodiments should be illustrated using the above-described functional module division as an example when performing scene-oriented neural representation and model training adaptive sampling. The functions described above can be assigned to different functional modules as needed, that is, the internal structure of the terminal or server can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the scene-oriented neural representation and model training adaptive sampling system provided in the above embodiments and the scene-oriented neural representation and model training adaptive sampling method embodiments belong to the same concept. The specific implementation process is detailed in the scene-oriented neural representation and model training adaptive sampling method embodiments, and will not be repeated here.

[0056] Based on the same inventive concept, the embodiment also provides a computing device, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-described scene-oriented neural representation and model training adaptive sampling method, specifically including the following steps: S1, establish a scene parameterized representation, perform scene encoding based on the scene parameterized representation to establish the scene latent space and obtain the scene latent representation; S2, generates candidate scenes based on the latent representation of the scene and performs adaptive sampling; S3, based on the sampled scene, generates sample data and trains a neural model, then performs feedback evaluation to obtain the scene's feedback signal; S4 optimizes the latent space of the scene based on the similarity between feedback signals from different scenes through comparative learning for the next round of candidate scene generation and adaptive sampling.

[0057] The computing device provided in this embodiment, at the hardware level, includes not only a processor and memory, but also internal buses, network interfaces, memory, and other hardware required for business operations. The memory is non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the scene-oriented neural representation and model training adaptive sampling method described in S1-S4 above. Of course, besides software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0058] Based on the same inventive concept, the embodiments also provide a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the above-described scene-oriented neural representation and model training adaptive sampling method, specifically including the following steps: S1, establish a scene parameterized representation, perform scene encoding based on the scene parameterized representation to establish the scene latent space and obtain the scene latent representation; S2, generates candidate scenes based on the latent representation of the scene and performs adaptive sampling; S3, based on the sampled scene, generates sample data and trains a neural model, then performs feedback evaluation to obtain the scene's feedback signal; S4 optimizes the latent space of the scene based on the similarity between feedback signals from different scenes through comparative learning for the next round of candidate scene generation and adaptive sampling.

[0059] In this embodiment, the computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data.

[0060] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A scene-oriented neural representation and model training adaptive sampling method, characterized in that, Includes the following steps: Establish a scene parameterized representation, perform scene encoding based on the scene parameterized representation to build a scene latent space and obtain the scene latent representation; Candidate scenes are generated based on the latent representation of the scene and adaptively sampled; After generating sample data based on the sampled scene and training a neural model, feedback evaluation is performed to obtain the feedback signal of the scene. Based on the similarity between feedback signals from different scenarios, comparative learning is used to optimize the latent space of the scenarios for the next round of candidate scenario generation and adaptive sampling.

2. The scene-oriented neural representation and model training adaptive sampling method according to claim 1, characterized in that, Establish a parameterized representation of the scene, including: The 3D scene of the sample data to be generated is represented as a scene state. Each scene state includes at least multiple object states. Each object state includes at least the object body and object parameter information. The object body includes at least one of the following: object identity information, object geometry, and object encoding. The object parameter information includes at least one of the following: material properties, translation parameters, rotation parameters, scale parameters, and lighting parameters. The background is included as a special object in the object state set.

3. The scene-oriented neural representation and model training adaptive sampling method according to claim 1, characterized in that, Scene encoding based on scene parameterization representation establishes the scene latent space and obtains the scene latent representation, including: The object ontology in the scene is encoded to obtain object features; based on multiple object features and object parameter information, a set encoding method that is insensitive to object order is used to establish the scene latent space and obtain the scene-level latent representation.

4. The scene-oriented neural representation and model training adaptive sampling method according to claim 1, characterized in that, Candidate scenes are generated based on the latent representation of the scene and adaptively sampled, including: The current scene state is obtained by decoding the scene latent representation. Based on the current scene state, a candidate scene scheme is executed: local perturbation or jump of the current scene latent representation is performed in the scene latent space, object replacement is performed in the current scene state, and / or camera parameter perturbation is performed to generate candidate scenes. After generating sample data based on candidate scenes and training a neural model, feedback evaluation is performed to obtain feedback signals for the candidate scenes. Adaptive sampling and filtering of candidate scenes are performed based on feedback signals from candidate scenes.

5. The scene-oriented neural representation and model training adaptive sampling method according to claim 4, characterized in that, Decoding the latent representation of a scene yields the current scene state, including: query-based decoding to generate the scene state for a given query target, or set-based joint decoding combined with the encoding process to automatically generate the current scene state.

6. The scene-oriented neural representation and model training adaptive sampling method according to claim 1, characterized in that, After generating sample data based on the scene and training a neural model, feedback evaluation is performed to obtain feedback signals for the scene, including: The scene is rendered using multi-camera parameters to generate sample data. The model is trained using the sample data and feedback evaluation is performed to obtain the scene feedback signal. The feedback signal includes at least the training loss, rendering error, uncertainty estimation, gradient direction, gradient norm, teacher model-student model output distribution bias, sample novelty, or sample diversity index.

7. The scene-oriented neural representation and model training adaptive sampling method according to claim 4, characterized in that, Adaptive sampling and filtering of candidate scenes based on feedback signals includes: Informational evaluation is performed based on the feedback signals of candidate scenarios to obtain informational scores, which consist of normalized loss, gradient norm, and gradient direction correlation statistics, or may be further incorporated into the differences in output between the teacher model and the student model, sample novelty, and diversity constraints. Adaptive sampling and screening of candidate scenes based on informative scores includes: using weighted sampling, sorting, thresholding, or Markov chain Monte Carlo acceptance / rejection mechanisms to retain high-training-value scenes from multiple candidate scenes based on the improvement of informative scores of candidate scenes, their diversity relationship with existing samples, and proposal probabilities.

8. The scene-oriented neural representation and model training adaptive sampling method according to claim 1, characterized in that, Based on the similarity between feedback signals from different scenarios, comparative learning is used to optimize the latent space of the scene for the next round of candidate scene generation and adaptive sampling, including: Contrastive learning loss is constructed based on the similarity between feedback signals from different scenarios. Based on the contrastive learning loss, scenarios with similar training value are constrained to be close to each other in the scenario latent space, while scenarios with large differences in training value are kept far apart in the scenario latent space, thereby optimizing the scenario latent space.

9. A scene-oriented neural representation and model training adaptive sampling system, characterized in that, include: The scene representation module is used to establish a parameterized representation of the scene, perform scene encoding based on the parameterized representation to establish the latent space of the scene and obtain the latent representation of the scene; An adaptive sampling module is used to generate candidate scenes based on the latent representation of the scene and to adaptively sample them; The sample generation and training module is used to generate sample data based on the sampled scene, train the neural model, and then perform feedback evaluation to obtain the feedback signal of the scene. The spatial optimization module optimizes the latent space of a scene by comparative learning based on the similarity between feedback signals from different scenes, which is then used for the next round of candidate scene generation and adaptive sampling.

10. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that... When the one or more processors execute the executable code, they are used to implement the above-described scene-oriented neural representation and model training adaptive sampling method.

Images