Three-dimensional scene understanding and generating method and system combined with auto-reflection learning
By optimizing the 3D generation model through a self-reflective learning mechanism and a consistent reward signal, the problem of insufficient self-reflection in existing technologies is solved, and higher quality and lower cost 3D scene generation is achieved.
Patent Information
- Application Number
- CN202511472980.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-15
- Publication Date
- 2026-01-09
AI Technical Summary
Existing technologies lack a self-reflection mechanism in 3D scene understanding and generation, relying on external evaluation metrics, which leads to inconsistent generation quality and high computational overhead, making it difficult to improve the stability and reliability of the model.
A self-reflective learning mechanism is introduced to optimize the 3D generation model through self-evaluation information and consistency reward signals. The internal reward function is combined to evaluate geometric consistency and semantic rationality, thereby reducing the reliance on external evaluation models.
It improves the stability and reliability of 3D scene generation, reduces computational overhead, and enhances the model's self-reflection ability and generation quality.
Smart Images
Figure CN121304928A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence and machine learning technology, and in particular relates to a method and system for understanding and generating three-dimensional scenes by combining self-reflective learning. Background Technology
[0002] In recent years, with the development of deep learning, artificial intelligence models have made significant progress in fields such as natural language processing and computer vision.
[0003] In language model training, existing training paradigms mainly include Supervised Fine-Tuning (SFT) and Reinforcement Learning Based on Human Feedback (RLHF). SFT can directly guide model learning using high-quality data, but the model lacks self-evaluation capabilities and can only mimic data examples. RLHF optimizes model behavior through reward signals, but relies on an external reward model, resulting in opaque reward modeling and high training complexity. Furthermore, the traditional training process ends with a marker indicating the end of the model output (…). <eos>The process terminates after the output is completed, without utilizing the sequence space after the output is finished. This results in the model lacking a human-like ability to "reflect after the fact," making it difficult to further improve the reliability and robustness of the reasoning.
[0004] In the field of 3D scene understanding and generation, existing technologies mostly employ large-scale visual-language models (VLMs) combined with diffusion models to achieve 3D scene generation and spatial question answering. Typical methods include interfacing 2D image encoders with LLMs, or utilizing geometric information such as point clouds and depth maps to assist in 3D generation. These methods can synthesize new 3D scenes given reference images and viewpoint transformations, and support spatial reasoning tasks. However, existing technologies generally rely on supervised learning or the reconstruction loss of diffusion models, lacking a self-evaluation mechanism for the model's output. In other words, the model can only "generate," but cannot actively check and correct its generation quality during training.
[0005] The aforementioned existing technologies have the following shortcomings:
[0006] 1. Lack of self-reflection mechanism: Whether it is a language model or a 3D generative model, the training process ends immediately after the output is completed, and it fails to perform self-checking and quality evaluation in the "self-reflection space", resulting in the model lacking internal quality control capabilities.
[0007] 2. Over-reliance on external evaluation metrics: 3D generative models are usually evaluated offline using metrics such as FID, KID, and LPIPS. These metrics cannot be directly integrated into the model's training process, which prevents the model from autonomously optimizing its own generation quality during training.
[0008] 3. Insufficient consistency: Although existing methods can generate 3D scenes, inconsistencies between geometric structure and semantic description are prone to occur, especially when synthesizing from new perspectives, where spatial relationship errors or object detail distortions often occur.
[0009] 4. High inference overhead: Some methods attempt to improve generation quality through additional discriminators or reward models, but this introduces additional computational costs and reduces actual deployment efficiency.
[0010] Therefore, it is necessary to propose a new training method that enables the 3D understanding and generation model to enter a self-evaluation stage after output completion. This method combines an internal reward function to actively evaluate the geometric consistency and semantic rationality of the output, thereby reducing the dependence on external evaluation models and improving the stability and reliability of 3D scene generation. Summary of the Invention
[0011] To address the aforementioned technical issues, this invention proposes a method and system for understanding and generating 3D scenes that incorporates self-reflective learning. This method enables the 3D understanding and generation model to enter a self-evaluation phase after output completion, actively evaluating the geometric consistency and semantic rationality of the output using an internal reward function. This reduces reliance on external evaluation models and improves the stability and reliability of 3D scene generation.
[0012] To achieve the above objectives, this invention provides a method for understanding and generating 3D scenes by combining self-reflective learning, comprising:
[0013] Obtain a reference image, task instructions, and viewpoint transformation parameters; perform geometric-semantic feature encoding on the reference image to obtain a visual representation;
[0014] Based on the task instructions, viewpoint transformation parameters, and visual representation, and combined with 3D scene or language-visual features, the task result is output. The language-visual features are obtained by fusing the semantic vector obtained by encoding the task instructions through a language model with the visual vector obtained by encoding the reference image through geometric-semantic features.
[0015] The task results are input into the self-reflection space stage to obtain self-assessment information;
[0016] Based on the self-assessment information and objective evaluation indicators, a consistent reward signal is obtained.
[0017] Based on the task results, self-assessment information, and consistent reward signals, the objective function is obtained;
[0018] The objective function is used to optimize the 3D generation model to obtain the optimal 3D generation model. The optimal 3D generation model is used to achieve new perspective synthesis, point cloud reconstruction, and spatial relationship question answering based on language-visual features of a 3D scene, given a reference image and perspective transformation parameters.
[0019] Optionally, performing geometric-semantic feature encoding on the reference image to obtain a visual representation includes:
[0020] The reference image is digitally converted to obtain a digital image;
[0021] The digitized image is processed using a convolutional neural network to obtain high-level features;
[0022] Based on the high-level features, combined with Gaussian noise, noise-enhanced residual visual labels are obtained;
[0023] The noise-enhanced residual visual markers are linearly mapped and dimensionally aligned to obtain a visual marker sequence with uniform dimensions;
[0024] The visual representation is obtained based on the unified dimension visual tag sequence.
[0025] Optionally, the task results may include: point cloud representation or answer sequence.
[0026] Optionally, based on the task instructions, viewpoint transformation parameters, and visual representation, combined with 3D scene or language-visual features, the output task results include:
[0027] Based on the task instructions, perspective transformation parameters, and visual representation, obtain the condition information for the task execution phase;
[0028] The task instructions are encoded to obtain a semantic embedding matrix;
[0029] Random noise is introduced into the semantic embedding matrix to obtain conditional feature representations;
[0030] The conditional feature representation and the visual feature representation are fused to obtain the fused feature;
[0031] The fusion features and the conditional information are input into the diffusion generator, and combined with the 3D scene, a point cloud representation is obtained. The conditional information is used to ensure that the point cloud representation meets the viewpoint requirements.
[0032] The fused features and the conditional information are input into a sequence decoder, and combined with language-visual features, an answer sequence is obtained. The conditional information is used to ensure that the answer sequence and the image semantics are consistent.
[0033] Optionally, the task results can be input into the self-reflection space stage to obtain self-assessment information, including:
[0034] When the task result is a point cloud representation, the Chamfer distance method is used to calculate the difference between the point cloud representation and the real point cloud to obtain the three-dimensional evaluation result;
[0035] When the task result is an answer sequence, a cross-modal similarity model is used to map the answer sequence and the reference image to the same feature space, and cosine similarity is calculated to obtain spatial evaluation results;
[0036] Based on the three-dimensional assessment results and spatial assessment results, the self-assessment information is obtained.
[0037] Optionally, using the Chamfer distance method to calculate the difference between the point cloud representation and the true point cloud includes:
[0038] ;
[0039] in, For Chamfer distance, For point cloud representation, For a real point cloud, x is The point in the middle, y is The point in the middle, It is a 2-norm (in square form).
[0040] Optionally, calculating cosine similarity includes:
[0041] ;
[0042] in, For semantic similarity scores, For the answer sequence, A text encoding function that maps answer sequences to a semantic feature space. For reference image, This is a visual encoding function that maps a reference image to a semantic feature space.
[0043] Optionally, obtaining a consistent reward signal based on the self-assessment information combined with objective evaluation indicators includes:
[0044] ;
[0045] in, As a consistent reward signal, For self-assessment information, For true indicators, This represents the L1 norm.
[0046] Optionally, based on the task results, self-assessment information, and consistent reward signals, the objective function can be obtained as follows:
[0047] Based on the task results, the task loss is obtained;
[0048] Based on the self-assessment information, the assessed loss is obtained;
[0049] Based on the consistency reward signal, obtain the reinforcement learning loss;
[0050] The objective function is obtained by weighting and integrating the task loss, evaluation loss, and reinforcement learning loss.
[0051] The present invention also provides a three-dimensional scene understanding and generation system that combines self-reflective learning, including: an image acquisition module, a main task module, a self-evaluation module, a consistency reward module, and a model optimization module;
[0052] The image acquisition module is used to acquire a reference image, task instructions and viewpoint transformation parameters, and to perform geometric-semantic feature encoding on the reference image to obtain a visual representation. The language-visual features are obtained by fusing the semantic vector obtained by encoding the task instructions through a language model with the visual vector obtained by encoding the reference image through geometric-semantic features.
[0053] The main task module is used to output task results based on the task instructions, viewpoint transformation parameters, and visual representation, combined with three-dimensional scene or language-visual features;
[0054] The self-assessment module is used to input the task results into the self-reflection space stage to obtain self-assessment information;
[0055] The consistency reward module is used to obtain a consistency reward signal based on the self-assessment information and objective evaluation indicators.
[0056] The model optimization module is used to obtain an objective function based on the task results, self-evaluation information, and consistency reward signal; and to optimize the 3D generation model using the objective function to obtain the optimal 3D generation model. The optimal 3D generation model is used to achieve new perspective synthesis of 3D scenes, point cloud reconstruction, and spatial relationship question answering based on language-visual features, given a reference image and perspective transformation parameters.
[0057] Compared with the prior art, the present invention has the following advantages and technical effects:
[0058] (1) Introducing a noise enhancement mechanism in the input processing stage enables the model to maintain stable output even in the presence of disturbances or noise, which is more robust and generalizable than existing methods.
[0059] (2) Visual features, text prompts and geometric parameters are integrated at the same time during the task execution phase to achieve unified processing under cross-modal conditions. Compared with existing single-modal technologies, it can support more complex tasks.
[0060] (3) A self-reflection space is introduced after the task is completed, which enables the model to have self-reflection ability and actively output evaluation results such as geometric consistency, semantic rationality and format integrity, which significantly improves the reliability of the output compared with existing technologies.
[0061] (4) By internalizing objective indicators into training signals through a consistent reward mechanism, the dependence on external discriminators or reward models is reduced, which significantly reduces training and deployment overhead compared to the traditional RLHF method.
[0062] (5) By jointly optimizing the task loss, evaluation loss and reinforcement learning loss, the consistency between the model output and the real index is improved, and higher quality results can be obtained in both 3D generation and spatial question answering tasks compared with existing methods.
[0063] (6) The self-evaluation mechanism is only enabled during the training phase and does not add extra computational overhead during the inference phase. It is more efficient than the scheme that requires calling an external evaluation model during inference. Attached Figure Description
[0064] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0065] Figure 1 This is a flowchart of a three-dimensional scene understanding and generation method combining self-reflective learning according to an embodiment of the present invention;
[0066] Figure 2 This is an algorithm architecture diagram of a three-dimensional scene understanding and generation method that combines self-reflective learning according to an embodiment of the present invention. Detailed Implementation
[0067] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0068] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0069] This embodiment proposes a method for 3D scene understanding and generation that combines self-reflective learning, such as... Figure 1 As shown, the specific steps include:
[0070] Acquire reference images, task instructions, and viewpoint transformation parameters; perform geometric-semantic feature encoding on the reference images to obtain visual representations;
[0071] Based on the task instructions, viewpoint transformation parameters, and visual representation, combined with 3D scene or language-visual features, the task results are output. The language-visual features are obtained by fusing the semantic vector obtained by encoding the task instructions through a language model with the visual vector obtained by encoding the reference image through geometric-semantic features.
[0072] Input the task results into the self-reflection space to obtain self-assessment information;
[0073] Based on self-assessment information combined with objective evaluation indicators, consistent reward signals are obtained.
[0074] Based on task results, self-assessment information, and consistent reward signals, obtain the objective function;
[0075] The objective function is used to optimize the 3D generation model to obtain the optimal 3D generation model. The optimal 3D generation model is used to achieve new perspective synthesis of 3D scenes, point cloud reconstruction, and spatial relationship question answering based on language-visual features, given a reference image and perspective transformation parameters.
[0076] Specifically, such as Figure 2 As shown, this method specifically includes:
[0077] S1: Receives a reference image, task instructions, and viewpoint transformation parameters; performs geometric-semantic feature encoding on the reference image to obtain a visual representation.
[0078] S2: Based on the input task, perform the synthesis of a 3D scene from the target's perspective using a diffusion model or generate spatial relationship answers by combining language-visual features;
[0079] S3: After the main task output is completed, enter the post-completion stage and output self-evaluation information, including scores for geometric consistency, semantic rationality and format integrity.
[0080] S4: Calculate the true reward function (geometric error, semantic matching degree) and compare it with the model's self-score to generate a consistent reward;
[0081] S5: Jointly optimize task loss, evaluation loss, and consistency reward to update model parameters and improve the stability and reliability of 3D generation results.
[0082] Furthermore, geometric-semantic feature encoding is performed on the reference image to obtain a visual representation, including:
[0083] The reference image is digitally converted to obtain a digital image;
[0084] Using convolutional neural networks to process digital images and obtain high-level features;
[0085] Based on high-level features and combined with Gaussian noise, noise-enhanced residual visual labels are obtained;
[0086] The noise-enhanced residual visual labels are linearly mapped and dimensionally aligned to obtain a visual label sequence with uniform dimensions;
[0087] Visual representations are obtained based on a sequence of visual tags of a unified dimension.
[0088] Specifically, in step S1, the input visual information typically consists of one or more images. To facilitate computer processing, these images are first converted into digital matrices, with each pixel corresponding to a numerical value. Color images usually consist of three channels, with each pixel represented by a (R, G, B) triplet.
[0089] Next, this embodiment processes the image layer by layer using a convolutional neural network. Convolutional layers are used to extract local features, while pooling layers progressively compress the data dimensionality to retain the main information. As the number of layers increases, the output features gradually transition from edge and texture information to higher-level semantic information. The basic operation of a convolutional layer can be simplified as the input features being weighted by the convolutional kernel and then processed by an activation function, for example:
[0090] ;
[0091] in, This represents the output feature map of the l-th convolutional neural network, where * denotes the convolution operation. and These are the weights and biases of the convolutional kernel in layer l, respectively. This is the activation function.
[0092] As the network depth increases, convolution and pooling operations gradually extract higher-level features, resulting in an output feature dimension of [dimensionality missing]. , , To enhance robustness, Gaussian noise is superimposed after obtaining the high-level features. :
[0093] ;
[0094] Received This is called noise-enhanced residual visual labeling, which improves the model's adaptability to input perturbations during the training phase.
[0095] Furthermore, to achieve cross-modal fusion, this embodiment sends the tag to the MLP encoding module for linear mapping and dimension alignment, resulting in a visual tag sequence with unified dimensions. , used for feature fusion in the S2 stage.
[0096] Furthermore, the task results include point cloud representations or answer sequences.
[0097] Furthermore, based on task instructions, viewpoint transformation parameters, and visual representations, combined with 3D scene or language-visual features, the output task results include:
[0098] Based on task instructions, perspective transformation parameters, and visual representation, obtain conditional information for the task execution phase;
[0099] Encode the task instructions to obtain the semantic embedding matrix;
[0100] Random noise is introduced into the semantic embedding matrix to obtain conditional feature representations;
[0101] The conditional feature representation and the visual feature representation are fused to obtain the fused feature;
[0102] The fused features and conditional information are input into the diffusion generator and combined with the 3D scene to obtain a point cloud representation. The conditional information is used to ensure that the point cloud representation meets the viewpoint requirements.
[0103] The fused features and conditional information are input into the sequence decoder, and the answer sequence is obtained by combining language-visual features. The conditional information is used to ensure that the answer sequence and the image semantics are consistent.
[0104] Specifically, in step S2, this embodiment utilizes the visual markers output from S1, the input text prompts, and geometric / viewpoint parameters to complete the task execution phase. The goal of this phase is to generate results that meet the requirements based on different task needs (such as 3D scene generation or spatial question answering).
[0105] First, the inputs for the task execution phase include: one is the visual labeling of the noise-enhanced residuals from step S1. Its form is dimension is A set of vectors, where Indicates the number of visual markers. The three dimensions are: first, the feature dimension; second, the text prompt T, which is the task description in natural language, such as "generate a scene viewed from the right" or "answer the spatial relationship between objects"; and third, the geometric / viewpoint parameters P, which describe the camera pose or viewpoint transformation information in the scene, and can be a combination of rotation matrices and translation vectors. These inputs can also be represented using projection matrices, ray parameters, etc. Together, they constitute the conditional information for the task execution phase.
[0106] Next, this embodiment encodes the input text prompt words. The text prompt T is processed by a pre-trained language model (such as LLaMA3) to obtain the corresponding semantic embedding matrix:
[0107] ;
[0108] in, L represents the prompt word length, and D represents the embedding dimension. In this way, the original natural language instructions are transformed into a set of continuous vector representations, which facilitates subsequent combination with visual features.
[0109] Next, to enhance the model's expressive power and adaptability to input uncertainty, random noise is introduced into the text embedding. This is then concatenated with the embedding and used as input to the supernetwork. Through multi-layer fully connected layers and activation function operations, the conditional feature representation H is obtained:
[0110] ;
[0111] in, and These are the intermediate layer outputs of the conditional feature generation network, used to progressively extract features from the input semantics and noise. The final conditional feature representation H can be directly taken from the output of the last layer. It can also be made by , The intermediate results are combined to obtain the final product.
[0112] Then, the conditional feature H is compared with the visual label output from step S1. The fusion process then branches into different task sub-branches: In the 3D generation task, the fused representation U′ is used as a conditional input to the diffusion generator. The conditional diffusion model progressively denoises the random noise, restoring it to the feature representation of the target viewpoint. And decoded into point cloud or voxel representation. :
[0113] ;
[0114] For diffusion models under given conditions and information The predicted noise results are then used to iterate and denoise the target feature representation z over multiple time steps to obtain a progressively recovered representation. t And further decoded into point cloud or voxel representation P t .
[0115] In the spatial question answering task, the fused representation U′ is used as the conditional vector input sequence decoder, and conditional probability modeling is used to generate the answer sequence:
[0116] ;
[0117] in, This represents the conditional probability distribution for generating the answer sequence under conditional information H. This represents a symbol sequence of length N. The symbol sequence is obtained by sequential sampling using an autoregressive method. This is the answer sequence A output in this embodiment.
[0118] Finally, output the 3D scene. Alternatively, the answer sequence A is used, and an insertion is made at the end of the result. <post-completion>Mark it and proceed to the self-assessment stage.
[0119] Furthermore, the task results are input into the self-reflection space stage to obtain self-assessment information, including:
[0120] When the task result is a point cloud representation, the Chamfer distance method is used to calculate the difference between the point cloud representation and the real point cloud to obtain the three-dimensional evaluation result;
[0121] When the task result is an answer sequence, a cross-modal similarity model is used to map the answer sequence and the reference image to the same feature space, and cosine similarity is calculated to obtain spatial evaluation results;
[0122] Self-assessment information is obtained based on the three-dimensional assessment results and spatial assessment results.
[0123] Specifically, in step S3, this embodiment does not immediately terminate the output after the task is completed, but instead enters a special "self-reflection space" ( <post-completion>In the first stage, the main task outputs from the diffusion generator and the sequence decoder are denoted as P, respectively. t Both A and B are fed into the same space for self-evaluation. During this stage, the model will self-evaluate the main task results it generates, thus possessing a certain degree of self-reflection ability.
[0124] First, the inputs for this step include: one, the results from step S2, which can be the 3D point cloud / voxel Pt from the 3D generation task, or the answer sequence A from the spatial question-and-answer task; and two, the baseline information used for evaluation. For example, the real point cloud P from the 3D generation process. gt , or semantic references paired with input images during spatial question answering.
[0125] Next, after detecting <post-completion>After labeling, the model automatically switches to self-evaluation mode. This mode differs from the main task output; instead of generating a scenario or answer, it outputs a series of evaluation scores and explanatory text to measure the quality of the previous stage's results. This "self-generated evaluation" approach creates an additional "quality control loop" within the model, without relying entirely on an external discriminator or reward model.
[0126] When the task is 3D generation, the system will check the generated point cloud P. t Compared with real point cloud P gt The differences.
[0127] The calculation uses the Chamfer distance (CD), which measures the geometric proximity of two point cloud sets.
[0128] ;
[0129] in, For Chamfer distance, For point cloud representation, For a real point cloud, x is The point in the middle, y is The point in the middle, It is a 2-norm.
[0130] The smaller the value, the closer the geometry of the generated point cloud is to the real point cloud.
[0131] Then, when the task is spatial question answering, the model checks whether the generated answer is semantically consistent with the reference image.
[0132] This embodiment preferably uses a cross-modal similarity model (such as CLIP) to map the answer text and the reference image to the same feature space and calculate the cosine similarity:
[0133] ;
[0134] in, For semantic similarity scores, For the answer sequence, A text encoding function that maps answer sequences to a semantic feature space. For reference image, This is a visual encoding function that maps a reference image to a semantic feature space.
[0135] The higher the score, the more consistent the semantic relationship between the answer and the image.
[0136] Next, the system will check whether the overall structure of the output is standardized. Specifically, the output must include two parts: the result of the main task (point cloud / text answer); and self-assessment content (geometric / semantic / format scores and explanations). If either part is missing, the output is considered incomplete.
[0137] Finally, the model in <post-completion>The stage outputs a structured self-assessment result, including scores for each dimension. , representing geometric consistency, semantic reasonableness, and format completeness, respectively; the self-evaluation text, for example: "The generated scene is geometrically largely consistent with the real point cloud, but there are slight deviations in the details; the semantic description meets the requirements." This self-evaluation result can be used as an optimization signal input for subsequent steps during training, or provided to the user for reference during the inference stage (if needed).
[0138] Furthermore, based on self-assessment information combined with objective evaluation indicators, consistent reward signals are obtained, including:
[0139] ;
[0140] in, As a consistent reward signal, For self-assessment information, For true indicators, This represents the L1 norm.
[0141] Specifically, in step S4, the purpose of this embodiment is to compare the self-evaluation result output by the model in stage S3 with the objective evaluation index, thereby calculating a consistency reward signal. This signal is not only used to optimize the model's parameters during training, but also effectively constrains the model to output results that are closer to the true evaluation during the self-evaluation stage.
[0142] In this step, the input data includes two parts: the model's self-scoring score and the actual evaluation metric. The model's self-scoring score... Derived from the S3 stage, it includes geometric consistency score, semantic reasonableness score, and format integrity score, for example... Real evaluation indicators The results are calculated by the system using objective functions, including: geometric metrics (e.g., calculating the error between the generated point cloud and the real point cloud using Chamfer distance or Hausdorff distance), semantic metrics (e.g., calculating the semantic matching degree between the answer text and the reference image using cross-modal similarity functions such as CLIP similarity), and format metrics (checking whether the output meets the requirements through rules, such as whether it contains the main task results and self-evaluation results).
[0143] Next, the true metrics are calculated item by item. The true geometric metrics are obtained using the Chamfer distance calculation method in step S3.
[0144] Then, the model self-scoring obtained in stage S3 is used. Compared with the actual indicators calculated by the system A step-by-step comparison was conducted.
[0145] If the model's self-assessment score is close to the actual metric, it indicates that the model's self-assessment is accurate; if the difference is large, it indicates that the model's self-assessment is biased and needs to be corrected through training and optimization. This comparison mechanism is equivalent to establishing an objective reference for the model's "self-assessment ability".
[0146] Next, based on the differences between the two, this embodiment calculates the consistency reward signal. The specific formula is: ;
[0147] in, This represents the L1 norm, which is the sum of the absolute differences across the fractional dimensions. When... and When the results are completely identical, the difference is 0 and the reward is 1, indicating that the model's self-evaluation is completely correct. When the difference is large, the reward is close to 0, indicating that the model's self-evaluation deviates from the true evaluation. This reward function is simple and intuitive, and can effectively guide the model to gradually correct its evaluation criteria.
[0148] Finally, the output of phase S4 is the consistency reward value. It will serve as a reinforcement learning signal during the training and optimization process, working together with task loss and evaluation loss to update model parameters.
[0149] Furthermore, based on task results, self-assessment information, and consistent reward signals, the objective function is obtained as follows:
[0150] Based on the task results, determine the task loss;
[0151] Based on self-assessment information, assess the loss;
[0152] Based on the consistent reward signal, obtain the reinforcement learning loss;
[0153] The objective function is obtained by weighting and integrating the task loss, evaluation loss, and reinforcement learning loss.
[0154] Specifically, in step S5, this embodiment integrates the various signals generated in the previous steps and trains the model using a joint optimization approach. The goal of this step is to simultaneously improve the model's performance on the main task and its reliability in self-evaluation and self-correction.
[0155] First, the inputs for stage S5 include three types of loss or reward signals:
[0156] Mission loss This comes from the S2 task execution phase and is used to measure the difference between the model's main task result and the standard answer. In 3D generation tasks, reconstruction loss is typically used, such as the Chamfer distance between point clouds or the cross-entropy of voxels; in spatial question answering tasks, the cross-entropy loss of the answer sequence is used to ensure that the output answer matches the standard answer.
[0157] assess loss This comes from the S3 self-reflection space self-assessment stage, used to constrain the model's generated self-assessment score to be close to the true indicator value. For example, mean squared error (MSE) is used to measure the model's self-assessment. Compared with real indicators Differences:
[0158] ;
[0159] Reinforcement learning loss From the S4 consistency reward calculation phase. Through consistency rewards... The model is guided to continuously adjust its parameters during training to keep its self-scoring score close to the true metric.
[0160] In this embodiment, the Proximal Policy Optimization (PPO) or Group Relative Policy Optimization (GRPO) methods are preferably used to implement reinforcement learning updates.
[0161] Next, this embodiment weights and integrates the three types of losses to form a joint optimization objective function. This function has the following form:
[0162] ;
[0163] in: Ensure the main task result is correct; Ensure that the self-assessment score is reasonable; Enhance the model's self-correction ability by leveraging consistency rewards; and This is a hyperparameter used to balance the contribution ratios of the three types of losses. It can be flexibly adjusted. , Trainers can achieve different optimization goals in different application scenarios. For example, in applications requiring high-precision self-evaluation, α can be improved; in applications requiring strong robustness, β can be improved.
[0164] Then, using the total loss function The model parameters are calculated and updated using the backpropagation algorithm. In the supervised learning part (task loss and evaluation loss), the parameter update process follows the standard stochastic gradient descent (SGD) or Adam optimizer; in the reinforcement learning part (based on consistency reward), the policy gradient method is used to update the model policy, so that the model outputs more reliable scores in future self-evaluations.
[0165] This dual-track optimization of "supervised learning + reinforcement learning" not only maintains the model's efficient learning ability in traditional tasks, but also endows the model with new self-evaluation and self-correction functions.
[0166] This embodiment also provides a 3D scene understanding and generation system that combines self-reflective learning, including: an image acquisition module, a main task module, a self-evaluation module, a consistency reward module, and a model optimization module;
[0167] The image acquisition module is used to acquire reference images, task instructions, and viewpoint transformation parameters. It performs geometric-semantic feature encoding on the reference image to obtain a visual representation. The language-visual features are obtained by fusing the semantic vector of the task instructions encoded by the language model with the visual vector of the reference image encoded by the geometric-semantic features.
[0168] The main task module is used to output task results based on task instructions, viewpoint transformation parameters, and visual representations, combined with 3D scene or language-visual features;
[0169] The self-assessment module is used to input task results into the self-reflection space stage to obtain self-assessment information;
[0170] The consistency reward module is used to obtain consistency reward signals based on self-assessment information and objective evaluation indicators.
[0171] The model optimization module is used to obtain the objective function based on the task results, self-evaluation information, and consistency reward signal. The objective function is used to optimize the 3D generation model to obtain the optimal 3D generation model. The optimal 3D generation model is used to achieve new perspective synthesis of 3D scenes, point cloud reconstruction, and spatial relationship question answering based on language-visual features, given a reference image and perspective transformation parameters.
[0172] The following is a brief application of this embodiment: In an autonomous driving system, an onboard camera captures real-time images of the road ahead as input reference images. The system receives task instructions including "generate a 3D scene 10 meters ahead of the vehicle" and "determine if there is an obstacle to the left front." Based on the reference image and the vehicle's viewpoint parameters, the following steps are performed:
[0173] Feature extraction: The camera images are converted into digital matrices, geometric-semantic features are extracted through a convolutional neural network, and noise-enhanced residual labels are introduced to obtain robust visual representations;
[0174] Conditional fusion: Combining driving task instructions and vehicle perspective parameters to form conditional information and fusing it with visual representation;
[0175] Task execution: In the 3D generation branch, the diffusion generator generates a 3D point cloud from the target's perspective to construct a 3D environment model in front of the vehicle; in the spatial question-and-answer branch, the sequence decoder outputs the answer to the question, such as "There is an obstacle to the left front";
[0176] Self-reflection phase: After completing the main task, the model enters the self-reflection space, calculates the Chamfer distance between the generated point cloud and the real radar point cloud, and checks the rationality of spatial question answering based on image-text similarity.
[0177] Consistency Rewards and Optimization: Compare the model's self-scoring with the actual metrics, generate consistency reward signals, and optimize model parameters.
[0178] Through the above embodiments, this embodiment can achieve high-precision 3D modeling of the road ahead in autonomous driving scenarios and has the ability to answer questions about spatial relationships. Because the model incorporates a self-reflection mechanism and consistency rewards during training, this embodiment can significantly reduce the environmental perception error rate and improve the safety and reliability of autonomous driving.
[0179] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims. < / eos>
Claims
1. A method for understanding and generating 3D scenes by combining self-reflective learning, characterized in that, include: Obtain a reference image, task instructions, and viewpoint transformation parameters; perform geometric-semantic feature encoding on the reference image to obtain a visual representation; Based on the task instructions, viewpoint transformation parameters, and visual representation, and combined with 3D scene or language-visual features, the task result is output. The language-visual features are obtained by fusing the semantic vector obtained by encoding the task instructions through a language model with the visual vector obtained by encoding the reference image through geometric-semantic features. The task results are input into the self-reflection space stage to obtain self-assessment information; Based on the self-assessment information and objective evaluation indicators, a consistent reward signal is obtained. Based on the task results, self-assessment information, and consistent reward signals, the objective function is obtained; The objective function is used to optimize the 3D generation model to obtain the optimal 3D generation model. The optimal 3D generation model is used to achieve new perspective synthesis, point cloud reconstruction, and spatial relationship question answering based on language-visual features of a 3D scene, given a reference image and perspective transformation parameters.
2. The method for understanding and generating three-dimensional scenes by combining self-reflective learning according to claim 1, characterized in that, Geometric-semantic feature encoding of the reference image to obtain a visual representation includes: The reference image is digitally converted to obtain a digital image; The digitized image is processed using a convolutional neural network to obtain high-level features; Based on the high-level features, combined with Gaussian noise, noise-enhanced residual visual labels are obtained; The noise-enhanced residual visual markers are linearly mapped and dimensionally aligned to obtain a visual marker sequence with uniform dimensions; The visual representation is obtained based on the unified dimension visual tag sequence.
3. The method for understanding and generating three-dimensional scenes by combining self-reflective learning according to claim 1, characterized in that, The task results include: point cloud representation or answer sequence.
4. The method for understanding and generating three-dimensional scenes by combining self-reflective learning according to claim 3, characterized in that, Based on the task instructions, viewpoint transformation parameters, and visual representation, combined with 3D scene or language-visual features, the output task results include: Based on the task instructions, perspective transformation parameters, and visual representation, obtain the condition information for the task execution phase; The task instructions are encoded to obtain a semantic embedding matrix; Random noise is introduced into the semantic embedding matrix to obtain conditional feature representations; The conditional feature representation and the visual feature representation are fused to obtain the fused feature; The fusion features and the conditional information are input into the diffusion generator, and combined with the 3D scene, a point cloud representation is obtained. The conditional information is used to ensure that the point cloud representation meets the viewpoint requirements. The fused features and the conditional information are input into a sequence decoder, and combined with language-visual features, an answer sequence is obtained. The conditional information is used to ensure that the answer sequence and the image semantics are consistent.
5. The method for understanding and generating three-dimensional scenes by combining self-reflective learning according to claim 3, characterized in that, The task results are input into the self-reflection space stage to obtain self-assessment information, including: When the task result is a point cloud representation, the Chamfer distance method is used to calculate the difference between the point cloud representation and the real point cloud to obtain the three-dimensional evaluation result; When the task result is an answer sequence, a cross-modal similarity model is used to map the answer sequence and the reference image to the same feature space, and cosine similarity is calculated to obtain spatial evaluation results; Based on the three-dimensional assessment results and spatial assessment results, the self-assessment information is obtained.
6. The method for understanding and generating three-dimensional scenes by combining self-reflective learning according to claim 5, characterized in that, The Chamfer distance method is used to calculate the difference between the point cloud representation and the true point cloud, including: ； in, For Chamfer distance, For point cloud representation, For a real point cloud, x is The point in the middle, y is The point in the middle, It is a 2-norm.
7. The method for understanding and generating three-dimensional scenes by combining self-reflective learning according to claim 5, characterized in that, Calculating cosine similarity includes: ； in, For semantic similarity scores, For the answer sequence, A text encoding function that maps answer sequences to a semantic feature space. For reference image, This is a visual encoding function that maps a reference image to a semantic feature space.
8. The method for understanding and generating three-dimensional scenes by combining self-reflective learning according to claim 1, characterized in that, Based on the self-assessment information combined with objective evaluation indicators, the consistency reward signal is obtained as follows: ； in, As a consistent reward signal, For self-assessment information, For true indicators, This represents the L1 norm.
9. The method for understanding and generating three-dimensional scenes by combining self-reflective learning according to claim 1, characterized in that, Based on the task results, self-assessment information, and consistent reward signals, the objective function is obtained as follows: Based on the task results, the task loss is obtained; Based on the self-assessment information, the assessed loss is obtained; Based on the consistency reward signal, obtain the reinforcement learning loss; The objective function is obtained by weighting and integrating the task loss, evaluation loss, and reinforcement learning loss.
10. A three-dimensional scene understanding and generation system combining self-reflective learning, implemented according to the method of any one of claims 1-9, characterized in that, include: The system includes an image acquisition module, a main task module, a self-evaluation module, a consistency reward module, and a model optimization module. The image acquisition module is used to acquire a reference image, task instructions and viewpoint transformation parameters, and to perform geometric-semantic feature encoding on the reference image to obtain a visual representation. The language-visual features are obtained by fusing the semantic vector obtained by encoding the task instructions through a language model with the visual vector obtained by encoding the reference image through geometric-semantic features. The main task module is used to output task results based on the task instructions, viewpoint transformation parameters, and visual representation, combined with three-dimensional scene or language-visual features; The self-assessment module is used to input the task results into the self-reflection space stage to obtain self-assessment information; The consistency reward module is used to obtain a consistency reward signal based on the self-assessment information and objective evaluation indicators. The model optimization module is used to obtain an objective function based on the task results, self-evaluation information, and consistency reward signal; and to optimize the 3D generation model using the objective function to obtain the optimal 3D generation model. The optimal 3D generation model is used to achieve new perspective synthesis of 3D scenes, point cloud reconstruction, and spatial relationship question answering based on language-visual features, given a reference image and perspective transformation parameters.