Complex interaction scene reconstruction and control method

By performing interaction space modeling and multi-scale factor decomposition in complex interactive scenarios, and combining interaction-aware language embedding, the limitations of the processing of interaction relationships of multiple objects in complex scenarios in the prior art are solved, and precise reconstruction and natural language control of multiple interactive objects are realized.

CN119690237AInactive Publication Date: 2025-03-25SHANGHAI ARTIFICIAL INTELLIGENCE INNOVATION CENT
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411112391.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2025-03-25
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art is difficult to effectively handle independent interactions between multiple objects in complex scenarios, lacks support for natural language interactions, and has limited processing capabilities when scene topology changes.

Method used

Interaction space modeling is performed based on the camera perspective and control variables of a specific interactive object, high-dimensional functions are used to describe the interactive scene, and the scene is decomposed into multiple locally deformable fields through multi-scale interactive space factor decomposition. At the same time, a multi-scale sampling strategy and interactive perception language embedding are adopted to realize natural language positioning and control individual interactive objects.

Benefits of technology

It realizes precise reconstruction and control of multiple interactive objects in complex scenarios, improves scene-level interaction understanding and control capabilities, enhances user-scene interaction and naturalness, and effectively solves the problem of interaction probability feature storage and sampling conflict.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119690237A_ABST
    Figure CN119690237A_ABST
Patent Text Reader

Abstract

The invention relates to a complex interaction scene reconstruction and control method, which comprises the following steps: carrying out interaction space modeling based on a camera view angle and a control variable of a specific interaction object, and describing an interaction scene by adopting a high-dimensional function; based on multi-scale interaction space factorization, the interaction scene is decomposed into a plurality of local deformable fields; a multi-scale sampling strategy is employed, and interaction-aware language embedding is utilized to locate and control individual interaction objects using natural languages. Compared with the prior art, the method has the advantages that the challenging problem that multiple interactive objects exist in reconstruction and control of a complex scene in a monocular video is solved, the interactive scene can be reconstructed and controlled at the scene level, and by embedding a natural language into the modeling and control process of the interactive scene, the interactive scene can be reconstructed and controlled. Accurate reconstruction and accurate control of a plurality of interaction objects in a complex scene are realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of monocular video scene modeling, and in particular to a reconstruction and control method for a complex interactive scene. Background Art

[0002] In the real world, complex scenes containing multiple interacting objects often need to be modeled and controlled. These scenes may involve virtual reality environments, robotics operations, game development, and other fields. Traditional methods mainly focus on modeling the interactions of a single object, using object-level interaction modeling techniques. However, with the increase in the number of interacting objects in complex scenes and the complexity of the interaction relationships between objects, traditional methods often seem to be unable to effectively capture and control the independent interactions between multiple objects.

[0003] Existing research extends NeRF (Neural Radiance Fields) to dynamic scene reconstruction, and related methods are mainly divided into time-varying methods, deformable-canonical methods, and hybrid representation methods. Among them, the time-varying method usually directly simulates the change of the radiation field over time, but it is often difficult to separate dynamic and static objects; the deformable-canonical method decouples the dynamic deformable field and the static canonical space, and models the four-dimensional scene by deforming the points with the deformable field to query the canonical features. However, these two methods face challenges in dealing with scene topology changes. The hybrid representation method achieves high-quality reconstruction and fast rendering by utilizing time-space feature planes, four-dimensional hash coding, dynamic voxels, or triple fields. However, it is limited to reconstructing dynamic scenes and lacks the ability to control and understand interactive scenes.

[0004] In addition, methods based on vision-language grounded models have inspired many language-embedded 3D scene representation methods, such as open vocabulary segmentation, 3D visual question answering, and 3D language localization. However, these methods are usually limited to static scene understanding and lack generalization capabilities when the topology of interactive scenes changes.

[0005] In order to deal with the control of reconstructed assets or neural fields in avatar and robotic tasks, some methods proposed the concept of controllable scene representation. For example, CoNeRF (Controllable Neural Radiance Fields) extends HyperNeRF (A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields) and introduces fine-grained controllable neural fields to finely control the scene. However, these methods usually lack natural language interaction capabilities and mainly rely on manual control. Moreover, most methods focus on the interaction of a single or a few objects, ignoring the interaction between different scene parts, which limits their application in the real world.

[0006] In summary, when faced with complex scene modeling containing multiple interactive objects, traditional interactive scene modeling methods are limited to the interaction modeling of a single object, usually focusing on the interaction modeling of a single object, such as a drawer, a toy car, or a face. This leads to the limitations of these methods in dealing with the interaction relationship between multiple objects in complex scenes. As the number of interactive objects in the scene increases, the interaction space becomes more and more high-dimensional and complex. Traditional methods are often unable to effectively capture and model this high-dimensional interaction space, resulting in difficulties in accurately modeling complex interactive scenes. This not only increases the computational time and memory cost, but also greatly increases the time required to train the model. In addition, natural language is an intuitive and necessary interface for interacting with 3D scenes, but existing technologies face challenges in dealing with interactive scenes. In particular, existing methods often perform poorly in dealing with the consistency problem of interaction changes. For example, methods such as LERF (Language Embedded Radiance Fields), OpenNeRF (OpenNeural Radiance Fields), and GARField often fail significantly when the scene topology changes.

[0007] That is, existing methods are often unable to effectively handle independent interactions between multiple objects in complex scenes, lack support for natural language interactions, and have limited ability to handle changes in scene topology. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for reconstructing and controlling a complex interactive scene, which can realize accurate reconstruction and control of multiple interactive objects in a complex scene.

[0009] The purpose of the present invention can be achieved by the following technical solution: A method for reconstructing and controlling a complex interactive scene, comprising the following steps:

[0010] S1. Based on the camera perspective and control variables of specific interactive objects, interactive space modeling is performed, and high-dimensional functions are used to describe the interactive scene;

[0011] S2, based on multi-scale interaction space factorization, decompose the interaction scene into multiple local deformable fields;

[0012] S3, adopts a multi-scale sampling strategy and exploits interaction-aware language embedding to locate and control individual interacting objects using natural language.

[0013] Furthermore, the specific process of step S1 is as follows: for a (3+α)-dimensional non-rigid interactive scene χ, there are α control variables k=[k1,κ2,...,kα] corresponding to α objects, and their representation is described by a high-dimensional function:

[0014] y=ρ(x,k,H;θ)

[0015] Among them, ρ is the representation model, x∈R3 is the spatial coordinate, k∈Rα is the control variable, H is a set of pre-selected additional parameters, θ is the storage scene information, and the function outputs the scene attribute y given the position x and k on the ray r.

[0016] Furthermore, the scene attribute y is specifically represented by color, occupancy, signed distance, density or BRDF (Bidirectional Reflectance Distribution Function) parameters.

[0017] Furthermore, after the interaction space is modeled in step S1, the interaction objects in the interaction space show mutual independence, and the interaction features Fα∈θ are unevenly distributed in the (3+α)-dimensional interaction space χ and clustered to the cluster center ci, i≤α.

[0018] Furthermore, the specific process of step S2 is: using a set of hyperplanes to divide the (3+α)-dimensional interaction space into non-overlapping regions R={R1, R2, ...Rα}, each sub-region Ri contains a local 4D deformable field θd to project the interaction feature θ(p) at the sampling point p∈R3+α into the 4-dimensional space R4 for representation.

[0019] Furthermore, the multi-scale sampling strategy in step S3 specifically solves the conflict problem of interactive probability feature storage and sampling through feature exclusion and probability rejection mechanism.

[0020] Furthermore, the characteristic repulsion mechanism specifically introduces repulsion loss to enhance the characteristic difference between different deformable fields, thereby promoting the separation of the deformable fields.

[0021] Furthermore, the rejection loss is specifically:

[0022]

[0023] in, and is the ground truth mask of the ray, Fsi and Fsj are the last layer features of the interaction probability decoder, and in the training iteration, a ray pair (ri, rj) is randomly selected and applied To promote the separation of interaction probability features in the local deformable space, and at the same time incorporate spatial information through the weight 1 / (ci-cj+δ) This encourages the model to focus on local sampling conflicts. The centers ci and cj of the ray pairs are obtained by ∑i=1Mwixi, where wi and xi are the weight and 3D position of each sample, respectively, and M is the number of samples.

[0024] Furthermore, the probability rejection mechanism is specifically as follows: if the deformable probability at p is less than a preset threshold s, a probability rejection operation is performed: d=argmax_iθ(k,θ_s(x))iifθ(κ,θ_s(x))i≥selse-1, and the rejected samples are considered to be outside the hyperplane and do not belong to the deformable field.

[0025] Furthermore, the interaction-aware language embedding process in step S3 includes: for any sampling point p, projecting it to pd=[x|κd], retrieving the local language feature group through d, and performing bilinear interpolation using κd to obtain a language embedding that adapts to the interaction variable changes of the surrounding clip features.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] The present invention models the interactive space based on the camera perspective and control variables of a specific interactive object, and uses high-dimensional functions to describe the interactive scene; then, based on multi-scale interactive space factorization, the interactive scene is decomposed into multiple local deformable fields; finally, a multi-scale sampling strategy is adopted, and interaction-aware language embedding is used to locate and control individual interactive objects using natural language. In this way, multiple interactive objects in a complex scene can be directly reconstructed and controlled from a single video, and interactive scenes can be reconstructed and controlled at the scene level. By embedding natural language into the modeling and control process of the interactive scene, accurate reconstruction and precise control of multiple interactive objects in a complex scene are achieved.

[0028] The present invention adopts multi-scale interactive space factor decomposition and feature compression technology to realize an efficient interactive space modeling method, which can effectively compress the high-dimensional interactive space into a compact four-dimensional feature plane, can more accurately model complex interactive scenes, improve the accuracy and stability of modeling, and realize a more compact storage and modeling scheme. Compared with the existing technology, the model has fewer model parameters and can more effectively utilize computing resources and memory.

[0029] The present invention achieves independent modeling and precise control of interactive objects by decomposing the interactive scene into multiple local deformable fields and adopting a multi-scale interactive probability sampling strategy. Through feature repulsion and probability rejection mechanisms, the present invention can effectively solve the problem of interactive probability feature storage and sampling conflicts, and improve the accuracy and stability of modeling.

[0030] The present invention introduces an interaction-aware language embedding method, which can generate different language embeddings according to different interaction states, thereby realizing the positioning and control of interactive objects, thereby providing a new way to achieve more intuitive and natural interaction, and can locate and control interactive objects according to natural language instructions, thereby improving the interactivity and naturalness between users and scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0032] Figure 2 It is a schematic diagram of the application process of the embodiment;

[0033] Figure 3 It is a schematic diagram of hyperplane factorization;

[0034] Figure 4a This is a schematic diagram of boundary sampling conflict;

[0035] Figure 4b This is a schematic diagram for comparing rendering quality;

[0036] Figure 4c Embedding schematic diagram for interaction-aware language;

[0037] Figure 5a A schematic diagram comparing the scene modeling capabilities of the method of the present invention and other existing methods;

[0038] Figure 5b A schematic diagram showing a comparison of rendering performance between the method of the present invention and other existing methods;

[0039] Figure 5c A schematic diagram showing the comparison of scene modeling efficiency between the method of the present invention and other existing methods;

[0040] Figure 6 to Figure 8Schematic diagram comparing the application effects of the method of the present invention and other existing methods on the OminiSim dataset. DETAILED DESCRIPTION

[0041] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Example

[0043] like Figure 1 As shown, a method for reconstructing and controlling a complex interactive scene includes the following steps:

[0044] S1. Based on the camera perspective and control variables of specific interactive objects, interactive space modeling is performed, and high-dimensional functions are used to describe the interactive scene;

[0045] S2, based on multi-scale interaction space factorization, decompose the interaction scene into multiple local deformable fields;

[0046] S3, adopts a multi-scale sampling strategy and exploits interaction-aware language embedding to locate and control individual interacting objects using natural language.

[0047] This embodiment applies the above solution, such as Figure 2 As shown, given the camera view and control variables of a specific interactive object, a series of 3D points are sampled in the local deformable field that simulates the interactive motion of the specific interactive object, and then the interactive object with novel interactive motion states is generated through volume rendering. In addition, interaction-aware language embedding is used to locate and control a single interactive object using natural language.

[0048] The goal of this proposal is to develop a representation method that can model α interactive articulated objects in a complex scene χ from randomly captured monocular videos {Ii}i=1T in a rendering-based self-supervised manner. Given a control variable κ=[κ1,κ2,...,κα] representing the motion state of the object in each frame of video, and a camera pose {P}i=1T for each frame of video. Given a camera viewpoint and a control variable κ for a specific interactive object, a series of 3D points are sampled in a local deformable field that simulates the interactive motion of the specific interactive object, and then interactive objects with novel interactive motion states are generated through volume rendering. In addition, interaction-aware language embeddings are used to locate and control individual interactive objects using natural language. Specifically:

[0049] 1. Interactive Space Modeling

[0050] Assume that there are α control variables κ = [κ1, κ2, ..., κα] corresponding to α objects in a non-rigid interactive scene χ, and its representation is described by a high-dimensional function: y = ρ(x, x, H; θ), where ρ is the representation model, x∈R3 is the spatial coordinate, κ∈Rα is the control variable, H is a set of optional additional parameters (such as viewing direction), and θ stores scene information. The function outputs the scene attribute y given the position x and κ on the ray r, where y can be represented by color, occupancy, signed distance, density and BRDF (Bidirectional Reflectance Distribution Function) parameters. In this embodiment, we focus on color, probability and language embedding. It should be noted that, unlike 3D static scene or 4D dynamic scene modeling, the sampling points p = [x|κ]∈R3+α in the interactive scene are high-dimensional and topologically variable, making the optimization of scene feature storage θ and representation model ρ very time-consuming or memory-intensive.

[0051] 2. Multi-scale Interaction Space Factorization

[0052] For a (3+α)-dimensional interaction space χ = R(3+α) containing α control variables, the goal is to explicitly represent the high-dimensional space as a compact storage, thereby reducing memory usage and improving optimization results. Figure 2 As shown in , the interacting objects exhibit mutual independence, and the interaction features Fα∈θ are unevenly distributed in the (3+α)-dimensional interaction space χ and clustered to cluster centers ci,i≤α. Therefore, there exists a set of hyperplanes that divide the space into non-overlapping regions R = {R1, R2, ... Rα}, and each sub-region Ri contains a local 4D deformable field θd, as Figure 3 As shown. Therefore, the interaction feature θ(p) at the sampling point p∈R3+α can be represented by projecting it into a compact 4-dimensional space R4.

[0053] Figure 3 An illustration of hyperplane factorization is presented for compact storage and efficient rendering by maintaining multiple local deformable fields in 4D space for each interacting object and projecting the high-dimensional interaction features into a compact 4D space, which is then further compressed into a multi-scale feature plane.

[0054] 3. Feature Rejection and Probability Rejection

[0055] A potential problem is that optimizing the interactive probability decoder with different masks may lead to blurred boundaries in the local deformable field, which in turn leads to conflicts in ray sampling and feature storage. Figure 3As shown in the figure, consider two adjacent local deformable regions Ri and Rj, and a point p in the high-dimensional space. Assuming that p moves from the cluster center i of ci to the cluster center j of cj, the probability that p belongs to Ri gradually decreases, while the probability that it belongs to Rj increases. In order to avoid sampling conflicts and boundary feature oscillations, this scheme introduces a repulsion loss to enhance the feature differences between different deformable fields, thereby promoting the separation of deformable fields:

[0056]

[0057] in, and is the ground truth mask of the ray, Fsi and Fsj are Figure 2 The last layer of features of the interaction probability decoder in , in the training iteration, randomly selects a ray pair (ri, rj) and applies To promote the separation of interaction probability features in the local deformable space, and at the same time incorporate spatial information through the weight 1 / (ci-cj+δ) This encourages the model to focus on local sampling conflicts. The centers ci and cj of the ray pairs are obtained by ∑i=1Mwixi, where wi and xi are the weight and three-dimensional position of each sample, respectively, and M is the number of samples. In addition, if the deformable probability at p is less than the threshold s, this scheme also proposes a probabilistic rejection operation: d=argmax_iθ(κ,θ_s(x))iifθ(κ,θ_s(x))i≥selse-1. The rejected samples are considered to be outside the hyperplane and do not belong to the deformable field. Figure 4a and Figure 4b As shown, the probabilistic rejection operation helps the model achieve higher rendering quality and also demonstrates its effectiveness in alleviating boundary sampling conflicts.

[0058] 4. Interaction-aware language embedding

[0059] In interactive scenarios, language embedding is a complex and storage-intensive task, as 3D extraction faces the dual challenges of high-dimensional optimization and inconsistent interactive scene changes, e.g. Figure 4cIn , the topology of the transformer changes significantly before and after deformation, which leads to the failure of SAM segmentation or LERF-based approaches. With the proposed multi-scale interaction space factorization, language features are efficiently stored in a lightweight plane instead of a 3D field in LERF by indexing them by maximum probability sampling. For any sample point p, it is projected to pd = [x|κd], the local language feature group is retrieved by d, and bilinear interpolation is performed using κd to obtain a language embedding that is adapted to the interaction variable changes of the surrounding clip features, with a storage complexity of O(C×α×dim), which is much smaller than language extraction methods such as LERF in 3D scenes.

[0060] In order to verify the effectiveness of this solution, this embodiment uses the method proposed by the present invention to compare with other existing methods. Figure 5a to Figure 5c As shown in the figure ("Ours" in the figure refers to the method proposed in the present invention), the scene modeling capability, rendering performance and efficiency are compared respectively: compared with static or deformable scenes, the method proposed in the present invention can model $(3+\alpha)$-dimensional interactive scenes with a language basis and achieve SOTA rendering quality on the simdata dataset. It is worth noting that the method proposed in the present invention has significant advantages in parameter efficiency, maintains a constant complexity regardless of the number of interactive instances, and meets the rendering time at $1200\times 1200$ resolution. Compared with other methods such as CoGS~\cite{yu2023cogs}, MK-Planes~\cite{kplanes_2023}, the number of parameters of these methods increases linearly or quadratically.

[0061] In addition, this embodiment is also extensively evaluated on synthetic data OminiSim, CoNeRF-Synthetic, real data sets InterReals, and CoNeRF-Control, and has achieved significant improvements in the evaluation of indicators such as new view synthesis, geometric depth L1, language localization, and network parameter quantity. Table 1 shows the quantitative results of CoNeRF synthetic and controllable data sets. The method proposed in the present invention achieves the best results in all indicators of synthetic scenes and the best PSNR (Peak signal-to-noise ratio) on the controllable data set. Table 2 shows the quantitative results of the OminiSim data set. The method proposed in the present invention outperforms previous work in most indicators and obtains the best PSNR with a significant advantage on the challenge subset. Table 3 shows the quantitative results of the InterReal data set. The method proposed in the present invention performs well in most settings and has obvious advantages in PSNR, SSIM (Structural Similarity) and LPIPS (Learned Perceptual Image Patch Similarity) on the challenge subset. Table 4 shows the comparison of geometric structure depth L1 error of the OminiSim dataset. The proposed method surpasses most previous works on the selected subset. Table 5 shows the comparison of language localization of the OminiSim dataset. The proposed method obtains the highest mIOU (Mean Intersection over Union) score.

[0062] Table 1

[0063]

[0064] Table 2

[0065]

[0066] Table 3

[0067]

[0068] Table 4

[0069]

[0070] Table 5

[0071]

[0072] Figure 6The figure is a visualization comparison of view synthesis of the OminiSim dataset. The RGB rendering of the proposed method is compared with that of the SOTA method on three scenes: #rs, #ihlen and #pomaria. Boxes of different colors represent different interactive objects in the scene. Figure 7 It shows that the proposed method performs well in reconstructing the complex details of interactive objects and maintaining the geometric integrity of static background. Figure 8 It shows that the basic performance of the LiveScene method proposed in the present invention shows clearer boundaries than other methods.

[0073] It can be seen that compared with the prior art, the method proposed in the present invention has the following advantages:

[0074] Fewer model parameters: The present invention adopts a more compact storage and modeling method. Compared with the existing technology, the model parameters are fewer, which can better cope with the increased scene complexity and more effectively utilize computing resources and memory.

[0075] Better modeling effect: By introducing multi-scale interaction space factor decomposition and feature compression technology, the present invention can more accurately model complex interaction scenes and improve the accuracy and stability of modeling. Among them, through the proposed multi-scale interaction space factor decomposition, the high-dimensional interaction space is compressed into a locally deformable 4D scene, thereby realizing efficient modeling of complex interaction scenes.

[0076] Interactive language embedding: The present invention introduces an interactively perceived language embedding method, which locates and controls interactive objects according to natural language instructions, improves the interactivity and naturalness of users and scenes, enables more natural interaction between users and scenes, and improves user experience and the convenience of scene control.

[0077] Solving conflict problems: Through feature repulsion and probability rejection mechanisms, the present invention can effectively solve the conflict problems of interactive probability feature storage and sampling, and improve the accuracy and stability of scene modeling.

[0078] The present invention is the first to propose a scene-level language embedded interactive scene reconstruction and control method, which has unique innovation and foresight, can bring new technical breakthroughs and solutions to the interactive scene modeling and control field, and has significant advantages in modeling accuracy, computational efficiency and user experience, etc., can bring important technical progress and innovation to the interactive scene modeling and control field. This technical breakthrough can more effectively handle the modeling and control of complex scenes in the real world, and provides important support for research and application in the fields of content generation, animation production, virtual reality, robotics and world understanding. The method of the present invention is applied to practice and can be extended to many robot projects, such as autonomous driving, navigation, robot operation and grasping, interactive physical world understanding, etc., and can provide ideal low-cost reconstruction and control methods for many robot tasks, thereby promoting the development of the field. It is also worth noting that the existing world understanding models are mostly limited to pure language understanding, video understanding, and 3D point cloud understanding, while the method proposed in the present invention can increase the interactive characteristics of training data and provide further support for physical world understanding.

Claims

1. A method for reconstructing and controlling a complex interactive scene, characterized in that: The following steps are involved: S1. Based on the camera perspective and control variables of specific interactive objects, interactive space modeling is performed, and high-dimensional functions are used to describe the interactive scene; S2, based on multi-scale interaction space factorization, decompose the interaction scene into multiple local deformable fields; S3, adopts a multi-scale sampling strategy and exploits interaction-aware language embedding to locate and control individual interacting objects using natural language.

2. The method for reconstructing and controlling a complex interactive scene according to claim 1, characterized in that: The specific process of step S1 is: for a (3+α)-dimensional non-rigid interactive scene χ, there are α control variables κ=[κ1,κ2,...,κα] corresponding to α objects, and their representation is described by a high-dimensional function: y=ρ(x,κ,H;θ) Among them, ρ is the representation model, x∈R3 is the spatial coordinate, κ∈Rα is the control variable, H is a set of pre-selected additional parameters, θ is the storage scene information, and the function outputs the scene attribute y given the position x and κ on the ray r.

3. The method for reconstructing and controlling a complex interactive scene according to claim 2, characterized in that: The scene attribute y is specifically represented by color, occupancy, signed distance, density or BRDF parameter.

4. The method for reconstructing and controlling a complex interactive scene according to claim 2, characterized in that: After the interaction space is modeled in step S1, the interaction objects in the interaction space show mutual independence, and the interaction features Fα∈θ are unevenly distributed in the (3+α)-dimensional interaction space χ and clustered to the cluster center ci, i≤α.

5. The method for reconstructing and controlling a complex interactive scene according to claim 4, characterized in that: The specific process of step S2 is: using a set of hyperplanes to divide the (3+α)-dimensional interaction space into non-overlapping regions R={R1, R2, ...Rα}, each sub-region Ri contains a local 4D deformable field θd to project the interaction feature θ(p) at the sampling point p∈R3+α into the 4-dimensional space R4 for representation.

6. The method for reconstructing and controlling a complex interactive scene according to claim 5, characterized in that: The multi-scale sampling strategy in step S3 specifically solves the conflict problem of interactive probability feature storage and sampling through feature exclusion and probability rejection mechanism.

7. The method for reconstructing and controlling a complex interactive scene according to claim 6, characterized in that: The characteristic repulsion mechanism specifically introduces repulsion loss to enhance the characteristic difference between different deformable fields, thereby promoting the separation of the deformable fields.

8. The method for reconstructing and controlling a complex interactive scene according to claim 7, characterized in that: The rejection loss is specifically: in, and is the ground truth mask of the ray, Fsi and Fsj are the last layer features of the interaction probability decoder, and in the training iteration, a ray pair (ri, rj) is randomly selected and applied To promote the separation of interaction probability features in the local deformable space, and at the same time incorporate spatial information through the weight 1 / (ci-cj+δ) This encourages the model to focus on local sampling conflicts. The centers ci and cj of the ray pairs are obtained by ∑i=1Mwixi, where wi and xi are the weight and 3D position of each sample, respectively, and M is the number of samples.

9. The method for reconstructing and controlling a complex interactive scene according to claim 7, characterized in that: The probability rejection mechanism is specifically as follows: if the deformable probability at point p is less than a preset threshold s, a probability rejection operation is performed: d=argmax_iθ(κ,θ_s(x))iifθ(κ,θ_s(x))i≥selse-1, and the rejected samples are considered to be outside the hyperplane and do not belong to the deformable field.

10. The method for reconstructing and controlling a complex interactive scene according to claim 5, characterized in that: The interaction-aware language embedding process in step S3 includes: for any sampling point p, projecting it to pd=[x|κd], retrieving the local language feature group through d, and performing bilinear interpolation using κd to obtain a language embedding that adapts to the interaction variable changes of the surrounding clip features.

Citation Information

Patent Citations

  • Large-scale three-dimensional scene semantics and building segmentation method based on neural radiation representation

    CN117726810A

  • Object recommendation method and device

    CN117893279A

  • Monocular indoor scene reconstruction method based on triangular three planes

    CN118447182A