Voxel segmentation system and method oriented to physical dynamic visual understanding

Abstract

The invention provides a voxel segmentation system and method oriented to physical dynamic visual understanding, and the system comprises a bidirectional dynamic module which is used for learning a space-time deformation field of voxels; the semantic volume slot attention module is used for deducing global time-invariant semantic features from the local time-variant voxel features, fusing the global time-invariant semantic features with 2D semantic features and determining three-dimensional global time-invariant semantic features; and the neural renderer module taking the object as the center is used for predicting the color value of each voxel taking the object as the center and generating a three-dimensional rendered image corresponding to a preset visual angle by adopting a preset combined neural radiation field. According to the method and the device, the object-centered decomposition problem of the unsupervised 3D dynamic scene is solved by adopting the preset neural radiation field, the semantic features are introduced to enhance the object-centered representation learning, and the understanding of the real world scene is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision and deep learning technology, and in particular, to a voxel segmentation system and method for physical dynamic visual understanding. Background Art

[0002] Unsupervised learning of the physical world, such as unsupervised 3D dynamic scene decomposition, is a crucial task with enormous application value in numerous fields, including autonomous driving and the metaverse. However, this task is challenging due to the complex coupling between spatial and temporal information. Existing methods primarily exploit the consistency of dynamic information between consecutive video frames but often ignore 3D features, resulting in multi-view mismatches in 2D object segmentation.

[0003] Unlike previous 2D methods, we naturally ensure 3D-consistent scene understanding through explicit voxel grid modeling and achieve explicit knowledge of object geometry and physical interactions in dynamic scenes.

[0004] However, the classic standard spatial rendering pipeline does not support object-centric learning. There are two key challenges: first, how to decouple the potential dynamics of each object from its visual appearance and capture the local geometric structure of each object; second, how to integrate the global semantic prior of the scene (usually extracted from the 2D base model) into 3D volume rendering, which is crucial for inferring 3D geometry in the presence of only partial observations, complex real-world structures, and severe occlusions. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the present invention aims to provide a voxel segmentation system and method for physical dynamic visual understanding.

[0006] To achieve the above objectives, according to one aspect of the present disclosure, a voxel segmentation system for physical dynamic visual understanding is provided, comprising:

[0007] A bidirectional dynamic module, which inputs the three-dimensional voxel grid data of the object, is used to learn the spatiotemporal deformation field of the voxels, and outputs the three-dimensional voxel grid after the spatiotemporal deformation;

[0008] a semantic volume slot attention module, which inputs the spatiotemporally deformed 3D voxel grid, derives global time-invariant semantic features from local time-varying voxel features, fuses 2D semantic features, determines 3D global time-invariant semantic features, and outputs a 3D voxel grid with semantic information;

[0009] The object-centered neural renderer module inputs the three-dimensional voxel grid after spatiotemporal deformation, the three-dimensional voxel grid with semantic information, the global object-centered features and the preset perspective, and is used to predict the object-centered color value of each voxel, and uses a preset combined neural radiation field rendering to generate a three-dimensional rendered image corresponding to the preset perspective, and outputs the three-dimensional rendered image corresponding to the preset perspective.

[0010] Optionally, the bidirectional dynamic module includes:

[0011] The reverse dynamic module is used to learn the displacement information of the voxels at each moment relative to the initial moment and determine the position information of the voxels at the initial moment;

[0012] The forward dynamic module is used to learn the displacement information of the voxel at the initial moment relative to each moment, and determine the position information of the voxel at each moment.

[0013] Optionally, the semantic volume slot attention module includes:

[0014] The semantic volume slot attention module includes:

[0015] A semantic voxel grid, used to divide the three-dimensional voxel grid after the spatiotemporal deformation into semantic voxel grids with semantic labels;

[0016] A slot feature is used to derive a global time-invariant semantic feature from the local time-varying voxel features of the semantic voxel grid with the semantic label through a slot attention mechanism;

[0017] A pre-trained 2D semantic feature extractor is used to extract 2D semantic features from 2D image frames of a monocular video, and fuse the 2D semantic features with the global time-invariant semantic features to determine the three-dimensional global time-invariant semantic features.

[0018] Optionally, the object-centric neural renderer module includes:

[0019] A preset multi-layer perceptron neural network is used to input the spatiotemporally deformed three-dimensional voxel grid and the three-dimensional voxel grid with semantic information, and output a color value of each voxel centered on the object;

[0020] A preset combined neural radiation field is used to input the object-centered color value of each voxel, the global object-centered feature and the preset perspective, and output a generated three-dimensional rendered image corresponding to the preset perspective, wherein the global object-centered feature represents a set of learnable parameters.

[0021] According to a second aspect of the present disclosure, a voxel segmentation method for physical dynamic visual understanding is provided, comprising:

[0022] According to the 3D voxel grid data of the object, a bidirectional dynamic module is used to determine the 3D voxel grid after time-space deformation;

[0023] Determining a three-dimensional voxel grid having semantic information using a semantic volume slot attention module based on the three-dimensional voxel grid after the spatiotemporal deformation;

[0024] Based on the three-dimensional voxel grid after spatiotemporal deformation, the three-dimensional voxel grid with semantic information, the global object-centric features and the preset perspective, an object-centric neural renderer module is used to determine the three-dimensional rendered image corresponding to the preset perspective, and the global object-centric features represent a set of learnable parameters.

[0025] Optionally, the bidirectional dynamic module includes a reverse dynamic module and a forward dynamic module.

[0026] Optionally, the determining of the spatiotemporally deformed three-dimensional voxel grid using a bidirectional dynamic module based on the three-dimensional voxel grid data of the object includes:

[0027] Using the reverse dynamic module to learn the displacement information of the voxels at each moment relative to the initial moment, and determine the position information of the voxels at the initial moment;

[0028] The forward dynamic module is used to simultaneously learn the displacement information of the voxels at the initial moment relative to each moment, and determine the position information of the voxels at each moment;

[0029] The three-dimensional voxel grid after the spatiotemporal deformation is determined according to the position information of the voxels at the initial moment and the position information of the voxels at each moment.

[0030] Optionally, the semantic volume slot attention module includes a semantic voxel grid, slot features, and a pre-trained 2D semantic feature extractor.

[0031] Optionally, determining the three-dimensional voxel grid having semantic information by using a semantic volume slot attention module according to the spatiotemporally deformed three-dimensional voxel grid includes:

[0032] Using the semantic voxel grid, the three-dimensional voxel grid after the spatiotemporal deformation is divided into semantic voxel grids with semantic labels;

[0033] Using the slot features to extract each local time-varying voxel feature in the semantic voxel grid with the semantic label through a slot attention mechanism;

[0034] deriving a global time-invariant semantic feature based on each local time-varying voxel feature in the semantic voxel grid with the semantic label;

[0035] The pre-trained 2D semantic feature extractor is used to extract 2D semantic information from the 2D image frames of the monocular video, and the 2D semantic information and the global time-invariant semantic features are fused based on the slot attention mechanism to determine the three-dimensional global time-invariant semantic features and the three-dimensional voxel grid with semantic information.

[0036] Optionally, the object-centric neural renderer module includes a preset multi-layer perceptron neural network and a preset combined neural radiation field.

[0037] Optionally, determining a three-dimensional rendered image corresponding to a preset perspective using an object-centric neural renderer module based on the spatiotemporally deformed three-dimensional voxel grid, the three-dimensional voxel grid with semantic information, a global object-centric feature, and a preset perspective includes:

[0038] Inputting the spatiotemporally deformed three-dimensional voxel grid and the three-dimensional voxel grid with semantic information into the preset multi-layer perceptron neural network, and outputting the color value of each voxel centered on the object;

[0039] The object-centered color value of each voxel, the global object-centered feature, and the preset viewing angle are input into the preset combined neural radiation field to determine a three-dimensional rendered image corresponding to the preset viewing angle.

[0040] Optionally, the method of training the bidirectional dynamic module, the semantic volume slot attention module, and the object-centric neural renderer module comprises:

[0041] A pre-trained model is used to extract feature maps from preset monocular video image frames to determine feature maps;

[0042] Using the feature map to train the bidirectional dynamic module, the semantic volume slot attention module, and the non-combination neural renderer, to determine a trained bidirectional dynamic module, a trained semantic volume slot attention module, and a trained non-combination neural renderer, wherein the semantic volume slot attention module includes a geometric voxel grid, a color-related feature grid, a semantic-related feature grid, and slot features, and the trained semantic volume slot attention module includes a trained geometric voxel grid, a trained color-related feature grid, a trained semantic-related feature grid, and a trained slot feature;

[0043] Initializing a voxel grid centered on the object, and determining the initialized voxel grid centered on the object;

[0044] The inverse dynamic module in the trained bidirectional dynamic module, the trained geometric voxel grid, the trained color-related feature grid and the initialized object-centered voxel grid are jointly optimized, and the global object-centered features and the object-centered neural renderer module are retrained to determine the trained bidirectional dynamic module, the trained volumetric slot attention module and the trained object-centered neural renderer module.

[0045] Optionally, initializing the voxel grid centered on the object and determining the initialized voxel grid centered on the object includes:

[0046] Taking the portion of the voxel grid centered on the object that is larger than a preset density threshold as a valid voxel;

[0047] Determine the valid voxel as a node of a feature map;

[0048] Constructing connections between the nodes based on the positions, colors, speeds, and semantic similarities of adjacent nodes of the nodes of the feature graph to determine the feature graph;

[0049] Performing clustering processing on the feature graph using a connected graph component algorithm to determine a graph clustering result of the feature graph;

[0050] interpolating the object-centered voxel grid corresponding to the node of the non-graph clustering result using a nearest neighbor interpolation method according to the graph clustering result of the feature graph to determine an interpolated object-centered voxel grid;

[0051] A conditional random field algorithm is used to smooth the interpolated object-centered voxel grid to determine an initialized object-centered voxel grid.

[0052] Compared with the prior art, the embodiments of the present disclosure have at least one of the following beneficial effects:

[0053] Through the above technical solution, an object-centered representation is adopted, and a bidirectional dynamic module is used to learn the spatiotemporal variation field of voxels to capture the local spatial structure of each object and perceive the changes of objects over time, so as to realize the object-centered voxelization method to infer the occupancy probability distribution of objects at each spatial position, thereby promoting 3D consistent scene decomposition; a semantic volume slot attention module is adopted to introduce semantic features to enhance object-centered representation learning, and the global non-time-varying semantic features derived from the local time-varying voxel features are fused with 2D semantic features, thereby promoting the 2D semantic features to 3D, integrating the prior knowledge of object geometry, and improving the ability to interpret complex real-world scenes; an object-centered neural renderer module is adopted to introduce a preset neural radiation field to solve the object-centered decomposition problem of unsupervised 3D dynamic scenes, and generate three-dimensional rendered images from a new perspective to improve the understanding of real-world scenes.

[0054] The semantic volume slot attention module of the embodiment of the present disclosure can project the learned 3D voxel grid to 2D and use a pre-trained 2D semantic feature extractor to extract 2D semantic features. The 2D semantic features can be used as supervision when training the semantic volume slot attention module to integrate prior knowledge of object geometry, help understand the structure of unfamiliar objects, and enhance the module's ability to interpret complex real-world scenes.

[0055] The embodiments of the present disclosure implement an unsupervised voxelization method, which can allow fine-grained separation of object-centric information in 3D space to achieve better scene understanding; and can also directly manipulate three-dimensional voxel grids or bidirectional dynamic modules to achieve direct scene editing, such as object removal, replacement and trajectory modification, without the need for additional training. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Other features, objects and advantages of the present disclosure will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0057] Figure 1 It is a block diagram of a voxel segmentation system for physical dynamic visual understanding according to an exemplary embodiment.

[0058] Figure 2 The present invention is a flowchart showing an execution flow of a voxel segmentation system for physical dynamic visual understanding according to an exemplary embodiment.

[0059] Figure 3 It is a flowchart of a voxel segmentation method for physical dynamic visual understanding according to an exemplary embodiment.

[0060] Figure 4It is a flowchart of a warm-up phase, an object-centered voxel grid initialization phase, and a multi-grid joint optimization phase training according to an exemplary embodiment.

[0061] Figure 5 The figure is a schematic diagram showing rendering and segmentation effects according to an exemplary embodiment. DETAILED DESCRIPTION

[0062] The present disclosure is described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art further understand the present disclosure, but are not intended to limit the present disclosure in any way. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the scope of the present disclosure. These modifications and improvements are all within the scope of protection of the present disclosure.

[0063] Figure 1 It is a block diagram of a voxel segmentation system for physical dynamic visual understanding according to an exemplary embodiment. Figure 2 The present invention is a flowchart showing an execution flow of a voxel segmentation system for physical dynamic visual understanding according to an exemplary embodiment.

[0064] like Figure 1 、 Figure 2 As shown, the present disclosure provides a voxel segmentation system for physical dynamic visual understanding, including: a bidirectional dynamic module, a semantic volume slot attention module, and an object-centric neural renderer module.

[0065] The bidirectional dynamic module inputs the 3D voxel grid data of the object, is used to learn the spatiotemporal deformation field of the voxels, and outputs the 3D voxel grid after spatiotemporal deformation.

[0066] Among them, the bidirectional dynamic module can learn the relative position transformation relationship between the voxel at any time t and the initial time, and determine the corresponding position information of the voxel at each moment after spatiotemporal deformation.

[0067] The semantic volume slot attention module takes as input a 3D voxel grid that has been deformed in space and time. It is used to derive global time-invariant semantic features from local time-varying voxel features, fuse 2D semantic features, determine the 3D global time-invariant semantic features, and output a 3D voxel grid with semantic information.

[0068] Among them, the semantic volume slot attention module adopts the slot attention mechanism.

[0069] The object-centric neural renderer module inputs a three-dimensional voxel grid that has undergone spatiotemporal deformation, a three-dimensional voxel grid with semantic information, global object-centric features, and a preset perspective. It is used to predict the object-centric color value of each voxel, and uses a preset combined neural radiation field rendering to generate a three-dimensional rendered image corresponding to the preset perspective, and outputs a three-dimensional rendered image corresponding to the preset perspective.

[0070] Among them, the global object-centric features represent a set of learnable parameters, and the global object-centric features are optimized and obtained simultaneously during the process of training the object-centric neural renderer module.

[0071] The preset combined Neural Radiance Fields (NeRF) are used to render pixels and generate a three-dimensional rendered image.

[0072] Preset perspectives can adopt different camera perspectives.

[0073] Through the above technical solution, an object-centered representation is adopted, and a bidirectional dynamic module is used to learn the spatiotemporal variation field of voxels to capture the local spatial structure of each object and perceive the changes of objects over time, so as to realize the object-centered voxelization method to infer the occupancy probability distribution of objects at each spatial position, thereby promoting 3D consistent scene decomposition; a semantic volume slot attention module is adopted to introduce semantic features to enhance object-centered representation learning, and the global non-time-varying semantic features derived from the local time-varying voxel features are fused with 2D semantic features, thereby promoting the 2D semantic features to 3D, integrating the prior knowledge of object geometry, and improving the ability to interpret complex real-world scenes; an object-centered neural renderer module is adopted to introduce a preset neural radiation field to solve the object-centered decomposition problem of unsupervised 3D dynamic scenes, and generate three-dimensional rendered images from a new perspective to improve the understanding of real-world scenes.

[0074] In a possible embodiment, the bidirectional dynamic module includes a reverse dynamic module and a forward dynamic module.

[0075] The reverse dynamic module is used to learn the displacement information of the voxels at each moment relative to the initial moment and determine the position information of the voxels at the initial moment.

[0076] Among them, the reverse dynamic module is represented by f ψ (x i ,t), where x i Represents the position information of the voxel at the initial moment, t represents time t, and the reverse dynamic module learns the displacement of the voxel at time t relative to the initial moment.

[0077] The forward dynamic module is used to learn the displacement information of the voxel at the initial moment relative to each moment and determine the position information of the voxel at each moment.

[0078] Among them, the forward dynamic module represents f′ ξ (x i ,t),x i Represents the position information of the voxel at the initial moment, t represents the time t, and the forward dynamic module learns the displacement of the voxel at the initial moment relative to the time t.

[0079] The following formula is used to calculate the position information of the voxel at the initial moment and the position information of the voxel at each moment:

[0080] x i′ =x i +Δx i

[0081] Among them, x i' Represents the position information of the voxel at time t, x i Indicates the position information of the voxel at the initial moment, Δx i Represents the displacement of the voxel at time t relative to the initial time.

[0082] In the present disclosure, the reverse dynamic module and the forward dynamic module simultaneously learn the displacement information of the voxel at each moment relative to the initial moment and the displacement information of the voxel at the initial moment relative to each moment. The reverse dynamic module and the forward dynamic module are trained simultaneously using cycle consistency loss, that is, the reverse dynamic module and the forward dynamic module work and are trained simultaneously.

[0083] In one possible embodiment, the semantic volume slot attention module includes: a semantic voxel grid, slot features, and a pre-trained 2D semantic extractor.

[0084] The semantic voxel grid is used to divide the three-dimensional voxel grid after spatiotemporal deformation into a semantic voxel grid with semantic labels.

[0085] where the semantic voxel grid is denoted as ν Sem .

[0086] Slot features are used to derive global time-invariant semantic features from the local time-varying voxel features of the semantic voxel grid with semantic labels through the slot attention mechanism.

[0087] Among them, the slot feature is a set of learnable object-centered slot features S.

[0088] In the present disclosure, the semantic voxel grid ν Sem And a set of learnable object-centered slot features S, trained using the slot attention method, the slot attention formula is as follows:

[0089]

[0090] Among them, A i represents the attention weight, N represents the number of slots, represents semantic features, S represents the slot feature centered on the object, τ represents the temperature coefficient, and D s represents the dimension of the slot feature, Represents the updated semantic features.

[0091] A pre-trained 2D semantic feature extractor is used to extract 2D semantic features from 2D image frames of monocular video, and fuse the 2D semantic features with global time-invariant semantic features to determine three-dimensional global time-invariant semantic features.

[0092] Among them, the 2D image frame is represented as a 2D image obtained by projecting a monocular video of a dynamic scene shot by a monocular camera onto a 2D feature map using a three-dimensional semantic voxel grid with semantic labels, and using a pre-trained 2D semantic feature extractor to extract 2D semantic features in the 2D image to obtain a 2D image frame.

[0093] The 2D semantic features can also be used as supervision during the training of the semantic volume slot attention module, and compared and learned with the 3D voxel features, thereby improving the ability of the slot features to derive global time-invariant semantic features from the local time-varying voxel features in the 3D voxel features.

[0094] The 2D semantic features are fused with the 3D global time-invariant semantic features to form a unified 3D semantic feature representation, that is, three-dimensional global time-invariant semantic features, thereby upgrading the 2D semantic features to 3D.

[0095] The semantic volume slot attention module disclosed in the present invention can project the learned 3D voxel grid into 2D and use a pre-trained 2D semantic feature extractor to extract 2D semantic features. The 2D semantic features can serve as supervision when training the semantic volume slot attention module to integrate prior knowledge of object geometry, help understand the structure of unfamiliar objects, and enhance the module's ability to interpret complex real-world scenes.

[0096] In one possible embodiment, the object-centric neural renderer module includes a preset multi-layer perceptron neural network and a preset combined neural radiance field.

[0097] The object-centric neural renderer module learns a multi-layer perceptron neural network, a pre-defined multi-layer perceptron network, to predict the object-centric color value of voxels.

[0098] The preset multi-layer perceptron neural network is used to input a three-dimensional voxel grid that has undergone spatiotemporal deformation and a three-dimensional voxel grid with semantic information, and output the color value of each voxel centered on the object.

[0099] Among them, the global object-centered features are a set of learnable parameters, expressed as

[0100] The preset multi-layer perceptron neural network transforms the global object-centered features Perform linear combination to obtain the features after linear combination Then a multi-layer perceptron (MLP) is used to learn the mapping function to output the color value of the voxel centered on the object:

[0101]

[0102] in, represents color-related features, d represents the viewing direction of the camera, represents the features after linear combination, c i Represents the color value of the voxel centered at the object.

[0103] A preset combined neural radiation field is used to input the object-centered color value of each voxel, the global object-centered feature and the preset perspective, and output a generated three-dimensional rendered image corresponding to the preset perspective.

[0104] Among them, the preset combined neural radiation field also requires the input of the density value of the retrieved voxel. It adopts the volume rendering method to output the color of the pixel according to the input voxel color value centered on the object and the retrieved density value to form a three-dimensional rendered image.

[0105] Figure 3 It is a flowchart of a voxel segmentation method for physical dynamic visual understanding according to an exemplary embodiment.

[0106] like Figure 3 As shown, the present disclosure further provides a voxel segmentation method for physical dynamic visual understanding, including S11 to S13. The voxel segmentation method for physical dynamic visual understanding disclosed in the present disclosure is implemented using the above-mentioned voxel segmentation system for physical dynamic visual understanding, which includes a bidirectional dynamic module, a semantic volume slot attention module, and an object-centric neural renderer module.

[0107] S11, according to the three-dimensional voxel grid data of the object, using a bidirectional dynamic module, to determine the three-dimensional voxel grid after the spatiotemporal deformation.

[0108] The three-dimensional voxel grid after the spatiotemporal deformation includes the position information corresponding to the voxels at each moment after the spatiotemporal deformation and the relative position transformation relationship between the voxels at any moment t and the initial moment.

[0109] S12, based on the three-dimensional voxel grid after spatiotemporal deformation, a semantic volume slot attention module is used to determine the three-dimensional voxel grid with semantic information.

[0110] Among them, the semantic volume slot attention module adopts the slot attention mechanism.

[0111] S13, based on the spatiotemporally deformed three-dimensional voxel grid, the three-dimensional voxel grid with semantic information, the global object-centered features and the preset perspective, an object-centered neural renderer module is used to determine a three-dimensional rendered image corresponding to the preset perspective.

[0112] Among them, the global object-centric features represent a set of learnable parameters, and the global object-centric features are optimized and obtained simultaneously during the process of training the object-centric neural renderer module.

[0113] Through the above technical solution, an object-centered representation is adopted, and a bidirectional dynamic module is used to learn the spatiotemporal variation field of voxels to capture the local spatial structure of each object and perceive the changes of objects over time, so as to realize the object-centered voxelization method to infer the occupancy probability distribution of objects at each spatial position, thereby promoting 3D consistent scene decomposition; a semantic volume slot attention module is adopted to introduce semantic features to enhance object-centered representation learning, and the global non-time-varying semantic features derived from the local time-varying voxel features are fused with 2D semantic features, thereby promoting the 2D semantic features to 3D, integrating the prior knowledge of object geometry, and improving the ability to interpret complex real-world scenes; an object-centered neural renderer module is adopted to introduce a preset combined neural radiation field to solve the object-centered decomposition problem of unsupervised 3D dynamic scenes, and generate three-dimensional rendered images from a new perspective to improve the understanding of real-world scenes.

[0114] In a possible embodiment, the bidirectional dynamic module includes a reverse dynamic module and a forward dynamic module.

[0115] In a possible embodiment, S11, using a bidirectional dynamic module to determine a three-dimensional voxel grid after spatiotemporal deformation based on the three-dimensional voxel grid data of the object, may include S21 to S23.

[0116] S21, using a reverse dynamic module to learn the displacement information of the voxels at each moment relative to the initial moment, and determine the position information of the voxels at the initial moment.

[0117] Among them, the reverse dynamic module is represented by fψ (x i ,t), where x i Represents the position information of the voxel at the initial moment, t represents time t, and the reverse dynamic module learns the displacement of the voxel at time t relative to the initial moment.

[0118] S22, using a forward dynamic module to simultaneously learn the displacement information of the voxels at the initial moment relative to each moment, and determine the position information of the voxels at each moment.

[0119] Among them, the forward dynamic module represents f′ ξ (x i ,t),x i Represents the position information of the voxel at the initial moment, t represents the time t, and the forward dynamic module learns the displacement of the voxel at the initial moment relative to the time t.

[0120] The following formula is used to calculate the position information of the voxel at the initial moment and the position information of the voxel at each moment:

[0121] x i′ =x i +Δx i

[0122] Among them, x i' Represents the position information of the voxel at time t, x i Indicates the position information of the voxel at the initial moment, Δx i Represents the displacement of the voxel at time t relative to the initial time.

[0123] In the present disclosure, step S21 and step S22 are performed simultaneously.

[0124] S23 , determining a three-dimensional voxel grid after the spatiotemporal deformation according to the voxel position information at the initial moment and the voxel position information at each moment.

[0125] In one possible embodiment, the semantic volume slot attention module includes a semantic voxel grid, slot features, and a pre-trained 2D semantic feature extractor.

[0126] In a possible embodiment, S12, using a semantic volume slot attention module to determine a three-dimensional voxel grid with semantic information based on the three-dimensional voxel grid after spatiotemporal deformation, may include S31 to S34.

[0127] S31, using a semantic voxel grid to divide the spatiotemporally deformed three-dimensional voxel grid into semantic voxel grids with semantic labels.

[0128] where the semantic voxel grid is denoted as ν Sem .

[0129] S32, slot features are used to extract each local time-varying voxel feature in the semantic voxel grid with semantic labels through the slot attention mechanism.

[0130] Among them, the slot feature is a set of learnable object-centered slot features S.

[0131] In the present disclosure, the semantic voxel grid ν Sem And a set of learnable object-centered slot features S, trained using the slot attention method, the slot attention formula is as follows:

[0132]

[0133] Among them, A i represents the attention weight, N represents the number of slots, represents semantic features, S represents the slot feature centered on the object, τ represents the temperature coefficient, and D s represents the dimension of the slot feature, Represents the updated semantic features.

[0134] S33: derive a global time-invariant semantic feature based on each local time-varying voxel feature in the semantic voxel grid with the semantic label.

[0135] S34 uses a pre-trained 2D semantic feature extractor to extract 2D semantic information from the 2D image frames of the monocular video, and fuses the 2D semantic information with the global non-time-varying semantic features based on the slot attention mechanism to determine the three-dimensional global non-time-varying semantic features and the three-dimensional voxel grid with semantic information.

[0136] Among them, the 2D image frame is represented as a 2D image obtained by projecting a monocular video of a dynamic scene shot by a monocular camera onto a 2D feature map using a three-dimensional semantic voxel grid with semantic labels, and using a pre-trained 2D semantic feature extractor to extract 2D semantic features in the 2D image to obtain a 2D image frame.

[0137] The 2D semantic features can also be used as supervision during the training of the semantic volume slot attention module, and compared and learned with the 3D voxel features, thereby improving the ability of the slot features to derive global time-invariant semantic features from the local time-varying voxel features in the 3D voxel features.

[0138] The 2D semantic features are fused with the 3D global time-invariant semantic features to form a unified 3D semantic feature representation, that is, three-dimensional global time-invariant semantic features, thereby upgrading the 2D semantic features to 3D.

[0139] The semantic volume slot attention module disclosed in the present invention can project the learned 3D voxel grid into 2D and use a pre-trained 2D semantic feature extractor to extract 2D semantic features. The 2D semantic features can serve as supervision when training the semantic volume slot attention module to integrate prior knowledge of object geometry, help understand the structure of unfamiliar objects, and enhance the module's ability to interpret complex real-world scenes.

[0140] In one possible embodiment, the object-centric neural renderer module includes a preset multi-layer perceptron neural network and a preset combined neural radiance field.

[0141] Among them, the object-centered neural renderer module learns a multi-layer perceptron neural network to predict the color value of the voxel centered on the object, that is, the preset multi-layer perceptron neural network.

[0142] In a possible embodiment, S13, based on the three-dimensional voxel grid after spatiotemporal deformation, the three-dimensional voxel grid with semantic information, the global object-centered features and the preset perspective, an object-centered neural renderer module is used to determine the three-dimensional rendered image corresponding to the preset perspective, which may include S41 to S42.

[0143] S41, inputting the spatiotemporally deformed three-dimensional voxel grid and the three-dimensional voxel grid with semantic information into a preset multi-layer perceptron neural network, and outputting the color value of each voxel centered on the object.

[0144] Among them, the global object-centered features are a set of learnable parameters, expressed as

[0145] The preset multi-layer perceptron neural network transforms the global object-centered features Perform linear combination to obtain the features after linear combination Then a multi-layer perceptron (MLP) is used to learn the mapping function to output the color value of the voxel centered on the object:

[0146]

[0147] in, represents color-related features, d represents the viewing direction of the camera, represents the features after linear combination, c i Represents the color value of the voxel centered at the object.

[0148] S42 , inputting the object-centered color value of each voxel, the global object-centered feature, and the preset viewing angle into a preset combined neural radiation field to determine a three-dimensional rendered image corresponding to the preset viewing angle.

[0149] Among them, the preset combined neural radiation field also requires the input of the density value of the retrieved voxel. It adopts the volume rendering method to output the color of the pixel according to the input voxel color value centered on the object and the retrieved density value to form a three-dimensional rendered image.

[0150] Through this disclosure, an unsupervised voxelization method is implemented, which can allow fine-grained separation of object-centric information in 3D space, achieving better scene understanding; and can also directly manipulate three-dimensional voxel grids or bidirectional dynamic modules to achieve direct scene editing, such as object removal, replacement and trajectory modification, without the need for additional training.

[0151] Figure 4 It is a flowchart of a warm-up phase, an object-centered voxel grid initialization phase, and a multi-grid joint optimization phase training according to an exemplary embodiment.

[0152] like Figure 4 As shown, in one possible embodiment, a method for training a bidirectional dynamic module, a semantic volume slot attention module, and an object-centric neural renderer module includes S14 to S17.

[0153] The process of training the bidirectional dynamic module, the semantic volume slot attention module, and the object-centric neural renderer module includes three stages: the warm-up stage, the object-centric voxel grid initialization stage, and the multi-grid joint optimization stage.

[0154] Among them, the warm-up phase is used to learn basic geometric, semantic and dynamic priors.

[0155] In the present disclosure, S14 to S15 represent a preheating stage.

[0156] S14: extract a feature map from a preset monocular video image frame using a pre-trained model to determine a feature map.

[0157] As an example, the pre-trained model can use the pre-trained DINOv2, and the preset monocular video image frame is represented as The feature map is represented as

[0158] S15, using feature maps to train the bidirectional dynamic module, the semantic volume slot attention module and the non-combination neural renderer to determine the trained bidirectional dynamic module, the trained semantic volume slot attention module and the trained non-combination neural renderer.

[0159] Among them, the semantic volume slot attention module includes a geometric voxel grid, a color-related feature grid, a semantic-related feature grid and a slot feature, and the trained semantic volume slot attention module includes a trained geometric voxel grid, a trained color-related feature grid, a trained semantic-related feature grid and a trained slot feature.

[0160] where the geometric voxel grid is denoted as ν Opac , the color-related feature grid is denoted as ν Color , the semantically relevant feature grid is denoted as ν Sem , slot features are represented as S, non-combination neural renderer N φ′ (·).

[0161] The bidirectional dynamic module includes the reverse dynamic module f ψ (x i ,t) and the forward dynamic module f′ ξ (x i ,t).

[0162] The non-combinatorial neural renderer is only applied during the warm-up phase. The non-combinatorial neural renderer is an object-centric neural renderer module that removes global object features.

[0163] Specifically, the method of using feature maps to train the bidirectional dynamic module, the semantic volume slot attention module, and the non-combination neural renderer includes:

[0164] The feature map is rendered using a bidirectional dynamic module, a semantic volume slot attention module, and a non-combination neural renderer to determine the predicted feature map.

[0165] Compare the predicted feature map with the original feature map and determine the loss function;

[0166] Gradient rotation is performed according to the loss function to train the bidirectional dynamic module, the semantic volume slot attention module and the non-combination neural renderer, and the trained bidirectional dynamic module, the trained semantic volume slot attention module and the trained non-combination neural renderer are determined.

[0167] In the present disclosure, S16 represents the object-centered voxel grid initialization stage.

[0168] S16, initializing a voxel grid centered on the object, and determining the initialized voxel grid centered on the object.

[0169] In this disclosure, S17 represents the multi-grid joint optimization stage.

[0170] S17, jointly optimize the reverse dynamic module, the trained geometric voxel grid, the trained color-related feature grid, and the initialized object-centered voxel grid in the trained bidirectional dynamic module, and retrain the global object-centered features and the object-centered neural renderer module to determine the trained bidirectional dynamic module, the trained volume slot attention module, and the trained object-centered neural renderer module.

[0171] In a possible embodiment, S16 , initializing a voxel grid centered on the object, and determining the initialized voxel grid centered on the object, includes: S51 to S55 .

[0172] Initialize the voxel grid ν centered on the object Occ , including: (1) generating feature maps and calculating connected components; (2) using conditional random fields for post-processing.

[0173] S51 : taking a portion of a voxel grid centered on the object and having a density greater than a preset threshold as a valid voxel.

[0174] Among them, the preset density threshold is expressed as δ den .

[0175] S52: Determine the valid voxels as nodes of the feature map.

[0176] S53, constructing connections between nodes based on the positions, colors, speeds, and semantic similarities of adjacent nodes of the nodes in the feature graph, and determining the feature graph.

[0177] Among them, the similarity is measured by the bi-norm distance between the position, color, speed and semantic features of the adjacent nodes of the feature map and the corresponding feature vector of the node.

[0178] Specifically, for adjacent nodes, if the similarity of all the above-mentioned positions, colors, speeds, and semantic features is higher than a preset threshold, that is, the two-norm distance of the feature vector is lower than the preset two-norm distance threshold, a connection line is constructed between the adjacent nodes to determine the feature map.

[0179] S53, using a connected graph component algorithm to perform clustering processing on the feature graph to determine a graph clustering result of the feature graph.

[0180] Among them, the connected graph component algorithm is implemented on the feature graph to determine M graph clusters, each of which represents an object.

[0181] The number of preset objects is N. If M≤N, M graph clusters are used as clustering results; if M>N, N clustering results are determined in descending order according to the contribution of each graph cluster to the rendering result, and the remaining graph clusters are left blank.

[0182] S54 , based on the graph clustering result of the feature graph, interpolate the object-centered voxel grid corresponding to the node of the non-graph clustering result using the nearest neighbor interpolation method to determine the interpolated object-centered voxel grid.

[0183] As an example, if the first N graph clusters are selected as the clustering results, the remaining voxel grids are interpolated to determine an interpolated voxel grid centered on the object.

[0184] S55 , using a conditional random field algorithm to smooth the interpolated voxel grid centered on the object, to determine an initialized voxel grid centered on the object.

[0185] Among them, the conditional random field algorithm smoothes the voxel grid and removes noise by calculating the similarity of color, position and semantic features between adjacent pixels, and also uses RGB information to refine the 2D mask prediction.

[0186] Moreover, the refined mask is used as a pseudo-label in the subsequent stage to solve the problem that the direct application of the refined 2D mask prediction affects the 4D consistency of the segmentation result, and to achieve fine-tuning of the 4D object-centered voxel grid ν Occ , ensuring consistency of object-centric representation.

[0187] The conditional random field algorithm is used to reduce the noise introduced by the connected component algorithm and smooth the segmentation results.

[0188] like Figure 4 As shown, in a possible embodiment, S17, jointly optimizing the reverse dynamic module, the trained geometric voxel grid, the trained color-related feature grid, and the initialized object-centered voxel grid in the trained bidirectional dynamic module, and retraining the global object-centered feature and the object-centered neural renderer module, and determining the trained bidirectional dynamic module, the trained volume slot attention module, and the trained object-centered neural renderer module, may include:

[0189] The multi-grid joint optimization stage is used to train the reverse dynamic module f ψ (·), the trained geometric voxel grid ν Opac , trained color-related feature grid ν Color and an initialized object-centered voxel grid ν Occ Fine-tune and restart training of global object-centric features and object-centric neural renderer N φ (·).

[0190] The entire training is carried out in an end-to-end manner, and all modules work together to contribute to object-centric dynamic scene understanding.

[0191] In this stage, the following loss function is used for optimization:

[0192] L Dyn =L Render +α p L Point +α b L Bg-Ent +α c L CE

[0193] in, Represents the loss function of this stage, represents the rendering loss, represents the RGB loss per point, α p Represents the weight of each point RGB loss, represents the background entropy loss, α b represents the weight of background entropy loss, represents the cross entropy loss, α c The weight representing the cross entropy loss

[0194] Among them, the first three are used to focus on scene reconstruction:

[0195]

[0196]

[0197]

[0198] in, represents the RGB value obtained by model rendering, C(r) represents the RGB value of the observed image, Represents all sampling rays, r represents the current sampling ray, Indicates the transparency of the current light, P indicates the number of sampling points on each light, Indicates the average color of all sampling points

[0199] In particular, L CE represents the voxel grid ν centered on the object Occ Rendered 2D segmentation and the pseudo labels obtained from the previous stage The cross entropy loss between .

[0200]

[0201] in, represents a pseudo label, represents 2D segmentation, LCE represents the cross entropy loss between 2D segmentation and pseudo labels.

[0202] Figure 5 The figure is a schematic diagram showing rendering and segmentation effects according to an exemplary embodiment.

[0203] like Figure 5 As shown, the first row of images represents the rendered RGB images, the second row of images represents the object-centered mask images of the first row of images, the third row of images represents the rendered RGB images, and the fourth row of images represents the object-centered mask images of the third row of images.

[0204] like Figure 5 As shown, the present disclosure provides a specific application embodiment, which adopts a voxel segmentation system and method for physical dynamic visual understanding provided by the present disclosure to solve the voxel-level modeling and segmentation problems of complex geometric structures and dynamic objects in real scenes.

[0205] In this embodiment, a monocular camera captures a real-world video of a scene, including the dynamic process of a person watering plants with a handheld kettle. Based on this video, a model is trained to achieve 3D voxel reconstruction and accurate segmentation of the scene.

[0206] The following technical difficulties exist in the reconstruction and segmentation tasks of the above-mentioned dynamic scenes: (1) The structure of plant branches and leaves is complex and requires fine-grained segmentation; (2) The scene background is cluttered and there are multiple objects interfering; (3) Some objects are only visible for part of the time in the video sequence and are not continuously in the field of view of the monocular camera.

[0207] The technical solution of this embodiment includes the following steps:

[0208] (1) Warm-up phase: First, the pre-trained visual base model is used to train the video frame sequence. Perform semantic feature extraction to obtain the semantic feature map sequence corresponding to the video frame sequence Then, the inverse dynamic module is used to construct the spatiotemporal deformation field, using the video frame sequence And the corresponding semantic feature map sequence For supervision signals, the geometric voxel grid, color-related feature grid, semantic-related feature grid, slot features, reverse dynamic module and non-combination neural renderer are initially trained. At the same time, the forward dynamic module is trained using cycle consistency loss. The spatial resolution of all voxel grids at this stage is uniformly set to 160 3 After training, the slot features can represent the semantic information of different objects in the scene (such as human hands, kettles, plants, and decorations), while the forward dynamic module effectively records the unique motion patterns of each object, making it easier to distinguish objects with different motion patterns, such as a moving hand, a slightly swaying plant, and a stationary decoration.

[0209] (2) Object-centered voxel grid initialization stage: Based on the geometric voxel grid, color-related feature grid, semantic-related feature grid and forward dynamic module obtained in the warm-up stage training, a feature map is constructed, and then the connected graph component algorithm is applied to perform cluster analysis to obtain several clusters. Each cluster clearly and roughly represents the probability distribution of each voxel belonging to a specific object (such as a hand, a kettle, a plant or various decorations), forming an occupancy probability grid. The clustering process comprehensively considers the multi-dimensional features of the object, such as color information, semantic features and motion patterns. Next, the occupancy probability grid is rendered to obtain a two-dimensional mask prediction map, and the conditional random field algorithm is used to smooth and refine the two-dimensional mask prediction map. The processed results are used as pseudo labels to fine-tune the occupancy probability grid to further improve the fineness of the segmentation.

[0210] (3) Multi-grid joint optimization stage: using the original video image frame sequence As supervision signals, the inverse dynamics module, geometric voxel grid, color-related feature grid and occupancy probability grid are fine-tuned in an end-to-end manner, and a set of global object-centric features and object-centric neural renderers are trained from scratch to obtain new perspective images through combined neural rendering.

[0211] The above describes specific embodiments of the present disclosure. It should be understood that the present disclosure is not limited to the specific embodiments described above, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present disclosure. The above preferred features may be used in any combination as long as they do not conflict with each other.

Claims

1. A voxel segmentation system for physical dynamic visual understanding, characterized by: include: A bidirectional dynamic module, which inputs the three-dimensional voxel grid data of the object, is used to learn the spatiotemporal deformation field of the voxels, and outputs the three-dimensional voxel grid after the spatiotemporal deformation; a semantic volume slot attention module, which inputs the spatiotemporally deformed 3D voxel grid, derives global time-invariant semantic features from local time-varying voxel features, fuses 2D semantic features, determines 3D global time-invariant semantic features, and outputs a 3D voxel grid with semantic information; The object-centered neural renderer module inputs the three-dimensional voxel grid after spatiotemporal deformation, the three-dimensional voxel grid with semantic information, the global object-centered features and the preset perspective, and is used to predict the object-centered color value of each voxel, and uses a preset combined neural radiation field rendering to generate a three-dimensional rendered image corresponding to the preset perspective, and outputs the three-dimensional rendered image corresponding to the preset perspective.

2. The voxel segmentation system for physical dynamic visual understanding according to claim 1, characterized in that: The bidirectional dynamic module includes: The reverse dynamic module is used to learn the displacement information of the voxels at each moment relative to the initial moment and determine the position information of the voxels at the initial moment; The forward dynamic module is used to learn the displacement information of the voxel at the initial moment relative to each moment, and determine the position information of the voxel at each moment.

3. The voxel segmentation system for physical dynamic visual understanding according to claim 1, characterized in that: The semantic volume slot attention module includes: A semantic voxel grid, used to divide the three-dimensional voxel grid after the spatiotemporal deformation into semantic voxel grids with semantic labels; A slot feature is used to derive a global time-invariant semantic feature from the local time-varying voxel features of the semantic voxel grid with the semantic label through a slot attention mechanism; A pre-trained 2D semantic feature extractor is used to extract 2D semantic features from 2D image frames of a monocular video, and fuse the 2D semantic features with the global time-invariant semantic features to determine the three-dimensional global time-invariant semantic features.

4. The voxel segmentation system for physical dynamic visual understanding according to claim 1, characterized in that: The object-centric neural renderer module includes: A preset multi-layer perceptron neural network is used to input the spatiotemporally deformed three-dimensional voxel grid and the three-dimensional voxel grid with semantic information, and output a color value of each voxel centered on the object; A preset combined neural radiation field is used to input the object-centered color value of each voxel, the global object-centered feature and the preset perspective, and output a generated three-dimensional rendered image corresponding to the preset perspective, wherein the global object-centered feature represents a set of learnable parameters.

5. A voxel segmentation method for physical dynamic visual understanding, characterized by: include: According to the 3D voxel grid data of the object, a bidirectional dynamic module is used to determine the 3D voxel grid after the spatiotemporal deformation; Determining a three-dimensional voxel grid having semantic information using a semantic volume slot attention module based on the three-dimensional voxel grid after the spatiotemporal deformation; Based on the three-dimensional voxel grid after spatiotemporal deformation, the three-dimensional voxel grid with semantic information, the global object-centric features and the preset perspective, an object-centric neural renderer module is used to determine the three-dimensional rendered image corresponding to the preset perspective, and the global object-centric features represent a set of learnable parameters.

6. The method according to claim 5, characterized in that The bidirectional dynamic module includes a reverse dynamic module and a forward dynamic module; The method of determining the spatiotemporally deformed three-dimensional voxel grid using a bidirectional dynamic module based on the three-dimensional voxel grid data of the object includes: Using the reverse dynamic module to learn the displacement information of the voxels at each moment relative to the initial moment, and determine the position information of the voxels at the initial moment; The forward dynamic module is used to simultaneously learn the displacement information of the voxels at the initial moment relative to each moment, and determine the position information of the voxels at each moment; The three-dimensional voxel grid after the spatiotemporal deformation is determined according to the position information of the voxels at the initial moment and the position information of the voxels at each moment.

7. The method according to claim 5, characterized in that The semantic volume slot attention module includes a semantic voxel grid, slot features, and a pre-trained 2D semantic feature extractor; Determining a three-dimensional voxel grid having semantic information using a semantic volume slot attention module based on the spatiotemporally deformed three-dimensional voxel grid includes: Using the semantic voxel grid, the three-dimensional voxel grid after the spatiotemporal deformation is divided into semantic voxel grids with semantic labels; Using the slot features to extract each local time-varying voxel feature in the semantic voxel grid with the semantic label through a slot attention mechanism; deriving a global time-invariant semantic feature based on each local time-varying voxel feature in the semantic voxel grid with the semantic label; The pre-trained 2D semantic feature extractor is used to extract 2D semantic information from the 2D image frames of the monocular video, and the 2D semantic information and the global time-invariant semantic features are fused based on the slot attention mechanism to determine the three-dimensional global time-invariant semantic features and the three-dimensional voxel grid with semantic information.

8. The method according to claim 5, characterized in that The object-centric neural renderer module includes a preset multi-layer perceptron neural network and a preset combined neural radiation field; The method further comprises: determining a three-dimensional rendered image corresponding to a preset perspective using an object-centered neural renderer module based on the spatiotemporally deformed three-dimensional voxel grid, the three-dimensional voxel grid with semantic information, a global object-centered feature, and a preset perspective, including: Inputting the spatiotemporally deformed three-dimensional voxel grid and the three-dimensional voxel grid with semantic information into the preset multi-layer perceptron neural network, and outputting the color value of each voxel centered on the object; The object-centered color value of each voxel, the global object-centered feature, and the preset viewing angle are input into the preset combined neural radiation field to determine a three-dimensional rendered image corresponding to the preset viewing angle.

9. The method according to claim 5, characterized in that The method of training the bidirectional dynamic module, the semantic volume slot attention module, and the object-centric neural renderer module comprises: A pre-trained model is used to extract feature maps from preset monocular video image frames to determine feature maps; Using the feature map to train the bidirectional dynamic module, the semantic volume slot attention module, and the non-combination neural renderer, to determine a trained bidirectional dynamic module, a trained semantic volume slot attention module, and a trained non-combination neural renderer, wherein the semantic volume slot attention module includes a geometric voxel grid, a color-related feature grid, a semantic-related feature grid, and slot features, and the trained semantic volume slot attention module includes a trained geometric voxel grid, a trained color-related feature grid, a trained semantic-related feature grid, and a trained slot feature; Initializing a voxel grid centered on the object, and determining an initialized voxel grid centered on the object; The inverse dynamic module in the trained bidirectional dynamic module, the trained geometric voxel grid, the trained color-related feature grid and the initialized object-centered voxel grid are jointly optimized, and the global object-centered features and the object-centered neural renderer module are retrained to determine the trained bidirectional dynamic module, the trained volumetric slot attention module and the trained object-centered neural renderer module.

10. The method according to claim 9, characterized in that Initializing the voxel grid centered on the object and determining the initialized voxel grid centered on the object include: Taking the portion of the voxel grid centered on the object that is larger than a preset density threshold as a valid voxel; Determine the valid voxel as a node of a feature map; Constructing connections between the nodes based on the positions, colors, speeds, and semantic similarities of adjacent nodes of the nodes of the feature graph to determine the feature graph; Performing clustering processing on the feature graph using a connected graph component algorithm to determine a graph clustering result of the feature graph; interpolating the object-centered voxel grid corresponding to the node of the non-graph clustering result using a nearest neighbor interpolation method according to the graph clustering result of the feature graph to determine an interpolated object-centered voxel grid; A conditional random field algorithm is used to smooth the interpolated object-centered voxel grid to determine an initialized object-centered voxel grid.

Images