Three-dimensional semantic scene completion method and device based on geometric and time sequence modeling hierarchical context alignment and medium

Abstract

The invention relates to a three-dimensional semantic scene completion method and device based on geometric and time sequence modeling hierarchical context alignment, and a medium. The method comprises the following steps: obtaining depth features and context features, sensing a cross attention mechanism through depth confidence, supplementing information of a low depth confidence region by using the context features, and generating related features of a current frame; historical frame related features are obtained from the attitude network, cross-frame feature affinity is calculated, the historical frame related features are dynamically optimized, and multi-layer historical frame aggregation features are obtained; and in the unified space, the depth hypothesis of the time feature voxels is adopted as a distance axis, the volume feature voxels are projected to the unified space, global alignment and combination are performed on the geometric feature voxels and the time feature voxels, and final aggregation features are obtained. Compared with the prior art, by introducing a hierarchical context alignment mechanism based on geometry and time sequence modeling, a complex 3D scene can be more accurately understood, and the method is especially suitable for environmental perception in an automatic driving system.

Description

Technical Field

[0001] The present invention relates to the field of 3D visual perception, and in particular to a three-dimensional semantic scene completion method based on hierarchical context alignment of geometry and temporal modeling. Background Art

[0002] 3D semantic occupancy prediction (SOP) is one of the key tasks in fields such as autonomous driving and robot navigation. It aims to infer a comprehensive understanding of the 3D scene, including the position, shape, and semantic information of objects, from limited 2D image observations. Traditionally, such tasks mainly rely on LiDAR (Light Detection and Ranging) data because LiDAR can provide accurate 3D geometric information. However, due to the high cost and large size of LiDAR equipment, camera-based 3D semantic occupancy prediction has gradually become a research hotspot in recent years. Camera-based methods are not only cost-effective but also easy to integrate into various mobile platforms.

[0003] Existing camera-based SOP methods usually assist occupancy representation learning by aggregating contextual features to alleviate problems such as occlusion or blur. Although these methods have improved the ability to understand 3D scenes to a certain extent, they have significant problems in handling cross-frame feature alignment. Specifically, features at the same position in different frames may have different semantic meanings, which makes the context fusion results unreliable, thereby affecting the stability of representation learning. Chinese patent application CN202410593387.X discloses a semantic scene completion method, electronic device and storage medium. Although previous work has achieved reliable feature aggregation through temporal modeling and context learning, it has ignored the uncertainty in depth distribution estimation during the voxel feature establishment process, resulting in unreliable context fusion results. Summary of the Invention

[0004] The purpose of the present invention is to overcome the defect of unreliable context fusion results in the above-mentioned prior art and to provide a three-dimensional semantic scene completion method based on hierarchical context alignment of geometric and temporal modeling.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A 3D semantic scene completion method based on hierarchical context alignment of geometric and temporal modeling, comprising the following steps:

[0007] Obtain the depth features and context features of the current frame, and based on the depth confidence-aware cross-attention mechanism, use the context features to supplement the information of the low depth confidence area to generate the relevant features of the current frame;

[0008] Obtain historical frame related features from the posture network, calculate cross-frame feature affinity based on historical frame related features and current frame related features, and dynamically optimize historical frame related features based on cross-frame feature affinity to obtain multi-layer historical frame aggregated features;

[0009] In a unified space, for the aggregated features of multiple layers of historical frames, the depth hypothesis of the temporal feature voxels is used as the distance axis, and the volume features are projected into the unified space through voxel pooling operations; based on semantic consistency, the transformed geometric feature voxels and temporal feature voxels are globally aligned and combined with the features related to the current frame to obtain the final aggregated features and generate semantic scene completion results.

[0010] As an optimal technical solution, the depth features and context features are based on the LSS-based three-dimensional visual network paradigm to generate voxel feature volumes through the depth network and the context network.

[0011] As a preferred technical solution, the generation of the current frame related features is specifically as follows:

[0012] Adopt cross attention mechanism to flatten the depth feature F in spatial and depth dimensions d To obtain the query Q d , flatten the context features to get the key K c Sum V c , the optimized interaction feature F I :

[0013] F I =φ q (Q d )⊙C d (φ k (K c ) T V c )

[0014] Among them, φ q represents the softmax function along each row of the input matrix; G c Represents the context feature F c The global context vector of k represents the softmax function along each column of the input matrix; ⊙ represents the element-wise product; C d Indicates that the deep feature F d The confidence map obtained by projection is as follows:

[0015]

[0016] Where φ represents the softmax function applied to the depth dimension; WTA represents the winner-takes-all operation; F d is the depth feature, d iDepth proxy value in depth feature;

[0017] Interaction feature F I Further obtain the current frame related features through the 3D visual network paradigm

[0018] As a preferred technical solution, the cross-frame feature affinity calculation is specifically as follows:

[0019] For historical frame related features Features related to the current frame 3D dilated convolutions with different expansion rates are used to construct multiple historical context groups H with different pattern scales. i And multiple sets of context C for the current frame i :

[0020] Calculate the affinity A between the historical multiple contexts and the previous frame multiple contexts at scale i i , and the affinity matrix A of different group scales i Connect and aggregate along the channel dimension to obtain cross-frame affinity

[0021] As a preferred technical solution, in each group of affinity A i During calculation, the historical multiple-group context and the current frame multiple-group context are subtracted from their respective average values.

[0022] As an optimal technical solution, the posture network adopts the PoseNet model.

[0023] As a preferred technical solution, the dynamic optimization of historical frame related features based on affinity is specifically as follows:

[0024] Use 3D deformable convolution for dynamic feature aggregation, and combine the features sampled by deformable convolution with cross-frame affinity Multiply to aggregate effective related features. Within a sampling window, the feature aggregation process is expressed as follows:

[0025]

[0026] Among them, V def represents the output feature vector after deformable convolution processing; K w Indicates the total number of sampling points in the sampling window; w k represents the spatial feature weight; V tem Represents the template feature map; p represents the center coordinate of the current feature point; p k represents the standard sampling offset relative to the center point; Δp k Indicates the additional offset corresponding to the kth sampling point; a k Indicates the affinity from cross-frame affinity weight of ;

[0027] Through a multi-stage deformable block with multiple cascaded 3D deformable convolutions, the output feature vectors of multiple different 3D deformable convolution outputs are concatenated along the channel dimension, and then passed through a 3D convolution layer to obtain multi-layer historical frame aggregation features.

[0028] As an optimal technical solution, the geometric feature voxels and the temporal feature voxels are globally aligned and combined to obtain the final aggregated features, as follows:

[0029] The current frame related features and multi-layer historical frame aggregation features As input, The depth hypothesis is used as the distance axis, and the voxel pooling operation is used to splash the volume features into a unified space; the zero convolution aggregation is used and Get the aggregate volume:

[0030]

[0031] Among them, Zero_Conv represents the zero convolution in the standard ControlNet, Voxel_Pool represents the standard voxel pooling operation; V com Represents the final aggregated feature.

[0032] As a second aspect of the present invention, there is provided an electronic device, comprising:

[0033] one or more processors;

[0034] a memory for storing one or more programs;

[0035] When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional semantic scene completion method based on geometric and temporal modeling hierarchical context alignment as described above.

[0036] As a third aspect of the present invention, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the three-dimensional semantic scene completion method based on hierarchical context alignment of geometric and temporal modeling as described above.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1) The camera-based 3D semantic occupancy prediction method proposed in this paper introduces a geometric confidence perception module to explicitly model geometric structures with deeply distributed confidence, thereby achieving reliable volumetric feature alignment. Furthermore, a hierarchical context alignment mechanism is introduced to separate geometric and temporal context alignment, reducing errors in the alignment process. Furthermore, through global alignment and combination, hierarchical context modeling and alignment of both geometry and temporal context are simultaneously considered, ensuring the reliability of the context fusion results, improving the stability of representation learning, and enabling more accurate understanding of complex 3D scenes. This method is particularly suitable for environmental perception in autonomous driving systems.

[0039] 2) To globally align geometric and temporal context within a unified space and achieve semantically consistent aggregation, this paper designs a Deep Hypothesis Transform (DHBT) module to achieve a stable geometric-temporal volume combination. This transform aims to create a uniform context distribution, facilitate more reliable volumetric feature alignment, and improve adaptability to dynamically changing scenes, making it particularly suitable for applications such as autonomous driving.

[0040] 3) The present invention has stronger adaptability to dynamic scenes. Through the cross-frame pattern affinity module CPA module, historical frame related features are obtained from the camera posture network through depth confidence perception and camera posture prior information. Then, a cross-frame feature affinity module is constructed based on the historical frame related features and the current frame related features to measure the context feature correlation. The historical frame related features are dynamically optimized according to the cross-frame feature affinity, thereby improving the ability to capture fine-grained features. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of a three-dimensional semantic scene completion method based on hierarchical context alignment of geometric and temporal modeling according to the present invention. DETAILED DESCRIPTION

[0042] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0043] Example 1

[0044] This paper proposes a 3D semantic scene completion method based on hierarchical context alignment of geometric and temporal modeling. By separating geometric and temporal context alignment, the error in the alignment process is reduced and the accuracy of prediction is improved. And through global alignment and combination, the reliability of the context fusion results is ensured and the stability of representation learning is improved. Figure 1 As shown, the specific steps include:

[0045] Step 1: Design a geometric confidence perception module to model geometric information through deep distributed confidence perception and integrate it with the corresponding context features to generate current frame related features.

[0046] Furthermore, the geometric confidence perception module is specifically:

[0047] The voxel feature volume is generated by the deep network and the context network based on the LSS (lift, splat, shoot) 3D visual network paradigm. The deep features from the deep network and the context features from the context network are then used as input. The confidence information of the depth distribution is explicitly indicated through the depth confidence-aware cross-attention mechanism, and the relevant context is used to supplement the information of the low-confidence area:

[0048]

[0049] Among them, C d Indicates that the deep feature F d The confidence map obtained by projection; WTA stands for winner-take-all operation; φ represents the softmax function applied to the depth dimension; F d is the depth feature, d i is the depth proxy value in the depth feature, D max is the maximum depth.

[0050] Then, the general cross-attention mechanism is used to flatten the deep features F in both spatial and depth dimensions. d To obtain the query Q d .

[0051] Similarly, flatten the context features to get the key K c Sum V c , F I The interaction features obtained for optimization are:

[0052] F I =Atten(Q d ,K c ,V c )

[0053] =φ q (Q d )⊙C d (G c ),

[0054] =φ q (Q d )⊙C d (φ k (K c ) T V c ),

[0055] Among them, φ q and φ k Represents the softmax function along each row and column of the input matrix respectively; G c Represents the context feature F c The global context vector of ; ⊙ represents the element-wise product.

[0056] Interaction feature F I Further, the current frame related features are obtained through the mainstream LSS (lift, splat, shoot) 3D visual network paradigm

[0057] Step 2: Obtain historical frame related features from the posture network, and then construct a cross-frame feature affinity module (CPA) based on the historical frame related features and the current frame related features to measure the context feature correlation and dynamically optimize the historical frame related features based on the cross-frame feature affinity.

[0058] In this embodiment, the posture network adopts the PoseNet paradigm of temporal video depth estimation.

[0059] Furthermore, the cross-frame feature affinity is calculated based on the historical frame-related features and the current frame-related features, as follows:

[0060] Given the historical frame related features and the current frame related features, 3D hole convolution with different expansion rates is used to construct multiple sets of context features. Specifically, the historical frame related features After a set of dilated convolutions, multiple sets of historical context H are generated. i :

[0061]

[0062] Among them, GN and δ represent group normalization and GELU activation respectively. In this embodiment, the Atrous convolution i The expansion rates are 1, 2, and 4 respectively.

[0063] Current frame related features It is processed in a similar way to generate multiple sets of context C for the current frame i :

[0064]

[0065] Among them, GN and δ represent group normalization and GELU activation respectively. In this embodiment, the Atrous convolution i The expansion rates are 1, 2, and 4 respectively.

[0066] Next, the present invention considers different pattern scales of multiple sets of context features and calculates the affinity A of each scale i i , these independent affinities are further aggregated along the channel dimension. Meanwhile, the respective mean values ​​are subtracted during the affinity calculation of each group to achieve scale-aware isolation, which is expressed as:

[0067]

[0068] Among them, the affinity matrix A of different group scales i Connect along the channel dimension to obtain cross-frame affinity Input context matrix C i and H i is considered as a high-dimensional vector with different groups. and Represents the average context matrix for each group scale.

[0069] Furthermore, the historical frame-related features are dynamically optimized based on affinity, specifically:

[0070] Given cross-frame affinity The features related to historical frames are aggregated using a general 3D deformable convolution to perform dynamic feature aggregation. The features sampled by the deformable convolution are compared with the cross-frame affinity. Multiply together to aggregate effective related features.

[0071] Specifically, the present invention introduces corresponding weights based on feature affinity and deformable position offsets to further enhance the model's adaptability to complex scenes and matching accuracy.

[0072] Specifically, in a w Within the sampling window of sampling points, the feature aggregation process adopted by the present invention can be expressed as the following mathematical form:

[0073]

[0074] Where: V def represents the output feature vector after deformable convolution processing; K w Indicates the total number of sampling points in the sampling window; w k represents the spatial feature weight; V tem Represents the template feature map; p represents the center coordinate of the current feature point; p k represents the standard sampling offset relative to the center point; Δp k Indicates the additional offset corresponding to the kth sampling point; a k Represents the affinity matrix from the cross-frame mode affinity weight.

[0075] This calculation method can adaptively adjust the sampling position and weight distribution according to the image content, thereby effectively improving the robustness and accuracy of three-dimensional feature matching. It is suitable for various computer vision tasks including three-dimensional reconstruction, stereo matching, and video target tracking.

[0076] In order to utilize the aggregated relevant features and further reason about dynamic modeling through hierarchical context, this paper considers contextual information from different feature levels to optimize the refinement process and constructs a multi-level deformable block with three cascaded 3D deformable convolutions. Output of multi-layer historical frame aggregated features for:

[0077]

[0078] in, Denotes different 3D deformable convolutions, Concat denotes concatenation along the channel dimension, and W denotes a general 3D convolution layer.

[0079] Step 3: Globally align the geometric and temporal contexts in a unified space using a depth-hypothesis-based transformation (DHBT). First, the depth hypothesis of the temporal feature voxels is used as the distance axis, and the volumetric features are projected into the unified space through voxel pooling. The transformed geometric and temporal voxels are then globally aligned and combined based on semantic consistency to improve reliability.

[0080] Furthermore, the depth hypothesis-based transformation module globally aligns the geometric context and the temporal context as follows:

[0081] Given the current frame related features and multi-layer historical frame aggregation features As input, The depth hypothesis is used as the distance axis, and the standard voxel pooling operation is used to splash the volume features into a unified space. Then the zero convolution aggregation in the standard ControlNet is used. and To achieve reliable information exchange, the aggregate volume V is obtained com :

[0082]

[0083] Among them, Zero_Conv represents the zero convolution in the standard ControlNet, Voxel_Pool represents the standard voxel pooling operation; V com Represents the final aggregated features, which are used to generate semantic scene completion results.

[0084] Example 2

[0085] As a second aspect of the present invention, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional semantic scene completion method based on hierarchical context alignment as described above. In addition to the above-mentioned processors, memories, and interfaces, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0086] Example 3

[0087] As a third aspect of the present invention, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the three-dimensional semantic scene completion method based on hierarchical context alignment as described above. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities as described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (FlashCard), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0088] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A 3D semantic scene completion method based on hierarchical context alignment of geometric and temporal modeling, characterized by the following steps: include: Obtain the depth features and context features of the current frame, and based on the depth confidence-aware cross-attention mechanism, use the context features to supplement the information of the low depth confidence area to generate the relevant features of the current frame; Obtain historical frame related features from the posture network, calculate cross-frame feature affinity based on historical frame related features and current frame related features, and dynamically optimize historical frame related features based on cross-frame feature affinity to obtain multi-layer historical frame aggregated features; In a unified space, for the aggregated features of multiple layers of historical frames, the depth hypothesis of the temporal feature voxels is used as the distance axis, and the volume features are projected into the unified space through voxel pooling operations; based on semantic consistency, the transformed geometric feature voxels and temporal feature voxels are globally aligned and combined with the features related to the current frame to obtain the final aggregated features and generate semantic scene completion results.

2. A 3D semantic scene completion method based on geometric and temporal modeling hierarchical context alignment according to claim 1, characterized in that: The depth features and context features generate voxel feature volumes through a depth network and a context network in a three-dimensional visual network paradigm based on LSS.

3. The 3D semantic scene completion method based on geometric and temporal modeling hierarchical context alignment according to claim 1, characterized in that: The generation of the current frame related features is specifically as follows: Adopt cross attention mechanism to flatten the depth feature F in spatial and depth dimensions d To get the query Q d , flatten the context features to get the key K c Sum V c , the optimized interaction feature F I : F I =φ q (Q d )⊙C d (f k (K c ) T V c ) Among them, φ q represents the softmax function along each row of the input matrix; G c Represents the context feature F c The global context vector of k represents the softmax function along each column of the input matrix; ⊙ represents the element-wise product; C d Indicates that the deep feature F d The confidence map obtained by projection is as follows: Where φ represents the softmax function applied to the depth dimension; WTA represents the winner-takes-all operation; F d is the depth feature, d i Depth proxy value in depth feature; Interaction feature F I Further obtain the current frame related features through the 3D visual network paradigm 4. The 3D semantic scene completion method based on geometric and temporal modeling hierarchical context alignment according to claim 1, characterized in that: The cross-frame feature affinity calculation is specifically as follows: For historical frame related features Features related to the current frame 3D dilated convolutions with different expansion rates are used to construct multiple historical context groups H with different pattern scales. i And multiple sets of context C for the current frame i : Calculate the affinity A of the historical multiple contexts and the previous frame multiple contexts at scale i i , and the affinity matrix A of different group scales i Connect and aggregate along the channel dimension to obtain cross-frame affinity 5. The 3D semantic scene completion method based on geometric and temporal modeling hierarchical context alignment according to claim 4, characterized in that: In each group, the affinity A i During calculation, the historical multiple-group context and the current frame multiple-group context are subtracted from their respective average values.

6. The 3D semantic scene completion method based on geometric and temporal modeling hierarchical context alignment according to claim 1, characterized in that: The posture network adopts the PoseNet model.

7. The method for completing a 3D semantic scene based on hierarchical context alignment of geometric and temporal modeling according to claim 1, characterized in that: The dynamic optimization of historical frame related features based on affinity is specifically as follows: Use 3D deformable convolution for dynamic feature aggregation, and combine the features sampled by deformable convolution with cross-frame affinity Multiply to aggregate effective related features. Within a sampling window, the feature aggregation process is expressed as follows: Among them, V def represents the output feature vector after deformable convolution processing; K w Indicates the total number of sampling points in the sampling window; w k represents the spatial feature weight; V tem Represents the template feature map; p represents the center coordinate of the current feature point; p k represents the standard sampling offset relative to the center point; Δp k Indicates the additional offset corresponding to the kth sampling point; a k Indicates the affinity from cross-frame affinity weight of ; Through a multi-stage deformable block with multiple cascaded 3D deformable convolutions, the output feature vectors of multiple different 3D deformable convolution outputs are concatenated along the channel dimension, and then passed through a 3D convolution layer to obtain multi-layer historical frame aggregation features.

8. The 3D semantic scene completion method based on geometric and temporal modeling hierarchical context alignment according to claim 1, characterized in that: The geometric feature voxels and the temporal feature voxels are globally aligned and combined to obtain the final aggregated features, as follows: The current frame related features and multi-layer historical frame aggregation features As input, The depth hypothesis is used as the distance axis, and the voxel pooling operation is used to splash the volume features into a unified space; the zero convolution aggregation is used and Get the aggregate volume: Among them, Zero_Conv represents the zero convolution in the standard ControlNet, Voxel_Pool represents the standard voxel pooling operation; V com Represents the final aggregated feature.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional semantic scene completion method based on geometric and temporal modeling hierarchical context alignment as described in any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the three-dimensional semantic scene completion method based on geometric and temporal modeling hierarchical context alignment are implemented as described in any one of claims 1 to 8.

Images