A scene perception and reconstruction system based on geometric semantic information fusion
The scene perception and reconstruction system, which integrates geometric and semantic information, solves the problem of insufficient VO and VPS performance in urban scenes, improves panoramic segmentation and positioning accuracy, fills gaps in point cloud reconstruction, and achieves high-fidelity scene reconstruction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YUNNAN UNIV
- Filing Date
- 2024-12-17
- Publication Date
- 2026-07-24
AI Technical Summary
Existing technologies suffer from problems such as insufficient VO and VPS performance, poor depth information estimation, inconsistent segmentation, limited dynamic object filtering performance, and holes in sparse point cloud reconstruction in accurate perception and modeling of urban scenes.
A scene perception and reconstruction system based on geometric semantic information fusion is adopted, including a video panorama segmentation module, a visual odometry module, and a scene rendering module. Through multi-scale feature map fusion, adaptive geometric fusion, kernel update, and rendering techniques, the estimation performance and robustness of the system are improved.
It improves the accuracy and positioning precision of panoramic segmentation, fills in the gaps in point cloud reconstruction, enhances the robustness and scalability of the system, and achieves higher-fidelity scene reconstruction.
Smart Images
Figure CN119810317B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a scene perception and reconstruction system based on geometric semantic information fusion. Background Technology
[0002] With the explosive growth of large-scale deployments of intelligent agents such as autonomous vehicles, quadruped robots, and humanoid robots, the ability to comprehensively understand urban scenes has become increasingly important. This capability is crucial for intelligent agents to perform tasks such as perception, localization, and obstacle avoidance. However, due to the diverse sizes of objects, complex geometric structures, and changing road conditions, accurate perception and modeling of urban scenes are extremely challenging. Furthermore, scenes are often presented through limited viewpoints and sparse data, which also presents modeling challenges. Existing video panorama segmentation technologies include VIP-DeepLab, HybridTracker, and Video K-Net. These methods either utilize geometric information such as optical flow and depth information, or employ end-to-end vision methods, improving the accuracy of panorama segmentation. Existing localization and 3D reconstruction technologies include DROID-SLAM and PVO. These methods leverage the interaction of multiple modalities such as optical flow, depth information, and semantics, combining tasks such as visual odometry (VO), optical flow, depth, and segmentation into a unified framework.
[0003] However, the VO and VPS performance of these methods still need improvement to support practical applications. VIP-DeepLab ignores the correlation information between consecutive frames, and the estimation accuracy of depth information is poor, resulting in holes. HybridTracker: only focuses on the improvement of optical flow for instance target tracking, without considering that the background also needs segmentation consistency. Video K-Net: in complex scenes, segmentation will cause inter-class confusion, and learning-based tracking will have low-level matching errors between adjacent frames. DROID-SLAM: methods that rely solely on optical flow have limited performance in filtering out dynamic objects, and fine reconstruction requires a large number of views. PVO: the geometric information fusion module has weak robustness and cannot be well used in other network structures. In driving scenarios, it can only reconstruct sparse point cloud maps with a large number of holes. Summary of the Invention
[0004] To address the aforementioned technical problems, this invention provides a system for scene perception and reconstruction based on geometric semantic information fusion.
[0005] To achieve the above technical solution, the specific steps are as follows:
[0006] A scene perception and reconstruction system based on geometric semantic information fusion includes: a video panorama segmentation (VPS) module, a visual odometry (VO) module, and a scene rendering (SR) module;
[0007] In the VPS module, select the target frame image I from the image sequence of the monocular video. t Image I of the adjacent frame t-1 Input the VPS module with ResNet50 as the backbone network into each module to extract multi-scale feature maps Z. t and Z t-1 Multi-scale feature maps are obtained through downsampling, including: feature maps after downsampling by half the original size of the input image, feature maps after downsampling by one-quarter the original size of the input image, and feature maps after downsampling by one-eighth the original size of the input image.
[0008] In the VO module, the depth D in the monocular video is obtained using the Dense Bundle Adjustment Layer (DBA) technique. t , position ζ t and optical flow F t And by comparing the image of the next frame adjacent to the downsampled target frame with the depth D t , position ζ t and optical flow F t The distortion is performed to obtain the distorted feature map. Distortion is achieved through optical flow, expressed as follows:
[0009] I t (x)≈I t-1 (x+f(x))
[0010] In the formula, I t (x) represents the image pixel value at position x at time t; t-1 f(x) represents the predicted image pixel value at time t-1; f(x) represents the optical flow field, which represents the displacement vector from time t to time t-1; x is the pixel position vector in the image.
[0011] Multiscale feature map z t and the distorted feature map Adaptive geometry fusion is performed; the steps are as follows:
[0012] Step 1: Establish a three-layer pyramid network structure. The scale of the first layer is the scale after downsampling by half, the scale of the second layer is the scale after downsampling by one-quarter, and the scale of the third layer is the scale after downsampling by one-eighth.
[0013] Step 2: Convert the multi-scale feature map z t and the distorted feature map Feature concatenation is performed according to the channel dimension to obtain a concatenated feature map z with a channel dimension of 2C. i ;
[0014] Step 3: Combine the spliced feature map z i Input the pyramid network structure at the corresponding scale;
[0015] Step 4: At each scale, stitch together the feature maps z. i The input to the i-th channel attention mechanism first goes through a global average pooling (equivalent to taking the average of all pixels in an image and then keeping only this value and a pixel) to transform the feature map into 1×1×2C, and then performs a one-dimensional convolution on the feature map to the weight Ai of each channel;
[0016] The kernel size is obtained using an adaptive formula: Where y and b represent the first and second learning parameters, b = y = 1, |*| odd This represents the odd number closest to *.
[0017] Each channel convolution is element-wise multiplied with the concatenated feature map to obtain 2C weighted feature maps;
[0018] The channel attention mechanism is ECA-Net;
[0019] Step 5: At the end of each pyramid network structure, 2C weighted feature maps are input into a 3D convolutional layer with a kernel size of n×n×2C to obtain a feature map with channels reduced to C after fusion.
[0020] The size of the convolution kernel, n, varies with the resolution of the input video image. For an image with a resolution of 375*1242, n is 3. n can be adjusted by multiples of the number of pixels.
[0021] After processing the feature maps at each scale, a fused feature map for that branch is generated. The expression is as follows:
[0022]
[0023] In the formula, f avg This indicates a global average pooling operation, and c represents concatenation based on channel characteristics. σ represents element-wise multiplication; σ represents the sigmoid function, which maps real numbers to the range (0, 1); W g and b g These are the learning parameters of the first neural network and the learning parameters of the second neural network, respectively.
[0024] Step 6: Decode and output the panoramic segmentation result; the specific process is as follows:
[0025] First, the first frame of the monocular video is input into the first feature map group embedding operation for initial training, i.e. there is no geometric information fusion stage.
[0026] Next, the feature maps will be fused. Input the first feature map group embedding operation;
[0027] Feature map group embedding is a preparatory operation for kernel update head operations, which enables the kernel to have group awareness; that is, given a feature map F and a kernel prediction mask M... p-1 The group characteristics F of each core k The calculation is performed pixel-by-pixel multiplication, and the expression is as follows:
[0028]
[0029] Where u and v represent the height and width coordinates of a pixel; the feature map F has a size of H×W×C, and the feature map F is...
[0030] Next, Perform the first core header update operation f s1 Second core update header operation f s2 Third core update header operation f s3 ;
[0031] The kernel update head operation includes three steps: feature map group embedding, adaptive kernel update, and kernel interaction.
[0032] The formula for updating the kernel header is expressed as follows:
[0033] K p M p =f sp (K p-1 M p-1 ,F)
[0034] In the formula, K p Representative kernels (including semantic kernels and instance kernels 0 and M) p The middle represents the prediction of the mask, f sp The kernel update head (which includes three steps: feature map group embedding, adaptive kernel update, and kernel interaction) represents the kernel update head, and F represents the feature map (in this embodiment, it refers to the feature map input after adaptive geometric fusion). );
[0035] The feature map group embedding expression is:
[0036]
[0037] Where u and v represent the height and width coordinates of a pixel; the feature map F has a size of H×W×C, and the feature map F is...
[0038] Adaptive kernel update is due to the kernel prediction mask M p-1 There may be errors in the extracted group features F. k Noise may be mixed in; therefore, element-wise multiplication is used to fuse the kernel features K. p-1 Group characteristics:
[0039]
[0040] In the formula, This represents element-wise multiplication; where, to control for the influence of noise, the model learns the first gating parameter G. F Second gating parameter G k Control F respectively k and K p-1 Its importance in kernel updates; the expression is as follows:
[0041] G F =σ(φ3(F) G )),G K =σ(φ4(F) G ))
[0042] The final kernel update formula is:
[0043] K * =G F ·φ5(F k )+G K ·φ6(K i-1 )
[0044] In the formula, φ N This represents using the Nth fully connected layer (FC) + layer normalization (LN), where σ is sigmoid; N = {1, ..., 6};
[0045] Furthermore, G F and G k Similar to the self-attention mechanism in Transformers, its output is computed by a weighted sum of values; in Transformers, the weights assigned to each value are typically computed using a dot product compatibility function of Q and K; similarly, adaptive kernel updates are essentially applied to the kernel features K. p-1 Features F G A weighted summation was performed; its weight G F and G k Calculated by element-wise multiplication, it can be regarded as another type of compatibility function;
[0046] At input time t, p = 3, which means that the kernel header is updated 3 times, while at time t-1, p = 1.
[0047] Kernel interaction simply employs a multi-head attention mechanism, then connects it to a feedforward neural network (FFN) to obtain a new kernel K. p Then, a new mask prediction is generated using the formula:
[0048] M p =g p (K p )F
[0049] In the formula, g p It is a network consisting of fully connected layers (FC), layer normalization (LN), and ReLU activation functions, followed by a fully connected layer; in addition, K p It is also used for classification score prediction in instance and panoptic segmentation;
[0050] Finally, after f s3 After the kernel interaction operation of the stage, the panoramic segmentation result and multiple instance kernels t are obtained;
[0051] Furthermore, Z t-1 After performing the first feature map group embedding operation, f is performed. s1 The first core update header operation of the phase, the first core update header operation f s1 The results are processed through two operations: generating multiple instance cores t-1 and updating the header of the first core f. s1 The result and f s3 The feature map group embedding operation of the stage is followed by kernel fusion operation;
[0052] To maintain segmentation consistency, this invention uses input kernels from two frames for fusion. First, a kernel update operation is performed, using the kernel features of the current frame to update the kernel of the previous frame. Then, a self-attention layer and a feedforward layer are used to fuse the kernel of the previous frame into the current frame.
[0053] The number of instance cores t-1 and instance cores t are the same, and core linking operations are performed.
[0054] Kernel connection operations are further operations beyond supervised learning, namely, connecting K kernels during training and inference. t and K t-1 This forces the kernel to interact in the time dimension. This invention employs a Multi-Head Self-Attention (MHSA) layer and a feedforward network (FFN) to learn the correspondence between each query, thereby obtaining the updated query and allowing for complete association between queries. The process is as follows:
[0055] Kl =FFN(MHSA9K) t ,K t-1 ,K t-1 )+K t )
[0056] In the formula, Q, K, and V are K t K t-1 and K t-1 In this way, the kernel of the reference frame is passed to the keyframe via the affinity matrix between kernels; then, the concatenated kernel K... l It is sent to the tracking head; during training, this operation is optimized only on the matched instance kernels; K t Represents instance kernel t, K t-1 Indicates instance core t-1; K l This indicates the instance core output after the connection;
[0057] Perform a kernel embedding operation between the kernel linking operation result and the instance kernel t-1;
[0058] Kernel embedding is simply implemented using a fully connected layer, extracting embedding features for each kernel; after kernel embedding, one-dimensional features are obtained, which can be used for similarity matching.
[0059] In the VO module, there are two steps: localization and reconstruction. The panoramic segmentation result plays a role in filtering out dynamic objects in the VO module, providing more accurate localization. However, reconstruction also requires monocular video (i.e., video with RGB and panoramic information) as input. Different point cloud maps are reconstructed based on the input video stream. Finally, the VO module outputs RGB point cloud, panoramic point cloud, and pose information.
[0060] In the SR module, monocular video (i.e., video with RGB and panoramic information), RGB point cloud, panoramic point cloud, and pose information are used as outputs to obtain the final rendering result; as follows:
[0061] The SR module employs a point-based rendering method from NERF technology. It trains sparse point clouds (RGB point clouds and panoramic point clouds) against real images from the same viewpoint using an ω-net neural network to construct a loss function. This loss function is then converted into an explicit neural field through neural network rendering. Learning from the sparse input view, it synthesizes a high-fidelity RGB image and its corresponding panoramic annotated view. Here, the sparse point cloud P = {p1, p2, ..., p...} A The point cloud is obtained from the input driving scene image and known camera parameters through the VO module, and the number of points is automatically generated by the computer.
[0062] During the rasterization stage, neural descriptors N_d = {nd_1, nd_2, ..., nd_A} are calculated from the point cloud P and updated by propagating gradients through the input data via a neural network. Each 3D point P_A = {x_A, y_A, z_A} is associated with a neural descriptor vector nd_A, which encodes local geometric and photometric features.
[0063] To avoid updating the descriptors of occluded points, a Z-buffer was constructed to filter the occluded point cloud. Monte Carlo sampling and random image patch sampling were then used to train the point cloud and the image, respectively. Monte Carlo sampling focused on the synthesis quality of the samples, retraining on samples with poor synthesis quality, while random image patch sampling divided the image into consistent image patches for training, keeping the proportion of image patch pixels to less than 15% of the total image pixels, thereby reducing the consumption of computational resources.
[0064] Beneficial effects of the present invention
[0065] In this invention, the video panorama segmentation module outputs panorama segmentation results by fusing geometric information obtained from the visual odometry module. The visual odometry module estimates camera pose, depth, and optical flow, and improves the system's estimation performance by obtaining semantic information from the video panorama segmentation module. These two modules promote each other through recursive interaction.
[0066] This invention improves the confidence of motion estimation by combining panoramic segmentation information and optical flow, thereby enhancing the accuracy of camera pose estimation.
[0067] Compared to traditional PVO, this invention incorporates a channel attention mechanism module that focuses on information between channels, assigns weights to assist in fusion, and improves robustness.
[0068] This invention takes into account multi-stage decoding and provides decoders at different stages to solve for multi-fusion adaptive training output;
[0069] This invention uses the SR module to address the issues of voids and sparseness in the point cloud generated by PVO in driving scenarios. Attached Figure Description
[0070] Figure 1 This is a flowchart of the present invention;
[0071] Figure 2 This is a schematic diagram of the network structure of the present invention;
[0072] Figure 3 This is a diagram of the panoramic segmentation network structure of the present invention;
[0073] Figure 4 This is a structural diagram of the adaptive geometry fusion of the present invention;
[0074] Figure 5 These are the experimental results of the VPS module on the VKITTI2 dataset of this invention;
[0075] Figure 6 These are the experimental results of the SR module of this invention on the VKITTI2 dataset;
[0076] Figure 7 The results are experimental results of this invention on the VKITTI2 dataset. Detailed Implementation
[0077] The present invention will be further described in detail below with reference to specific embodiments.
[0078] like Figure 1 and Figure 2 As shown, a scene perception and reconstruction system based on geometric semantic information fusion includes: a video panorama segmentation (VPS) module, a visual odometry (VO) module, and a scene rendering (SR) module.
[0079] In the VPS module, select the target frame image I from the image sequence of the monocular video. t Image I of the adjacent frame t-1 Input the VPS module with ResNet50 as the backbone network into each module to extract multi-scale feature maps Z. t and Z t-1 Multi-scale feature maps are obtained through downsampling, including: feature maps after downsampling by half the original size of the input image, feature maps after downsampling by one-quarter the original size of the input image, and feature maps after downsampling by one-eighth the original size of the input image.
[0080] In the VO module, the depth D in the monocular video is obtained using the Dense Bundle Adjustment Layer (DBA) technique. t , position ζ t and optical flow F t And by comparing the image of the next frame adjacent to the downsampled target frame with the depth D t , position ζ t and optical flow F t The distortion is performed to obtain the distorted feature map. Distortion is achieved through optical flow, expressed as follows:
[0081] I t (x)≈I t-1 (x+f(x))
[0082] In the formula, I t (x) represents the image pixel value at position x at time t;t-1 f(x) represents the predicted image pixel value at time t-1; f(x) represents the optical flow field, which represents the displacement vector from time t to time t-1; x is the pixel position vector in the image.
[0083] Furthermore, such as Figure 3 and Figure 4 As shown, the multi-scale feature map z t and the distorted feature map Adaptive geometry fusion is performed; the steps are as follows:
[0084] Step 1: Establish a three-layer pyramid network structure. The scale of the first layer is the scale after downsampling by half, the scale of the second layer is the scale after downsampling by one-quarter, and the scale of the third layer is the scale after downsampling by one-eighth.
[0085] Step 2: Convert the multi-scale feature map z t and the distorted feature map Feature concatenation is performed according to the channel dimension to obtain a concatenated feature map z with a channel dimension of 2C. i In this embodiment, i = 4;
[0086] Step 3: Combine the spliced feature map z i Input the pyramid network structure at the corresponding scale;
[0087] Step 4: At each scale, stitch together the feature maps z. i The input to the i-th channel attention mechanism first goes through a global average pooling (equivalent to taking the average of all pixels in an image and then keeping only this value and a pixel) to transform the feature map into 1×1×2C, and then performs a one-dimensional convolution on the feature map to the weight Ai of each channel;
[0088] The kernel size is obtained using an adaptive formula: Where y and b represent the first and second learning parameters, b = y = 1, |*| odd This represents the odd number closest to *.
[0089] Each channel convolution is element-wise multiplied with the concatenated feature map to obtain 2C weighted feature maps;
[0090] The channel attention mechanism is ECA-Net;
[0091] In this embodiment, the weights are automatically generated by the network and the weight Ai of each channel is different;
[0092] Step 5: At the end of each pyramid network structure, 2C weighted feature maps are input into a 3D convolutional layer with a kernel size of n×n×2C to obtain a feature map with channels reduced to C after fusion.
[0093] The size n of the convolution kernel varies with the resolution of the input video image. For an image with a resolution of 375*1242, n is 3. n can be adjusted by multiples of the number of pixels. In this embodiment, n is 3.
[0094] After processing the feature maps at each scale, a fused feature map for that branch is generated. The expression is as follows:
[0095]
[0096] In the formula, f avg This indicates a global average pooling operation, and c represents concatenation based on channel characteristics. σ represents element-wise multiplication; σ represents the sigmoid function, which maps real numbers to the range (0, 1); W g and b g These are the learning parameters of the first neural network and the learning parameters of the second neural network, respectively.
[0097] In this embodiment, W g and b g The values are learned through a neural network, and their specific values depend on the training process.
[0098] Step 6: Decode and output the panoramic segmentation result; the specific process is as follows:
[0099] First, the first frame of the monocular video is input into the first feature map group embedding operation for initial training, i.e. there is no geometric information fusion stage.
[0100] Next, the feature maps will be fused. Input the first feature map group embedding operation;
[0101] Feature map group embedding is a preparatory operation for kernel update head operations, which enables the kernel to have group awareness; that is, given a feature map F and a kernel prediction mask M... p-1 The group characteristics F of each core k The calculation is performed pixel-by-pixel multiplication, and the expression is as follows:
[0102]
[0103] Where u and v represent the height and width coordinates of a pixel; the feature map F has a size of H×W×C, and the feature map F is... In this embodiment, p∈(1,4);
[0104] Next, Perform the first core header update operation f s1 Second core update header operation f s2 Third core update header operation f s3 ;
[0105] The kernel update head operation includes three steps: feature map group embedding, adaptive kernel update, and kernel interaction.
[0106] The formula for updating the kernel header is expressed as follows:
[0107] K p M p =f sp (K p-1 M p-1 ,F)
[0108] In the formula, K p Representation kernel (including semantic kernel and instance kernel), M p The middle represents the prediction of the mask, f sp The kernel update head (which includes three steps: feature map group embedding, adaptive kernel update, and kernel interaction) represents the kernel update head, and F represents the feature map (in this embodiment, it refers to the feature map input after adaptive geometric fusion). );
[0109] The feature map group embedding expression is:
[0110]
[0111] Where u and v represent the height and width coordinates of a pixel; the feature map F has a size of H×W×C, and the feature value F is z i t * In this embodiment, p∈(1,4);
[0112] Adaptive kernel update is due to the kernel prediction mask M p-1 There may be errors in the extracted group features F. k Noise may be mixed in; therefore, element-wise multiplication is used to fuse the kernel features K. p-1 Group characteristics:
[0113]
[0114] In the formula, This represents element-wise multiplication; where, to control for the influence of noise, the model learns the first gating parameter G. F Second gating parameter G k Control F respectively k and K p-1 Its importance in kernel updates; the expression is as follows:
[0115] G F =σ(φ3(F) G )),G K =σ(φ4(F) G ))
[0116] The final kernel update formula is:
[0117] K * =G F ·φ5(F k )+G K ·φ6(K i-1 )
[0118] In the formula, φ N This represents using the Nth fully connected layer (FC) + layer normalization (LN), where σ is sigmoid; N = {1, ..., 6};
[0119] Furthermore, G F and G k Similar to the self-attention mechanism in Transformers, its output is computed by a weighted sum of values; in Transformers, the weights assigned to each value are typically computed using a dot product compatibility function of Q and K; similarly, adaptive kernel updates are essentially applied to the kernel features K. p-1 Features F G A weighted summation was performed; its weight G F and G k Calculated by element-wise multiplication, it can be regarded as another type of compatibility function;
[0120] At input time t, p = 3, which means that the kernel header is updated 3 times, while at time t-1, p = 1.
[0121] Kernel interaction simply employs a multi-head attention mechanism, then connects it to a feedforward neural network (FFN) to obtain a new kernel K. p Then, a new mask prediction is generated using the formula:
[0122] M p =g p (K p )F
[0123] In the formula, g p It is a network consisting of fully connected layers (FC), layer normalization (LN), and ReLU activation functions, followed by a fully connected layer; in addition, K p It is also used for classification score prediction in instance and panoptic segmentation;
[0124] Finally, after f s3 After the kernel interaction operation of the stage, the panoramic segmentation result and multiple instance kernels t are obtained;
[0125] Furthermore, Z t-1 After performing the first feature map group embedding operation, f is performed. s1 The first core update header operation of the phase, the first core update header operation f s1The results are processed through two operations: generating multiple instance cores t-1 and updating the header of the first core f. s1 The result and f s3 The feature map group embedding operation of the stage is followed by kernel fusion operation;
[0126] To maintain segmentation consistency, this invention uses input kernels from two frames for fusion. First, a kernel update operation is performed, using the kernel features of the current frame to update the kernel of the previous frame. Then, a self-attention layer and a feedforward layer are used to fuse the kernel of the previous frame into the current frame.
[0127] The number of instance cores t-1 and instance cores t are the same, and core linking operations are performed.
[0128] Kernel connection operations are further operations beyond supervised learning, namely, connecting K kernels during training and inference. t and K t-1 This forces the kernel to interact in the time dimension. This invention employs a Multi-Head Self-Attention (MHSA) layer and a feedforward network (FFN) to learn the correspondence between each query, thereby obtaining the updated query and allowing for complete association between queries. The process is as follows:
[0129] K l =FFN(MHSA(K) t ,K t-1 ,K t-1 )+K t )
[0130] In the formula, Q, K, and V are K t K t-1 and K t-1 In this way, the kernel of the reference frame is passed to the keyframe via the affinity matrix between kernels; then, the concatenated kernel K... l It is sent to the tracking head; during training, this operation is optimized only on the matched instance kernels; K t Represents instance kernel t, K t-1 Indicates instance core t-1; K l This indicates the instance core output after the connection;
[0131] Perform a kernel embedding operation between the kernel linking operation result and the instance kernel t-1;
[0132] Kernel embedding is simply implemented using a fully connected layer, extracting embedding features for each kernel; after kernel embedding, one-dimensional features are obtained, which can be used for similarity matching.
[0133] In the VO module, there are two steps: localization and reconstruction. The panoramic segmentation result plays a role in filtering out dynamic objects in the VO module, providing more accurate localization. However, reconstruction also requires monocular video (i.e., video with RGB and panoramic information) as input. Different point cloud maps are reconstructed based on the input video stream. Finally, the VO module outputs RGB point cloud, panoramic point cloud, and pose information.
[0134] In the SR module, monocular video (i.e., video with RGB and panoramic information), RGB point cloud, panoramic point cloud, and pose information are used as outputs to obtain the final rendering result; as follows:
[0135] The SR module employs a point-based rendering method from NERF technology. It trains sparse point clouds (RGB point clouds and panoramic point clouds) against real images from the same viewpoint using an ω-net neural network to construct a loss function. This loss function is then converted into an explicit neural field through neural network rendering. Learning from the sparse input view, it synthesizes a high-fidelity RGB image and its corresponding panoramic annotated view. Here, the sparse point cloud P = {p1, p2, ..., p...} A The point cloud is obtained from the input driving scene image and known camera parameters through the VO module, and the number of points is automatically generated by the computer.
[0136] During the rasterization stage, neural descriptors N_d = {nd_1, nd_2, ..., nd_A} are calculated from the point cloud P and updated by propagating gradients through the input data via a neural network. Each 3D point P_A = {x_A, y_A, z_A} is associated with a neural descriptor vector nd_A, which encodes local geometric and photometric features.
[0137] To avoid updating the descriptors of occluded points, a Z-buffer was constructed to filter the occluded point cloud. Monte Carlo sampling and random image patch sampling were then used to train the point cloud and the image, respectively. Monte Carlo sampling focused on the synthesis quality of the samples, retraining on samples with poor synthesis quality, while random image patch sampling divided the image into consistent image patches for training, keeping the proportion of image patch pixels to less than 15% of the total image pixels, thereby reducing the consumption of computational resources.
[0138] This invention evaluates segmentation accuracy by calculating Video Panorama Segmentation Quality (VPQ). As shown in Table 1, it is clearly observable that this method predicts the most accurate segmentation mask in dynamic scenes compared to existing technologies such as PVO and Video K-Net. This is because this method successfully fuses geometric information to maintain consistency between frame information.
[0139] Table 1 Performance of Video Panorama Segmentation
[0140]
[0141] In the table, (↑) indicates that the larger the index number, the better the performance. It can be found that existing technologies PVO, PVO+AG Fusion (AG Fusion represents the adaptive geometric information fusion module), Video K-Net, and Video K-Net+PVOFusion do not perform as well as the present invention in calculating the video panorama segmentation quality (VPQ) on the two datasets VKITTI2 and VIPER.
[0142] As shown in Table 2, the system of this invention achieves the best localization results compared to the other two methods on both synthetic and real datasets. This is also because good segmentation results help the VO filter out dynamic objects, thereby estimating more accurate camera pose.
[0143] Table 2 Positioning Performance
[0144]
[0145] In the table, (↑) indicates that the larger the indicator number, the better the performance. It can be observed that, when comparing the results of existing technologies DROID-SLAM, PVO, PVO+AG Fusion, and this invention on the VKITTI2 and KITTI datasets using root mean square error, the error of this invention is smaller than that of the existing technologies.
[0146] The experimental results of this method were quantitatively analyzed by calculating the peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and perceptual loss (LPIPS). The calculation results are shown in Table 3.
[0147] Table 3 Performance of Rendering Reconstruction Quality
[0148]
[0149] In the table, KITTI and VKITTI are dataset names, and the numbers 09, 10, and 20 represent video sequence numbers. Psnr is the peak signal-to-weight ratio, ssim is the structural similarity, and lpips is the learned perceptual patch similarity. It can be seen that when more accurate localization and point cloud results are used as input, the rendering quality can be improved, demonstrating the improved coupling between multiple modules in this invention.
[0150] The method of the present invention has excellent robustness and scalability. Even when using various techniques in the attention mechanism of the fusion module, segmentation performance can still be improved, as shown in Table 4.
[0151] Table 4 Performance of Video Panorama Segmentation
[0152]
[0153] In the table, (↑) indicates that the larger the index number, the better the performance. Video K-Net uses the adaptive geometric information fusion module of this invention, and replaces the channel attention technology in the adaptive geometric information fusion module with three different ones for comparison, including SE, CBMA and ECA. It can be seen that the video panorama segmentation quality of this invention using the channel attention mechanism ECA is the best on the VKITTI2 dataset.
[0154] In this embodiment, the VKITTI2 dataset is used for experiments, such as... Figure 5 As shown in the experimental results, it is clear that the present invention successfully and completely segmented the background and tracked the instance target.
[0155] like Figure 6 The experimental results shown are from the VKITTI2 dataset. It is clear from the results that this invention successfully reconstructs the point cloud scene and synthesizes a dense viewpoint through the rendering module.
[0156] like Figure 7 The experimental results shown are from the VKITTI2 dataset. It is clear from the results that this invention successfully demonstrates the ability to synthesize novel perspectives by rotating the image 5° left and right.
Claims
1. A scene perception and reconstruction system based on geometric semantic information fusion, characterized in that, The system includes: a video panorama segmentation module, a visual odometry module, and a scene rendering module; The target frame image and the image of the adjacent frame in the monocular video are input into the video panorama segmentation module to obtain a multi-scale feature map. The multi-scale feature map includes: the scale after downsampling by half, the scale after downsampling by one-quarter, and the scale after downsampling by one-eighth. The target frame is any frame in a monocular video or a selected frame; Depth, pose, and optical flow of monocular video are obtained through a visual odometry module; After warping the multi-scale feature map of the adjacent frame of the target frame with depth, pose and optical flow, adaptive geometric fusion is performed with the multi-scale feature map of the target frame image. The features after adaptive geometric fusion are decoded to obtain the panoramic segmentation result. The adaptive geometry fusion steps are as follows: A three-layer pyramid network structure is established. The scale of the first layer is the scale after downsampling by half, the scale of the second layer is the scale after downsampling by one-quarter, and the scale of the third layer is the scale after downsampling by one-eighth. Multi-scale feature maps of the target frame image and distorted multi-scale feature maps Features are concatenated according to the channel dimension to obtain a concatenated feature map with a channel dimension of 2C; Input the stitched feature map into a pyramid network structure of the corresponding scale; At each scale, stitch feature maps Enter the first The channel attention mechanism first transforms the feature map into a 1×1×2C form using global average pooling, and then performs a one-dimensional convolution on the feature map to obtain the weights for each channel. Each channel weight is multiplied element-wise with the concatenated feature map to obtain 2C weighted feature maps; At the end of each pyramid network structure, 2C weighted feature maps are input into a kernel with a size of... The 3D convolutional layer yields a fused feature map with channels reduced to C; After processing the feature maps at each scale, a fused feature map for that branch is generated. ; The decoded output panorama segmentation results include: Fuse feature maps , , Perform the first feature map group embedding operation in the first branch respectively. f s1 The first core update header operation of the phase, f s2 The second core update header operation in the phase, f s3 The second feature map group embedding operation of the stage, and f s3 The output of the second feature map group embedding operation in the stage is input to f s3 During the third core header update operation of this phase; The multi-scale features of the (t-1)th frame of the monocular video are input into the second branch, and the following operations are performed sequentially: first feature map group embedding and... f s1 The first core update header operation of the phase is based on f s1 The output of the first core update header operation in this phase generates a multi-core instance core pt-1; and then... f s1 The output of the first kernel update head in the first stage is compared with the first branch. f s3 The output of the feature map group embedding operation in the stage is subjected to kernel fusion operation, and the fusion result is input into the first branch. f s3 The third core update header operation in this phase; The kernel update head operation includes three steps: feature map group embedding, adaptive kernel update, and kernel interaction. final f s3 The output of the third core update head in the stage goes through f s3 After the kernel interaction operation of the stage, the panoramic segmentation result and multiple instance kernels are obtained. pt ; instance core pt -1 and instance core pt The quantities are the same, and a core linking operation is performed; Compare the kernel join operation result with the instance kernel. pt -1 is used for kernel embedding operation, which is implemented through a fully connected layer to extract embedding features for each kernel; after kernel embedding, one-dimensional features are obtained for similarity matching. The scene rendering module extracts RGB point clouds and panoramic view clouds from the visual odometry module. The scene rendering module then synthesizes the panoramic view clouds and RGB point clouds into a high-quality scene reconstruction, resulting in the final result.
2. The scene perception and reconstruction system based on geometric semantic information fusion according to claim 1, characterized in that, The distortion is achieved through optical flow, expressed as follows: In the formula, Indicates time Time and location Image pixel values at that location; Indicates time Image pixel values predicted at any given time; Represents the optical flow field, representing the time... Time The displacement vector; It is a vector of pixel positions in the image.