Scene perception and reconstruction system based on geometric semantic information fusion

Abstract

The invention discloses a scene perception and reconstruction system based on geometric semantic information fusion. Belongs to the technical field of computer vision. The system comprises a video panorama segmentation (VPS) module, a visual odometer (VO) module and a scene rendering (SR) module, wherein the VPS module obtains a panorama segmentation result by fusing the VO module and adaptively fusing geometric information; the VO module estimates the pose, depth and optical flow of the camera, and obtains semantic information from the VPS module to improve the estimation performance; the two modules promote each other through recursion interaction; and the SR module aims to synthesize a vivid scene through the sparse point cloud map. The core motivation of the technology is to enhance the association between VO and VPS tasks, thereby realizing finer 3D scene understanding. The method is a unified sensing, positioning and reconstruction method, and compared with a direct convolution mode, the self-adaptive geometric information fusion can support more real application scenes and has high expansibility.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a scene perception and reconstruction system based on geometric semantic information fusion. Background Art

[0002] As autonomous vehicles, quadruped robots, or humanoid robots usher in a period of large-scale deployment of intelligent agents, the ability to understand the overall urban scene becomes increasingly important. This ability is essential for intelligent agents to perform tasks such as perception, positioning, and obstacle avoidance. However, due to the diverse sizes of objects, complex geometric structures, and changing road conditions, accurate perception and modeling of urban scenes are extremely challenging. In addition, scenes are usually presented through limited perspectives and sparse data, which also brings modeling challenges. Existing video panoramic segmentation technologies include: VIP-DeepLab, HybridTracker, and Video K-Net. These methods either use geometric information such as optical flow and depth information, or adopt end-to-end visual methods to improve the accuracy of panoramic segmentation. Existing positioning and 3D reconstruction technologies include: DROID-SLAM and PVO technologies. These methods use the interaction of multiple modalities such as optical flow, depth information, and semantics to combine tasks such as visual odometry (VO), optical flow, depth, and segmentation into a unified framework.

[0003] However, the VO and VPS performance of these methods still need to be improved to support practical applications. VIP-DeepLab ignores the correlation information between the previous and next frames, and the estimation accuracy of the depth information is poor, resulting in holes. HybridTracker: It only focuses on the improvement of optical flow for instance target tracking, without considering that the background also needs segmentation consistency. Video K-Net: In complex scenes, segmentation will be confused between classes, and learning-based tracking will have low-level matching errors between adjacent frames. DROID-SLAM: Methods that rely solely on optical flow have limited filtering performance for dynamic objects, and fine reconstruction requires a large number of viewpoints. PVO: The geometric information fusion module is weakly robust and cannot be used well in other network structures. In driving scenarios, it can only reconstruct sparse point cloud maps with a large number of holes. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a system of scene perception and reconstruction technology based on geometric semantic information fusion.

[0005] To implement the above technical solution, the details are as follows:

[0006] A scene perception and reconstruction system based on geometric semantic information fusion, including: a video panoptic segmentation (VPS) module, a visual odometry (VO) module and a scene rendering (SR) module;

[0007] In the VPS module, select the target frame image I in the image sequence of the monocular video t and the image I of the adjacent frame t-1 , respectively input the VPS module with ResNet50 as the backbone network to extract the multi-scale feature map Z t and Z t-1 ; The multi-scale feature map is obtained by downsampling, including: a feature map after downsampling by half of the original size of the input image, a feature map after downsampling by one quarter of the original size of the input image, and a feature map after downsampling by one eighth of the original size of the input image;

[0008] In the VO module, the depth D in the monocular video is obtained through the dense bundle adjustment layer (DBA) technology in the VO module. t 、Positionζ t and optical flow F t , and by comparing the image of the adjacent frame of the downsampled target frame with the depth D t 、Positionζ t and optical flow F t Distort and obtain the distorted feature map The distortion is achieved through optical flow, and the expression is as follows:

[0009] I t (x)≈I t-1 (x+f(x))

[0010] In the formula, I t (x) represents the image pixel value at position x at time t; t-1 (x) represents the predicted image pixel value at time t-1; f(x) represents the optical flow field, which represents the displacement vector from time t to time t-1; x is the pixel position vector in the image;

[0011] The multi-scale feature map z t And the distorted feature map Perform adaptive geometric fusion; the steps are as follows:

[0012] Step 1: Establish a three-layer pyramid network structure. The scale of the first layer is the scale after downsampling by half, the scale of the second layer is the scale after downsampling by one quarter, and the scale of the third layer is the scale after downsampling by one eighth.

[0013] Step 2: Multi-scale feature map z t And the distorted feature map Perform feature concatenation according to the channel dimension to obtain a concatenated feature map z with a channel dimension of 2C i ;

[0014] Step 3: Splice the feature map z i Input the pyramid network structure of the corresponding scale;

[0015] Step 4: At each scale, concatenate feature maps z i Input the i-th channel attention mechanism, first through a global average pooling (equivalent to taking the average of all pixels in an image and then retaining only this value and one pixel) to convert the feature map into 1×1×2C, and then perform a one-dimensional convolution on the feature map to the weight Ai of each channel;

[0016] The convolution kernel size is obtained through the adaptive formula: Where y, b represent the first learning parameter and the second learning parameter, b=y=1,|*| odd represents the odd number closest to *;

[0017] Multiply each channel convolution with the concatenated feature map element by element to obtain 2C feature maps with weights;

[0018] The channel attention mechanism is ECA-Net;

[0019] Step 5: At the end of each layer of the pyramid network structure, 2C weighted feature maps are input into a 3D convolutional layer with a kernel size of n×n×2C to obtain a feature map after channel reduction to C fusion;

[0020] The size n of the convolution kernel varies with the resolution of the input video image. For an image with a resolution of 375*1242, n is 3. n can be adjusted by multiples of the number of pixels.

[0021] After processing the feature maps of each scale, the fused feature map of the branch is generated The expression is as follows:

[0022]

[0023] In the formula, f avg represents the global average pooling operation, c represents the channel-by-channel feature concatenation, represents element-by-element multiplication; σ represents the sigmoid function, which maps real numbers to the range (0, 1); W g and b g are the first neural network learning parameters and the second neural network learning parameters respectively;

[0024] Step 6: Decode and output the panoramic segmentation result; the specific process is as follows:

[0025] First, the first frame of the monocular video is input into the first feature map group embedding operation for initialization training, that is, there is no geometric information fusion stage;

[0026] Afterwards, the fusion feature map Input the first feature map group embedding operation;

[0027] Feature map group embedding is an operation in the preparation stage of the kernel update head operation, which can make the kernel group-aware; that is, given a feature map F and a kernel prediction mask M p-1 , the group feature F of each kernel k Calculated by pixel-by-pixel multiplication, the expression is as follows:

[0028]

[0029] Among them, u, v represent the coordinates of the height and width of the pixel; the size of the feature map F is H×W×C, and the feature map F is

[0030] Next, Perform the first core update header operation f s1 , the second core updates the header operation f s2 , the third core updates the header operation f s3 ;

[0031] The kernel update head operation includes three steps: feature map group embedding, adaptive kernel update and kernel interaction;

[0032] The formula for the core update header is expressed as:

[0033] K p ,M p =f sp (K p-1 ,M p-1 ,F)

[0034] In the formula, K p Representative core (including semantic core and instance core 0, M p represents the prediction of the mask, f sp represents the kernel update header (including three steps: feature map group embedding, adaptive kernel update and kernel interaction), F represents the feature map (in this embodiment, it refers to the feature map input after adaptive geometric fusion). );

[0035] The feature map group embedding expression is:

[0036]

[0037] Among them, u, v represent the coordinates of the height and width of the pixel; the size of the feature map F is H×W×C, and the feature map F is

[0038] Adaptive kernel update is due to the kernel prediction mask M p-1 There may be errors, the extracted group features F k Noise may be mixed in; therefore, the kernel feature K is fused by element-by-element multiplication. p-1 With group features:

[0039]

[0040] In the formula, Represents element-by-element multiplication; in order to control the impact of noise, the model learns the first gating parameter G F and the second gating parameter G k , respectively control F k and K p-1 Importance in kernel updates; the expression is as follows:

[0041] G F =σ(φ 3 (F G )),G K =σ(φ 4 (F G ))

[0042] The final kernel update formula is:

[0043] K * =G F ·φ 5 (F k )+G K ·φ 6 (K i-1 )

[0044] In the formula, φ N represents the use of the Nth fully connected layer (FC) + layer normalization (LN), σ is sigmoid; N = {1,…,6};

[0045] Furthermore, G. F and G k Similar to the self-attention mechanism in Transformer, its output is calculated by weighted summation of values; in Transformer, the weight assigned to each value is usually calculated by the dot product compatibility function of Q and K; similarly, the adaptive kernel update essentially performs a kernel feature update on K. p-1 and group feature F G A weighted sum is performed; its weight G F and G k It is calculated by element-by-element multiplication and can be regarded as another compatibility function;

[0046] When inputting a frame at time t, p = 3, which means that the core updates the header three times, while the frame at time t-1 p = 1;

[0047] Kernel interaction simply uses a multi-head attention mechanism and then connects it to a feedforward neural network (FFN) to get a new kernel K p . A new mask prediction is then generated using the formula:

[0048] M p =g p (K p )F

[0049] In the formula, g p It is a network consisting of a fully connected layer (FC), layer normalization (LN) and ReLU activation function, followed by a fully connected layer; in addition, K p It is also used for classification score prediction in instance and panoptic segmentation;

[0050] Finally, after f s3 After the kernel interaction operation in the stage, the panoramic segmentation result and multiple instance kernels t are obtained;

[0051] Furthermore, Z t-1 After the first feature map group embedding operation, f s1 The first core update header operation of the stage, the first core update header operation f s1 The results of the two operations are respectively performed, including: generating multiple instance cores t-1 and updating the first core head operation f s1 The result is similar to f s3 The kernel fusion operation is performed on the feature map group embedding operation of the stage;

[0052] Kernel fusion operation To maintain segmentation consistency, the present invention uses input kernels from two frames for fusion; first, a kernel update operation is performed to update the kernel of the previous frame using the kernel features of the current frame; then, a self-attention layer and a feed-forward layer are used to fuse the kernel of the previous frame into the current frame;

[0053] The number of instance cores t-1 and instance cores t is the same, and the core connection operation is performed;

[0054] The kernel connection operation is a further operation beyond supervised learning, that is, connecting K in the training and inference process. t and K t-1 ; This forces the kernel to interact in the time dimension. The present invention uses a self-attention layer (MHSA: Multi Head Self Attention) and a feed-forward network (FFN) to learn the correspondence between each query, thereby obtaining updated queries, allowing full association between queries; the process is as follows:

[0055] K l =FFN(MHSA9K t ,K t-1 ,K t-1 )+K t )

[0056] Where Q, K and V are K t , K t-1 and K t-1 ; In this way, the kernel of the reference frame is transferred to the key frame through the affinity matrix between kernels; then, the connected kernel K l is sent to the tracking head; during training, this operation is optimized only on the matching instance kernels; K t represents the instance kernel t, K t-1 represents the instance kernel t-1; K l Represents the instance core output after connection;

[0057] Perform a kernel embedding operation on the kernel connection operation result and the instance kernel t-1;

[0058] The kernel embedding operation is simply implemented with a fully connected layer to extract embedded features for each kernel; after kernel embedding, a one-dimensional feature is obtained, which can be used for similarity matching.

[0059] In the VO module, the VO module is divided into two steps: positioning and reconstruction. The panoramic segmentation result helps the VO module to filter out dynamic objects and provides more accurate positioning; but reconstruction also requires the input of monocular video (that is, video with RGB and panoramic information). Different point cloud maps are reconstructed according to the input video stream. Finally, the RGB point cloud, panoramic point cloud and posture information are output in the VO module.

[0060] In the SR module, the monocular video (i.e., video with RGB and panoramic information), RGB point cloud, panoramic point cloud, and pose information are used as output to obtain the final rendering result; the details are as follows:

[0061] The SR module uses the point-based rendering method in NERF technology to train the sparse point cloud (RGB point cloud and panoramic point cloud) with the real image at the same viewing angle through the ω-net neural network loss, and converts it into an explicit neural field through neural network rendering. It learns from the sparse input view to synthesize a high-fidelity RGB image and its corresponding panoramic annotation view. 1 ,p 2 ,…,p A}The number of point clouds is automatically generated by the computer from the input driving scene image and known camera parameters through the VO module;

[0062] In the rasterization stage, the neural descriptor N_d = {nd_1, nd_2, ..., nd_A} is calculated from the point cloud P and updated by propagating gradients of the input data through a neural network. Each 3D point P_A = {x_A, y_A, z_A} is assigned a neural descriptor vector nd_A to encode local geometric and photometric features.

[0063] In order to avoid updating the descriptors of occluded points, the occluded point cloud is filtered by constructing a Z-buffer. Monte Carlo sampling and random image block sampling are used to sample and train the point cloud and image respectively. Monte Carlo sampling samples the synthetic quality of the samples and retrains the samples with poor synthetic quality, while random image block sampling divides the image into consistent image blocks for training. The ratio of image block pixels to the entire image pixels is controlled within 15%, thereby reducing the use of computing resources.

[0064] Beneficial effects of the present invention

[0065] In the present invention, the video panoramic segmentation module outputs the panoramic segmentation result by fusing the geometric information obtained by the visual odometry module. The visual odometry module estimates the camera pose, depth and optical flow, and improves the estimation performance of the system by obtaining semantic information from the video panoramic segmentation module. The two modules promote each other through recursive interaction.

[0066] The present invention improves the confidence of motion estimation by combining panoptic segmentation information and optical flow, thereby improving the accuracy of camera pose estimation.

[0067] Compared with traditional PVO, the present invention adds a channel attention mechanism module to focus on the information between channels and assign weights to assist fusion to improve robustness.

[0068] The present invention takes into account multi-stage decoding and provides decoders at different stages to solve multi-fusion adaptive training outputs;

[0069] The present invention uses the SR module to supplement the problem of holes and sparseness in the point cloud generated by PVO in driving scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is a flow chart of the present invention;

[0071] Figure 2 It is a schematic diagram of the network structure of the present invention;

[0072] Figure 3 This is a diagram of the panoramic segmentation network structure of the present invention;

[0073] Figure 4 It is a structural diagram of the adaptive geometric fusion of the present invention;

[0074] Figure 5 The experimental results of the VPS module of the present invention on the VKITTI2 dataset;

[0075] Figure 6 The experimental results of the SR module of the present invention on the VKITTI2 dataset;

[0076] Figure 7 This is the experimental result of the present invention on the VKITTI2 dataset. DETAILED DESCRIPTION

[0077] The present invention is further described in detail below in conjunction with specific embodiments.

[0078] like Figure 1 and Figure 2 As shown, a scene perception and reconstruction system based on geometric semantic information fusion includes: a video panoramic segmentation (VPS) module, a visual odometry (VO) module and a scene rendering (Sence Render, SR) module;

[0079] In the VPS module, select the target frame image I in the image sequence of the monocular video t and the image I of the adjacent frame t-1 , respectively input the VPS module with ResNet50 as the backbone network to extract the multi-scale feature map Z t and Z t-1 ; The multi-scale feature map is obtained by downsampling, including: a feature map after downsampling by half of the original size of the input image, a feature map after downsampling by one quarter of the original size of the input image, and a feature map after downsampling by one eighth of the original size of the input image;

[0080] In the VO module, the depth D in the monocular video is obtained through the dense bundle adjustment layer (DBA) technology in the VO module. t 、Positionζ t and optical flow F t , and by comparing the image of the adjacent frame of the downsampled target frame with the depth D t 、Positionζ t and optical flow F t Distort and obtain the distorted feature map The distortion is achieved through optical flow, and the expression is as follows:

[0081] I t (x)≈I t-1 (x+f(x))

[0082] In the formula, I t (x) represents the image pixel value at position x at time t;t-1 (x) represents the predicted image pixel value at time t-1; f(x) represents the optical flow field, which represents the displacement vector from time t to time t-1; x is the pixel position vector in the image;

[0083] Further, such as Figure 3 and Figure 4 As shown, the multi-scale feature map z t And the distorted feature map Perform adaptive geometric fusion; the steps are as follows:

[0084] Step 1: Establish a three-layer pyramid network structure. The scale of the first layer is the scale after downsampling by half, the scale of the second layer is the scale after downsampling by one quarter, and the scale of the third layer is the scale after downsampling by one eighth.

[0085] Step 2: Multi-scale feature map z t And the distorted feature map Perform feature concatenation according to the channel dimension to obtain a concatenated feature map z with a channel dimension of 2C i ; In this embodiment, i=4;

[0086] Step 3: Splice the feature map z i Input the pyramid network structure of the corresponding scale;

[0087] Step 4: At each scale, concatenate feature maps z i Input the i-th channel attention mechanism, first through a global average pooling (equivalent to taking the average of all pixels in an image and then retaining only this value and one pixel) to convert the feature map into 1×1×2C, and then perform a one-dimensional convolution on the feature map to the weight Ai of each channel;

[0088] The convolution kernel size is obtained through the adaptive formula: Among them, y, b represent the first learning parameter and the second learning parameter, b = y = 1, |*| odd represents the odd number closest to *;

[0089] Multiply each channel convolution with the concatenated feature map element by element to obtain 2C feature maps with weights;

[0090] The channel attention mechanism is ECA-Net;

[0091] In this embodiment, the weights are automatically generated by the network and the weight Ai of each channel is different;

[0092] Step 5: At the end of each layer of the pyramid network structure, 2C weighted feature maps are input into a 3D convolutional layer with a kernel size of n×n×2C to obtain a feature map after channel reduction to C fusion;

[0093] The size n of the convolution kernel varies with the resolution of the input video image. For an image with a resolution of 375*1242, n is 3. n can be adjusted by multiples of the number of pixels. In this embodiment, n is 3.

[0094] After processing the feature maps of each scale, the fused feature map of the branch is generated The expression is as follows:

[0095]

[0096] In the formula, f avg represents the global average pooling operation, c represents the channel-by-channel feature concatenation, represents element-by-element multiplication; σ represents the sigmoid function, which maps real numbers to the range (0, 1); W g and b g are the first neural network learning parameters and the second neural network learning parameters respectively;

[0097] In this embodiment, W g and b g Learned by neural network, the specific value depends on training;

[0098] Step 6: Decode and output the panoramic segmentation result; the specific process is as follows:

[0099] First, the first frame of the monocular video is input into the first feature map group embedding operation for initialization training, that is, there is no geometric information fusion stage;

[0100] Afterwards, the fusion feature map Input the first feature map group embedding operation;

[0101] Feature map group embedding is an operation in the preparation stage of the kernel update head operation, which can make the kernel group-aware; that is, given a feature map F and a kernel prediction mask M p-1 , the group feature F of each kernel k Calculated by pixel-by-pixel multiplication, the expression is as follows:

[0102]

[0103] Among them, u, v represent the coordinates of the height and width of the pixel; the size of the feature map F is H×W×C, and the feature map F is In this embodiment, p∈(1,4);

[0104] Next, Perform the first core update header operation f s1 , the second core updates the header operation f s2 , the third core updates the header operation f s3 ;

[0105] The kernel update head operation includes three steps: feature map group embedding, adaptive kernel update and kernel interaction;

[0106] The formula for the core update header is expressed as:

[0107] K p ,M p =f sp (K p-1 ,M p-1 ,F)

[0108] In the formula, K p Representative core (including semantic core and instance core), M p represents the prediction of the mask, f sp represents the kernel update header (including three steps: feature map group embedding, adaptive kernel update and kernel interaction), F represents the feature map (in this embodiment, it refers to the feature map input after adaptive geometric fusion). );

[0109] The feature map group embedding expression is:

[0110]

[0111] Among them, u, v represent the coordinates of the height and width of the pixel; the size of the feature map F is H×W×C, and the feature F is z i t * ; In this embodiment, p∈(1,4);

[0112] Adaptive kernel update is due to the kernel prediction mask M p-1 There may be errors, the extracted group features F k Noise may be mixed in; therefore, the kernel feature K is fused by element-by-element multiplication. p-1 With group features:

[0113]

[0114] In the formula, Represents element-by-element multiplication; in order to control the impact of noise, the model learns the first gating parameter G F and the second gating parameter G k , respectively control F k and K p-1 Importance in kernel updates; the expression is as follows:

[0115] G F =σ(φ 3 (F G )),G K =σ(φ 4 (F G ))

[0116] The final kernel update formula is:

[0117] K * =G F ·φ 5 (F k )+G K ·φ 6 (K i-1 )

[0118] In the formula, φ N represents the use of the Nth fully connected layer (FC) + layer normalization (LN), σ is sigmoid; N = {1,…,6};

[0119] Furthermore, G. F and G k Similar to the self-attention mechanism in Transformer, its output is calculated by weighted summation of values; in Transformer, the weight assigned to each value is usually calculated by the dot product compatibility function of Q and K; similarly, the adaptive kernel update essentially performs a kernel feature update on K. p-1 and group feature F G A weighted sum is performed; its weight G F and G k It is calculated by element-by-element multiplication and can be regarded as another compatibility function;

[0120] When inputting a frame at time t, p = 3, which means that the core updates the header three times, while the frame at time t-1 p = 1;

[0121] Kernel interaction simply uses a multi-head attention mechanism and then connects it to a feedforward neural network (FFN) to get a new kernel K p . A new mask prediction is then generated using the formula:

[0122] M p =g p (K p )F

[0123] In the formula, g p It is a network consisting of a fully connected layer (FC), layer normalization (LN) and ReLU activation function, followed by a fully connected layer; in addition, K p It is also used for classification score prediction in instance and panoptic segmentation;

[0124] Finally, after f s3 After the kernel interaction operation in the stage, the panoramic segmentation result and multiple instance kernels t are obtained;

[0125] Furthermore, Z t-1 After the first feature map group embedding operation, fs1 The first core update header operation of the stage, the first core update header operation f s1 The results of the two operations are respectively performed, including: generating multiple instance cores t-1 and updating the first core head operation f s1 The result is similar to f s3 The kernel fusion operation is performed on the feature map group embedding operation of the stage;

[0126] Kernel fusion operation To maintain segmentation consistency, the present invention uses input kernels from two frames for fusion; first, a kernel update operation is performed to update the kernel of the previous frame using the kernel features of the current frame; then, a self-attention layer and a feed-forward layer are used to fuse the kernel of the previous frame into the current frame;

[0127] The number of instance cores t-1 and instance cores t is the same, and the core connection operation is performed;

[0128] The kernel connection operation is a further operation beyond supervised learning, that is, connecting K t and K t-1 ; This forces the kernel to interact in the time dimension. The present invention uses a self-attention layer (MHSA: Multi Head Self Attention) and a feed-forward network (FFN) to learn the correspondence between each query, thereby obtaining updated queries, allowing full association between queries; the process is as follows:

[0129] K l =FFN(MHSA(K t ,K t-1 ,K t-1 )+K t )

[0130] Where Q, K and V are K t , K t-1 and K t-1 ; In this way, the kernel of the reference frame is transferred to the key frame through the affinity matrix between kernels; then, the connected kernel K l is sent to the tracking head; during training, this operation is optimized only on the matching instance kernels; K t represents the instance kernel t, K t-1 represents the instance kernel t-1; K l Represents the instance core output after connection;

[0131] Perform a kernel embedding operation on the kernel connection operation result and the instance kernel t-1;

[0132] The kernel embedding operation is simply implemented with a fully connected layer to extract embedded features for each kernel; after kernel embedding, a one-dimensional feature is obtained, which can be used for similarity matching.

[0133] In the VO module, the VO module is divided into two steps: positioning and reconstruction. The panoramic segmentation result helps the VO module to filter out dynamic objects and provides more accurate positioning; but reconstruction also requires the input of monocular video (that is, video with RGB and panoramic information). Different point cloud maps are reconstructed according to the input video stream. Finally, the RGB point cloud, panoramic point cloud and posture information are output in the VO module.

[0134] In the SR module, the monocular video (i.e., video with RGB and panoramic information), RGB point cloud, panoramic point cloud, and pose information are used as output to obtain the final rendering result; the details are as follows:

[0135] The SR module uses the point-based rendering method in NERF technology to train the sparse point cloud (RGB point cloud and panoramic point cloud) with the real image at the same viewing angle through the ω-net neural network loss, and converts it into an explicit neural field through neural network rendering. It learns from the sparse input view to synthesize a high-fidelity RGB image and its corresponding panoramic annotation view. 1 ,p 2 ,…,p A}The number of point clouds is automatically generated by the computer from the input driving scene image and known camera parameters through the VO module;

[0136] In the rasterization stage, the neural descriptor N_d = {nd_1, nd_2, ..., nd_A} is calculated from the point cloud P and updated by propagating gradients of the input data through a neural network. Each 3D point P_A = {x_A, y_A, z_A} is assigned a neural descriptor vector nd_A to encode local geometric and photometric features.

[0137] In order to avoid updating the descriptors of occluded points, the occluded point cloud is filtered by constructing a Z-buffer. Monte Carlo sampling and random image block sampling are used to sample and train the point cloud and image respectively. Monte Carlo sampling samples the synthetic quality of the samples and retrains the samples with poor synthetic quality, while random image block sampling divides the image into consistent image blocks for training. The ratio of image block pixels to the entire image pixels is controlled within 15%, thereby reducing the use of computing resources.

[0138] The present invention evaluates the accuracy of segmentation by calculating the video panoramic segmentation quality (VPQ). As shown in Table 1, it can be clearly observed that compared with the prior art PVO and Video K-Net, the present method predicts the most accurate segmentation mask in dynamic scenes. This is because the present method successfully integrates geometric information to keep the information consistent between frames.

[0139] Table 1 Performance of video panoptic segmentation

[0140]

[0141] In the table, (↑) indicates that the larger the index number, the better the performance. It can be found that the prior art PVO, PVO+AG Fusion (AG Fusion represents the adaptive geometric information fusion module), the prior art Video K-Net and Video K-Net+PVOFusion are not as good as the present invention in the results of calculating the video panoramic segmentation quality (VPQ) under the two datasets VKITTI2 and VIPER.

[0142] It can be seen from Table 2 that on both the synthetic dataset and the real dataset, the proposed system achieves the best positioning results in comparison with the other two methods. This is also because good segmentation results also help VO filter out dynamic objects, thereby estimating more accurate camera poses.

[0143] Table 2 Positioning performance

[0144]

[0145] In the table, (↑) indicates that the larger the index number, the better the performance. It can be found that the prior art DROID-SLAM, PVO, PVO+AG Fusion and the present invention are compared by the root mean square error on the data sets VKITTI2 and KITTI, and it can be found that the error of the present invention is smaller than that of the prior art.

[0146] The experimental results of this method are quantitatively analyzed by calculating the peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) and perceptual loss (LPIPS). The calculation results are shown in Table 3.

[0147] Table 3 Performance of rendering reconstruction quality

[0148]

[0149] In the table, KITTI and VKITTI are the names of the datasets, and the numbers 09, 10, and 20 represent the video sequence numbers. Psnr is the peak signal ratio, ssim is the structural similarity, and lpips is the learning-perceptual patch similarity. It can be seen that when more accurate positioning and point cloud results are used as input, the rendering quality can be improved, demonstrating the improvement of the coupling between multiple modules of the present invention.

[0150] The method of the present invention has excellent robustness and scalability. The use of multiple technologies in the attention mechanism of the fusion module can still improve the segmentation performance, as shown in Table 4;

[0151] Table 4 Performance of video panoptic segmentation

[0152]

[0153] In the table, (↑) represents that the larger the index number, the better the performance. Video K-Net adopts the adaptive geometric information fusion module of the present invention, and replaces the channel attention technology in the adaptive geometric information fusion module with three different ones for comparison, including SE, CBMA, and ECA. It can be seen that the channel attention mechanism used in the present invention is ECA, and the video panoramic segmentation quality effect on the dataset VKITTI2 is the best.

[0154] In this embodiment, the VKITTI2 dataset is used for experiments. Figure 5 As shown, it can be clearly seen from the experimental results that the present invention successfully and completely segments the background and tracks the instance target.

[0155] like Figure 6 As shown, the experimental results on the VKITTI2 dataset. It can be clearly seen from the experimental results that the present invention successfully reconstructs the point cloud scene and synthesizes a dense perspective through the rendering module.

[0156] like Figure 7 As shown in FIG. 1 , the experimental results on the VKITTI2 dataset are shown. It can be clearly seen from the experimental results that the present invention successfully demonstrates the ability to synthesize new perspectives by rotating 5° left and right.

Claims

1. A scene perception and reconstruction system based on geometric semantic information fusion, characterized in that: The system comprises: a video panorama segmentation module, a visual odometer module and a scene rendering module; Inputting the image of the target frame and the image of the adjacent frame of the target frame in the monocular video into the video panorama segmentation module to obtain a multi-scale feature map, wherein the multi-scale feature map includes: a scale after downsampling by half, a scale after downsampling by one quarter, and a scale after downsampling by one eighth; The target frame is any frame or a selected frame in the monocular video; The depth, pose and optical flow of monocular video are obtained through the visual odometry module; After the multi-scale feature map of the target frame adjacent to the target frame is distorted with the depth, pose and optical flow, it is adaptively geometrically fused and decoded with the multi-scale feature map of the target frame to obtain the panoramic segmentation result; Through the scene rendering module, the visual odometry module extracts RGB point cloud and panoramic point cloud, and through the scene rendering module, the panoramic point cloud and RGB point cloud are synthesized into high-quality scene reconstruction to obtain the final result.

2. A scene perception and reconstruction system based on geometric semantic information fusion according to claim 1, characterized in that: The distortion is achieved through optical flow, and the expression is as follows: I t (x)≈I t-1 (x+f(x)) In the formula, I t (x) represents the image pixel value at position x at time t; t-1 (x) represents the predicted image pixel value at time t-1; f(x) represents the optical flow field, which represents the displacement vector from time t to time t-1; x is the pixel position vector in the image.

3. The scene perception and reconstruction system based on geometric semantic information fusion according to claim 1, characterized in that: The steps of the adaptive geometric fusion are as follows: Step 1: Establish a three-layer pyramid network structure. The scale of the first layer is the scale after downsampling by half, the scale of the second layer is the scale after downsampling by one quarter, and the scale of the third layer is the scale after downsampling by one eighth. Step 2: The multi-scale feature map of the target frame image and the distorted multi-scale feature map are spliced ​​according to the channel dimension to obtain a spliced ​​feature map with a channel dimension of 2C; Step 3: Input the concatenated feature map into the pyramid network structure of the corresponding scale; Step 4: At each scale, concatenate feature maps z i Input the i-th channel attention mechanism, firstly transform the feature map into 1×1×2C through a global average pooling, then perform a one-dimensional convolution on the feature map to the weight Ai of each channel, and multiply each channel convolution with the concatenated feature map element by element to obtain 2C feature maps with weights Step 5: At the end of each layer of the pyramid network structure, 2C weighted feature maps are input into a 3D convolutional layer with a kernel size of n×n×2C to obtain a feature map after channel reduction to C fusion; Step 6: Decode and output the panoramic segmentation result.

Images