Visual map data processing method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of visual SLAM positioning and mapping, and discloses a visual map data processing method and device, computer equipment and a storage medium. Firstly, panoramic semantic segmentation is carried out on an original image, a panoramic segmented image is generated, the panoramic segmented image comprises semantic information of each pixel, and the semantic information comprises prior dynamic semantic information. And performing optical flow estimation processing on the original image to obtain an optical flow image. Then, based on the panoramic segmented image, the optical flow image and the original image, feature points of a dynamic area are removed in real time, and a static feature image with higher precision is obtained; and finally, under the condition of performing motion tracking based on the static feature image and determining that the static feature image is a key frame, generating a dense point cloud map endowed with semantic information based on the original image, the static feature image and the panoramic segmentation image, and removing a point cloud with prior dynamic semantic information from the dense point cloud map. And a more accurate static environment map is constructed to update the point cloud map.

Description

Technical Field

[0001] This invention relates to the field of visual SLAM localization and mapping technology, and in particular to a visual map data processing method, apparatus, computer equipment and storage medium. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is commonly used for navigation in indoor intelligent robots, but it typically assumes a static environment, thus its pose estimation relies on the static scene. In dynamic scenes, the robustness and real-time performance of SLAM are affected.

[0003] In related technologies, multi-view geometry-based methods calculate camera pose using consecutive frames and coarsely estimate the dense 3D optical flow of the image. Outliers can be eliminated by calculating the Mahalanobis distance between matching points in two frames and comparing it with a threshold. However, multi-view geometry-based methods suffer from insufficient robustness in most dynamic scenes. Summary of the Invention

[0004] The embodiments described in this specification aim to at least partially solve one of the technical problems in the related art. To this end, the embodiments described in this specification propose a visual map data processing method, apparatus, computer device, and storage medium.

[0005] This specification provides a visual map data processing method, the method comprising: The original image is subjected to panoramic semantic segmentation to generate a panoramic segmented image, wherein the panoramic segmented image includes semantic information of each pixel, and the semantic information includes prior dynamic semantic information; Optical flow estimation is performed on the original image to obtain an optical flow image; Based on the panoramic segmentation image, the optical flow image, and the original image, a static feature image is obtained; When motion tracking is performed based on the static feature image and the static feature image is determined to be a keyframe, a dense point cloud map with semantic information is generated based on the original image, the static feature image and the panoramic segmentation image, and point clouds with the prior dynamic semantic information are removed from the dense point cloud map to update the point cloud map.

[0006] In one implementation, obtaining a static feature image based on the panoramic segmentation image, the optical flow image, and the original image includes: Feature points are extracted from the original image to generate a feature image; Based on the mutual verification of the panoramic segmentation image and the optical flow image, a target dynamic region mask image is generated, wherein the target dynamic region mask image includes a target dynamic region mask region and the target dynamic region mask image includes semantic information of each pixel; Based on the target dynamic region mask image, a mask is overlaid at the corresponding position of the target dynamic region mask area in the feature image to obtain a static feature image.

[0007] In one implementation, motion tracking based on the static feature image and determining whether the static feature image is a keyframe includes: Pose tracking is performed based on the static feature image to obtain the pose tracking result; If the pose tracking result is normal, local map tracking is performed based on the static feature image to obtain the tracking result; If the pose tracking result fails, relocation tracking is performed based on the static feature image, followed by local map tracking, to obtain the tracking result. Based on the tracking results and the preset keyframe selection strategy, it is determined whether the static feature image is a keyframe.

[0008] In one implementation, based on the original image, the static feature image, and the panoramic segmentation image, a dense point cloud map with semantic information is generated, and point clouds with the prior dynamic semantic information are removed from the dense point cloud map to update the point cloud map, including: Based on the original image, the static feature image, and the target dynamic region mask image, a dense point cloud map with semantic information is generated, and point clouds with the prior dynamic semantic information are removed from the dense point cloud map to update the point cloud map.

[0009] In one implementation, generating a dense point cloud map with semantic information based on the original image, the static feature image, and the target dynamic region mask image, and removing point clouds with the prior dynamic semantic information from the dense point cloud map to update the point cloud map, includes: A dense point cloud map is generated based on the original image and the static feature image. Based on the target dynamic region mask image, semantic information is assigned to the dense point cloud map and point clouds with the prior dynamic semantic information are removed from the dense point cloud map to update the point cloud map.

[0010] In one embodiment, the method further includes: When the static feature image is a keyframe, local map optimization is performed based on the static feature image and the target dynamic region mask image to update the point cloud map.

[0011] In one implementation, when the static feature image is a keyframe, performing local map optimization based on the static feature image and the target dynamic region mask image to update the point cloud map includes: Based on the static feature image and the target dynamic region mask image, the feature points in the local space are optimized, redundant keyframes and low-quality map points are deleted, and semantic information is assigned to update the point cloud map.

[0012] This specification provides a visual map data processing apparatus, the apparatus comprising: A panoramic semantic segmentation module is used to perform panoramic semantic segmentation on the original image to generate a panoramic segmented image, wherein the panoramic segmented image includes semantic information of each pixel, and the semantic information includes prior dynamic semantic information. The optical flow estimation processing module is used to perform optical flow estimation processing on the original image to obtain an optical flow image; A static feature image acquisition module is used to obtain a static feature image based on the panoramic segmentation image, the optical flow image, and the original image; The point cloud map update module is used to generate a dense point cloud map with semantic information based on the original image, the static feature image and the panoramic segmentation image, after motion tracking is performed based on the static feature image and the static feature image is determined to be a keyframe, and then remove point clouds with the prior dynamic semantic information from the dense point cloud map to update the point cloud map.

[0013] This specification provides a computer device comprising: a memory, and one or more processors communicatively connected to the memory; the memory stores instructions executable by the one or more processors, the instructions being executed by the one or more processors to cause the one or more processors to perform the steps of the method described in any of the above embodiments.

[0014] This specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0015] This specification provides a computer program product that includes instructions that, when executed by a processor of a computer device, enable the computer device to perform the steps of the method described in any of the above embodiments.

[0016] In the above-described implementation, firstly, panoramic semantic segmentation is performed on the original image to generate a panoramic segmentation image. The panoramic segmentation image includes semantic information for each pixel, including prior dynamic semantic information. Optical flow estimation is then performed on the original image to obtain an optical flow image. Next, based on the panoramic segmentation image, the optical flow image, and the original image, feature points in dynamic regions are removed in real time to obtain a more accurate static feature image, thereby improving the robustness and real-time performance of subsequent motion tracking. Finally, with motion tracking based on the static feature image and the static feature image identified as a keyframe, a dense point cloud map with semantic information is generated based on the original image, the static feature image, and the panoramic segmentation image. Points with prior dynamic semantic information are removed from the dense point cloud map to construct a more accurate static environment map. This map is then used to update the point cloud map, further improving the localization and mapping accuracy of the visual map data processing system in dynamic scenes. Attached Figure Description

[0017] Figure 1a A flowchart illustrating the visual map data processing method provided in the embodiments of this specification; Figure 1b A schematic diagram of the structure of the Panoptic FCN network provided for the embodiments of this specification; Figure 1c A schematic diagram of the structure of the PWC-Net network provided for the implementation of this specification; Figure 2 A schematic diagram illustrating the process of obtaining static feature images provided for embodiments of this specification; Figure 3 A flowchart illustrating the process of determining whether a static feature image is a keyframe, provided for the implementation of this specification. Figure 4 A schematic diagram illustrating the process of updating a point cloud map as provided in the embodiments of this specification; Figure 5 A flowchart illustrating the visual map data processing method provided in the embodiments of this specification; Figure 6 A schematic diagram of a visual map data processing apparatus provided for embodiments of this specification; Figure 7 An internal structural diagram of a computer device provided for embodiments of this specification. Detailed Implementation

[0018] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0019] Simultaneous Localization and Mapping (SLAM) is commonly used for navigation in indoor intelligent robots, but it typically assumes a static environment, thus its pose estimation relies on the static scene. In dynamic scenes, the robustness and real-time performance of SLAM are affected.

[0020] In related technologies, there are two main methods for handling dynamic objects in dynamic scenes within SLAM systems: (1) Based on multi-view geometry and related improved methods. These methods calculate the camera pose using consecutive frames and roughly estimate the dense 3D optical flow of the image. Outliers can be eliminated by calculating the Mahalanobis distance between matching points in two frames and comparing it with a threshold. In addition, the Random Sample Consensus (RANSAC) algorithm can be used to calculate the identity matrix of adjacent frames and perform matrix operations with the previous frame to obtain the transformed image. After subtracting from the current frame image, the pixel values ​​of static regions are close to 0, while the pixel values ​​of dynamic regions are greater than 0. After post-processing, the segmentation result is obtained.

[0021] (2) Methods that do not rely on camera motion. These methods use Mask R-CNN (Region-Convolutional Neural Network) to perform semantic segmentation on images and identify highly dynamic regions in the environment. Pixels that do not belong to dynamic regions and their neighboring pixels are treated as static regions, and then the static regions are embedded into the DSO (DirectSparse Odometry) system, which can achieve better results. In addition, the DynaSLAM method based on the ORB-SLAM2 framework uses the SSD (Single Shot MultiBox Detector) network to label dynamic objects in the scene based on prior information, filters out dynamic feature points, and uses the remaining static feature points to build the map. There is also a method called DS-SLAM, which combines motion consistency detection to remove dynamic feature points in dynamic scenes.

[0022] However, multi-view geometry-based methods suffer from insufficient robustness in most dynamic scenes, while methods relying on prior semantic information are limited by the types and number of dynamic objects in the training set. When dynamic objects not in the training set appear in the scene, the system's localization accuracy may decrease.

[0023] Based on the above analysis, this specification provides a visual map data processing method. First, panoramic semantic segmentation is performed on the original image to generate a panoramic segmentation image. The panoramic segmentation image includes semantic information for each pixel, including prior dynamic semantic information. Optical flow estimation is then performed on the original image to obtain an optical flow image. Next, based on the panoramic segmentation image, the optical flow image, and the original image, feature points in dynamic regions are removed in real time to obtain a more accurate static feature image, thereby improving the robustness and real-time performance of subsequent motion tracking. Finally, with motion tracking based on the static feature image and the static feature image identified as a keyframe, a dense point cloud map with semantic information is generated based on the original image, the static feature image, and the panoramic segmentation image. Points with prior dynamic semantic information are removed from the dense point cloud map to construct a more accurate static environment map. This is used to update the point cloud map and further improve the positioning and mapping accuracy of the visual map data processing system in dynamic scenes.

[0024] This specification provides a visual map data processing method. Please refer to [link / reference]. Figure 1a The visual map data processing method may include the following steps: S110. Perform panoramic semantic segmentation on the original image to generate a panoramic segmented image.

[0025] The panoramic segmentation image includes semantic information for each pixel, which includes prior dynamic semantic information.

[0026] Specifically, an image acquisition device (such as an RGB-D depth camera) is used to capture images of the visible environment to obtain the original image. This original image can be a depth image. After obtaining the original image, it is input into a panoramic segmentation network for further processing. The panoramic segmentation network performs the dual tasks of semantic segmentation and instance segmentation, identifying static backgrounds and dynamic foreground objects in the image and performing panoramic-level semantic segmentation. It assigns a corresponding semantic label to each pixel in the image, ultimately outputting a panoramic segmented image. Each pixel in the panoramic segmented image is associated with semantic information, including prior dynamic semantic information determined based on prior knowledge. This helps the system distinguish between static background and dynamic foreground regions, effectively removing feature points in dynamic regions during subsequent processing and improving the accuracy of the constructed map. The panoramic segmentation network can be box-based panoramic segmentation, such as the Panoptic FPN network (Panoptic Feature Pyramid Network), or box-free panoramic segmentation, such as DeeperLab.

[0027] For example, a panoptic segmentation network could be a Panoptic FCN network. Panoptic FCN networks predict the object and background by unifying them into feature descriptors (kernels), thus eliminating complex post-processing steps and the fusion of information from different branches. See also... Figure 1b , Figure 1b The structure of the Panoptic FCN network is shown. The Panoptic FCN network mainly consists of three parts: Kernel Generator, Kernel Fusion, and Feature Encoder.

[0028] First, a kernel generator is introduced to generate kernel weights for both the thing and the stuff. Specifically, the kernel generator consists of two branches: a kernel head and a position head. The position head is responsible for simultaneously predicting the positions of both the thing and the stuff. The thing is located and classified by predicting its center position, while the stuff is located and classified by predicting its region. Then, the kernel head generates the corresponding kernel weights based on this positional information.

[0029] Then, kernel weights from different stages are averaged using kernel fusion to obtain kernel weights for all things and stuff. A thresholding operation is then used to remove similar kernel weights, ultimately generating kernel weights for M things and N kernel weights for N stuff.

[0030] Finally, the feature encoder encodes the high-resolution features and performs a convolution operation on the encoded features using M+N kernel weights obtained through Kernel Fusion to obtain the final prediction result. Each output channel corresponds to a mask prediction of a thing or stuff.

[0031] S120. Perform optical flow estimation processing on the original image to obtain an optical flow image.

[0032] Specifically, after obtaining the original image, it is input into an optical flow estimation network for further processing. The optical flow estimation network performs optical flow estimation on the original image, calculating a two-dimensional motion vector for each pixel in the image, thereby automatically identifying regions with high motion amplitude and using them as candidate regions for dynamic objects. Subsequently, the processing focuses on these candidate regions: by analyzing the continuity and consistency of the optical flow direction, the network can identify and extract the most representative dynamic feature points (such as corners or edges of moving objects), thus generating a complete optical flow image. In this optical flow image, the regions corresponding to dynamic objects typically have vector amplitudes significantly higher than the background, and these enhanced dynamic feature points and their motion directions are clearly presented.

[0033] For example, the optical flow estimation network can be the PWC-Net network. The PWCNet network utilizes multi-scale features to replace the concatenation of sub-networks, and its design follows three core principles: pyramid feature extraction, optical flow mapping (warping), and matching relevance cost measurement (Cost Volume). Neither the warping nor the cost volume contains any learned parameters, which significantly reduces the size of the network model while ensuring network effectiveness. On the Sintel dataset (1024×436), the PWC-Net network runs at approximately 35 fps. See also... Figure 1c , Figure 1c The structure of the PWC-Net network is shown. The PWC-Net network uses a pyramid structure to extract features at multiple scales from the two input images. Each feature scale corresponds to a different receptive field, enabling the capture of more feature information. The warping layer applies the calculated optical flow to the target image to generate mapped image data. The mapping result of each layer is further refined based on the optical flow estimation result of the previous layer. The cost volume is used to calculate the correlation measure (i.e., cost) between the features of input image 1 and the features of image 2 after optical flow mapping.

[0034] The optical flow estimator is a simple multi-layer convolutional neural network (CNN). Its inputs are the Cost Volume calculated in the current layer, the features of image 1 in the current layer, and the concatenation of the upsampled optical flow from the previous layer. The output is the optical flow prediction for the current layer. The context network effectively improves the optical flow estimation by expanding the receptive field of each output unit. It takes the optical flow estimate and features from the penultimate layer of the optical flow estimator and outputs a refined optical flow estimate.

[0035] S130. Based on the panoramic segmentation image, optical flow image, and original image, obtain a static feature image.

[0036] S140. When motion tracking is performed based on static feature images and the static feature images are determined as keyframes, a dense point cloud map with semantic information is generated based on the original image, the static feature image and the panoramic segmentation image. Point clouds with prior dynamic semantic information are removed from the dense point cloud map to update the point cloud map.

[0037] In some cases, because the original image represents a real dynamic scene, it may contain moving objects such as pedestrians, vehicles, and animals, as well as mobile objects like bags carried by pedestrians. The feature points of these moving objects can cause mismatches between frames, leading to issues like lost feature tracking and trajectory drift. This negatively impacts the system's positioning accuracy and robustness, significantly reducing the accuracy of the constructed environment map. Furthermore, in highly dynamic scenes, rapid object movement can cause image blurring, and low-quality feature points become unstable between adjacent frames. Therefore, it is necessary to remove dynamic objects to improve the accuracy of subsequent motion tracking.

[0038] Specifically, the panoramic segmentation image, imbued with prior dynamic semantic information based on prior knowledge, is used in conjunction with instantaneous motion information provided by optical flow images to perform spatiotemporal cross-validation of the prior dynamic regions, accurately filtering out pixels that are actually in motion. Then, these validated dynamic regions are removed from the original image, resulting in a static feature image containing a static background and stationary objects. Motion tracking is then performed using the static feature image; by analyzing motion changes in the image, the relative displacement and changes of objects in the scene are determined. Based on the motion tracking results, a preset keyframe selection strategy is used to determine whether the static feature image should be selected as a keyframe. If the static feature image is determined to be a keyframe, a dense point cloud map with semantic information is generated based on the original image, the static feature image, and the panoramic segmentation image. Furthermore, to improve the static consistency and robustness of the point cloud map in dynamic scenes, the system removes point clouds with prior dynamic semantic information (e.g., 3D points belonging to non-static objects such as pedestrians and vehicles) based on the semantic information provided by the panoramic segmentation image. By fusing and updating point cloud maps that have removed prior dynamic semantic information with point cloud maps generated based on historical keyframes, a semantic map that is more consistent with the structure of the real static environment is constructed.

[0039] In the above implementation, firstly, panoramic semantic segmentation is performed on the original image to generate a panoramic segmentation image. The panoramic segmentation image includes semantic information for each pixel, including prior dynamic semantic information. Optical flow estimation is then performed on the original image to obtain an optical flow image. Next, based on the panoramic segmentation image, the optical flow image, and the original image, feature points in dynamic regions are removed in real time to obtain a more accurate static feature image, thereby improving the robustness and real-time performance of subsequent motion tracking. Finally, with motion tracking based on the static feature image and the static feature image identified as a keyframe, a dense point cloud map with semantic information is generated based on the original image, the static feature image, and the panoramic segmentation image. Points with prior dynamic semantic information are removed from the dense point cloud map to construct a more accurate static environment map. This map is then used to update the point cloud map, further improving the localization and mapping accuracy of the visual map data processing system in dynamic scenes.

[0040] In some implementations, please refer to Figure 2 Obtaining a static feature image based on the panoramic segmentation image, optical flow image, and original image may include the following steps: S210. Extract feature points from the original image to generate a feature image.

[0041] S220: Based on the mutual verification of the panoramic segmentation image and the optical flow image, a target dynamic region mask image is generated.

[0042] S230. Based on the target dynamic region mask image, cover the corresponding position of the target dynamic region mask area in the feature image to obtain a static feature image.

[0043] The target dynamic region mask image includes the target dynamic region mask area, and the target dynamic region mask image includes semantic information for each pixel.

[0044] Specifically, feature extraction algorithms are used to extract feature points from the original image to obtain feature points that can effectively represent the key structural features of the image, and corresponding feature images are generated accordingly. Since panoramic segmentation images may be inaccurate due to occlusion or category confusion, and optical flow images are prone to errors in sparse texture regions or under varying illumination, both panoramic segmentation images and optical flow images are considered together. Cross-validation between the two is performed to improve the robustness and accuracy of dynamic region detection, generating a target dynamic region mask image that includes a more precise target dynamic region mask area. Then, using the target dynamic region mask image, the target dynamic region mask area is mapped onto the feature image. Mask information is used to cover the corresponding positions of feature points falling within the target dynamic region mask area, achieving direct removal of feature points and suppressing interference from dynamic objects on subsequent visual tasks (such as SLAM and 3D reconstruction). Finally, a static feature image with higher reliability and scene consistency is obtained. For example, the feature extraction algorithm includes ORB (Oriented Fast and Rotated BRIEF).

[0045] In the above embodiments, feature points are extracted from the original image to generate a feature image. Based on mutual verification between the panoramic segmentation image and the optical flow image, a target dynamic region mask image is generated, including a more accurate target dynamic region mask area. Based on the target dynamic region mask image, a mask is overlaid at the corresponding position of the target dynamic region mask area in the feature image to obtain a more accurate static feature image.

[0046] In some implementations, please refer to Figure 3 Motion tracking based on static feature images and determining whether a static feature image is a keyframe can include the following steps: S310. Perform pose tracking based on static feature images to obtain pose tracking results.

[0047] S320. If the pose tracking result is normal, perform local map tracking based on the static feature image to obtain the tracking result.

[0048] S330. If the pose tracking result fails, perform relocation tracking based on the static feature image and then perform local map tracking to obtain the tracking result.

[0049] S340. Based on the tracking results and the preset keyframe selection strategy, determine whether the static feature image is a keyframe.

[0050] Specifically, in the visual localization process based on static feature images, pose tracking is first performed to roughly estimate the camera pose of the current frame (i.e., the static feature image) relative to the previous frame, obtaining the pose tracking result. If this pose tracking result is normal (i.e., tracking is successful and has high confidence), the local map tracking module is then started. This module optimizes and outputs a more accurate tracking result by matching the static features of the current frame (i.e., the static feature image) with 3D landmarks in the local map. Conversely, if the pose tracking result fails (e.g., due to excessively fast movement or occlusion causing feature tracking loss), relocalization tracking will be performed. The relocalization process re-estimates the camera pose by matching the static features of the current frame (i.e., the static feature image) with keyframes in the global map; after successful relocalization, local map tracking will continue to ensure the continuity and stability of subsequent tracking, ultimately obtaining a usable tracking result. After obtaining the tracking result in any of the above situations, a preset keyframe selection strategy will be used to determine whether the current frame (i.e., the static feature image) should be selected as a keyframe. The preset keyframe selection strategy may include ensuring that the time interval or camera movement distance (spatial baseline) between the current frame and the previous keyframe meets preset requirements to ensure that there is effective parallax between keyframes; that the number of successfully tracked static feature points in the current frame meets preset requirements and is evenly distributed to robustly support camera pose estimation; and that the scene content observed in the current frame has changed significantly compared to existing keyframes, such as the appearance of new areas or features not fully covered by existing maps.

[0051] In the above implementation, pose tracking is performed based on static feature images to obtain pose tracking results. If the pose tracking results are normal, local map tracking is performed based on static feature images to obtain tracking results. If the pose tracking results fail, relocation tracking is performed based on static feature images, followed by local map tracking to obtain tracking results. Based on the tracking results and a preset keyframe selection strategy, it is determined whether the static feature images are keyframes, providing a basis for whether to update the point cloud map subsequently.

[0052] In some implementations, a dense point cloud map with semantic information is generated based on the original image, static feature image, and panoramic segmentation image, and point clouds with prior dynamic semantic information are removed from the dense point cloud map to update the point cloud map. This may include: generating a dense point cloud map with semantic information based on the original image, static feature image, and target dynamic region mask image, and removing point clouds with prior dynamic semantic information from the dense point cloud map to update the point cloud map.

[0053] Specifically, since the target dynamic region mask image is obtained by cross-verifying the panoramic segmentation image with the optical flow image, it can more accurately represent the target dynamic region in the static image. The target dynamic region mask image includes the target dynamic region mask area and contains semantic information for each pixel. Therefore, after obtaining the target dynamic region mask image, a dense point cloud map with semantic information is generated based on the original image, the static feature image, and the target dynamic region mask image. Further, to improve the static consistency and robustness of the point cloud map in dynamic scenes, the system removes point clouds with prior dynamic semantic information (e.g., 3D points belonging to non-static objects such as pedestrians and vehicles) based on the semantic information provided by the target dynamic region mask image. The point cloud map with prior dynamic semantic information removed is then fused and updated with the point cloud map generated based on historical keyframes, thereby constructing a semantic map that better conforms to the structure of the real static environment.

[0054] In the above embodiments, a dense point cloud map with semantic information is generated based on the original image, static feature image and target dynamic region mask image, and point clouds with prior dynamic semantic information are removed from the dense point cloud map to update the point cloud map, thereby further improving the positioning and mapping accuracy of the visual map data processing system in dynamic scenes.

[0055] In some implementations, please refer to Figure 4 Based on the original image, static feature image, and target dynamic region mask image, a dense point cloud map with semantic information is generated. Point clouds with prior dynamic semantic information are removed from the dense point cloud map to update the point cloud map. This may include the following steps: S410. Generate a dense point cloud map based on the original image and static feature image.

[0056] S420. Based on the target dynamic region mask image, semantic information is assigned to the dense point cloud map and point clouds with prior dynamic semantic information are removed from the dense point cloud map to update the point cloud map.

[0057] Specifically, based on the original image and static feature image, pixel-level registration and semantic association are performed in a unified spatiotemporal coordinate system. Utilizing depth values, color, normal vectors, and geometric context information, a dense point cloud map containing both accurate 3D coordinates and rich visual / semantic attributes is generated through fusion filtering, anomaly removal, and pixel-level optimization. After generating the dense point cloud map from the static feature image of keyframes, a target dynamic region mask image is used to map the target dynamic region mask area onto the dense point cloud map. Mask information is used to cover the corresponding positions of point clouds falling within the target dynamic region mask area, thereby directly removing point clouds with prior dynamic semantic information, effectively improving the reliability and scene consistency of the point cloud map. The dense point cloud map with removed point clouds possessing prior dynamic semantic information is then fused and updated with the point cloud map generated based on historical keyframes, thus constructing a semantic map that more closely matches the structure of the real static environment.

[0058] In the above embodiments, a dense point cloud map is generated based on the original image and static feature image. Based on the target dynamic region mask image, semantic information is assigned to the dense point cloud map and point clouds with prior dynamic semantic information are removed from the dense point cloud map to update the point cloud map, thereby further improving the positioning and mapping accuracy of the visual map data processing system in dynamic scenes.

[0059] In some implementations, the method may further include: when the static feature image is a keyframe, performing local map optimization based on the static feature image and the target dynamic region mask image to update the point cloud map.

[0060] Specifically, when the static feature image serves as the keyframe, bundle adjustment optimization is performed within a local sliding window based on the static feature image and multiple neighboring keyframes associated with it. This optimization process simultaneously corrects the pose of these keyframes and the 3D coordinates of all observed static map points. Meanwhile, to enhance the semantic expressiveness of the map, semantic information included in the target dynamic region mask image is used to assign semantic information to the corrected map points, achieving a simultaneous improvement in the geometric accuracy and semantic integrity of the local point cloud map. This results in a more interpretable semantic map suitable for high-level tasks such as path planning, human-computer interaction, or augmented reality.

[0061] In the above embodiments, when the static feature image is the keyframe, local map optimization is performed based on the static feature image and the target dynamic region mask image to update the point cloud map, thereby further improving the positioning and mapping accuracy of the visual map data processing system in dynamic scenes.

[0062] In some implementations, when the static feature image is a keyframe, local map optimization based on the static feature image and the target dynamic region mask image is performed to update the point cloud map. This may include: optimizing feature points in the local space based on the static feature image and the target dynamic region mask image, deleting redundant keyframes and low-quality map points and assigning semantic information to update the point cloud map.

[0063] Specifically, after identifying static feature images as keyframes, they are added to the map. The quality of map points is checked, and low-quality map points are removed. Further, multiple neighboring keyframes that are temporally adjacent and spatially co-located with the keyframe are selected to form a local temporal-spatial sliding window. The local mapping thread optimizes the static feature images based on this window using Local Bundle Adjustment (BA). This process aims to minimize reprojection errors, effectively suppressing accumulated errors and identifying and removing outliers to obtain a more accurate point cloud map. After completing the local BA optimization, redundant keyframes are further removed. Simultaneously, to enhance the semantic expressiveness of the map, semantic information included in the target dynamic region mask image is used to imbue the obtained more accurate point cloud map with semantic information, achieving a simultaneous improvement in the geometric accuracy and semantic integrity of the local point cloud map. This results in a more interpretable semantic map suitable for high-level tasks (such as path planning, human-computer interaction, or augmented reality).

[0064] It should be noted that this method also includes loop closure detection using a global map and keyframes. If a loop is detected, the cumulative error of the odometry is calculated by optimization, then all matched point clouds are aligned and duplicate points are merged. Finally, graph optimization is performed on the similarity constraints to achieve global consistency.

[0065] In the above embodiments, feature points in the local space are optimized based on static feature images and target dynamic region mask images. Redundant keyframes and low-quality map points are deleted and semantic information is assigned to update the point cloud map, thereby further improving the positioning and mapping accuracy of the visual map data processing system in dynamic scenes.

[0066] This specification provides an example of a visual map data processing method, applied to a system architecture based on the ORB-SLAM2 framework, by adding a dynamic region detection thread and a dense point cloud creation thread. Please refer to... Figure 5The process begins by acquiring the current image frame using a depth camera. A Panoptic FCN network is then used to perform panoramic semantic segmentation on the current image frame, yielding a panoramic segmentation image as the semantic result. A PWC-Net network is used to perform optical flow estimation on the current image frame, obtaining an optical flow image, which includes dynamic region optical flow. ORB feature points are extracted from the current image frame to generate a feature image. The panoramic segmentation image and the optical flow image are cross-validated to generate a target dynamic region mask image. Based on the target dynamic region mask image, a mask is overlaid at the corresponding positions of the target dynamic region mask area in the feature image to obtain a static feature image. Pose tracking is performed based on the static feature image to obtain the pose tracking result. If the pose tracking result is normal, local map tracking is performed based on the static feature image to obtain the tracking result. If the pose tracking result fails, relocalization tracking is performed based on the static feature image, followed by local map tracking to obtain the tracking result. Based on the tracking result and a preset keyframe selection strategy, it is determined whether the static feature image is a keyframe. Once the static feature image is determined to be a keyframe, it is sent to the local map optimization thread. This thread optimizes the pose and map point coordinates of the current frame and its co-view keyframes through methods such as local bundle adjustment, thereby updating the point cloud map. Simultaneously, the system continuously performs loop closure detection; upon identifying a loop closure candidate frame, it performs loop closure correction, eliminating accumulated errors through graph optimization to achieve global consistency.

[0067] A dense point cloud map is created based on the current image frame and static feature image. Then, based on the target dynamic region mask image, semantic information is added to the dense point cloud map, and point clouds with prior dynamic semantic information are removed from the dense point cloud map to update the point cloud map.

[0068] This specification provides a visual map data processing device 600. Please refer to [link / reference]. Figure 6 The visual map data processing device 600 includes: a panoramic semantic segmentation module 610, an optical flow estimation processing module 620, a static feature image acquisition module 630, and a point cloud map update module 640.

[0069] The panoramic semantic segmentation module 610 is used to perform panoramic semantic segmentation on the original image to generate a panoramic segmented image, wherein the panoramic segmented image includes semantic information of each pixel, and the semantic information includes prior dynamic semantic information. The optical flow estimation processing module 620 is used to perform optical flow estimation processing on the original image to obtain an optical flow image; The static feature image acquisition module 630 is used to obtain a static feature image based on the panoramic segmentation image, the optical flow image, and the original image; The point cloud map update module 640 is used to generate a dense point cloud map with semantic information based on the original image, the static feature image and the panoramic segmentation image, after motion tracking is performed based on the static feature image and the static feature image is determined to be a keyframe, and to remove point clouds with the prior dynamic semantic information from the dense point cloud map in order to update the point cloud map.

[0070] For a detailed description of the visual map data processing device, please refer to the description of the visual map data processing method above, which will not be repeated here.

[0071] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a visual map data processing method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0072] Those skilled in the art will understand that Figure 7 The structures shown are merely block diagrams of some structures related to the solutions disclosed in this specification, and do not constitute a limitation on the computer device to which the solutions disclosed in this specification are applied. Specifically, the computer device may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0073] In some embodiments, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps described above.

[0074] This specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in any of the above embodiments.

[0075] One embodiment of this specification provides a computer program product including instructions that, when executed by a processor of a computer device, enable the computer device to perform the steps of the method described in any of the above embodiments.

[0076] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). In addition, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning paper or other media, followed by editing, interpreting or otherwise processing as necessary, and then stored in computer memory.

Claims

1. A visual map data processing method, characterized in that, The method includes: The original image is subjected to panoramic semantic segmentation to generate a panoramic segmented image, wherein the panoramic segmented image includes semantic information of each pixel, and the semantic information includes prior dynamic semantic information; Optical flow estimation is performed on the original image to obtain an optical flow image; Based on the panoramic segmentation image, the optical flow image, and the original image, a static feature image is obtained; When motion tracking is performed based on the static feature image and the static feature image is determined to be a keyframe, a dense point cloud map with semantic information is generated based on the original image, the static feature image and the panoramic segmentation image, and point clouds with the prior dynamic semantic information are removed from the dense point cloud map to update the point cloud map.

2. The method according to claim 1, characterized in that, The process of obtaining a static feature image based on the panoramic segmentation image, the optical flow image, and the original image includes: Feature points are extracted from the original image to generate a feature image; Based on the mutual verification of the panoramic segmentation image and the optical flow image, a target dynamic region mask image is generated, wherein the target dynamic region mask image includes a target dynamic region mask region and the target dynamic region mask image includes semantic information of each pixel; Based on the target dynamic region mask image, a mask is overlaid at the corresponding position of the target dynamic region mask area in the feature image to obtain a static feature image.

3. The method according to claim 1, characterized in that, Motion tracking based on the static feature image and determining whether the static feature image is a keyframe includes: Pose tracking is performed based on the static feature image to obtain the pose tracking result; If the pose tracking result is normal, local map tracking is performed based on the static feature image to obtain the tracking result; If the pose tracking result fails, relocation tracking is performed based on the static feature image, followed by local map tracking, to obtain the tracking result. Based on the tracking results and the preset keyframe selection strategy, it is determined whether the static feature image is a keyframe.

4. The method according to claim 2, characterized in that, Based on the original image, the static feature image, and the panoramic segmentation image, a dense point cloud map with semantic information is generated, and point clouds with the prior dynamic semantic information are removed from the dense point cloud map to update the point cloud map, including: Based on the original image, the static feature image, and the target dynamic region mask image, a dense point cloud map with semantic information is generated, and point clouds with the prior dynamic semantic information are removed from the dense point cloud map to update the point cloud map.

5. The method according to claim 4, characterized in that, The step of generating a dense point cloud map with semantic information based on the original image, the static feature image, and the target dynamic region mask image, and then removing point clouds with the prior dynamic semantic information from the dense point cloud map to update the point cloud map, includes: A dense point cloud map is generated based on the original image and the static feature image. Based on the target dynamic region mask image, semantic information is assigned to the dense point cloud map and point clouds with the prior dynamic semantic information are removed from the dense point cloud map to update the point cloud map.

6. The method according to claim 2, characterized in that, The method further includes: When the static feature image is a keyframe, local map optimization is performed based on the static feature image and the target dynamic region mask image to update the point cloud map.

7. The method according to claim 6, characterized in that, When the static feature image is a keyframe, the step of performing local map optimization based on the static feature image and the target dynamic region mask image to update the point cloud map includes: Based on the static feature image and the target dynamic region mask image, the feature points in the local space are optimized, redundant keyframes and low-quality map points are deleted, and semantic information is assigned to update the point cloud map.

8. A visual map data processing device, characterized in that, The device includes: A panoramic semantic segmentation module is used to perform panoramic semantic segmentation on the original image to generate a panoramic segmented image, wherein the panoramic segmented image includes semantic information of each pixel, and the semantic information includes prior dynamic semantic information. The optical flow estimation processing module is used to perform optical flow estimation processing on the original image to obtain an optical flow image; A static feature image acquisition module is used to obtain a static feature image based on the panoramic segmentation image, the optical flow image, and the original image; The point cloud map update module is used to generate a dense point cloud map with semantic information based on the original image, the static feature image and the panoramic segmentation image, after motion tracking is performed based on the static feature image and the static feature image is determined to be a keyframe, and then remove point clouds with the prior dynamic semantic information from the dense point cloud map to update the point cloud map.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images