Visual navigation mapping method and device, computer equipment and storage medium
By using dynamically perceived Gaussian point selection and multi-scale scene modeling, the Gaussian scene model in 3DGS SLAM technology was optimized, solving the problems of high memory usage and missing or redundant scene details, and achieving efficient real-time visual navigation and mapping.
Patent Information
- Application Number
- CN202511058209.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-28
- Publication Date
- 2025-11-07
AI Technical Summary
In existing 3DGS SLAM technology, global dense Gaussian modeling leads to excessive memory usage, making it difficult to deploy on mobile devices. At the same time, fixed-scale Gaussian representation suffers from near-field detail loss and far-field redundancy in complex scenes.
By using a dynamic-aware Gaussian point selection, multi-scale scene modeling, and adaptive optimization framework, Gaussian points in the target dynamic and static regions are selected, multi-scale Gaussian modeling is performed, and the Gaussian scene model is optimized through dynamic-static partitioning Gaussian updates.
It improves positioning accuracy and mapping efficiency in dynamic scenarios, reduces resource consumption, and provides real-time visual navigation and mapping capabilities for embedded devices.
Smart Images
Figure CN120912704A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer vision, and particularly relates to a visual navigation mapping method and device, a computer device and a storage medium. BACKGROUND
[0002] 3DGSSLAM (3D Geometric-Semantic Simultaneous Localization and Mapping) technology is an important research direction in the field of robot navigation and environment perception in recent years. With the rapid development of applications such as autonomous driving, service robots and augmented reality, the demand for high-precision and high-robustness environment perception and positioning technology is growing.
[0003] In the existing 3DGS SLAM technology (such as SplaTAM), global dense Gaussian modeling leads to high memory occupation (>8GB), which is difficult to deploy mobile devices, and fixed scale Gaussian expression has the problems of near-field detail loss and far-field redundancy in complex scenes. SUMMARY
[0004] Therefore, the present application aims to overcome the deficiencies in the prior art and provide a visual navigation mapping method, device, computer device and storage medium, which uses dynamically perceived Gaussian point screening, multi-scale scene modeling and adaptive optimization framework to provide efficient real-time visual navigation and mapping capabilities for embedded devices.
[0005] The present application provides the following technical solutions: In a first aspect, the present application provides a visual navigation mapping method, comprising: obtaining an RGB image and a depth map of a current video frame, and obtaining original Gaussian points according to the RGB image and the depth map; screening target dynamic region Gaussian points and target static region Gaussian points from the original Gaussian points according to the RGB image; performing multi-scale Gaussian modeling on the target dynamic region Gaussian points and the target static region Gaussian points according to the depth map to obtain a hierarchical Gaussian scene model; performing dynamic and static partition Gaussian updating on the hierarchical Gaussian scene model by re-executing the steps of obtaining the RGB image and the depth map of the current video frame and obtaining the original Gaussian points according to the RGB image and the depth map to obtain an updated hierarchical Gaussian scene model; performing visual navigation mapping according to the hierarchical Gaussian scene model.
[0006] In an embodiment, the filtering the original Gaussian points according to the RGB image to obtain target dynamic region Gaussian points and target static region Gaussian points comprises: inputting the RGB image into a semantic segmentation network to obtain a probability map, the probability map comprising a probability of each pixel in the RGB image belonging to a dynamic object; filtering the original Gaussian points according to the probability map to obtain initial dynamic region Gaussian points and initial static region Gaussian points; filtering the initial dynamic region Gaussian points and the initial static region Gaussian points based on a preset confidence threshold to obtain the target dynamic region Gaussian points, the target static region Gaussian points and backup dynamic region Gaussian points.
[0007] In an embodiment, the filtering the original Gaussian points according to the probability map to obtain initial dynamic region Gaussian points and initial static Gaussian points comprises: performing binary processing on the probability map based on a preset probability threshold to obtain a dynamic object mask; filtering the original Gaussian points according to the dynamic object mask to obtain the initial dynamic region Gaussian points and the initial static Gaussian points.
[0008] In an embodiment, the preset confidence threshold comprises a first confidence threshold and a second confidence threshold, and the filtering the initial dynamic region Gaussian points and the initial static region Gaussian points based on the preset confidence threshold to obtain the target dynamic region Gaussian points, the target static region Gaussian points and backup dynamic region Gaussian points comprises: performing confidence decay on the initial dynamic Gaussian points to obtain candidate dynamic region Gaussian points; performing confidence enhancement on the initial static region Gaussian points to obtain candidate static region Gaussian points; if the confidence of the candidate dynamic region Gaussian points is higher than the first confidence threshold, the candidate dynamic region Gaussian points are taken as the target dynamic region Gaussian points; if the confidence of the candidate dynamic region Gaussian points is lower than or equal to the first confidence threshold, the candidate dynamic region Gaussian points are taken as the backup dynamic region Gaussian points; if the confidence of the candidate static region Gaussian points is higher than the second confidence threshold, the candidate static region Gaussian points are taken as the target static region Gaussian points; if the confidence of the candidate static region Gaussian points is lower than or equal to the second confidence threshold, the candidate static region Gaussian points are deleted.
[0009] In an embodiment, the multi-scale Gaussian modeling of the target dynamic region Gaussian point and the target static region Gaussian point according to the depth map obtains a hierarchical Gaussian scene model, and the hierarchical Gaussian scene model comprises: determining a near field region, a far field region and a middle field region according to the depth map; performing multi-scale Gaussian modeling of the near field region, the far field region and the middle field region according to the target dynamic region Gaussian point and the target static region Gaussian point respectively, to obtain the hierarchical Gaussian scene model.
[0010] In an embodiment, the dynamic and static partition Gaussian updating of the hierarchical Gaussian scene model by re-executing the steps of obtaining the RGB image and the depth map of the current video frame and obtaining the original Gaussian point according to the RGB image and the depth map of the next video frame obtains an updated hierarchical Gaussian scene model, and the dynamic and static partition Gaussian updating comprises: determining whether the backup dynamic region Gaussian point matches the next video frame; if yes, performing dynamic region Gaussian point updating of the hierarchical Gaussian scene model according to the Gaussian point corresponding to the next video frame in the backup dynamic region Gaussian point; if no, performing the steps of obtaining the RGB image and the depth map of the current video frame and obtaining the original Gaussian point according to the RGB image and the depth map to perform dynamic region Gaussian point updating of the hierarchical Gaussian scene model; determining a static region updating node based on a preset interval frame number, and re-executing the steps of obtaining the RGB image and the depth map of the current video frame and obtaining the original Gaussian point according to the RGB image and the depth map at the static region updating node to update the key frame static region in the hierarchical Gaussian scene model; taking the hierarchical Gaussian scene model after dynamic region Gaussian point updating and the hierarchical Gaussian scene model after key frame static region updating as the updated hierarchical Gaussian scene model.
[0011] In an embodiment, the method further comprises: obtaining an error gradient according to the RGB image corresponding to the next video frame and the plane projection of the hierarchical Gaussian scene model; determining a dynamic region learning rate of the hierarchical Gaussian scene model according to the error gradient; performing Gaussian point updating of the hierarchical Gaussian scene model based on the dynamic region learning rate.
[0012] In a second aspect, the application provides a visual navigation mapping device, comprising: The acquisition module is configured to acquire an RGB image and a depth map of a current video frame, and acquire original Gaussian points according to the RGB image and the depth map; The screening module is configured to screen target dynamic region Gaussian points and target static region Gaussian points from the original Gaussian points according to the RGB image; The construction module is configured to perform multi-scale Gaussian modeling on the target dynamic region Gaussian points and the target static region Gaussian points according to the depth map, and obtain a hierarchical Gaussian scene model; The updating module is configured to perform dynamic and static partition Gaussian updating on the hierarchical Gaussian scene model by re-executing the steps of acquiring the RGB image and the depth map of the current video frame, and acquiring original Gaussian points according to the RGB image and the depth map, to obtain an updated hierarchical Gaussian scene model; The mapping module is configured to perform visual navigation mapping according to the hierarchical Gaussian scene model.
[0013] In a third aspect, the present application provides a computer device, which comprises a memory and a processor, and the memory stores a computer program, and the computer program is executed by the processor to implement the visual navigation mapping method according to the first aspect.
[0014] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the visual navigation mapping method according to the first aspect.
[0015] The visual navigation mapping method, device, computer device and storage medium disclosed by the present application acquire an RGB image and a depth map of a current video frame, and acquire original Gaussian points according to the RGB image and the depth map; target dynamic region Gaussian points and target static region Gaussian points are screened from the original Gaussian points according to the RGB image; multi-scale Gaussian modeling is performed on the target dynamic region Gaussian points and the target static region Gaussian points according to the depth map, to obtain a hierarchical Gaussian scene model; dynamic and static partition Gaussian updating is performed on the hierarchical Gaussian scene model by re-executing the steps of acquiring the RGB image and the depth map of the current video frame, and acquiring original Gaussian points according to the RGB image and the depth map, to obtain an updated hierarchical Gaussian scene model; and visual navigation mapping is performed according to the hierarchical Gaussian scene model. In this way, through dynamic perception Gaussian point screening, multi-scale scene modeling and adaptive Gaussian updating, the positioning accuracy, mapping efficiency and resource utilization rate in a dynamic scene are improved, and reliable real-time visual navigation and mapping capabilities are provided for embedded application scenarios such as humanoid robots and unmanned vehicles. BRIEF DESCRIPTION OF DRAWINGS
[0016] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiments will be briefly introduced as follows. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the protection scope of the present application. In the various drawings, similar components are denoted by similar reference numerals.
[0017] Figure 1 A flowchart of the visual navigation mapping method proposed in the embodiment is shown; Figure 2 Another flowchart of the visual navigation mapping method proposed in the embodiment is shown; Figure 3 A structural diagram of the visual navigation mapping device proposed in the embodiment is shown.
[0018] Explanation of the drawing identifiers: 300 - visual navigation mapping device; 301 - acquisition module; 302 - screening module; 303 - construction module; 304 - update module; 305 - mapping module. DETAILED DESCRIPTION
[0019] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments.
[0020] The components of the embodiments of the present application generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.
[0021] In the following, the terms "include", "have", and their synonymous words used in various embodiments of the present application are only intended to represent a specific feature, number, step, operation, element, component, or combination of the foregoing, and should not be understood as first excluding the presence or possibility of adding one or more features, numbers, steps, operations, elements, components, or combinations of the foregoing.
[0022] In addition, the terms "first", "second", "third", and the like are only used for differentiation in description, and should not be understood as indicating or implying relative importance.
[0023] Unless specifically defined, all other terms used herein have the meanings as commonly understood by one of ordinary skill in the art to which various embodiments of the present disclosure belong. The terms used herein, such as those defined in generally used dictionaries, are to be interpreted as having the same meanings as those in the context of related technology and are not to be interpreted as having idealized or overly formal meanings, unless clearly defined in various embodiments of the present disclosure.
[0024] Embodiment 1 The embodiment of the present disclosure provides a visual navigation mapping method, which is used for implementing a dynamic perception 3D Gaussian spray SLAM system on an Orin platform, and utilizes a dynamic perception Gaussian point screening, a multi-scale scene modeling and an adaptive optimization framework to provide an embedded device with efficient real-time visual navigation and mapping capabilities.
[0025] Referring to Figure 1 The visual navigation mapping method includes steps S101-S105, and each step is described in detail below.
[0026] In step S101, an RGB image, a depth map and inertial data of a current video frame are acquired, and original Gaussian points are acquired according to the RGB image and the depth map.
[0027] In the embodiment, the RGB image, the depth map and the inertial data of the current video frame are collected in real time by an RGB-D camera and an IMU sensor, and the RGB image and the depth map are aligned and down-sampled, and the original Gaussian points are further acquired according to the RGB image and the depth map. The Gaussian point is a 3D space point described by a Gaussian distribution model, which is used to construct the three-dimensional structure of the scene. The RGB image provides color and dynamic information, and the depth map provides position and scale basis, which are used for initialization and updating of the Gaussian point.
[0028] In step S102, target dynamic region Gaussian points and target static region Gaussian points are screened from the original Gaussian points according to the RGB image.
[0029] In the embodiment, the target dynamic region Gaussian points and the target static region Gaussian points are screened from the original Gaussian points according to the RGB image. The target dynamic region Gaussian points and the target static region Gaussian points are Gaussian points with higher confidence, which are used for scene modeling.
[0030] Referring to Figure 2 In a specific embodiment, the preset confidence threshold includes a first confidence threshold and a second confidence threshold, and step S102 includes steps S1021-S1023, which are described in detail below.
[0031] Step S1021, input the RGB image into a semantic segmentation network to obtain a probability map, the probability map including probabilities of each pixel in the RGB image belonging to a dynamic object.
[0032] In this embodiment, the RGB image is input into a pre-trained semantic segmentation network to obtain a probability map including probabilities of each pixel in the RGB image belonging to a dynamic object, which is used to screen the dynamic region Gaussian points and the static region Gaussian points. The semantic segmentation network can be Mask R-CNN, DeepLab, etc.
[0033] Step S1022, screen the original Gaussian points according to the probability map to obtain initial dynamic region Gaussian points and initial static Gaussian points.
[0034] In this embodiment, the original Gaussian points are screened by using the dynamic object probability information reflected by the probability map to determine whether each original Gaussian point belongs to a dynamic region Gaussian point, thereby obtaining initial dynamic region Gaussian points and initial static Gaussian points.
[0035] In a specific embodiment, step S1022 includes: performing binaryzation processing on the probability map based on a preset probability threshold to obtain a dynamic object mask; and performing Gaussian point screening on the original Gaussian points according to the dynamic object mask to obtain the initial dynamic region Gaussian points and the initial static Gaussian points.
[0036] In this embodiment, the probability map is binaryzation processed based on a preset probability threshold to obtain a dynamic object mask, which is used to mark the dynamic region in the image and provide a basis for subsequent Gaussian point screening (distinguishing dynamic / static region Gaussian points) and confidence updating (dynamic region attenuation and static region enhancement) of dynamic perception.
[0037] Further, the dynamic object mask is matched with the projection of the original Gaussian points on the image plane to determine which Gaussian points fall into the dynamic region marked by the mask, thereby identifying the initial dynamic region Gaussian points and the initial static region Gaussian points.
[0038] Step S1023, screen the initial dynamic region Gaussian points and the initial static region Gaussian points based on a preset confidence threshold to obtain the target dynamic region Gaussian points, the target static region Gaussian points, and the backup dynamic region Gaussian points.
[0039] In this embodiment, the initial dynamic region Gaussian points and the initial static region Gaussian points are confidence-screened by using a preset confidence threshold to obtain target dynamic region Gaussian points and target static region Gaussian points with higher confidence, and backup dynamic region Gaussian points with lower confidence, and the initial static region Gaussian points with lower confidence are directly deleted.
[0040] In a specific embodiment, the preset confidence threshold comprises a first confidence threshold and a second confidence threshold, and step S1023 comprises: confidence attenuation of the initial dynamic Gaussian point to obtain a candidate dynamic region Gaussian point; confidence enhancement of the initial static region Gaussian point to obtain a candidate static region Gaussian point; if the confidence of the candidate dynamic region Gaussian point is higher than the first confidence threshold, the candidate dynamic region Gaussian point is taken as the target dynamic region Gaussian point; if the confidence of the candidate dynamic region Gaussian point is lower than or equal to the first confidence threshold, the candidate dynamic region Gaussian point is taken as the backup dynamic region Gaussian point; if the confidence of the candidate static region Gaussian point is higher than the second confidence threshold, the candidate static region Gaussian point is taken as the target static region Gaussian point; if the confidence of the candidate static region Gaussian point is lower than or equal to the second confidence threshold, the candidate static region Gaussian point is deleted.
[0041] In the embodiment, the initial dynamic Gaussian point is subjected to confidence attenuation based on a preset attenuation rule to obtain a candidate dynamic region Gaussian point. The preset attenuation rule is, for example, linear attenuation, exponential attenuation or the like based on a timestamp or the number of observations.
[0042] Meanwhile, the initial static region Gaussian point is subjected to confidence enhancement by using a co-view to obtain a candidate static region Gaussian point. The co-view is a correlation graph reflecting the observation overlap relationship of the same space region (or Gaussian point) among multiple frames of images. Through the co-view, it can be known that the static region Gaussian point is commonly observed by multiple frames. The more the number of observations, the higher the confidence according to the rule, so as to enhance the confidence.
[0043] Further, the first confidence threshold is lower than the second confidence threshold, the second confidence threshold is used to mark a Gaussian point that stably exists, and the first confidence threshold is used to determine a Gaussian point that needs to be eliminated. If the confidence of the candidate dynamic region Gaussian point is higher than the first confidence threshold, the candidate dynamic region Gaussian point is taken as the target dynamic region Gaussian point; if the confidence of the candidate dynamic region Gaussian point is lower than or equal to the first confidence threshold, the candidate dynamic region Gaussian point is taken as the backup dynamic region Gaussian point, and if the backup dynamic region Gaussian point is not reactivated subsequently, it is finally eliminated, so as to realize confidence attenuation and life cycle management of the dynamic region Gaussian point; if the confidence of the candidate static region Gaussian point is higher than the second confidence threshold, the candidate static region Gaussian point is taken as the target static region Gaussian point; if the confidence of the candidate static region Gaussian point is lower than or equal to the second confidence threshold, the candidate static region Gaussian point is deleted.
[0044] Step S103: multi-scale Gaussian modeling of the target dynamic region Gaussian point and the target static region Gaussian point according to the depth map to obtain a hierarchical Gaussian scene model.
[0045] In the embodiment, the scene is divided into three regions of near field, middle field and far field according to the depth map, and the target dynamic region Gaussian points and the target static region Gaussian points are modeled by multi-scale Gauss according to the three regions, so as to obtain a hierarchical Gaussian scene model, and then solve the problems of missing details in the near field and redundancy in the far field existing in the fixed-scale Gaussian expression in a complex scene.
[0046] In a specific embodiment, step S103 comprises: determining a near field region, a far field region and a middle field region according to the depth map; and modeling the near field region, the far field region and the middle field region by multi-scale Gauss according to the target dynamic region Gaussian points and the target static region Gaussian points respectively, to obtain the hierarchical Gaussian scene model.
[0047] In the embodiment, the near field region, the far field region and the middle field region are determined according to the depth map; and the near field region, the far field region and the middle field region are modeled by multi-scale Gauss according to the target dynamic region Gaussian points and the target static region Gaussian points respectively, to obtain the hierarchical Gaussian scene model.
[0048] Specifically, the near field region (0-5m) extracts the RGB color information and the depth data of the region, initializes a large number of target dynamic region Gaussian points and target static region Gaussian points according to a density greater than 100 points / m³, the position of each Gaussian point is calculated by the depth data, and the color is assigned by the RGB image pixel. Each target dynamic region Gaussian point and target static region Gaussian point is assigned an anisotropic covariance matrix, and the matrix parameters are dynamically adjusted by fitting the details such as surface texture and edge of the near field object (such as expanding the scale along the surface direction and reducing the scale in the vertical direction.
[0049] The middle field region adopts a moderate distribution of about 50 points / m³, is simplified to a semi-anisotropic or approximately spherical covariance matrix, and the attribute update frequency is reduced.
[0050] The far field region initializes Gaussian points according to a density less than 10 points / m³, uses a spherical covariance instead of a complex matrix, and reduces the update frequency. After reducing the memory occupation, a sparse Gaussian point set reflecting the overall structure of the far field (such as the outline of a distant building) is obtained, which meets the demand of navigation on the general geometric relationship of the far field.
[0051] Step S104, performing dynamic and static partition Gaussian update on the hierarchical Gaussian scene model by re-executing the steps of obtaining the RGB image and the depth map of the next video frame and obtaining the original Gaussian points according to the RGB image and the depth map, to obtain an updated hierarchical Gaussian scene model.
[0052] In the embodiment, the Gaussian point information of the dynamic region and the static region is continuously updated according to the next video frame based on the multi-frame observation data, for example, the position, the color and the covariance matrix, so as to obtain the updated hierarchical Gaussian scene model, ensure the consistency of the detail features, obtain the high-density Gaussian point set which can accurately reflect the geometric shape, the surface details and the color change of the scene object, and realize the fine three-dimensional reconstruction and realistic rendering of the scene.
[0053] In a specific embodiment, the step S105 comprises: judging whether the backup dynamic region Gaussian point matches the next video frame; if yes, performing dynamic region Gaussian point update on the hierarchical Gaussian scene model according to the Gaussian point corresponding to the next video frame in the backup dynamic region Gaussian point; if no, performing the steps of obtaining the RGB image and the depth map of the current video frame, and obtaining the original Gaussian point according to the RGB image and the depth map, and performing dynamic region Gaussian point update on the hierarchical Gaussian scene model; determining a static region update node based on a preset interval frame number, and at the static region update node, re-performing the steps of obtaining the RGB image and the depth map of the current video frame, and obtaining the original Gaussian point according to the RGB image and the depth map, to update the key frame static region in the hierarchical Gaussian scene model; taking the hierarchical Gaussian scene model after dynamic region Gaussian point update and the hierarchical Gaussian scene model after key frame static region update as the updated hierarchical Gaussian scene model.
[0054] In the embodiment, the CPU thread processes the static region parameter update, and the GPU CUDA kernel function accelerates the dynamic region optimization. In this embodiment, the developer is allowed to define a GPU CUDA kernel function to specify the parameter update task to be executed in parallel.
[0055] The method based on image feature matching is used to judge whether the backup dynamic region Gaussian point matches the next video frame, that is, whether the backup dynamic region Gaussian point is perceived by the next video frame. Specifically, the feature points extracted from the next video frame are matched with the feature points in the corresponding region of the backup dynamic region Gaussian point in the temporary storage pool using a feature matching algorithm; and a matching degree index is calculated according to the matching result, such as the number of matching points and the average distance between the matching points. If the number of matching points exceeds a certain threshold value and the average distance is less than a set value, it is considered that the region corresponding to the dynamic Gaussian point in the temporary storage pool is perceived again in the subsequent frame.
[0056] If yes, the confidence of the backup dynamic region Gaussian point is recalculated according to the next video frame, and when the confidence rises above the first confidence threshold, the backup dynamic region Gaussian point is activated to update the hierarchical Gaussian scene model.
[0057] If the region corresponding to the backup dynamic region Gaussian point is not re-perceived, step S101 is continued to update the dynamic region Gaussian point of the hierarchical Gaussian scene model.
[0058] Meanwhile, a low-frequency update strategy is implemented for the static region. Specifically, a static region update node is determined based on a preset interval frame number, and at the static region update node, step S101 is re-executed on the next video frame to update the key frame static region in the hierarchical Gaussian scene model, so that the calculation overhead is reduced, and a static scene Gaussian point set with a preserved core structure and stable precision is obtained, meeting the demand of long-term navigation for consistency of the static environment.
[0059] Finally, the hierarchical Gaussian scene model after the dynamic region Gaussian point is updated and the hierarchical Gaussian scene model after the key frame static region is updated are taken as the updated hierarchical Gaussian scene model.
[0060] Step S105, visual navigation mapping is performed according to the hierarchical Gaussian scene model.
[0061] In this embodiment, the hierarchical Gaussian scene model is used as the environment representation basis, and the output camera pose is combined to perform visual navigation mapping, so as to realize autonomous positioning, path planning and scene understanding of embedded devices such as robots and unmanned vehicles.
[0062] In a specific embodiment, the method further includes: obtaining an error gradient according to a plane projection of an RGB image corresponding to the next video frame and the hierarchical Gaussian scene model; determining a dynamic region learning rate of the hierarchical Gaussian scene model according to the error gradient; and updating a Gaussian point of the hierarchical Gaussian scene model based on the dynamic region learning rate.
[0063] In the embodiment, an error gradient is obtained according to a plane projection of an RGB image corresponding to a next video frame and a hierarchical Gaussian scene model; a dynamic region learning rate of the hierarchical Gaussian scene model is adaptively adjusted according to the error gradient; and a Gaussian point of the hierarchical Gaussian scene model is updated based on the dynamic region learning rate, so as to realize fast updating of the Gaussian point position and covariance by using the dynamic region learning rate, and to adaptively improve the updating speed of the hierarchical Gaussian scene model. The learning rate is improved more significantly as the error gradient is larger, thereby maximizing the hardware resource utilization rate and realizing efficient and accurate real-time visual navigation and mapping.
[0064] The visual navigation and mapping method provided in the embodiment acquires an RGB image and a depth map of a current video frame, and acquires original Gaussian points according to the RGB image and the depth map; target dynamic region Gaussian points and target static region Gaussian points are screened from the original Gaussian points according to the RGB image; a hierarchical Gaussian scene model is obtained by performing multi-scale Gaussian modeling on the target dynamic region Gaussian points and the target static region Gaussian points according to the depth map; the hierarchical Gaussian scene model is updated by re-executing the steps of acquiring the RGB image and the depth map of the current video frame and acquiring the original Gaussian points according to the RGB image and the depth map, to obtain an updated hierarchical Gaussian scene model; and visual navigation and mapping are performed according to the hierarchical Gaussian scene model. In this way, the positioning accuracy, mapping efficiency and resource utilization rate in a dynamic scene are improved by dynamic perception-based Gaussian point screening, multi-scale scene modeling and Gaussian updating, thereby providing reliable real-time visual navigation and mapping capability for embedded application scenarios such as humanoid robots and unmanned vehicles.
[0065] Embodiment 2 In addition, the embodiment of the present disclosure provides a visual navigation and mapping device 300, please see Figure 3 , comprising: An acquisition module 301 is configured to acquire an RGB image and a depth map of a current video frame, and acquire original Gaussian points according to the RGB image and the depth map; A screening module 302 is configured to screen target dynamic region Gaussian points and target static region Gaussian points from the original Gaussian points according to the RGB image; A construction module 303 is configured to perform multi-scale Gaussian modeling on the target dynamic region Gaussian points and the target static region Gaussian points according to the depth map, to obtain a hierarchical Gaussian scene model; An updating module 304 is configured to perform dynamic and static partition Gaussian updating on the hierarchical Gaussian scene model by re-executing the steps of acquiring the RGB image and the depth map of the current video frame and acquiring the original Gaussian points according to the RGB image and the depth map, to obtain an updated hierarchical Gaussian scene model. The mapping module 305 performs visual navigation mapping according to the hierarchical Gaussian scene model.
[0066] Optionally, the screening module 302 is further configured to input the RGB image into a semantic segmentation network to obtain a probability map, the probability map including probabilities of each pixel in the RGB image belonging to a dynamic object; screen the original Gaussian points according to the probability map to obtain initial dynamic region Gaussian points and initial static region Gaussian points; and screen the initial dynamic region Gaussian points and the initial static region Gaussian points based on a preset confidence threshold to obtain the target dynamic region Gaussian points, the target static region Gaussian points, and backup dynamic region Gaussian points.
[0067] Optionally, the screening module 302 is further configured to perform binaryzation processing on the probability map based on a preset probability threshold to obtain a dynamic object mask; and perform Gaussian point screening on the original Gaussian points according to the dynamic object mask to obtain the initial dynamic region Gaussian points and the initial static Gaussian points.
[0068] Optionally, the preset confidence threshold includes a first confidence threshold and a second confidence threshold, and the screening module 302 is further configured to perform confidence decay on the initial dynamic Gaussian points to obtain candidate dynamic region Gaussian points; perform confidence enhancement on the initial static region Gaussian points to obtain candidate static region Gaussian points; if a confidence of the candidate dynamic region Gaussian points is higher than the first confidence threshold, the candidate dynamic region Gaussian points are taken as the target dynamic region Gaussian points; if the confidence of the candidate dynamic region Gaussian points is lower than or equal to the first confidence threshold, the candidate dynamic region Gaussian points are taken as the backup dynamic region Gaussian points; if a confidence of the candidate static region Gaussian points is higher than the second confidence threshold, the candidate static region Gaussian points are taken as the target static region Gaussian points; and if the confidence of the candidate static region Gaussian points is lower than or equal to the second confidence threshold, the candidate static region Gaussian points are deleted.
[0069] Optionally, the construction module 303 is further configured to determine a near-field region, a far-field region, and a middle-field region according to the depth map; and perform multi-scale Gaussian modeling on the near-field region, the far-field region, and the middle-field region according to the target dynamic region Gaussian points and the target static region Gaussian points to obtain the hierarchical Gaussian scene model.
[0070] Optionally, the updating module 304 is further configured to determine whether the next video frame corresponds to a Gaussian point in the backup dynamic region Gaussian point; if yes, perform dynamic region Gaussian point updating on the hierarchical Gaussian scene model according to the Gaussian point corresponding to the next video frame in the backup dynamic region Gaussian point; if no, perform dynamic region Gaussian point updating on the hierarchical Gaussian scene model by re-executing the steps of obtaining the RGB image and the depth map of the current video frame and obtaining the original Gaussian point according to the RGB image and the depth map; determine a static region updating node based on a preset interval frame number, and at the static region updating node, re-execute the steps of obtaining the RGB image and the depth map of the current video frame and obtaining the original Gaussian point according to the RGB image and the depth map to update the key frame static region in the hierarchical Gaussian scene model; and take the hierarchical Gaussian scene model after dynamic region Gaussian point updating and the hierarchical Gaussian scene model after key frame static region updating as the updated hierarchical Gaussian scene model.
[0071] Optionally, the updating module 304 is further configured to obtain an error gradient based on a plane projection of the next video frame and the hierarchical Gaussian scene model; determine a dynamic region learning rate of the hierarchical Gaussian scene model according to the error gradient; and perform Gaussian point updating on the hierarchical Gaussian scene model based on the dynamic region learning rate.
[0072] The device provided in the embodiments of the present disclosure can perform the steps of the visual navigation mapping method provided in Embodiment 1, and thus will not be described in detail.
[0073] The visual navigation mapping device provided in the present embodiment obtains the RGB image and the depth map of the current video frame, and obtains the original Gaussian point according to the RGB image and the depth map; filters target dynamic region Gaussian points and target static region Gaussian points from the original Gaussian points according to the RGB image; performs multi-scale Gaussian modeling on the target dynamic region Gaussian points and the target static region Gaussian points according to the depth map to obtain a hierarchical Gaussian scene model; performs dynamic and static partition Gaussian updating on the hierarchical Gaussian scene model by re-executing the steps of obtaining the RGB image and the depth map of the current video frame and obtaining the original Gaussian point according to the RGB image and the depth map to obtain an updated hierarchical Gaussian scene model; and performs visual navigation mapping according to the hierarchical Gaussian scene model. In this way, the positioning accuracy, the mapping efficiency and the resource utilization rate in a dynamic scene are improved through dynamic perception Gaussian point filtering, multi-scale scene modeling and Gaussian updating, and reliable real-time visual navigation and mapping capabilities are provided for embedded application scenarios such as humanoid robots and unmanned vehicles.
[0074] Embodiment 3 In addition, the disclosure embodiments provide a computer device, comprising a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to implement the visual navigation mapping method in embodiment 1.
[0075] The device provided by the embodiments of the disclosure can execute the steps of the visual navigation mapping method provided in embodiment 1, and details are not repeated.
[0076] Embodiment 4 The disclosure embodiments provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the visual navigation mapping method in embodiment 1.
[0077] In the embodiment, the computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0078] The computer readable storage medium provided by the embodiments can implement the visual navigation mapping method provided in embodiment 1, and details are not repeated here.
[0079] In all examples shown and described herein, any specific values should be interpreted as merely exemplary and not as a limitation, and thus other examples of the example embodiments can have different values.
[0080] It should be noted that: similar labels and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.
[0081] The above-described embodiments only express several embodiments of the present application, which are described in detail and specifically, but should not be understood as limiting the scope of the present application. It should be noted that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are within the scope of protection of the present application.
Claims
1. A visual navigation mapping method, characterized in that, The method comprises the following steps: obtaining an RGB image and a depth image of a current video frame, and obtaining original Gaussian points according to the RGB image and the depth image; screening target dynamic region Gaussian points and target static region Gaussian points from the original Gaussian points according to the RGB image; performing multi-scale Gaussian modeling on the target dynamic region Gaussian points and the target static region Gaussian points according to the depth image to obtain a hierarchical Gaussian scene model; performing dynamic and static partition Gaussian updating on the hierarchical Gaussian scene model by re-executing the steps of obtaining the RGB image and the depth image of the current video frame and obtaining the original Gaussian points according to the RGB image and the depth image to obtain an updated hierarchical Gaussian scene model; performing visual navigation mapping according to the hierarchical Gaussian scene model.
2. The visually navigated mapping method of claim 1, wherein, The step of screening target dynamic region Gaussian points and target static region Gaussian points from the original Gaussian points according to the RGB image comprises the following steps: inputting the RGB image into a semantic segmentation network to obtain a probability map, wherein the probability map comprises the probability of each pixel in the RGB image belonging to a dynamic object; screening the original Gaussian points according to the probability map to obtain initial dynamic region Gaussian points and initial static region Gaussian points; screening the initial dynamic region Gaussian points and the initial static region Gaussian points based on a preset confidence threshold to obtain the target dynamic region Gaussian points, the target static region Gaussian points and backup dynamic region Gaussian points.
3. The visually navigated mapping method of claim 2, wherein, The step of screening the original Gaussian points according to the probability map to obtain initial dynamic region Gaussian points and initial static Gaussian points comprises the following steps: performing binary processing on the probability map based on a preset probability threshold to obtain a dynamic object mask; screening the original Gaussian points according to the dynamic object mask to obtain the initial dynamic region Gaussian points and the initial static Gaussian points.
4. The visually navigated mapping method of claim 2, wherein, The preset confidence threshold comprises a first confidence threshold and a second confidence threshold, and the step of screening the initial dynamic region Gaussian points and the initial static region Gaussian points based on the preset confidence threshold to obtain the target dynamic region Gaussian points, the target static region Gaussian points and backup dynamic region Gaussian points comprises the following steps: performing confidence decay on the initial dynamic Gaussian points to obtain candidate dynamic region Gaussian points; performing confidence enhancement on the initial static region Gaussian points to obtain candidate static region Gaussian points; if the confidence of the candidate dynamic region Gaussian points is higher than the first confidence threshold, the candidate dynamic region Gaussian points are taken as the target dynamic region Gaussian points; if the confidence of the candidate dynamic region Gaussian points is lower than or equal to the first confidence threshold, the candidate dynamic region Gaussian points are taken as the backup dynamic region Gaussian points; if the confidence of the candidate static region Gaussian points is higher than the second confidence threshold, the candidate static region Gaussian points are taken as the target static region Gaussian points; if the confidence of the candidate static region Gaussian points is lower than or equal to the second confidence threshold, the candidate static region Gaussian points are deleted.
5. The visually navigated mapping method of claim 1, wherein, The multi-scale Gaussian modeling of the target dynamic region Gaussian point and the target static region Gaussian point according to the depth map comprises: determining a near-field region, a far-field region and a middle-field region according to the depth map; performing multi-scale Gaussian modeling on the near-field region, the far-field region and the middle-field region according to the target dynamic region Gaussian point and the target static region Gaussian point respectively, to obtain the hierarchical Gaussian scene model.
6. The visually navigated mapping method of claim 2, wherein, The dynamic and static partition Gaussian updating of the hierarchical Gaussian scene model by re-executing the steps of acquiring the RGB image and the depth map of the current video frame and acquiring the original Gaussian point according to the RGB image and the depth map on the next video frame comprises: determining whether the backup dynamic region Gaussian point matches the next video frame; if yes, performing dynamic region Gaussian point updating on the hierarchical Gaussian scene model according to the Gaussian point corresponding to the next video frame in the backup dynamic region Gaussian point; if no, performing dynamic region Gaussian point updating on the hierarchical Gaussian scene model by executing the steps of acquiring the RGB image and the depth map of the current video frame and acquiring the original Gaussian point according to the RGB image and the depth map; determining a static region updating node based on a preset interval frame number, and re-executing the steps of acquiring the RGB image and the depth map of the current video frame and acquiring the original Gaussian point according to the RGB image and the depth map on the next video frame at the static region updating node to update the key frame static region in the hierarchical Gaussian scene model; taking the hierarchical Gaussian scene model after dynamic region Gaussian point updating and the hierarchical Gaussian scene model after key frame static region updating as the updated hierarchical Gaussian scene model.
7. The visually navigated mapping method of claim 1, wherein, The method further comprises: obtaining an error gradient according to the RGB image corresponding to the next video frame and the plane projection of the hierarchical Gaussian scene model; determining a dynamic region learning rate of the hierarchical Gaussian scene model according to the error gradient; performing Gaussian point updating on the hierarchical Gaussian scene model based on the dynamic region learning rate.
8. A visual navigation mapping device, characterized by The method further comprises: an acquiring module configured to acquire an RGB image and a depth map of a current video frame and acquire original Gaussian points according to the RGB image and the depth map; a screening module configured to screen target dynamic region Gaussian points and target static region Gaussian points from the original Gaussian points according to the RGB image; a constructing module configured to perform multi-scale Gaussian modeling on the target dynamic region Gaussian points and the target static region Gaussian points according to the depth map, to obtain a hierarchical Gaussian scene model; an updating module configured to perform dynamic and static partition Gaussian updating on the hierarchical Gaussian scene model by re-executing the steps of acquiring the RGB image and the depth map of the current video frame and acquiring the original Gaussian point according to the RGB image and the depth map on the next video frame, to obtain an updated hierarchical Gaussian scene model; a mapping module configured to perform visual navigation mapping according to the hierarchical Gaussian scene model.
9. A computer device, comprising: A computer program product comprising a computer readable medium storing a computer program, the computer program being loadable into an internal memory of a computer and comprising instructions that, when executed by the computer, cause the computer to perform the method of any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, A computer program product comprising a computer readable medium storing a computer program, the computer program being loadable into an internal memory of a computer and comprising instructions that, when executed by the computer, cause the computer to perform the method of any one of claims 1 to 7.