NeRF map construction method and device based on visual slam

Abstract

The application relates to the technical field of computer vision, in particular to a NeRF map construction method and device based on visual SLAM, wherein the method comprises the following steps: in a dynamic scene, acquiring a static target feature point of a current frame image of a camera; tracking the static target feature point, solving a current frame camera pose of the camera, and selecting a key frame according to the number of projection inliers in a tracking process and tracking time; sampling pixel points in the key frame according to ORB features, inputting a key frame sampling result into a pre-established NeRF map model for training, obtaining NeRF map model parameters, and obtaining a NeRF map construction result. The application can generate a dense NeRF map in real time for a dynamic scene, overcome the problem of false matching in the dynamic scene, maintain simplicity with high efficiency, eliminate the influence of dynamic targets, complete high-quality background reconstruction, and effectively meet the positioning and mapping application requirements in different scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a NeRF map construction method and device based on visual SLAM. BACKGROUND

[0002] In the related art, the sensor of the traditional SLAM scheme is mainly a laser radar, but the laser radar is high in cost, and the point cloud information can only reflect the distance and angle of the surrounding object points, discarding the rich information such as texture and semantics of the surrounding environment, and the interaction with people is poor. Therefore, the camera gradually becomes the mainstream sensor of the SLAM scheme in recent years, and the SLAM scheme using the camera as the main sensor is called visual SLAM. Some researches use a deep learning network to detect potential moving targets in images to remove potential dynamic feature points, thereby improving the running ability of the visual SLAM system in a dynamic scene. However, this method based on the deep learning network removes all objects that may move in the image, even if these objects do not actually move (such as stationary vehicles and humans, etc.), and when there are some non-potential moving targets that are forced to move (such as books and cabinets that are moved, etc.), the deep network cannot remove them. In addition, the visual SLAM system also faces the challenge of constructing a high-fidelity dense map in a dynamic scene in real time to adapt to the complex task requirements of an unmanned system. At present, some traditional visual SLAM methods can complete real-time dense mapping using an RGB-D camera or with the aid of GPU acceleration.

[0003] However, when the visual SLAM method in the related art performs real-time dense mapping, since the point cloud or octree is used to represent the map, it is impossible to make a reasonable prediction for the unobserved or occluded part of the scene by a moving object, and it is also impossible to use the subsequent observation information of the system to optimize the part of the map that has been constructed, resulting in a large number of holes and missing parts in the reconstruction result, so that high-fidelity real-time dense reconstruction cannot be achieved, and the applicability in actual scenes is not strong, which needs to be solved urgently. SUMMARY

[0004] The present application provides a NeRF map construction method and device based on visual SLAM to solve the problems that the visual SLAM method in the related art cannot make a reasonable prediction for the unobserved or occluded part of the scene by a moving object when performing real-time dense mapping, and cannot use the subsequent observation information of the system to optimize the part of the map that has been constructed, resulting in a large number of holes and missing parts in the reconstruction result, so that high-fidelity real-time dense reconstruction cannot be achieved, and the applicability in actual scenes is not strong.

[0005] The first aspect embodiment of the application provides a NeRF map construction method based on visual SLAM, comprising the following steps: in a dynamic scene, obtaining static target feature points of a current frame image of a camera; tracking the static target feature points, so as to solve a current frame camera pose of the camera according to the static target feature points in the tracking process, obtain a number of projection points and a tracking time, and select a first key frame according to the number of projection points and the tracking time; performing pixel point sampling according to ORB feature guidance in the first key frame to obtain a first key frame sampling result, inputting the first key frame sampling result into a pre-established NeRF map model for training, obtaining NeRF map model parameters, and obtaining a NeRF map construction result.

[0006] Optionally, in an embodiment of the application, the obtaining of the static target feature points of the current frame image of the camera comprises: performing instance segmentation on the current frame image according to a configuration file to obtain a target instance segmentation mask, dividing the target instance segmentation mask into a dynamic instance segmentation mask and a static instance segmentation mask; calculating dynamic scores of objects represented by the dynamic instance segmentation mask and the static instance segmentation mask based on a target geometric constraint condition, determining static target feature points of the current frame image based on the dynamic scores and an empirical threshold.

[0007] Optionally, in an embodiment of the application, before the instance segmentation on the current frame image according to the configuration file, the method comprises: performing instance segmentation on a predefined instance according to a predefined target category to obtain a predefined dynamic instance and a predefined static instance; storing detection confidence and corresponding semantic color of the predefined dynamic instance and the predefined static instance in a file to obtain the configuration file.

[0008] Optionally, in an embodiment of the application, the method further comprises: selecting a second key frame in a historical key frame set; performing pixel point sampling in a static area of the second key frame to obtain a second sampling result; inputting the second sampling result into the pre-established NeRF map model for training.

[0009] The second aspect embodiment of the application provides a device for constructing a NeRF map based on visual SLAM, comprising: an acquisition module configured to acquire static target feature points of a current frame image of a camera in a dynamic scene; a first selection module configured to track the static target feature points, so as to solve a current frame camera pose of the camera according to the static target feature points in the tracking process, obtain a number of projection points and a tracking time, and select a first key frame according to the number of projection points and the tracking time; and a construction module configured to sample pixel points in the first key frame, obtain a first sampling result, and input the first sampling result into a pre-established NeRF map model for training, so as to obtain a NeRF map construction result.

[0010] Optionally, in an embodiment of the application, the acquisition module comprises: a segmentation unit configured to perform instance segmentation on the current frame image according to a configuration file, to obtain a target instance segmentation mask, and to divide the target instance segmentation mask into a dynamic instance segmentation mask and a static instance segmentation mask; and a determination unit configured to calculate a dynamic score of an object represented by the dynamic instance segmentation mask and the static instance segmentation mask based on a target geometric constraint condition, to determine static target feature points of the current frame image based on the dynamic score and an empirical threshold.

[0011] Optionally, in an embodiment of the application, the device further comprises: a segmentation module configured to perform instance segmentation on a predefined instance according to a predefined target category before performing instance segmentation on the current frame image according to the configuration file, to obtain a predefined dynamic instance and a predefined static instance; and a storage module configured to store a detection confidence and a corresponding semantic color of the predefined dynamic instance and the predefined static instance in a file, to obtain the configuration file.

[0012] Optionally, in an embodiment of the application, the device further comprises: a second selection module configured to select a second key frame from a set of historical key frames; a sampling module configured to sample pixel points in a static region of the second key frame, to obtain a second sampling result; and a training module configured to input the second sampling result into the pre-established NeRF map model for training.

[0013] The third aspect embodiment of the application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for constructing a NeRF map based on visual SLAM as described in the above embodiments.

[0014] The fourth aspect of the application provides a computer readable storage medium storing a computer program, which, when executed by a processor, implements the above method for constructing a NeRF map based on visual SLAM.

[0015] The fifth aspect of the application provides a computer program product comprising a computer program, which, when executed, implements the above method for constructing a NeRF map based on visual SLAM.

[0016] The embodiments of the application can overcome the problem of false matching in a dynamic scene based on the dynamic point rejection method of instance segmentation and geometric constraint. A dense NeRF map is generated in real time for a dynamic scene. Thus, the problem of decreased positioning accuracy of existing SLAM technology in a dynamic scene is solved. The dynamic point rejection of instance segmentation and geometric constraint is performed using the image characteristics in a dynamic scene, which is efficient while maintaining the simplicity of practical applications. The map can eliminate the influence of dynamic targets and complete high-quality background reconstruction, achieving good performance in a dynamic scene and effectively meeting the positioning and mapping application requirements in different scenes. In related technologies, when a visual SLAM method performs real-time dense mapping, since a point cloud or octree is used to represent the map, it cannot make reasonable predictions for unobserved or occluded parts in the scene, and cannot use subsequent observation information of the system to optimize the built part, resulting in a large number of holes and missing parts in the reconstruction result, thus it cannot achieve high-fidelity real-time dense reconstruction, and has poor applicability in actual scenes.

[0017] Additional aspects and advantages of the application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and / or additional aspects and advantages of the application will become apparent and be readily appreciated from the following description, including the appended drawings, wherein:

[0019] Figure 1 A flowchart of a method for constructing a NeRF map based on visual SLAM according to an embodiment of the application;

[0020] Figure 2 A schematic diagram of a dense NeRF map constructed in real time in dynamic sequences fr3-w-xyz and fr2-desk-with-person for an embodiment of the application;

[0021] Figure 3 A flowchart of a method for constructing a NeRF map based on visual SLAM according to an embodiment of the application;

[0022] Figure 4 FIG. 1 is a structural schematic diagram of a visual SLAM-based NeRF map construction device according to an embodiment of the present application;

[0023] Figure 5 FIG. 2 is a structural schematic diagram of an electronic device according to an embodiment of the present application.

[0024] Reference signs:

[0025] 10-visual SLAM-based NeRF map construction device: 100-acquisition module, 200-first selection module, and 300-construction module; 501-memory, 502-processor, and 503-communication interface. DETAILED DESCRIPTION

[0026] Embodiments of the present application are described in detail below with reference to the accompanying drawings, examples of which are shown in the drawings, in which the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0027] A visual SLAM-based NeRF map construction method and device of an embodiment of the present application are described below with reference to the accompanying drawings. In view of the problem that, when a visual SLAM method in related technologies mentioned in the background art above performs real-time dense mapping, since a point cloud or an octree is used to represent a map, a reasonable prediction cannot be made for unobserved or occluded parts in a scene by a moving object, and the observed information of the system later cannot be used to optimize the mapped parts, resulting in a large number of holes and missing parts in the reconstruction result, thus high-fidelity real-time dense reconstruction cannot be achieved, and the applicability in actual scenes is not strong, the present application provides a visual SLAM-based NeRF map construction method, in which, based on a dynamic point rejection method of instance segmentation and geometric constraint, the problem of false matching in a dynamic scene can be overcome. A dense NeRF map is generated in real time for a dynamic scene. Thus, the problem of decreased positioning accuracy of the existing SLAM technology in a dynamic scene is solved, and the dynamic point rejection of instance segmentation and geometric constraint is performed by using the image characteristics in a dynamic scene, which is efficient while keeping the simplicity of actual application. The map can reject the influence of dynamic targets, complete high-quality background reconstruction, achieve good performance in a dynamic scene, and effectively meet the positioning and mapping application requirements in different scenes. In related technologies, when a visual SLAM method performs real-time dense mapping, since a point cloud or an octree is used to represent a map, a reasonable prediction cannot be made for unobserved or occluded parts in a scene by a moving object, and the observed information of the system later cannot be used to optimize the mapped parts, resulting in a large number of holes and missing parts in the reconstruction result, thus high-fidelity real-time dense reconstruction cannot be achieved, and the applicability in actual scenes is not strong.

[0028] Before the method for constructing a NeRF map based on visual SLAM in the embodiments of the application is explained, the SLAM system involved in the embodiments of the application is explained.

[0029] With the vigorous development of information technology such as artificial intelligence, unmanned systems also have higher and higher autonomy and intelligence, and have the ability to make decisions in specific environments. Autonomous intelligent systems (AIS) can complete general tasks without or with limited human intervention, and are widely used in automatic driving, medical care, national security and deep space exploration, and many other fields, such as various service robots and military drones that protect national security. However, in order for unmanned systems such as robots to achieve truly autonomous movement, they also need to have the ability to perceive and understand the surrounding environment, which is reflected in the ability of the robot to autonomously construct an environmental map of the surrounding environment and locate the relative position in the constructed environmental map. Therefore, the SLAM (Simultaneous Localization and Mapping) technology emerged as the times required, and became one of the core components of unmanned systems to normally operate and make decisions in unknown environments. Moreover, the SLAM technology also has a wide range of applications in intelligent manufacturing, virtual reality, smart cities and many other fields.

[0030] In actual applications, many factors will greatly interfere with the accuracy and stability of the visual SLAM system, and the most significant factor is the scene features. Since most visual SLAM solutions strongly rely on feature extraction and matching, when there are moving targets in the scene, the feature extraction and matching between images will be greatly affected, thereby affecting the stable operation of the visual SLAM system.

[0031] Specifically, Figure 1 A flowchart of a method for constructing a NeRF map based on visual SLAM provided by the embodiments of the application.

[0032] As Figure 1 shown, the method for constructing a NeRF map based on visual SLAM includes the following steps:

[0033] In step S101, in a dynamic scene, the static target feature points of the current frame image of the camera are obtained.

[0034] In some embodiments, many factors can greatly interfere with the accuracy and stability of the visual SLAM system, among which the most significant factor is the scene features. Since most visual SLAM schemes rely heavily on feature extraction and matching, when there are moving targets in the scene, the feature extraction and matching between images will be greatly affected, thereby affecting the stable operation of the visual SLAM system.

[0035] Therefore, in the feature extraction of the current frame image of the camera, a target geometric constraint condition can be calculated in the predefined dynamic instance segmentation mask, so as to remove some influencing factors or unnecessary feature points in the actual application scene according to the target geometric constraint condition, for example, remove dynamic feature points, and only obtain the remaining static target feature points.

[0036] Next, the process of obtaining the static target feature points of the current frame image in the embodiment of the present application is further described.

[0037] Optionally, in an embodiment of the present application, before performing instance segmentation on the current frame image according to the configuration file, it includes: performing instance segmentation on the predefined instance according to the predefined target category to obtain the predefined dynamic instance and the predefined static instance; and storing the detection confidence and the corresponding semantic color of the predefined dynamic instance and the predefined static instance in the file to obtain the configuration file.

[0038] In some embodiments, when obtaining the reserved feature points of the current frame image, a configuration file needs to be used. Based on this, before performing instance segmentation on the current frame image according to the configuration file, the present application needs to store the predefined instance segmentation category information (static or dynamic), the detection confidence, and the semantic color corresponding to each category, the camera intrinsic parameter, the scale factor, and the preset feature point number in the configuration file to obtain the configuration file.

[0039] For example, the present application can first pre-train the instance segmentation network YOLOv8 on the COCO dataset. The pre-trained instance segmentation network YOLOv8 is used to segment the predefined instance to obtain the predefined classification result, such as the predefined dynamic instance and the predefined static instance, and store it in the configuration file.

[0040] Optionally, in an embodiment of the present application, obtaining the static target feature points of the current frame image of the camera includes: performing instance segmentation on the current frame image according to the configuration file to obtain a target instance segmentation mask, so as to divide the target instance segmentation mask into a dynamic instance segmentation mask and a static instance segmentation mask; based on a target geometric constraint condition, calculating a dynamic score of an object represented by the dynamic instance segmentation mask and the static instance segmentation mask, respectively, so as to determine the static target feature points of the current frame image based on the dynamic score and an empirical threshold.

[0041] In actual execution, when the reserved feature points of the current frame image are acquired, the dynamic feature points can be removed according to the target geometric constraint condition calculated in the predefined dynamic instance segmentation mask, so as to determine the static target feature points of the image.

[0042] Further, when the reserved feature points of the current frame image are determined, the target instance segmentation mask can be divided into a dynamic instance segmentation mask and a static instance segmentation mask according to the configuration file after the instance segmentation of the current frame image is performed to obtain the target instance segmentation mask. Then, the dynamic scores of the objects represented by the dynamic instance segmentation mask and the static instance segmentation mask of the current frame image can be calculated based on the target geometric constraint condition, so as to determine the reserved feature points of the current frame image based on the dynamic scores and the empirical threshold. The specific process can be represented as follows:

[0043] (1) According to the feature pyramid layer number (the feature pyramid is a common technology used in object detection, image segmentation, target tracking and other tasks in computer vision, and the layer number is usually determined by fusing feature maps of different scales) and the preset feature point (which can be understood as a feature point manually selected in some specific task or defined based on prior knowledge, such as a corner point or an edge point) number in the configuration file, the ORB feature points (composed of key points and descriptors, wherein the key points refer to special places in the image, such as corner points, which are the positions of the feature points in the image; the descriptors refer to the pixel information around the key points) of each frame of the current frame image are extracted, and at the same time, a data matrix is managed and maintained by the SLAM system to store the descriptors of these feature points.

[0044] (2) All ORB feature points in the current key frame are screened to remove overlapping feature points.

[0045] (3) The ORB feature points remaining after removing the overlapping feature points are subjected to Delaunay triangulation processing to form a triangular mesh, and the three-dimensional coordinates of each vertex of the triangular mesh are calculated according to the image data and depth information of the current key frame, and the length of each edge of the triangular mesh is further calculated, and the calculation formula can be represented as follows:

[0046]

[0047] wherein X C , Y C , and Z C represent the three-dimensional space coordinates of any one feature point used to construct the Delaunay triangular mesh of the current frame, u and v are the corresponding image coordinates, f x , f y , c x , and cy Indicates camera intrinsic parameters, X Ci Y Ci Z Ci X Cj Y Cj Z Cj Let d represent the three-dimensional spatial coordinates of the i-th and y-th vertices on any edge of the Delaunay triangulation, respectively. c，j This represents the distance between the i-th vertex and the ,-th vertex in the Delaunay triangulation of the current frame.

[0048] (4) Based on the feature matching relationship between ORB feature points in the current keyframe (mainly based on the similarity between feature point descriptors), the triangular mesh of the current keyframe is mapped to the previous keyframe. The three-dimensional coordinates of each vertex of the triangular mesh are calculated by backprojection based on the image data and depth information of the previous keyframe. The length of each side of the triangular mesh is further calculated. The calculation formula can be expressed as follows:

[0049]

[0050] Among them, X l Y l Z l Let u′ and v′ represent the 3D spatial coordinates of any feature point used to construct the Delaunay triangulation in the previous frame, and f be its corresponding image coordinates. x f y c x c y Indicates camera intrinsic parameters, X li Y li Z li X lj Y lj Z lj These represent X in the previous frame and X in the current frame, respectively. Ci Y Ci Z Ci X Cj Y Cj Z Cj The 3D spatial coordinates of the i-th and j-th vertices on an edge in a matched triangular mesh, d l，ij This represents the distance between the i-th vertex and the th vertex matched in the Delaunay triangulation of the previous frame.

[0051] (5) Calculate a change ratio based on the length of each edge in the triangular mesh in the current keyframe and the length in the previous keyframe. The calculation formula can be expressed as follows:

[0052]

[0053] Next, an adaptive threshold for the change ratio of an edge is calculated based on all edges located in the background region in the current keyframe. In this example, twice the average change ratio of edges in the background region of the current keyframe is used to represent the adaptive threshold τ. τ can be set to a minimum of 0.05 to avoid the adaptive threshold becoming too strict. The specific definition can be expressed as follows:

[0054]

[0055] Where N is the number of edges in the current keyframe where both ends of the feature points are located within the background region; w ij The change ratio σ ij The weighting coefficient, e ij For the edge.

[0056] This application's embodiments can comprehensively consider historical observation data of feature points and triangle edges, combining the number of times the edges are observed in historical keyframes and the standard deviation of their lengths to weight the change ratio. This enhances the weight of edges that remain stable during observation in the calculation, and reduces the weight of unstable edges or edges with fewer historical observations when calculating the adaptive threshold. The weighting coefficient w ij The specific definition can be expressed as follows:

[0057] w ij =w n w s (7)

[0058]

[0059]

[0060] Where n is the edge e ij The number of times it is observed in historical keyframes, w n The observation count weight is positively correlated with the observation count n, and reaches its maximum value of 1 when the edge has n ≥ 3 observations in the historical keyframes. ij For edge e ij The standard deviation of length in historical keyframes, d k，ij For edge e ij The length w in the k-th observable historical keyframe s To observe the standard deviation weights, when edge e ij The more stable the historical observations of the length, i.e., the more stable the standard deviation s, the better. ij The smaller the value, the greater the weight. The constant 0.05 (in meters) represents the common depth measurement error of RGB-D cameras within their normal operating range.

[0061] (6) judge whether the change ratio of each edge in the foreground region (main object or region that needs to be recognized, extracted or analyzed) is greater than a threshold value, and if greater than the threshold value, mark as a dynamic edge, otherwise mark as a static edge.

[0062] (7) calculate a dynamic score according to whether the edge in each instance segmentation mask is a dynamic edge,

[0063] s i =∑ j F(e ij ) (10)

[0064]

[0065] wherein F(e ij ) is a function of scoring each edge e ij according to the change ratio and the position of the end points, wherein the corresponding feature points of the edge e ij are p i and p j . As described above, in the embodiments of the present application, only the feature points in a certain potential dynamic object mask region are calculated for the dynamic score. For each dynamic edge, the position of the other end feature point p j in the image can be considered. If p j is in the background region, it is considered that the change in the length of the dynamic edge is mainly caused by the movement of p i , so the dynamic score s i is increased by 1; if p j is also in a certain potential dynamic object mask region, it is considered that p i and p j may both be moving, so the dynamic scores s i and s j are both increased by 0.5.

[0066] If the dynamic score is greater than an empirical threshold value, for example, 5, it is indicated that the object represented by the instance segmentation mask is indeed moving, and the ORB feature points in the entire dynamic target instance segmentation mask need to be removed. At this time, a data matrix can be reconstructed and managed to save the descriptors of the remaining ORB feature points after the dynamic feature points are removed, so as to determine the reserved feature points of the current frame image according to the descriptors.

[0067] In step S102, the static target feature points are tracked, so as to solve the camera pose of the current frame of the camera according to the static target feature points in the tracking process, obtain the number of projection points and the tracking time, and select the first key frame according to the number of projection points and the tracking time.

[0068] In some embodiments, after obtaining the static target feature points of the current frame image, the application can track the static target feature points and calculate the initial pose of the camera of the current frame according to the static target feature points in the tracking process. Specifically, the initial pose of the camera of the current frame can be calculated by a uniform linear motion model, but is not limited thereto, i.e., it is assumed that the camera motion in the time period from the previous frame to the current frame is consistent with the camera motion in the previous time period. According to the remaining matching feature points, the initial pose of the camera of the current frame is calculated by minimizing the re-projection error.

[0069] Further, after obtaining the initial pose of the camera of the current frame and determining that the current frame is a key frame, the key frame needs to be managed and processed, and the initial pose of the camera of the current frame is optimized by maintaining the key frame co-view to determine the operation of the current SLAM system. Specifically, in the case that the system positioning is not stable enough (i.e., the number of co-view points of the current frame and the reference frame is small) and the quality of the current frame is high, the current frame is inserted into the key frame co-view as a key frame, and the feature points of the current frame are projected to the world coordinate system according to the pose of the camera of the current frame and the depth of the feature points, and are restored as map points.

[0070] For example, the initial pose of the camera of the current frame is calculated by the uniform linear motion model and the remaining matching feature points by minimizing the re-projection error, and the formula can be represented as:

[0071] min R,t，X ∑ i ||u i -v i || (12)

[0072] wherein R represents the rotation matrix of the initial pose of the camera of the current frame, t represents the translation matrix of the initial pose of the camera of the current frame, X represents the three-dimensional coordinates of the image feature points in the world coordinate system, u i represents the image coordinates of the matching feature points of the current frame participating in the calculation of minimizing the re-projection error, and v i represents the re-projection point image coordinates of the matching feature points of the previous frame projected to the current frame.

[0073] It should be noted that since formula (12) is a nonlinear optimization problem, the Gauss-Newton method can be used for solving in the embodiments of the application, but is not limited thereto, and the skilled in the art can solve it by other ways, which is only exemplarily described herein and is not specifically explained.

[0074] Then the pose can be optimized by maintaining a keyframe co-view. Specifically, a keyframe co-view can be maintained and managed using a SLAM system, which can be composed of keyframes, map points, and co-view weighted edges, but is not limited to. During system operation, when the interval between the last reference frame and the current frame is more than 30 frames, or the number of co-view points between the current frame and the reference frame is less than a threshold Th_pt = 30, it can be determined whether to insert the keyframe co-view into the keyframe according to the actual situation. For example, the quality of the current frame is judged, and if the current frame has at least Th ft = 15 recoverable map points, and the pose thereof satisfies Then the current frame can be inserted into the keyframe co-view as a keyframe, and the feature points belonging to the keyframe can be projected to the world coordinate system according to the depth of the point and the camera pose of the frame to recover as map points. When two keyframes have more than 15 co-observable map points, a weighted edge can be constructed between the two keyframes, and the weight is the number of co-view points of the two keyframes.

[0075] Further, when selecting the first keyframe in the keyframe co-view to determine whether the current frame is a keyframe, the selection can be performed according to the following conditions, but is not limited to:

[0076] When a certain keyframe satisfies the following conditions, the keyframe can be deleted: (1) more than 90% of the map points of the keyframe can be observed by other 5 keyframes; (2) the keyframe is separated from the current frame by more than 10 keyframes. When a certain map point satisfies one of the following conditions, the point can be deleted: (1) the number of keyframes that can observe the map point is less than 25% of the number of keyframes that can theoretically observe the point; (2) after the map point is first created, the next two consecutive keyframes cannot observe the map point.

[0077] It should be noted that the various insertion conditions, deletion conditions, or selection conditions involved in maintaining the keyframe co-view to optimize the pose in the embodiments of the present application are only exemplary, and the person skilled in the art can select or adjust them according to the actual situation during actual application, and the embodiments of the present application do not make specific limitations.

[0078] In step S103, pixel point sampling is performed in the first keyframe to obtain a first sampling result, and the first sampling result is input into a pre-established NeRF map model for training to obtain a NeRF map construction result.

[0079] Optionally, in an embodiment of the present application, it further includes: selecting a second keyframe in the set of historical keyframes; performing pixel point sampling in the second keyframe to obtain a second sampling result; and inputting the second sampling result into the pre-established NeRF map model for training.

[0080] As a possible implementation manner, after the first key frame of the current frame image is selected, the ORB feature points can be used to guide the pixel point sampling in the static area of the current key frame to obtain a first sampling result. A key frame can be randomly selected from the historical key frame set, and a pixel point can be randomly sampled in the static area of the key frame. The above operations are performed M times to obtain a complete pixel point sampling result in the global optimization process. The complete pixel point sampling result in the global optimization process is taken as a second sampling result.

[0081] The NeRF map model is input into the training in combination with the first sampling result, the second sampling result, and the calculated pose. The training of the NeRF map model is ended after the tracking of the visual SLAM system is completed, and the final NeRF map model parameter, that is, the dense NeRF map, is obtained. The output result can obtain the camera pose and the NeRF map construction result.

[0082] The specific process can be represented as follows:

[0083] (1) determining the static area in the current key frame according to the detected dynamic object;

[0084] (2) performing pixel point sampling in the static area of the current key frame image according to the ORB feature points to input the NeRF map model;

[0085] (3) randomly selecting a key frame from the historical key frame set and performing random pixel point sampling to input the NeRF map model;

[0086] (4) performing the training of the NeRF map model in the idle period of the mapping thread, and ending the training of the NeRF map model after the tracking of the visual SLAM system is completed to obtain the final NeRF map model parameter, that is, the dense NeRF map.

[0087] (5) writing the camera pose in the form of a quaternion into a text file and saving, and saving the NeRF map in the form of a model parameter file.

[0088] Further, the embodiment of the application can construct a visual SLAM system based on a Linux platform, and implement on the dynamic scene image sequence of the public RGB-D dataset TUM.

[0089] (1) Table 1 is an experimental data table of the system positioning accuracy-absolute trajectory error (ATE) comparison (unit: m) of the embodiment of the application and two other open source RGB-D (ORB-SLAM2 and DS-SLAM) and the NeRF map construction method based on visual SLAM proposed in the embodiment of the application in a dynamic scene, which can be as follows:

[0090] Table 1

[0091]

[0092]

[0093] Table 1 uses the RMSE statistics of absolute trajectory error ATE as an evaluation index, the smaller the value, the better the performance. From the table, it can be seen that on the widely used TUM dataset, the positioning error of the NeRF map construction method based on visual SLAM proposed in the embodiment of the application is less than that of the other two open source SLAM methods in four different dynamic scenes, which shows that the NeRF map construction method based on visual SLAM proposed in the application has better positioning accuracy in dynamic scenes.

[0094] (2) Further, Figure 2 The schematic diagram of the dense NeRF map constructed in real time by an embodiment of the application in dynamic sequences fr3-w-xyz and fr2-desk-with-person is shown in Figure 6. Figure 2 As shown in Figure 6, it can be seen that the dense NeRF map constructed in real time by the embodiment of the application in the dynamic sequence has a high-fidelity three-dimensional scene reconstruction effect.

[0095] The application will be described in detail below with one specific embodiment.

[0096] Figure 3 The flowchart of the NeRF map construction method based on visual SLAM of an embodiment of the application is shown in Figure 7. Figure 3 As shown in Figure 7:

[0097] Step 1: Obtain the current frame image of the camera for instance segmentation, and classify all instance segmentation results into static instances and dynamic instances.

[0098] Step 2: Feature extraction for the current frame image, remove dynamic feature points according to geometric constraints in the dynamic instance segmentation mask.

[0099] Step 3: In the tracking process, solve the current frame camera pose according to the remaining static target feature points.

[0100] Step 4: Select key frames according to the number of inlier points in the projection and the tracking time in the tracking process, and manage and maintain the co-view of the key frames.

[0101] Step 5: Sample the pixel points in each key frame and input them into the NeRF map model for training to obtain the NeRF map model parameters, i.e. the dense NeRF map.

[0102] Step 6: Output the results, i.e. the camera pose and the NeRF map.

[0103] The NeRF map construction method based on visual SLAM according to the embodiment of the application can overcome the problem of false matching in a dynamic scene based on the dynamic point elimination method of instance segmentation and geometric constraint. A dense NeRF map is generated in real time for the dynamic scene. Thus, the problem of decreased positioning accuracy of the existing SLAM technology in a dynamic scene is solved. The dynamic point elimination of instance segmentation and geometric constraint is performed by using the image characteristics in the dynamic scene, which is efficient and simple in practical application. The map can eliminate the influence of dynamic targets and complete high-quality background reconstruction, and better performance is achieved in a dynamic scene, effectively meeting the positioning and mapping application requirements in different scenes. In the related art, when the visual SLAM method performs real-time dense mapping, since the point cloud or octree is used to represent the map, the unobserved or occluded part in the scene cannot be reasonably predicted, and the subsequent observation information of the system cannot be used to optimize the part of the built map, resulting in a large number of holes and missing parts in the reconstruction result, so that high-fidelity real-time dense reconstruction cannot be achieved, and the applicability in actual scenes is not strong.

[0104] Secondly, the NeRF map construction device based on visual SLAM according to the embodiment of the application is described with reference to the accompanying drawings.

[0105] Figure 4 FIG. 1 is a structural schematic diagram of the NeRF map construction device based on visual SLAM according to the embodiment of the application.

[0106] As shown in FIG. 1, the NeRF map construction device 10 based on visual SLAM includes an acquisition module 100, a first selection module 200, and a construction module 300. Figure 4

[0107] The acquisition module 100 is configured to acquire static target feature points of a current frame image of a camera in a dynamic scene.

[0108] The first selection module 200 is configured to track the static target feature points, to solve a current frame camera pose of the camera according to the static target feature points in the tracking process, to obtain a number of projection points and a tracking time, and to select a first key frame according to the number of projection points and the tracking time.

[0109] The construction module 300 is configured to perform pixel point sampling according to ORB features in the first key frame to obtain a first key frame sampling result, and to input the first key frame sampling result into a pre-established NeRF map model to perform training, to obtain NeRF map model parameters, and to obtain a NeRF map construction result.

[0110] Optionally, in an embodiment of the application, the acquisition module 100 includes a segmentation unit and a determination unit.

[0111] ​The segmentation unit is configured to perform instance segmentation on the current frame image according to a configuration file to obtain a target instance segmentation mask, and divide the target instance segmentation mask into a dynamic instance segmentation mask and a static instance segmentation mask.

[0112] The determining unit is configured to calculate a dynamic score of an object represented by the dynamic instance segmentation mask and the static instance segmentation mask based on a target geometric constraint condition, and determine a static target feature point of the current frame image based on the dynamic score and an empirical threshold.

[0113] Optionally, in an embodiment of the present application, the method further comprises a first training module and a storage module.

[0114] The segmentation module is configured to perform instance segmentation on predefined instances according to predefined target categories before performing instance segmentation on the current frame image according to the configuration file to obtain predefined dynamic instances and predefined static instances.

[0115] The storage module is configured to store the detection confidence of the predefined dynamic instances and the predefined static instances and the corresponding semantic color in a file to obtain the configuration file.

[0116] Optionally, in an embodiment of the present application, the method further comprises a second selection module, a sampling module and a training module.

[0117] The second selection module is configured to select a second key frame from the set of historical key frames.

[0118] The sampling module is configured to perform pixel point sampling in the second key frame to obtain a second sampling result.

[0119] The training module is configured to input the second sampling result into a pre-established NeRF map model for training.

[0120] It should be noted that the foregoing explanation and description of the embodiment of the method for constructing a NeRF map based on visual SLAM also applies to the embodiment of the device for constructing a NeRF map based on visual SLAM, which will not be described here again.

[0121] The NeRF map construction device based on visual SLAM provided by the embodiment of the application can overcome the problem of false matching in a dynamic scene based on the dynamic point elimination method of instance segmentation and geometric constraint. A dense NeRF map is generated in real time for a dynamic scene. Thus, the problem of decreased positioning accuracy of the existing SLAM technology in a dynamic scene is solved. The dynamic point elimination of instance segmentation and geometric constraint is performed by using the image characteristics in a dynamic scene, which is efficient and simple in actual application. The map can eliminate the influence of dynamic targets and complete high-quality background reconstruction. Better performance is achieved in a dynamic scene, and the positioning and mapping application requirements in different scenes are effectively met. In the related art, when a visual SLAM method is used for real-time dense mapping, since a point cloud or an octree is used to represent the map, the unobserved or occluded part in the scene cannot be reasonably predicted, and the subsequent observation information of the system cannot be used to optimize the part of the map, resulting in a large number of holes and missing parts in the reconstruction result, so that high-fidelity real-time dense reconstruction cannot be achieved, and the applicability in an actual scene is not strong.

[0122] Figure 5 The structure schematic diagram of the electronic device provided by the embodiment of the application is provided. The electronic device can include:

[0123] The memory 501, the processor 502, and the computer program stored in the memory 501 and executable on the processor 502.

[0124] The processor 502 implements the NeRF map construction method based on visual SLAM provided in the above embodiment when executing the program.

[0125] Further, the electronic device further includes:

[0126] The communication interface 503 is used for communication between the memory 501 and the processor 502.

[0127] The memory 501 is used to store the computer program executable on the processor 502.

[0128] The memory 501 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0129] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 5 Only one thick line is used in the figure to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0130] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can complete communication between each other through an internal interface.

[0131] The processor 502 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0132] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the above method for constructing a NeRF map based on visual SLAM.

[0133] The embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program can run computer instructions, and the computer instructions are executed by a processor to implement the method for constructing a NeRF map based on visual SLAM provided by the embodiments of the present application.

[0134] In the description of the application, reference to "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that a particular feature, structure, material, or characteristic being described is included in at least one embodiment or example of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment or example. Furthermore, the described specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples. In addition, the usage of "N" means at least two, for example, two, three or the like, unless explicitly stated otherwise.

[0135] Furthermore, the terms "first", "second", or the like, are used merely as a designation of certain elements or features of the application, and do not imply or connote relative importance or a specific order of precedence. Thus, features defined with "first", "second", etc. can include at least one of the features, either explicitly or implicitly.

[0136] Any process or method descriptions or blocks in flow charts or otherwise described herein represent embodiments of modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions or steps, and alternate implementations are possible. In some embodiments, the processes or methods described in flow charts or otherwise described herein are not necessarily performed in the order shown or discussed, including, for example, performing or depending from other operations or stages, in parallel, in reverse order, or in other orders.

[0137] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of executable instructions stored in a computer readable medium, which can be executed by an instruction execution system, apparatus or device, such as a computer-based system, a processor-based system, or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or a combination of the above. For the purposes of this specification, a "computer readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus or device. The computer readable medium can be a computer readable storage medium or a computer readable signal medium. The computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a propagation medium. The computer readable signal medium can include, but is not limited to, a computer readable medium that facilitates transfer of the program from one place to another. A specific example of a computer readable medium is a non-transitory computer-readable storage medium. A specific example of a computer readable signal medium is a source or destination of the computer readable medium. Another specific example of a computer readable signal medium is a computer readable signal travelling through space. Thus, a computer readable medium can take many forms of hardware to carry out the program for use by or in connection with the instruction execution system, apparatus or device.

[0138] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware and in another embodiment, the hardware can be implemented using any or a combination of the following technologies, which are each well known in the art: a discrete logic circuit(s) having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array(s) (PGA), a field programmable gate array (FPGA), etc.

[0139] Those of skill in the art would understand that the steps of the methods carried out above can be carried out wholly or partly by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, and when executed, includes one or a combination of the steps of the method embodiments.

[0140] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0141] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for NeRF map construction based on visual SLAM, characterized in that, The method comprises the following steps: In a dynamic scene, obtaining static target feature points of a current frame image of a camera; Tracking the static target feature points to solve the current frame camera pose of the camera according to the static target feature points in the tracking process, to obtain the number of projection points and the tracking time, and to select a first key frame according to the number of projection points and the tracking time; In the first key frame, pixel point sampling is performed according to ORB feature guidance to obtain a first key frame sampling result, and the first key frame sampling result is input into a pre-established NeRF map model for training to obtain NeRF map model parameters, so as to obtain a NeRF map construction result; Wherein, the obtaining of the static target feature points of the current frame image of the camera comprises: performing instance segmentation on the current frame image according to a configuration file to obtain a target instance segmentation mask, dividing the target instance segmentation mask into a dynamic instance segmentation mask and a static instance segmentation mask; based on a target geometric constraint condition, the dynamic score of the object represented by the dynamic instance segmentation mask and the static instance segmentation mask is calculated respectively, and based on the dynamic score and an empirical threshold, the static target feature points of the current frame image are determined; The method comprises: generating a triangular mesh based on ORB feature points of each frame in the current frame image; calculating lengths of each edge in the triangular mesh in the current key frame according to image data and depth information of the current key frame ; mapping the triangular mesh into a previous key frame according to a feature matching relationship between the ORB feature points of the current key frame, and calculating lengths of each edge in the triangular mesh in the previous key frame according to image data and depth information back projection of the previous key frame ; combining the lengths and the lengths to calculate a change ratio of the lengths of each edge in the triangular mesh; weighting the change ratio based on the ORB feature points and historical observation data of each edge in the triangular mesh to obtain a weight of each edge for calculating an adaptive threshold; combining the weight and an adaptive threshold for calculating the change ratio of the edge according to all edges located in a background region in the current key frame; and determining static target feature points of the current frame image based on the change ratio and the adaptive threshold. Wherein, the calculation formula of the adaptive threshold value is: wherein, is an adaptive threshold, is the number of feature points at both ends of the edge in the current key frame that are located in the background region; is the change ratio is a weighting coefficient, is an edge; Wherein, the calculation formula of the weight is: wherein, is an edge the number of times it was observed in historical keyframes, is an observation weight, is an edge the standard deviation of the length in historical keyframes, is an edge the length in the k historical keyframe that can be observed, is an observation standard deviation weight, is the length of each edge in the triangular mesh in the previous keyframe.

2. The method of claim 1, wherein, Before performing instance segmentation on the current frame image according to the configuration file, comprising: Performing instance segmentation on a predefined instance according to a predefined target category to obtain a predefined dynamic instance and a predefined static instance; The detection confidence and the corresponding semantic color of the predefined dynamic instance and the predefined static instance are stored in a file to obtain the configuration file.

3. The method of claim 1, wherein, Further comprising: Selecting a second key frame in a historical key frame set; Pixel point sampling is performed in the static area of the second key frame to obtain a second sampling result; The second sampling result is input into the pre-established NeRF map model for training.

4. A device for NeRF map construction based on visual SLAM, characterized in that, Comprise: The acquisition module is used for obtaining static target feature points of a current frame image of a camera in a dynamic scene; The selection module is used for tracking the static target feature points to solve the current frame camera pose of the camera according to the static target feature points in the tracking process, to obtain the number of projection points and the tracking time, and to select a first key frame according to the number of projection points and the tracking time; The construction module is used for performing pixel point sampling in the first key frame according to ORB feature guidance to obtain a first key frame sampling result, and inputting the first key frame sampling result into a pre-established NeRF map model for training to obtain NeRF map model parameters, so as to obtain a NeRF map construction result; The acquisition module comprises: a segmentation unit configured to perform instance segmentation on the current frame image according to a configuration file to obtain a target instance segmentation mask, and divide the target instance segmentation mask into a dynamic instance segmentation mask and a static instance segmentation mask; and a determination unit configured to calculate a dynamic score of an object represented by the dynamic instance segmentation mask and the static instance segmentation mask based on a target geometric constraint condition, and determine a static target feature point of the current frame image based on the dynamic score and an empirical threshold. The determining unit comprises: generating a triangular mesh based on the ORB feature points of each frame in the current frame image; calculating the length of each edge in the triangular mesh in the current key frame according to the image data and depth information of the current key frame ; mapping the triangular mesh into the previous key frame according to the feature matching relationship between the ORB feature points of the current key frame, and calculating the length of each edge in the triangular mesh in the previous key frame according to the image data and depth information of the previous key frame ; combining the length and the length , calculating the change ratio of the length of each edge of the triangular mesh; weighting the change ratio based on the ORB feature points and the historical observation data of each edge of the triangular mesh to obtain the weight of each edge for calculating the adaptive threshold; combining the weight and the adaptive threshold for calculating the change ratio of the edge according to all edges located in the background region in the current key frame; determining the static target feature points of the current frame image based on the change ratio and the adaptive threshold. The adaptive threshold is calculated according to the following formula: wherein, is an adaptive threshold, is the number of edges whose two end feature points are both located in the background region in the current key frame; is the change ratio of the weighting coefficient, is an edge; The weight is calculated according to the following formula: wherein, is an edge the number of times observed in historical keyframes, is an observation number weight, is an edge the standard deviation of length in historical keyframes, is an edge the length in the k historical keyframe that can be observed, is an observation standard deviation weight, is the length of each edge in the triangular mesh in the previous keyframe.

5. The apparatus of claim 4, wherein, Further comprising: A segmentation module configured to perform instance segmentation on predefined instances according to a predefined target category before performing instance segmentation on the current frame image according to a configuration file to obtain predefined dynamic instances and predefined static instances; A storage module configured to store a detection confidence and a corresponding semantic color of the predefined dynamic instances and the predefined static instances in a file to obtain the configuration file.

6. An electronic device, comprising: The computer program is stored in the memory and executable on the processor, and the processor executes the program to implement the NeRF map construction method based on visual SLAM according to any one of claims 1-3. The program is executed by the processor to implement the NeRF map construction method based on visual SLAM according to any one of claims 1-3.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed to implement the NeRF map construction method based on visual SLAM according to any one of claims 1-3.

8. A computer program product comprising a computer program, characterized in that, ​

Images