NeRF map construction method and device based on visual SLAM

Abstract

The invention relates to the technical field of computer vision, in particular to a NeRF map construction method and device based on visual SLAM, and the method comprises the steps: obtaining a static target feature point of a current frame image of a camera in a dynamic scene; tracking the static target feature points, solving the current frame camera pose of the camera, and selecting a key frame according to the number of projection inner points and the tracking time in the tracking process; pixel point sampling is conducted in the key frame according to ORB feature guidance, a key frame sampling result is input into a pre-established NeRF map model for training, NeRF map model parameters are obtained, and a NeRF map construction result is obtained. According to the method, the dense NeRF map can be generated in real time for the dynamic scene, the problem of mismatching in the dynamic scene is solved, simplicity is kept while high efficiency is achieved, the influence caused by a dynamic target is eliminated, high-quality background reconstruction is completed, and the positioning and mapping application requirements in different scenes are effectively met.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a NeRF map construction method and device based on visual SLAM. Background Art

[0002] In the related art, the sensor of the traditional SLAM solution is mainly laser radar, but due to the high cost of laser radar, and the point cloud information can only reflect the distance and angle of the surrounding object points, it discards the rich information such as the texture and semantics of the surrounding environment, and has poor interaction with people. Therefore, the camera has gradually become the mainstream sensor of the SLAM solution in recent years, and the SLAM solution with the camera as the main sensor is called visual SLAM. Some studies use deep learning networks to detect potential moving targets in images, so as to remove potential dynamic feature points and improve the operation ability of the visual SLAM system in dynamic scenes. However, this method based on deep learning networks will remove all objects that may move in the image, even if these objects do not actually move (such as stationary vehicles and humans, etc.), and when there are some non-potential moving targets that are forced to move in the environment (such as books and cabinets that are moved), the deep network cannot remove them. In addition, the visual SLAM system also faces the challenge of building high-fidelity dense maps in real time in dynamic scenes to adapt to the complex task requirements of unmanned systems. At present, some traditional visual SLAM methods can complete real-time dense mapping using RGB-D cameras or with the help of GPU acceleration.

[0003] However, when the visual SLAM method in the related technology builds dense maps in real time, it uses point clouds or octrees to represent the maps, and cannot make reasonable predictions about the unobserved or occluded parts of the scene. It is also unable to use the system's subsequent observation information to optimize the mapped parts, resulting in a large number of holes and missing parts in the reconstruction results. Therefore, it is impossible to achieve high-fidelity real-time dense reconstruction, and the applicability in actual scenarios is not strong, which needs to be solved urgently. Summary of the invention

[0004] The present application provides a NeRF map construction method and device based on visual SLAM to solve the problems that when the visual SLAM method in the related art is used for real-time dense mapping, due to the use of point cloud or octree to represent the map, it is impossible to make a reasonable prediction for the unobserved or blocked parts of the scene by moving objects, and it is impossible to use the subsequent observation information of the system to optimize the mapped parts, resulting in a large number of holes and missing parts in the reconstruction results. Therefore, it is impossible to achieve high-fidelity real-time dense reconstruction and the applicability in actual scenes is not strong.

[0005] The first aspect of the present application provides a NeRF map construction method based on visual SLAM, comprising the following steps: in a dynamic scene, obtaining static target feature points of a current frame image of a camera; tracking the static target feature points, so as to solve the current frame camera pose of the camera according to the static target feature points during the tracking process, and obtain the number of projection points and the tracking time, so as to select a first key frame according to the number of points in the projection and the tracking time; sampling pixels in the first key frame according to the ORB feature guidance to obtain a first key frame sampling result, and inputting the first key frame sampling result into a pre-established NeRF map model for training to obtain NeRF map model parameters to obtain a NeRF map construction result.

[0006] Optionally, in one embodiment of the present application, obtaining static target feature points of a current frame image of the camera includes: performing instance segmentation on the current frame image according to a configuration file to obtain a target instance segmentation mask to divide the target instance segmentation mask into a dynamic instance segmentation mask and a static instance segmentation mask; based on target geometric constraints, respectively calculating the dynamic scores of the objects represented by the dynamic instance segmentation mask and the static instance segmentation mask to determine the static target feature points of the current frame image based on the dynamic scores and an empirical threshold.

[0007] Optionally, in one embodiment of the present application, before performing instance segmentation on the current frame image according to the configuration file, it includes: performing instance segmentation on predefined instances according to predefined target categories to obtain predefined dynamic instances and predefined static instances; storing the detection confidence and corresponding semantic colors of the predefined dynamic instances and the predefined static instances into a file to obtain the configuration file.

[0008] Optionally, in one embodiment of the present application, it also includes: selecting a second key frame from the historical key frame set; sampling pixels in a static area of ​​the second key frame to obtain a second sampling result; and inputting the second sampling result into the pre-established NeRF map model for training.

[0009] The second aspect of the present application provides a NeRF map construction device based on visual SLAM, including: an acquisition module, which acquires static target feature points of the current frame image of the camera in a dynamic scene; a first selection module, which is used to track the static target feature points, so as to solve the current frame camera pose of the camera according to the static target feature points during the tracking process, and obtain the number of projection points and tracking time, so as to select a first key frame according to the number of points in the projection and the tracking time; a construction module, which is used to sample pixel points in the first key frame to obtain a first sampling result, and input the first sampling result into a pre-established NeRF map model for training to obtain a NeRF map construction result.

[0010] Optionally, in one embodiment of the present application, the acquisition module includes: a segmentation unit, used to perform instance segmentation on the current frame image according to a configuration file to obtain a target instance segmentation mask, so as to divide the target instance segmentation mask into a dynamic instance segmentation mask and a static instance segmentation mask; a determination unit, used to calculate the dynamic scores of the objects represented by the dynamic instance segmentation mask and the static instance segmentation mask respectively based on the target geometric constraints, so as to determine the static target feature points of the current frame image based on the dynamic scores and the empirical threshold.

[0011] Optionally, in one embodiment of the present application, it also includes: a segmentation module, which is used to perform instance segmentation on predefined instances according to predefined target categories before performing instance segmentation on the current frame image according to the configuration file, so as to obtain predefined dynamic instances and predefined static instances; and a storage module, which is used to store the detection confidence and corresponding semantic color of the predefined dynamic instances and the predefined static instances into a file to obtain the configuration file.

[0012] Optionally, in one embodiment of the present application, it also includes: a second selection module, used to select a second key frame from the historical key frame set; a sampling module, used to sample pixels in the static area of ​​the second key frame to obtain a second sampling result; and a training module, used to input the second sampling result into the pre-established NeRF map model for training.

[0013] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the NeRF map construction method based on visual SLAM as described in the above embodiment.

[0014] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the program is executed by a processor, it implements the above-mentioned NeRF map construction method based on visual SLAM.

[0015] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned NeRF map construction method based on visual SLAM.

[0016] The embodiment of the present application can overcome the mismatching problem in dynamic scenes based on the dynamic point elimination method of instance segmentation and geometric constraints. A dense NeRF map is generated in real time for dynamic scenes. Thus, the problem of reduced positioning accuracy of existing SLAM technology in dynamic scenes is achieved. The image characteristics in dynamic scenes are used to perform instance segmentation and dynamic point elimination with geometric constraints. The efficiency is high while maintaining the simplicity of practical application. The map can eliminate the influence of dynamic targets, complete high-quality background reconstruction, and achieve good performance in dynamic scenes, effectively meeting the positioning and mapping application requirements in different scenes. When the visual SLAM method in the related art is densely mapped in real time, due to the use of point clouds or octrees to represent the map, it is impossible to make reasonable predictions for the parts of the scene that are not observed or blocked by moving objects, and it is impossible to use the subsequent observation information of the system to optimize the parts that have been mapped, resulting in a large number of holes and missing parts in the reconstruction results. Therefore, it is impossible to achieve high-fidelity real-time dense reconstruction, and the applicability in actual scenes is not strong.

[0017] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0019] Figure 1 A flowchart of a NeRF map construction method based on visual SLAM provided according to an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a dense NeRF map constructed in real time under the dynamic sequences fr3-w-xyz and fr2-desk-with-person according to an embodiment of the present application;

[0021] Figure 3 This is a flowchart of a NeRF map construction method based on visual SLAM according to an embodiment of the present application;

[0022] Figure 4 A schematic diagram of the structure of a NeRF map construction device based on visual SLAM according to an embodiment of the present application;

[0023] Figure 5 Schematic diagram of the structure of an electronic device according to an embodiment of the present application.

[0024] Reference numerals:

[0025] 10-NeRF map construction device based on visual SLAM: 100-acquisition module, 200-first selection module and 300-construction module; 501-memory, 502-processor and 503-communication interface. DETAILED DESCRIPTION

[0026] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0027] The following describes the NeRF map construction method and device based on visual SLAM of the embodiment of the present application with reference to the accompanying drawings. When the visual SLAM method in the related art mentioned in the above background technology is used for real-time dense mapping, since the map is represented by point cloud or octree, it is impossible to make a reasonable prediction for the unobserved or occluded parts of the scene, and it is impossible to use the subsequent observation information of the system to optimize the mapped parts, resulting in a large number of holes and missing parts in the reconstruction results. Therefore, it is impossible to achieve high-fidelity real-time dense reconstruction, and the applicability in actual scenes is not strong. The present application provides a NeRF map construction method based on visual SLAM, in which a dynamic point elimination method based on instance segmentation and geometric constraints can be used to overcome the mismatching problem in dynamic scenes. A dense NeRF map is generated in real time for dynamic scenes. As a result, the problem of reduced positioning accuracy of existing SLAM technology in dynamic scenes is solved. The image characteristics in dynamic scenes are used to perform instance segmentation and dynamic point elimination with geometric constraints. The efficiency is high while maintaining the simplicity of practical application. The map can eliminate the influence of dynamic targets and complete high-quality background reconstruction. It achieves good performance in dynamic scenes and effectively meets the positioning and mapping application requirements in different scenes. When the visual SLAM method in the related art is used for real-time dense mapping, it is impossible to make reasonable predictions for the unobserved or occluded parts of the scene due to the use of point clouds or octrees to represent the map, and it is impossible to use the subsequent observation information of the system to optimize the mapped parts, resulting in a large number of holes and missing parts in the reconstruction results. Therefore, it is impossible to achieve high-fidelity real-time dense reconstruction, and it is not applicable in actual scenes.

[0028] Before explaining the NeRF map construction method based on visual SLAM in the embodiment of the present application, the SLAM system involved in the embodiment of the present application is first explained.

[0029] With the vigorous development of information technology such as artificial intelligence, unmanned systems have also become more and more autonomous and intelligent, and have the ability to make decisions in specific environments. Autonomous intelligent unmanned systems (AIS) can complete general tasks without or with limited human involvement. They are widely used in many fields such as autonomous driving, health care, national defense security, and deep space exploration, such as various service robots and military drones that defend national security. However, unmanned systems such as robots need to have the ability to perceive and understand the surrounding environment in order to achieve true autonomous movement. This is reflected in the robot's ability to autonomously build a map of the surrounding environment and locate the relative position in the built environment map. Therefore, Simultaneous Localization and Mapping (SLAM) technology came into being and became one of the core components for unmanned systems to operate normally and make decisions in unknown environments. In addition, SLAM technology is also widely used in many fields such as intelligent manufacturing, virtual reality, and smart cities.

[0030] In practical applications, many factors will have a significant impact on the accuracy and stability of the visual SLAM system, among which the most significant factor is the scene features. Since most visual SLAM solutions strongly rely on feature extraction and matching, when there are moving targets in the scene, the feature extraction and matching between images will be greatly affected, thus affecting the stable operation of the visual SLAM system.

[0031] Specifically, Figure 1 A flowchart of a NeRF map construction method based on visual SLAM provided in an embodiment of the present application.

[0032] like Figure 1 As shown, the NeRF map construction method based on visual SLAM includes the following steps:

[0033] In step S101, in a dynamic scene, static target feature points of a current frame image of a camera are obtained.

[0034] In some embodiments, many factors will have a significant impact on the accuracy and stability of the visual SLAM system, among which the most significant factor is the scene feature. Since most visual SLAM solutions rely more on feature extraction and matching, when there are moving targets in the scene, the feature extraction and matching between images will be greatly affected, thus affecting the stable operation of the visual SLAM system.

[0035] Therefore, when the embodiment of the present application performs feature extraction on the current frame image of the camera, a target geometric constraint condition can be calculated within a predefined dynamic instance segmentation mask, so as to remove some influencing factors or unnecessary feature points in the actual application scenario according to the target geometric constraint condition, such as removing dynamic feature points, etc., and only obtaining the retained static target feature points.

[0036] Next, the process of obtaining the static target feature points of the current frame image in the embodiment of the present application is further explained.

[0037] Optionally, in one embodiment of the present application, before performing instance segmentation on the current frame image according to the configuration file, it includes: performing instance segmentation on predefined instances according to predefined target categories to obtain predefined dynamic instances and predefined static instances; storing the detection confidence and corresponding semantic colors of the predefined dynamic instances and the predefined static instances into a file to obtain the configuration file.

[0038] In some embodiments, a configuration file is required to obtain the retained feature points of the current frame image. Based on this, before performing instance segmentation on the current frame image according to the configuration file, the present application needs to first store the predefined instance segmentation category information (static or dynamic), detection confidence, and the semantic color corresponding to each category, camera intrinsic parameters, scale factor, and preset feature point quantity into the configuration file to obtain the configuration file.

[0039] For example, the present application may first pre-train the instance segmentation network YOLOv8 on the COCO dataset. The pre-trained instance segmentation network YOLOv8 is used to segment the predefined instances, obtain predefined classification results, such as predefined dynamic instances and predefined static instances, etc., and store them in the configuration file.

[0040] Optionally, in one embodiment of the present application, obtaining static target feature points of a current frame image of a camera includes: performing instance segmentation on the current frame image according to a configuration file to obtain a target instance segmentation mask to divide the target instance segmentation mask into a dynamic instance segmentation mask and a static instance segmentation mask; based on target geometric constraints, respectively calculating the dynamic scores of the objects represented by the dynamic instance segmentation mask and the static instance segmentation mask to determine the static target feature points of the current frame image based on the dynamic scores and an empirical threshold.

[0041] In the actual implementation process, when obtaining the retained feature points of the current frame image, the dynamic feature points can be removed according to the target geometric constraints calculated in the predefined dynamic instance segmentation mask, so as to determine the static target feature points of the image.

[0042] Furthermore, when determining the retained feature points of the current frame image, the current frame image can be first instance segmented to obtain the target instance segmentation mask, and then the target instance segmentation mask can be divided into a dynamic instance segmentation mask and a static instance segmentation mask according to the configuration file. Then, based on the target geometric constraints, the dynamic scores of the objects represented by the dynamic instance segmentation mask and the static instance segmentation mask of the current frame image can be calculated respectively, so as to determine the retained feature points of the current frame image based on the dynamic score and the empirical threshold. The specific process can be expressed as follows:

[0043] (1) According to the number of feature pyramid layers in the configuration file (feature pyramid is a common technology used for object detection, image segmentation, target tracking and other tasks in computer vision. The number of layers is usually determined by fusing feature maps of different scales) and the number of preset feature points (which can be understood as feature points manually selected in advance in certain specific tasks or defined based on prior knowledge, such as corner points, edge points, etc.), the ORB feature points (composed of key points and descriptors, where key points refer to special places in the image, such as corner points, which are the positions of feature points in the image; descriptors refer to the pixel information used to quantify the description of the surroundings of key points) are extracted for each frame in the current frame image. At the same time, a data matrix is ​​managed and maintained by the SLAM system to store the descriptors of these feature points.

[0044] (2) Filter all ORB feature points in the current key frame and remove overlapping feature points.

[0045] (3) Delaunay triangulation is performed on the remaining ORB feature points except for the overlapping feature points to form a triangular network. The three-dimensional coordinates of each vertex of the triangular network are calculated by back-projection based on the image data and depth information of the current key frame, and the length of each edge of the triangular network is further calculated. The calculation formula can be expressed as follows:

[0046]

[0047] Among them, X C , Y C , Z C They represent the three-dimensional spatial coordinates of any feature point used to construct the Delaunay triangulation in the current frame, u and v are their corresponding image coordinates, and f x , f y , c x , cy represents the camera internal parameters, X Ci , Y Ci , Z Ci , X Cj , Y Cj , Z Cj Respectively represent the three-dimensional space coordinates of the i-th vertex and the ,th vertex on any edge in the Delaunay triangulation. c，j Represents the distance between the i-th vertex and the,th vertex in the Delaunay triangulation of the current frame.

[0048] (4) According to the feature matching relationship between the ORB feature points of the current key frame (mainly based on the similarity between the descriptors of the feature points), the triangle mesh of the current key frame is mapped to the previous key frame, and the three-dimensional coordinates of each vertex of the triangle mesh are calculated by back-projection based on the image data and depth information of the previous key frame, and the length of each edge of the triangle mesh is further calculated. The calculation formula can be expressed as follows:

[0049]

[0050] Among them, X l , Y l , Z l represents the 3D spatial coordinates of any feature point used to construct the Delaunay triangulation in the previous frame, u′, v′ are its corresponding image coordinates, and f x , f y , c x , c y represents the camera internal parameters, X li , Y li , Z li , X lj , Y lj , Z lj Respectively represent the previous frame and the current frame X Ci , Y Ci , Z Ci , X Cj , Y Cj , Z Cj The three-dimensional coordinates of the i-th and j-th vertices on an edge in the matching triangulated network, d l，ij Represents the distance between the i-th vertex and the ,th vertex matched in the previous frame's Delaunay triangulation.

[0051] (5) A change ratio is calculated based on the length of each edge in the triangle network in the current key frame and the length in the previous key frame. The calculation formula can be expressed as follows:

[0052]

[0053] Then, the adaptive threshold of the change ratio of an edge is calculated based on all the edges in the background area in the current key frame. In this example, the adaptive threshold τ is expressed as 2 times the average change ratio of the edges in the background area in the current key frame, and τ can be set to a minimum of 0.05, but is not limited to, to prevent the adaptive threshold from becoming too strict. The specific definition can be expressed as follows:

[0054]

[0055] Where N is the number of edges in the current key frame where both feature points at both ends are located in the background area; w ij is the change ratio σ ij The weighting coefficient, e ij For the edge.

[0056] The embodiment of the present application can comprehensively consider the historical observation data of feature points and triangle edges, and weight the change ratio in combination with the number of times the edge is observed in the historical key frame and the standard deviation of the length, so as to enhance the weight of the edge that remains stable during the observation process in the calculation, and reduce the weight of the edge that is unstable or has a small number of historical observations when calculating the adaptive threshold. ij The specific definition can be expressed as follows:

[0057] w ij =w n w s (7)

[0058]

[0059]

[0060] Where n is the edge e ij The number of times it has been observed in the historical keyframes, w n is the observation number weight, which is positively correlated with the observation number n, and reaches the maximum value 1 when the edge has n ≥ 3 observations in the historical key frame. ij For edge e ij The standard deviation of the length in the historical keyframes, d k，ij For edge e ij The length of the kth historical key frame that can be observed, w s is the observed standard deviation weight, when edge e ij The longer the historical observations are, the more stable the standard deviation s is. ij The smaller the value, the larger the weight. The constant 0.05 (in meters) is the depth value measurement error of a common RGB-D camera within the normal working range.

[0061] (6) Determine whether the change ratio of each edge located in the foreground area (the main object or area to be identified, extracted or analyzed) is greater than a threshold. If it is greater than the threshold, it is recorded as a dynamic edge, otherwise it is recorded as a static edge.

[0062] (7) Calculate a dynamic score based on whether the edge in each instance segmentation mask is a dynamic edge.

[0063] s i =∑ j F(e ij ) (10)

[0064]

[0065] Among them, F(e ij ) is the ratio of change and the position of the endpoints for each edge e ij The function of assigning points, where the edge e ij The corresponding feature points are p i , p j As described above, in the embodiment of the present application, only the feature points in a potential dynamic object mask area are calculated for the dynamic score. For each dynamic edge, the embodiment of the present application may consider the feature point p at the other end thereof. j At the position in the image, if p j In the background area, it is believed that the change in the length of the dynamic edge is mainly caused by p i The dynamic fraction s is caused by the movement of i Increase by 1; if p j If the object is also in a potential dynamic object mask area, then p i and p j may be in motion, so the dynamic score s i and j Both increase by 0.5.

[0066] If the dynamic score is greater than an empirical threshold, such as 5, it means that the object represented by the instance segmentation mask is indeed moving, and the ORB feature points in the entire dynamic target instance segmentation mask need to be removed. At this time, a data matrix can be reconstructed and managed to save the descriptors of the remaining ORB feature points after removing the dynamic feature points, so as to determine the retained feature points of the current frame image based on the descriptors.

[0067] Step S102, tracking the static target feature points, and solving the current frame camera pose of the camera according to the static target feature points during the tracking process, obtaining the number of projection points and tracking time, and selecting the first key frame according to the number of projection points and tracking time.

[0068] In other embodiments, after obtaining the static target feature points of the current frame image, the present application can track the static target feature points, and calculate the initial pose of the current frame camera based on the static target feature points during the tracking process. Specifically, the initial pose of the current frame camera can be, but is not limited to, calculated by a uniform linear motion model, that is, assuming that the camera motion from the previous frame to the current frame is consistent with the camera motion in the previous time period. Based on the remaining matching feature points, the initial pose of the current frame camera is calculated by minimizing the reprojection error.

[0069] Furthermore, after obtaining the initial pose of the current frame camera and determining that the current frame is a key frame, it is also necessary to manage and process the key frame, and optimize the initial pose of the current frame camera by maintaining the key frame co-view to determine the current SLAM system operation. Specifically, when the system positioning is not stable enough (i.e., the number of co-viewpoints between the current frame and the reference frame is small) and the quality of the current frame is high, the current frame can be inserted into the key frame co-view as a key frame, and the feature points of the current frame are projected to the world coordinate system according to the current frame camera pose and feature point depth, and restored to map points.

[0070] For example, the initial pose of the camera in the current frame is calculated by minimizing the reprojection error through the uniform linear motion model and the remaining matching feature points. The formula can be expressed as:

[0071] min R,t，X ∑ i ||u i -v i || (12)

[0072] Among them, R represents the rotation matrix of the initial position of the camera in the current frame, t represents the translation matrix of the initial position of the camera in the current frame, X represents the three-dimensional coordinates of the image feature point in the world coordinate system, and u i represents the image coordinates of the current frame matching feature points that participate in the calculation of minimizing the reprojection error, v i Indicates the image coordinates of the reprojected points of the previous frame’s matching feature points projected onto the current frame.

[0073] It should be noted that since formula (12) is a nonlinear optimization problem, the Gauss-Newton method can be used but is not limited to be used for solving it in the embodiments of the present application. Professional and technical personnel in this field can also solve it in other ways. This is only an illustrative explanation without further explanation.

[0074] Then the pose can be optimized by maintaining the keyframe co-view map. Specifically, the SLAM system can be used to maintain and manage a keyframe co-view map, where the keyframe co-view map can be, but is not limited to, composed of keyframes, map points, and co-view weighted edges. During system operation, when the interval between the previous reference frame and the current frame is more than 30 frames, or the number of co-view points between the current frame and the reference frame is less than the threshold Th_pt=30, it can be decided whether to insert the keyframe co-view map into the keyframe based on actual conditions. For example, to judge the quality of the current frame, if the current frame has at least Th ft=15 recoverable map points, and its pose satisfies the pose of the previous keyframe. Then the current frame can be inserted into the keyframe co-viewing area as a keyframe, and the feature points belonging to the keyframe can be projected to the world coordinate system according to the depth of the point and the camera pose of the frame, and restored to the map point. When there are more than 15 map points that can be observed together between two keyframes, a weighted edge can be constructed between the two keyframes, and the weight is the number of co-viewing points of the two keyframes.

[0075] Furthermore, when selecting the first key frame in the key frame co-viewing diagram, that is, determining whether the current frame is a key frame, the selection may be performed based on, but not limited to, the following conditions:

[0076] A keyframe can be deleted if it meets all of the following conditions: (1) more than 90% of the map points in the keyframe can be observed by the other five keyframes; (2) the keyframe is more than 10 keyframes away from the current frame. A map point can be deleted if it meets one of the following conditions: (1) the number of keyframes that can observe the map point is less than 25% of the theoretical number of keyframes that can observe the point; (2) after the map point is first created, the next two consecutive keyframes cannot observe the map point.

[0077] It should be noted that the various insertion conditions, deletion conditions or selection conditions involved in maintaining the key frame co-viewing image to optimize the posture in the embodiments of the present application are only for illustrative purposes. Professional and technical personnel in this field can choose or adjust them according to actual conditions when performing actual applications, and the embodiments of the present application do not impose specific restrictions.

[0078] Step S103, sampling pixels in the first key frame to obtain a first sampling result, and inputting the first sampling result into a pre-established NeRF map model for training to obtain a NeRF map construction result.

[0079] Optionally, in an embodiment of the present application, it also includes: selecting a second key frame in the historical key frame set; sampling pixels in the second key frame to obtain a second sampling result; and inputting the second sampling result into a pre-established NeRF map model for training.

[0080] As a possible implementation method, after selecting the first key frame of the current frame image, you can then use the ORB feature points to guide pixel sampling in the static area of ​​the current key frame to obtain a first sampling result; you can also randomly select a key frame in the historical key frame set, and then randomly sample a pixel in the static area of ​​the key frame, perform the above operation M times in total, obtain a complete pixel sampling result in the global optimization process, and use the complete pixel sampling result in the global optimization process as the second sampling result.

[0081] The first sampling result, the second sampling result and the calculated pose are input into the NeRF map model for training. After the visual SLAM system completes the tracking, the NeRF map model training ends to obtain the final NeRF map model parameters, that is, the dense NeRF map. The output result can obtain the camera pose and NeRF map construction result.

[0082] The specific process can be expressed as follows:

[0083] (1) Determine the static area in the current key frame based on the detected dynamic objects;

[0084] (2) In the static area of ​​the current key frame image, pixel sampling is performed according to the guidance of ORB feature points and input into the NeRF map model;

[0085] (3) Randomly select key frames from the historical key frame set and perform random pixel sampling, and input them into the NeRF map model;

[0086] (4) The NeRF map model is trained during the idle period of the mapping thread. After the visual SLAM system completes tracking, the NeRF map model training is terminated to obtain the final NeRF map model parameters, i.e., the dense NeRF map.

[0087] (5) Write the camera pose into a text file in the form of quaternion and save it, and save the NeRF map in the form of a model parameter file.

[0088] Furthermore, the embodiments of the present application can construct a visual SLAM system based on the Linux platform and implement it on the dynamic scene image sequence of the public RGB-D dataset TUM.

[0089] (1) Table 1 is a table of experimental data comparing the system positioning accuracy-absolute trajectory error (ATE) in dynamic scenes between the embodiment of the present application and two other open source RGB-D (ORB-SLAM2 and DS-SLAM) and the NeRF map construction method based on visual SLAM proposed in the embodiment of the present application, which can be shown as follows:

[0090] Table 1

[0091]

[0092]

[0093] Table 1 uses the RMSE statistical value of the absolute trajectory error ATE as an evaluation indicator. The smaller the value, the better the performance. As can be seen from the table, on the widely used TUM dataset, the positioning error of the NeRF map construction method based on visual SLAM proposed in the embodiment of the present application in four different dynamic scenes is smaller than that of the other two open source SLAM methods, which indicates that the NeRF map construction method based on visual SLAM proposed in the present application has better positioning accuracy in dynamic scenes.

[0094] (2) Furthermore, Figure 2 Schematic diagram of a dense NeRF map constructed in real time under the dynamic sequences fr3-w-xyz and fr2-desk-with-person according to an embodiment of the present application. Figure 2 As shown, it can be seen that the dense NeRF map constructed in real time under a dynamic sequence in the embodiment of the present application has a high-fidelity three-dimensional scene reconstruction effect.

[0095] The present application is described in detail below with reference to a specific embodiment.

[0096] Figure 3 FIG. 1 is a flowchart of a NeRF map construction method based on visual SLAM according to an embodiment of the present application. Figure 3 As shown:

[0097] Step 1: Get the current frame image of the camera for instance segmentation, and classify all instance segmentation results into static instances and dynamic instances.

[0098] Step 2: Extract features for the current frame image and remove dynamic feature points within the dynamic instance segmentation mask according to geometric constraints.

[0099] Step 3: During the tracking process, the camera pose of the current frame is solved based on the remaining static target feature points.

[0100] Step 4: Select key frames according to the number of projection internal points and tracking time during the tracking process, and manage and maintain the key frame co-visual graph.

[0101] Step 5: Sample pixels in each key frame and input them into the NeRF map model for training to obtain the NeRF map model parameters, i.e., the dense NeRF map.

[0102] Step 6: Output the results, namely the camera pose and NeRF map.

[0103] According to the NeRF map construction method based on visual SLAM proposed in the embodiment of the present application, the problem of mismatching in dynamic scenes can be overcome based on the dynamic point elimination method of instance segmentation and geometric constraints. A dense NeRF map is generated in real time for dynamic scenes. As a result, the problem of reduced positioning accuracy of existing SLAM technology in dynamic scenes is achieved. The image characteristics in dynamic scenes are used to perform instance segmentation and dynamic point elimination of geometric constraints. The efficiency is high while maintaining the simplicity of practical application. The map can eliminate the influence of dynamic targets, complete high-quality background reconstruction, and achieve better performance than in dynamic scenes, effectively meeting the positioning and mapping application requirements in different scenes. When the visual SLAM method in the related art is densely mapped in real time, due to the use of point clouds or octrees to represent the map, it is impossible to make reasonable predictions for the parts of the scene that are not observed or blocked by moving objects, and it is impossible to use the subsequent observation information of the system to optimize the parts that have been mapped, resulting in a large number of holes and missing parts in the reconstruction results. Therefore, it is impossible to achieve high-fidelity real-time dense reconstruction, and the applicability in actual scenes is not strong.

[0104] Next, the NeRF map construction device based on visual SLAM proposed in accordance with the embodiment of the present application is described with reference to the accompanying drawings.

[0105] Figure 4 It is a structural diagram of a NeRF map construction device based on visual SLAM in an embodiment of the present application.

[0106] like Figure 4 As shown, the NeRF map construction device 10 based on visual SLAM includes: an acquisition module 100, a first selection module 200 and a construction module 300.

[0107] The acquisition module 100 is used to acquire static target feature points of the current frame image of the camera in a dynamic scene.

[0108] The first selection module 200 is used to track the static target feature points, so as to solve the current frame camera pose of the camera according to the static target feature points during the tracking process, obtain the number of projection points and tracking time, and select the first key frame according to the number of points in the projection and the tracking time.

[0109] The construction module 300 is used to sample pixels in the first key frame according to the ORB feature guidance to obtain the first key frame sampling result, and input the first key frame sampling result into a pre-established NeRF map model for training to obtain NeRF map model parameters to obtain NeRF map construction results.

[0110] Optionally, in one embodiment of the present application, the acquisition module 100 includes: a segmentation unit and a determination unit.

[0111] The segmentation unit is used to perform instance segmentation on the current frame image according to the configuration file to obtain a target instance segmentation mask, so as to divide the target instance segmentation mask into a dynamic instance segmentation mask and a static instance segmentation mask.

[0112] A determination unit is used to calculate the dynamic scores of the objects represented by the dynamic instance segmentation mask and the static instance segmentation mask respectively based on the target geometric constraints, so as to determine the static target feature points of the current frame image based on the dynamic scores and the empirical threshold.

[0113] Optionally, in one embodiment of the present application, it also includes: a first training module and a storage module.

[0114] The segmentation module is used to perform instance segmentation on predefined instances according to predefined target categories before performing instance segmentation on the current frame image according to the configuration file, so as to obtain predefined dynamic instances and predefined static instances.

[0115] The storage module is used to store the detection confidences and corresponding semantic colors of the predefined dynamic instances and the predefined static instances into a file to obtain a configuration file.

[0116] Optionally, in one embodiment of the present application, it also includes: a second selection module, a sampling module and a training module.

[0117] The second selection module is used to select a second key frame from the historical key frame set.

[0118] The sampling module is used to sample pixels in the second key frame to obtain a second sampling result.

[0119] The training module is used to input the second sampling result into a pre-established NeRF map model for training.

[0120] It should be noted that the aforementioned explanation of the embodiment of the NeRF map construction method based on visual SLAM is also applicable to the NeRF map construction device based on visual SLAM in this embodiment, and will not be repeated here.

[0121] According to the NeRF map construction device based on visual SLAM proposed in the embodiment of the present application, the problem of mismatching in dynamic scenes can be overcome based on the dynamic point elimination method of instance segmentation and geometric constraints. A dense NeRF map is generated in real time for dynamic scenes. As a result, the problem of reduced positioning accuracy of existing SLAM technology in dynamic scenes is achieved. The image characteristics in dynamic scenes are used to perform instance segmentation and dynamic point elimination of geometric constraints. The efficiency is high while maintaining the simplicity of practical application. The map can eliminate the influence of dynamic targets, complete high-quality background reconstruction, and achieve better performance than in dynamic scenes, effectively meeting the positioning and mapping application requirements in different scenes. When the visual SLAM method in the related art is densely mapped in real time, due to the use of point clouds or octrees to represent the map, it is impossible to make reasonable predictions for the parts of the scene that are not observed or blocked by moving objects, and it is impossible to use the subsequent observation information of the system to optimize the parts that have been mapped, resulting in a large number of holes and missing parts in the reconstruction results. Therefore, it is impossible to achieve high-fidelity real-time dense reconstruction, and the applicability in actual scenes is not strong.

[0122] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0123] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0124] When the processor 502 executes the program, the NeRF map construction method based on visual SLAM provided in the above embodiment is implemented.

[0125] Furthermore, the electronic device further comprises:

[0126] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0127] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0128] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0129] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0130] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0131] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0132] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned NeRF map construction method based on visual SLAM.

[0133] The embodiment of the present application also provides a computer program product, including a computer program, which can run computer instructions. When the computer instructions are executed by a processor, the NeRF map construction method based on visual SLAM provided in the embodiment of the present application is implemented.

[0134] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0135] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0136] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0137] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0138] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0139] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0140] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0141] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A NeRF map construction method based on visual SLAM, characterized in that: The following steps are involved: In a dynamic scene, obtain the static target feature points of the current frame image of the camera; Tracking the static target feature points, so as to solve the current frame camera pose of the camera according to the static target feature points during the tracking process, obtain the number of projection points and the tracking time, and select the first key frame according to the number of points in the projection and the tracking time; In the first key frame, pixel sampling is performed according to the ORB feature guidance to obtain a first key frame sampling result, and the first key frame sampling result is input into a pre-established NeRF map model for training to obtain NeRF map model parameters to obtain a NeRF map construction result.

2. The method according to claim 1, characterized in that The step of obtaining static target feature points of the current frame image of the camera includes: Performing instance segmentation on the current frame image according to the configuration file to obtain a target instance segmentation mask, so as to divide the target instance segmentation mask into a dynamic instance segmentation mask and a static instance segmentation mask; Based on the target geometric constraint condition, the dynamic scores of the objects represented by the dynamic instance segmentation mask and the static instance segmentation mask are calculated respectively, so as to determine the static target feature points of the current frame image based on the dynamic scores and an empirical threshold.

3. The method according to claim 2, characterized in that Before performing instance segmentation on the current frame image according to the configuration file, the method includes: Perform instance segmentation on the predefined instances according to the predefined target categories to obtain predefined dynamic instances and predefined static instances; The detection confidences and corresponding semantic colors of the predefined dynamic instances and the predefined static instances are stored in a file to obtain the configuration file.

4. The method according to claim 1, characterized in that: Also includes: Selecting a second key frame from the historical key frame set; Perform pixel sampling in the static area of ​​the second key frame to obtain a second sampling result; The second sampling result is input into the pre-established NeRF map model for training.

5. A NeRF map construction device based on visual SLAM, characterized in that: include: An acquisition module is used to acquire static target feature points of the current frame image of the camera in a dynamic scene; A selection module is used to track the static target feature points, so as to solve the current frame camera pose of the camera according to the static target feature points during the tracking process, obtain the number of projection points and the tracking time, and select the first key frame according to the number of points in the projection and the tracking time; A construction module is used to sample pixels in the first key frame according to the ORB feature guidance to obtain a first key frame sampling result, and input the first key frame sampling result into a pre-established NeRF map model for training to obtain NeRF map model parameters to obtain a NeRF map construction result.

6. The device according to claim 5, characterized in that The acquisition module comprises: a segmentation unit, configured to perform instance segmentation on the current frame image according to a configuration file to obtain a target instance segmentation mask, so as to divide the target instance segmentation mask into a dynamic instance segmentation mask and a static instance segmentation mask; The determination unit is used to calculate the dynamic scores of the objects represented by the dynamic instance segmentation mask and the static instance segmentation mask respectively based on the target geometric constraint condition, so as to determine the static target feature points of the current frame image based on the dynamic scores and the empirical threshold.

7. The device according to claim 6, characterized in that Also includes: A segmentation module, used for performing instance segmentation on predefined instances according to predefined target categories before performing instance segmentation on the current frame image according to the configuration file, to obtain predefined dynamic instances and predefined static instances; The storage module is used to store the detection confidence and corresponding semantic color of the predefined dynamic instance and the predefined static instance into a file to obtain the configuration file.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the NeRF map construction method based on visual SLAM as described in any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the NeRF map construction method based on visual SLAM as described in any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed, it is used to implement the NeRF map construction method based on visual SLAM as described in any one of claims 1-4.

Images