Dynamic environment monocular multi-object slam method based on instance segmentation and three-dimensional reconstruction

Abstract

The application relates to the technical field of automatic driving and computer vision, in particular to a dynamic environment monocular multi-object SLAM method based on instance segmentation and three-dimensional reconstruction, which comprises the following steps: acquiring an image frame sequence collected in the driving process of an automatic driving vehicle; performing feature extraction on each frame image; using an instance segmentation network to obtain dynamic objects and static objects in the image on the key frame image; marking and removing the influence of dynamic feature points; obtaining the current frame dynamic feature points by using Bayesian probability propagation, and removing the dynamic feature points; obtaining the mask, the boundary box and the sparse 3D point cloud of the extracted multiple static objects, and obtaining the initial pose of the multiple static objects by using PCA; after multi-frame observation and sufficient data, the appearance implicit code and the optimized pose of the multiple static objects are obtained by using a three-dimensional reconstruction network, and three-dimensional reconstruction is performed on the multiple static objects. The application can reduce the absolute trajectory error of a traditional SLAM method in a dynamic environment, improve the system pose estimation accuracy and robustness, and improve the semantic degree of a map.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving and computer vision, and particularly relates to a dynamic environment monocular multi-object SLAM method based on instance segmentation and three-dimensional reconstruction. BACKGROUND

[0002] In recent years, with the vigorous development of the field of automatic driving and computer vision, simultaneous localization and mapping (SLAM) has become a key research field in the field, and positioning and semantic mapping in a dynamic scene is a popular research direction, and is widely applied in the fields of automatic driving, indoor and outdoor robots, etc.

[0003] Most of the current visual SLAM methods are based on the assumption that the observed environment is static, such as the classic SLAM systems: ORB-SLAM2, Vins-Mono, etc. However, the real environment will inevitably contain dynamic objects. The presence of dynamic objects will cause the traditional SLAM method to have insufficient feature points for pose estimation due to the dynamic objects occupying most of the field of view or being blocked, etc., thereby causing the system to drift or lose tracking, i.e., the positioning accuracy and robustness of the system in a dynamic scene decrease. Moreover, the current monocular SLAM method constructs a map that is too sparse, and cannot construct a semantic SLAM map. This will further lead to insufficient semantic information of the map. Some SLAM systems apply semantic segmentation and other deep learning methods to only exclude prior dynamic objects existing in the dynamic environment, and do not fully utilize the semantic information existing in the environment, i.e., the semantic information of the environment is not fully utilized.

[0004] Therefore, it is of great value to combine instance segmentation and three-dimensional reconstruction technology and apply it to the monocular SLAM system to improve the positioning accuracy and robustness of the SLAM system. SUMMARY

[0005] In order to overcome the technical defects of the current monocular SLAM method constructing a map that is too sparse, being unable to construct a semantic SLAM map, and insufficient utilization of environmental semantic information, the present application provides a dynamic environment monocular multi-object SLAM method based on instance segmentation and three-dimensional reconstruction.

[0006] The present application discloses a dynamic environment monocular multi-object SLAM method based on instance segmentation and three-dimensional reconstruction, comprising the following steps:

[0007] Step 1, acquiring an image frame sequence collected in the driving process of an automatic driving vehicle, performing feature extraction on each frame of image and classifying the same into key frames and ordinary frames;

[0008] Step 2: Use an instance segmentation network to obtain masks for both static and dynamic objects, and label and remove feature points within dynamic objects. The sub-steps are as follows:

[0009] Step 2.1: Use the trained SparseInst instance segmentation network to segment keyframes into instances, and obtain a set of prior semantic masks of size n×h×w, containing the corresponding training scores n×1.

[0010] Step 2.2: The prior semantic mask set includes prior dynamic object masks and pre-defined static object masks. The prior dynamic object masks are dynamic object masks of size a×h×w. The dynamic object masks are combined into dynamic object masks of size h×w. The pre-defined static object masks are dynamic object masks of size b×h×w. Based on the b×1 score output by the SparseInst instance segmentation network, c object masks with a corresponding score threshold greater than α are selected from the dynamic object masks. That is, static object masks of size c×h×w are selected. The OPENCV algorithm is used to calculate the two-dimensional bounding box corresponding to each static object mask. The resulting two-dimensional bounding box set is c×h×w.

[0011] Step 2.3: Extract feature points from the keyframe image, mark the initial movement probability of feature points within the dynamic object mask in the current keyframe image, update it to bel(x0), update the initial movement probability of the remaining static feature points to 1-bel(x0), and remove the dynamic feature points in the keyframe image matching feature point set.

[0012] Step 3: Use Bayesian probability propagation to propagate the dynamic feature points within the mask of dynamic objects in the keyframe image to the ordinary frames. The sub-steps are as follows:

[0013] Step 3.1: Extract ORB features from the new frame image and calculate descriptors. Perform feature matching between the current frame I and the previous frame image frame F, from which dynamic feature points have been removed. If the previous frame is a keyframe, then directly use the keyframe from which dynamic feature points have been removed for feature matching.

[0014] Step 3.2: After matching the two frames of images, obtain the camera pose changes R and t of the two frames. Project the static and dynamic feature points in the previous frame image into the current camera frame and calculate the matching pixel error d of all feature points.

[0015] Step 3.3: Match pixel differences based on feature points Update the movement probability of the corresponding feature point, i.e., update the feature point movement probability using the following formula:

[0016] (1)

[0017] Step 3.4, update the state of all feature points according to the initial movement probability bel(x0) in the key frame image and the measurement probability in the current image frame, wherein the measurement probability is the movement probability update shown in formula (1), and the movement probability state update is expressed as a Bayesian filter:

[0018] (2)

[0019] According to the Bayesian rule and conditional independence, the current frame observation z t only depends on the current state m t , and η is a normalization constant, wherein the state prediction can be obtained by formula (3):

[0020] (3)

[0021] The final feature point state is updated by formula (4), and the threshold and are used to determine whether a single feature point is dynamic or static:

[0022] (4)

[0023] After the determination, the dynamic feature points are marked in the current image frame, and the influence of the dynamic feature points on the feature matching in the adjacent frame is eliminated, and the current image frame after processing is matched with the next frame image;

[0024] Step 4, create an object, and the sub-steps are as follows:

[0025] Step 4.1, obtain c static object objects segmented in the key frame image, and calculate the three-dimensional sparse point cloud P corresponding to each static object, detect whether the corresponding object has been created, if the static object corresponding to the three-dimensional sparse point cloud P in the current key frame has been created, update the created static object, if not, create a static object object; wherein the process of creating a static object object is: using the current static object mask M and the two-dimensional bounding box B corresponding thereto, the three-dimensional sparse point cloud P, the initial pose T co,0 instantiation, create a static object object I n :

[0026] (5)

[0027] When the number of three-dimensional sparse point clouds P on the static object meets the set value, the initial pose T co,0 of the current static object is calculated by using the PCA algorithm according to formula (5):

[0028] ​(6);

[0029] When the number of three-dimensional sparse point clouds P on the static object does not satisfy the set value, go to the next static object;

[0030] Step 4.2, sequentially loop to detect c static object objects in the key frame image, and create a plurality of static objects according to the instantiation process in step 4.1;

[0031] Step 5, three-dimensional reconstruction of multiple static objects, the sub-steps are as follows:

[0032] Step 5.1, pre-training the DeepSDF three-dimensional reconstruction network using the ShapeNet point cloud dataset, at this time the trained object is a predefined static object, and a 32-dimensional or 64-dimensional training model can be trained, and the specific dimension is the same as the predefined dimension;

[0033] Step 5.2, when processing each key frame image, a plurality of static objects are detected by the SparseInst instance segmentation network, and the three-dimensional sparse point cloud P and the initial pose T on each static object are updated according to steps 4.1 and 4.2 co,0 ;

[0034] Step 5.3, when the number of observation frames of a single static object reaches γ and the number of three-dimensional sparse point clouds P satisfies the set value, the current static object mask M and the corresponding two-dimensional bounding box B, three-dimensional sparse point cloud P, initial pose T co,0 are input into the DeepSDF three-dimensional reconstruction network;

[0035] Step 5.4, using the 32-dimensional or 64-dimensional training model to infer the 32-dimensional or 64-dimensional implicit encoding Z of each static object and the optimized object pose T co ;

[0036] Step 5.5, using the Pangolin visualization drawing library, using the static object implicit encoding Z to perform three-dimensional reconstruction on each static object, to obtain the three-dimensional reconstruction result of the plurality of static objects, that is, to construct a multi-object semantic map.

[0037] The technical scheme provided by the application has the following advantages compared with the prior art: for a monocular camera model, the influence of dynamic objects existing in the key frame is eliminated by using the key frame instance segmentation result, and the dynamic feature points existing in the last frame are propagated to the current frame and their influence is eliminated by using Bayesian probability propagation, and a multi-object semantic map is constructed by combining the static objects obtained by instance segmentation according to the key frame and three-dimensional reconstruction technology; the application can reduce the absolute pose error and the relative pose error of the traditional monocular SLAM method, improve the accuracy and robustness of the pose estimation of the current monocular SLAM method, and enhance the semantic nature of the constructed map. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced in the following. Obviously, those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0040] Figure 1 A flow chart of the dynamic environment monocular multi-object SLAM method based on instance segmentation and three-dimensional reconstruction according to the present application;

[0041] Figure 2 A block diagram of a SLAM system involved in the dynamic environment monocular multi-object SLAM method according to the present application;

[0042] Figure 3 A flow chart of a three-dimensional reconstruction module in the dynamic environment monocular multi-object SLAM method based on instance segmentation and three-dimensional reconstruction according to the present application;

[0043] Figure 4 A comparison diagram of the running results of the method according to the present application and the real trajectory on the TUM dynamic dataset w_xyz sequence;

[0044] Figure 5 A comparison diagram of the running results of the method according to the present application and the ORB-SLAM2 algorithm and the real trajectory on the TUM dynamic dataset w_xyz sequence;

[0045] Figure 6 A multi-object reconstruction experiment diagram of the dynamic environment monocular multi-object SLAM method based on instance segmentation and three-dimensional reconstruction according to the present application. DETAILED DESCRIPTION

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced in the following. Obviously, those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0047] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of the present application, not all embodiments.

[0048] The specific embodiments of the present application will be described below with reference to the accompanying drawings. Figure 1 to the accompanying drawings Figure 6Detailed description is made to specific embodiments of the present application.

[0049] As a certain embodiment of the present application, a dynamic environment monocular multi-object SLAM method based on instance segmentation and three-dimensional reconstruction is disclosed, which involves a SLAM system including an instance segmentation network module, a probability propagation module, an object instantiation module and a three-dimensional reconstruction module, wherein the instance segmentation network module is used to obtain dynamic and static object masks for key frames using an instance segmentation network, specifically, the instance segmentation network module is used as a front-end image tracking, and a SparseInst instance segmentation network needs to be pre-trained, the SparseInst network is pre-trained using a coco dataset to obtain a trained model, and the SparseInst instance segmentation network is combined with an ORB-SLAM2 algorithm; the probability propagation module is used to obtain dynamic feature points on a normal frame using Bayesian probability propagation for feature points in a dynamic object mask obtained by the instance segmentation network module; the object instantiation module is used to extract an image mask, a two-dimensional bounding box and a three-dimensional sparse point cloud based on a static object obtained by the instance segmentation network module, and obtain an initial pose of the object using a PCA algorithm; the three-dimensional reconstruction module is used to obtain an implicit encoding and an optimized pose of the object based on sufficient observed data after obtaining observation data of multiple key frame images, and construct a static object model; specifically, the dynamic environment monocular multi-object SLAM method includes the following steps:

[0050] Step 1, obtaining image frame sequences collected during driving of an autonomous vehicle, performing feature extraction on each frame of image and classifying the frame of image into a key frame and a normal frame;

[0051] Step 2, obtaining dynamic and static object masks using an instance segmentation network, and marking and removing feature points in a dynamic object, the sub-steps are as follows:

[0052] Step 2.1, performing instance segmentation on a key frame using a trained SparseInst instance segmentation network to obtain a prior semantic mask set, the size of which is n x h x w, and the set contains a corresponding training score n x 1;

[0053] Step 2.2, the prior semantic mask set includes a prior dynamic object mask and a pre-set static object mask, the prior dynamic object mask is a dynamic object mask with a size of a x h x w, the dynamic object masks are combined into a dynamic object mask with a size of h x w; the pre-set static object mask is a dynamic object mask with a size of b x h x w, according to the b x 1 score output by the SparseInst instance segmentation network, c object masks in the dynamic object mask with a score threshold greater than a are selected, that is, a static object mask with a size of c x h x w is selected, and the two-dimensional bounding box corresponding to each static object mask is calculated by using the OPENCV algorithm, and the obtained two-dimensional bounding box set is c x h x w;

[0054] Step 2.3, the feature points in the key frame image are extracted, the initial moving probability of the feature points in the dynamic object mask in the current key frame image is marked and updated to bel(x0), and the initial moving probability of the remaining static feature points is updated to 1-bel(x0), and the dynamic feature points in the key frame image matching feature point set are removed;

[0055] Step 3, since the SparseInst instance segmentation network is difficult to run in real time when applied to the SLAM system, and the application to all image frames will cause waste of computer resources, the dynamic feature points in the dynamic object mask in the key frame image are propagated into the ordinary frame by using the Bayesian probability propagation, and the sub-steps are as follows:

[0056] Step 3.1, the ORB features in a new frame of image are extracted and the descriptors are calculated, the feature matching is performed between the current frame I and the image frame F from which the dynamic feature points have been removed, and if the last frame is a key frame, the feature matching is directly performed by using the key frame from which the dynamic feature points have been removed;

[0057] Step 3.2, after the matching of the two frames of images, the camera pose change R and t of the two frames are obtained, the static feature points and the dynamic feature points in the last frame image are projected into the current camera frame, and the matching pixel error d of all feature points is calculated;

[0058] Step 3.3, according to the feature point matching pixel difference the moving probability of the corresponding feature point is updated, that is, the feature point moving probability is updated by using the following formula:

[0059] (1),

[0060] wherein d1 and d2 are two upper and lower threshold values of the feature point matching pixel difference set by the method, the moving probability of the feature point is determined according to d1, d2 and the feature point matching pixel difference d, and z t is the state of the feature point observed by the current frame, and m t is the actual state of the feature point in the current frame, that is, dynamic or static.

[0061] Step 3.4, state update for all feature points, according to the initial movement probability bel(x0) in the key frame image and the measurement probability in the current image frame, wherein the measurement probability is the movement probability update shown in formula (1), the movement probability state update is expressed as a Bayesian filter:

[0062] (2),

[0063] Wherein, bel(m t ) is the movement state probability of a single feature point in the current frame image, z 1:t is the observation state of the feature point from the first frame to the current frame image, z 1:t-1 is the observation state of the feature point from the first frame image to the previous frame image, m0 is the feature point initialization state;

[0064] According to the Bayesian rule and conditional independence, the current frame observation z t only depends on the current state m t , and η is a normalization constant, wherein the state prediction can be obtained from formula (3):

[0065] (3),

[0066] Wherein the state transition probability P(M t | m t-1 )=0.95, that is, the probability of changing the state of the dynamic feature point after the next frame, and the same applies to the static feature point, and bel(m t-1 ) is the movement state probability of the corresponding feature point in the previous frame image; the final feature point state is updated by formula (4), and the threshold and are used to determine whether a single feature point is dynamic or static:

[0067] (4)

[0068] Wherein, P(m t ) is the feature point state bel(m t ) obtained from formula (2), θ d is the lower limit value of the dynamic feature point set by the method, and the dynamic feature point is higher than the value; θ s is the upper limit value of the static feature point set by the method, and the static feature point is lower than the value; the intermediate range is set as the position state; after the judgment, the dynamic feature points are marked in the current image frame, and the influence of the dynamic feature points on the feature matching in the adjacent frames is eliminated, and the current image frame after processing is matched with the next frame image;

[0069] Step 4: To improve the semantic nature of the SLAM map and the utilization rate of semantic information in the environment, object objects are created for static objects existing in the environment. The sub-steps for creating object objects are as follows:

[0070] Step 4.1: Obtain c static object segments from the keyframe image and calculate the 3D sparse point cloud P corresponding to each static object. Check if the corresponding object has been created. If it has been created, then extract the 3D sparse point cloud corresponding to the static object in the current keyframe. Update the static objects that have already been created; if no static object has been created, then create one. The process of creating a static object involves: using the current static object mask M and its corresponding 2D bounding box B, 3D sparse point cloud P, and initial pose T. co,0 Instantiate and create a static object I. n :

[0071] (5)

[0072] When the number of 3D sparse point clouds P on the static object meets the set value, the PCA algorithm is used to calculate the initial pose T of the current static object using formula (5). co,0 :

[0073] (6);

[0074] Where s is the camera scale; R co Let t be the rotation matrix of the static object in the current frame image. co Sim(3) is the translation vector of the current frame image of the static object. Sim(3) is the similarity transformation to be solved based on the matching points, that is, to solve the rotation matrix, translation vector and scale between the two coordinate systems. Here it is the similarity transformation between the object coordinate system and the camera coordinate system. When the number of 3D sparse point clouds P on the static object does not meet the set value, then switch to the next static object.

[0075] Step 4.2: Detect c static objects in the keyframe image in sequence, and create multiple static objects according to the instantiation process in Step 4.1;

[0076] Step 5: 3D reconstruction of multiple static objects, with the following sub-steps:

[0077] Step 5.1: Use the ShapeNet point cloud dataset to pre-train the DeepSDF 3D reconstruction network. At this time, the object being trained is a predefined static object, and a 32-dimensional or 64-dimensional training model can be obtained. The specific dimension can be the same as the pre-set dimension.

[0078] Step 5.2, when processing each key frame image, a plurality of static objects are detected by the SparseInst instance segmentation network, and the three-dimensional sparse point cloud P on each static object and the initial pose T are updated according to steps 4.1 and 4.2 co,0 ;

[0079] Step 5.3, when the number of single static object observation frames reaches γ and the number of three-dimensional sparse point clouds P meets the set value, the current static object mask M and the two-dimensional bounding box B corresponding thereto, the three-dimensional sparse point cloud P, the initial pose T co,0 are transmitted into the DeepSDF three-dimensional reconstruction network;

[0080] Step 5.4, the 32-dimensional or 64-dimensional implicit encoding Z of each static object and the optimized object pose T are inferred using a 32-dimensional or 64-dimensional training model co ;

[0081] Step 5.5, using the Pangolin visualization drawing library, the three-dimensional reconstruction of each static object is performed using the static object implicit encoding Z, so as to obtain the three-dimensional reconstruction result of the plurality of static objects, that is, to construct a multi-object semantic map.

[0082] From Figure 4 it can be seen that the multi-object semantic map constructed by the method of the present application has a very small gap from the real trajectory in camera pose estimation; from Figure 5 it can be seen that the method of the present application is an improvement based on the ORB-SLAM2 algorithm, and the camera pose estimation is more accurate; from Figure 6 it can be seen that the method of the present application has high feasibility in constructing a multi-object semantic map.

[0083] The above is only a specific embodiment of the present application, which enables those skilled in the art to understand or implement the present application. Although detailed description is made with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions described in the foregoing embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solutions of the embodiments, and they should be covered in the protection scope of the claims.

Claims

1. A dynamic environment monocular multi-object SLAM method based on instance segmentation and three-dimensional reconstruction, characterized in that, The method comprises the following steps: Step 1, acquiring image frame sequences collected in the driving process of an autonomous vehicle, performing feature extraction on each frame image and classifying the frame images into key frames and ordinary frames; Step 2, obtaining dynamic and static object masks using an instance segmentation network, and marking and removing feature points in the dynamic objects, the sub-steps being as follows: Step 2.1, performing instance segmentation on the key frames using the trained SparseInst instance segmentation network to obtain a prior semantic mask set with a size of n x h x w, which contains n x 1 training scores; Step 2.2, the prior semantic mask set includes a prior dynamic object mask and a pre-set static object mask, the prior dynamic object mask is a dynamic object mask with a size of a x h x w, and the dynamic object mask set is combined into a dynamic object mask with a size of h x w; The pre-set static object mask is a dynamic object mask with a size of b x h x w, according to the score b x 1 output by the SparseInst instance segmentation network, c object masks in the dynamic object mask with a score threshold greater than a are selected, that is, a static object mask with a size of c x h x w is selected, and the two-dimensional boundary box corresponding to each static object mask is calculated using the OPENCV algorithm, and a two-dimensional boundary box set with a size of c x h x w is obtained; Step 2.3, extracting feature points in the key frame image, marking the initial movement probability of the feature points in the dynamic object mask in the current key frame image, updating the initial movement probability of the remaining static feature points to 1-bel(x0), and removing the dynamic feature points in the key frame image matching feature point set; Step 3, propagating the dynamic feature points in the dynamic object mask in the key frame image into the ordinary frame using Bayesian probability propagation, the sub-steps being as follows: Step 3.1, extracting ORB features in a new frame of image and calculating descriptors, performing feature matching on the current frame I and the image frame F from which the dynamic feature points have been removed, and if the previous frame is a key frame, directly performing feature matching using the key frame from which the dynamic feature points have been removed; Step 3.2, obtaining the camera pose change R and t after matching the two frames of images, projecting the static feature points and the dynamic feature points in the previous frame of image into the current camera frame, and calculating the matching pixel error d of all feature points; Step 3.3, updating the movement probability of the corresponding feature points according to the feature point matching pixel error d, that is, updating the movement probability of the feature points using the following formula: （1）， Wherein, d1, d2 are two upper and lower limit thresholds of the feature point matching pixel difference set by the method, the movement probability of the feature point is determined according to d1, d2 and the feature point matching pixel difference d, z t The state of the feature point observed in the current frame, m t The actual state of the feature point in the current frame, that is, dynamic or static Step 3.4, updating the state of all feature points according to the initial movement probability bel(x0) in the key frame image and the measurement probability in the current image frame, wherein the measurement probability is the movement probability update shown in formula (1), and the movement probability state update is represented as a Bayesian filter: （2）， wherein bel(m t ) is the current frame image single feature point movement state probability, z 1:t is the first frame to the current frame image feature point observation state, z 1:t-1 is the first frame image to the last frame image feature point observation state, and m0 is the feature point initialization state. According to Bayes rule and conditional independence, the current frame observation z t Depend only on the current state m t η is a normalization constant, where the state prediction From equation (3), we have: （3）， where state transition probability P(M t ∣m t-1 )=0.95, i.e. the probability of dynamic feature points changing state after the next frame, and the same for static feature points, bel(m t-1 ) is the moving state probability of the corresponding feature points in the previous frame image. The final feature point state is updated using formula (4), and the threshold and are used to determine whether a single feature point is dynamic or static: （4）, wherein P(m t ) is the feature point state bel(m t ) obtained from equation (2), θ d is a lower limit value of dynamic feature points set by the method, and a value higher than the value is a dynamic feature point; θ s is an upper limit value of static feature points set by the method, and a value lower than the value is a static feature point; an intermediate range is set as a position state; after the judgment is completed, the dynamic feature points are marked in the current image frame, and the influence of the dynamic feature points on the feature matching in the adjacent frame is eliminated, and the current image frame after the processing is matched with the next frame image. Step 4, creating object objects, the sub-steps being as follows: Step 4.1, obtain c static object objects segmented in the key frame image, and calculate the three-dimensional sparse point cloud P corresponding to each static object, detect whether the corresponding object has been created, if it has been created, update the three-dimensional sparse point cloud P corresponding to the static object in the current key frame to the created static object, if not, create a static object object; wherein the process of creating a static object object is: using the current static object mask M and the two-dimensional bounding box B corresponding thereto, the three-dimensional sparse point cloud P, the initial pose T Update to the created static object, if not created, create a static object object; wherein the process of creating a static object object is: using the current static object mask M and the two-dimensional bounding box B corresponding thereto, the three-dimensional sparse point cloud P, the initial pose T co,0 Instantiation is carried out, and a static object object I is created n : （5） When the number of three-dimensional sparse point clouds P on the static object meets a set value, the initial pose T of the current static object is calculated using the PCA algorithm and formula (5) co,0 : （6）； where s is the scale of the camera; R co is the rotation matrix of the current frame image of the static object, t co is the translation vector of the current frame image of the static object, Sim(3) is to solve the similarity transformation according to the matching points, that is, to solve the rotation matrix, the translation vector and the scale between two coordinate systems; when the number of the three-dimensional sparse point cloud P on the static object does not satisfy the set value, the next static object is turned to. Step 4.2, sequentially and circularly detecting c static object objects in the key frame image, and creating multiple static objects according to the instantiation process in step 4.1; Step 5, three-dimensional reconstruction of multiple static objects, the sub-steps being as follows: Step 5.1, pre-training the DeepSDF three-dimensional reconstruction network using the ShapeNet point cloud dataset, at this time the trained object is a pre-defined static object, and a 32-dimensional or 64-dimensional training model is obtained, and the specific dimension is the same as the pre-set dimension; Step 5.2, when processing each key frame image, multiple static objects are detected by the SparseInst instance segmentation network, and the three-dimensional sparse point cloud P on each static object and the initial pose T are updated according to steps 4.1 and 4.2 co,0 ; Step 5.3, when the number of single static object observation frames reaches γ and the number of three-dimensional sparse point clouds P meets the set value, the current static object mask M and the two-dimensional bounding box B corresponding thereto, the three-dimensional sparse point cloud P, the initial pose T co,0 into the DeepSDF three-dimensional reconstruction network; Step 5.4, infer the 32-dimensional or 64-dimensional latent code Z of each static object and the optimized object pose T using the trained 32-dimensional or 64-dimensional model co ; Step 5.5, using the Pangolin visualization drawing library, using the static object implicit encoding Z to perform three-dimensional reconstruction on each static object, so as to obtain the three-dimensional reconstruction results of a plurality of static objects, that is, to construct a multi-object semantic map.

Images