Fast slam method and system for indoor dense point cloud in multi-transparent object environment

Abstract

The application discloses a kind of indoor dense point cloud fast SLAM method and system under the environment of multiple transparent objects, and the method comprises: collecting RGB picture to obtain frame by processing;Carrying out ORB feature point collection to obtain key point coordinate;Optimal pose is solved to the feature point matching of front and rear frames, and current frame Frame is sent into identification thread when being key frame, whether there is transparent object is judged using YoloV4-tiny, if there is, then send into DZ-SD-DIV module and extract main profile, and clustering is carried out to obtain voxel depth information;While key frame is input into fine profile extraction module and extracts profile, and the profile is input into GNN-SVO neural network and outputs predicted depth information;Two kinds of information are carried out depth information fusion, and then the fused key frame is transmitted into mapping thread.The method is carried out in the environment of multiple transparent objects or multiple black objects, and fast real-time synchronous positioning and dense mapping.

Description

Technical Field

[0001] This invention relates to the field of indoor scene technology, and in particular to a fast SLAM method and system for dense point clouds in indoor environments with multiple transparent objects. Background Technology

[0002] Currently, 3D reconstruction of transparent objects in dense point clouds is a relatively complex problem. This is because there are complex optical effects between transparent objects and light, such as reflection and refraction, which are related to the viewing angle. This makes it impossible to directly apply traditional reconstruction algorithms to transparent objects.

[0003] A traditional method for 3D reconstruction of transparent objects relies on the distortion of known or unknown calibration patterns caused by light refraction to infer and recover the object's geometric information. This method is only applicable to single refractive surfaces or simple parametric surface models and cannot be generalized to a wider range of object categories.

[0004] Another traditional method for 3D reconstruction of transparent objects is reverse rendering. This involves forward rendering an initial rough model based on reasonable assumptions, comparing the generated image with the acquired image, and linking the imaging residuals to surface deformation. The rough model is then progressively optimized. This method typically requires altering the internal properties of the target object to allow the imaging system to acquire the inherent geometric information of the transparent object's surface, but its reconstruction accuracy is not high enough.

[0005] Another traditional method for 3D reconstruction of transparent objects is tomography, which involves immersing the object in a solution with the same refractive index and using chemical pigments to color the solution and distinguish it from the object. If the transparent object is a homogeneous medium, theoretically, light will not refract inside the device after passing through it. The target object can be reconstructed by acquiring images from 360° using tomography. The drawback of this method is that it requires accurate matching of the relative refractive indices of the object and the solution, making it unsuitable for general applications.

[0006] The traditional methods described above cannot achieve real-time 3D reconstruction and require specific environments. Furthermore, these methods only achieve reconstruction, not localization. Current methods supporting dense SLAM reconstruction also exhibit anomalies in scenes with transparent objects. This is because the infrared light from the RGBD camera is absorbed or reflected by glass, dark materials, etc., preventing the infrared receiver from acquiring depth information. Summary of the Invention

[0007] The present invention aims to at least partially solve one of the technical problems in the related art.

[0008] Therefore, the first objective of this invention is to propose a fast SLAM method for indoor dense point clouds in environments with multiple transparent objects.

[0009] Therefore, the second objective of this invention is to propose a fast SLAM system for dense point clouds in indoor environments with multiple transparent objects.

[0010] Therefore, a third objective of the present invention is to provide a computer device.

[0011] Therefore, a fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0012] To achieve the above objectives, a first aspect of the present invention proposes a fast SLAM method for indoor dense point clouds in environments with multiple transparent objects, comprising the following steps: Step S1, acquiring RGB images and depth information images, and processing the RGB images and depth information images to obtain a frame; Step S2, acquiring ORB feature points from the RGB images in the frame to obtain key point coordinates for descriptor generation; Step S3, performing feature point matching on consecutive frames, comparing the distances between each pair of descriptor points in the two RGB images, and selecting the minimum distance to complete the matching of each pair of feature points; Step S4, after matching, solving for the optimal pose using the PnP algorithm and a tracking strategy, and then performing optimized tracking; Step S5, determining whether the current frame is a key frame, and if so, sending it to the transparent object recognition thread; Step S6, ... When the transparent object recognition thread receives the keyframe, it uses the YOLOv4-tiny lightweight deep learning framework to determine if there is a transparent object. If a transparent object exists, the keyframe is sent to the DZ-SD-DIV module for processing to extract the main outline of the transparent object. K-Mean clustering is performed, followed by dilation and erosion to obtain the voxel depth information of the main outline. In step S7, the keyframe is simultaneously input into the fine outline extraction module to finely extract the transparent object outline. The transparent object outline is then input into the GNN-SVO neural network to output predicted depth information. In step S8, the voxel depth information and the predicted depth information are fused, and the fused keyframe is then passed to the mapping thread. Simultaneously, the keyword is passed to the loop closure detection thread.

[0013] The fast SLAM method for dense point clouds in indoor environments with multiple transparent objects in this invention is mainly applied to indoor scenes. Based on ORB-SLAM, a pre-trained YOLOV4-tiny lightweight neural network is used to identify transparent objects. The voxel edges of transparent objects extracted by the DZ-SD-DIV module are fused with the depth information of the GNN-SVO neural network, followed by mapping and loop closure detection. In environments with multiple transparent or black objects, synchronous localization and dense mapping can be performed quickly and in real time.

[0014] In addition, the fast SLAM method for indoor dense point clouds in a multi-transparent object environment according to the above embodiments of the present invention may also have the following additional technical features:

[0015] Further, in one embodiment of the present invention, step S2 specifically includes: step S201, generating an image pyramid based on the RGB image in the frame; step S202, extracting key points of the image pyramid using the FAST feature point extraction strategy; step S203, calculating the centroid angle of the key points, and transforming the centroid angle to obtain the coordinates of the key points.

[0016] Furthermore, in one embodiment of the present invention, the tracking strategy includes: constant velocity model tracking, reference frame tracking, and relocation tracking, wherein the constant velocity model tracking has the highest priority, the reference frame tracking has the second highest priority, and the relocation tracking is used when the first two tracking strategies fail.

[0017] Furthermore, in one embodiment of the present invention, the specific processing procedure of the constant velocity model tracking is as follows: after each frame completes the pose tracking of the current frame, the velocity of the current frame is calculated; when the next frame is passed in, the pose of the current frame is initialized using the velocity of the previous frame; nonlinear optimization is performed, the optimal pose is solved by minimizing the reprojection error, and then the outliers in the optimized matching points are removed.

[0018] Furthermore, in one embodiment of the present invention, the specific processing procedure for reference frame tracking is as follows: the descriptor of the current frame is converted into a BoW vector, and the feature point matching between the current frame and the reference frame is accelerated by bag-of-words BoW; the pose of the previous frame is used as the initial value of the pose of the current frame, nonlinear optimization is performed, the optimal pose is solved by minimizing the reprojection error, and then the outliers in the optimized matching points are removed.

[0019] Furthermore, in one embodiment of the present invention, the specific process of the relocation tracking is as follows: calculate the bag-of-words vector of the feature points of the current frame, find the candidate keyframe similar to the current frame through bag-of-words search keyFrame; perform BoW matching between the current frame and the candidate keyframe; estimate the initial pose through the EPnP algorithm, perform nonlinear optimization, and solve the optimal pose by minimizing the reprojection error.

[0020] Furthermore, in one embodiment of the present invention, the DZ-SD-DIV module uses three variables of the RGBD ascending information map in the coupled keyframe to extract the main outline of the transparent object. The three variables include: voxels with a high probability of zero depth state, voxels with a high probability of depth anomaly state determined by the signed distance function, and voxels with a high probability of depth anomaly state determined by the depth information difference.

[0021] To achieve the above objectives, a second aspect of the present invention proposes a fast SLAM system for dense point clouds in indoor environments with multiple transparent objects, comprising: an image acquisition module for acquiring RGB images and depth information images, and processing the RGB images and depth information images to obtain a frame; a feature point acquisition module for acquiring ORB feature points in the RGB images of the frame to obtain key point coordinates for descriptor generation; a matching module for matching feature points between consecutive frames, selecting the minimum distance between each pair of descriptor points in the two RGB images to complete the matching of each pair of feature points; a solution module for solving the optimal pose using the PnP algorithm and tracking strategy after matching, and then performing optimized tracking; a judgment module for determining whether the current frame is a key frame and sending it to the transparent object recognition thread; and a first processing module. The first module is used to determine whether a transparent object exists when the transparent object recognition thread receives the keyframe (keyFrame). If a transparent object exists, the keyframe (keyFrame) is sent to the DZ-SD-DIV module for processing to extract the main contour of the transparent object. K-Mean clustering is performed, followed by dilation and erosion to obtain the voxel depth information of the main contour. The second module simultaneously inputs the keyframe (keyFrame) into the fine contour extraction module to finely extract the transparent object contour, and then inputs the transparent object contour into the GNN-SVO neural network to output predicted depth information. The third module fuses the voxel depth information and the predicted depth information, then sends the fused keyframe to the mapping thread, and simultaneously sends the keyword to the loop closure detection thread.

[0022] The indoor dense point cloud fast SLAM system in the multi-transparent object environment of this invention is mainly applied in indoor scenes. Based on ORB-SLAM, a pre-trained YOLOV4-tiny lightweight neural network is used to identify transparent objects. The voxel edges of transparent objects extracted by the DZ-SD-DIV module are fused with the depth information of the GNN-SVO neural network, and then mapping and loop closure detection are performed. In environments with multiple transparent objects or multiple black objects, synchronous localization and dense mapping can be performed quickly and in real time.

[0023] To achieve the above objectives, a third aspect of the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0024] To achieve the above objectives, a fourth aspect of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described above.

[0025] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0026] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0027] Figure 1 This is a flowchart of a fast SLAM method for indoor dense point clouds in a multi-transparent object environment according to an embodiment of the present invention;

[0028] Figure 2 This is a detailed execution diagram of a fast SLAM method for indoor dense point clouds in a multi-transparent object environment according to an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the structure of a fast SLAM system for dense point clouds in an indoor environment with multiple transparent objects, according to an embodiment of the present invention. Detailed Implementation

[0030] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0031] The following describes, with reference to the accompanying drawings, a fast SLAM method and system for indoor dense point clouds in a multi-transparent object environment according to embodiments of the present invention.

[0032] Figure 1 This is a flowchart of a fast SLAM method for indoor dense point clouds in a multi-transparent object environment according to an embodiment of the present invention.

[0033] like Figure 1 As shown, the fast SLAM method for indoor dense point clouds in a multi-transparent object environment includes the following steps:

[0034] In step S1, RGB images and depth information images are acquired, and the RGB images and depth information images are processed to obtain a frame.

[0035] Specifically, an RGBD camera is used to simultaneously acquire RGB images and depth information images, which are then sent to a computer for processing. A frame contains information such as an RGB image, a depth information image, a timestamp, and a frame ID.

[0036] In step S2, ORB feature points are collected from the RGB image in the frame to obtain the coordinates of key points for descriptor generation.

[0037] Furthermore, in one embodiment of the present invention, step S2 specifically includes:

[0038] Step S201: Generate an image pyramid based on the RGB image in the frame;

[0039] Step S202: Extract key points of the image pyramid using the FAST feature point extraction strategy;

[0040] Step S203: Calculate the centroid angle of the key point and transform the centroid angle to obtain the coordinates of the key point.

[0041] Specifically, ORB feature point acquisition is a crucial preprocessing step in this algorithm. The acquisition process consists of: image pyramid generation, keypoint extraction, calculation of keypoint centroid angles, and descriptor generation (using 256 bits).

[0042] 1) Generate an image pyramid based on the RGB images in the frame, using a scale factor of 1.2 by default, with 5 layers;

[0043] 2) Key point extraction uses the FAST feature point extraction strategy;

[0044] 3) Calculate the centroid angle of the key point:

[0045] First, define

[0046] Where, m pq Let B be the moment of the image patch, B be the image resolution, x and y be the horizontal and vertical coordinates, and I(x,y) be the gray value of the (x,y) point in the image. Using this formula, m can be calculated. 10 and m 01 Therefore, the centroid angle of the key point is:

[0047]

[0048] Based on this, the coordinates of the key points after the transformation of the feature points can be obtained as follows:

[0049]

[0050] The keypoint coordinates obtained using formula (3) are used for descriptor generation, and the descriptors use the BRIEF descriptor.

[0051] In step S3, feature point matching is performed on the previous and next frames. By comparing the distance between each descriptor point pair in the two RGB images, the minimum distance is selected to complete the matching of each pair of feature points.

[0052] The distance is defined by the following formula:

[0053]

[0054] Where, d i 1 represents the i-th position of the descriptor of feature point 1, d i 2 represents the i-th position of the descriptor of feature point 2.

[0055] In step S4, after matching is completed, the optimal pose is solved using the PnP algorithm and tracking strategy, and then optimized tracking is performed.

[0056] Furthermore, in one embodiment of the present invention, the tracking strategy includes: constant velocity model tracking, reference frame tracking, and relocation tracking, wherein constant velocity model tracking has the highest priority, reference frame tracking has the second highest priority, and relocation tracking is used when the first two tracking strategies fail.

[0057] Specifically, constant velocity model tracking: the tracking velocity of the previous frame is used to initialize the current frame, and then nonlinear optimization is performed. The key formulas are as follows:

[0058] After each frame completes pose tracking for that frame, the velocity for that frame is calculated. Indicates the pose of the current frame. This represents the pose of the previous frame at this moment, therefore the velocity formula for this frame is as follows:

[0059]

[0060] When the next frame is received, the pose of the current frame can be initialized using the velocity of the previous frame, as shown in the following formula:

[0061]

[0062] Then, nonlinear optimization is performed to solve for the optimal pose by minimizing the reprojection error, where n is the number of matched feature point pairs, K is the camera intrinsic parameter matrix, and T... c_w For the initial pose, X i The spatial coordinates of the i-th feature point are given by the following formula:

[0063]

[0064] Remove outliers from the optimized matching points.

[0065] Reference frame tracking:

[0066] The descriptor of the current frame is converted into a BoW vector, and the feature point matching between the current frame and the reference frame is accelerated by using bag-of-words BoW.

[0067] The pose of the previous frame is used as the initial value of the pose of the current frame. Nonlinear optimization is performed, and the optimal pose is solved by minimizing the reprojection error, as in formula (7).

[0068] Remove outliers from the optimized matching points.

[0069] Relocation tracking:

[0070] Calculate the bag-of-words vector of the feature points in the current frame, and find candidate keyframes similar to the current frame by searching the keyFrame using the bag-of-words vector.

[0071] Perform BoW matching between the current frame and the candidate keyframes;

[0072] The initial pose is estimated using the EPNP algorithm;

[0073] Nonlinear optimization is performed to solve for the optimal pose by minimizing the reprojection error, as in formula (7);

[0074] The optimized tracking process is as follows:

[0075] Find map points in the local map of the current frame that match the current frame, and reproject all map points into the current frame;

[0076] The pose is then further optimized using nonlinear methods.

[0077] Update the observation rate of map points in the current frame and analyze the effect of tracking the local map.

[0078] Count the number of inliers; if the number of inliers is greater than 25, the tracking is considered successful.

[0079] In step S5, it is determined whether the current frame is a key frame. If so, it is sent to the transparent object recognition thread.

[0080] The selection criteria for KeyFrame are as follows:

[0081] bool keyFrame =(c 1a ||c 1b ||c 1c )&&c2 (8)

[0082] In the formula, c 1a The number of frames between the current frame and the previous keyframe is greater than δ. max, default δ max 25; c1 b The number of frames between the current frame and the previous keyframe is greater than δ. min And the mapping thread is in an idle state, with a default δ min 10; c 1c c1 is the ratio of the number of inliers tracked in the current frame to the number of feature points in the reference keyframe, which is less than ε1. The default value of ε1 is 0.25. c2 is the ratio of the number of inliers tracked in the current frame to the number of feature points matched in the current frame, which is less than ε2, and the number of inliers tracked in the current frame is greater than 15. The default value of ε2 is 0.75.

[0083] When the current frame is determined to be a keyframe, it is sent to the transparent object recognition thread.

[0084] In step S6, when the transparent object recognition thread receives the keyframe, it uses the YoloV4-tiny lightweight deep learning framework to determine whether there is a transparent object. If there is a transparent object, the keyframe is sent to the DZ-SD-DIV module for processing. The main outline of the transparent object is extracted, clustered using the K-Mean algorithm, dilated, and then eroded to obtain the voxel depth information of the main outline.

[0085] It should be noted that the YOLOv4-tiny deep learning framework has approximately 6 million parameters, a network structure with 38 layers and 3 residual units, and uses LeakyReLU as the activation function. YOLOv4-tiny has been pre-trained using a massive transparent object dataset. In the cfg configuration, the batch size is 96, subdivisions are 48, momentum is 0.921, learning rate is 0.001, and max_batches are 50000.

[0086] Furthermore, in one embodiment of the present invention, the DZ-SD-DIV module uses three variables of the RGBD ascending information map in the coupled keyframe to extract the main outline of the transparent object. The three variables include: voxels with a high probability of zero depth state, voxels with a high probability of depth anomalous state determined by the signed distance function, and voxels with a high probability of depth anomalous state determined by the depth information difference.

[0087] Specifically, such as Figure 2 As shown, the main processing idea of ​​the DZ-SD-DIV module is to couple the three variables of the RGBD ascending information map in the keyframe to extract the main outline of the transparent object. The key formula is as follows:

[0088] voxels transparent =v Dz_H ∪(v SD_H ∩vDIV_H (9)

[0089] Calculate ZD H The process is as follows:

[0090]

[0091] in,

[0092]

[0093]

[0094]

[0095] In the formula, v Dz_H A voxel with a high probability of being in a zero-depth state; v Dz (X) represents the ZD value of voxel point X; σ is an adjustable threshold, with a default value of 0.5; Frames is the number of consecutive frames observed for the current voxel, which must be greater than 3. When the number of consecutive frames is greater than 10, the 10 most recent frames are taken; k indicates that the current frame is the k-th frame; depth k (x k (x) represents the pixel x in the depth information map of the k-th frame. k The depth value; d k (X) is the truncation symbol distance function for voxel point X in the k-th frame; δ is the truncation value, which defaults to 1.

[0096] Calculate v SD_H The process is as follows:

[0097] v SD_H ={X|V k (X)>αδ} (15)

[0098] Among them, V k (X)=Q k (X)-D 2 k (X) (16)

[0099]

[0100]

[0101]

[0102] In the formula, v SD_H Q is used to identify voxels with high probability of depth anomalies using the signed distance function. k (X) is the weight function for voxel X in the k-th frame; D k(X) is the truncation symbol distance function update value of voxel point X in the kth frame; α is a control variable, which defaults to 0.8.

[0103] Calculate v DIV_H The process is as follows:

[0104] v DIV_H ={div(X)>ρ 2} (20)

[0105]

[0106] Among them, v DIV_H This is a voxel used to determine high-probability depth anomalies using depth information difference; the div(X) function is the depth information difference function; ρ is a control variable, which defaults to 0.5.

[0107] Furthermore, after processing by the DZ-SD-DIV module, the voxel edges of the transparent object are obtained and projected onto the current frame, X. i Let x be the i-th point on the edge of the voxel. i Let be the coordinates of the voxel point in the current frame, as shown in the following formula:

[0108] x i =KT c_w X i (twenty two)

[0109] Due to the uncertainty of voxels, the resulting objects may be several transparent regions. The KD-Mean algorithm is needed for clustering. After obtaining the clustering results, the clustered regions are subjected to morphological dilation and erosion processing.

[0110] In step S7, the keyframe is simultaneously input into the fine contour extraction module to finely extract the contour of the transparent object, and the transparent object contour is input into the GNN-SVO neural network to output the predicted depth information.

[0111] Specifically, the region of the transparent object determined by the YoloV4-tiny deep learning framework is used as a mask and passed to the fine contour extraction module. This module uses the Gradcut algorithm to finely extract the contour of the transparent object.

[0112] Then, the keyframes of the detailed transparent object outline obtained in the previous step are fed into the GNN-SVO neural network, which is trained to perform indoor semi-transparent SVO to predict the depth information of the current frame.

[0113] In step S8, the voxel depth information and the predicted depth information are fused together, and the fused keyframes are then passed to the mapping thread, while the keywords are passed to the loop closure detection thread.

[0114] The specific formula for deep information fusion is as follows:

[0115]

[0116] in, The depth corresponding to the final predicted projection point x of the current frame; w1 and w2 are the weight information of the voxel point X output by the DZ-SD-DIV module in step 8 and the weight information of the predicted voxel depth in step 10, respectively; depth1(x) and depth2(x) are the depth information of the predicted pixel point x of the current frame in steps S6 and S7, respectively.

[0117] Furthermore, in the mapping thread, spatial points of all keyframes are mapped, and the color information of the spatial points uses the color points of the RGB images of the keyframes. Simultaneously, global nonlinear BA optimization is initiated to optimize the spatial points and pose.

[0118] The purpose of the loop closure detection thread is to detect whether the same location has been visited before, in order to establish constraints and reduce pose and spatial point errors. This algorithm uses the classic Bag-of-Words (BOW) method: the bag-of-words vector of the current frame is compared with the bag-of-words library. If the best keyframe is matched, loop closure correction is initiated, and constraints are established between the frames before and after the loop closure.

[0119] The fast SLAM method for dense point clouds in indoor environments with multiple transparent objects proposed in this invention is mainly applied to indoor scenes. Based on ORB-SLAM, a pre-trained YOLOV4-tiny lightweight neural network is used to identify transparent objects. The voxel edges of transparent objects extracted by the DZ-SD-DIV module are fused with the depth information of the GNN-SVO neural network, followed by mapping and loop closure detection. In environments with multiple transparent or black objects, synchronous localization and dense mapping can be performed quickly and in real time.

[0120] Next, referring to the accompanying drawings, a fast SLAM system for dense point clouds in an indoor environment with multiple transparent objects, according to an embodiment of the present invention, is described.

[0121] Figure 3 This is a schematic diagram of the structure of a fast SLAM system for dense point clouds in an indoor environment with multiple transparent objects, according to an embodiment of the present invention.

[0122] like Figure 3 As shown, the system 10 includes: an image acquisition module 100, a feature point acquisition module 200, a matching module 300, a solution module 400, a judgment module 500, a first processing module 600, a second processing module 700, and a mapping module 800.

[0123] The system comprises the following modules: Image Acquisition 100 acquires RGB and depth information images, processes them to obtain a frame; Feature Point Acquisition 200 performs ORB feature point acquisition on the RGB images in the frame to obtain keypoint coordinates for descriptor generation; Matching 300 performs feature point matching between consecutive frames, comparing the distances between descriptor point pairs in the two RGB images and selecting the minimum distance to complete the matching for each pair; Solving 400, after matching, solves for the optimal pose using the PnP algorithm and tracking strategy, and then optimizes the tracking; and Judgment 500 determines whether the current frame is a keyframe and sends it to the transparent object recognition thread. The first processing module 600, upon receiving a keyframe (keyFrame) in the transparent object recognition thread, uses the YOLOv4-tiny lightweight deep learning framework to determine if a transparent object exists. If a transparent object is found, the keyframe (keyFrame) is sent to the DZ-SD-DIV module for processing. The module extracts the main contour of the transparent object, performs clustering using the K-Mean algorithm, dilates it, and then performs erosion to obtain the voxel depth information of the main contour. The second processing module 700 simultaneously inputs the keyframe (keyFrame) into the fine contour extraction module to finely extract the transparent object's contour. This contour is then input into the GNN-SVO neural network to output predicted depth information. The mapping module 800 fuses the voxel depth information and predicted depth information, then sends the fused keyframe to the mapping thread, while simultaneously sending keywords to the loop closure detection thread.

[0124] The fast SLAM system for dense point clouds in indoor environments with multiple transparent objects, proposed according to embodiments of the present invention, is mainly applied in indoor scenes. Based on ORB-SLAM, a pre-trained YOLOV4-tiny lightweight neural network is used to identify transparent objects. The voxel edges of transparent objects extracted by the DZ-SD-DIV module are fused with the depth information of the GNN-SVO neural network, followed by mapping and loop closure detection. In environments with multiple transparent objects or multiple black objects, it can quickly perform synchronous localization and dense mapping in real time.

[0125] To implement the above embodiments, the present invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the fast SLAM method for indoor dense point clouds in a multi-transparent object environment as described in the foregoing embodiments.

[0126] To implement the above embodiments, the present invention also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the fast SLAM method for indoor dense point clouds in a multi-transparent object environment as described in the foregoing embodiments.

[0127] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0128] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0129] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0130] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0131] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0132] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0133] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0134] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for fast SLAM of indoor dense point cloud in a multi-transparent object environment, characterized in that, The method comprises the following steps: Step S1, collect an RGB picture and a depth information picture, and process the RGB picture and the depth information picture to obtain a frame Frame; Step S2, collect ORB feature points of the RGB picture in the frame Frame to obtain key point coordinates for description sub-generation; Step S3, match feature points of front and back frames, select the minimum distance to complete matching of each pair of feature points by comparing distances of each pair of description sub-points of two RGB images; Step S4, after matching, solve an optimal pose by a PnP algorithm and a tracking strategy, and then perform optimized tracking; Step S5, judge whether the current frame Frame is a key frame keyFrame, and if so, send the key frame keyFrame to a transparent object recognition thread; Step S6, when the transparent object recognition thread receives the key frame keyFrame, judge whether there is a transparent object by using a YoloV4-tiny lightweight deep learning framework, and if so, send the key frame keyFrame to a DZ-SD-DIV module for processing, extract a main contour of the transparent object, perform clustering by using a K-Mean algorithm, perform inflation and then perform corrosion to obtain voxel depth information of the main contour; Step S7, simultaneously input the key frame keyFrame into a fine contour extraction module to finely extract a transparent object contour, and input the transparent object contour into a GNN-SVO neural network to output predicted depth information; Step S8, perform depth information fusion on the voxel depth information and the predicted depth information, and then input the fused key frame into a mapping thread, and simultaneously input the key frame into a loop detection thread; The selection requirement of the key frame KeyFrame is as follows: （8） In the formula, is the number of frames from the current frame to the last key frame , by default is 25; is the number of frames from the current frame to the last key frame , and the mapping thread is in an idle state, by default is 10; is the ratio of the number of inner points tracked in the current frame to the number of feature points in the reference key frame , by default is 0.25; is the ratio of the number of inner points tracked in the current frame to the number of feature points matched in the current frame , and the number of inner points tracked in the current frame > 15, by default is 0.75; The DZ-SD-DIV module uses three variables of RGBD ascending information graphs in the key frame to extract the main contour of the transparent object, wherein the three variables include: a voxel in a high-probability zero-depth state, a voxel in a high-probability depth abnormal state determined by using a signed distance function, and a voxel in a high-probability depth abnormal state determined by using depth information difference; The main processing idea of the DZ-SD-DIV module is to couple the three variables of the RGBD ascending information graphs in the key frame to extract the main contour of the transparent object, and the key formula is as follows: （9） The process of computing is: （10） wherein (11) （12） （13） （14） In the formula, is a voxel of high probability zero depth state; represents the ZD value of the voxel point X; is an adjustable threshold value, and the default value is 0.5; Frames is the number of continuous frames in which the current voxel is observed, and the number of continuous frames needs to be greater than 3; k is the kth frame in the current frame; is the depth value of the pixel point in the depth information map of the kth frame; is the truncated signed distance function of the kth frame voxel point X; is a truncation value, and the default value is 1; The process of computing is: （15） wherein (16) （17） （18） （19） wherein, is the voxel with high-probability depth anomaly state determined by the signed distance function; is the weight function of the voxel in the frame k is the weight function of the voxel in the frame X is the truncated signed distance function update value of the voxel in the frame is the truncated signed distance function update value of the voxel in the frame k is the truncated signed distance function update value of the voxel in the frame X is the truncated signed distance function update value of the voxel in the frame is the control quantity, and the default value is 0.

8. The process of computing is: （20） （21） wherein, is the voxel for which the high probability depth anomaly state is determined; is the difference function for the depth information; is the control quantity, by default 0.5; After the DZ-SD-DIV module processing, the voxel edge of the transparent object is obtained, and projected to the current frame, is the i-th point of the voxel edge, is the corresponding coordinate of the voxel point in the current frame, and the formula is as follows: （22） Because of the uncertainty of the voxel, several transparent object regions are obtained, and the KD-Mean algorithm is used for clustering, and after obtaining the clustering result, morphological inflation and corrosion processing are performed on the clustered regions; The specific formula of the depth information fusion is as follows: （23） wherein, is the depth corresponding to the projection point x of the current frame for the final prediction; and are the weight information of the voxel point X output by the DZ-SD-DIV module in step 8 and the weight information of the predicted voxel depth in step 10, respectively; and are the depth information of the pixel point x of the current frame predicted in steps S6 and S7, respectively.

2. The indoor dense point cloud fast SLAM method in a multi-transparent object environment according to claim 1, characterized in that, The step S2 specifically comprises: Step S201, generate an image pyramid according to the RGB picture in the frame Frame; Step S202, extract key points of the image pyramid by using a FAST feature point extraction strategy; Step S203, calculate the centroid angle of the key points, and transform the centroid angle to obtain key point coordinates.

3. The indoor dense point cloud fast SLAM method in a multi-transparent object environment according to claim 1, characterized in that, The tracking strategy comprises constant speed model tracking, reference frame tracking and relocalization tracking, wherein the constant speed model tracking has the highest priority, the reference frame tracking has the second priority, and the relocalization tracking is used when the two tracking strategies fail.

4. The indoor dense point cloud fast SLAM method in a multi-transparent object environment according to claim 3, characterized in that, The specific processing procedure of the constant speed model tracking is as follows: After the pose tracking of the current frame is completed, the speed of the current frame is calculated; When the next frame is transmitted, the speed of the last frame is used to initialize the pose of the current frame; Nonlinear optimization is performed to solve the optimal pose with the minimum re-projection error, and outliers in the optimized matching points are removed.

5. The indoor dense point cloud fast SLAM method in a multi-transparent object environment according to claim 3, characterized in that, The specific processing procedure of the reference frame tracking is as follows: The descriptor of the current frame is converted into a BoW vector, and the feature point matching between the current frame and the reference frame is accelerated through the bag of words BoW; The pose of the last frame is used as the initial value of the pose of the current frame, nonlinear optimization is performed to solve the optimal pose with the minimum re-projection error, and outliers in the optimized matching points are removed.

6. The indoor dense point cloud fast SLAM method in a multi-transparent object environment according to claim 3, characterized in that, The specific procedure of the relocalization tracking is as follows: The bag of words vector of the feature points of the current frame is calculated, and the keyFrame similar to the current frame is found through bag of words search; The current frame and the candidate keyFrame are matched through BoW; The initial pose is estimated through the EPnP algorithm, nonlinear optimization is performed to solve the optimal pose with the minimum re-projection error.

7. The indoor dense point cloud fast SLAM method in a multi-transparent object environment according to claim 1, characterized in that, The DZ-SD-DIV module adopts three variables of the RGBD ascending information graph in the coupling keyFrame to extract the main contour of the transparent object, wherein the three variables comprise: a voxel in a high probability zero depth state, a voxel in a high probability depth abnormal state determined by using a signed distance function, and a voxel in a high probability depth abnormal state determined by using a depth information difference. 8.A fast SLAM system for indoor dense point cloud in multi-transparent object environment, characterized in that, The indoor dense point cloud fast SLAM method in the multi-transparent object environment according to any one of claims 1-7 comprises: a picture acquisition module configured to acquire an RGB picture and a depth information picture, and process the RGB picture and the depth information picture to obtain a frame Frame; a feature point acquisition module configured to acquire ORB feature points of the RGB picture in the frame Frame to obtain key point coordinates for descriptor generation; a matching module configured to match feature points of the front and rear frames, and select the minimum distance to complete the matching of each pair of feature points by comparing the distance of each pair of descriptor points of the two RGB pictures; a solving module configured to solve the optimal pose through a PnP algorithm and a tracking strategy after the matching is completed, and then perform optimization tracking; a judgment module configured to judge whether the current frame Frame is a keyFrame keyFrame, and send the current frame Frame to a transparent object identification thread. The first processing module is configured to determine whether there is a transparent object by using a YoloV4-tiny lightweight deep learning framework when the transparent object identification thread receives the key frame, and if there is a transparent object, send the key frame to a DZ-SD-DIV module for processing to extract the main contour of the transparent object, cluster by using a K-Mean algorithm, and erode after inflation to obtain voxel depth information of the main contour. The second processing module is configured to input the key frame into a fine contour extraction module to finely extract a transparent object contour, and input the transparent object contour into a GNN-SVO neural network to output predicted depth information. The mapping module is configured to fuse the voxel depth information and the predicted depth information, and input the fused key frame into a mapping thread, and input the key frame into a loop detection thread. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1-7.

Images