Visual SLAM method, device, terminal equipment and storage medium

By using the Transformer model in the visual SLAM method to semantically segment the RGB image and depth image, identify and eliminate dynamic areas, the problem of reduced graph construction accuracy in dynamic environments is solved, and higher graph construction accuracy is achieved.

CN113628334BActive Publication Date: 2024-11-15SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110805748.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-16
Publication Date
2024-11-15
Estimated Expiration
2041-07-16

AI Technical Summary

Technical Problem

The existing visual SLAM method reduces the accuracy of map construction due to the influence of dynamic objects in a dynamic environment.

Method used

By acquiring RGB images and depth images, semantic segmentation is performed using the Transformer model, dynamic areas are identified and eliminated, the target image is obtained, and then graph construction and loopback detection are performed.

Benefits of technology

It improves the accuracy of map construction in dynamic environments, can identify dynamic objects more accurately, and reduces the impact of dynamic objects on map construction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113628334B_ABST
    Figure CN113628334B_ABST
Patent Text Reader

Abstract

The present application is applicable to the field of image processing technology, and provides a visual SLAM method, apparatus, terminal device and storage medium. The visual SLAM method comprises: obtaining a current image frame, wherein the current image frame comprises an RGB image and a depth image; based on the RGB image and the depth image, semantically segmenting the current image frame through a Transformer model to obtain a dynamic area in the current image frame, wherein the dynamic area refers to the area where the dynamic objects in the current image frame are located; removing the dynamic area from the current image frame to obtain a target image; and performing mapping and loop detection based on the target image. The present application can solve the problem that dynamic objects affect the accuracy of mapping when performing visual SLAM in a dynamic environment in the prior art.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a visual SLAM method, apparatus, terminal device and storage medium. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) refers to a technology that uses visual sensors mounted on a robot to perceive the surrounding environment without prior information, build an environmental model during movement, and estimate its own positioning.

[0003] Current visual SLAM methods are usually based on the static assumption, that is, in the entire visual SLAM process, the objects in the default environment are static. However, most actual application environments are dynamic environments with dynamic objects, such as walking people and moving cars, which affect the accuracy of mapping. Summary of the invention

[0004] The embodiments of the present application provide a visual SLAM method, apparatus, terminal device and storage medium to solve the problem in the prior art that dynamic objects affect the accuracy of mapping when performing visual SLAM in a dynamic environment.

[0005] In a first aspect, an embodiment of the present application provides a visual SLAM method, the visual SLAM method comprising:

[0006] Acquire a current image frame, where the current image frame includes an RGB image and a depth image;

[0007] Based on the RGB image and the depth image, semantically segment the current image frame through a Transformer model to obtain a dynamic area in the current image frame, where the dynamic area refers to an area where a dynamic object in the current image frame is located;

[0008] Eliminating the dynamic area from the current image frame to obtain a target image;

[0009] Mapping and loop closure detection are performed based on the target image.

[0010] In a second aspect, an embodiment of the present application provides a visual SLAM device, the visual SLAM device comprising:

[0011] An image acquisition module is used to acquire a current image frame, wherein the current image frame includes an RGB image and a depth image;

[0012] A region segmentation module, configured to perform semantic segmentation on the current image frame through a Transformer model based on the RGB image and the depth image to obtain a dynamic region in the current image frame, where the dynamic region refers to a region where a dynamic object in the current image frame is located;

[0013] A region elimination module, used for eliminating the dynamic region from the current image frame to obtain a target image;

[0014] The mapping loop closure module is used to perform mapping and loop closure detection based on the target image.

[0015] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the visual SLAM method described in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the visual SLAM method described in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the steps of the visual SLAM method as described in the first aspect above.

[0018] As can be seen from the above, based on the RGB image and depth image in the current image frame, the present application can perform semantic segmentation on the current image frame through the Transformer model, obtain the dynamic area in the current image frame (i.e., the area where the dynamic object is located), and perform mapping and loop detection based on the image after removing the dynamic area, which can reduce the impact of dynamic objects in a dynamic environment on mapping and improve the accuracy of mapping. And when the present application performs semantic segmentation on the current image frame through the Transformer model, it makes full use of various information in the RGB image and the depth image, and can more accurately identify the dynamic objects in the current image frame and obtain a more accurate dynamic area. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1This is a schematic diagram of the implementation process of the visual SLAM method provided in Example 1 of the present application;

[0021] Figure 2 This is an example diagram of the structure of the Transformer encoder;

[0022] Figure 3 This is a schematic diagram of the implementation process of the visual SLAM method provided in Example 2 of the present application;

[0023] Figure 4 It is a schematic diagram of the structure of the visual SLAM device provided in Example 3 of the present application;

[0024] Figure 5 It is a structural diagram of the terminal device provided in Example 4 of the present application. DETAILED DESCRIPTION

[0025] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0026] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0027] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0028] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0029] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0031] Before describing the present application scheme, in order to facilitate the reader's understanding, the system architecture of the visual SLAM in this scheme is first explained.

[0032] The system architecture of visual SLAM mainly consists of five parts: image frame acquisition, visual odometer, backend optimization, mapping and loop detection.

[0033] Image frame acquisition: Capture image frames of the surrounding environment through the camera.

[0034] Visual odometry: Estimating the camera pose between adjacent image frames, including the application of algorithms such as feature point matching.

[0035] Backend optimization: Optimize the cumulative error generated by the visual odometry, including the application of algorithms such as filters and graph optimization.

[0036] Mapping: 3D map construction.

[0037] Loop detection: Detect whether the terminal device has reached the previous position to reduce the cumulative error.

[0038] In the embodiment of the present application, in order to solve the problem that dynamic objects affect the accuracy of mapping when performing visual SLAM in a dynamic environment in the prior art, it is proposed to perform semantic segmentation on the current image frame based on the RGB image and depth image in the current image frame through the Transformer model, and the dynamic area in the current image frame (i.e., the area where the dynamic object is located) can be obtained. Mapping and loop detection are performed based on the image after removing the dynamic area, which can reduce the impact of dynamic objects on mapping in a dynamic environment and improve the accuracy of mapping. And when the present application performs semantic segmentation on the current image frame through the Transformer model, various information in the RGB image and the depth image are fully utilized, and the dynamic objects in the current image frame can be more accurately identified to obtain a more accurate dynamic area.

[0039] The visual SLAM method provided in the embodiments of the present application can be applied to terminal devices such as unmanned vehicles, augmented reality (AR) / virtual reality (VR) devices, drones, sweeping robots, etc. The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0040] It should be understood that the size of the serial numbers of the steps in this embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0041] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0042] See also Figure 1 , is a schematic diagram of the implementation process of the visual SLAM method provided in the first embodiment of the present application, and the visual SLAM method is applied to a terminal device. Figure 1 As shown, the visual SLAM method may include the following steps:

[0043] Step 101, obtaining a current image frame, where the current image frame includes an RGB image and a depth image.

[0044] The terminal device may be equipped with a camera to collect image frames of the surrounding environment. The camera sends the collected image frames to the terminal device. The current image frame refers to the image frame collected by the camera at the current moment.

[0045] The camera has a simple structure, diversified installation methods, and no limitation on sensor detection distance. It has a wide range of applications and low cost. In addition, the image frames captured by the camera can extract relatively rich semantic information, map information, etc.

[0046] Optionally, the camera may be an RGBD camera. An RGBD camera can detect the depth of field of a shooting space, and an image frame shot by an RGBD camera includes an RGB image and a depth image. An RGBD camera may also be referred to as a depth camera.

[0047] RGB images can be understood as color images. Depth images are images that include depth information. The RGB image and depth image included in the same image frame are registered, that is, there is a one-to-one correspondence between the pixels of the RGB image and the depth image included in the same image frame.

[0048] There are three main types of RGBD cameras currently available: a binocular vision-based RGBD camera, a structured light-based RGBD camera, and a time-of-flight-based RGBD camera. All three types can be used in the embodiments of the present application.

[0049] The RGBD camera based on binocular vision uses two cameras at different perspectives to take pictures of the same space. The difference in pixels of the same object in the images taken by the two cameras is directly related to the depth of the object. Therefore, the depth information can be obtained by calculating the pixel deviation.

[0050] The RGBD camera based on structured light projects a coded structured light pattern into the target space, then uses the camera to capture an image of the target space containing the structured light pattern, and processes the image (for example, by matching calculation with a reference structured light image) to obtain depth information.

[0051] The RGBD camera based on the time-of-flight method emits laser pulses to the target space. After the laser pulses are reflected by the target, they are received by the receiving unit and the round-trip time of the laser pulses is recorded. The depth information is calculated based on this time.

[0052] Step 102: Based on the RGB image and the depth image, semantic segmentation is performed on the current image frame through the Transformer model to obtain the dynamic area in the current image frame.

[0053] The terminal device can input the current image frame into the Transformer model, and the Transformer model performs semantic segmentation on the current image frame based on the RGB image and depth image of the current image frame to obtain the dynamic area in the current image frame. The dynamic area in the current image frame refers to the area where the dynamic object in the current image frame is located or covered.

[0054] RGB images have rich surface texture and appearance information, and depth images have rich three-dimensional structure information and are insensitive to lighting changes. Therefore, when the Transformer model performs semantic segmentation on the current image frame based on the RGB image and depth image of the current image frame, it can make full use of the various information of the RGB image and depth image in the current image frame, and can more accurately identify dynamic objects in the current image frame, obtain more accurate dynamic areas, and further improve the accuracy of mapping.

[0055] In one embodiment, the Transformer model includes a converter, a first Transformer encoder, a second Transformer encoder, and a Transformer decoder; based on the RGB image and the depth image, the current image frame is semantically segmented by the Transformer model, and the dynamic area in the current image frame includes:

[0056] The RGB image and the depth image are converted by a converter to obtain an RGB information sequence and a depth information sequence, wherein the RGB information sequence includes image information and position information of more than two RGB sub-images, and the depth information sequence includes image information and position information of more than two depth sub-images;

[0057] Input the depth information sequence into the first Transformer encoder to obtain encoded data corresponding to the depth information sequence;

[0058] The encoded data corresponding to the depth information sequence and the RGB information sequence are sequentially input into the second Transformer encoder and the Transformer decoder to obtain the dynamic area in the current image frame.

[0059] Among them, converting the RGB image through the converter can be understood as position encoding the RGB image. Converting the depth image through the converter can be understood as position encoding the depth image.

[0060] The process of converting an RGB image through a converter may include: converting the RGB image x∈R H×W×C Divide evenly into The size of each grid image is Form a vectorized package p; use the linear projection function f:p→e∈R C , mapping each vectorized packet p into a potential C-dimensional embedding space, we can obtain a sequence of one-dimensional embedded information packets of the RGB image, where p i The linear map is e i ; Then encode the spatial information of each embedded information packet, and learn a specific position embedding information pos for each position i i , embedding it into the linear map e of the image bag i In the formation of sequence E = {e 1 +pos 1 ,e 2 +pos 2 ,…,e L +pos L}, the sequence is the RGB information sequence. Among them, H and W represent the length and width of the RGB image respectively; C represents the number of channels of the RGB image; L represents the total number of RGB sub-images, which can be understood as the length of the RGB information sequence; e can be understood as the image information of the RGB sub-image; pos can be understood as the position information of the RGB sub-image.

[0061] It should be noted that the process of converting the depth image through the converter can refer to the process of converting the RGB image through the above converter, which will not be repeated here.

[0062] The terminal device inputs the depth information sequence into the first Transformer encoder, and the first Transformer encoder may encode the depth information sequence to obtain encoded data corresponding to the depth information sequence.

[0063] The terminal device inputs the encoded data corresponding to the depth information sequence and the RGB information sequence into the second Transformer encoder and the Transformer decoder in sequence. The encoded data corresponding to the depth information sequence and the RGB information sequence can be first encoded by the second Transformer encoder to obtain target encoded data, and then the target encoded data is input into the Transformer decoder. The target encoded data can be decoded by the Transformer decoder to obtain the dynamic area in the current image frame.

[0064] Among them, the structure of the first Transformer encoder and the second Transformer encoder is the same, both including 12 layers of network, each layer of the network includes layer normalization (LN), multi-head self-attention (MSA) and multi-layer perceptron (MLP), and LN is performed before MSA and between MSA and MLP.

[0065] It should be noted that the Transformer encoder in the present application (ie, the first Transformer encoder and the second Transformer encoder) uses a 12-layer network to replace the 24-layer network of the existing Transformer encoder, thereby improving the recognition speed of dynamic areas.

[0066] LN is the net input s∈R of each layer. d Perform the following processing:

[0067]

[0068] Where μ∈R d ,δ∈R d , μ and δ are the mean and standard deviation of the net input respectively; Υ and β are the learned parameters respectively; d represents the dimension.

[0069] In the j-th layer network, the input of SA is based on the output of the j-1th layer The calculated (query, key, value) triples, where:

[0070]

[0071] Among them, W Q ∈R C×d , W K ∈R C×d , W V ∈R C×d , W Q , W K and W V are the learnable parameters of the three linear projection layers, and d represents the dimension of (query, key, value).

[0072] Self-attention can be expressed as:

[0073]

[0074] Among them, softmax represents the softmax function, also known as the normalized exponential function; T is the transpose symbol.

[0075] A multilayer perceptron is a union of m independent SA operations, and its output is the union of the m SAs:

[0076]

[0077] Among them, W o ∈R md×C , d is set to C / m.

[0078] The output of MSA through MLP is converted through residual connection as the output of the j-th layer network. The output of the j-th layer network can be expressed as follows:

[0079]

[0080] in,

[0081] The Transformer decoder can perform four decoding operations on the input target encoded data, each operation alternately using convolutional layers and 2x upsampling to convert the target encoded data from Convert the dimensions to H x W.

[0082] In one embodiment, before the encoded data corresponding to the depth information sequence and the RGB information sequence are sequentially input into the second Transformer encoder and the Transformer decoder, the method further includes:

[0083] Mapping the encoded data corresponding to the depth information sequence to the vector space of the RGB information sequence to obtain mapped data;

[0084] Inputting the encoded data corresponding to the depth information sequence and the RGB information sequence into the second Transformer encoder and the Transformer decoder in sequence includes:

[0085] The mapped data and RGB information sequence are input into the second Transformer encoder and Transformer decoder in sequence.

[0086] The terminal device can map the encoded data corresponding to the depth information sequence to the vector space of the RGB information sequence through linear mapping, so as to facilitate the encoding and decoding operations of the second Transformer encoder and Transformer decoder.

[0087] like Figure 2 The following is an example diagram of the structure of the Transformer encoder. Figure 2 The input data in refers to the mapped data and RGB information sequence.

[0088] Step 103, remove the dynamic area from the current image frame to obtain a target image.

[0089] The target image refers to the image after the dynamic area of ​​the current image frame is removed.

[0090] Among them, removing the dynamic area from the current image frame may refer to setting the RGBD values ​​of the pixel points in the dynamic area in the current image frame to a preset value, for example, the preset value is 0.

[0091] Step 104: perform mapping and loop closure detection based on the target image.

[0092] The terminal device can build a map and perform loop detection based on the feature points of the target image to achieve visual SLAM in a dynamic environment. Feature points can be understood as points where the grayscale value of the image changes dramatically or points with large curvature on the edge of the image.

[0093] In one embodiment, mapping and loop closure detection based on the target image includes:

[0094] Determine the camera pose according to the image coordinates of each feature point in the target image and the image coordinates of the feature points matching it in the key frame. The camera refers to the device that collects the current image frame and key frame.

[0095] Mapping and loop closure detection are performed based on the camera pose and target image.

[0096] The terminal device can extract each feature point in the target image and each feature point in the key frame, match each feature point in the target image with each feature point in the key frame, and use the matched feature points to calculate the transformation matrix between the target image and the key frame, which is the camera pose. Among them, the feature points in the image can use the ORB (Oriented FAST and Rotated BRIEF) feature extraction algorithm, or other feature extraction algorithms, which are not limited here.

[0097] The terminal device can update the key frame according to the camera pose and target image, and perform mapping and loop detection based on the updated key frame.

[0098] The embodiment of the present application is based on the RGB image and depth image in the current image frame. The current image frame can be semantically segmented through the Transformer model to obtain the dynamic area (i.e., the area where the dynamic object is located) in the current image frame. Mapping and loop detection are performed based on the image after the dynamic area is eliminated, which can reduce the impact of dynamic objects on mapping in a dynamic environment and improve the accuracy of mapping. And when the present application performs semantic segmentation on the current image frame through the Transformer model, it makes full use of various information in the RGB image and the depth image, can more accurately identify the dynamic objects in the current image frame, obtain a more accurate dynamic area, and has better adaptability to the dynamic object recognition task in the visual SLAM in a dynamic environment.

[0099] See also Figure 3 , is a schematic diagram of the implementation flow of the visual SLAM method provided in the second embodiment of the present application, and the visual SLAM method is applied to a terminal device. Figure 2 As shown, the visual SLAM method may include the following steps:

[0100] Step 301, obtaining a current image frame, where the current image frame includes an RGB image and a depth image.

[0101] This step is the same as step 101. For details, please refer to the relevant description of step 101, which will not be repeated here.

[0102] Step 302: Based on the RGB image and the depth image, semantic segmentation is performed on the current image frame through the Transformer model to obtain the dynamic area in the current image frame.

[0103] This step is the same as step 102. For details, please refer to the relevant description of step 102, which will not be repeated here.

[0104] Step 303, determining the distance between the epipolar line of each feature point in other areas and the epipolar line of the feature point matching therewith in the key frame according to the image coordinates of each feature point in other areas of the current image frame and the image coordinates of the feature points matching therewith in the key frame.

[0105] The other areas refer to areas other than the dynamic area in the current image frame.

[0106] In one embodiment, determining the distance between the epipolar line of each feature point in other areas and the epipolar line of the feature point matching therewith in the key frame according to the image coordinates of each feature point in other areas of the current image frame and the image coordinates of the feature point matching therewith in the key frame comprises:

[0107] Determine the homogeneous coordinates of each feature point in other regions and the homogeneous coordinates of the feature points matching therewith in the key frame according to the image coordinates of each feature point in other regions and the image coordinates of the feature points matching therewith in the key frame;

[0108] According to the homogeneous coordinates of each feature point in other regions and the homogeneous coordinates of the feature points matching therewith in the key frame, the distance between the epipolar line of each feature point in other regions and the epipolar line of the feature points matching therewith in the key frame is determined.

[0109] For the two matching feature points P obtained from other regions of the current image frame and the key frame respectively 1 and P 2 , feature point P 1 and P 2 The image coordinates can be expressed as:

[0110] P 1 '=[u 1 ,v 1 ] T P 2 '=[u 2 ,v 2 ] T

[0111] Feature point P 1 and P 2 The homogeneous coordinates of can be expressed as:

[0112] P 1 ”=[u 1 ,v 1 ,1] T P 2 ”=[u 2 ,v 2 ,1] T

[0113] Among them, u 1 and v1 Represents the feature point P 1 The image coordinate value, u 2 and v 2 Represents the feature point P 2 The image coordinates of the

[0114] Feature point P 1 The epipolar line can be expressed as:

[0115] E 1 =[X 1 ,Y 1 ,Z 1 ] T =FP 1 ”

[0116] Where F represents the fundamental matrix in the epipolar constraint.

[0117] Feature point P 1 The epipolar line and characteristic point P 2 The distance between the epipolar lines can be expressed as:

[0118]

[0119] Among them, P 2 ” T Indicates P 2 ” is the transposed matrix of .

[0120] After traversing each feature point in other regions of the current image frame and the feature points matched therewith in the key frame, the distance between the epipolar line of each feature point in other regions and the epipolar line of the corresponding feature point in the key frame can be obtained.

[0121] Step 304 : weighted average the distances between the epipolar lines of each feature point in other regions and the epipolar lines of the feature points matched therewith in the key frame to obtain a first epipolar line distance.

[0122] For example, the distances between the epipolar lines of each feature point in other areas and the epipolar lines of the feature points matching them in the key frame can be accumulated, and then divided by the total number of feature points in other areas. The average value of the distances between the epipolar lines of each feature point in other areas and the epipolar lines of the feature points matching them in the key frame can be obtained. This average value is the first epipolar distance.

[0123] Step 305 , determining the distance between the epipolar line of each feature point in the dynamic region and the epipolar line of the feature point matching therewith in the key frame according to the image coordinates of each feature point in the dynamic region and the image coordinates of the feature point matching therewith in the key frame.

[0124] In one embodiment, determining the distance between the epipolar line of each feature point in the dynamic region and the epipolar line of the feature point matching therewith in the key frame according to the image coordinates of each feature point in the dynamic region and the image coordinates of the feature point matching therewith in the key frame comprises:

[0125] Determine the homogeneous coordinates of each feature point in the dynamic area and the homogeneous coordinates of the feature points matching therewith in the key frame according to the image coordinates of each feature point in the dynamic area and the image coordinates of the feature points matching therewith in the key frame respectively;

[0126] According to the homogeneous coordinates of each feature point in the dynamic region and the homogeneous coordinates of the feature points matching therewith in the key frame, the distance between the epipolar line of each feature point in the dynamic region and the epipolar line of the feature points matching therewith in the key frame is determined.

[0127] It should be noted that the process of calculating the distance between the polar lines of each feature point in the dynamic area and the polar lines of the feature points matching it in the key frame can refer to the process of calculating the distance between the polar lines of each feature point in the other areas and the polar lines of the feature points matching it in the key frame, and will not be repeated here.

[0128] Step 306, performing weighted averaging of the distances between the epipolar lines of each feature point in the dynamic region and the epipolar lines of the feature points matched therewith in the key frame to obtain a second epipolar line distance;

[0129] For example, the distances between the epipolar lines of each feature point in the dynamic area and the epipolar lines of the feature points matching it in the key frame can be accumulated, and then divided by the total number of feature points in the dynamic area. The average value of the distances between the epipolar lines of each feature point in the dynamic area and the epipolar lines of the feature points matching it in the key frame can be obtained. This average value is the second epipolar distance.

[0130] Step 307, calculating the absolute value of the difference between the first epipolar distance and the second epipolar distance.

[0131] Step 308: If the absolute value of the difference is greater than a preset threshold, the dynamic area is removed from the current image frame to obtain a target image.

[0132] If the absolute value of the difference between the first epipolar distance and the second epipolar distance is greater than a preset threshold, it is determined that the object in the dynamic area semantically segmented by the Transformer model is a dynamic object. At this time, the dynamic area can be removed from the current image frame to reduce the impact of the dynamic object on the map. If the absolute value of the difference between the first epipolar distance and the second epipolar distance is less than or equal to the preset threshold, it is determined that the object in the dynamic area semantically segmented by the Transformer model may not be a dynamic object (that is, the dynamic area semantically segmented by the Transformer model is incorrect). At this time, the current image frame can be re-input into the Transformer model and semantically segmented again, or it can be directly used for mapping and loop detection without the need for removal.

[0133] Among them, the above method of determining the first epipolar distance and the second epipolar distance, calculating the absolute value of the difference between the first epipolar distance and the second epipolar distance, and comparing the absolute value of the difference with a preset threshold can be called an epipolar distance matching method.

[0134] Step 309: perform mapping and loop closure detection based on the target image.

[0135] This step is the same as step 104. For details, please refer to the relevant description of step 104, which will not be repeated here.

[0136] In actual application scenarios, each frame of the image can be transmitted to the tracking thread and the semantic segmentation thread at the same time. The tracking thread extracts ORB feature points from the image, while the semantic segmentation thread uses the Transformer-based image semantic segmentation algorithm (i.e., the Transformer model) to perform semantic segmentation on the image to obtain the dynamic area, and then transmits the results of the semantic segmentation thread to the tracking thread.

[0137] After the tracking thread obtains the results of the semantic segmentation thread, it further optimizes the dynamic area and calculates the transformation matrix between the current image frame and the key frame through the optimized data to obtain the camera pose, which is further optimized through local map tracking. The tracking thread selects the key frame and transfers the key frame to the semantic segmentation thread and the local mapping thread. At the same time, the key frame and camera pose are transferred to the point cloud drawing thread, which draws according to the key data.

[0138] The local map thread removes low-quality map points based on the inserted keyframes, and then optimizes and removes redundant or erroneous keyframes through the local bundle adjustment method.

[0139] Finally, the loop detection thread performs loop detection through keyframes and map points. Once a loop is detected, loop correction will perform a global bundle adjustment optimization to ensure global consistency and compensate for the drift accumulated during the SLAM process.

[0140] Based on the first embodiment, this embodiment can further detect the dynamic region semantically segmented by the Transformer model through the epipolar distance matching method to determine whether the dynamic region is a real dynamic region, thereby obtaining a more accurate dynamic region.

[0141] The embodiment of the present application is based on the RGB image and depth image in the current image frame. The current image frame can be semantically segmented through the Transformer model to obtain the dynamic area (i.e., the area where the dynamic object is located) in the current image frame. Mapping and loop detection are performed based on the image after the dynamic area is eliminated, which can reduce the impact of dynamic objects on mapping in a dynamic environment and improve the accuracy of mapping. And when the present application performs semantic segmentation on the current image frame through the Transformer model, it makes full use of various information in the RGB image and the depth image, can more accurately identify the dynamic objects in the current image frame, and obtain a more accurate dynamic area. The dynamic area semantically segmented by the Transformer model is detected by the polar line distance matching method, which can further improve the accuracy of the dynamic area, and has better adaptability to the dynamic object recognition task in the visual SLAM under a dynamic environment.

[0142] Corresponding to the visual SLAM method described in the above embodiment, Figure 4 A structural block diagram of a visual SLAM device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0143] Reference Figure 4 , the visual SLAM device comprises:

[0144] An image acquisition module 41 is used to acquire a current image frame, where the current image frame includes an RGB image and a depth image;

[0145] A region segmentation module 42 is used to perform semantic segmentation on the current image frame through a Transformer model based on the RGB image and the depth image to obtain a dynamic region in the current image frame, where the dynamic region refers to a region where a dynamic object in the current image frame is located;

[0146] The region elimination module 43 is used to eliminate the dynamic region from the current image frame to obtain a target image;

[0147] The mapping and loop closure module 44 is used to perform mapping and loop closure detection based on the target image.

[0148] Optionally, the Transformer model includes a converter, a first Transformer encoder, a second Transformer encoder and a Transformer decoder; the region segmentation module 42 includes:

[0149] An image conversion unit, used to convert the RGB image and the depth image respectively through a converter to obtain an RGB information sequence and a depth information sequence, wherein the RGB information sequence includes image information and position information of more than two RGB sub-images, and the depth information sequence includes image information and position information of more than two depth sub-images;

[0150] An information encoding unit, used to input the depth information sequence into a first Transformer encoder to obtain encoded data corresponding to the depth information sequence;

[0151] The information processing unit is used to input the encoded data corresponding to the depth information sequence and the RGB information sequence into the second Transformer encoder and the Transformer decoder in sequence to obtain the dynamic area in the current image frame.

[0152] Optionally, the region segmentation module 42 further includes:

[0153] An information mapping unit, used to map the encoded data corresponding to the depth information sequence to the vector space of the RGB information sequence to obtain mapped data;

[0154] The above information processing unit is specifically used for:

[0155] The mapped data and RGB information sequence are input into the second Transformer encoder and Transformer decoder in sequence.

[0156] Optionally, the above-mentioned visual SLAM device also includes:

[0157] A first determination module is used to determine the distance between the epipolar line of each feature point in other areas and the epipolar line of the feature point matching therewith in the key frame according to the image coordinates of each feature point in other areas of the current image frame and the image coordinates of the feature points matching therewith in the key frame, wherein the other areas refer to areas other than the dynamic area in the current image frame;

[0158] A first weighting module, used for weighted averaging the distances between the epipolar lines of each feature point in other regions and the epipolar lines of the feature points matched therewith in the key frame to obtain a first epipolar line distance;

[0159] A second determination module is used to determine the distance between the epipolar line of each feature point in the dynamic area and the epipolar line of the feature point matching therewith in the key frame according to the image coordinates of each feature point in the dynamic area and the image coordinates of the feature point matching therewith in the key frame;

[0160] A second weighting module is used to perform weighted averaging of the distances between the epipolar lines of each feature point in the dynamic area and the epipolar lines of the feature points matched therewith in the key frame to obtain a second epipolar line distance;

[0161] A difference calculation module, used to calculate the absolute value of the difference between the first epipolar distance and the second epipolar distance;

[0162] The above-mentioned area elimination module 43 is specifically used for:

[0163] If the absolute value of the difference is greater than a preset threshold, the dynamic area is removed from the current image frame to obtain the target image.

[0164] Optionally, the first determining module is specifically used to:

[0165] Determine the homogeneous coordinates of each feature point in other regions and the homogeneous coordinates of the feature points matching therewith in the key frame according to the image coordinates of each feature point in other regions and the image coordinates of the feature points matching therewith in the key frame;

[0166] Determine the distance between the epipolar line of each feature point in the other regions and the epipolar line of the feature point matching therewith in the key frame according to the homogeneous coordinates of each feature point in the other regions and the homogeneous coordinates of the feature point matching therewith in the key frame;

[0167] The second determination module is specifically used for:

[0168] Determine the homogeneous coordinates of each feature point in the dynamic area and the homogeneous coordinates of the feature points matching therewith in the key frame according to the image coordinates of each feature point in the dynamic area and the image coordinates of the feature points matching therewith in the key frame respectively;

[0169] According to the homogeneous coordinates of each feature point in the dynamic region and the homogeneous coordinates of the feature points matching therewith in the key frame, the distance between the epipolar line of each feature point in the dynamic region and the epipolar line of the feature points matching therewith in the key frame is determined.

[0170] Optionally, the above-mentioned mapping loop module 44 is specifically used for:

[0171] Determine the camera pose according to the image coordinates of each feature point in the target image and the image coordinates of the feature points matching it in the key frame. The camera refers to the device that collects the current image frame and key frame.

[0172] Mapping and loop closure detection are performed based on the camera pose and target image.

[0173] Optionally, the camera is an RGBD camera.

[0174] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0175] The device provided in the embodiment of the present application can be applied in the aforementioned method embodiments 1 and 2. For details, please refer to the description of the aforementioned method embodiments 1 and 2, which will not be repeated here.

[0176] Figure 5 Schematic diagram of the structure of the terminal device provided in the fourth embodiment of the present application. Figure 5 As shown, the terminal device 5 of this embodiment includes: one or more processors 50 (only one is shown in the figure), a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. When the processor 50 executes the computer program 52, the steps in the above-mentioned various visual SLAM method embodiments are implemented.

[0177] The terminal device 5 may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will appreciate that Figure 5 It is only an example of the terminal device 5 and does not constitute a limitation of the terminal device 5. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0178] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0179] The memory 51 may be an internal storage unit of the terminal device 5, such as a hard disk or memory of the terminal device 5. The memory 51 may also be an external storage device of the terminal device 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 5. Further, the memory 51 may also include both an internal storage unit of the terminal device 5 and an external storage device. The memory 51 is used to store the computer program and other programs and data required by the terminal device. The memory 51 may also be used to temporarily store data that has been output or is to be output.

[0180] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0181] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0182] The embodiment of the present application also provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0183] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0184] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0185] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0186] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0188] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A visual SLAM method, characterized in that: The visual SLAM method comprises: Acquire a current image frame, where the current image frame includes an RGB image and a depth image; Based on the RGB image and the depth image, semantically segment the current image frame through a Transformer model to obtain a dynamic area in the current image frame, where the dynamic area refers to an area where a dynamic object in the current image frame is located; Eliminating the dynamic area from the current image frame to obtain a target image; Performing mapping and loop closure detection based on the target image; The Transformer model includes a converter, a first Transformer encoder, a second Transformer encoder and a Transformer decoder; based on the RGB image and the depth image, the current image frame is semantically segmented by the Transformer model to obtain a dynamic area in the current image frame including: The RGB image and the depth image are converted by the converter to obtain an RGB information sequence and a depth information sequence, wherein the RGB information sequence includes image information and position information of more than two RGB sub-images, and the depth information sequence includes image information and position information of more than two depth sub-images; Inputting the depth information sequence into the first Transformer encoder to obtain encoded data corresponding to the depth information sequence; The encoded data corresponding to the depth information sequence and the RGB information sequence are sequentially input into the second Transformer encoder and the Transformer decoder to obtain the dynamic area in the current image frame.

2. The visual SLAM method according to claim 1, wherein Before sequentially inputting the encoded data corresponding to the depth information sequence and the RGB information sequence into the second Transformer encoder and the Transformer decoder, the method further includes: Mapping the encoded data corresponding to the depth information sequence to the vector space of the RGB information sequence to obtain mapped data; The step of sequentially inputting the encoded data corresponding to the depth information sequence and the RGB information sequence into the second Transformer encoder and the Transformer decoder comprises: The mapped data and the RGB information sequence are sequentially input into the second Transformer encoder and the Transformer decoder.

3. The visual SLAM method according to any one of claims 1 to 2, characterized in that Before removing the dynamic area from the current image frame, the method further includes: Determine, according to the image coordinates of each feature point in other areas of the current image frame and the image coordinates of the feature points matching therewith in the key frame, the distance between the epipolar line of each feature point in the other areas and the epipolar line of the feature points matching therewith in the key frame, wherein the other areas refer to areas other than the dynamic area in the current image frame; Taking a weighted average of the distances between the epipolar lines of each feature point in the other region and the epipolar lines of the feature points matched therewith in the key frame to obtain a first epipolar line distance; Determining the distance between the epipolar line of each feature point in the dynamic area and the epipolar line of the feature point matching therewith in the key frame according to the image coordinates of each feature point in the dynamic area and the image coordinates of the feature point matching therewith in the key frame; Taking a weighted average of the distances between the epipolar lines of each feature point in the dynamic region and the epipolar lines of the feature points matched therewith in the key frame to obtain a second epipolar line distance; calculating an absolute value of a difference between the first epipolar distance and the second epipolar distance; The removing the dynamic area from the current image frame to obtain a target image comprises: If the absolute value of the difference is greater than a preset threshold, the dynamic area is removed from the current image frame to obtain the target image.

4. The visual SLAM method as claimed in claim 3, wherein: The step of determining the distance between the epipolar line of each feature point in the other area and the epipolar line of the feature point matching therewith in the key frame according to the image coordinates of each feature point in the other area of ​​the current image frame and the image coordinates of the feature point matching therewith in the key frame comprises: Determine the homogeneous coordinates of each feature point in the other regions and the homogeneous coordinates of the feature points matching therewith in the key frame according to the image coordinates of each feature point in the other regions and the image coordinates of the feature points matching therewith in the key frame respectively; Determine the distance between the epipolar line of each feature point in the other region and the epipolar line of the feature point in the key frame matching therewith according to the homogeneous coordinates of each feature point in the other region and the homogeneous coordinates of the feature point in the key frame matching therewith; The determining, based on the image coordinates of each feature point in the dynamic area and the image coordinates of the feature point matching therewith in the key frame, the distance between the epipolar line of each feature point in the dynamic area and the epipolar line of the feature point matching therewith in the key frame comprises: Determine the homogeneous coordinates of each feature point in the dynamic area and the homogeneous coordinates of the feature points matching therewith in the key frame according to the image coordinates of each feature point in the dynamic area and the image coordinates of the feature points matching therewith in the key frame respectively; The distance between the epipolar line of each feature point in the dynamic region and the epipolar line of the feature point matching therewith in the key frame is determined according to the homogeneous coordinates of each feature point in the dynamic region and the homogeneous coordinates of the feature point matching therewith in the key frame.

5. The visual SLAM method according to any one of claims 1 to 2, characterized in that: The mapping and loop closure detection based on the target image includes: Determine the camera pose according to the image coordinates of each feature point in the target image and the image coordinates of the feature points matching therewith in the key frame, wherein the camera refers to the device that acquires the current image frame and the key frame; Mapping and loop closure detection are performed according to the camera pose and the target image.

6. The visual SLAM method as claimed in claim 5, characterized in that, The camera is an RGBD camera.

7. A visual SLAM device, characterized in that: The visual SLAM device comprises: An image acquisition module is used to acquire a current image frame, wherein the current image frame includes an RGB image and a depth image; A region segmentation module, configured to perform semantic segmentation on the current image frame through a Transformer model based on the RGB image and the depth image to obtain a dynamic region in the current image frame, where the dynamic region refers to a region where a dynamic object in the current image frame is located; A region elimination module, used for eliminating the dynamic region from the current image frame to obtain a target image; A mapping and loop closure module, used for performing mapping and loop closure detection based on the target image; The Transformer model includes a converter, a first Transformer encoder, a second Transformer encoder and a Transformer decoder; the region segmentation module includes: an image conversion unit, configured to convert the RGB image and the depth image respectively through the converter to obtain an RGB information sequence and a depth information sequence, wherein the RGB information sequence includes image information and position information of more than two RGB sub-images, and the depth information sequence includes image information and position information of more than two depth sub-images; an information encoding unit, configured to input the depth information sequence into the first Transformer encoder to obtain encoded data corresponding to the depth information sequence; An information processing unit is used to input the encoded data corresponding to the depth information sequence and the RGB information sequence into the second Transformer encoder and the Transformer decoder in sequence to obtain the dynamic area in the current image frame.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the visual SLAM method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the visual SLAM method according to any one of claims 1 to 6 are implemented.