Simultaneous positioning and map construction method, device, equipment and storage medium

Abstract

The present invention provides a method, apparatus, device, and storage medium for simultaneous positioning and mapping, which are applied to the field of image processing technology. The method comprises: obtaining a first image group captured by a camera at the current moment of a scene and performing semantic segmentation and depth estimation to obtain semantic segmentation results and a target depth image for each feature point in the first image group; the first image group comprises a first RGB image and a first initial depth image; determining the target category of each feature point in the first RGB image based on the semantic segmentation results and target depth image of each feature point in the first image group and the semantic segmentation results and target depth image of each feature point in each second image group; eliminating feature points in the first RGB image whose target category is dynamic to obtain a target RGB image, performing simultaneous positioning and mapping, and determining the camera's target pose and target map. The technical solution of the present invention can improve the accuracy of simultaneous positioning and mapping.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a simultaneous positioning and map building method, device, equipment and storage medium. Background Art

[0002] With the rapid development of autonomous robots, augmented reality, and drones, SLAM (Simultaneous Localization and Mapping) technology has become crucial for the successful operation of intelligent mobile robots in unknown environments. During the simultaneous localization and mapping process, SLAM distinguishes between static and dynamic objects in the environment, enabling precise positioning and higher-precision mapping.

[0003] In related technologies, dynamic objects and / or static objects in the environment are generally processed by combining deep learning with geometric methods, thereby achieving simultaneous positioning and map construction.

[0004] However, the above technologies have the problem of low accuracy in simultaneous positioning and map construction. Summary of the Invention

[0005] The present invention provides a simultaneous positioning and mapping method, device, equipment and storage medium to address the defect of low accuracy caused by the simultaneous positioning and mapping of dynamic or static objects in the environment after processing through deep learning combined with geometric methods in the prior art. The method achieves this by fusing the semantic information and depth information of the scene images collected by the camera at each consecutive moment, and accurately distinguishing the dynamic feature points and static feature points in the current frame image based on this to accurately eliminate the dynamic feature points, thereby accurately estimating the camera's position and posture and accurately constructing the scene map, thereby improving the accuracy of camera pose estimation and the accuracy and robustness of map construction, while reducing computing resource consumption, so as to meet the needs of different robot applications in complex dynamic environments.

[0006] The present invention provides a method for simultaneous positioning and map construction, comprising:

[0007] Obtain a first image group captured by the camera at the current moment of the scene, and perform semantic segmentation and depth estimation on the first image group to obtain semantic segmentation results of each feature point in the first image group and a first target depth image; the first image group includes a first RGB image and a first initial depth image;

[0008] Acquire multiple second image groups captured by the camera that are temporally correlated with the first image group, and obtain semantic segmentation results and second target depth images for each feature point in each second image group. Determine the target category corresponding to each feature point in the first RGB image based on the semantic segmentation results and first target depth images for each feature point in the first image group and the semantic segmentation results and second target depth images for each feature point in each second image group. The target category is used to characterize whether the feature point is a static feature point or a dynamic feature point. Each second image group includes a second RGB image and a second initial depth image.

[0009] According to the target categories corresponding to the feature points in the first RGB image, the feature points belonging to the dynamic target category in the first RGB image are eliminated to obtain the target RGB image corresponding to the first RGB image;

[0010] Simultaneous positioning and mapping are performed based on the target RGB image to determine the target pose corresponding to the camera and the target map corresponding to the scene.

[0011] According to a simultaneous positioning and mapping method provided by the present invention, the semantic segmentation results of each feature point in the first image group include semantic category labels of each feature point in the first RGB image, and the first target depth image includes depth values ​​of each feature point in the first RGB image. The method of determining the target category corresponding to each feature point in the first RGB image based on the semantic segmentation results of each feature point in the first image group and the first target depth image, the semantic segmentation results of each feature point in each second image group, and the second target depth image includes:

[0012] For each feature point in the first RGB image, determine change information of the semantic category label of the feature point based on the semantic category label corresponding to the feature point in the first RGB image and the semantic category label corresponding to the feature point in each second RGB image;

[0013] Determine change information of the spatial position of the feature point according to the depth value corresponding to the feature point in the first target depth image and the depth value corresponding to the feature point in each second target depth image;

[0014] According to the change information of the semantic category label and the spatial position of the feature point, the target category corresponding to the feature point is determined.

[0015] According to a simultaneous positioning and mapping method provided by the present invention, determining change information of the semantic category label of a feature point based on the semantic category label corresponding to the feature point in the first RGB image and the semantic category label corresponding to the feature point in each second RGB image includes:

[0016] Determine an average semantic category label corresponding to the feature point based on the semantic category label corresponding to the feature point in the first RGB image and the semantic category label corresponding to the feature point in each second RGB image;

[0017] According to the semantic category labels of the feature points in each frame of RGB images and the average semantic category labels corresponding to the feature points, the change information of the semantic category labels of the feature points in each frame of RGB images is determined; the above-mentioned each frame of RGB images includes a first RGB image and each second RGB image.

[0018] According to a simultaneous positioning and mapping method provided by the present invention, the above-mentioned determination of change information of the spatial position of the feature point based on the depth value corresponding to the feature point in the first target depth image and the depth value corresponding to the feature point in each second target depth image includes:

[0019] Determine the spatial position corresponding to the feature point in each frame of the RGB image according to the depth value corresponding to the feature point in the first target depth image, the depth value corresponding to the feature point in each second target depth image, the two-dimensional position corresponding to the feature point in the first RGB image, and the two-dimensional position corresponding to the feature point in each second RGB image;

[0020] According to the spatial positions corresponding to the feature points in each frame of RGB image, the average spatial position corresponding to the feature points is determined;

[0021] According to the spatial position of the feature point in each frame of RGB image and the average spatial position corresponding to the feature point, the change information of the spatial position of the feature point in each frame of RGB image is determined.

[0022] According to a simultaneous positioning and mapping method provided by the present invention, the change information of the semantic category label of the feature point includes the change information of the semantic category label of the feature point in each frame of RGB image, and the change information of the spatial position of the feature point includes the change information of the spatial position of the feature point in each frame of RGB image. The above-mentioned determination of the target category corresponding to the feature point based on the change information of the semantic category label and the change information of the spatial position of the feature point includes:

[0023] Calculate the spatiotemporal consistency quantization value corresponding to the feature point based on the change information of the semantic category label of the feature point in each frame of RGB image and the change information of the spatial position of the feature point under each frame of RGB image;

[0024] The target category corresponding to the feature point is determined based on the spatiotemporal consistency quantization value corresponding to the feature point and a preset threshold.

[0025] According to a simultaneous positioning and mapping method provided by the present invention, the simultaneous positioning and mapping based on the target RGB image is performed to determine the target pose corresponding to the camera and the target map corresponding to the scene, including:

[0026] Obtain static feature points of the target category in the target RGB image, and calculate the corresponding projection positions of the static feature points on the image plane of the camera;

[0027] Obtain the semantic category labels corresponding to each region in the target RGB image and obtain the current position of the camera;

[0028] The current camera pose is used as a node, the projection position of the static feature points and the semantic category labels of each region are used as the edges of the node, and a graphical model and the corresponding objective function are constructed.

[0029] The objective function is iteratively solved to determine the target pose corresponding to the camera and the target map corresponding to the scene; the target map is composed of static feature points.

[0030] According to a simultaneous positioning and mapping method provided by the present invention, the semantic segmentation and depth estimation processing is performed on the first image group to obtain the semantic segmentation results of each feature point in the first image group and the first target depth image, including:

[0031] Performing semantic segmentation and depth estimation on the first image group using a preset lightweight semantic segmentation and depth estimation joint network to obtain semantic segmentation results of each feature point in the first image group and a first target depth image;

[0032] Among them, in the lightweight semantic segmentation and depth estimation joint network, the semantic segmentation processing and the depth estimation processing share some convolutional layers, and the information in the two parts of the semantic segmentation processing and the depth estimation processing are linked across layers.

[0033] The present invention also provides a simultaneous positioning and map building device, comprising the following modules:

[0034] A depth semantic perception module is used to obtain a first image group captured by the camera at the current moment of the scene, and perform semantic segmentation and depth estimation on the first image group to obtain semantic segmentation results of each feature point in the first image group and a first target depth image; the first image group includes a first RGB image and a first initial depth image;

[0035] A feature point category determination module is configured to obtain a plurality of second image groups captured by the camera that are temporally correlated with the first image group, obtain a semantic segmentation result for each feature point in each second image group, and a second target depth image, and determine a target category corresponding to each feature point in the first RGB image based on the semantic segmentation result for each feature point in the first image group and the first target depth image, and the semantic segmentation result for each feature point in each second image group and the second target depth image; the target category is used to characterize whether the feature point is a static feature point or a dynamic feature point, and each second image group includes a second RGB image and a second initial depth image;

[0036] A dynamic feature point removal module is used to remove dynamic feature points in the first RGB image according to the target categories corresponding to the feature points in the first RGB image, so as to obtain a target RGB image corresponding to the first RGB image;

[0037] The positioning and mapping module is used to simultaneously perform positioning and mapping based on the target RGB image, determine the target pose corresponding to the camera and the target map corresponding to the scene.

[0038] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the simultaneous positioning and mapping method as described above is implemented.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described simultaneous positioning and mapping methods.

[0040] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any one of the above-described simultaneous positioning and mapping methods.

[0041] The present invention provides a simultaneous positioning and mapping method, apparatus, device and storage medium, which obtains a first image group including a first RGB image and a first initial depth image captured by the camera at the current moment of the scene, and performs semantic segmentation and depth estimation on the first image group to obtain semantic segmentation results of each feature point in the first image group and a first target depth image, and simultaneously obtains multiple second image groups captured by the camera and having time correlation with the first image group, and obtains semantic segmentation results and second target depth images of each feature point in each second image group, and determines the target category of each feature point in the first RGB image based on the multiple semantic segmentation results and the multiple target depth images, and then eliminates the feature points in the first RGB image whose target category belongs to dynamic to obtain a target RGB image, and then performs simultaneous positioning and mapping based on the target RGB image to determine the target posture of the camera and the target map of the scene; wherein the target category is used to characterize whether the feature point belongs to a static feature point or a dynamic feature point, and each second image group includes a second RGB image and a second initial depth image. In this method, the semantic segmentation results and depth information of the scene images collected by the camera at each consecutive moment can be fused, and the dynamic feature points and static feature points in the current frame image can be accurately distinguished to accurately eliminate the dynamic feature points, thereby accurately estimating the camera's pose and accurately constructing the scene map, thereby achieving the purpose of improving the accuracy of camera pose estimation and the accuracy and robustness of map construction. At the same time, the robot does not need to rely on the rigid motion assumption, so it can reduce computing resource consumption and meet the needs of different robot applications in complex dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 This is one of the flow charts of the simultaneous positioning and map building method provided by the present invention.

[0044] Figure 2 This is the second flow chart of the simultaneous positioning and map building method provided by the present invention.

[0045] Figure 3 This is the third flow chart of the simultaneous positioning and map building method provided by the present invention.

[0046] Figure 4 It is a detailed flowchart of the simultaneous positioning and map construction method provided by the present invention.

[0047] Figure 5 It is a structural schematic diagram of the simultaneous positioning and map building device provided by the present invention.

[0048] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0050] In the area of ​​simultaneous localization and mapping, some technologies use deep learning techniques to provide semantic information for simultaneous localization and mapping, but these techniques cannot accurately determine the real-time motion state of objects. Other technologies use deep learning combined with geometric methods to process dynamic and / or static objects in the environment to achieve simultaneous localization and mapping. However, these approaches still have many limitations, such as limited semantic understanding, resulting in low accuracy in simultaneous localization and mapping, poor handling of complex dynamic scenes, and low computational efficiency. Some technologies also use ORB-SLAM3 (a SLAM system that supports vision, vision plus inertial navigation, and hybrid mapping) for simultaneous localization and mapping. While this performs well in static environments, it is significantly affected by dynamic feature points in dynamic environments. Other technologies use DynaSLAM (a visual SLAM system designed specifically for dynamic scenes) for simultaneous localization and mapping, which relies on rigid motion assumptions and has high computational overhead. Therefore, embodiments of the present invention provide a simultaneous localization and mapping method, apparatus, device, and storage medium that address these technical issues.

[0051] The following combination Figures 1-4 The simultaneous localization and mapping method of the present invention is described.

[0052] It should be noted that the execution subject of the embodiments of the present invention can be a simultaneous positioning and mapping device, an electronic device that includes the simultaneous positioning and mapping device, or other devices or equipment, such as a SLAM system. The electronic device here can be a terminal or a server. In the case of a terminal, the electronic device can be a robot or an electronic device in a robot. The following embodiments are described using an electronic device as an example of the execution subject.

[0053] Figure 1 This is one of the flow charts of the simultaneous positioning and map building method provided by the present invention, such as Figure 1 As shown, the method includes the following steps:

[0054] Step 102: Obtain a first image group captured by the camera at the current moment of the scene, and perform semantic segmentation and depth estimation on the first image group to obtain the semantic segmentation results of each feature point in the first image group and a first target depth image; the first image group includes a first RGB image and a first initial depth image.

[0055] The camera is a camera mounted on the robot. This can be one or multiple cameras. These cameras can include standard cameras that capture red, green, and blue (RGB) images of a scene, or depth cameras that capture depth data of objects in the scene. Alternatively, the robot can be equipped with a lidar (LiDAR) to collect depth data of objects in the scene. The scene can be one in which the robot needs to perform simultaneous localization and mapping (SLAM), either indoors or outdoors.

[0056] The robot's camera can capture data about the scene and its objects at the current moment, obtaining an RGB image and depth data. The RGB image captured at the current moment is referred to as the first RGB image, and the depth data captured at the current moment is referred to as the first initial depth image (also referred to as D). This image includes depth information / depth values ​​for each point in the scene. The first RGB image and the first initial depth image are referred to as the first image group. Objects in the scene can include dynamic objects (such as pedestrians and moving vehicles) and static objects (such as buildings and trees), and each object can include multiple points.

[0057] After obtaining the first RGB image and the first initial depth image in the first image group, the information on the two images can be fused for semantic segmentation and depth estimation. As an optional embodiment, a preset lightweight semantic segmentation and depth estimation joint network can be used to perform semantic segmentation and depth estimation on the first image group to obtain the semantic segmentation results of each feature point in the first image group and the first target depth image.

[0058] The lightweight semantic segmentation and depth estimation joint network can be denoted as Lightweight Semantic Segmentation and Depth Estimation Network, or LSDEN for short. The lightweight semantic segmentation and depth estimation joint network can be an improved network based on a deep learning network framework, such as a network based on the deep learning network framework TensorFlow or PyTorch.

[0059] The lightweight semantic segmentation and depth estimation joint network is used to simultaneously perform semantic segmentation processing and depth estimation processing on the input RGB image and the initial depth image. In the lightweight semantic segmentation and depth estimation joint network, the semantic segmentation processing and the depth estimation processing share some convolutional layers, and the information in the semantic segmentation processing and the depth estimation processing are cross-layer linked. That is, when performing semantic segmentation processing and depth estimation processing on the first RGB image and the first initial depth image, the two tasks of semantic segmentation processing and depth estimation processing can share some underlying convolutional layers (for example, the two tasks share the convolutional layer used to extract features), which can reduce the amount of computation and improve the efficiency of feature extraction. At the same time, during the processing of the two tasks, the semantic segmentation-related information obtained in the middle will be fused with the depth information through a cross-convolutional layer multi-source information fusion method to better estimate the depth information of the points in the scene through the semantic segmentation information, and at the same time, the depth information of the depth estimation is used to better perform semantic segmentation on the points in the scene, thereby improving the accuracy of the semantic segmentation results and the accuracy of the depth estimation. In addition, the lightweight semantic segmentation and depth estimation joint network can divide the input RGB image and the initial depth image into multiple feature maps of different scales, perform semantic segmentation and depth estimation at different scales, and obtain the final semantic segmentation result and target depth image through upsampling and fusion operations.

[0060] The lightweight joint semantic segmentation and depth estimation network described above can be pre-trained. The training process can include the following: 1. Data preparation: Collect a large number of RGB-D image groups / image datasets containing different scenes, objects, and lighting conditions, manually annotate them with semantic category labels and depth information, and divide the image dataset / image dataset into training, validation, and test sets according to a certain ratio. 2. Network training: Use a deep learning framework (such as TensorFlow or PyTorch) to build the LSDEN network, and use the cross-entropy loss function and mean squared error loss function for semantic segmentation and depth estimation, respectively. During training, the network's performance is continuously optimized by adjusting parameters such as the learning rate, optimizer (such as the Adam optimizer), and number of iterations to minimize the network's loss function value on the validation set.

[0061] After the lightweight semantic segmentation and depth estimation joint network is trained, the first RGB image and the first initial depth image collected at the current moment can be input into the lightweight semantic segmentation and depth estimation joint network to perform semantic segmentation processing and depth estimation processing at the same time, and the semantic segmentation result and depth image corresponding to the first RGB image can be obtained. The semantic segmentation results and depth image corresponding to the first RGB image are used to obtain the final semantic segmentation results of each feature point and the first target depth image.

[0062] Here, the first RGB image includes the RGB value of each pixel, and the first initial depth image includes the depth information / depth value corresponding to each pixel on the first RGB image.

[0063] The semantic segmentation results and depth image corresponding to the first RGB image output by the lightweight semantic segmentation and depth estimation joint network are obtained to obtain the final semantic segmentation results of each feature point and the first target depth image, which can include the following methods:

[0064] Method 1: The semantic segmentation result corresponding to the first RGB image output by the lightweight semantic segmentation and depth estimation joint network includes the semantic segmentation result of each pixel therein, and the depth image also includes the depth value corresponding to each pixel therein. The feature points in the first RGB image are extracted by using an improved feature extraction algorithm (such as an improved method based on FAST feature points and ORB descriptors), and the position and feature description information of the extracted feature points are recorded. The feature description information of the feature points may include information describing the local appearance of the feature points. Then, based on the position of the feature points, the semantic segmentation result corresponding to the feature points is found in the semantic segmentation result corresponding to the first RGB image, and the depth value corresponding to the feature points is found in the depth image. The semantic segmentation results corresponding to the feature points are obtained through the semantic segmentation results corresponding to the feature points, and the first target depth image is obtained through the depth values ​​corresponding to the feature points.

[0065] Method 2: The semantic segmentation result corresponding to the first RGB image output by the lightweight semantic segmentation and depth estimation joint network includes the semantic segmentation results of each feature point, and the depth image includes the depth value of each feature point. The semantic segmentation result corresponding to the first RGB image can be directly used as the semantic segmentation result of each feature point in the first image group, and the depth image output by the lightweight semantic segmentation and depth estimation joint network can be directly used as the first target depth image.

[0066] It should be noted that the above-mentioned feature points may be representative points in a scene or object, such as boundary points, corner points, etc. The number of feature points may be one or more, which is set according to actual conditions.

[0067] In addition, the semantic segmentation result of each feature point mentioned above can include the category label corresponding to the feature point, such as pedestrians, vehicles, buildings, etc. The first target depth image includes the depth value of each feature point. This depth value is combined with the semantic information of the feature point, that is, it is combined with the understanding of the scene content. It can distinguish objects / feature points with different category labels in the first RGB image, and distinguish the boundaries and structures between objects / feature points with different category labels in the first RGB image, which can assist in subsequent more accurate simultaneous positioning and map construction.

[0068] Step 104: Acquire multiple second image groups captured by the camera that are temporally correlated with the first image group, and obtain semantic segmentation results and second target depth images for each feature point in each second image group. Determine the target category corresponding to each feature point in the first RGB image based on the semantic segmentation results and first target depth images for each feature point in the first image group and the semantic segmentation results and second target depth images for each feature point in each second image group.

[0069] In this step, each second image group includes a second RGB image and a second initial depth image, that is, each second image group includes an RGB image and an initial depth image captured by the camera at the same time. The acquisition time of each second image group can be continuous with the acquisition time of the first image group, that is, the camera can continuously capture multiple frames of RGB images and multiple frames of initial depth images to accurately analyze the real-time motion state of feature points in the first RGB image at the current moment. The acquisition time of each second image group can be before the current moment and continuous with the current moment, that is, each second image group is continuously captured at multiple moments before the current moment, or after the current moment and continuous with the current moment, that is, each second image group is continuously captured at a moment after the current moment, or each second image group can be partially captured before the current moment and partially captured after the current moment, and all are continuous. The number of second image groups captured here can be set according to actual conditions, for example, 5 to 10 consecutive second image groups.

[0070] After obtaining each second image group captured at multiple moments continuous with the current moment, semantic segmentation processing and depth estimation processing can be performed on each group of second image groups in the manner of the above step 102 to obtain the semantic segmentation results of each feature point corresponding to each second image group and the second target depth image; or after extracting the feature points in the above step 102, the extracted feature points can be tracked and semantic segmentation processing and depth estimation processing can be performed in each second image group to obtain the semantic segmentation results of each feature point corresponding to each second image group and the second target depth image.

[0071] Afterwards, the semantic segmentation results and the first target depth image corresponding to each feature point in the first image group at the current moment, and the semantic segmentation results and the second target depth image of each feature point in the second image group at each consecutive moment can be used to analyze whether the semantic information and depth information of the feature point have changed at multiple consecutive moments, that is, whether the feature point has changed in time and space. Based on this information on whether or not the feature point has changed, the target category of the feature point in the first image group, that is, the target category of the feature point in the first RGB image, can be determined. The above target category is used to characterize whether the feature point is a static feature point or a dynamic feature point.

[0072] Optionally, if a feature point changes in at least one of time and space (for example, a change in the semantic category label of the same feature point at different times is considered a change in time, and a change in the depth value of the same feature point at different times is considered a change in space), the target category of the feature point is determined to be a dynamic feature point, i.e., the feature point is a dynamically changing point. If the feature point does not change in either time or space, the target category of the feature point is determined to be a static feature point, i.e., the feature point is an unchanging / fixed point.

[0073] Step 106 : According to the target categories corresponding to the feature points in the first RGB image, the feature points in the first RGB image whose target categories are dynamic are removed to obtain a target RGB image corresponding to the first RGB image.

[0074] In this step, after obtaining the target categories corresponding to each feature point in the first RGB image, feature points with dynamic target categories can be removed from the first RGB image. For example, the pixel values ​​corresponding to the dynamic feature points can be filled with the pixel values ​​corresponding to the background. After removing the dynamic feature points, the final RGB image at the current moment is obtained, recorded as the target RGB image. This target RGB image only includes static feature points, eliminating the interference of dynamic feature points, thereby more accurately locating the camera and mapping it simultaneously.

[0075] Step 108 : Perform simultaneous positioning and mapping based on the target RGB image to determine the target pose corresponding to the camera and the target map corresponding to the scene.

[0076] In this step, after obtaining the target RGB image, the camera's current pose is calculated using information such as the pixel values ​​and positions of each pixel in the target RGB image. This pose is then used to calculate the spatial positions of each feature point in the scene. This information is then used to construct a map of the scene. The resulting map is then referred to as the target map.

[0077] As can be seen from the above description, the simultaneous localization and mapping method in this embodiment uses multiple views acquired by a camera to distinguish between dynamic and static feature points. It is suitable for dynamic environments and can effectively distinguish between static and dynamic points in dynamic scenes. The positioning and mapping accuracy are high. The system corresponding to this method can also be called an improved multi-view semantic SLAM system suitable for dynamic environments.

[0078] In this embodiment, a first image group including a first RGB image and a first initial depth image captured by the camera for the scene at the current moment is obtained, and semantic segmentation and depth estimation processing are performed on the first image group to obtain semantic segmentation results of each feature point in the first image group and a first target depth image. At the same time, a plurality of second image groups captured by the camera and having a time correlation with the first image group are obtained, and semantic segmentation results and second target depth images of each feature point in each second image group are obtained. The target category of each feature point in the first RGB image is determined based on the plurality of semantic segmentation results and the plurality of target depth images, and then the feature points in the first RGB image whose target category belongs to dynamic are eliminated to obtain a target RGB image. Then, simultaneous positioning and mapping are performed based on the target RGB image to determine the target pose of the camera and the target map of the scene. The target category is used to characterize whether the feature point belongs to a static feature point or a dynamic feature point, and each second image group includes a second RGB image and a second initial depth image. In this method, the semantic segmentation results and depth information of the scene images collected by the camera at each consecutive moment can be fused, and the dynamic feature points and static feature points in the current frame image can be accurately distinguished to accurately eliminate the dynamic feature points, thereby accurately estimating the camera's pose and accurately constructing the scene map, thereby achieving the purpose of improving the accuracy of camera pose estimation and the accuracy and robustness of map construction. At the same time, the robot does not need to rely on the rigid motion assumption, so it can reduce computing resource consumption and meet the needs of different robot applications in complex dynamic environments.

[0079] The following embodiment describes a possible implementation method for determining the target category corresponding to each feature point in the first RGB image based on the semantic segmentation results of the RGB images and the initial depth image collected at multiple consecutive moments and the target depth image.

[0080] Figure 2 This is the second flow chart of the simultaneous positioning and map building method provided by the present invention. Figure 2 As shown, in the above step 104, determining the target category corresponding to each feature point in the first RGB image based on the semantic segmentation result of each feature point in the first image group and the first target depth image, the semantic segmentation result of each feature point in each second image group and the second target depth image, can include the following steps:

[0081] Step 202 : For each feature point in the first RGB image, determine change information of the semantic category label of the feature point based on the semantic category label corresponding to the feature point in the first RGB image and the semantic category label corresponding to the feature point in each second RGB image.

[0082] The semantic segmentation results of each feature point in the first image group include the semantic category labels of each feature point in the first RGB image, and the first target depth image includes the depth value of each feature point in the first RGB image. The semantic segmentation results of each feature point in the second image group include the semantic category labels of each feature point in the second RGB image, and the second target depth image includes the depth value of each feature point in the second RGB image.

[0083] In this embodiment, a spatio-temporal consistency based dynamic feature point filtering algorithm (STCDF) can be used to determine the target category of each feature point in the first RGB image, thereby distinguishing between dynamic feature points and static feature points in the first RGB image.

[0084] Optionally, for the same feature point in the first RGB image, the average semantic category label corresponding to the feature point can be determined based on the semantic category label corresponding to the feature point in the first RGB image and the semantic category label corresponding to the feature point in each second RGB image; based on the semantic category label of the feature point in each frame of RGB image and the average semantic category label corresponding to the feature point, the change information of the semantic category label of the feature point in each frame of RGB image is determined; the above-mentioned frames of RGB images include the first RGB image and each second RGB image.

[0085] Among them, for the same feature point, the semantic category label corresponding to the feature point in the first RGB image and the semantic category label corresponding to each second RGB image can be directly summed and averaged to obtain the average semantic category label corresponding to the feature point; or corresponding weights can be set for the first RGB image and each second RGB image respectively, and then the semantic category label corresponding to the feature point in the first RGB image and the semantic category label corresponding to the second RGB image can be weighted summed and averaged to obtain the average semantic category label corresponding to the feature point.

[0086] After obtaining the average semantic category label corresponding to each feature point, the absolute value of the difference between the semantic category label corresponding to the feature point in each frame of RGB image and its average semantic category label can be calculated to obtain multiple semantic category label differences, and these semantic category label differences are used as the change information of the semantic category label of the feature point in each frame of RGB image.

[0087] According to the above method, the change information of the semantic category labels of all feature points in the first RGB image in each frame of RGB image can be calculated, which will not be repeated here.

[0088] Step 204 : determining change information of the spatial position of the feature point according to the depth value corresponding to the feature point in the first target depth image and the depth value corresponding to the feature point in each second target depth image.

[0089] The first RGB image also includes the two-dimensional position / two-dimensional coordinate (x, y) of each feature point, and each second RGB image also includes the two-dimensional position / two-dimensional coordinate (x, y) of each feature point.

[0090] In this step, after obtaining the depth value corresponding to each feature point in the first RGB image in the first target depth image and the depth value corresponding to each second target depth image, optionally, for the same feature point in the first RGB image, the spatial position corresponding to the feature point in each frame of RGB image can be determined based on the depth value corresponding to the feature point in the first target depth image, the depth value corresponding to the feature point in each second target depth image, the two-dimensional position corresponding to the feature point in the first RGB image, and the two-dimensional position corresponding to the feature point in each second RGB image; based on the spatial position corresponding to the feature point in each frame of RGB image, the average spatial position corresponding to the feature point is determined; based on the spatial position of the feature point in each frame of RGB image and the average spatial position corresponding to the feature point, the change information of the spatial position of the feature point in each frame of RGB image is determined. Each frame of RGB image includes the first RGB image and each second RGB image.

[0091] For the same feature point, the depth value z corresponding to the feature point can be obtained through the first target depth image, and the two-dimensional position (x, y) corresponding to the feature point can be obtained through the first RGB image. Then, combined with the camera intrinsic parameters, the two-dimensional position (x, y) of the feature point can be converted into a position (X, Y, Z) in three-dimensional space to obtain the spatial position of the feature point in the first RGB image. Similarly, the spatial position of the feature point in each second RGB image can be obtained in this way. Afterwards, the spatial position of the feature point in each frame of the RGB image can be directly summed along the axial direction and the average is taken to obtain the average spatial position of the feature point; or the spatial position of the feature point in each frame of the RGB image can be weighted summed along the axial direction and the average is taken to obtain the average spatial position of the feature point.

[0092] After obtaining the average spatial position of each feature point, the absolute value of the difference between the spatial position of the feature point in each frame of RGB image and its corresponding average spatial position can be calculated to obtain multiple spatial position differences, and these spatial position differences can be used as the change information of the spatial position of the feature point in each frame of RGB image.

[0093] According to the above method, the spatial position change information of all feature points in the first RGB image in each frame of RGB image can be calculated, which will not be repeated here.

[0094] Step 206 : Determine the target category corresponding to the feature point based on the change information of the semantic category label and the change information of the spatial position of the feature point.

[0095] The change information of the semantic category label of the feature point includes the change information of the semantic category label of the feature point in each frame of RGB image, the change information of the spatial position of the feature point includes the change information of the spatial position of the feature point in each frame of RGB image, and each frame of RGB image includes a first RGB image and each second RGB image.

[0096] After obtaining the change information of the semantic category label of each feature point in each frame of RGB image and the change information of the spatial position of each feature point in each frame of RGB image, as an option, for the same feature point in the first RGB image, the spatiotemporal change information of the feature point in the same frame of RGB image can be determined based on the change information of the semantic category label and the change information of the spatial position of the feature point in the same frame of RGB image; the weight corresponding to each frame of RGB image is obtained; and the target category corresponding to the feature point is determined based on the weight corresponding to each frame of RGB image and the spatiotemporal change information in each frame of RGB image. The determining the target category corresponding to the feature point based on the weight corresponding to each frame of RGB image and the spatiotemporal change information in each frame of RGB image can include: for each feature point, calculating the spatiotemporal consistency quantization value corresponding to the feature point based on the spatiotemporal change information of the feature point in the same frame of RGB image and the weight corresponding to each frame of RGB image; and determining the target category corresponding to the feature point based on the spatiotemporal consistency quantization value corresponding to the feature point and a preset threshold.

[0097] Alternatively, as an option, the spatiotemporal consistency quantization value corresponding to the feature point can be calculated directly based on the change information of the semantic category label of the feature point in each frame of RGB image and the change information of the spatial position of the feature point in each frame of RGB image; and the target category corresponding to the feature point can be determined based on the spatiotemporal consistency quantization value corresponding to the feature point and a preset threshold.

[0098] Specifically, for each feature point, the change information of the semantic category label and the change information of the spatial position in each RGB image frame can be directly summed or weighted to obtain the spatiotemporal variation information of the feature point in the same or each RGB image frame. The spatiotemporal variation information of the feature point in the same or each RGB image frame can then be multiplied by the corresponding weight of the corresponding RGB image frame to obtain the weighted value of the feature point in each RGB image frame. The weighted values ​​of the feature point in each RGB image frame are then summed to obtain the sum of the variation information of the feature point in each RGB image frame, which is recorded as the spatiotemporal consistency quantization value. The spatiotemporal consistency quantization value of the feature point is then compared with a preset threshold. If the spatiotemporal consistency quantization value of the feature point is greater than the preset threshold, the target category corresponding to the feature point is determined to be a dynamic feature point. If the spatiotemporal consistency quantization value of the feature point is not greater than (less than or equal to) the preset threshold, the target category corresponding to the feature point is determined to be a static feature point. The preset threshold is determined through experiments and data analysis.

[0099] Alternatively, the above-mentioned dynamic feature point screening algorithm STCDF based on spatiotemporal consistency can be expressed in the following formula and its variant form:

[0100] .

[0101] Where i represents the i-th feature point in the first RGB image; t represents the t-th frame RGB image, T represents the total number of consecutive RGB images; C i represents the spatiotemporal consistency quantization value of the i-th feature point; w t represents the weight corresponding to the RGB image of the tth frame; S i,t Indicates the semantic category label corresponding to the i-th feature point in the t-th frame RGB image; Represents the average semantic category label corresponding to the i-th feature point; represents the change information of the semantic category label of the i-th feature point in the t-th frame RGB image; P i,t Indicates the spatial position corresponding to the i-th feature point in the t-th frame RGB image; Represents the average spatial position corresponding to the i-th feature point; Represents the change information of the spatial position of the i-th feature point in the t-th frame RGB image; represents the parameter that balances the weights of semantic category labels and spatial positions.

[0102] In this embodiment, the semantic category label change information and spatial position change information of the feature point are determined by the semantic category label of the feature point in each RGB image frame and the depth value in each depth image, and the dynamic and static categories of the feature point are determined accordingly. This can effectively distinguish dynamic points from static points in the first RGB image, improving the accuracy of subsequent simultaneous localization and mapping. In addition, by calculating the average semantic category label of the feature point in each RGB image frame and combining it with the semantic category label in each RGB image frame to determine the change information of the semantic category label of the feature point, this method is simple and effective, thereby improving the efficiency and accuracy of determining the change information of the semantic category label of the feature point, thereby improving the accuracy of subsequent simultaneous localization and mapping and reducing the computational complexity of simultaneous localization and mapping. In addition, by calculating the average spatial position of the feature point in each RGB image frame and combining it with the spatial position in each RGB image frame to determine the change information of the spatial position of the feature point, this method is simple and effective, thereby improving the efficiency and accuracy of determining the change information of the spatial position of the feature point, thereby improving the accuracy of subsequent simultaneous localization and mapping and reducing the computational complexity of simultaneous localization and mapping. Furthermore, the spatiotemporal consistency quantification value of the feature point is calculated based on the change information of the semantic category label and the change information of the spatial position of the feature point, and the target category of the feature point is determined by comparing it with the threshold. This can improve the efficiency and accuracy of distinguishing dynamic points from static points, thereby improving the accuracy of subsequent simultaneous positioning and map construction and reducing the computational complexity of simultaneous positioning and map construction.

[0103] The following example illustrates a possible implementation method for simultaneous positioning and map construction based on target RGB images.

[0104] Figure 3 This is the third flow chart of the simultaneous positioning and map building method provided by the present invention, such as Figure 3 As shown, the simultaneous positioning and mapping in step 108 based on the target RGB image to determine the target pose corresponding to the camera and the target map corresponding to the scene may include the following steps:

[0105] Step 302: Obtain static feature points of the target category in the target RGB image, and calculate the corresponding projection positions of the static feature points on the image plane of the camera.

[0106] Step 304: Obtain the semantic category labels corresponding to each region in the target RGB image and obtain the current position of the camera.

[0107] In step 306 , the current position of the camera is used as a node, the projection position of the static feature points and the semantic category labels of each region are used as edges of the node, and a graph model and an objective function corresponding to the graph model are constructed.

[0108] Step 308 , iteratively solve the objective function to determine the target pose corresponding to the camera and the target map corresponding to the scene; the target map is composed of static feature points.

[0109] In this embodiment, an improved multi-view geometry optimization module is used to perform simultaneous positioning and map construction, and in the processing process of the improved multi-view geometry optimization module, a graph-based optimization for multiview geometry optimization method (Graph-based Optimization for Multiview Geometry, GOMG) is introduced to perform simultaneous positioning and map construction.

[0110] After obtaining the target RGB image, the static feature points and their corresponding position information in the target RGB image can be obtained. At the same time, when each camera captures the RGB image and the initial depth image at the current moment, an image plane corresponding to each camera is formed. Then, the corresponding projection position of the static feature points on the image plane of each camera can be obtained based on the corresponding position information of the static feature points in the target RGB image, which is recorded as the observation point. At the same time, when each camera captures the image, the current pose of the camera can be obtained, which is recorded as the current pose or pose transformation matrix (which includes parameters such as the camera's translation and rotation).

[0111] In addition, the target RGB image can be divided into multiple regions in advance, each region can include one or more feature points, or each region can include one or more pixels. Then, the semantic category label corresponding to each region can be obtained through the semantic category label of the feature point in each region. At the same time, a deep learning network that is pre-trained to identify the semantic category label of the region can be used to identify each region in the target RGB image and obtain the predicted semantic category label of each region.

[0112] Later, during the operation of the robot / SLAM system, the current camera pose can be used as a node, and the projected positions of static feature points, as well as the semantic category labels and predicted semantic category labels (i.e., semantic constraints) of each region can be used as edges to construct a graph model and an objective function corresponding to the graph model. This objective function can include minimizing the reprojection error and the semantic consistency error. The objective function of the graph model constructed above is shown in the following formula:

[0113] .

[0114] Where E is the total error of the objective function, ρ is the robust kernel function, π is the projection function, T j is the pose transformation matrix of the j-th camera, P i is the world coordinate of the i-th feature point (which can be calculated using the calculation method of the spatial position of the feature point in the previous embodiment), p i,j is the observation point / projection position of the i-th feature point on the image plane of the j-th camera, S k is the semantic category label of the kth region in the target RGB image, is the predicted semantic label of the kth region in the target RGB image, and µ is the error weight of the semantic category label.

[0115] After determining the objective function, a nonlinear optimization algorithm (such as the Levenberg-Marquardt algorithm) can be used to iteratively optimize the objective function of the graph model to continuously update the camera pose and map until convergence (for example, if the change in the objective function value E after multiple consecutive iterations is less than the preset threshold, it converges). Finally, the optimal target pose of the camera at the current moment and the map composed of each three-dimensional point in the scene are obtained.

[0116] In this embodiment, by constructing a graphical model that includes camera pose, feature points, and semantic information, and utilizing the constraints between nodes for joint optimization, the accuracy of camera pose estimation and map construction can be improved. Furthermore, compared to traditional methods based solely on geometric constraints, the GOMG-based method in this embodiment can better utilize semantic information to constrain the optimization process, reduce error accumulation, and effectively improve the accuracy of simultaneous positioning and mapping.

[0117] The following is a detailed example to illustrate the simultaneous positioning and map construction method of the present invention. Figure 4 The detailed flow diagram of the simultaneous localization and mapping method provided by the present invention is shown. The simultaneous localization and mapping method in the embodiment of the present invention can be implemented using a SLAM system. The SLAM system may include a depth semantic perception module, a dynamic feature point screening module, and a multi-view geometry optimization module. Based on these modules, the method may include:

[0118] The robot's cameras capture an RGB image and an initial depth image at the current moment. The depth semantic perception module then extracts features using the LSDEN network to obtain a feature map. This feature map is then processed for semantic segmentation and depth estimation to obtain the current semantic segmentation result and the target depth image. Simultaneously, the robot's multiple cameras can capture multiple frames of RGB images and initial depth images at multiple moments consecutive to the current moment, and the LSDEN network is used to obtain the semantic segmentation results and target depth image at each moment.

[0119] Afterwards, the dynamic feature point screening module uses the STCDF algorithm to calculate the spatiotemporal consistency quantization value of the feature points based on the semantic segmentation results and target depth image at the current moment and the semantic segmentation results and target depth images at subsequent moments, and performs threshold screening based on the calculated spatiotemporal consistency quantization value of the feature points to eliminate the dynamic points in the RGB image at the current moment and obtain the target RGB image that only includes static feature points.

[0120] Then, the multi-view geometry optimization module uses the GOMG method to perform pose estimation and map construction based on the target RGB image, and continuously iteratively optimizes the objective function of the constructed graph model to finally obtain the camera's precise pose and constructed map.

[0121] From the above description, it can be seen that the technical solution of this embodiment has the following technical effects:

[0122] Through the innovative design of the deep semantic perception module, the semantic and depth information of the environment can be obtained more accurately, which improves the richness and accuracy of information compared to traditional methods and provides a better foundation for subsequent processing. The spatiotemporal consistency algorithm of the dynamic feature point screening module effectively utilizes the information of multiple frames of images, can more accurately identify dynamic feature points, reduce misjudgments, and thus significantly improve the accuracy of camera pose estimation. The positioning error in complex dynamic scenes is significantly reduced compared to traditional technologies. The improved multi-view geometry optimization module uses graph optimization and semantic information fusion to further improve the accuracy and robustness of map construction, and the generated map performs better in terms of details and consistency. While ensuring performance improvement, the overall system of this embodiment reduces computing resource consumption through the optimization of network structure and algorithm, making it more suitable for resource-constrained platforms, such as small mobile robots, and has a wider range of application scenarios.

[0123] The simultaneous positioning and mapping apparatus provided by the present invention is described below. The simultaneous positioning and mapping apparatus described below and the simultaneous positioning and mapping method described above can be referenced to each other.

[0124] Figure 5 This is a schematic diagram of the structure of the simultaneous positioning and map building device provided by the present invention, see Figure 5 As shown, the device may include:

[0125] The depth semantic perception module 510 is used to obtain a first image group captured by the camera at the current moment of the scene, and perform semantic segmentation and depth estimation on the first image group to obtain semantic segmentation results of each feature point in the first image group and a first target depth image; the first image group includes a first RGB image and a first initial depth image;

[0126] The feature point category determination module 520 is configured to obtain a plurality of second image groups captured by the camera and having a temporal correlation with the first image group, obtain a semantic segmentation result and a second target depth image for each feature point in each second image group, and determine a target category corresponding to each feature point in the first RGB image based on the semantic segmentation result and the first target depth image for each feature point in the first image group and the semantic segmentation result and the second target depth image for each feature point in each second image group. The target category is used to characterize whether a feature point is a static feature point or a dynamic feature point. Each second image group includes a second RGB image and a second initial depth image.

[0127] A dynamic feature point removal module 530 is configured to remove dynamic feature points in the first RGB image according to the target categories corresponding to the feature points in the first RGB image, thereby obtaining a target RGB image corresponding to the first RGB image.

[0128] The positioning and mapping module 540 is used to perform simultaneous positioning and mapping based on the target RGB image, and determine the target pose corresponding to the camera and the target map corresponding to the scene.

[0129] In some embodiments, the semantic segmentation results of each feature point in the first image group include semantic category labels of each feature point in the first RGB image, the first target depth image includes depth values ​​of each feature point in the first RGB image, and the feature point category determination module 520 includes:

[0130] a semantic label change information determining unit, configured to determine, for each feature point in the first RGB image, change information of the semantic category label of the feature point based on the semantic category label corresponding to the feature point in the first RGB image and the semantic category label corresponding to the feature point in each second RGB image;

[0131] A spatial position change information determining unit, configured to determine change information of the spatial position of a feature point based on a depth value corresponding to the feature point in the first target depth image and a depth value corresponding to the feature point in each second target depth image;

[0132] The feature point category determination unit is used to determine the target category corresponding to the feature point according to the change information of the semantic category label and the change information of the spatial position of the feature point.

[0133] Optionally, the above-mentioned semantic label change information determination unit is specifically used to determine the average semantic category label corresponding to the feature point based on the semantic category label corresponding to the feature point in the first RGB image and the semantic category label corresponding to the feature point in each second RGB image; determine the change information of the semantic category label of the feature point in each frame of RGB image based on the semantic category label of the feature point in each frame of RGB image and the average semantic category label corresponding to the feature point; the above-mentioned frames of RGB images include the first RGB image and each second RGB image.

[0134] Optionally, the above-mentioned spatial position change information determination unit is specifically used to determine the spatial position corresponding to the feature point in each frame of RGB image based on the depth value corresponding to the feature point in the first target depth image, the depth value corresponding to the feature point in each second target depth image, the two-dimensional position corresponding to the feature point in the first RGB image, and the two-dimensional position corresponding to the feature point in each second RGB image; determine the average spatial position corresponding to the feature point according to the spatial position corresponding to the feature point in each frame of RGB image; determine the change information of the spatial position of the feature point in each frame of RGB image according to the spatial position of the feature point in each frame of RGB image and the average spatial position corresponding to the feature point.

[0135] Optionally, the change information of the semantic category label of the above-mentioned feature point includes the change information of the semantic category label of the feature point in each frame of RGB image, and the change information of the spatial position of the feature point includes the change information of the spatial position of the feature point under each frame of RGB image. The above-mentioned feature point category determination unit is specifically used to calculate the spatiotemporal consistency quantization value corresponding to the feature point based on the change information of the semantic category label of the feature point in each frame of RGB image and the change information of the spatial position of the feature point under each frame of RGB image; and determine the target category corresponding to the feature point based on the spatiotemporal consistency quantization value corresponding to the feature point and a preset threshold.

[0136] In some embodiments, the positioning and map construction module 540 is specifically used to obtain static feature points belonging to the target category in the target RGB image, and calculate the corresponding projection positions of the static feature points on the image plane of the camera; obtain the semantic category labels corresponding to each area in the target RGB image and obtain the current posture of the camera; use the current posture of the camera as a node, the projection positions of the static feature points and the semantic category labels of each area as edges of the node, and construct a graph model and an objective function corresponding to the graph model; iteratively solve the objective function to determine the target posture corresponding to the camera and the target map corresponding to the scene; the above-mentioned target map is composed of static feature points.

[0137] In some embodiments, the above-mentioned depth semantic perception module 510 is specifically used to perform semantic segmentation and depth estimation processing on the first image group using a preset lightweight semantic segmentation and depth estimation joint network to obtain the semantic segmentation results of each feature point in the first image group and the first target depth image; wherein, in the lightweight semantic segmentation and depth estimation joint network, the semantic segmentation processing and the depth estimation processing share some convolutional layers, and the information in the semantic segmentation processing and the depth estimation processing are linked across layers.

[0138] It should be noted here that the above-mentioned device provided by the embodiment of the present invention can implement all the method steps implemented by the above-mentioned method embodiment and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be described in detail here.

[0139] Figure 6 An example of a physical structure diagram of an electronic device is shown below. Figure 6As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communications interface 620 and the memory 630 communicate with each other via the communications bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a simultaneous positioning and map construction method, the method comprising: obtaining a first image group captured by the camera of the scene at the current moment, and performing semantic segmentation and depth estimation processing on the first image group to obtain semantic segmentation results of each feature point in the first image group and a first target depth image; the first image group includes a first RGB image and a first initial depth image; obtaining a plurality of second image groups captured by the camera that are temporally correlated with the first image group and obtaining semantic segmentation results of each feature point in each second image group and a second target depth image, and performing semantic segmentation processing on the first image group based on the semantic segmentation results of each feature point in the first image group and the first initial depth image; The target depth image, the semantic segmentation results of each feature point in each second image group, and the second target depth image are used to determine the target category corresponding to each feature point in the first RGB image; the above target category is used to characterize whether the feature point is a static feature point or a dynamic feature point, and each second image group includes a second RGB image and a second initial depth image; according to the target category corresponding to each feature point in the first RGB image, the feature points whose target category in the first RGB image is dynamic are eliminated to obtain the target RGB image corresponding to the first RGB image; simultaneous positioning and map construction are performed based on the target RGB image to determine the target pose corresponding to the camera and the target map corresponding to the scene.

[0140] Furthermore, the logic instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0141] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the simultaneous positioning and map construction method provided by the above methods, which includes: obtaining a first image group captured by the camera of the scene at the current moment, and performing semantic segmentation and depth estimation on the first image group to obtain the semantic segmentation results of each feature point in the first image group and a first target depth image; the first image group includes a first RGB image and a first initial depth image; obtaining multiple second image groups captured by the camera that are temporally correlated with the first image group, and obtaining the semantic segmentation results of each feature point in each second image group and a first target depth image; Two target depth images, and determine the target category corresponding to each feature point in the first RGB image based on the semantic segmentation results of each feature point in the first image group and the first target depth image, the semantic segmentation results of each feature point in each second image group and the second target depth image; the above target category is used to characterize whether the feature point belongs to a static feature point or a dynamic feature point, and each second image group includes a second RGB image and a second initial depth image; according to the target category corresponding to each feature point in the first RGB image, the feature points whose target category in the first RGB image is dynamic are eliminated to obtain the target RGB image corresponding to the first RGB image; simultaneous positioning and map construction are performed based on the target RGB image to determine the target pose corresponding to the camera and the target map corresponding to the scene.

[0142] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the simultaneous positioning and map construction method provided by the above-mentioned methods, the method comprising: obtaining a first image group captured by the camera of the scene at the current moment, and performing semantic segmentation and depth estimation processing on the first image group to obtain the semantic segmentation results of each feature point in the first image group and a first target depth image; the first image group includes a first RGB image and a first initial depth image; obtaining a plurality of second image groups captured by the camera that are temporally correlated with the first image group and obtaining the semantic segmentation results of each feature point in each second image group and a second target depth image, and according to the first image The semantic segmentation results of each feature point in the group and the first target depth image, the semantic segmentation results of each feature point in each second image group and the second target depth image are used to determine the target category corresponding to each feature point in the first RGB image; the above target category is used to characterize whether the feature point belongs to a static feature point or a dynamic feature point, and each second image group includes a second RGB image and a second initial depth image; according to the target category corresponding to each feature point in the first RGB image, the feature points whose target category in the first RGB image is dynamic are eliminated to obtain the target RGB image corresponding to the first RGB image; simultaneous positioning and map construction are performed based on the target RGB image to determine the target pose corresponding to the camera and the target map corresponding to the scene.

[0143] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0144] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for simultaneous positioning and mapping, characterized in that: include: Obtain a first image group captured by the camera at the current moment of the scene, and perform semantic segmentation and depth estimation on the first image group using a joint semantic segmentation and depth estimation network to obtain semantic segmentation results for each feature point in the first image group and a first target depth image; The first image group includes a first RGB image and a first initial depth image; Acquire multiple second image groups captured by the camera that are temporally correlated with the first image group, and obtain semantic segmentation results and second target depth images for each feature point in each of the second image groups based on the semantic segmentation and depth estimation joint network, and determine the target category corresponding to each feature point in the first RGB image based on the semantic segmentation results and first target depth images for each feature point in the first image group and the semantic segmentation results and second target depth images for each feature point in each of the second image groups; the target category is used to characterize whether the feature point is a static feature point or a dynamic feature point, and each of the second image groups includes a second RGB image and a second initial depth image; The plurality of second image groups are second image groups at consecutive moments; According to the target categories corresponding to the respective feature points in the first RGB image, feature points belonging to the dynamic target category in the first RGB image are eliminated to obtain a target RGB image corresponding to the first RGB image; Perform simultaneous positioning and mapping based on the target RGB image to determine the target pose corresponding to the camera and the target map corresponding to the scene; The semantic segmentation result of each feature point in the first image group includes a semantic category label of each feature point in the first RGB image, the first target depth image includes a depth value of each feature point in the first RGB image, and determining the target category corresponding to each feature point in the first RGB image based on the semantic segmentation result of each feature point in the first image group and the first target depth image, and the semantic segmentation result of each feature point in each second image group and the second target depth image, includes: For each feature point in the first RGB image, determining change information of the semantic category label of the feature point based on the semantic category label corresponding to the feature point in the first RGB image and the semantic category label corresponding to the feature point in each of the second RGB images; Determining change information of the spatial position of the feature point according to the depth value corresponding to the feature point in the first target depth image and the depth value corresponding to the feature point in each of the second target depth images; Determine the target category corresponding to the feature point based on the change information of the semantic category label and the change information of the spatial position of the feature point; specifically include: for the same feature point in the first RGB image, directly sum or weighted sum the change information of the semantic category label and the change information of the spatial position of the feature point in the same frame of RGB image to determine the spatiotemporal change information of the feature point in the same frame of RGB image; obtain the weight corresponding to each frame of RGB image; determine the target category corresponding to the feature point based on the weight corresponding to each frame of RGB image and the spatiotemporal change information under each frame of RGB image.

2. The simultaneous positioning and mapping method according to claim 1, wherein: The determining, based on the semantic category label corresponding to the feature point in the first RGB image and the semantic category label corresponding to the feature point in each of the second RGB images, the change information of the semantic category label of the feature point includes: Determining an average semantic category label corresponding to the feature point based on the semantic category label corresponding to the feature point in the first RGB image and the semantic category label corresponding to the feature point in each of the second RGB images; Determine change information of the semantic category label of the feature point in each RGB image frame based on the semantic category label of the feature point in each RGB image frame and the average semantic category label corresponding to the feature point; each RGB image frame includes the first RGB image and each second RGB image.

3. The simultaneous positioning and mapping method according to claim 1, wherein: The determining, according to the depth value corresponding to the feature point in the first target depth image and the depth value corresponding to the feature point in each of the second target depth images, the change information of the spatial position of the feature point includes: Determine, according to the depth value corresponding to the feature point in the first target depth image, the depth value corresponding to the feature point in each of the second target depth images, the two-dimensional position corresponding to the feature point in the first RGB image, and the two-dimensional position corresponding to the feature point in each of the second RGB images, the spatial position corresponding to the feature point in each frame of the RGB image; Determine the average spatial position corresponding to the feature point based on the spatial position corresponding to each frame of the RGB image; According to the spatial position of the feature point in each frame of RGB image and the average spatial position corresponding to the feature point, the change information of the spatial position of the feature point in each frame of RGB image is determined.

4. The simultaneous positioning and mapping method according to claim 1, wherein: The change information of the semantic category label of the feature point includes change information of the semantic category label of the feature point in each frame of RGB image, the change information of the spatial position of the feature point includes change information of the spatial position of the feature point in each frame of RGB image, and determining the target category corresponding to the feature point based on the change information of the semantic category label and the change information of the spatial position of the feature point includes: Calculate the spatiotemporal consistency quantization value corresponding to the feature point based on the change information of the semantic category label of the feature point in each frame of RGB image and the change information of the spatial position of the feature point in each frame of RGB image; The target category corresponding to the feature point is determined according to the spatiotemporal consistency quantization value corresponding to the feature point and a preset threshold.

5. The simultaneous positioning and mapping method according to any one of claims 1 to 4, characterized in that: The simultaneous positioning and mapping according to the target RGB image to determine the target pose corresponding to the camera and the target map corresponding to the scene includes: Obtaining static feature points of the target category in the target RGB image, and calculating the corresponding projection positions of the static feature points on the image plane of the camera; Obtaining semantic category labels corresponding to each region in the target RGB image and obtaining the current pose of the camera; Taking the current pose of the camera as a node, the projection position of the static feature point and the semantic category label of each region as the edge of the node, and constructing a graph model and an objective function corresponding to the graph model; The objective function is iteratively solved to determine a target pose corresponding to the camera and a target map corresponding to the scene; the target map is composed of the static feature points.

6. The simultaneous positioning and mapping method according to any one of claims 1 to 4, characterized in that: The performing semantic segmentation and depth estimation processing on the first image group to obtain a semantic segmentation result of each feature point in the first image group and a first target depth image includes: Performing semantic segmentation and depth estimation processing on the first image group using a preset lightweight semantic segmentation and depth estimation joint network to obtain semantic segmentation results of each feature point in the first image group and a first target depth image; In the lightweight semantic segmentation and depth estimation joint network, the semantic segmentation processing and the depth estimation processing share some convolutional layers, and the information in the semantic segmentation processing and the depth estimation processing are linked across layers.

7. A simultaneous positioning and mapping device, characterized in that: include: a depth semantic perception module, configured to obtain a first image group captured by the camera of the scene at the current moment, and perform semantic segmentation and depth estimation on the first image group using a joint semantic segmentation and depth estimation network to obtain semantic segmentation results for each feature point in the first image group and a first target depth image; The first image group includes a first RGB image and a first initial depth image; a feature point category determination module, configured to obtain a plurality of second image groups captured by the camera and having a temporal correlation with the first image group, and to obtain a semantic segmentation result and a second target depth image for each feature point in each of the second image groups based on the semantic segmentation and depth estimation joint network, and to determine a target category corresponding to each feature point in the first RGB image based on the semantic segmentation result and the first target depth image for each feature point in the first image group and the semantic segmentation result and the second target depth image for each feature point in each of the second image groups; the target category is used to characterize whether the feature point is a static feature point or a dynamic feature point, and each of the second image groups includes a second RGB image and a second initial depth image; The plurality of second image groups are second image groups at consecutive moments; a dynamic feature point removal module, configured to remove feature points belonging to dynamic target categories in the first RGB image according to target categories corresponding to the feature points in the first RGB image, and obtain a target RGB image corresponding to the first RGB image; A positioning and mapping module, configured to simultaneously perform positioning and mapping based on the target RGB image, and determine the target pose corresponding to the camera and the target map corresponding to the scene; The semantic segmentation result of each feature point in the first image group includes a semantic category label of each feature point in the first RGB image, the first target depth image includes a depth value of each feature point in the first RGB image, and the feature point category determination module includes: a semantic label change information determining unit, configured to determine, for each feature point in the first RGB image, change information of the semantic category label of the feature point based on the semantic category label corresponding to the feature point in the first RGB image and the semantic category label corresponding to the feature point in each of the second RGB images; a spatial position change information determining unit, configured to determine change information of the spatial position of the feature point based on a depth value corresponding to the feature point in the first target depth image and a depth value corresponding to the feature point in each of the second target depth images; The feature point category determination unit is used to determine the target category corresponding to the feature point based on the change information of the semantic category label and the change information of the spatial position of the feature point; specifically comprising: for the same feature point in the first RGB image, directly summing or weighted summing the change information of the semantic category label and the change information of the spatial position of the feature point in the same frame of RGB image to determine the spatiotemporal change information of the feature point in the same frame of RGB image; obtaining the weight corresponding to each frame of RGB image; and determining the target category corresponding to the feature point based on the weight corresponding to each frame of RGB image and the spatiotemporal change information in each frame of RGB image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the simultaneous positioning and mapping method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the simultaneous positioning and mapping method according to any one of claims 1 to 6 is implemented.

Images