Visual synchronous positioning and mapping method and device based on semantics in dynamic scene

Abstract

The invention provides a semantic-based visual synchronous localization and mapping method and device in a dynamic scene, and the method comprises the steps: obtaining a target image, inputting the target image into a semantic segmentation network, and obtaining a semantic segmentation image; wherein the target image comprises color information and depth information; the semantic segmentation image comprises a plurality of feature points and a mask of a dynamic target in the target image; determining dynamic feature points and potential dynamic feature points in the mask of the dynamic target and the mask neighborhood of the dynamic target through a random sample consistency algorithm based on depth information on the basis of the plurality of feature points; and feature points except the dynamic feature points and the potential dynamic feature points are determined as static feature points, a target three-dimensional map point is generated based on the target image and the static feature points, and a target three-dimensional map is constructed based on the target three-dimensional map point. Through the method provided by the invention, the positioning precision of the unmanned aerial vehicle in a dynamic scene is improved.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a semantic-based visual synchronous positioning and mapping method and device in a dynamic scene. Background Art

[0002] In fire rescue scenarios, firefighting drones need to achieve self-positioning and navigation without external navigation infrastructure in order to quickly reach the source of the fire and carry out fire extinguishing and search and rescue missions.

[0003] Traditional visual simultaneous localization and mapping (SLAM) technologies, such as ORB-SLAM3, perform well in static scenes and can provide high-precision pose estimation and map construction. However, in dynamic scenes, such as fire scenes, there are a large number of moving dynamic objects in the environment (such as rescuers, vehicles, and flames). These dynamic objects can cause serious positioning errors and map distortion in traditional SLAM algorithms. In addition, traditional SLAM algorithms lack a mechanism to handle dynamic objects and cannot effectively distinguish and eliminate dynamic feature points, resulting in a significant decrease in positioning accuracy and robustness.

[0004] How to achieve high-precision positioning and map construction of UAVs in complex dynamic environments is a technical problem that needs to be solved at present. Summary of the invention

[0005] The present invention provides a semantic-based visual synchronous positioning and mapping method and device in a dynamic scene, so as to solve the defects existing in the prior art.

[0006] The present invention provides a semantic-based visual synchronous positioning and mapping method in a dynamic scene, comprising the following steps: Acquire a target image, and input the target image into a semantic segmentation network to obtain a semantic segmentation image; wherein the target image includes: color information and depth information; the semantic segmentation image includes: a plurality of feature points and a mask of a dynamic target in the target image; Determine dynamic feature points and potential dynamic feature points in the mask of the dynamic target and in the mask neighborhood of the dynamic target based on the plurality of feature points by using a random sample consistency algorithm based on depth information; Feature points other than the dynamic feature points and the potential dynamic feature points are determined as static feature points, target three-dimensional map points are generated based on the target image and the static feature points, and a target three-dimensional map is constructed based on the target three-dimensional map points.

[0007] According to a semantic-based visual synchronous positioning and mapping method in a dynamic scene provided by the present invention, the target image is acquired and the target image is input into a semantic segmentation network to obtain a semantic segmentation image, including: Obtain a target image, and input the target image into a semantic segmentation network to obtain a semantic segmentation file of a binary mask; The semantic segmentation file of the binary mask is fused with the target image to obtain the semantic segmentation image.

[0008] According to a semantic-based visual synchronous positioning and mapping method in a dynamic scene provided by the present invention, the dynamic feature points and potential dynamic feature points are determined in the mask of the dynamic target and the mask neighborhood of the dynamic target based on the multiple feature points through a random sample consistency algorithm based on depth information, including: Calculating an initial depth variance between a first feature point and a second feature point; wherein the first feature point and the second feature point are: any two feature points within the mask of the dynamic target; The target depth variance of the third feature point is calculated based on the initial depth variance; wherein the third feature point is: any feature point in the dynamic target mask except the first feature point and the second feature point or any feature point in the neighborhood of the dynamic target mask; When the target depth variance is less than a preset variance threshold, adding the third feature point to a dynamic feature point set or a potential dynamic feature point set; The above steps are repeatedly performed until the mask of the dynamic target and the feature points in the neighborhood of the mask of the dynamic target are judged.

[0009] According to a semantic-based visual synchronous positioning and mapping method in a dynamic scene provided by the present invention, the calculating of the initial depth variance between the first feature point and the second feature point includes: An initial depth variance between the first feature point and the second feature point is calculated based on the depth value of the first feature point, the depth value of the second feature point, and an initial average depth value of the first feature point and the second feature point.

[0010] According to a semantic-based visual synchronous positioning and mapping method in a dynamic scene provided by the present invention, the target depth variance of the third feature point is calculated based on the initial depth variance, including: The target depth variance is calculated based on the depth value of the third feature point, the initial depth variance and the target average depth value; wherein the target average depth value is: an average value of the depth value of the third feature point and the initial average depth value.

[0011] According to a semantic-based visual synchronous positioning and mapping method in a dynamic scene provided by the present invention, the target three-dimensional map points are generated based on the target image and the static feature points, and the target three-dimensional map is constructed based on the target three-dimensional map points, including: Based on the target image and the static feature points, generating initial three-dimensional map points; The initial three-dimensional map points and camera pose are optimized by a local bundle adjustment method to generate target three-dimensional map points, and the target three-dimensional map is constructed based on the target three-dimensional map points and the optimized camera pose.

[0012] According to a semantic-based visual synchronous positioning and mapping method in a dynamic scene provided by the present invention, after generating target three-dimensional map points based on the target image and the static feature points, and constructing a target three-dimensional map based on the target three-dimensional map points, the method further includes: Loop closure detection and global bundle adjustment method optimization are performed on the target three-dimensional map to improve the accuracy of the target three-dimensional map.

[0013] The present invention also provides a semantic-based visual synchronous positioning and mapping device in a dynamic scene, comprising the following modules: A semantic segmentation module, used to obtain a target image and input the target image into a semantic segmentation network to obtain a semantic segmentation image; wherein the target image includes: color information and depth information; the semantic segmentation image includes: multiple feature points and a mask of a dynamic target in the target image; A feature point determination module, configured to determine, based on the plurality of feature points, dynamic feature points and potential dynamic feature points in the mask of the dynamic target and in the mask neighborhood of the dynamic target by using a random sample consistency algorithm based on depth information; A map construction module is used to determine feature points other than the dynamic feature points and the potential dynamic feature points as static feature points, generate target three-dimensional map points based on the target image and the static feature points, and construct a target three-dimensional map based on the target three-dimensional map points.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for semantic-based visual synchronous positioning and mapping in a dynamic scene as described in any one of the above-mentioned methods is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a semantic-based visual synchronous positioning and mapping method in a dynamic scene as described in any of the above.

[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the semantic-based visual synchronous positioning and mapping method in a dynamic scene as described in any of the above.

[0017] The present invention provides a method and device for visual synchronous positioning and mapping based on semantics in a dynamic scene, by acquiring a target image and inputting the target image into a semantic segmentation network to obtain a semantic segmentation image; wherein the target image includes: color information and depth information; the semantic segmentation image includes: multiple feature points and a mask of a dynamic target in the target image; based on the multiple feature points, a random sample consistency algorithm based on depth information is used to determine dynamic feature points and potential dynamic feature points in the mask of the dynamic target and the mask neighborhood of the dynamic target; feature points other than the dynamic feature points and the potential dynamic feature points are determined as static feature points, and target three-dimensional map points are generated based on the target image and the static feature points, and a target three-dimensional map is constructed based on the target three-dimensional map points. It can be seen from this that the present invention can effectively identify and eliminate dynamic feature points and potential dynamic feature points through a method combining semantic segmentation and geometric constraints, thereby significantly improving the robustness and positioning accuracy of the drone in dynamic scenes and improving the quality of map construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of the semantic-based visual synchronous positioning and mapping method in a dynamic scene provided by the present invention.

[0020] Figure 2 It is a schematic diagram of the result of semantic segmentation of the target image provided by the present invention.

[0021] Figure 3 It is a schematic diagram of the results of the semantic-based visual synchronous positioning and mapping method in a dynamic scene provided by the present invention.

[0022] Figure 4 It is a complete flow chart of the semantic-based visual synchronous positioning and mapping method in dynamic scenes provided by the present invention.

[0023] Figure 5 It is a structural schematic diagram of a semantic-based visual synchronous positioning and mapping device in a dynamic scene provided by the present invention.

[0024] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Before describing the method of the present invention in detail, the terms involved in the present invention are first explained schematically.

[0027] Visual SLAM: Simultaneous positioning and mapping technology, which uses cameras to obtain environmental images and uses image processing technology to achieve drone self-positioning and environmental map construction.

[0028] Semantic segmentation: Segment an image into regions with semantic labels, such as people, cars, buildings, etc.

[0029] Geometric constraints: Use geometric information to match and eliminate feature points.

[0030] FastestDet: An ultra-lightweight, anchor-free real-time object detection algorithm designed for platforms with limited computing resources, with extremely fast detection speed and low power consumption, suitable for edge devices.

[0031] EfficientViT: A lightweight visual Transformer model for efficient semantic segmentation tasks.

[0032] Depth-RANSAC: A random sample consensus algorithm based on depth information, used to process noisy data and remove dynamic feature points and potential dynamic feature points.

[0033] Combine the following Figure 1-Figure 6 The present invention describes a semantic-based visual synchronous positioning and mapping method and device in a dynamic scene.

[0034] Figure 1 is a flow chart of a semantic-based visual synchronous positioning and mapping method in a dynamic scene provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 100, acquiring a target image, and inputting the target image into a semantic segmentation network to obtain a semantic segmentation image; wherein the target image includes: color information and depth information; the semantic segmentation image includes: multiple feature points and masks of dynamic targets in the target image.

[0035] It should be noted that the embodiment of the present invention applies the semantic segmentation method to visual SLAM to cope with the challenges in dynamic scenes. Semantic segmentation technology can identify different objects (such as people, cars, buildings, etc.) in the image by segmenting the image into regions with semantic labels, thereby providing more environmental information for the SLAM algorithm.

[0036] Specifically, step 100 acquires a target image and inputs the target image into a semantic segmentation network to obtain a semantic segmentation image, including: Obtain a target image, and input the target image into a semantic segmentation network to obtain a semantic segmentation file of a binary mask; The semantic segmentation file of the binary mask is fused with the target image to obtain the semantic segmentation image.

[0037] It should be noted that in this embodiment, the target image may be an RGB-D image. An RGB-D image is an image that contains both color (RGB) and depth (D) information. The RGB channel provides the color information of the object, and the D channel records the distance or depth value of the pixel point in space.

[0038] In one embodiment, the original RGB-D image is cropped into a 512*512*3 image and input into the semantic segmentation network, and the output is a text file of a binary mask. Specifically, the semantic segmentation network used in this embodiment is based on the FastestDet framework, and the backbone network is replaced by EfficientViT (Efficient Vision Transformer). The network has 8 channels, the query and key dimensions are 8, the value dimension is 4, the number of attention heads is 8, and the deep convolution kernel size is [5, 5, 5, 5]. The input of the network is a 512*512*3 RGB image, and then multiple images are spliced. The output is a text file of a binary mask, and the mask and the original image can be fused to generate a visualization result. Labels are added to all segmented objects, and the undefined background is set to -1, the static background is set to 0, and other dynamic objects such as people are labeled in turn to achieve semantic segmentation of objects in the scene. Figure 2 is a schematic diagram of the result of semantic segmentation of the target image provided by the present invention. The semantic segmentation result is as follows Figure 2 shown.

[0039] Step 200: Determine dynamic feature points and potential dynamic feature points in the mask of the dynamic target and in the mask neighborhood of the dynamic target based on the multiple feature points by using a random sample consistency algorithm based on depth information.

[0040] It should be noted that this embodiment eliminates feature mismatch pairs based on the Depth-RANSAC (random sample consistency based on depth information) algorithm. Depth-RANSAC is a method for denoising data samples containing noise. Considering the high computational cost of pairwise feature point matching, Depth-RANSAC extracts four point pairs between two frames to infer the homography matrix parameters between adjacent frames. Based on the segmentation results generated by the semantic segmentation network of the above embodiment, the semantic label of the object can be obtained. The objects in the image are divided into dynamic targets (such as "people"), static objects, and potential dynamic targets (such as "chairs"). Potential dynamic targets are a type of object that changes state with dynamic targets, so they also need to be eliminated in the visual SLAM of this embodiment.

[0041] The use of Depth-RANSAC comes from three assumptions: (1) The semantic information obtained through semantic segmentation can obtain the dynamic target contour with high confidence; (2) Feature points that appear on the same semantic information have almost the same depth values; (3) Objects with similar depth values ​​around dynamic semantic information are potential dynamic targets.

[0042] The above assumptions indicate that the depth values ​​of dynamic targets in the environment fluctuate within a certain range, and objects with similar depth values ​​around semantic information are potential dynamic targets with high confidence. In real environments, most of the above assumptions can be met.

[0043] Specifically, step 200 determines the dynamic feature points and potential dynamic feature points in the mask of the dynamic target and the mask neighborhood of the dynamic target based on the multiple feature points through a random sample consistency algorithm based on depth information, including: Step 210: Calculate an initial depth variance between a first feature point and a second feature point; wherein the first feature point and the second feature point are any two feature points within the mask of the dynamic target.

[0044] Specifically, step 210 includes: An initial depth variance between the first feature point and the second feature point is calculated based on the depth value of the first feature point, the depth value of the second feature point, and an initial average depth value of the first feature point and the second feature point.

[0045] In one embodiment, the mask of the dynamic target is obtained using semantic information, and the Depth-RANSAC algorithm is used to find feature points in the high-confidence dynamic semantic information mask, and these points are stored in a set, the dynamic point set is S, and the temporary dynamic feature point set is S1. Then, two feature points are selected and their initial depth variance is calculated.

[0046]

[0047] in, represents the initial depth variance, represents the depth value of the first feature point, represents the depth value of the second feature point, Represents the initial average depth value of the first feature point and the second feature point.

[0048]

[0049] Step 220: Calculate the target depth variance of the third feature point based on the initial depth variance; wherein the third feature point is: any feature point in the dynamic target mask except the first feature point and the second feature point or any feature point in the neighborhood of the dynamic target mask.

[0050] Specifically, step 220 includes: The target depth variance is calculated based on the depth value of the third feature point, the initial depth variance and the target average depth value; wherein the target average depth value is: an average value of the depth value of the third feature point and the initial average depth value.

[0051] In one embodiment, for the current feature point P3 and the calculated initial depth variance , calculate the new depth variance .

[0052]

[0053] in, represents the target depth variance, represents the depth value of the third feature point, Indicates the target average depth value.

[0054] Step 230: When the target depth variance is less than a preset variance threshold, add the third feature point to a dynamic feature point set or a potential dynamic feature point set.

[0055] In one embodiment, for a feature point Pi, if its depth variance If the following conditions are met, the feature point is considered to be an interior point:

[0056] in, Indicates the preset variance threshold.

[0057] If the current number of internal points Greater than the maximum number of recorded inliers , then update the maximum number of internal points and the set of dynamic feature points, and expand the neighborhood.

[0058]

[0059] Step 240: Repeat the above steps until the mask of the dynamic target and the feature points in the neighborhood of the mask of the dynamic target are judged.

[0060] Specifically, when the neighborhood is no longer expanded, all recorded dynamic feature points (dynamic feature points and potential dynamic feature points) are removed and only static feature points are retained. Figure 3 is a schematic diagram of the results of the semantic-based visual synchronous positioning and mapping method in a dynamic scene provided by the present invention. The results before and after elimination are as follows Figure 3 As shown in the figure, (a) and (b) are feature point displays of ORB-SLAM3, and (c) and (d) are feature point displays of the algorithm of the embodiment of the present invention. FD-SLAM (FastestDet-SLAM algorithm) removes the dynamic feature points of the body part in the image and the potential dynamic feature points of the chair part.

[0061] Step 300: Determine feature points other than the dynamic feature points and the potential dynamic feature points as static feature points, generate target three-dimensional map points based on the target image and the static feature points, and construct a target three-dimensional map based on the target three-dimensional map points.

[0062] Specifically, step 300 generates target three-dimensional map points based on the target image and the static feature points, and constructs a target three-dimensional map based on the target three-dimensional map points, including: Based on the target image and the static feature points, generating initial three-dimensional map points; The initial three-dimensional map points and camera pose are optimized by a local bundle adjustment method to generate target three-dimensional map points, and the target three-dimensional map is constructed based on the target three-dimensional map points and the optimized camera pose.

[0063] In one embodiment, new three-dimensional map points are obtained based on the RGB-D image and the remaining static feature points, and local bundle adjustment optimization is performed to optimize the camera pose and map points simultaneously.

[0064] Furthermore, by querying the visual word bag database to select loop detection targets, semantic information detection and loop fusion are used to perform loop detection and global BA (global bundle adjustment) optimization to improve map accuracy.

[0065] The above is a step description of the method for visual synchronous positioning and mapping based on semantics in dynamic scenes provided by the present invention. From the description of the above steps, it can be seen that according to the method for visual synchronous positioning and mapping based on semantics in dynamic scenes provided by the present invention, a semantic segmentation image is obtained by acquiring a target image and inputting the target image into a semantic segmentation network; wherein the target image contains: color information and depth information; the semantic segmentation image contains: multiple feature points and a mask of a dynamic target in the target image; based on the multiple feature points, a random sample consistency algorithm based on depth information is used to determine dynamic feature points and potential dynamic feature points in the mask of the dynamic target and the mask neighborhood of the dynamic target; feature points other than the dynamic feature points and the potential dynamic feature points are determined as static feature points, and target three-dimensional map points are generated based on the target image and the static feature points, and a target three-dimensional map is constructed based on the target three-dimensional map points. It can be seen from this that the present invention can effectively identify and eliminate dynamic feature points and potential dynamic feature points through a method combining semantic segmentation and geometric constraints, thereby significantly improving the robustness and positioning accuracy of the drone in dynamic scenes and improving the quality of map construction.

[0066] Figure 4 This is a complete flow chart of the semantic-based visual synchronous positioning and mapping method in dynamic scenes provided by the present invention. Figure 4 The complete process of the semantic-based visual synchronous positioning and mapping method in dynamic scenes provided by the present invention is described.

[0067] 1. Input the original RGB-D image and perform semantic segmentation to preliminarily detect dynamic targets.

[0068] 2. Design a dynamic target mask augmentation method and use the Depth-RANSAC algorithm to calculate the depth variance between feature points within the mask and in the neighborhood.

[0069] 3. Eliminate the determined dynamic feature points and potential dynamic feature points, use the retained static feature points to generate new map points and optimize them.

[0070] 4. Perform loop detection and global BA optimization.

[0071] The semantic-based visual synchronous positioning and mapping method in dynamic scenes provided by the embodiment of the present invention combines semantic information and geometric information to effectively remove dynamic feature points and potential dynamic feature points, thereby improving the robustness of the algorithm in dynamic scenes. The accuracy of pose estimation is enhanced through the EfficientViT and Depth-RANSAC methods, and high-quality maps are generated. Among them, EfficientViT is a lightweight model suitable for real-time execution of edge devices, which improves the real-time and practicality of the algorithm.

[0072] Furthermore, the present invention selects the TUM dataset to quantitatively test FD-SLAM and qualitatively demonstrates the effect of feature extraction in a dynamic environment. The present invention selects five sequences, namely w_xyz, w_half, w_rpy, w_static and s_static. Among them, walking (w) is a high dynamic sequence and sitting (s) is a low dynamic sequence. The experimental configuration of the present invention is 64 GB memory, Intel Core i9 CPU and NVIDIA GeForce RTX 3070 GPU.

[0073] The present invention selects the absolute trajectory error (ATE) to measure the degree of positioning drift, which can well reflect the accuracy. The relative pose error (RPE) is used to measure the pose between the image frames, which is helpful to evaluate the translation and rotation drift. The accuracy of the pose estimation of SLAM can be tested by the root mean square error (RMSE), and the robustness can be tested by the standard deviation (SD).

[0074] Table 1

[0075] Table 2

[0076] Tables 1 and 2 show the ATE and RPE of ORB-SLAM3, DS-SLAM, RDS-SLAM, Dyna-SLAM and FD-SLAM in five sequences. Table 3 shows the comparison between FD-SLAM and ORB-SLAM3. The accuracy and robustness of FD-SLAM are greatly improved compared with ORB-SLAM3. Although DS-SLAM, Dyna-SLAM and RDS-SLAM have obvious improvements, FD-SLAM has better performance than them. According to the ATE results, the RMSE can be improved by up to 98.41% compared with ORB-SLAM3, and the SD can be improved by up to 98.56%.

[0077] Table 3

[0078] The following is a description of the semantic-based visual synchronous positioning and mapping device for dynamic scenes provided by the present invention. The semantic-based visual synchronous positioning and mapping device for dynamic scenes described below and the semantic-based visual synchronous positioning and mapping method for dynamic scenes described above can be referenced to each other.

[0079] Figure 5 is a schematic diagram of the structure of the semantic-based visual synchronous positioning and mapping device in a dynamic scene provided by the present invention, such as Figure 5 As shown, the present invention provides a semantic-based visual synchronous positioning and mapping device in a dynamic scene, comprising: The semantic segmentation module 501 is used to obtain a target image and input the target image into a semantic segmentation network to obtain a semantic segmentation image; wherein the target image includes: color information and depth information; the semantic segmentation image includes: multiple feature points and a mask of a dynamic target in the target image; A feature point determination module 502 is used to determine dynamic feature points and potential dynamic feature points in the mask of the dynamic target and the mask neighborhood of the dynamic target based on the plurality of feature points by using a random sample consistency algorithm based on depth information; The map construction module 503 is used to determine the feature points other than the dynamic feature points and the potential dynamic feature points as static feature points, generate target three-dimensional map points based on the target image and the static feature points, and construct a target three-dimensional map based on the target three-dimensional map points.

[0080] The semantic-based visual synchronous positioning and mapping device in a dynamic scene provided by the present invention obtains a semantic segmentation image by acquiring a target image and inputting the target image into a semantic segmentation network; wherein the target image includes: color information and depth information; the semantic segmentation image includes: multiple feature points and a mask of a dynamic target in the target image; based on the multiple feature points, a random sample consistency algorithm based on depth information is used to determine dynamic feature points and potential dynamic feature points in the mask of the dynamic target and the mask neighborhood of the dynamic target; feature points other than the dynamic feature points and the potential dynamic feature points are determined as static feature points, target three-dimensional map points are generated based on the target image and the static feature points, and a target three-dimensional map is constructed based on the target three-dimensional map points. It can be seen from this that the present invention can effectively identify and eliminate dynamic feature points and potential dynamic feature points through a method combining semantic segmentation and geometric constraints, thereby significantly improving the robustness and positioning accuracy of the drone in dynamic scenes and improving the quality of map construction.

[0081] Based on the above embodiment, in this embodiment, the semantic segmentation module 501 is specifically used for: Obtain a target image, and input the target image into a semantic segmentation network to obtain a semantic segmentation file of a binary mask; The semantic segmentation file of the binary mask is fused with the target image to obtain the semantic segmentation image.

[0082] Based on the above embodiment, in this embodiment, the feature point determination module 502 is specifically used for: Calculating an initial depth variance between a first feature point and a second feature point; wherein the first feature point and the second feature point are: any two feature points within the mask of the dynamic target; The target depth variance of the third feature point is calculated based on the initial depth variance; wherein the third feature point is: any feature point in the dynamic target mask except the first feature point and the second feature point or any feature point in the neighborhood of the dynamic target mask; When the target depth variance is less than a preset variance threshold, adding the third feature point to a dynamic feature point set or a potential dynamic feature point set; The above steps are repeatedly performed until the mask of the dynamic target and the feature points in the neighborhood of the mask of the dynamic target are judged.

[0083] Based on the above embodiment, in this embodiment, the device further includes a calculation module, which is specifically used to: An initial depth variance between the first feature point and the second feature point is calculated based on the depth value of the first feature point, the depth value of the second feature point, and an initial average depth value of the first feature point and the second feature point.

[0084] Based on the above embodiments, in this embodiment, the calculation module is specifically used for: The target depth variance is calculated based on the depth value of the third feature point, the initial depth variance and the target average depth value; wherein the target average depth value is: an average value of the depth value of the third feature point and the initial average depth value.

[0085] Based on the above embodiment, in this embodiment, the map construction module 503 is specifically used for: Based on the target image and the static feature points, generating initial three-dimensional map points; The initial three-dimensional map points and camera pose are optimized by a local bundle adjustment method to generate target three-dimensional map points, and the target three-dimensional map is constructed based on the target three-dimensional map points and the optimized camera pose.

[0086] Based on the above embodiment, in this embodiment, the device further includes an optimization module, which is specifically used to: After generating target three-dimensional map points based on the target image and the static feature points and constructing a target three-dimensional map based on the target three-dimensional map points, loop closure detection and global bundle adjustment method optimization are performed on the target three-dimensional map to improve the accuracy of the target three-dimensional map.

[0087] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may be a robot or other electronic device, and the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the semantic-based visual synchronous positioning and mapping method in a dynamic scene, including: Acquire a target image, and input the target image into a semantic segmentation network to obtain a semantic segmentation image; wherein the target image includes: color information and depth information; the semantic segmentation image includes: a plurality of feature points and a mask of a dynamic target in the target image; Determine dynamic feature points and potential dynamic feature points in the mask of the dynamic target and in the mask neighborhood of the dynamic target based on the plurality of feature points by using a random sample consistency algorithm based on depth information; Feature points other than the dynamic feature points and the potential dynamic feature points are determined as static feature points, target three-dimensional map points are generated based on the target image and the static feature points, and a target three-dimensional map is constructed based on the target three-dimensional map points.

[0088] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0089] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the semantic-based visual synchronous positioning and mapping method in a dynamic scene provided by the above methods, including: Acquire a target image, and input the target image into a semantic segmentation network to obtain a semantic segmentation image; wherein the target image includes: color information and depth information; the semantic segmentation image includes: a plurality of feature points and a mask of a dynamic target in the target image; Determine dynamic feature points and potential dynamic feature points in the mask of the dynamic target and in the mask neighborhood of the dynamic target based on the plurality of feature points by using a random sample consistency algorithm based on depth information; Feature points other than the dynamic feature points and the potential dynamic feature points are determined as static feature points, target three-dimensional map points are generated based on the target image and the static feature points, and a target three-dimensional map is constructed based on the target three-dimensional map points.

[0090] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the semantic-based visual synchronous positioning and mapping method in a dynamic scene provided by the above methods, including: Acquire a target image, and input the target image into a semantic segmentation network to obtain a semantic segmentation image; wherein the target image includes: color information and depth information; the semantic segmentation image includes: a plurality of feature points and a mask of a dynamic target in the target image; Determine dynamic feature points and potential dynamic feature points in the mask of the dynamic target and in the mask neighborhood of the dynamic target based on the plurality of feature points by using a random sample consistency algorithm based on depth information; Feature points other than the dynamic feature points and the potential dynamic feature points are determined as static feature points, target three-dimensional map points are generated based on the target image and the static feature points, and a target three-dimensional map is constructed based on the target three-dimensional map points.

[0091] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semantic-based visual synchronous positioning and mapping method in dynamic scenes, characterized in that: include: Acquire a target image, and input the target image into a semantic segmentation network to obtain a semantic segmentation image; wherein the target image includes: color information and depth information; the semantic segmentation image includes: a plurality of feature points and a mask of a dynamic target in the target image; Determine dynamic feature points and potential dynamic feature points in the mask of the dynamic target and in the mask neighborhood of the dynamic target based on the plurality of feature points by using a random sample consistency algorithm based on depth information; Feature points other than the dynamic feature points and the potential dynamic feature points are determined as static feature points, target three-dimensional map points are generated based on the target image and the static feature points, and a target three-dimensional map is constructed based on the target three-dimensional map points.

2. The semantic-based visual synchronous positioning and mapping method in dynamic scenes according to claim 1, characterized in that: The step of acquiring a target image and inputting the target image into a semantic segmentation network to obtain a semantic segmentation image includes: Obtain a target image, and input the target image into a semantic segmentation network to obtain a semantic segmentation file of a binary mask; The semantic segmentation file of the binary mask is fused with the target image to obtain the semantic segmentation image.

3. The semantic-based visual synchronous positioning and mapping method in dynamic scenes according to claim 1, characterized in that: The determining of dynamic feature points and potential dynamic feature points in the mask of the dynamic target and the mask neighborhood of the dynamic target based on the plurality of feature points through a random sample consistency algorithm based on depth information includes: Calculating an initial depth variance between a first feature point and a second feature point; wherein the first feature point and the second feature point are: any two feature points within the mask of the dynamic target; The target depth variance of the third feature point is calculated based on the initial depth variance; wherein the third feature point is: any feature point in the dynamic target mask except the first feature point and the second feature point or any feature point in the neighborhood of the dynamic target mask; When the target depth variance is less than a preset variance threshold, adding the third feature point to a dynamic feature point set or a potential dynamic feature point set; The above steps are repeatedly performed until the mask of the dynamic target and the feature points in the neighborhood of the mask of the dynamic target are judged.

4. The semantic-based visual synchronous positioning and mapping method in dynamic scenes according to claim 3, characterized in that: The calculating the initial depth variance between the first feature point and the second feature point includes: An initial depth variance between the first feature point and the second feature point is calculated based on the depth value of the first feature point, the depth value of the second feature point, and an initial average depth value of the first feature point and the second feature point.

5. The semantic-based visual synchronous positioning and mapping method in dynamic scenes according to claim 3, characterized in that: The step of calculating a target depth variance of a third feature point based on the initial depth variance includes: The target depth variance is calculated based on the depth value of the third feature point, the initial depth variance and the target average depth value; wherein the target average depth value is: an average value of the depth value of the third feature point and the initial average depth value.

6. The semantic-based visual synchronous positioning and mapping method in dynamic scenes according to claim 1, characterized in that: The step of generating target three-dimensional map points based on the target image and the static feature points, and constructing a target three-dimensional map based on the target three-dimensional map points comprises: Based on the target image and the static feature points, generating initial three-dimensional map points; The initial three-dimensional map points and camera pose are optimized by a local bundle adjustment method to generate target three-dimensional map points, and the target three-dimensional map is constructed based on the target three-dimensional map points and the optimized camera pose.

7. The semantic-based visual synchronous positioning and mapping method in dynamic scenes according to claim 1, characterized in that: After generating target three-dimensional map points based on the target image and the static feature points, and constructing a target three-dimensional map based on the target three-dimensional map points, the method further includes: Loop closure detection and global bundle adjustment method optimization are performed on the target three-dimensional map to improve the accuracy of the target three-dimensional map.

8. A semantic-based visual synchronous positioning and mapping device in a dynamic scene, characterized in that: include: A semantic segmentation module, used to obtain a target image and input the target image into a semantic segmentation network to obtain a semantic segmentation image; wherein the target image includes: color information and depth information; the semantic segmentation image includes: multiple feature points and a mask of a dynamic target in the target image; A feature point determination module, configured to determine, based on the plurality of feature points, dynamic feature points and potential dynamic feature points in the mask of the dynamic target and in the mask neighborhood of the dynamic target by using a random sample consistency algorithm based on depth information; A map construction module is used to determine feature points other than the dynamic feature points and the potential dynamic feature points as static feature points, generate target three-dimensional map points based on the target image and the static feature points, and construct a target three-dimensional map based on the target three-dimensional map points.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for semantic-based visual synchronous positioning and mapping in a dynamic scene as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for semantic-based visual synchronous positioning and mapping in a dynamic scene as described in any one of claims 1 to 7 is implemented.

Images