Synchronous localization and mapping method and device

Abstract

The invention relates to the technical field of computer vision, and discloses a synchronous localization and map construction method and device, and the method comprises the steps: obtaining multi-frame continuous image data of a current environment in real time, and extracting a visual feature point corresponding to each frame of image data; respectively performing semantic segmentation processing and optical flow calculation processing on the image data to generate a corresponding semantic mask and an optical flow mask; eliminating dynamic feature points in the visual feature points based on the semantic mask and the optical flow mask, and retaining static points; and generating a semantic point cloud based on the static points, and incrementally constructing a semantic octree map based on the semantic point cloud and an octree module. According to the method, the positioning accuracy and the mapping practicability in the dynamic environment can be improved, and the problems that the positioning accuracy of a traditional visual SLAM technology in the dynamic environment is seriously reduced and the subsequent upper-layer application requirements cannot be met are solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method and device for simultaneous positioning and map construction. Background Art

[0002] With the rapid development of artificial intelligence and computer vision technologies, mobile robots are increasingly being used in numerous fields, such as logistics, service, and cleaning, significantly improving production efficiency and intelligence. Simultaneous Localization and Mapping (SLAM) technology, at the core of autonomous robot navigation, uses sensors to acquire real-time environmental data, enabling robot pose estimation and map construction.

[0003] Currently, most mainstream visual SLAM systems assume a static environment. While they offer stable performance in static scenes, they still face significant limitations in dynamic environments. The presence of dynamic objects, such as pedestrians or moving furniture, significantly degrades system performance, reduces positioning accuracy, and renders map construction unreliable. Even the recently emerging semantic SLAM technology, which incorporates semantic information for object recognition and classification, is largely limited to eliminating a priori dynamic objects. It lacks effective identification and processing methods for the numerous non-a priori dynamic objects present in dynamic environments, making stable operation difficult in complex dynamic scenes. Furthermore, to meet the demands of subsequent upper-layer robotic applications, the traditional approach of constructing only sparse point cloud maps is no longer suitable. Constructing richer, denser, and more semantically informed maps is becoming the trend. However, existing technologies often face challenges in achieving this goal, such as high computational costs and poor real-time performance, limiting their widespread application in real-world scenarios. Consequently, traditional SLAM technology suffers from inaccurate positioning in dynamic environments and poor mapping practicality.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0005] The embodiments of the present application provide a method and device for simultaneous positioning and map construction, which generates a mask through semantic segmentation and optical flow calculation, and constructs a semantic map after removing dynamic feature points, thereby achieving high-precision positioning and incremental semantic map construction in a dynamic environment, improving positioning accuracy and map construction practicality in a dynamic environment, and solving the problem that traditional visual SLAM technology suffers from a serious decline in positioning accuracy in a dynamic environment and cannot meet the needs of subsequent upper-level applications.

[0006] In a first aspect, an embodiment of the present application provides a method for simultaneous positioning and map construction, comprising:

[0007] Acquire multiple frames of continuous image data of the current environment in real time and extract the visual feature points corresponding to each frame of image data;

[0008] Performing semantic segmentation processing and optical flow calculation processing on the image data respectively to generate corresponding semantic masks and optical flow masks;

[0009] Eliminating dynamic feature points from the visual feature points based on the semantic mask and the optical flow mask, and retaining static feature points;

[0010] A semantic point cloud is generated based on the static points, and a semantic octree map is incrementally constructed based on the semantic point cloud and an octree module.

[0011] Optionally, in some embodiments of the present application, performing semantic segmentation processing and optical flow calculation processing on the image data respectively to generate corresponding semantic masks and optical flow masks includes:

[0012] Performing frame-by-frame semantic segmentation processing on the image data based on a semantic segmentation network to generate a semantic mask of the dynamic object;

[0013] Based on the optical flow network, motion analysis is performed on adjacent frame images to generate optical flow masks of dynamic areas.

[0014] Optionally, in some embodiments of the present application, performing frame-by-frame semantic segmentation processing on the image data based on a semantic segmentation network to generate a semantic mask of a dynamic object includes:

[0015] Input the current frame image into the pre-trained semantic segmentation model and output the pixel-level classification results and the corresponding object category labels;

[0016] According to the classification result, a corresponding binary mask is generated for the dynamic object category, wherein the dynamic object area is marked as a first preset value and the static background is marked as a second preset value;

[0017] Morphological processing is performed on the binary mask to expand the mask range of the dynamic object area and generate a semantic mask corresponding to the dynamic object.

[0018] Optionally, in some embodiments of the present application, performing motion analysis on adjacent frame images based on an optical flow network to generate an optical flow mask of a dynamic area includes:

[0019] Based on the optical flow network, optical flow calculation is performed on two adjacent frames of images to obtain the pixel-level motion vector field;

[0020] generating a normalized grayscale image according to the amplitude value of the motion vector field;

[0021] Performing adaptive threshold segmentation on the grayscale image to distinguish the static background from the dynamic area and generate an initial optical flow mask;

[0022] The initial optical flow mask is morphologically optimized to obtain the optical flow mask corresponding to the dynamic area.

[0023] Optionally, in some embodiments of the present application, the removing dynamic feature points from the visual feature points based on the semantic mask and the optical flow mask and retaining static feature points includes:

[0024] fusing the semantic mask and the optical flow mask to generate a comprehensive mask of the dynamic feature area;

[0025] Dynamic feature points in the visual feature points are eliminated according to the comprehensive mask, and static feature points are retained.

[0026] Optionally, in some embodiments of the present application, fusing the semantic mask and the optical flow mask to generate a comprehensive mask of the dynamic feature area includes:

[0027] performing a logical OR operation on the semantic mask and the optical flow mask to generate a joint dynamic region;

[0028] A connected domain analysis is performed on the joint dynamic region, and noise regions with an area smaller than a preset threshold are eliminated to form a comprehensive mask corresponding to the dynamic feature region.

[0029] Optionally, in some embodiments of the present application, generating a semantic point cloud based on the static points, and incrementally constructing a semantic octree map based on the semantic point cloud and an octree module includes:

[0030] Generate a three-dimensional semantic point cloud based on the static points of the key frame, and mark the three-dimensional semantic point cloud with corresponding semantic tags;

[0031] Performing clustering optimization processing on the three-dimensional semantic point cloud to obtain an optimized static semantic point cloud;

[0032] The optimized static semantic point cloud is input into the octree module to incrementally update the semantic octree map.

[0033] Optionally, in some embodiments of the present application, performing clustering optimization processing on the three-dimensional semantic point cloud to obtain an optimized static semantic point cloud includes:

[0034] Perform Euclidean clustering on the current point cloud cluster to obtain independent point cloud clusters;

[0035] Projecting each of the independent point cloud clusters onto the mask area of ​​the historical frame and calculating the dynamic area ratio;

[0036] If the proportion of the dynamic area exceeds a preset threshold, the independent point cloud cluster is determined to be a missed dynamic object and is removed.

[0037] Optionally, in some embodiments of the present application, after generating a semantic point cloud based on the static points and incrementally constructing a semantic octree map based on the semantic point cloud and an octree module, the method further includes:

[0038] According to the semantic labels and spatial distribution of static objects in the semantic octree map, the initial navigation path is generated based on the shortest path algorithm;

[0039] Detect new dynamic obstacle areas through the current frame image and optical flow mask acquired in real time and mark them as temporary restricted areas;

[0040] Mapping the temporary restricted area to the semantic octree map, dynamically adjusting the initial navigation path to bypass the restricted area, and obtaining an adjusted navigation path;

[0041] The robot motion instructions are updated according to the adjusted navigation path, and the dynamic obstacle information in the semantic octree map is updated synchronously.

[0042] In a second aspect, an embodiment of the present application provides a simultaneous positioning and map building device, comprising:

[0043] The data acquisition module is used to acquire multiple frames of continuous image data of the current environment in real time and extract the visual feature points corresponding to each frame of image data;

[0044] A mask generation module is used to perform semantic segmentation processing and optical flow calculation processing on the image data respectively to generate corresponding semantic masks and optical flow masks;

[0045] A feature processing module, configured to remove dynamic feature points from the visual feature points based on the semantic mask and the optical flow mask, and retain static feature points;

[0046] A map construction module is used to generate a semantic point cloud based on the static points, and to incrementally construct a semantic octree map based on the semantic point cloud and the octree module.

[0047] In a third aspect, an embodiment of the present application provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the synchronous positioning and map construction method as described in the first aspect are implemented.

[0048] In a fourth aspect, an embodiment of the present application provides a storage medium storing a computer program that can be loaded by a processor and execute the simultaneous positioning and map construction method as described in the first aspect.

[0049] The present application provides a method and device for synchronous positioning and map construction. First, multiple frames of continuous image data of the current environment are acquired in real time, and the visual feature points corresponding to each frame of image data are extracted; then, semantic segmentation processing and optical flow calculation processing are performed on the image data respectively to generate corresponding semantic masks and optical flow masks; then, dynamic feature points in the visual feature points are eliminated based on the semantic mask and optical flow mask, and static feature points are retained; finally, a semantic point cloud is generated based on the static points, and a semantic octree map is incrementally constructed based on the semantic point cloud and the octree module. In the synchronous positioning and map construction scheme provided by the present application, semantic masks and optical flow masks are generated by semantic segmentation and optical flow calculation, dynamic feature points are accurately identified and eliminated, and the problem of decreased positioning accuracy of traditional SLAM in dynamic environments is effectively solved. After retaining static points, semantic point clouds are generated, and semantic octree maps are incrementally constructed based on the semantic point cloud and the octree module, thereby achieving richer, denser and more semantic map construction, meeting the subsequent upper-layer application requirements of the robot, while reducing computing costs and improving the real-time performance of map construction. It can be seen that this application can achieve high-precision positioning and incremental semantic map construction in dynamic environments, improve positioning accuracy and mapping practicality in dynamic environments, and provide a feasible solution for high-precision positioning and semantic map construction in dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 This is an application environment diagram of the synchronous positioning and map construction method provided in an embodiment of the present application;

[0052] Figure 2 Schematic diagram of the process of the simultaneous positioning and map construction method provided in the embodiment of the present application;

[0053] Figure 3 This is a flowchart of the semantic mask generation process provided by the embodiment of the present application;

[0054] Figure 4 This is a schematic diagram of the process of generating an optical flow mask provided in an embodiment of the present application;

[0055] Figure 5 This is a flow chart of the dynamic feature point selection process provided by the embodiment of the present application;

[0056] Figure 6 This is a schematic diagram of the process of generating a semantic point cloud according to an embodiment of the present application;

[0057] Figure 7 This is another flowchart of the simultaneous positioning and map building method provided by an embodiment of the present application;

[0058] Figure 8 is a structural diagram of a synchronous positioning and map building device provided in an embodiment of the present application;

[0059] Figure 9 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of systems and methods consistent with aspects of the present application, as detailed in the appended claims.

[0061] It should be noted that, in this document, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive descriptions such as inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0062] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0063] In the subsequent description, the use of suffixes such as "module", "component" or "unit" to represent elements is only for the purpose of facilitating the description of the present application and has no specific meaning. Therefore, "module", "component" or "unit" can be used interchangeably.

[0064] With the rapid development of artificial intelligence and computer vision technologies, mobile robots are increasingly being used in logistics, service, cleaning, and other fields, significantly improving production efficiency and intelligence. Simultaneous Localization and Mapping (SLAM) technology, as the core support for autonomous robot navigation, uses sensors (such as cameras and lidar) to acquire real-time environmental data, enabling robot pose estimation and map construction.

[0065] Most mainstream visual SLAM systems are based on the assumption of a static environment, so they often achieve relatively stable performance in static scenes. However, in real-world environments, dynamic objects can interfere with the accuracy and stability of the SLAM system. Semantic SLAM has been a major innovation in SLAM research in recent years. By combining computer vision, deep learning, and traditional SLAM technology, the system can understand and utilize semantic information in the environment. Using deep neural networks, SLAM systems can detect and classify targets in images captured by visual sensors, extracting high-level semantic information from the scene and integrating it into the SLAM system's map-building and positioning processes. However, the methods used by common semantic SLAM systems are computationally expensive and slow to run. Furthermore, due to the presence of non-a priori dynamic objects, semantic segmentation alone cannot completely eliminate dynamic objects in the environment.

[0066] In order to solve the above-mentioned technical problems and overcome the defects of the existing technology, the embodiment of the present application provides a method and device for synchronous positioning and map construction, which generates a mask through semantic segmentation and optical flow calculation, constructs a semantic map after removing dynamic feature points, and realizes high-precision positioning and incremental semantic map construction in a dynamic environment, solving the problem that the positioning accuracy of traditional visual SLAM technology is seriously reduced in a dynamic environment and cannot meet the needs of subsequent upper-level applications.

[0067] Figure 1 FIG. 1 is an application environment diagram of a method for synchronous positioning and map construction in an embodiment. Figure 1, the synchronous positioning and map construction method is applied to the synchronous positioning and map construction system. The synchronous positioning and map construction system includes a terminal 110 and a server 120. The terminal 110 and the server 120 are connected through a network. The terminal 110 can be a desktop terminal or a mobile terminal. The mobile terminal can be at least one of a mobile phone, a tablet computer, a laptop computer, etc. The server 120 can be implemented as an independent server or a server cluster composed of multiple servers. The server 120 can be used to obtain multiple frames of continuous image data of the current environment in real time, and extract the visual feature points corresponding to each frame of image data; perform semantic segmentation processing and optical flow calculation processing on the image data respectively to generate corresponding semantic masks and optical flow masks; remove dynamic feature points from the visual feature points based on the semantic mask and the optical flow mask, and retain static feature points; generate a semantic point cloud based on the static points, and incrementally construct a semantic octree map based on the semantic point cloud and the octree module.

[0068] See also Figure 2 , Figure 2 : This is a flow chart of a method for simultaneous positioning and map building provided in one embodiment of the present application. This embodiment mainly uses the application of the method for simultaneous positioning and map building to a computer device as an example for illustration. The method for simultaneous positioning and map building provided in one embodiment of the present application may specifically include the following steps:

[0069] S1. Acquire multiple frames of continuous image data of the current environment in real time and extract the visual feature points corresponding to each frame of image data;

[0070] Specifically, in step S1, a sensor such as a camera captures an image of the environment in real time. Feature point detection algorithms such as ORB (Oriented FAST and Rotated BRIEF) are used to extract distinctive visual feature points from each frame. These feature points are used for subsequent positioning and map construction. A variety of feature point detection algorithms, such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded-Up Robust Features), can also be used, depending on the specific application scenario and computing resources to improve the efficiency and robustness of feature point detection.

[0071] S2. Perform semantic segmentation and optical flow calculation on the image data to generate corresponding semantic masks and optical flow masks;

[0072] Specifically, for step S2, on the one hand, a pre-trained semantic segmentation network (such as RTMDet-Ins-tiny) is used to process the image frame by frame, classify and annotate different objects in the image at the pixel level, generate a semantic mask, and distinguish dynamic objects from static backgrounds. On the other hand, the optical flow field between adjacent frames is calculated through an optical flow network (such as LiteFlowNet2) to obtain an optical flow mask and mark the moving area. By continuously optimizing the structure and parameters of the semantic segmentation network and the optical flow network, the recognition accuracy of dynamic objects and the accuracy of optical flow calculation can be improved, while reducing the computing cost and time delay. For example, the network can be pruned, quantized, and other operations to adapt to resource-constrained embedded devices.

[0073] S3. Remove dynamic feature points from visual feature points based on semantic mask and optical flow mask, and retain static feature points;

[0074] Specifically, for step S3, the semantic mask and the optical flow mask are fused to generate a comprehensive mask for determining the position of dynamic feature points in the image. Then, the extracted visual feature points are screened according to the comprehensive mask, and the feature points located in the dynamic area are removed, retaining only the static feature points to improve the accuracy of positioning. A more effective mask fusion algorithm and feature point screening strategy can also be further adopted to comprehensively consider semantic information, optical flow information and other properties of feature points (such as stability, distribution, etc.), more accurately remove dynamic feature points, and avoid mistakenly removing static feature points, thereby improving the reliability and stability of positioning.

[0075] S4. Generate a semantic point cloud based on static points, and incrementally construct a semantic octree map based on the semantic point cloud and octree modules;

[0076] Specifically, in step S4, the retained static points are combined with depth information (which can be directly acquired by a depth camera or calculated through methods such as stereo vision and structured light) to generate a 3D semantic point cloud. The semantic point cloud is then fused with existing map data, and the octree module incrementally constructs and updates the semantic octree map, achieving a semantic and dense representation of the environment. During the semantic point cloud generation and map construction process, additional optimization algorithms, such as filtering, smoothing, and data association, can be introduced to improve point cloud quality and map accuracy.

[0077] This embodiment acquires multi-frame continuous image data in real time and extracts visual feature points, uses semantic segmentation and optical flow calculation to generate semantic masks and optical flow masks respectively, and then accurately eliminates dynamic feature points. After retaining static points, semantic point clouds are generated using pixels of the static background, and an incremental semantic map is constructed based on this. This effectively solves the problem of decreased positioning accuracy of traditional SLAM in dynamic environments, realizes high-precision, dense and semantic dynamic SLAM, improves the navigation performance and autonomy of the robot, enables it to operate stably and efficiently in complex dynamic scenes, and meets the high requirements of the robot for positioning and map construction in practical applications.

[0078] Optionally, in some embodiments, step S2 of “performing semantic segmentation processing and optical flow calculation processing on the image data respectively to generate corresponding semantic masks and optical flow masks” may specifically include:

[0079] S21. Perform frame-by-frame semantic segmentation on the image data based on the semantic segmentation network to generate semantic masks of dynamic objects;

[0080] Specifically, each frame of the image is input into a pre-trained semantic segmentation network, which can classify each pixel in the image and determine the object category to which it belongs. For dynamic objects, such as pedestrians and vehicles, the semantic segmentation network will mark them out and generate corresponding semantic masks. In this mask, the dynamic object area and the static background area are represented by different numerical values, thereby realizing the recognition and marking of dynamic objects. By continuously optimizing the structure and parameters of the semantic segmentation network, the accuracy and speed of dynamic object recognition can be improved. For example, more advanced deep learning architectures such as the DeepLab series can be adopted, or existing networks can be improved, such as adding attention mechanisms and improving feature fusion methods, to better handle dynamic object recognition in complex scenes. At the same time, the network can also be compressed and accelerated, such as through pruning, quantization and other technologies, so that it can run efficiently on resource-constrained embedded devices.

[0081] S22. Perform motion analysis on adjacent frame images based on the optical flow network to generate an optical flow mask for the dynamic area;

[0082] Specifically, the optical flow network calculates the motion vectors of pixels in two adjacent image frames to generate an optical flow field. The optical flow field reflects the temporal motion of pixels in the image, specifically the direction and speed of each pixel's movement. By analyzing the optical flow field, dynamic regions in the image—those pixels with noticeable motion—can be identified. This allows the generation of an optical flow mask to identify these dynamic regions.

[0083] This embodiment processes image data based on a semantic segmentation network and an optical flow network, respectively, to generate semantic masks of dynamic objects and optical flow masks of dynamic areas, thereby achieving accurate recognition of dynamic objects from two dimensions: semantic category and motion state. This provides a reliable basis for the subsequent elimination of dynamic feature points, effectively improving the positioning and map construction performance in dynamic environments, and enhancing the robustness and accuracy of the system.

[0084] Optionally, in some embodiments, step S21 of “performing frame-by-frame semantic segmentation processing on the image data based on the semantic segmentation network to generate a semantic mask of the dynamic object” may specifically include:

[0085] S211. Input the current frame image into the pre-trained semantic segmentation model and output the pixel-level classification results and the corresponding object category labels;

[0086] Specifically, the current frame image is a real-time image of the environment captured by a sensor such as a camera. This image is then fed into a pre-trained semantic segmentation model. This model, trained on a large amount of labeled data, can classify each pixel in the image, determine its object category (e.g., pedestrian, vehicle, chair, etc.), and assign a category label to each pixel. Ultimately, the classification result for each pixel and the corresponding object category label are obtained, thus achieving pixel-level classification of objects in the image. When training the semantic segmentation model, data augmentation techniques such as rotation, flipping, and scaling can be used to increase the diversity of the training data and improve the model's ability to recognize dynamic objects in different scenes and poses. Furthermore, the training dataset can be continuously updated and expanded to include more types and forms of dynamic objects, making the model more versatile and adaptable.

[0087] S212. Based on the classification result, a binary mask corresponding to the dynamic object category is generated, wherein the dynamic object area is marked as a first preset value and the static background is marked as a second preset value;

[0088] Specifically, based on the pixel-level classification results output by the semantic segmentation model, determine which pixels belong to the dynamic object category. Then, a binary mask is generated for these dynamic object pixels, usually marking the dynamic object area with a specific value (such as 255) and the static background area with another value (such as 0). This can separate the dynamic object from the background, facilitating subsequent processing. The dynamic object category can also be further refined, generating different masks for different types of dynamic objects, so that different strategies can be adopted for different types of dynamic objects in subsequent processing. At the same time, when generating the binary mask, methods such as multi-threshold segmentation can be used to improve the accuracy of the mask.

[0089] S213. Perform morphological processing on the binary mask, expand the mask range of the dynamic object area, and generate a semantic mask corresponding to the dynamic object;

[0090] Specifically, morphological processing is an image processing technique that uses operations such as dilation and erosion on a binary mask to change its shape and range. In this embodiment, dilation is performed on the binary mask to appropriately expand the mask range of the dynamic object area to ensure complete coverage of the dynamic object. The semantic mask corresponding to the dynamic object generated in this way not only includes the core area of ​​the dynamic object but also some edge areas, thereby improving the recognition range and accuracy of the dynamic object. During the morphological processing process, appropriate structuring elements and processing parameters can be selected based on the characteristics of the dynamic object and the actual application scenario to achieve the optimal mask expansion effect. For example, for smaller dynamic objects, a smaller structuring element can be used for dilation; for larger dynamic objects, the size of the structuring element can be appropriately increased. Furthermore, multiple morphological operations can be combined, such as performing a dilation operation followed by an erosion operation to remove some noise points and smooth the mask boundaries.

[0091] This embodiment inputs the current frame image into a pre-trained semantic segmentation model, outputs pixel-level classification results and object category labels, and generates a binary mask of the dynamic object based on this. The mask range is then optimized through morphological processing to generate a semantic mask corresponding to the dynamic object, thereby achieving accurate recognition and effective distinction of dynamic objects, providing a reliable basis for subsequent dynamic feature point removal, and significantly improving the accuracy and robustness of semantic segmentation in dynamic environments.

[0092] In a specific embodiment, Figure 3 As shown, Figure 3 The process of semantic mask generation is provided, and the specific process is as follows:

[0093] Step 1: The "Semantic Module" receives the input RGB image and verifies its format. If the format is incorrect, proceed to step 2; if the format is correct, proceed to step 3.

[0094] Step 2: If the format is wrong, a full background empty mask (pure black mask) is generated, indicating that no operation is performed, and go to step 10;

[0095] Step 3: The "semantic module" sends the RGB image to the Python semantic segmentation server through the ROS Action communication protocol;

[0096] Step 4: The server calls the pre-trained semantic segmentation model to perform pixel-level classification on the image and output the probability and category label;

[0097] Step 5: Transfer the segmentation results from CUDA memory to CPU memory and convert them into a universal tensor format. Create an initial full background mask (a pure black image with a grayscale value of 0);

[0098] Step 6: Determine whether the current object index exceeds the number of objects. If yes, go to step 6, otherwise go to step 7.

[0099] Step 7: Determine whether the correct probability of the current object category is greater than 50%. If so, go to step 8. Otherwise, add 1 to the index and go to step 6.

[0100] Step 8: Add the current object's mask to the overall mask image, displaying the grayscale value with the corresponding category ID. Meanwhile, increase the object index by 1 and go back to step 6.

[0101] Step 9: Use morphological algorithm to dilate the mask and expand the mask range;

[0102] Step 10: Return the overall mask.

[0103] Optionally, in some embodiments, step S22 of “performing motion analysis on adjacent frame images based on an optical flow network to generate an optical flow mask of a dynamic area” may specifically include:

[0104] S221. Based on the optical flow network, optical flow calculation is performed on two adjacent frames of images to obtain a pixel-level motion vector field;

[0105] Specifically, two adjacent image frames are fed into an optical flow network, which calculates the motion vector of each pixel between the adjacent frames—that is, the direction and magnitude of the pixel's displacement in the time dimension. These motion vectors together form a pixel-level motion vector field, which reflects the movement of pixels in the image.

[0106] S222. Generate a normalized grayscale image based on the amplitude value of the motion vector field;

[0107] Specifically, the motion vector amplitude value of each pixel in the motion vector field is normalized and mapped to a grayscale range of 0-255 to generate a normalized grayscale image. This converts motion information into a grayscale image, facilitating subsequent image processing operations. Different normalization strategies, such as linear normalization and nonlinear normalization, can be employed to suit different application scenarios and requirements.

[0108] S223. Perform adaptive threshold segmentation on the grayscale image to distinguish the static background from the dynamic area and generate an initial optical flow mask;

[0109] Specifically, an adaptive threshold segmentation algorithm is used to process the normalized grayscale image, automatically determining the threshold and segmenting the grayscale image into a static background and a dynamic region. This method generates an initial optical flow mask, in which the dynamic region and the static background are marked with different numerical values. In addition to adaptive threshold segmentation, other image segmentation techniques, such as clustering-based segmentation and edge detection-based segmentation, can also be used to improve segmentation accuracy and noise resistance.

[0110] S224. Perform morphological optimization on the initial optical flow mask to obtain an optical flow mask corresponding to the dynamic area;

[0111] Specifically, morphological operations such as dilation and erosion are performed on the initial optical flow mask to remove noise, fill holes, and smooth boundaries, thereby obtaining a more accurate and complete optical flow mask corresponding to the dynamic region. During the morphological optimization process, the appropriate morphological operation sequence and parameters can be selected based on the characteristics of the optical flow mask in the actual application to achieve the best optimization effect. At the same time, other image post-processing techniques such as median filtering and connected domain analysis can be combined to further improve the quality of the optical flow mask.

[0112] This embodiment performs optical flow calculation on adjacent frame images based on the optical flow network to obtain a pixel-level motion vector field, and generates an initial optical flow mask through steps such as normalized grayscale image generation and adaptive threshold segmentation. Finally, the optical flow mask corresponding to the dynamic area is obtained through morphological optimization, which achieves accurate capture and effective distinction of dynamic areas, improves the quality and reliability of the optical flow mask, and provides strong support for subsequent dynamic feature point removal.

[0113] In a specific embodiment, Figure 4 As shown, Figure 4 The process of optical flow mask generation is provided, and the specific process is as follows:

[0114] Step 1: The "Optical Flow Module" receives the input RGB image and verifies its format. If the format is incorrect, proceed to step 2; if the format is correct, proceed to step 3.

[0115] Step 2: If the format is wrong, generate an empty mask and go to step 9;

[0116] Step 3: The "optical flow module" sends the RGB image to the Python optical flow calculation server through the ROS Action communication protocol;

[0117] Step 4: The server calls the optical flow estimation algorithm to calculate the pixel motion vectors between adjacent frames and generate optical flow field data;

[0118] Step 5: Optical flow result processing: transfer the optical flow calculation results from CUDA video memory to CPU memory and convert them into Numpy array format.

[0119] Step 6: Extract the optical flow amplitude (Euclidean distance of motion vector), normalize it to the range of 0-255, and generate the initial grayscale image;

[0120] Step 7: Apply OTSU adaptive threshold segmentation to the grayscale image to distinguish the static background (grayscale value 0) from the dynamic area (grayscale value 255) to obtain a binary mask;

[0121] Step 8: Use morphological algorithm to dilate the mask and expand the mask range;

[0122] Step 9: Return to the mask.

[0123] Optionally, in some embodiments, step S3 of “eliminating dynamic feature points from visual feature points based on semantic masks and optical flow masks, and retaining static feature points” may specifically include:

[0124] S31. Fusing the semantic mask and the optical flow mask to generate a comprehensive mask of the dynamic feature area;

[0125] Specifically, the semantic mask can identify the area where dynamic objects are located in the image, while the optical flow mask can display the dynamic areas caused by motion. The two are fused, that is, the two mask images are subjected to logical operations such as bitwise OR and bitwise AND operations. The bitwise OR operation can merge all areas marked as dynamic in the semantic mask and the optical flow mask to form a more comprehensive dynamic feature area comprehensive mask, which can ensure that all possible dynamic areas are covered. During the fusion process, more complex fusion algorithms can be used, such as weighted fusion, to assign different weights to the semantic mask and the optical flow mask according to their reliability and importance in different scenarios, so as to more accurately determine the dynamic feature areas.

[0126] S32. Remove dynamic feature points from the visual feature points based on the comprehensive mask and retain static feature points;

[0127] Specifically, the generated comprehensive mask is used to screen the extracted visual feature points. Each feature point is checked one by one to see if it is within the dynamic feature area marked by the comprehensive mask. If so, the feature point is removed; if not, the feature point is retained as a static feature point for subsequent positioning and map construction. In order to improve the efficiency and accuracy of removing dynamic feature points, a fast matching algorithm between feature points and mask areas can be established, such as data structures such as spatial indexes or hash tables, to quickly locate the mask area where the feature points are located. At the same time, other properties of the feature points, such as the stability of the feature points, the size of the eigenvalues, etc., can be combined to further verify whether the feature points are dynamic feature points and reduce the occurrence of false removals.

[0128] This embodiment fuses the semantic mask and the optical flow mask to generate a comprehensive mask of the dynamic feature area, and based on this, removes dynamic feature points from the visual feature points, retains static feature points, effectively integrates semantic and motion information, comprehensively identifies dynamic feature areas, improves the quality of static feature points, enhances the robustness and adaptability of the system, and provides a reliable data foundation for subsequent high-precision positioning and semantic map construction.

[0129] Optionally, in some embodiments, step S31 of “fusing the semantic mask and the optical flow mask to generate a comprehensive mask of the dynamic feature region” may specifically include:

[0130] S311. Perform a logical OR operation on the semantic mask and the optical flow mask to generate a joint dynamic region;

[0131] Specifically, a logical OR operation is performed on the semantic mask and the optical flow mask. That is, any location marked as a dynamic region in either the semantic mask or the optical flow mask is also marked as dynamic in the joint dynamic region. This allows the dynamic object regions identified by semantic segmentation and the motion regions obtained by the optical flow calculation to be merged, initially forming a more comprehensive dynamic region labeling. In addition to the logical OR operation, other fusion strategies can also be used, such as logical AND operations and weighted summation. For example, the logical AND operation can more strictly filter out regions identified as dynamic by both semantic segmentation and optical flow calculations; the weighted summation can assign different weights based on the reliability of different masks before fusion.

[0132] S312. Performing connected domain analysis on the joint dynamic region, removing noise regions whose area is smaller than a preset threshold, and forming a comprehensive mask corresponding to the dynamic feature region;

[0133] Specifically, a connected domain analysis is performed on the joint dynamic area to find the interconnected pixel areas. Then, the connected domain is screened according to its area size, and small areas with an area smaller than a preset threshold are eliminated. These small areas are likely to be caused by noise or false detection. The larger connected domains that are finally retained form a comprehensive mask corresponding to the dynamic feature area. When performing connected domain analysis, different connectivity definitions (such as 4-connected, 8-connected) can be used to adapt to different scene requirements. In addition, the connected domain can be further screened in combination with other features (such as shape, position, etc.), not just based on area size. For example, connected domains with irregular shapes or positions that do not conform to the characteristics of dynamic objects are eliminated.

[0134] This embodiment generates a joint dynamic region by performing a logical OR operation on the semantic mask and the optical flow mask, and forms a comprehensive mask after removing the noise region through connected domain analysis. It effectively integrates semantic information and optical flow information, accurately identifies dynamic feature regions, and removes noise interference. It improves the accuracy and robustness of the comprehensive mask and provides a high-quality basis for the subsequent removal of dynamic feature points.

[0135] In a specific embodiment, Figure 5 As shown, Figure 5 The process of dynamic feature point removal is provided, the specific process is as follows:

[0136] Step 1: The "dynamic point removal module" synchronously receives the semantic mask (marking the dynamic object area) output by the semantic segmentation module and the optical flow mask (marking the motion area) generated by the optical flow module;

[0137] Step 2: Check whether the semantic mask is empty. If it is, go to step 3; otherwise, go to step 4.

[0138] Step 3: Generate a full background mask with the same size as the input image (grayscale value 0, indicating no dynamic objects);

[0139] Step 4: Check if the optical flow mask is empty, if so go to step 5, otherwise go to step 6;

[0140] Step 5: Generate a full background mask (grayscale value 0, indicating no motion area);

[0141] Step 6: Merge the semantic mask and optical flow mask, with pure black as the background and other masks merged.

[0142] Step 7: Check whether the current feature point number exceeds the number of feature points. If yes, then end directly, otherwise go to step 8;

[0143] Step 8: Check whether the current feature point is within the mask, that is, the landing point is not black. If so, return directly. Otherwise, continue to check the next feature point and go to step 7.

[0144] Optionally, in some embodiments, step S4 of “generating a semantic point cloud based on static points, and incrementally constructing a semantic octree map based on the semantic point cloud and the octree module” may specifically include:

[0145] S41. Generate a three-dimensional semantic point cloud based on the static points of the key frame, and mark the corresponding semantic tags on the three-dimensional semantic point cloud;

[0146] Specifically, the static feature points in the keyframes are combined with the camera's internal and external parameters and depth information to convert these 2D feature points into 3D point cloud data through projection and other methods. Simultaneously, based on the results of semantic segmentation, each 3D point cloud data point is assigned a corresponding semantic label, such as a different object category, such as ground, wall, or table. When generating 3D point cloud data, multi-view geometry methods can be used to fuse information from multiple keyframes to improve the accuracy and density of the point cloud.

[0147] S42. Perform clustering optimization on the three-dimensional semantic point cloud to obtain an optimized static semantic point cloud;

[0148] Specifically, cluster analysis is performed on the generated 3D semantic point cloud, grouping similar point cloud points into the same cluster. Cluster optimization can remove noise points and merge similar point cloud clusters, resulting in a more concise and accurate semantic point cloud. Clustering algorithms such as Euclidean clustering and DBSCAN can be used. To improve clustering effectiveness, comprehensive clustering can be performed by combining geometric features of the point cloud (such as normal direction and curvature) with semantic labels.

[0149] S43. Input the optimized static semantic point cloud into the octree module and incrementally update the semantic octree map;

[0150] Specifically, the optimized 3D semantic point cloud data is input into the octree module, which then fuses the new point cloud data with existing map data. For example, the module spatially registers the currently generated 3D point cloud data with existing historical map data, finding the coordinate transformation relationship between them so that the newly generated point cloud data can be accurately integrated into the existing map. Finally, the semantic octree map is incrementally constructed and updated. This approach reflects environmental changes in real time, improving the map's timeliness and accuracy. When inputting point cloud data into the octree module, timestamp information can be combined to estimate and predict the motion state of dynamic obstacles, providing more accurate information for subsequent path planning.

[0151] This embodiment generates a three-dimensional semantic point cloud based on the static points of the key frames and marks the semantic tags. After clustering and optimizing the three-dimensional semantic point cloud, it inputs it into the octree module to incrementally update the semantic octree map, thereby achieving an effective fusion of semantic information and environmental geometric structure, constructing an efficient, accurate and semantically rich incremental map, and significantly improving the quality and practicality of the map.

[0152] Optionally, in some embodiments, step S43 of “performing clustering optimization processing on the three-dimensional semantic point cloud to obtain an optimized static semantic point cloud” may specifically include:

[0153] S431. Perform Euclidean clustering on the current point cloud cluster to obtain independent point cloud clusters;

[0154] Specifically, the currently generated point cloud cluster is clustered and segmented according to the Euclidean distance, and the point cloud cluster is divided into multiple independent clusters. Each independent cluster is relatively independent in space and represents a possible object or part of an object.

[0155] S432. Project each independent point cloud cluster onto the mask area of ​​the historical frame and calculate the dynamic area ratio;

[0156] Specifically, each independent point cloud cluster is projected onto the mask area of ​​the previous frame. By comparing the overlap between the projected area and the dynamic area mask in the historical frame, the percentage of points in the point cloud cluster that are located in the dynamic area is calculated. This step can determine whether the current point cloud cluster is related to the dynamic objects in the historical frame. The camera's motion model and the spatial distribution of the point cloud can be considered during the projection process to improve the accuracy of the projection. At the same time, the calculation of the dynamic area ratio can also be combined with other factors, such as the speed and direction of movement of the point cloud, to make a comprehensive judgment.

[0157] S433. If the dynamic area ratio exceeds the preset threshold, the independent point cloud cluster is determined to be a missed dynamic object and is removed;

[0158] Specifically, if the dynamic area ratio of an independent point cloud cluster exceeds a preset threshold, the cluster is considered a missed dynamic object and removed from the generated point cloud sequence to ensure that the remaining point cloud clusters primarily represent static objects. The preset threshold can be adjusted to achieve optimal removal results based on different application scenarios and environments. Furthermore, machine learning methods can be combined to automatically determine a reasonable threshold by learning from large amounts of sample data, improving the accuracy and adaptability of the removal.

[0159] This embodiment performs Euclidean clustering on the current point cloud cluster, projects it to the historical frame mask area, calculates the proportion of dynamic areas, and eliminates missed dynamic objects, effectively solving the problem of missed detection of dynamic objects, improving the accuracy of semantic point clouds, optimizing the quality of semantic maps, and enhancing the adaptability and robustness of the system in dynamic environments.

[0160] In a specific embodiment, Figure 6 As shown, Figure 6 The process of generating semantic point cloud is provided. The specific process is as follows:

[0161] Step 1: The "mapping module" receives the key frame data transmitted by the main thread of the SLAM system;

[0162] Step 2: Traverse all pixels of the RGB image and check whether the current pixel coordinates exceed the image boundary. If yes, go to step 5, otherwise go to step 3.

[0163] Step 3: Query the grayscale value of the pixel position corresponding to the semantic mask to determine whether it belongs to the dynamic object area. If so, determine the next pixel and go to step 2. Otherwise, go to step 4.

[0164] Step 4: The current pixel represents a static object. The pixel is projected into the three-dimensional space through the depth map and camera parameters to obtain a point cloud. The point cloud is given semantic information based on the semantic segmentation result, and the next pixel is processed, and then the process goes to step 2.

[0165] Step 5: Apply the Euclidean clustering algorithm to the generated original point cloud and divide it into independent point cloud clusters based on the spatial distance threshold.

[0166] Step 6: Check whether the current point cloud cluster index exceeds the number of point cloud clusters. If yes, go to step 10; otherwise, go to step 7.

[0167] Step 7: Project the current point cloud cluster to the previous frame and count the proportion of dynamic pixels in the projection area;

[0168] Step 8: Whether more than 50% of the projection area of ​​the point cloud cluster falls within the dynamic object area in the mask of the previous frame, it means that the current point cloud cluster is a missed dynamic object and go to step 9. Otherwise, determine the next point cloud cluster and go to step 6.

[0169] Step 9: The current point cloud cluster is a dynamic object, and it is deleted from the generated point cloud sequence;

[0170] Step 10: Return the optimized pure static object point cloud.

[0171] Optionally, in some embodiments, after step S4 of "generating a semantic point cloud based on static points, and incrementally constructing a semantic octree map based on the semantic point cloud and the octree module", a step of generating a robot navigation path based on the semantic octree map and updating dynamic obstacle information in real time is also included. This step may specifically include:

[0172] S51. Generate an initial navigation path based on the shortest path algorithm according to the semantic labels and spatial distribution of static objects in the semantic octree map;

[0173] Specifically, the semantic labels of static objects and their spatial distribution information in the semantic octree map are used to determine the traversable areas and obstacle locations in the environment. Then, based on a shortest path algorithm (such as A* or Dijkstra), an initial navigation path is generated from the starting point to the target point, avoiding known static obstacles.

[0174] S52. Detect new dynamic obstacle areas using the current frame image and optical flow mask acquired in real time and mark them as temporary restricted areas.

[0175] Specifically, the current frame image is processed in real time and combined with an optical flow mask to identify newly added dynamic obstacle areas in the image. The optical flow mask can help determine which areas show motion, thus identifying these areas as potentially new dynamic obstacles. Once a new dynamic obstacle is detected, it is marked as a temporary restricted area to alert the robot to avoid it. When mapping the temporary restricted area to the map, the uncertainty of the obstacle and the predicted motion can be taken into account, and a certain extended boundary can be added to the obstacle to improve obstacle avoidance safety.

[0176] S53 maps the temporary restricted area to the semantic octree map, dynamically adjusts the initial navigation path to bypass the restricted area, and obtains the adjusted navigation path;

[0177] Specifically, the detected temporary restricted areas are converted to the coordinate system of the semantic octree map, and the obstacle information in the map is updated. Then, the navigation path is replanned based on the updated map, and the initial path is adjusted to bypass the temporary restricted areas. This generates a new adjusted navigation path, ensuring that the robot can safely avoid dynamic obstacles.

[0178] S54 updates the robot motion instructions according to the adjusted navigation path and simultaneously updates the dynamic obstacle information in the semantic octree map;

[0179] Specifically, based on the replanned navigation path, corresponding robot motion instructions are generated to control the robot to move along the new path. Simultaneously, information about detected dynamic obstacles is updated in the semantic octree map so that subsequent path planning can account for these dynamic changes. To improve the robot's responsiveness and flexibility, more real-time motion control algorithms, such as model predictive control (MPC), can be employed. Furthermore, when updating the semantic octree map, timestamp information can be combined to estimate and predict the motion state of dynamic obstacles, providing more accurate information for subsequent path planning.

[0180] This embodiment generates an initial navigation path based on a semantic octree map, and detects newly added dynamic obstacle areas in real time for marking and path adjustment, thereby achieving efficient path planning and reliable obstacle avoidance in a dynamic environment. This improves the robot's navigation performance and autonomous decision-making capabilities, enabling it to operate stably and safely in complex and changing environments.

[0181] To facilitate understanding of the simultaneous positioning and map building method provided in this embodiment, Figure 7 As shown, this embodiment also provides a specific implementation method of the simultaneous positioning and map construction method, and the specific process is as follows:

[0182] 1. Frame processing

[0183] (1) ORB feature extraction: ORB features are extracted from the image frame by frame to provide feature point basis for subsequent tracking and map construction.

[0184] (2) Waiting for results: After feature extraction is completed, enter the waiting state and wait for the result feedback of the subsequent processing module.

[0185] (3) Outlier removal: Perform preliminary screening of the extracted feature points to remove obvious abnormal points (outliers) and improve the quality and reliability of the feature points.

[0186] (4) Tracking: Use the feature points of the previous frame to match the feature points of the current frame, estimate the movement of the camera, and realize the position tracking of the robot.

[0187] (5) New keyframe identification: Based on certain strategies (such as the number of feature points, tracking error, etc.), determine whether the current frame needs to be a new keyframe. If so, mark the frame as a keyframe and enter the keyframe processing flow.

[0188] 2. Semantic Segmentation

[0189] (1) RTMDet-Ins: performs semantic segmentation on the input image, outputs pixel-level classification results and corresponding object category labels, and identifies dynamic objects in the image.

[0190] (2) Object mask: Based on the semantic segmentation results, a corresponding binary mask is generated for the dynamic object category, where the dynamic object area is marked with a specific value and the static background is marked with another value. The binary mask is morphologically processed to expand the mask range of the dynamic object area and generate a semantic mask corresponding to the dynamic object.

[0191] 3. Optical flow calculation

[0192] (1) LiteFlowNet2: It calculates the optical flow of two adjacent frames and obtains the pixel-level motion vector field, which reflects the movement of pixels in the image.

[0193] (2) OTSU: Normalize the motion vector amplitude value obtained by optical flow calculation to generate a normalized grayscale image, and apply OTSU adaptive threshold segmentation to distinguish static background from dynamic areas and generate an initial optical flow mask.

[0194] (3) Dynamic area mask: Morphologically optimize the initial optical flow mask to obtain the optical flow mask corresponding to the dynamic area.

[0195] 4. Dynamic Point Culling

[0196] The semantic mask obtained by semantic segmentation and the optical flow mask obtained by optical flow calculation are fused to generate a comprehensive mask of the dynamic feature area. Then, dynamic feature points in the visual feature points are eliminated based on the comprehensive mask, and static feature points are retained, providing a reliable feature point basis for subsequent map construction.

[0197] 5. Keyframe Processing

[0198] (1) Keyframe insertion: Insert the newly identified keyframes into the map in preparation for map update and optimization.

[0199] (2) Map point removal: According to certain rules (such as the number of observations, projection error, etc.), unreliable map points in the map are removed to reduce the redundancy and noise of the map.

[0200] (3) New map point generation: Based on the key frames, new map points are generated through methods such as feature matching and triangulation to enrich the map information.

[0201] (4) Local bundle adjustment (BA) optimization: With the new keyframe as the center, the local map is optimized by Bundle Adjustment (BA), adjusting the camera pose and map point positions to improve the accuracy and consistency of the map.

[0202] (5) Redundant keyframe removal: Identify and remove redundant keyframes in the map, simplify the map structure, and improve the efficiency and real-time performance of the map.

[0203] 6. Dense Mapping

[0204] (1) Keyframe insertion and selection: In the dense mapping module, keyframes are further inserted and selected to prepare for the generation of dense point clouds.

[0205] (2) Point cloud generation: Based on the depth information and camera parameters of the key frame, the pixels of the key frame are projected into three-dimensional space to generate dense point cloud data.

[0206] (3) Local culling: Perform local culling on the generated dense point cloud to remove the point cloud points that do not meet the geometric constraints or observation conditions, thereby improving the quality of the point cloud.

[0207] (4) Filtering: Filter the point cloud to smooth the surface of the point cloud, reduce the influence of noise points, and make the point cloud more regular and continuous.

[0208] (5) Probability update: Based on the new observation information, the probability model of the point cloud is updated to reflect the reliability and existence probability of the point cloud.

[0209] 7. Loopback and Optimization

[0210] (1) Loop detection: By comparing the feature similarity between the current keyframe and the previous keyframe, it detects whether a loop has occurred, that is, the robot returns to the location it visited before.

[0211] (2) Global BA optimization: Based on loop closure detection, global BA optimization is performed on the entire map to adjust all camera poses and map point positions, eliminate cumulative errors, and improve the global consistency of the map.

[0212] Based on ORB-SLAM3, this embodiment adds a lightweight semantic segmentation network RTMDet-Ins-tiny and an optical flow network LiteFlowNet2. By identifying the surrounding environment, it generates a corresponding dynamic object mask, and thereby eliminates a priori and non-prior dynamic objects in the environment, thereby improving the positioning accuracy of visual SLAM and allowing the system to have good real-time performance. It also further expands the system architecture, adds a dense mapping thread, and uses keyframe data to build a semantic map to provide more comprehensive environmental modeling information. Based on the use of dynamic object masks to eliminate dynamic object point clouds, this method uses KD-Tree and Euclidean clustering methods to segment point cloud clusters, and eliminates missed areas of the semantic segmentation network and optical flow network by projecting to the previous frame. After filtering, a semantic point cloud is obtained, and a semantic octree map is incrementally constructed.

[0213] In summary, this embodiment solves the problem of serious decline in positioning accuracy of traditional visual SLAM in dynamic environments. It also solves the problem that the current semantic SLAM only eliminates a priori dynamic objects (such as humans) and ignores the impact of non-a priori dynamic objects (such as chairs pushed by people) that may exist in dynamic environments on positioning; and adds a semantic octree map to solve the problem that the ORB-SLAM3 framework only constructs a sparse point cloud map and cannot be used for subsequent robot upper-layer applications.

[0214] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0215] To facilitate better implementation of the simultaneous positioning and mapping method of the present application, the present invention also provides a simultaneous positioning and mapping device based on the aforementioned simultaneous positioning and mapping method. The terms herein have the same meanings as in the aforementioned simultaneous positioning and mapping method. For specific implementation details, please refer to the description in the method embodiment.

[0216] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of the simultaneous positioning and map building device provided in an embodiment of the present application. The simultaneous positioning and map building device may include a data acquisition module 201, a mask generation module 202, a feature processing module 203, and a map building module 204. The specific structure may be as follows:

[0217] The data acquisition module 201 is used to acquire multiple frames of continuous image data of the current environment in real time and extract visual feature points corresponding to each frame of image data;

[0218] The mask generation module 202 is used to perform semantic segmentation processing and optical flow calculation processing on the image data to generate corresponding semantic masks and optical flow masks;

[0219] A feature processing module 203 is used to remove dynamic feature points from visual feature points based on semantic masks and optical flow masks, and retain static feature points;

[0220] The map construction module 204 is used to generate a semantic point cloud based on static points, and to incrementally construct a semantic octree map based on the semantic point cloud and the octree module.

[0221] Optionally, in some embodiments, the mask generation module 202 may specifically include:

[0222] The semantic mask unit is used to perform frame-by-frame semantic segmentation processing on image data based on the semantic segmentation network to generate semantic masks of dynamic objects;

[0223] The optical flow mask unit is used to perform motion analysis on adjacent frame images based on the optical flow network and generate optical flow masks of dynamic areas.

[0224] Optionally, in some embodiments, the semantic mask unit is specifically used to: input the current frame image into a pre-trained semantic segmentation model, output pixel-level classification results and corresponding object category labels; based on the classification results, generate a corresponding binary mask for the dynamic object category, wherein the dynamic object area is marked as a first preset value and the static background is marked as a second preset value; perform morphological processing on the binary mask, expand the mask range of the dynamic object area, and generate a semantic mask corresponding to the dynamic object.

[0225] Optionally, in some embodiments, the optical flow mask unit is specifically used to: perform optical flow calculation on two adjacent frames of images based on the optical flow network to obtain a pixel-level motion vector field; generate a normalized grayscale image according to the amplitude value of the motion vector field; perform adaptive threshold segmentation on the grayscale image to distinguish between static background and dynamic areas, and generate an initial optical flow mask; perform morphological optimization on the initial optical flow mask to obtain an optical flow mask corresponding to the dynamic area.

[0226] Optionally, in some embodiments, the feature processing module 203 may specifically include:

[0227] A fusion unit is used to fuse the semantic mask and the optical flow mask to generate a comprehensive mask of the dynamic feature area;

[0228] The elimination unit is used to eliminate dynamic feature points from the visual feature points according to the comprehensive mask and retain the static feature points.

[0229] Optionally, in some embodiments, the fusion unit can be specifically used to: perform a logical OR operation on the semantic mask and the optical flow mask to generate a joint dynamic area; perform a connected domain analysis on the joint dynamic area, eliminate noise areas with an area smaller than a preset threshold, and form a comprehensive mask corresponding to the dynamic feature area.

[0230] Optionally, in some embodiments, the map construction module 204 may specifically include:

[0231] The point cloud unit is used to generate a 3D semantic point cloud based on the static points of the key frame and mark the 3D semantic point cloud with corresponding semantic labels;

[0232] The clustering unit is used to perform clustering optimization processing on the three-dimensional semantic point cloud to obtain an optimized static semantic point cloud;

[0233] The update unit is used to input the optimized static semantic point cloud into the octree module and incrementally update the semantic octree map.

[0234] Optionally, in some embodiments, the clustering unit can be specifically used to: perform Euclidean clustering on the current point cloud cluster to obtain independent point cloud clusters; project each independent point cloud cluster to the mask area of ​​the historical frame and calculate the dynamic area ratio; if the dynamic area ratio exceeds a preset threshold, the independent point cloud cluster is determined to be a missed dynamic object and is eliminated.

[0235] Optionally, in some embodiments, the device further includes a navigation module, which can be specifically used to:

[0236] According to the semantic labels and spatial distribution of static objects in the semantic octree map, the initial navigation path is generated based on the shortest path algorithm;

[0237] Detect new dynamic obstacle areas through the current frame image and optical flow mask acquired in real time and mark them as temporary restricted areas;

[0238] Mapping the temporary restricted area into the semantic octree map, dynamically adjusting the initial navigation path to bypass the restricted area, and obtaining the adjusted navigation path;

[0239] The robot motion instructions are updated according to the adjusted navigation path, and the dynamic obstacle information in the semantic octree map is updated synchronously.

[0240] The specific definitions of the simultaneous positioning and mapping device can be found in the definitions of the simultaneous positioning and mapping method above and will not be repeated here. Each module in the aforementioned simultaneous positioning and mapping device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of these modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0241] The synchronous positioning and map construction device provided in this embodiment generates semantic masks and optical flow masks through semantic segmentation and optical flow calculation, accurately identifies and eliminates dynamic feature points, effectively solves the problem of decreased positioning accuracy of traditional SLAM in dynamic environments, and generates semantic point clouds after retaining static points, thereby incrementally constructing semantic octree maps based on semantic point clouds and octree modules, achieving richer, denser and more semantic map construction, meeting the subsequent upper-level application requirements of the robot, while reducing computing costs and improving the real-time performance of map construction.

[0242] In addition, the present invention also provides an electronic device, such as Figure 9 , which shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0243] The electronic device may include one or more processors 301 of processing cores, one or more computer-readable storage media memories 302, a power supply 303, an input unit 304 and other components. Those skilled in the art will appreciate that Figure 9 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0244] The processor 301 is the control center of the electronic device. It connects all parts of the electronic device using various interfaces and lines. By running or executing software programs and / or modules stored in the memory 302 and accessing data stored in the memory 302, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 301.

[0245] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and the simultaneous positioning and map construction method by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0246] The electronic device also includes a power supply 303 for supplying power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 303 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0247] The electronic device may further include an input unit 304, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0248] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 301 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application programs stored in the memory 302 to implement various functions as follows:

[0249] Acquire multiple frames of continuous image data of the current environment in real time and extract the visual feature points corresponding to each frame of image data; perform semantic segmentation processing and optical flow calculation on the image data respectively to generate corresponding semantic masks and optical flow masks; remove dynamic feature points from the visual feature points based on the semantic mask and optical flow mask, and retain static feature points; generate semantic point clouds based on static points, and incrementally construct semantic octree maps based on the semantic point clouds and octree modules.

[0250] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0251] The embodiment of the present application generates semantic masks and optical flow masks through semantic segmentation and optical flow calculation, accurately identifies and eliminates dynamic feature points, effectively solves the problem of decreased positioning accuracy of traditional SLAM in dynamic environments, and generates semantic point clouds after retaining static points, thereby incrementally constructing semantic octree maps based on semantic point clouds and octree modules, achieving richer, denser and more semantic map construction, meeting the subsequent upper-level application requirements of the robot, while reducing computing costs and improving the real-time performance of map construction.

[0252] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0253] To this end, an embodiment of the present application provides a storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps of any of the simultaneous positioning and map building methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:

[0254] Acquire multiple frames of continuous image data of the current environment in real time and extract the visual feature points corresponding to each frame of image data; perform semantic segmentation processing and optical flow calculation on the image data respectively to generate corresponding semantic masks and optical flow masks; remove dynamic feature points from the visual feature points based on the semantic mask and optical flow mask, and retain static feature points; generate semantic point clouds based on static points, and incrementally construct semantic octree maps based on the semantic point clouds and octree modules.

[0255] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0256] The storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0257] Since the instructions stored in the storage medium can execute the steps of any of the simultaneous positioning and map construction methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any of the simultaneous positioning and map construction methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0258] The above is a detailed introduction to a synchronous positioning and map construction method and device provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core ideas. At the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for simultaneous positioning and map construction, characterized in that: The steps include: Acquire multiple frames of continuous image data of the current environment in real time and extract the visual feature points corresponding to each frame of image data; Performing semantic segmentation processing and optical flow calculation processing on the image data respectively to generate corresponding semantic masks and optical flow masks; Eliminating dynamic feature points from the visual feature points based on the semantic mask and the optical flow mask, and retaining static points; A semantic point cloud is generated based on the static points, and a semantic octree map is incrementally constructed based on the semantic point cloud and an octree module.

2. The method for simultaneous positioning and map construction according to claim 1, wherein: The performing semantic segmentation processing and optical flow calculation processing on the image data respectively to generate corresponding semantic masks and optical flow masks includes: Performing frame-by-frame semantic segmentation processing on the image data based on a semantic segmentation network to generate a semantic mask of the dynamic object; Based on the optical flow network, motion analysis is performed on adjacent frame images to generate optical flow masks of dynamic areas.

3. The method for simultaneous positioning and map construction according to claim 2, wherein: The step of performing frame-by-frame semantic segmentation processing on the image data based on the semantic segmentation network to generate a semantic mask of a dynamic object includes: Input the current frame image into the pre-trained semantic segmentation model and output the pixel-level classification results and the corresponding object category labels; According to the classification result, a corresponding binary mask is generated for the dynamic object category, wherein the dynamic object area is marked as a first preset value and the static background is marked as a second preset value; Morphological processing is performed on the binary mask to expand the mask range of the dynamic object area and generate a semantic mask corresponding to the dynamic object.

4. The method for simultaneous positioning and map construction according to claim 2, wherein: The method of performing motion analysis on adjacent frame images based on an optical flow network to generate an optical flow mask of a dynamic area includes: Based on the optical flow network, optical flow calculation is performed on two adjacent frames of images to obtain the pixel-level motion vector field; generating a normalized grayscale image according to the amplitude value of the motion vector field; Performing adaptive threshold segmentation on the grayscale image to distinguish the static background from the dynamic area and generate an initial optical flow mask; The initial optical flow mask is morphologically optimized to obtain the optical flow mask corresponding to the dynamic area.

5. The method for simultaneous positioning and map construction according to claim 1, wherein: The removing of dynamic feature points from the visual feature points based on the semantic mask and the optical flow mask and retaining static feature points includes: fusing the semantic mask and the optical flow mask to generate a comprehensive mask of the dynamic feature area; Dynamic feature points in the visual feature points are eliminated according to the comprehensive mask, and static feature points are retained.

6. The method for simultaneous positioning and map construction according to claim 5, wherein: The fusing of the semantic mask and the optical flow mask to generate a comprehensive mask of the dynamic feature area includes: performing a logical OR operation on the semantic mask and the optical flow mask to generate a joint dynamic region; A connected domain analysis is performed on the joint dynamic region, and noise regions with an area smaller than a preset threshold are eliminated to form a comprehensive mask corresponding to the dynamic feature region.

7. The method for simultaneous positioning and map construction according to claim 5, wherein: Generating a semantic point cloud based on the static points, and incrementally constructing a semantic octree map based on the semantic point cloud and an octree module, includes: Generate a three-dimensional semantic point cloud based on the static points of the key frame, and mark the three-dimensional semantic point cloud with corresponding semantic tags; Performing clustering optimization processing on the three-dimensional semantic point cloud to obtain an optimized static semantic point cloud; The optimized static semantic point cloud is input into the octree module to incrementally update the semantic octree map.

8. The method for simultaneous positioning and map construction according to claim 7, wherein: The clustering optimization process is performed on the three-dimensional semantic point cloud to obtain an optimized static semantic point cloud, including: Perform Euclidean clustering on the current point cloud cluster to obtain independent point cloud clusters; Projecting each of the independent point cloud clusters onto the mask area of ​​the historical frame and calculating the dynamic area ratio; If the proportion of the dynamic area exceeds a preset threshold, the independent point cloud cluster is determined to be a missed dynamic object and is removed.

9. The method for simultaneous positioning and map construction according to claim 1, wherein: After generating a semantic point cloud based on the static points and incrementally constructing a semantic octree map based on the semantic point cloud and an octree module, the method further includes: According to the semantic labels and spatial distribution of static objects in the semantic octree map, the initial navigation path is generated based on the shortest path algorithm; Detect new dynamic obstacle areas through the current frame image and optical flow mask acquired in real time and mark them as temporary restricted areas; Mapping the temporary restricted area to the semantic octree map, dynamically adjusting the initial navigation path to bypass the restricted area, and obtaining an adjusted navigation path; The robot motion instructions are updated according to the adjusted navigation path, and the dynamic obstacle information in the semantic octree map is updated synchronously.

10. A synchronous positioning and map building device, characterized in that: include: The data acquisition module is used to acquire multiple frames of continuous image data of the current environment in real time and extract the visual feature points corresponding to each frame of image data; A mask generation module is used to perform semantic segmentation processing and optical flow calculation processing on the image data respectively to generate corresponding semantic masks and optical flow masks; A feature processing module, configured to remove dynamic feature points from the visual feature points based on the semantic mask and the optical flow mask, and retain static feature points; A map construction module is used to generate a semantic point cloud based on the static points, and to incrementally construct a semantic octree map based on the semantic point cloud and the octree module.

Images