A real-time dynamic semantic mapping method and system based on multi-sensor fusion

Abstract

The application discloses a kind of real-time dynamic semantic mapping method and system based on multi-sensor fusion, comprising: extracting the feature points of each frame color image in color image sequence, obtain first feature point set, color image sequence and depth image sequence are sequentially segmented by semantic segmentation network, and the semantic segmentation result of each frame is obtained;Dynamic object segmentation network is sequentially segmented to event frame sequence, and the dynamic object area of each frame is obtained, based on first feature point set, the feature points in each frame dynamic object area are removed, and second feature point set is obtained;Each frame in second feature point set is matched with the feature points of previous frame, and global optimal camera pose sequence is obtained by combining nonlinear optimization method;Based on semantic segmentation result, global optimal camera pose sequence and depth image, semantic octree mapping is completed.The application significantly improves the accuracy and real-time performance of semantic mapping.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of semantic mapping, and particularly relates to a real-time dynamic semantic mapping method and system based on multi-sensor fusion. BACKGROUND

[0002] Visual SLAM is a SLAM system taking images as the main source of environmental perception information, and can be applied to fields such as unmanned driving and augmented reality, and is a popular research direction in recent years. Typical visual SLAM algorithms take camera pose estimation as the main target, and reconstruct a 3D map through multi-view geometry theory. In order to improve the data processing speed, some visual SLAM algorithms first extract sparse image features, and realize inter-frame estimation and loop closure detection through matching between feature points. The ORB feature is widely used in the field of visual SLAM due to its good robustness, optimal distinguishing ability and fast processing speed.

[0003] The deep learning algorithm is a mainstream recognition algorithm in the current field of computer vision, which learns the hierarchical feature representation of images by relying on a multi-layer neural network, and can achieve higher recognition accuracy compared with traditional recognition methods. At the same time, deep learning can associate images with semantics, and combined with SLAM technology, a semantic map of the environment is generated, and a semantic knowledge base of the environment is constructed, which is used for robot cognition and task reasoning, and improves the service ability of the robot and the intelligence of human-computer interaction.

[0004] The key of semantic SLAM lies in the accurate recognition of object targets in the environment, and the deep learning technology which has arisen in recent years is the most potential and advantageous object recognition method at present, so the combination of deep learning and semantic SLAM has attracted widespread attention from researchers in the field. The challenge of generating semantic information lies in the accurate classification of object targets at the level or pixel level.

[0005] Octree map is a commonly used map form in navigation, and has good compression performance itself. Point cloud map is usually very large, and a 640 pixel x 480 pixel image will generate about 300,000 spatial points, which requires a large amount of storage space. Moreover, importantly, its largeness is not necessary, and the point cloud map provides a lot of unnecessary details, such as wrinkles on the carpet, shadows in dark places, etc. In octree, information about whether it is occupied is stored in the node. When all the child nodes of a certain block are occupied or not, there is no need to expand this node. It is a flexible, compressed and updatable map form; when the octree is limited to 10 layers, the total volume that can be modeled is about 8 raised to the power of 10 cubic centimeters, that is, 1073 cubic meters, which is enough to model a room.

[0006] The general environment is generally divided into dynamic environment and static environment, unlike static environment, dynamic environment has some objects that can move autonomously, such as people, animals, vehicles and the like. At present, most visual SLAM systems can obtain specific pose and map in static environment. But in the scene with dynamic objects, its positioning and mapping ability is weak. Because the moving object will greatly interfere with the structure and positioning. In recent years, some visual SLAM solutions combined with semantic network have achieved good results.

[0007] For example, the Chinese patent document with publication number CN112132897A, but it has the following shortcomings:

[0008] (1) Only the CNN network based on the RGB image is used on the semantic segmentation, without combining the depth information. And the network is large, which is not suitable for the NX board with limited calculation amount.

[0009] (2) The dynamic object is judged by calculating whether the difference between each key feature point and the projected depth value and the parallax angle is greater than the threshold value, which can filter out the static person while filtering out the dynamic object. But this method will produce different degrees of error in the actual motion process, especially when the light is unstable; and the dynamic target is easy to produce motion blur and ghosting, which affects feature extraction, camera pose calculation and dynamic object region segmentation.

[0010] (3) The mask based on semantic segmentation is needed to judge the dynamic object region. But the category of semantic segmentation must be trained in advance, if there is a moving object of unknown category, semantic segmentation cannot segment it out, and then it is difficult to judge the dynamic object region with the mask; which will affect the removal of feature points in the dynamic region, and then affect the calculation of camera pose. SUMMARY

[0011] One of the purposes of the present application is to provide a real-time dynamic semantic mapping method based on multi-sensor fusion, which improves the accuracy and real-time performance of semantic mapping.

[0012] To achieve the above purpose, the technical scheme adopted by the present application is:

[0013] A real-time dynamic semantic mapping method based on multi-sensor fusion, the real-time dynamic semantic mapping method based on multi-sensor fusion, comprising:

[0014] Step 1, training semantic segmentation network and dynamic object segmentation network;

[0015] Step 2, obtaining color image, depth image and event frame of target environment, matching images based on time stamp, obtaining color image sequence, depth image sequence and event frame sequence;

[0016] Step 3, extracting feature points of each frame of the color image sequence to obtain a first feature point set, performing semantic segmentation on the color image sequence and the depth image sequence frame by frame through the semantic segmentation network to obtain a semantic segmentation result of each frame, the semantic segmentation result including a semantic mask and a confidence of the color image;

[0017] Step 4, performing dynamic object segmentation on the event frame sequence frame by frame through the dynamic object segmentation network to obtain a dynamic object region of each frame, removing feature points in the dynamic object region of each frame based on the first feature point set to obtain a second feature point set;

[0018] Step 5, performing feature matching on each frame of the second feature point set with feature points of a previous frame, and obtaining a global optimal camera pose sequence through a nonlinear optimization method;

[0019] Step 6, completing semantic octree mapping based on the semantic segmentation result, the global optimal camera pose sequence and the depth image.

[0020] The following also provides several optional modes, but not as an additional limitation to the above overall scheme, just a further supplement or preferred, without technical or logical contradiction, each optional mode can be combined with the above overall scheme alone, and can also be combined between multiple optional modes.

[0021] As preferred, the training of the semantic segmentation network and the dynamic object segmentation network includes:

[0022] The semantic segmentation network is first trained based on a public data set, and after training, the color image and the depth image collected and labeled by the depth camera are fine-tuned, the network weight obtained is loaded to the semantic segmentation network, and the training of the semantic segmentation network is completed;

[0023] The dynamic object segmentation network is first trained based on a synthetic data set, and after training, the event frame collected and labeled by the event camera is fine-tuned, the network weight obtained is loaded to the dynamic object segmentation network, and the training of the dynamic object segmentation network is completed.

[0024] As preferred, the event frame is obtained as follows:

[0025] An event sequence is collected based on an event camera, the event sequence being (x, y, p, t), wherein (x, y) is an event coordinate, p is a positive or negative polarity, and t represents time information of the current event;

[0026] A time window is set, the event sequence is converted into an event frame according to the length of the time window, and the converted event frame is divided into three channels, wherein:

[0027] All events with positive polarity p are accumulated, and the number of accumulated events at each (x, y) position is taken as the pixel value at (x, y) coordinates to obtain a first channel of the event frame;

[0028] All events with negative polarity p are accumulated, and the number of accumulated events at each (x, y) position is taken as the pixel value at (x, y) coordinates to obtain a second channel of the event frame;

[0029] The average time surface is taken as a third channel of the event frame, and the number of accumulated events at each (x, y) position within the time window is I x,y The average time surface is

[0030] As preferred, the semantic octree mapping is completed based on the semantic segmentation result, the globally optimal camera pose sequence and the depth image, comprising:

[0031] Taking the semantic segmentation result, the globally optimal camera pose and the depth image belonging to the same frame;

[0032] Mapping the semantic segmentation result to a three-dimensional coordinate system according to the globally optimal camera pose and the depth image to obtain a three-dimensional coordinate point;

[0033] Voxel filtering is performed on the three-dimensional coordinate point;

[0034] Taking the voxel filtered three-dimensional coordinate point for semantic fusion to complete the semantic mapping for the current frame;

[0035] Continuing to take the semantic segmentation result, the globally optimal camera pose and the depth image of the next frame for semantic mapping to realize real-time dynamic semantic mapping.

[0036] As preferred, the mapping of the semantic segmentation result to a three-dimensional coordinate system according to the globally optimal camera pose and the depth image to obtain a three-dimensional coordinate point, comprises:

[0037] Taking the two-dimensional coordinate point P of a pixel in the semantic segmentation result as (u, v);

[0038] According to the depth image, the depth information at the two-dimensional coordinate point (u, v) is obtained as Z;

[0039] According to the camera intrinsic parameters (f x , f y , c x , c y ), the three-dimensional coordinate point P c converted to the camera coordinate system at the two-dimensional coordinate point (u, v) is obtained as Then, the three-dimensional coordinate point P W in the world coordinate system is obtained through the globally optimal camera pose (R, t) transformation ct, where R is a rotation matrix and t is a translation vector.

[0040] As preferred, the voxel filtering on the three-dimensional coordinate points comprises:

[0041] setting a maximum search radius and a mapping radius;

[0042] discarding the three-dimensional coordinate points with a distance greater than the maximum search radius from the origin;

[0043] proportionally mapping the three-dimensional coordinate points with a distance less than or equal to the maximum search radius and greater than the mapping radius from the origin to within the mapping radius, and mapping according to the direction from the current point to the origin;

[0044] performing voxel filtering on the three-dimensional coordinate points mapped within the mapping radius, and mapping the three-dimensional coordinate points reserved after the voxel filtering to their original positions;

[0045] taking the three-dimensional coordinate points originally within the mapping radius and the three-dimensional coordinate points reserved after the voxel filtering as the final three-dimensional coordinate points after the voxel filtering.

[0046] As preferred, the semantic fusion is a maximum fusion method or a Bayesian fusion method.

[0047] As preferred, the semantic fusion is a maximum fusion method, and the taking the three-dimensional coordinate points after the voxel filtering to perform semantic fusion comprises:

[0048] a. determining a dynamic object region in the current image according to the segmentation result of the dynamic object segmentation network, judging whether an object is segmented in the dynamic object region according to the segmentation result of the semantic segmentation network, if no object is segmented, marking the dynamic object region as static background and performing step d, if an object is segmented, proceeding to the next step;

[0049] b. calculating the intersection-over-union of the dynamic object region and the region where the object segmented by the semantic segmentation network is located, if the intersection-over-union is greater than or equal to a threshold, marking the union of the dynamic object region and the region where the object segmented by the semantic segmentation network is located as a dynamic target; otherwise, marking the dynamic object region as a dynamic target;

[0050] c. matching the three-dimensional coordinate points in the dynamic target with the semantics of the same three-dimensional coordinate points in the previous frame, when the three-dimensional coordinate points at the same position are matched to a static background point for the first time, replacing the semantics of the three-dimensional coordinate points with the semantics of the static background point, and in subsequent matching, if the semantics at the same position in the previous frame are the same, the semantics remain unchanged, and the confidence of the position is the average of the two; if the semantics at the same position in the previous frame are different, replacing the semantics of the three-dimensional coordinate points with the semantics with greater confidence, and the confidence is a preset multiple of the greater confidence;

[0051] d, determining whether the determination of all dynamic object regions is completed, if not, returning to step a; otherwise, completing semantic fusion.

[0052] As preferred, the semantic fusion is a Bayesian fusion method, and the semantic fusion of the voxel-filtered three-dimensional coordinate points comprises:

[0053] The classification results of the semantic information of each pixel output by the semantic segmentation network are re-divided, the first class and the second class with the highest confidence are taken as a new first class and a new second class, and the remaining classes are taken as a new third class, and semantic fusion is performed based on the three re-divided classes.

[0054] The real-time dynamic semantic mapping method based on multi-sensor fusion provided by the application combines the texture information of the RGB image and the depth information of the depth map for semantic segmentation, and uses tensorRT to optimize the network, so that the purpose of real-time mapping is achieved. In terms of judging dynamic objects, the event camera has unique advantages, is robust to changes in light, and will not produce motion blur and ghosting due to the movement of dynamic objects, thereby affecting the segmentation of dynamic regions. Moreover, the event camera dynamic object segmentation is class-independent, and there is no need to label the class of dynamic objects in advance. As long as the object is in motion, it can be segmented, which can significantly improve the real-time performance and accuracy of mapping.

[0055] The second object of the application is to provide a real-time dynamic semantic mapping system based on multi-sensor fusion, which improves the accuracy and real-time performance of semantic mapping.

[0056] To achieve the above object, the technical scheme adopted by the application is:

[0057] A real-time dynamic semantic mapping system based on multi-sensor fusion, comprising a processor and a memory storing a plurality of computer instructions, wherein the computer instructions are executed by the processor to implement the steps of the real-time dynamic semantic mapping method based on multi-sensor fusion. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 A flowchart of the real-time dynamic semantic mapping method based on multi-sensor fusion of the application;

[0059] Figure 2 An embodiment structure diagram of the dynamic object segmentation network of the application;

[0060] Figure 3 A schematic diagram of voxel mapping of the application. DETAILED DESCRIPTION

[0061] With reference to the drawings and the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of the present application.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0063] In order to overcome the defects of semantic mapping in the prior art, the embodiment proposes a real-time dynamic semantic mapping method based on multi-sensor fusion. The prior art uses RGB images for semantic segmentation, and the semantic segmentation network is large, which is not conducive to running on the development board with limited memory and computation amount. In the embodiment, the texture information of the RGB image and the depth information of the depth image are combined for semantic segmentation, and the network is optimized by tensorRT to achieve the purpose of real-time mapping.

[0064] In the prior art, when calculating whether the feature point belongs to a dynamic object, it needs to rely on the stability of light, and the motion blur of the object will have a great influence on the result. In the embodiment, the unique advantage of the event camera is used to judge the dynamic object, which is robust to changes in light and will not produce motion blur and ghosting due to the movement of dynamic objects, thereby affecting the segmentation of dynamic regions.

[0065] In the prior art, when judging the dynamic object region, it needs to rely on the mask of the semantic segmentation result, and the semantic segmentation needs to know the category of the dynamic object in advance. The embodiment uses the category independence of the event camera dynamic object segmentation, which does not need to mark the category of the dynamic object in advance, and can segment any moving object.

[0066] The prior art does not perform voxel filtering on the mapped point cloud, and the semantic map occupies a large amount of memory. In the embodiment, in the semantic mapping stage, the voxel filtering is performed on the more concentrated point cloud through line mapping, thereby reducing the storage space required by the map.

[0067] Specifically, the real-time dynamic semantic mapping method based on multi-sensor fusion of the embodiment mainly includes the following four parallel parts:

[0068] Semantic segmentation part: semantic segmentation is performed by combining RGB images and depth maps.

[0069] Event camera part: parallel to the RGB-D camera, distinguish dynamic objects and static backgrounds.

[0070] SLAM part: extract feature points, remove feature points of dynamic objects, and calculate camera pose using SLAM algorithm.

[0071] Octree mapping part: create semantic octree map using semantic segmentation results, depth image and camera pose, and perform voxel filtering on point cloud through line mapping to optimize storage space of map.

[0072] As shown in Figure 1 Based on the above four parallel parts, the real-time dynamic semantic mapping method based on multi-sensor fusion of the embodiment includes the following steps:

[0073] Step 1, training semantic segmentation network and dynamic object segmentation network.

[0074] When training the semantic segmentation network and the dynamic object segmentation network, first, a predetermined number of color images, depth images and event frames are collected and labeled.

[0075] The semantic segmentation network is first trained based on a public dataset (such as the SUNRGB-D dataset or the NYUv2 dataset), and after training, the color images and depth images collected and labeled by the depth camera are fine-tuned, and the network weights obtained are loaded into the semantic segmentation network to complete the training of the semantic segmentation network.

[0076] The dynamic object segmentation network is first trained based on a synthetic dataset (such as the synthetic data of ETH), and after training, the event frames collected and labeled by the event camera are fine-tuned, and the network weights obtained are loaded into the dynamic object segmentation network to complete the training of the dynamic object segmentation network.

[0077] When training the network of the embodiment, first, training is performed on a public or synthetic dataset, and then network migration is performed using collected data to reduce data collection pressure. The semantic segmentation network of the embodiment is a deep neural network ESANet, and one network structure of the dynamic object segmentation network provided in the embodiment is as shown in Figure 2 From the output side to the input side, it includes convolution layers (convolution kernel size 5*5, channel number 8, step 2), convolution layers (convolution kernel size 5*5, channel number 16, step 2), convolution layers (convolution kernel size 5*5, channel number 32, step 2), depth convolution layers (convolution kernel size 5*5, channel number 16, step 2), depth convolution layers (convolution kernel size 5*5, channel number 8, step 2) and depth convolution layers (convolution kernel size 5*5, channel number 2, step 2) connected in sequence.

[0078] Step 2, obtain color images, depth images and event frames of the target environment, match images based on timestamps, and obtain color image sequences, depth image sequences and event frame sequences.

[0079] The embodiment acquires a target environment color image and a depth image of a set number of frames through an RGB-D camera, obtains events through an event camera and processes the events into an event frame; matches the images based on timestamps to obtain a color image sequence, a depth image sequence and an event frame sequence.

[0080] Compared with a traditional frame-based camera, an event camera can capture luminance changes at almost infinite frame rates and record events at specific time points and image positions. In particular, for moving scenes, an event camera has great advantages in data rate, speed and dynamic range. A relatively new event camera such as DAVIS (Dynamic and Active-pixel Vision Sensor) has an IMU (Inertial Measurement Unit) module, which can measure linear acceleration and angular velocity in three axes and is often used to obtain three-dimensional motion information of the camera.

[0081] The process of processing events obtained by the event camera into an event frame is as follows: based on the event camera, an event sequence is collected, the event sequence is (x, y, p, t), where (x, y) is the event coordinates, p is the positive or negative polarity, the positive polarity indicates that the light intensity at the current time is stronger than the light intensity at the previous time and the change exceeds a certain threshold; otherwise, the negative polarity indicates that the light intensity is weaker; t represents the time information of the current event; a time window (in this embodiment, the time window is 5 ms) is set, and the event sequence is converted into an event frame according to the length of the time window, and the converted event frame is divided into three channels:

[0082] All events with positive polarity are accumulated, and the number of accumulated events at each (x, y) position is taken as the pixel value at the (x, y) coordinates to obtain the first channel of the event frame.

[0083] All events with negative polarity are accumulated, and the number of accumulated events at each (x, y) position is taken as the pixel value at the (x, y) coordinates to obtain the second channel of the event frame.

[0084] The average time surface is taken as the third channel of the event frame, and the number of accumulated events (including positive and negative polarities) at each (x, y) position in the time window is I x,y , and the average time surface is where ∑t is the sum of the time information of each event in the accumulated events.

[0085] In other embodiments, in order to improve the accuracy of the event frame, motion compensation calculation can also be performed on the event frame after the event frame is calculated. Let the coordinates of the i-th event point be g i , and the time information be t i , then the coordinates after motion compensation are wherein p0 denotes a camera mapping model from a camera coordinate system to a pixel coordinate system, Z(g i ) denotes a depth at g i , time t i ; and denotes a camera pose transformation from time t i to t k , t k is the time information of the kth event point, and the kth event point can be the last event point of the ith event point.

[0086] Step 3, extracting feature points of each frame of the color image sequence to obtain a first feature point set, performing semantic segmentation on the color image sequence and the depth image sequence frame by frame through a semantic segmentation network to obtain a semantic segmentation result of each frame, and the semantic segmentation result includes a semantic mask and a confidence of the color image.

[0087] In order to facilitate extraction, the feature points in this embodiment are ORB feature points. It is easily understood that the ORB feature points are a preferred type of feature point used in this embodiment, and in other embodiments, the ORB feature points can be replaced by other corner features, or can be replaced by feature points learned based on a neural network (such as SuperPoint).

[0088] In terms of semantic segmentation, this embodiment combines the texture information of the RGB image and the depth information of the depth map for semantic segmentation, and uses a tensorRT optimized network to realize real-time semantic mapping on an NX board.

[0089] Step 4, performing dynamic object segmentation on the event frame sequence frame by frame through a dynamic object segmentation network to obtain a dynamic object region of each frame, and removing feature points in the dynamic object region of each frame based on the first feature point set to obtain a second feature point set.

[0090] Similarly, when removing the feature points in the dynamic object region of each frame, the ORB feature points in this region are removed, and in other embodiments, the ORB feature points can be replaced by other corner features, or can be replaced by feature points learned based on a neural network (such as SuperPoint).

[0091] In terms of judging dynamic objects, this embodiment takes advantage of the unique advantages of the event camera, is robust to changes in light, and will not produce motion blur and ghosting due to the movement of dynamic objects. By using a dynamic segmentation network, the dynamic objects can be stably segmented in a complex background. Moreover, the mask of semantic segmentation is not needed to judge the dynamic object region, which is class-independent and does not need to train the classes of dynamic objects in advance. As long as the object is in motion, it can be segmented.

[0092] Step 5: Perform feature matching between each frame in the second feature point set and the feature points of the previous frame, and combine the nonlinear optimization method to obtain the globally optimal camera pose sequence.

[0093] In this embodiment, the tracking and mapping module is built on the ORB-SLAM3 framework and consists of three parallel threads: a tracking thread, a local mapping thread, and a loop closure detection thread. RGB images and depth images are input into the system. The tracking thread extracts ORB features and feature point descriptors from the RGB images. By finding matching feature points in the local map and calculating the fundamental matrix F or homography matrix H, bundle adjustment (BA) is used to minimize reprojection errors, thereby estimating the camera pose.

[0094] It should be noted that the global basis identification (BA) and loop closure detection steps in the ORB-SLAM3 framework are not mandatory and can be accelerated by removing these two parts. Furthermore, the ORB-SLAM3 framework is not strictly necessary; calculating the homography matrix H or the fundamental matrix F to obtain the camera pose are also feasible methods.

[0095] Step 6: Construct a semantic octree based on the semantic segmentation results, the globally optimal camera pose sequence, and the depth image.

[0096] Step 6a: Point cloud computing: Take the semantic segmentation results, the global optimal camera pose and depth image belonging to the same frame; map the semantic segmentation results to the three-dimensional coordinate system according to the global optimal camera pose and depth image to obtain the three-dimensional coordinate points.

[0097] The semantic segmentation result is a two-dimensional image, which needs to be mapped to a three-dimensional coordinate system. The specific mapping process is as follows: Take the two-dimensional coordinate point P (u, v) of a pixel in the semantic segmentation result; obtain the depth information Z at the two-dimensional coordinate point (u, v) from the depth image; and then, based on the camera intrinsic parameters (f... x ,f y ,c x ,c y The two-dimensional coordinate point (u,v) is transformed into the three-dimensional coordinate point P in the camera coordinate system. c for Then, the three-dimensional coordinate point P in the world coordinate system is obtained through the global optimal camera pose (R,t) transformation. W =RP c +t, where R is the rotation matrix and t is the translation vector. Where f x f y c represents the focal length in the x and y directions, respectively. x c y These are the offsets of the optical axis relative to the coordinate center of the projection plane.

[0098] Step 6b: Point cloud voxel filtering: Voxel filtering is performed on the 3D coordinate points.

[0099] In this embodiment, when performing voxel filtering, the maximum search radius and mapping radius are first set. Since the computational cost of building an octree by searching for distant points is large, this embodiment limits the maximum search radius to 5 meters (which can be freely adjusted). Because points farther from the origin will be sparser than points closer to the origin, voxel filtering cannot be uniformly controlled; therefore, a mapping radius of 2 meters is set (which can be freely adjusted).

[0100] In this embodiment, 3D coordinate points whose distance from the origin is greater than the maximum search radius are discarded. Points within the mapping radius do not require filtering and are directly inserted into the octree map. However, point clouds between the mapping radius and the maximum search radius require voxel filtering, specifically by projecting them into the mapping radius according to the direction vector from the origin to the point.

[0101] like Figure 3 As shown, mapping is performed along the direction from the current point to the origin, that is, along the reverse direction of the ray from the origin to the current point, reducing the distance between the current point and the origin by a certain ratio, thus mapping the current point into the mapping radius. When projected into the mapping radius, these points will be more densely packed, allowing for more effective voxel filtering. Therefore, voxel filtering is performed on the projected points, and then they are mapped back to their original positions. This saves some storage space.

[0102] In this embodiment, voxel filtering is performed on the three-dimensional coordinate points mapped to the mapping radius, and the three-dimensional coordinate points retained after voxel filtering are mapped back to their original positions. Finally, the original three-dimensional coordinate points located within the mapping radius and the three-dimensional coordinate points retained after voxel filtering are taken as the final three-dimensional coordinate points after voxel filtering.

[0103] Mapping can be performed at a fixed ratio (e.g., 0.4 times), or a resolution can be given, and then the direction vector from the origin to the current point can be calculated. In this case, the coordinates of the mapped point can be calculated by adding the origin coordinates to the direction vector and then multiplying the mapping radius by the resolution. This mapping method is more reasonable and helps to improve the filtering effect.

[0104] In the experimental example, only 26,596 points remained after filtering out 307,200 points. This shows that in the semantic mapping stage, this embodiment can significantly reduce the storage space required for the semantic map by performing voxel filtering on the point cloud through line mapping.

[0105] Step 6c: semantic fusion part: the same three-dimensional coordinate point in the subsequent frame can be judged as the same semantic, or can be judged as different semantics, at this time the three-dimensional coordinate point needs to be fused. Usually, there are two methods of Bayesian fusion method and maximum fusion method for semantic fusion. In order to reduce the calculation overhead, the maximum fusion method is adopted in this embodiment.

[0106] Maximum fusion method: if the semantic of the three-dimensional coordinate point is the same as the semantic of the previous frame, the semantic after fusion is still the same, but the confidence is the average of the two, that is, conf=(conf1+conf2) / 2; if the semantics are different, the semantic with higher confidence is taken, and the confidence is 90% (adjustable) of the higher confidence.

[0107] Specifically, the maximum fusion method is used for semantic fusion, including the following steps:

[0108] a, mark the dynamic object and the static background separately: determine the dynamic object region in the current image according to the segmentation result of the dynamic object segmentation network, judge whether an object is segmented in the dynamic object region according to the segmentation result of the semantic segmentation network, if no object is segmented, mark the dynamic object region as static background and execute step d; if an object is segmented, proceed to the next step.

[0109] b, calculate the intersection over union (IOU) of the dynamic object region and the region where the object segmented by the semantic segmentation network is located, if the intersection over union is greater than or equal to a threshold (for example, IOU>=40% (the ratio can be adjusted)), mark the union of the dynamic object region and the region where the object segmented by the semantic segmentation network is located as a dynamic target; otherwise, mark the dynamic object region as a dynamic target.

[0110] c, match the three-dimensional coordinate point in the dynamic target with the semantic of the same three-dimensional coordinate point in the previous frame, when the three-dimensional coordinate point at the same position is matched to a static background point for the first time, replace the semantic of the three-dimensional coordinate point with the semantic of the static background point, and in subsequent matching, if the semantic at the same position is the same as that of the previous frame, the semantic does not change, and the confidence of the position is the average of the two; if the semantics at the same position are different from those of the previous frame, replace the semantic of the three-dimensional coordinate point with the semantic with higher confidence, and the confidence is a preset multiple of the larger confidence, the preset multiple is between 0 and 1, preferably 0.9, which can be adjusted according to actual needs.

[0111] d, judge whether the judgment of all dynamic object regions is completed, if not, return to step a; otherwise, complete the semantic fusion.

[0112] The embodiment is aimed at updating the semantics of dynamic object points to the semantics of static background points when the three-dimensional coordinate points of dynamic targets are matched to static background points in subsequent frames, but the semantics of static background points are not updated to the semantics of dynamic object points. And when the semantics of dynamic targets is updated to the semantics of static background points for the first time, the static semantics is directly replaced, and the subsequent semantic fusion method is used for updating, so as to realize the continuous fusion of semantics in the octree map.

[0113] When the embodiment performs continuous fusion of semantics in the octree map, the semantics of the three-dimensional coordinate points of the current frame and the three-dimensional coordinate points of the last frame are fused continuously. The last frame here should be understood as the last frame after the semantic fusion, so as to always maintain the most accurate semantics. In real-time dynamic semantic mapping, semantic mapping is performed according to the semantic segmentation result of each frame, the globally optimal camera pose and the depth image.

[0114] In other embodiments, the maximum fusion method can be replaced by the Bayesian fusion method in the case of small calculation overhead requirement. When the embodiment performs semantic fusion based on the Bayesian fusion method, the classification results of the semantic information of each pixel output by the semantic segmentation network are re-divided, the first class and the second class with the highest confidence are taken as the new first class and the new second class, and the remaining classes are taken as the new third class. The three classes after re-dividing are used for semantic fusion.

[0115] Since the conventional Bayesian fusion method relies not only on the current frame to update the confidence, but also needs to save the semantic information of several consecutive frames, and then element-wise multiplies their confidences, and then normalizes to obtain the fused semantic map. In order to reduce the calculation overhead of the Bayesian fusion method, the embodiment re-divides the classes, only stores the semantic information of 3 classes, label1, label2 and others, and performs element-wise multiplication based on the three classes, and then normalizes the operation.

[0116] Generally, semantic segmentation will have dozens to hundreds of classes. If all are saved, each point needs to save dozens of classes and the probability of each class, and each class needs to be multiplied when multiplied, resulting in a large amount of calculation. The embodiment only saves 3 kinds of classes after optimization, occupies small memory space, and has small calculation amount when multiplied.

[0117] The Bayesian fusion method and the maximum fusion method mentioned in this embodiment are both based on the semantic segmentation (classification) result of the semantic segmentation network. The difference between the Bayesian fusion method and the maximum fusion method is that the maximum fusion method only uses the information of the previous and next two frames, while the Bayesian fusion method uses the information of several continuous frames, multiplies the confidence of each pixel of the semantic segmentation result of the several continuous frames, and then normalizes the obtained result. In the optimized Bayesian fusion method provided in this embodiment, the steps of confidence multiplication and normalization remain unchanged, but the number of stored categories is reduced from more than 100 categories to 3 categories, so the number of categories participating in multiplication is also reduced, and the calculation amount is reduced, thereby achieving optimization.

[0118] It should be noted that if the dynamic object segmentation network does not segment out the dynamic object region when performing dynamic object segmentation on a certain event frame in step 4, the feature points of the current frame in the first feature point set do not need to be removed, and the feature matching in step 5 can be directly performed based on the feature points of the current frame in the first feature point set.

[0119] In another embodiment, a real-time dynamic semantic mapping system based on multi-sensor fusion is provided, comprising a processor and a memory storing a plurality of computer instructions, which are executed by the processor to implement the steps of the real-time dynamic semantic mapping method based on multi-sensor fusion.

[0120] For specific limitations of the real-time dynamic semantic mapping system based on multi-sensor fusion, refer to the limitations of the real-time dynamic semantic mapping method based on multi-sensor fusion in the above, which will not be repeated here.

[0121] The memory and the processor are directly or indirectly electrically connected to realize the transmission or interaction of data. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The memory stores a computer program that can run on the processor, and the processor realizes the real-time dynamic semantic mapping method based on multi-sensor fusion in the embodiment of the application by running the computer program stored in the memory.

[0122] The memory can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and the like. The memory is configured to store a program. The processor is configured to execute the program after receiving an execution instruction.

[0123] The processor can be an integrated circuit chip with processing capability. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), and the like. The processor can implement or execute the methods, steps, and logical block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0124] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above-described embodiments are not described, but it should be considered that any combination of the technical features is within the scope of the present disclosure, as long as the combination does not result in contradictions.

[0125] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be considered as a limitation on the scope of the present application. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A real-time dynamic semantic mapping method based on multi-sensor fusion, characterized in that, The real-time dynamic semantic mapping method based on multi-sensor fusion includes: Step 1: Train the semantic segmentation network and the dynamic object segmentation network, including: The semantic segmentation network is first trained on a public dataset. After training, it is fine-tuned using color and depth images captured and labeled by a depth camera. The resulting network weights are then loaded into the semantic segmentation network to complete the training of the semantic segmentation network. The dynamic object segmentation network is first trained on a synthetic dataset. After training, it is fine-tuned using event frames captured and labeled by an event camera. The resulting network weights are then loaded into the dynamic object segmentation network to complete the training of the dynamic object segmentation network. Step 2: Obtain color images, depth images, and event frames of the target environment; match images based on timestamps to obtain color image sequences, depth image sequences, and event frame sequences. Step 3: Extract feature points from each frame of the color image sequence to obtain the first feature point set. Then, perform semantic segmentation on the color image sequence and depth image sequence frame by frame using the semantic segmentation network to obtain the semantic segmentation results for each frame. The semantic segmentation results include the semantic mask and confidence of the color image. Step 4: Perform dynamic object segmentation on the event frame sequence frame by frame using the dynamic object segmentation network to obtain the dynamic object region of each frame. Remove the feature points in the dynamic object region of each frame based on the first feature point set to obtain the second feature point set. Step 5: Perform feature matching between each frame in the second feature point set and the feature points of the previous frame, and combine the nonlinear optimization method to obtain the globally optimal camera pose sequence. Step 6: Based on the semantic segmentation results, the globally optimal camera pose sequence, and the depth image, complete the semantic octree mapping.

2. The real-time dynamic semantic mapping method based on multi-sensor fusion as described in claim 1, characterized in that, The event frame is obtained as follows: The event sequence is acquired based on an event camera. The event sequence is (x, y, p, t), where (x, y) are the event coordinates, p is the positive or negative polarity, and t represents the time information of the current event. Set a time window, and convert the event sequence into event frames according to the length of the time window. The converted event frames are divided into three channels, of which: Accumulate all events with positive p, and use the cumulative number of events at each (x,y) position as the pixel value at the (x,y) coordinate to obtain the first channel of the event frame; Accumulate all events with negative p polarity, and use the cumulative number of events at each (x,y) position as the pixel value at the (x,y) coordinate to obtain the second channel of the event frame; Using the average time plane as the third channel of the event frame, the cumulative number of events at each (x,y) position within the time window is: Then the average time surface is .

3. The real-time dynamic semantic mapping method based on multi-sensor fusion as described in claim 1, characterized in that, The step of constructing a semantic octree based on the semantic segmentation results, the globally optimal camera pose sequence, and the depth image includes: Take the semantic segmentation results, the globally optimal camera pose, and the depth image belonging to the same frame; The semantic segmentation results are mapped to a 3D coordinate system based on the globally optimal camera pose and depth image to obtain 3D coordinate points; Voxel filtering is applied to the three-dimensional coordinate points; Semantic fusion is performed on the three-dimensional coordinate points after voxel filtering to complete the semantic mapping for the current frame; The semantic segmentation result of the next frame, the globally optimal camera pose, and the depth image are then used to construct a semantic map, thus achieving real-time dynamic semantic mapping.

4. The real-time dynamic semantic mapping method based on multi-sensor fusion as described in claim 3, characterized in that, The step of mapping the semantic segmentation results to a three-dimensional coordinate system based on the globally optimal camera pose and depth image to obtain three-dimensional coordinate points includes: Take the two-dimensional coordinates P of a pixel in the semantic segmentation result as... ; Two-dimensional coordinate points are obtained from the depth image. The depth information at that location is Z; According to camera internal parameters Obtain two-dimensional coordinates Transformed to 3D coordinates in the camera coordinate system for Then, the three-dimensional coordinates in the world coordinate system are obtained through the global optimal camera pose (R,t) transformation. Where R is the rotation matrix, t is the translation vector, and f x f y c represents the focal length in the x and y directions, respectively. x c y These are the offsets of the optical axis relative to the coordinate center of the projection plane.

5. The real-time dynamic semantic mapping method based on multi-sensor fusion as described in claim 3, characterized in that, The voxel filtering of the three-dimensional coordinate points includes: Set the maximum search radius and mapping radius; Discard 3D coordinate points whose distance from the origin is greater than the maximum search radius; 3D coordinate points whose distance from the origin is less than or equal to the maximum search radius and greater than the mapping radius are mapped proportionally to the mapping radius, and the mapping is performed according to the direction from the current point to the origin; Voxel filtering is performed on the 3D coordinate points mapped to the mapping radius, and the 3D coordinate points retained after voxel filtering are mapped back to their original positions; The original 3D coordinate points located within the mapping radius and the 3D coordinate points retained after voxel filtering are taken as the final 3D coordinate points after voxel filtering.

6. The real-time dynamic semantic mapping method based on multi-sensor fusion as described in claim 3, characterized in that, The semantic fusion is either maximum value fusion or Bayesian fusion.

7. The real-time dynamic semantic mapping method based on multi-sensor fusion as described in claim 6, characterized in that, The semantic fusion method is the maximum value fusion method. Therefore, the semantic fusion of the three-dimensional coordinate points after voxel filtering includes: a. Determine the dynamic object region in the current image based on the segmentation result of the dynamic object segmentation network. Determine whether an object has been segmented in the dynamic object region based on the segmentation result of the semantic segmentation network. If no object has been segmented, mark the dynamic object region as static background and execute step d. If an object has been segmented, proceed to the next step. b. Calculate the intersection-union ratio (IUGR) between the dynamic object region and the object region segmented by the semantic segmentation network. If the IUGR is greater than or equal to the threshold, mark the union of the dynamic object region and the object region segmented by the semantic segmentation network as the dynamic target; otherwise, mark the dynamic object region as the dynamic target. c. Match the semantics of the 3D coordinates of the dynamic target with the semantics of the same 3D coordinates in the previous frame. When the 3D coordinates of the same position are first matched with a static background point, replace the semantics of the 3D coordinates with the semantics of the static background point. In subsequent matches, if the semantics of the same position are the same as those in the previous frame, the semantics remain unchanged, and the confidence level of the position is the average of the two. If the semantics of the same position are different from those in the previous frame, replace the semantics of the 3D coordinates with the semantics of the position with the one with the higher confidence level, and the confidence level is a preset multiple of the higher confidence level. d. Determine whether the judgment of all dynamic object regions has been completed. If not, return to step a; otherwise, complete semantic fusion.

8. The real-time dynamic semantic mapping method based on multi-sensor fusion as described in claim 6, characterized in that, The semantic fusion is a Bayesian fusion method, and the semantic fusion of the three-dimensional coordinate points after voxel filtering includes: The classification results of the semantic segmentation network, which classifies the semantic information of each pixel, are reclassified. The first and second categories with the highest confidence are taken as the new first and second categories, and the remaining categories are taken as the new third category. Semantic fusion is then performed based on the three reclassified categories.

9. A real-time dynamic semantic mapping system based on multi-sensor fusion, comprising a processor and a memory storing a plurality of computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the real-time dynamic semantic mapping method based on multi-sensor fusion as described in any one of claims 1 to 8.

Images