Bionic polarization semantic SLAM method based on neural radiation field

Abstract

The invention provides a bionic polarization semantic SLAM (Simultaneous Localization and Mapping) method based on a neural radiation field. A system comprises three parts, namely a bionic polarization pose estimation module based on factor graph optimization, a dense mapping module based on the neural radiation field and a semantic information positioning module based on bionic polarization infrared rays. The system firstly collects multi-sensor data, establishes a state equation and a measurement equation, and optimizes and estimates the pose by using a factor graph. Secondly, dense mapping is carried out by using the estimated pose and laser radar data and by adopting a neural implicit representation (NeRF-LOAM) model constructed by a laser radar odometer and a map through voxel embedding map generation and grid reconstruction. And finally, processing the polarization infrared fusion image through Yolov8 and SegNeXt, and realizing semantic positioning in combination with laser radar features. According to the system, the anti-interference performance and the environmental adaptability are remarkably enhanced, the mapping training time is shortened, and the system can be applied to multiple fields such as automatic driving, intelligent transportation and intelligent logistics.

Description

Technical Field

[0001] The invention belongs to the field of autonomous navigation technology and relates to a bionic polarization semantic SLAM method based on neural radiation field. Background Art

[0002] Neural radiance field is a technology that uses the differentiable characteristics of implicit representation of deep neural networks to optimize and render three-dimensional scenes. Semantic representation refers to the information-level perception of the surrounding environment through semantic segmentation technology. Bionic polarization navigation uses polarization image technology to extract feature points by introducing scene polarization information, effectively solving the problem of sparse feature extraction in low-texture environments. Simultaneous Localization and Mapping (SLAM) is the process of a moving body calculating its own position and building an environmental map based on sensor information.

[0003] SLAM technology has been widely used in aerospace, intelligent transportation, Internet of Things and other fields. Neural Radiance Fields (NeRF) research mainly focuses on the quality of 3D reconstructed maps, while traditional SLAM focuses on the accuracy of tracking trajectories, using filter theory to minimize the motion body posture and map landmarks to estimate the error covariance.

[0004] In recent years, graph optimization technology has developed rapidly, and SLAM methods based on graph optimization have emerged in an endless stream. These methods usually establish the position and posture transformation relationship between multiple nodes and nodes based on key frames, and maintain key nodes to ensure map capacity and accuracy, reducing the amount of calculation. Factor graph optimization algorithms are mainly used for trajectory tracking, and cannot effectively generate perception of the surrounding environment or perform fine dense mapping. Therefore, the introduction of neural radiation fields and semantic methods can effectively make up for the problems of sparse environmental perception and mapping. Although neural radiation fields can perform dense mapping, they require an effective training process, so using neural radiation fields alone cannot achieve the fastest trajectory tracking. Combining neural radiation fields with factor graph optimization can give full play to their respective advantages and complement each other.

[0005] Bionic polarization cameras can obtain polarization information of scenes for feature extraction. However, the research on feature extraction and navigation positioning using polarization information is still relatively limited. The research on combining visible light cameras with infrared cameras in low-light scenes is also advancing. Therefore, combining bionic polarization cameras with infrared cameras can effectively extract features from low-texture and low-light scenes. This fusion helps to improve the accuracy and robustness of the navigation system. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a bionic polarization semantic SLAM method based on neural radiation field, which uses laser radar to establish neural radiation field, uses YOLOv8 and SegNeXt to identify semantic information, and uses factor graph method to provide initial pose state for neural radiation field training, thereby accelerating the convergence speed of the model to achieve high-precision positioning and fast three-dimensional dense mapping. At the same time, the innovative introduction of bionic polarization camera and infrared camera for feature extraction in low-light and low-texture scenes enhances the robustness in dim scenes, underground environments and outdoor low-texture open scenes.

[0007] To achieve the above object, the present invention adopts the following technical solution:

[0008] A bionic polarization semantic SLAM method based on neural radiation field, the method comprising the following steps:

[0009] Step 1: Select the carrier attitude, speed, position and inertial measurement unit bias as the system state, establish the state model, collect lidar, inertial measurement unit, geomagnetic sensor, bionic polarization camera and infrared camera data, establish lidar odometer, bionic polarization infrared visual odometer, inertial measurement model and geomagnetic measurement model, state model, lidar odometer, and initialize the bionic polarization pose estimation module, dense mapping module, and semantic information positioning module;

[0010] Step 2: Input the laser radar, inertial measurement unit, geomagnetic sensor, bionic polarization camera and infrared camera data into the bionic polarization pose estimation module based on factor graph optimization, integrate the angular velocity measurement of the inertial measurement unit between two laser radar frames in the front end part to obtain the pitch angle, roll angle and heading angle, calculate the change of the heading angle of the carrier by using complementary filtering for the heading angle of the inertial measurement unit and the geomagnetic heading angle calculated by geomagnetism, correct the motion distortion of the laser radar model, and use the laser radar, inertial measurement unit, geomagnetic sensor, bionic polarization camera and infrared camera data and the feature image obtained by the semantic information positioning module based on bionic polarization infrared in the back end part to design the laser radar odometer factor, bionic polarization infrared visual inertial odometer factor, inertial measurement unit pre-integration factor, geomagnetic measurement factor and closed-loop detection factor and error function, establish a bionic polarization factor graph, solve the bionic polarization factor graph by iSAM2 algorithm, obtain the optimal estimate, and output the attitude, velocity, position and inertial measurement unit bias;

[0011] Step 3: The obtained optimal estimated carrier posture and position and the laser radar data collected in step 1 are input into a dense mapping module based on neural radiation field. The dense mapping module is established based on the neural implicit representation (NeRF-LOAM) model of laser radar odometer and map construction, including a neural mapping submodule and a grid reconstruction submodule. The neural mapping submodule uses the collected laser radar data and the optimal estimated carrier position and posture to train a neural implicit distance function model, generate dynamic voxels, output a voxel embedding map, and save key frames; the voxel embedding map and the key frames are passed into the grid reconstruction submodule in the NeRF-LOAM model, the key frames and the neural implicit distance function model are used to optimize the voxel embedding map, and the dense map is reconstructed by the marching cube method;

[0012] Step 4: Input the bionic polarization camera and infrared camera data collected in step 1 into the semantic information positioning module based on bionic polarization infrared to obtain semantic information positioning in the three-dimensional scene, wherein the semantic information positioning module includes an object recognition submodule based on Yolov8 and an image segmentation submodule based on SegNeXt. First, the polarization degree and polarization angle of each polarization image in step 1 are calculated to obtain a grayscale image, a polarization degree image and a polarization angle image, and a polarization RGB image is synthesized. The polarization RGB image and the infrared image are input into the object recognition submodule based on Yolov8 to identify the object name and the labeled color; the image segmentation submodule based on SegNeXt performs ORB feature extraction on the polarization RGB image and the infrared image to obtain polarization feature points and infrared feature points, associates the polarization image and the infrared image feature points, uses the feature encoding and decoding in the SegNeXt model to perform image segmentation, outputs the target boundaries in the polarization image and the infrared image, combines the target name and the labeled color, the target boundary and the visual depth map in the process of establishing the bionic polarization infrared visual inertial odometer factor in the bionic polarization pose estimation module, and obtains semantic information positioning;

[0013] Step 5: The position and posture output by the bionic polarization navigation factor graph are used as the position and posture of the carrier in the three-dimensional map; the dense map optimized by the grid reconstruction submodule in the dense mapping based on the neural radiation field is used as the framework of the three-dimensional map; the semantic positioning information output by the semantic information positioning module based on the fusion of bionic polarization and infrared features is projected onto the three-dimensional map, and the position, outline, category and color of each target are marked, and the fusion generates a bionic polarization navigation dense semantic map.

[0014] Preferably, in step 1, the carrier attitude, speed, position and sensor bias are selected as the state of the bionic polarization system, and a state model is established. In the world coordinate system (W system), the state node at time k is expressed as follows:

[0015] ,

[0016] for The state node of the integrated navigation system at time, T represents the transpose of the vector, express The rotation matrix at the moment, the superscript w represents the world coordinate system, represents the special orthogonal group of three-dimensional rotation matrices, express The position vector of the object in the world coordinate system at this moment, express The velocity of the object in the world coordinate system at the moment in the directions of x, y, and z axes. Indicates the sensor bias.

[0017] The carrier state model is:

[0018] ,

[0019] The state transfer function for:

[0020] ,

[0021] in, The rotation matrix between time k-1 and time k is expressed as follows:

[0022] ,

[0023] ,

[0024] ,

[0025] in, Indicates the time interval, represents the process noise, represents the rotation matrix at time k-1, Indicated in The amount of rotation change in time, represents the position at time k-1, represents the speed at time k-1, represents the carrier acceleration vector measured by the inertial measurement unit at time k-1, represents the inertial measurement unit bias at time k-1, represents the identity matrix, represents the Euclidean norm, represents the rotation vector, A skew-symmetric matrix representing the angular velocity vector, express The carrier angular velocity vector measured by the inertial measurement unit at the moment, , and They represent the x, y and z axis components of the carrier angular velocity vector measured by the inertial measurement unit at time k-1 respectively.

[0026] Step 1 describes the establishment of the laser radar odometer, bionic polarized infrared visual odometer, inertial measurement model and geomagnetic measurement model as follows:

[0027] The LiDAR odometry includes:

[0028] Divide the point cloud into edge feature points and surface feature points ,The residual model is established by matching the corresponding feature points in the feature point set from k-1 to k, and the Gauss-Newton method is used to solve the optimal transformation:

[0029] ,

[0030] in, express arrive The distance between the edge feature point and its corresponding matching point at the moment, is the distance between the surface feature point and its corresponding matching point. The feature point classification standard is based on the concave-convex change rate of the scanned surface. Calculation status and The relative transformation between ,in It means that the laser radar is The posture of the moment, represents the k moment The edge feature points in represents the k moment Surface feature points in ;

[0031] Bionic polarization infrared visual odometer includes:

[0032] The polarized RGB image provided by the semantic information positioning module and the feature point map extracted from the infrared image are associated with the dense depth map stitched from multiple frames of lidar point clouds to obtain a visual depth map. The KLT pyramid optical flow is then used to track adjacent frames to obtain the relative transformation between adjacent frames assisted by the visual depth map. , in It shows the bionic polarization infrared camera The position of the moment.

[0033] Inertial Measurement Unit Measurement Model:

[0034] ,

[0035] ,

[0036] is the accelerometer measurement at time k, is the true value of the accelerometer at time k, is the angular velocity measurement of the gyroscope at time k, is the true value of the gyroscope angular velocity at time k, is the accelerometer zero bias at time k, is the gyro zero bias at time k, and are the noise of the accelerometer and gyroscope at time k;

[0037] Geomagnetic measurement model:

[0038] ,

[0039] ,

[0040] is the measurement function of the geomagnetic sensor, represents the magnetic heading angle, is the noise of the geomagnetic sensor, express The magnetic induction intensity output by the magnetic sensor at this moment, They represent the magnetic induction intensity in the x, y, and z directions in the carrier coordinate system, respectively. represents the roll angle calculated in the geomagnetic coordinate system, Represents the pitch angle calculated in the geomagnetic coordinate system;

[0041] The coordinate systems of the laser radar, inertial measurement unit, geomagnetic sensor, bionic polarization camera and infrared camera in the carrier coordinate system are converted to the world coordinate system, and the bionic polarization pose estimation module, dense mapping module and semantic information positioning module are initialized.

[0042] Preferably, step 2 integrates the angular velocity measurement of the inertial measurement unit between two frames of lidar to obtain the pitch angle, roll angle and heading angle, and uses complementary filtering to calculate the change in the heading angle of the carrier for the heading angle of the inertial measurement unit and the geomagnetic heading angle obtained by geomagnetic solution to correct the motion distortion of the lidar model.

[0043] The complementary filtering technology of heading angle fusion is calculated as follows:

[0044] ,

[0045] ,

[0046] represents the weight, represents the variance of the inertial measurement unit, represents the variance of the geomagnetic field, yes arrive The heading angle change obtained by integrating the inertial measurement unit and geomagnetic field at a certain time. express arrive The heading angle change obtained by integrating the angular velocity of the inertial measurement unit over time, express arrive The change of geomagnetic heading angle over time.

[0047] Secondly, the laser radar odometer factor, bionic polarized infrared visual inertial odometer factor, inertial measurement unit pre-integration factor, geomagnetic heading vector measurement factor, and closed-loop detection factor are established. The factors are established as follows:

[0048] LiDAR odometry factor:

[0049] The relative transformation between two LiDAR frames can be obtained from the LiDAR odometer In order to reduce the computational complexity, a sliding window is introduced ,in is the sensor state at the kth moment, including the position, attitude, velocity and sensor bias in the world coordinate system, n is the total number of key frames, m is the total number of features in the sliding window, It is the first time that the first The inverse depth of features is calculated using the LiDAR odometer to calculate the sliding window The relative transformation of the i-th lidar keyframe in and the previous keyframe is used as a constraint between the two frames to establish the lidar odometry factor.

[0050] Bionic polarized infrared visual inertial odometry factor:

[0051] Relative conversion via biomimetic polarization infrared visual odometry Computing the priors of visual residuals Define the visual inertial odometry residual model and obtain middle The optimal conversion of:

[0052] ,

[0053] in, represents the Huber norm, represents the marginal prior residual, The Hessian matrix representing the edge prior information, represents the prior of visual residual, represents the prior of the inertial residual, represents the covariance matrix of the inertial measurement unit, represents the predicted feature measurement value on the unit sphere after converting the first observation in the i-th key frame to the j-th key frame, represents the feature set observed at least twice in the current sliding window, represents the set of inertial measurement unit measurements, Indicates the image observed in the i-th key frame. The residual of the feature and the j-th key frame observing the feature, Represents the pre-integrated value of the inertial measurement unit from time k-1 to time k;

[0054] Using gravity inertial measurement, the pitch angle and roll angle can be observed in the visual inertial odometer, so the bionic polarized infrared visual inertial odometer factor only establishes 4 degrees of freedom constraints, including position and heading angle. Keyframe in the sliding window Added to the factor graph with the previous keyframe in the sliding window Create edge nodes:

[0055] ,

[0056] ,

[0057] in, represents the estimated value of the position of the i-th key frame in the world coordinate system, represents the estimated value of the jth key frame position in the world coordinate system, Represents the rotation matrix from the world coordinate system to the camera coordinate system, represents the position transformation between the i-th key frame and the j-th key frame, represents the estimated heading angle of the i-th key frame, represents the estimated heading angle of the jth key frame, Represents the heading angle transformation between the i-th key frame and the j-th key frame;

[0058] The marginal residual of the bionic polarized infrared visual inertial odometry factor is defined as:

[0059] ,

[0060] in, Represents the position of the i-th key frame in the world coordinate system, Represents the position of the jth key frame in the world coordinate system and the rolling angle in the visual odometer coordinate system and pitch angle is a fixed estimate obtained from a monocular visual-inertial odometry, Represents the position in the world coordinate system, represents the heading angle of the i-th key frame, represents the heading angle of the jth key frame, Represent the transformation matrix from the world coordinate system to the visual coordinate system, calculate the visual motion transformation, use the motion transformation as an inter-frame constraint, and establish the bionic polarized infrared visual inertial odometry factor;

[0061] Inertial measurement unit pre-integration factor:

[0062] Using the measurement results of the inertial measurement unit, the carrier is and The speed between ,Location and rotation The calculation is as follows:

[0063] ,

[0064] ,

[0065] ,

[0066] = represents the gravitational acceleration obtained by the accelerometer of the inertial measurement unit, where , and They represent the components of the gravity vector on the x, y and z axes in the b system respectively.

[0067] The time is obtained by the inertial measurement unit pre-integration method. and The speed change between , position change and posture change , and then , and Substituting into the pre-integration measurement model, we have:

[0068] ,

[0069] ,

[0070] ,

[0071] in, = represents the gravitational acceleration obtained by the accelerometer of the inertial measurement unit, where , and They represent the components of the gravity vector on the x, y and z axes in the b system respectively. represents the pre-integrated rotation measurement noise, represents the speed measurement noise pre-integrated from k-1 to k, represents the position measurement noise from the pre-integrated k-1 to k time, It indicates the change of posture obtained by pre-integration calculation from time k-1 to time k. It indicates the speed change obtained by pre-integration calculation from time k-1 to k. It represents the position change obtained from the pre-integration calculation k-1 to k. The error function of the pre-integration factor of the inertial measurement unit is:

[0072] ,

[0073] in, and They represent the error of attitude change, position change, and speed change respectively:

[0074] ,

[0075] ,

[0076] ,

[0077] Geomagnetic heading measurement factor:

[0078] Calculated by the accelerometer of the inertial measurement unit , solve for the roll angle and pitch angle In the b system, the output of the magnetic sensor is , calculate the magnetic heading angle The formula is as follows:

[0079] ,

[0080] ,

[0081] The magnetic heading angle is obtained by calculation, and the geomagnetic heading vector measurement factor is established. The error function of the geomagnetic heading vector measurement factor is:

[0082] ,

[0083] Closed loop detection factor:

[0084] The loop closure detection factor uses a loop closure detection method based on Euler distance, assuming that the new state is added to the factor graph, and the graph is first searched for the closest The previous state of is a candidate node returned. Then the scan matching method is used to try to match the key frame of the state With the previous child keyframe , after the match is successful, it is converted to the w coordinate system to obtain the relative transformation , and are the rotation and displacement calculated by scan matching, The subscript indicates the matched keyframe and the current frame , adding the transformation as a loop closure factor to the factor graph, It is a custom loop search distance setting. The error function of the loop detection factor is:

[0085] ,

[0086] represents the translation error, Represents the rotation error.

[0087] Based on the factors established above, a bionic polarization navigation factor graph is constructed, and an error function containing position, velocity, attitude, and sensor bias is defined:

[0088] ,

[0089] in, That is, the state node of the integrated navigation system at the kth moment The best estimate of is the covariance of the lidar noise, represents the inter-frame constraints of the lidar odometry, represents the relative transformation between the current frame and the previous keyframe i, is the noise covariance of the IMU accelerometer and gyroscope, is the noise covariance of the geomagnetic sensor, represents the inter-frame constraint between the jth keyframe and the previous i-th keyframe in the sliding window of the visual odometry, Representation of sliding window in bionic polarized infrared visual inertial odometry The set of all edge nodes in the robot is constructed, and the iSAM2 algorithm is used to solve the bionic polarization navigation factor graph to obtain the optimal estimate and output the attitude, velocity, position and inertial measurement unit bias.

[0090] Preferably, in step 3, the position and posture obtained in step 2 and the lidar data collected in step 1 are input into the neural implicit representation of lidar odometer and map construction (NeRF-LOAM) model to construct a dense mapping module based on neural radiation field. The lidar data and the position and posture obtained in step 2 are input into the neural mapping submodule in the NeRF-LOAM model to train the neural implicit distance function model. , which is expressed as follows:

[0091] ,

[0092] in, , represents the posture of aligning the i-th key frame interpolation output of the bionic polarization navigation factor map with the current scanning frame of the lidar, represents the original point in lidar coordinates, Represents the sampling point transformed to the world coordinate system, Indicates sampling point quilt The trilinear interpolation of neighboring voxels is surrounded by embeddings, Indicates that the parameter is The neural implicit network is used to obtain dynamic voxel generation through the neural implicit distance function model, establish a voxel embedding map, set a threshold, generate a lidar scanning keyframe when the time interval, displacement distance or rotation angle exceeds the threshold, and save the keyframe to the keyframe buffer in the mesh reconstruction submodule.

[0093] At the same time, due to the use of the position and attitude output by the bionic polarization navigation factor graph, the position and attitude have been corrected for laser radar motion distortion, reducing the increase in training time caused by the difficulty of laser radar z-axis constraint, and effectively accelerating the convergence of the model. Output voxel grid map and keyframe. The voxel grid map divides the scene using an octree-based method. The position and attitude of the receiving factor graph are converted to the world coordinate system. Points that are not in the existing voxels are assigned to the newly generated voxels. These voxels are added to the octree together with their corresponding voxel features. In order to quickly locate the required voxel features, the three-dimensional voxel coordinates are encoded into a unique scalar value, thereby using an efficient and scalable method for dynamically generating voxel features.

[0094] As described in step 3, the voxel embedding map and key frames are passed into the grid reconstruction submodule in the NeRF-LOAM model, the voxel embedding map is optimized using the key frames and the neural implicit distance function model, and the dense map is reconstructed using the marching cube method.

[0095] Preferably, the object recognition submodule based on Yolov8 and the image segmentation submodule based on SegNeXt described in step 4 establish a semantic information positioning module, firstly calculate the polarization degree and polarization angle of each polarization image in step 1 to obtain the grayscale image, polarization degree image and polarization angle image, and synthesize the polarization RGB image. Taking a four-channel sensor as an example, its light intensity output is , , , , the calculation method is as follows:

[0096] Using the Stokes vector Calculate the degree of polarization With polarization angle :

[0097] ,

[0098] in, and , and The difference is 90°. and , and The difference is 45°, so the degree of polarization and polarization angle can be calculated:

[0099] ,

[0100] ,

[0101] Calculate the grayscale image according to the polarization degree and polarization angle , polarization degree diagram , Polarization angle diagram :

[0102] (1) Calculate the grayscale image. Each unit of the original image calculates a new pixel. The grayscale value g of this pixel can be obtained as follows, where g is an integer:

[0103] ,

[0104] , , , Respectively represent the light transmission Directional polarizer produces grayscale values.

[0105] (2) Calculate the polarization degree map. A new pixel is calculated for each unit of the original image, and the polarization degree of this pixel is calculated by the following formula:

[0106] ,

[0107] The gray value of each pixel is mapped by the polarization degree Value instead.

[0108] (3) Calculate the polarization angle map. Each unit of the original image calculates a new pixel, and the polarization angle of this pixel is calculated by the following formula: Linear mapping to integers between 0 and 255:

[0109] ,

[0110] The gray value of each pixel is mapped by the polarization angle Value instead.

[0111] The grayscale image, polarization degree image, and polarization angle image are synthesized into a polarization RGB image. The Red channel of this image is the polarization degree image, the Green channel is the grayscale image, and the Blue channel is the polarization angle image. The polarization RGB image and the infrared image are passed into the Yolov8 model for object recognition, and the object label and color information are saved.

[0112] In step 4, the SegNeXt-based image segmentation submodule performs ORB feature extraction on the polarized RGB image and the infrared image to obtain polarized feature points and infrared feature points, and inputs the polarized RGB image and the infrared image into the SegNeXt-based semantic segmentation submodule, extracts ORB features on the image, extracts features of the texture environment through the polarized RGB image, and extracts features of the lighting environment through the infrared image. Associate the feature points of the polarized image and the infrared image, use the feature encoding and decoding in the SegNeXt model to perform image segmentation, and output the target boundaries in the polarized image and the infrared image. The encoder adopts a new multi-scale convolutional attention, which includes three parts: deep separable convolution that aggregates local information, multi-branch deep separable convolution that captures multi-scale context, and 1x1 convolution that models the relationship between different channels. The decoder uses a lightweight Hambuger to further model the global context. Combine the target name and annotation color, target boundary, and the visual depth map in the process of establishing the bionic polarized infrared visual inertial odometer factor in the bionic polarized pose estimation module to obtain semantic information positioning;

[0113] Preferably, in step 5, the position and posture output by the bionic polarization navigation factor graph are used as the position and posture of the carrier in the three-dimensional map; the dense map optimized by the grid reconstruction submodule in the dense mapping based on the neural radiation field is used as the framework of the three-dimensional map; the semantic positioning information output by the semantic information positioning module based on the fusion of bionic polarization and infrared features is projected onto the three-dimensional map, and the position, outline, category and color of each target are marked, and the fusion generates a bionic polarization navigation dense semantic map.

[0114] The beneficial effects of the present invention are:

[0115] The present invention uses laser radar to establish a neural radiation field, uses YOLOv8 and SegNeXt to identify semantic information, and uses the factor graph method to provide the initial pose state for the training of the neural radiation field, thereby accelerating the convergence of the model. At the same time, the innovative introduction of bionic polarization cameras and infrared cameras to extract features in low-light and low-texture scenes provides more stable feature information for the visual part of the vision / radar / inertial / geomagnetic combined navigation, and enhances the robustness in dim scenes, underground environments, and outdoor low-texture open scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0116] Figure 1 This is a flow chart of a bionic polarization semantic SLAM method based on neural radiation field in the present invention. DETAILED DESCRIPTION

[0117] The present invention will be described in detail below in conjunction with the accompanying drawings, which serve as a part of this specification and illustrate the principles of the present invention through implementation. Other aspects, features and advantages of the present invention will become apparent from the detailed description.

[0118] The present invention relates to a bionic polarization semantic SLAM method based on neural radiation field. The method selects the position, speed, attitude and sensor bias in the world coordinate system as the state vector, establishes a laser radar odometer, a bionic polarization infrared visual odometer, an inertial measurement unit measurement model and a geomagnetic sensor measurement model, collects multi-sensor data, and establishes a bionic polarization system state equation. In the front end, the inertial measurement unit measurement and the geomagnetic sensor measurement are used to calculate the angle change by using complementary filtering fusion, correct the laser radar model motion distortion, and establish a laser radar odometer and a bionic polarization infrared visual odometer. In the back-end part, the inertial measurement unit pre-integration factor is established through the inertial measurement unit data and the pre-integration method; the geomagnetic heading vector measurement factor is established using the gravity vector and geomagnetic measurement; the dense depth map generated by the splicing of multi-frame lidar point clouds is associated with the polarization RGB map provided by the semantic information positioning module and the feature map generated by the infrared image extraction feature fusion to obtain the visual depth map and motion measurement, and the joint residual model is established by minimizing the visual residual and the inertial measurement unit residual, and the bionic polarized infrared visual inertial odometer factor is established; the corrected lidar data is scanned and matched between the key frame and the voxel map, the relative transformation is calculated, and the lidar odometer factor is established; the loopback factor is obtained through Euclidean space scan matching. Based on the above factors, the bionic polarization navigation factor map is constructed, the position, velocity, attitude and sensor bias error functions are defined, the iSAM2 optimization algorithm is used to realize pose estimation, and the position, velocity, attitude and sensor zero bias are updated.

[0119] Using the pose, attitude and raw data of the laser radar output by the bionic polarization factor map, and the neural implicit representation (NeRF-LOAM) model constructed by the laser radar odometer and map, a dense mapping module based on the neural radiation field is constructed. The pose results and laser radar data are passed to the neural mapping submodule in NeRF-LOAM to accelerate the convergence of the model, output voxel grid maps and key frames, and save the key frames to the key frame buffer for grid reconstruction in NeRF-LOAM. The key frames are selected and optimized in the grid reconstruction submodule, and finally the dense map is established using the marching cube method.

[0120] The semantic information positioning system based on the bionic polarization camera and infrared camera receives the image data of the bionic polarization camera and the infrared camera, calculates the polarization degree and polarization angle of each polarization image to obtain the grayscale image, polarization degree image and polarization angle image, and synthesizes the polarization RGB image, and inputs the polarization RGB image and infrared image into YOLOv8 to identify the object mark and color in the scene; the semantic segmentation operation is performed through the semantic segmentation convolutional network architecture SegNeXt, firstly ORB feature extraction is performed on the polarization RGB image and infrared image, and the features are output to the encoder and decoder of SegNeXt for image segmentation, and the boundary of the target in the image is output. The carrier position and posture output by the bionic polarization factor map, the distance between the carrier and the target provided by the visual depth map, the boundary of the target provided by the image segmentation submodule based on SegNeXt, and the target and the labeled color identified by the object recognition submodule based on Yolov8 are used to obtain semantic information positioning.

[0121] Finally, the position and posture output by the bionic polarization navigation factor map are used as the position and posture of the carrier in the three-dimensional map; the dense map optimized by the grid reconstruction submodule in the dense mapping based on the neural radiation field is used as the framework of the three-dimensional map; the semantic positioning information output by the semantic information positioning module based on the fusion of bionic polarization and infrared features is projected onto the three-dimensional map, and the position, outline, category and color of each target are marked, and finally a bionic polarization navigation dense semantic map is generated.

[0122] Please refer to the present invention Figure 1 A flow chart of a bionic polarization semantic SLAM method based on neural radiation field is shown. The specific technical solution includes the following steps:

[0123] Step 1: Select the carrier attitude, speed, position and inertial measurement unit bias as the system state, establish the state model, collect lidar, inertial measurement unit, geomagnetic sensor, bionic polarization camera and infrared camera data, establish lidar odometer, bionic polarization infrared visual odometer, inertial measurement model and geomagnetic measurement model, state model, lidar odometer, and initialize the bionic polarization pose estimation module, dense mapping module, and semantic information positioning module;

[0124] Step 2: Input the laser radar, inertial measurement unit, geomagnetic sensor, bionic polarization camera and infrared camera data into the bionic polarization pose estimation module based on factor graph optimization, integrate the angular velocity measurement of the inertial measurement unit between two laser radar frames in the front end part to obtain the pitch angle, roll angle and heading angle, calculate the change of the heading angle of the carrier by using complementary filtering for the heading angle of the inertial measurement unit and the geomagnetic heading angle calculated by geomagnetism, correct the motion distortion of the laser radar model, and use the laser radar, inertial measurement unit, geomagnetic sensor, bionic polarization camera and infrared camera data and the feature image obtained by the semantic information positioning module based on bionic polarization infrared in the back end part to design the laser radar odometer factor, bionic polarization infrared visual inertial odometer factor, inertial measurement unit pre-integration factor, geomagnetic measurement factor and closed-loop detection factor and error function, establish a bionic polarization factor graph, solve the bionic polarization factor graph by iSAM2 algorithm, obtain the optimal estimate, and output the attitude, velocity, position and inertial measurement unit bias;

[0125] Step 3: The obtained optimal estimated carrier posture and position and the laser radar data collected in step 1 are input into a dense mapping module based on neural radiation field. The dense mapping module is established based on the neural implicit representation (NeRF-LOAM) model of laser radar odometer and map construction, including a neural mapping submodule and a grid reconstruction submodule. The neural mapping submodule uses the collected laser radar data and the optimal estimated carrier position and posture to train a neural implicit distance function model, generate dynamic voxels, output a voxel embedding map, and save key frames; the voxel embedding map and the key frames are passed into the grid reconstruction submodule in the NeRF-LOAM model, the key frames and the neural implicit distance function model are used to optimize the voxel embedding map, and the dense map is reconstructed by the marching cube method;

[0126] Step 4: Input the bionic polarization camera and infrared camera data collected in step 1 into the semantic information positioning module based on bionic polarization infrared to obtain semantic information positioning in the three-dimensional scene, wherein the semantic information positioning module includes an object recognition submodule based on Yolov8 and an image segmentation submodule based on SegNeXt. First, the polarization degree and polarization angle of each polarization image in step 1 are calculated to obtain a grayscale image, a polarization degree image and a polarization angle image, and a polarization RGB image is synthesized. The polarization RGB image and the infrared image are input into the object recognition submodule based on Yolov8 to identify the object name and the labeled color; the image segmentation submodule based on SegNeXt performs ORB feature extraction on the polarization RGB image and the infrared image to obtain polarization feature points and infrared feature points, associates the polarization image and the infrared image feature points, uses the feature encoding and decoding in the SegNeXt model to perform image segmentation, outputs the target boundaries in the polarization image and the infrared image, combines the target name and the labeled color, the target boundary and the visual depth map in the process of establishing the bionic polarization infrared visual inertial odometer factor in the bionic polarization pose estimation module, and obtains semantic information positioning;

[0127] Step 5: The position and posture output by the bionic polarization navigation factor graph are used as the position and posture of the carrier in the three-dimensional map; the dense map optimized by the grid reconstruction submodule in the dense mapping based on the neural radiation field is used as the framework of the three-dimensional map; the semantic positioning information output by the semantic information positioning module based on the fusion of bionic polarization and infrared features is projected onto the three-dimensional map, and the position, outline, category and color of each target are marked, and the fusion generates a bionic polarization navigation dense semantic map.

[0128] Furthermore, in step 1, the carrier posture, speed, position and sensor bias are selected as the state of the bionic navigation system, and a state model is established. In the world coordinate system (w system), the state node at time k is expressed as follows:

[0129] ,

[0130] express The state node of the integrated navigation system at time, T represents the transpose of the vector, express The rotation matrix at the moment, the superscript w represents the world coordinate system, represents the special orthogonal group of three-dimensional rotation matrices, express The position vector of the object in the world coordinate system at this moment, express The velocity vector of the object pointing to the x, y, and z axes in the world coordinate system at that moment, Represents the inertial measurement unit sensor bias;

[0131] The carrier state model is:

[0132] ,

[0133] is the state transfer function, represents process noise.

[0134] Step 1: Establish the carrier state model , LiDAR odometer, bionic polarized infrared visual odometer, inertial measurement model acceleration measurement Angular velocity measurement and geomagnetic measurement models Each sensor measurement model. Collect the data of each sensor, and the bionic polarization system is initialized according to the rotation matrix from the carrier coordinate system (b system) to the world coordinate system (w system) , convert the coordinate systems of the laser radar, inertial measurement unit, geomagnetic sensor, bionic polarization camera and infrared camera in the carrier coordinate system to the world coordinate system, and initialize the bionic polarization pose estimation module, dense mapping module and semantic information positioning module.

[0135] Furthermore, in step 2, the angular velocity measurement of the inertial measurement unit between the two laser radar frames is integrated to obtain the pitch angle, roll angle and heading angle, and the heading angle of the inertial measurement unit and the heading angle calculated by the geomagnetic solution are calculated using complementary filtering to calculate the change in the heading angle of the carrier. , correct the motion distortion of the laser radar model. Establish factors including the laser radar odometer factor, bionic polarized infrared visual inertial odometer factor, inertial measurement unit pre-integration factor, geomagnetic heading vector measurement factor, and closed-loop detection factor. The factors are established as follows:

[0136] The relative transformation between two LiDAR frames can be obtained from the LiDAR odometer . Calculate the sliding window Lidar keyframes within Compute relative transformation with previous keyframe to establish lidar odometry factor.

[0137] The bionic polarized infrared visual inertial odometry factor is optimized using a lidar-assisted visual inertial map, and the bionic polarized infrared visual odometry is calculated by splicing multiple frames of lidar point clouds into a dense depth map. Priors with visual residuals , define the residual model of visual inertial odometry, obtain the maximum a posteriori estimate to calculate the visual motion transformation, and calculate it in the sliding window The edge node between the current keyframe and the previous keyframe defines the edge residual of the bionic polarized infrared visual inertial odometry factor ,Establish bionic polarized infrared visual inertial odometry factor.

[0138] The inertial measurement unit pre-integration factor obtains the velocity change between time k-1 and k through the inertial measurement unit pre-integration method. , position change and posture change . Define the error function of the inertial measurement unit pre-integration factor .

[0139] The geomagnetic heading measurement factor is calculated by the accelerometer of the inertial measurement unit , solve for the roll angle and pitch angle In the carrier coordinate system, the output of the magnetic sensor is , calculate the magnetic heading angle Establish geomagnetic heading measurement factor and define the error function of geomagnetic heading measurement factor .

[0140] The loop closure detection factor uses a loop closure detection method based on Euler distance to obtain a new state For example, is added to the factor graph, and the graph is first searched for the closest The previous state of is a candidate node returned. Then the scan matching method is used to try to match the key frame of the state With the previous child keyframe , after the match is successful, it is converted to the w coordinate system to obtain the relative transformation , and then add the transformation as a loop factor to the factor graph, Sets the custom loop search distance. Defines the error function of the loop detection factor .

[0141] Finally, based on the above established factors, the bionic polarization navigation factor map is constructed, and the error function containing position, velocity, attitude and sensor bias is defined. , and use the iSAM2 algorithm to solve the bionic polarization navigation factor graph, obtain the optimal estimate, and output the attitude, velocity, position and inertial measurement unit bias.

[0142] Furthermore, in step 3, the obtained optimal estimated position, posture and collected lidar data are input into a dense mapping module based on neural radiation field, and the neural mapping submodule uses the collected lidar data and the optimal estimated carrier position and posture to train a neural implicit distance function model. . At the same time, due to the use of the posture information output by the bionic polarization navigation factor map, the posture has been corrected for laser radar motion distortion, reducing the increase in training time caused by the difficulty of laser radar z-axis constraint, and effectively accelerating the convergence of the model. Output the voxel embedding map and save the key frame; pass the voxel embedding map and key frame into the grid reconstruction submodule in the NeRF-LOAM model, use the key frame and the neural implicit distance function model to optimize the voxel embedding map, and complete the reconstruction of the dense map through the moving cube method;

[0143] Furthermore, in step 4, the object recognition submodule based on Yolov8 and the image segmentation submodule based on SegNeXt are used to establish a semantic information positioning module. First, the polarization degree and polarization angle are calculated for each polarization image in step 1 to obtain a grayscale image, a polarization degree image, and a polarization angle image, and then a polarization RGB image is synthesized. Taking a four-channel sensor as an example, its light intensity output is , , , Calculate the Stokes vector from the light intensity output , and then use the Stokes vector Calculate the degree of polarization With polarization angle ,in and , and The difference is 90°. and , and The difference is 45°.

[0144] Calculate the grayscale image according to the polarization degree and polarization angle , polarization degree diagram , Polarization angle diagram The grayscale image, polarization degree image, and polarization angle image are synthesized into a polarization RGB image, and the polarization RGB image and infrared image are input into the object recognition submodule based on Yolov8 to identify the object name and mark the color.

[0145] The polarized RGB image and infrared image are passed to the semantic segmentation submodule based on SegNeXt. First, ORB feature extraction is performed on the image. The features of the texture environment are extracted from the polarized RGB image, and the features of the lighting environment are extracted from the infrared image. The feature points of the polarized image and the infrared image are associated, and the image segmentation is performed using the feature encoding and decoding in the SegNeXt model. The target boundaries in the polarized image and the infrared image are output.

[0146] The bionic polarization factor map provides the carrier position and posture, the depth map provides the distance between the carrier and the target, the SegNeXt-based image segmentation submodule provides the target boundary, and the Yolov8-based object recognition submodule identifies the target and annotates the color to obtain semantic information positioning.

[0147] Furthermore, the position and posture output by the fused bionic polarization navigation factor map described in step 5 are used as the position and posture of the carrier in the three-dimensional map; the dense map optimized by the grid reconstruction submodule in the dense mapping based on the neural radiation field is used as the framework of the three-dimensional map; the semantic positioning information output by the semantic information positioning module fused with bionic polarization and infrared features is projected onto the three-dimensional map, and the position, outline, category and color of each target are marked, so as to finally generate a bionic polarization navigation dense semantic map.

[0148] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A bionic polarization semantic SLAM method based on neural radiation field, characterized in that: The method comprises the following steps: Step 1: Select the carrier attitude, speed, position and inertial measurement unit bias as the system state, establish the state model, collect lidar, inertial measurement unit, geomagnetic sensor, bionic polarization camera and infrared camera data, establish lidar odometer, inertial measurement model, geomagnetic measurement model, bionic polarization infrared visual odometer, and initialize the bionic polarization pose estimation module, dense mapping module, and semantic information positioning module; Step 2: Input the laser radar, inertial measurement unit, geomagnetic sensor, bionic polarization camera and infrared camera data into the bionic polarization pose estimation module based on factor graph optimization, use the inertial measurement unit and geomagnetic sensor to correct the motion distortion of the laser radar in the front end, and use the laser radar, inertial measurement unit, geomagnetic sensor, bionic polarization camera and infrared camera data and the feature image obtained by the semantic information positioning module based on bionic polarization infrared in the back end to design the laser radar odometer factor, bionic polarization infrared visual inertial odometer factor, inertial measurement unit pre-integration factor, geomagnetic heading factor and closed-loop detection factor and error function, establish the bionic polarization navigation factor graph, and use iSAM2 optimization to obtain the optimal estimation of the carrier attitude, speed, position and inertial measurement unit bias; Step 3: The obtained optimal estimated carrier posture and position and the laser radar data collected in step 1 are input into a dense mapping module based on neural radiation field. The dense mapping module is established based on the neural implicit representation (NeRF-LOAM) model of laser radar odometer and map construction, including a neural mapping submodule and a grid reconstruction submodule. The neural mapping submodule uses the collected laser radar data and the optimal estimated carrier position and posture to train the neural implicit distance function, establish a voxel embedding map and save key frames; the grid reconstruction submodule uses the key frames and the trained neural implicit distance function to optimize the voxel embedding map, and completes the reconstruction of the dense map through the marching cube method; Step 4: Input the bionic polarization camera and infrared camera data collected in step 1 into the semantic information positioning module based on bionic polarization infrared to obtain semantic information positioning in the three-dimensional scene, wherein the semantic information positioning module includes an object recognition submodule based on Yolov8 and an image segmentation submodule based on SegNeXt, the object recognition submodule is used to recognize the target name and annotation color in the image, and the image segmentation submodule is used to segment the target boundary in the image, and the target name and annotation color, the target boundary and the visual depth map in the process of establishing the bionic polarization infrared visual inertial odometer factor in the bionic polarization pose estimation module are combined to obtain semantic information positioning; Step 5: Use the optimal estimated carrier posture and position as the carrier's posture, use the reconstructed dense map as the framework of the overall three-dimensional map, locate based on the obtained semantic information in the three-dimensional scene, project the two-dimensional image into the three-dimensional map, obtain the category and color of each object in the two-dimensional image, and generate a bionic polarization dense semantic map.

2. The bionic polarization semantic SLAM method based on neural radiation field according to claim 1, characterized in that: In step 1, the state model is represented as follows in the world coordinate system, and the state node at time k is: , for The state node of the integrated navigation system at time, T represents the transpose of the vector, express The rotation matrix at the moment, the superscript w represents the world coordinate system, represents the special orthogonal group of three-dimensional rotation matrices, express The position vector of the object in the world coordinate system at this moment, express The velocity of the object in the world coordinate system at the moment in the directions of x, y, and z axes. Indicates sensor bias; The carrier state model is: , represents the state transition function, express -1 time process noise; The coordinate systems of the laser radar, inertial measurement unit, geomagnetic sensor, bionic polarization camera and infrared camera in the carrier coordinate system are converted to the world coordinate system, and the bionic polarization pose estimation module, dense mapping module and semantic information positioning module are initialized.

3. The bionic polarization semantic SLAM method based on neural radiation field according to claim 1 is characterized in that: The step 2 includes integrating the angular velocity measurement of the inertial measurement unit between two frames of the laser radar at the front end to obtain the pitch angle, roll angle and heading angle of the inertial measurement unit, and calculating the heading angle change of the carrier based on the complementary filtering of the heading angle of the inertial measurement unit and the heading angle calculated by the geomagnetic solution to correct the motion distortion of the laser radar.

4. The bionic polarization semantic SLAM method based on neural radiation field according to claim 1, characterized in that: In the step 2, a laser radar odometer factor, a bionic polarization infrared visual inertial odometer factor, an inertial measurement unit pre-integration factor, a geomagnetic heading factor, and a closed-loop detection factor and an error function are established in the back-end part, and the establishment of a bionic polarization navigation factor map includes: Introducing Sliding Window ,in represents the state node of the integrated navigation system at the kth moment, n is the total number of key frames, m is the total number of features in the sliding window, It is the first time that the first The inverse depth of features is calculated using the LiDAR odometer to calculate the sliding window The relative transformation between the i-th lidar keyframe and the previous keyframe in , using the relative transformation as a constraint between the two frames, and establishing the lidar odometry factor; The collected LiDAR data is used to stitch multiple LiDAR point clouds into a dense depth map. The polarized RGB map provided by the semantic information positioning module is associated with the feature points extracted from the infrared image to obtain a visual depth map. The depth map is used to assist in calculating the prior of the visual residual. , define the visual inertial odometry residual model, and obtain the sliding window by minimizing the visual inertial odometry residual middle The optimal conversion of: ， in, represents the Euclidean norm, represents the Huber norm, represents the marginal prior residual, The Hessian matrix representing the edge prior information, represents the prior of visual residual, represents the prior of the inertial residual, represents the covariance matrix of the inertial measurement unit, represents the predicted feature measurement value on the unit sphere after converting the first observation in the i-th key frame to the j-th key frame, represents the feature set observed at least twice in the current sliding window, represents the set of inertial measurement unit measurements, Indicates the image observed in the i-th key frame. The residual of the feature and the j-th key frame observing the feature, Represents the pre-integrated value of the inertial measurement unit from time k-1 to time k; The i-th keyframe in the sliding window added to the factor graph establishes an edge node with the previous j-th keyframe in the sliding window: ， ， represents the estimated value of the position of the i-th key frame in the world coordinate system, represents the estimated value of the jth key frame position in the world coordinate system, Represents the rotation matrix from the world coordinate system to the camera coordinate system, represents the position transformation between the i-th key frame and the j-th key frame, represents the estimated heading angle of the i-th key frame, represents the estimated heading angle of the jth key frame, Represents the heading angle transformation between the i-th key frame and the j-th key frame; Defining the marginal residual of biomimetic polarized infrared visual inertial odometry factor for: ， in, Represents the position of the i-th key frame in the world coordinate system, Represents the position of the jth key frame in the world coordinate system and the rolling angle in the visual odometer coordinate system and pitch angle is a fixed estimate obtained from a monocular visual-inertial odometry, Represents the position in the world coordinate system, represents the heading angle of the i-th key frame, represents the heading angle of the jth key frame, Represent the transformation matrix from the world coordinate system to the visual coordinate system, calculate the visual motion transformation, use the motion transformation as an inter-frame constraint, and establish the bionic polarized infrared visual inertial odometry factor; Using the inertial measurement unit pre-integration method, the time is obtained and The speed change between , position change and posture change , taking the pre-integration result as the inter-frame constraint, establish the inertial measurement unit pre-integration factor, the error function of the inertial measurement unit pre-integration factor for: ， in, , and They represent the error of attitude change, position change, and speed change respectively; The gravity acceleration is obtained from the accelerometer of the inertial measurement unit = ,in , and Respectively represent the components of the gravity vector on the x, y and z axes in the b system, and solve the geomagnetic roll angle and geomagnetic pitch angle , in the carrier coordinate system, the output of the magnetic sensor is , calculate the magnetic heading angle , the magnetic heading angle value is used as the observation prior constraint to establish the geomagnetic heading factor and the error function of the geomagnetic heading factor for: ， in, A measurement model representing the geomagnetic heading factor; Calculate the Euclidean distance and then scan and match to obtain the relative transformation , and are the rotation and displacement calculated by scan matching, Indicates the matched keyframe and the relative transformation between the current frame j, the error function of the closed-loop detection factor is: ， represents the translation error, represents the rotation error; Construct the bionic polarization navigation factor graph and define the error function containing position, velocity, attitude and sensor bias: ， in, That is, the state node of the integrated navigation system at the kth moment The best estimate of is the covariance of the lidar noise, represents the inter-frame constraints of the lidar odometry, represents the relative transformation between the current frame and the previous keyframe i, is the noise covariance of the IMU accelerometer and gyroscope, is the noise covariance of the geomagnetic sensor, represents the inter-frame constraint between the jth keyframe and the previous i-th keyframe in the sliding window of the visual odometry, Representation of sliding window in bionic polarized infrared visual inertial odometry The set of all edge nodes in the robot is constructed, and the iSAM2 algorithm is used to solve the bionic polarization navigation factor graph to obtain the optimal estimate and output the attitude, velocity, position and inertial measurement unit bias.

5. The bionic polarization semantic SLAM method based on neural radiation field according to claim 1, characterized in that: The step 3 includes, in the neural mapping submodule, using radar data and posture to train a neural implicit distance function, a neural implicit distance function model , which is expressed as follows: ， in, , represents the posture of aligning the i-th key frame interpolation output of the bionic polarization navigation factor map with the current scanning frame of the lidar, represents the original point in lidar coordinates, Represents the sampling point transformed to the world coordinate system, Indicates sampling point The trilinear interpolation surrounded by the embedding of h neighboring voxels, Indicates that the parameter is A neural implicit network is used to obtain dynamic voxel generation through a neural implicit distance function model, a voxel embedding map is established, and a threshold is set. When the time interval, displacement distance, or rotation angle exceeds the threshold, a lidar scanning key frame is generated, and the lidar scanning key frame is saved to the key frame buffer of the mesh reconstruction submodule; The voxel embedding map and lidar scanning keyframes are passed into the mesh reconstruction submodule in the NeRF-LOAM model. The voxel embedding map is optimized using the keyframes and the neural implicit distance function, and the dense map is reconstructed using the marching cubes method.

6. The bionic polarization semantic SLAM method based on neural radiation field according to claim 1, characterized in that: The step 4 includes using the Stokes vector Calculate the degree of polarization With polarization angle , according to the degree of polarization With polarization angle Calculate the grayscale image separately , polarization degree diagram , Polarization angle diagram , the grayscale image , polarization degree diagram , Polarization angle diagram Synthesize the polarized RGB image, input the polarized RGB image and the infrared image into the object recognition submodule based on Yolov8, and identify the object name and mark the color; The polarized RGB image and the infrared image are respectively input into the semantic segmentation submodule based on SegNeXt, the ORB features are extracted, the feature points extracted from the polarized RGB image and the infrared image are associated, and the target boundary of the polarized and infrared fused image is output; the optimal carrier posture and position, the distance between the carrier and the target, the boundary of the target provided by the visual depth map, and the object name and annotation color recognized by the object recognition submodule are fused to obtain the semantic information positioning in the three-dimensional scene.

Images