An infrared / visible light fusion all-day autonomous positioning method and system guided by semantic information in a dynamic scenario

Abstract

The present application provides an infrared / visible light fusion all-day autonomous positioning method and system guided by semantic information in a dynamic scenario. The method includes: sending a visible light image and an infrared image into a trained fusion model guided by semantic information to obtain a fused image; extracting feature points from the fused image; fusing the fused image with IMU data, and removing dynamic feature points of the fused image based on a dual-threshold elimination method of the IMU and the RGB-D camera; estimating the camera pose using the remaining feature points; and optimizing the estimated camera pose. The advantages of the present application are as follows: on the basis of a traditional visual odometer, a generative adversarial network guided by semantic information is used to fuse infrared and visible light images, and a dual-threshold dynamic target feature point elimination strategy based on the IMU and the RGB-D camera is used to effectively avoid the influence of dynamic objects on the visual positioning accuracy and improve the accuracy of all-day autonomous positioning, especially in a night environment.

Description

Technical Field

[0001] This application belongs to the field of navigation technology, and particularly relates to an infrared / visible light fusion all-weather autonomous positioning method and system guided by semantic information in a dynamic scenario. Background Technique

[0002] Autonomous positioning technology determines its own position and attitude through information observed by sensors in a completely unknown environment, providing a data basis for subsequent tasks such as map construction and navigation. Autonomous positioning is not only a core component of unmanned systems, but also the key to ensuring their stable operation in complex environments.

[0003] Traditional unmanned systems often use the fusion of satellite navigation and inertial navigation. However, the satellite positioning system is greatly affected by the environment. When working in valleys, building clusters, or indoors, satellite signals are easily blocked or multipath effects occur; during wartime, signals are easily blocked or interfered. When satellite positioning fails, pure inertial navigation is easily affected by sensor errors, system noise, and integral cumulative errors. After long-term use, position and velocity information drifts, and additional time is required to correct the accumulated errors, resulting in instability and inaccuracy of the navigation system. Pure vision positioning was first proposed by Stanford University in the 1980s. However, due to the limitations of sensor accuracy and processor performance, it has been in the experimental research stage for a long time. With the progress of electronic information technology in the 21st century, vision-based positioning has made great progress. Currently, the front-end image processing of vision positioning technology can be divided into feature point methods and direct methods, and the back-end state estimation can be divided into filtering-based algorithms and optimization-based algorithms.

[0004] Although pure vision positioning algorithms have made great progress, the accuracy and stability of position and attitude estimation based on image information still do not meet the accuracy requirements. Multi-sensor fusion is an important development direction for vision positioning. Visual sensors and inertial measurement units (IMUs) are highly complementary and both have relatively low prices, so the vision / inertial fusion method has broad applications. The earliest vision / inertial system was proposed by Anastasios Mourikis et al. from the University of California, Riverside, USA in 2007. Subsequently, a series of classic algorithms such as VI-ORB, VINS-Mono, and ORB-SLAM3 emerged.

[0005] The above technologies are almost all used in scenarios with sufficient light. The autonomous positioning technology of vision-based unmanned systems in extreme lighting conditions such as low-light environments and complete darkness is still in the exploratory stage. Currently, the mainstream method is to adopt infrared or infrared-fused visible light strategies. However, so far, domestic and foreign research is still in the theoretical research stage, and no all-weather autonomous positioning system that meets real-time and stable operation has been implemented.

[0006] In 2016, Borges, Vidas et al. proposed an infrared visual odometer based on optical flow tracking to detect known objects, thereby constraining the scale of monocular navigation. However, since this algorithm does not optimize the map, the positioning error is relatively large. In 2017, Chen et al. proposed a visual odometry algorithm RGB-T that fuses infrared images and visible light images. During the day, infrared images are used to improve the accuracy of visible light. At night, the infrared features can work independently. Then, FAST corners ("Features from Accelerated Segment Test" corner detection algorithm) are extracted from the images for tracking to achieve the fusion of heterogeneous image observation information. In 2020, Zhao et al. proposed an infrared feature extraction network ThermalPoint based on the existing visible light feature extraction network Superpoint, effectively solving the problems of low texture and photometric changes in infrared images. However, the algorithm cannot solve the cumulative error generated during the long-term operation of the odometer, and the authors did not give the test situation of outdoor operation. In 2023, Wang et al. proposed the first edge-based monocular thermal inertial odometer ETIO, which uses the binary image extracted from the edges to replace the original image for pose estimation to overcome the problem of poor quality of thermal infrared images. The IMU pre-integration is combined with the reprojection error of all edge feature observations for pose graph optimization, and real-time estimation is performed on the sliding window of the latest state. However, the edge feature data volume is large, making it difficult to optimize and manage for a long time. It can only perform local matching. When the camera movement direction is approximately parallel to the edge direction, the edge depth cannot be initialized.

[0007] In addition, existing mainstream visual SLAM algorithms assume that objects in the environment are stationary or have low motion. However, in practical applications, most scenes are complex and dynamic. Methods such as the RANSAC algorithm (Random Sample Consensus) can solve the problems brought by dynamic targets that occupy a small and small number of pixels in the image to a certain extent. However, when the number of feature points on the dynamic target is large, this method will fail, and the SLAM system cannot distinguish the motion of the carrier and the object, resulting in inaccurate positioning and serious deviation in mapping, greatly reducing the robustness and accuracy of the system.

[0008] In 2018, Yu et al. from Tsinghua University proposed DS-SLAM, which combines the semantic segmentation network SegNet with a method for detecting mobile consistency, filtering out the dynamic parts of the scene. Compared with ORB-SLAM2, DS-SLAM adds a semantic segmentation and a thread for constructing a dense semantic octree map. The problems of DS-SLAM are as follows: 1) The time overhead of semantic segmentation is 37.6 ms, which is roughly equivalent to that of the tracking thread, and it is difficult to ensure real-time performance; 2) The types of objects that can be recognized in the semantic segmentation network are limited, restricting its scope of application; 3) The outlier detection method based on the epipolar constraint cannot find all outliers, and this method will fail when the object moves along the epipolar direction. DynaSLAM, proposed in 2018, is also based on ORB-SLAM2 and adds the ability to detect dynamic objects and repair the background. When using an RGB-D camera, it uses a method that combines multi-view geometry and deep learning to detect moving objects, and when using a monocular or binocular camera, it only uses deep learning methods. The disadvantages of this method are that 1) removing all potential dynamic points may result in too few remaining static feature points, affecting the SLAM effect; 2) DynaSLAM is not real-time, and the time consumption is mainly due to the Region Growing algorithm. RDS SLAM, proposed in 2020, is based on ORB-SLAM3, adds a semantic thread and a semantic-based optimization thread, uses the movement probability to update and propagate semantic information, and uses a data association algorithm to remove outliers in tracking. RDS SLAM solves the real-time problem of semantic-based methods, but only removes outliers for key frames, and there are limitations in removing disturbances for large and violently moving dynamic targets. Summary of the Invention

[0009] The purpose of this application is to overcome the defect that the visible light vision / inertial fusion navigation technology fails in scenarios with a large number of dynamic objects under extreme lighting conditions such as low-light environments without GNSS signals or complete darkness.

[0010] To achieve the above purpose, this application proposes an all-weather autonomous positioning method for infrared / visible light fusion guided by semantic information in a dynamic scene, including:

[0011] Step 1: Send the visible light image and the infrared image into a trained fusion model guided by semantic information to obtain a fused image;

[0012] Step 2: Extract feature points from the fused image;

[0013] Step 3: Further fuse the fused image with IMU data, and remove the dynamic feature points of the fused image based on the dual-threshold elimination method of the IMU and the RGB-D camera;

[0014] Step 4: Estimate the camera pose using the remaining feature points;

[0015] Step 5: Optimize the estimated camera pose.

[0016] As an improvement of the above method, the fusion model is a generative adversarial network model.

[0017] As an improvement of the above method, the training process of the fusion model includes:

[0018] First, use the object detection algorithm to detect the objects and foreground targets that interfere with positioning in the infrared image, extract the foreground targets in the infrared image to obtain the foreground image, delete the foreground targets in the visible light image, and only keep the background image after removing the foreground targets; second, connect the infrared image and the visible light image in the channels and send them into the generator to output the fused image; third, separate the fused image into the fused foreground image and the fused background image; finally, send the fused foreground / background image and the foreground / background image before fusion into the foreground discriminator and the background discriminator respectively to judge the authenticity. If it is false, use the generator to regenerate.

[0019] As an improvement of the above method, the loss function of the fusion model includes 1 generator loss function and 2 discriminator loss functions;

[0020] The generator loss function L G is:

[0021] L G = L content + L edge + αL adversarial

[0022] where L content represents the content loss:

[0023]

[0024] where H and W represent the image height and width respectively; ||·|| F represents the Frobenius norm; ξ represents the weight coefficient; I ir represents the infrared image; I vi represents the visible light image; I f represents the fused image; represents the gradient of each pixel point in the fused image; represents the gradient of each pixel point in the visible light image;

[0025] L edge represents the edge loss:

[0026]

[0027] Among them, G canny represents the weight based on the Canny operator;

[0028] L adversarial represents that the adversarial loss is the loss function of the conventional FusionGAN model; α is the weight coefficient;

[0029] The loss functions of the 2 discriminators include the foreground discriminator loss function L front and the background discriminator loss function L back ;

[0030] The foreground discriminator loss function L front is:

[0031]

[0032] Among them, N represents the number of fused images; a and b respectively represent the labels of the foreground discriminator for the fused image I f and the visible light image I vi ; D front (I f ) and D front (I ir ) respectively represent the discrimination results of the foreground discriminator for the fused image and the visible light image;

[0033] The background discriminator loss function L back is:

[0034]

[0035] Among them, c and d respectively represent the labels of the background discriminator for the fused image I f and the visible light image I vi ; D back (I f ) and D back (I vi ) respectively represent the discrimination results of the background discriminator for the fused image and the visible light image.

[0036] As an improvement of the above method, for the extraction of feature points from the fused image, the ORB image feature detection and description method is used for the extraction of feature points.

[0037] As an improvement of the above method, for the further fusion of the fused image and the IMU data, it further includes pre-integrating the IMU data, and the pre-integration model is:

[0038]

[0039] Among them, P wbj represents the position of the IMU data at time j in the world coordinate system; Pwbi Denote the position of the IMU data at time \(i\) in the world coordinate system; Denote the velocity of the IMU data at time \(i\) in the world coordinate system; Denote the velocity of the IMU data at time \(j\) in the world coordinate system; \(\Delta t\) represents the time difference between time \(i\) and time \(j\); \(g\) w Denote the gravitational acceleration in the world coordinate system; \(q\) wbi Denote the attitude quaternion for the IMU data at time \(i\) when rotating from the body coordinate system to the world coordinate system; \(q\) wbj Denote the attitude quaternion for the IMU data at time \(j\) when rotating from the body coordinate system to the world coordinate system; Denote the pre-integrated quantity of the attitude change; Denote the acceleration of the IMU data at time \(t\) in the body coordinate system; Denote the angular velocity of the IMU data in the body coordinate system; Denote the multiplication of quaternions;

[0040] The pre-integrated quantities of displacement change, velocity change, and attitude change are:

[0041]

[0042] Among them, Denote the pre-integrated quantity of displacement change; Denote the pre-integrated quantity of velocity change.

[0043] As an improvement of the above method, the double-threshold elimination method for the IMU and RGB-D camera includes:

[0044] Calculate the coordinates of the current frame feature point \(p2\) in the camera coordinate system \(I2\) using the data of the RGB-D camera

[0045]

[0046] Among them, \((u2, v2)\) represents the camera coordinate system coordinates of \(p2\); \(d2\) represents the depth obtained by the current frame camera; \(f\) x 、\(f\) y Represent the focal lengths of the camera in the \(x\) direction and \(y\) direction respectively;

[0047] The coordinates \(P1\) of the feature point \(p1\) matched in the previous frame in the camera coordinate system \(I1\) cam1 (X1 cam1 , Y1 cam1 , Z1 cam1 ):

[0048]

[0049] Among them, (u1, v1) represents the camera coordinate system coordinates of p1; d1 represents the depth obtained by the camera in the previous frame;

[0050] Substitute the joint calibration parameters of the camera and IMU and the IMU pre-integration quantities into the kinematic formula to obtain the rotation and translation matrices (R 21 , t 21 ) from the previous frame to this frame, and calculate the coordinates P1 of the feature point p1 in the previous frame in the camera coordinate system I2 cam2 :

[0051] P1 cam2 = R 21 P1 cam1 + t 21

[0052] Calculate the Chebyshev distance ψ between two three-dimensional points in the camera coordinate system I2:

[0053] ψ = max(|X1 cam2 - X2 cam2 |, |Y1 cam2 - Y2 cam2 |, |Z1 cam2 - Z2 cam2 |)

[0054] Among them, X1 cam2 represents the x coordinate of the feature point p1 in the previous frame in the camera coordinate system I2; Y1 cam2 represents the y coordinate of the feature point p1 in the previous frame in the camera coordinate system I2; Z1 cam2 represents the z coordinate of the feature point p1 in the previous frame in the camera coordinate system I2;

[0055] When the Chebyshev distance ψ is greater than the first set threshold, the feature point is determined to be a dynamic feature point;

[0056] When the number of dynamic feature points in a target is greater than the second set threshold, the target is determined to be a dynamic target, and all feature points in the target are removed.

[0057] As an improvement of the above method, the camera pose is estimated using the remaining feature points, specifically by using the non-linear optimization PnP method for camera pose estimation.

[0058] As an improvement of the above method, the estimated camera pose is optimized, specifically by the ORB-SLAM3 processing method.

[0059] This application also provides an infrared / visible light fusion all-day autonomous positioning system guided by semantic information in a dynamic scene, which is implemented based on the above method. The system includes:

[0060] The fused image module is used to send the visible light image and the infrared image into a trained fusion model guided by semantic information to obtain a fused image;

[0061] The feature point extraction module is used to extract feature points from the fused image;

[0062] The dynamic feature point removal module is used to fuse the fused image with the IMU data, and based on the double-threshold removal method of the IMU and the RGB-D camera, remove the dynamic feature points of the fused image;

[0063] The camera pose estimation module is used to estimate the camera pose using the remaining feature points;

[0064] The optimization module is used to optimize the estimated camera pose.

[0065] Compared with the prior art, the advantages of this application are as follows:

[0066] 1. Infrared images and visible light images often contain various complementary information. Especially at night, the infrared image can filter out the influence of lights, and can accurately highlight targets with a large temperature difference between the surface temperature of pedestrians, vehicles, etc. and the ambient temperature; at the same time, the visible light image also contains more texture information. Traditional fusion methods usually calibrate the fusion parameters according to the observations of algorithm designers, while the present invention pre-labels the objects that need to be focused on in the training set, and uses semantic information to guide the fusion of infrared and visible light, which can make the fusion more purposeful, and the fused image can better detect potential dynamic targets.

[0067] 2. Traditional feature point-based SLAM algorithms often have poor accuracy in dynamic scenarios. The present invention proposes a double-threshold feature point removal method based on the IMU and the RGB-D camera. By calculating two thresholds, the Chebyshev distance of the corresponding three-dimensional coordinates of the matching feature points in the unified coordinate system and the number of dynamic feature points of the same target, the true dynamicity of the object is judged, and all feature points of the dynamic target are removed, avoiding the influence of mis-matched and feature points with an unclear movement range in the dynamic target on the accuracy.

[0068] 3. When the external lighting conditions change, a single sensor cannot handle all working scenarios. The present invention uses a generative adversarial network guided by semantic information to fuse infrared and visible light images, which can better detect external targets, not only avoid the influence of dynamic targets, but also enable the visual autonomous positioning technology to work all day long.

[0069] 4. Based on the traditional visual odometry, the present invention uses a semantic information-guided generative adversarial network to fuse infrared and visible light images, and uses a dual-threshold dynamic target feature point elimination strategy based on IMU and RGB-D cameras, effectively avoiding the influence of dynamic objects on the visual positioning accuracy and improving the accuracy of all-weather, especially autonomous positioning in night environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 The figure shows a schematic diagram of the training process of infrared-visible light fusion images;

[0071] Figure 2 The figure shows a schematic diagram of dynamic feature points;

[0072] Figure 3 The figure shows a schematic diagram of eliminating moving objects;

[0073] Figure 4 The figure shows a flow chart of dual-threshold dynamic object feature point elimination;

[0074] Figure 5 The figure shows a schematic diagram of reprojection error;

[0075] Figure 6 The figure shows a flow chart of the semantic information-guided infrared / visible light fusion all-weather autonomous positioning method in a dynamic scene. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0076] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings.

[0077] Aiming at the problem that the visible light vision / inertial fusion navigation technology fails in scenes with more dynamic objects under extreme illuminations such as low-light environments without GNSS signals or complete darkness, the present application introduces infrared feature-based visual navigation into the traditional visual inertial fusion navigation system, and uses the semantic information of dynamic objects to guide the fusion of infrared and visible light. A framework based on a generative adversarial network (GAN) is used to model the image fusion problem as an adversarial game problem between a generator and a discriminator, and the discriminator is used to force the fusion result generated by the generator to be an infrared enhanced image whose probability distribution tends to be consistent with the target distribution. According to the features of the fused image, the feature point method is used for feature tracking, and the IMU data and the extracted image features are jointly used to perform an optimal estimation of the pose state quantity.

[0078] The present invention adopts a visual odometry calculation framework with tight coupling of an RGB-D camera, an infrared camera, and an IMU. First, a model for fusing infrared and visible light guided by semantic information is trained. The visible light image and the infrared image are sent into the model to obtain a fused image. The fused image contains the features of the infrared and visible light images, which is more convenient for the target recognition algorithm to detect potential dynamic objects. Feature points are extracted at the front end of the visual odometer, and then target detection algorithms such as YOLOv8 are used to detect potentially moving targets. Geometric constraints are used to judge the true dynamics of the targets, and the feature points on the dynamic targets are removed. The remaining feature points are fused with the IMU information to estimate the rough camera motion. The backend is consistent with ORB-SLAM3, and the pose is optimized again through BA (Bundle Adjustment), and finally a pose estimation result with higher accuracy is obtained.

[0079] Embodiment 1

[0080] As Figure 6 shown, the all-weather autonomous positioning method for infrared / visible light fusion guided by semantic information in a dynamic scene provided by this application includes:

[0081] Step 1: Send the visible light image and the infrared image into a trained fusion model guided by semantic information to obtain a fused image.

[0082] The training process of the infrared and visible light image fusion model is as Figure 1 shown. The fusion model is a generative adversarial network model. First, the infrared image I ir Use target detection algorithms such as YOLOv8 to detect objects such as people and vehicles and foreground targets that interfere with positioning, take out the foreground targets detected in the infrared image to obtain a foreground image, and correspondingly delete the target in the visible light image I vi to only retain the background image after removing the target; then, connect the infrared image and the visible light image in channels and send them into the generator to output a fused image; again, separate the fused foreground image and the fused background image from the fused image using the above method; finally, send the fused foreground / background image and the foreground / background image before fusion into the foreground discriminator and the background discriminator respectively to judge the authenticity. If it is false, let the generator regenerate. In the repeated generative adversarial process, the generated image tends to be consistent with the image we need in terms of probability distribution.

[0083] The loss function of the fusion model is divided into the generator loss function and two discriminator loss functions. The generator loss function is defined as:

[0084] L G =L content +L edge +αL adversarial

[0085] Among them, the content loss L content makes the pixel intensity of the fused image similar to that of the infrared image and the gradient change similar to that of the visible light image. The edge loss L edge retains the edge information of the fused image. The adversarial loss L adversarial is defined in relation to the discriminator and is the loss function of the conventional FusionGAN model, which helps train the generator to "fool" the discriminator. α is the weight coefficient.

[0086] The content loss is defined as follows:

[0087]

[0088] Among them, H and W respectively represent the image height and width, and ||·|| F represents the Frobenius norm, ξ represents the weight coefficient, and I f represents the fused image, represents the gradient of each pixel point in the fused image, represents the gradient of each pixel point in the visible light image. The first term aims to retain the infrared information in the fused image, and the second term aims to retain the gradient information contained in the visible light image.

[0089] The edge loss is defined as:

[0090]

[0091] The definition of the edge loss is based on retaining the infrared information in the first term of the content loss and using the weight G canny based on the canny operator, making the training more focused on the edge information based on the infrared image.

[0092] The discriminator is designed to distinguish the fused image from the foreground / background image based on the features extracted from the fused image and the visible light background image / infrared foreground image.

[0093] The loss functions of the foreground and background discriminators are respectively defined as:

[0094]

[0095] Among them, N represents the number of fused images, a, c and b, d represent the labels of the discriminator for the fused image I f and the visible light image I vi , and D · (I f ) and D · (I vi ) respectively represent the discrimination results of the current discriminator for the fused image and the visible light image.

[0096] In this application, "semantic information guidance" refers to using object detection algorithms to detect objects such as people and vehicles as semantic information, that is, "possibly moving objects". In the generative adversarial fusion, the discriminator focuses on whether the fused image can be well detected for the target.

[0097] Step 2: Extract feature points from the fused image.

[0098] Feature point extraction can be performed using methods such as ORB. The first step of ORB (Oriented Fast and Rotated Brief) feature detection uses the Oriented FAST corner detection algorithm to find key points in the image, and then uses the improved BRIEF descriptor to describe the surrounding image area of the feature points (key points).

[0099] Oriented FAST corners add descriptions of scale and rotation on the basis of FAST corners by constructing an image pyramid and the gray centroid method. The process of FAST corner detection is as follows:

[0100] (1) Select a pixel p in the image, and its brightness is I p ;

[0101] (2) Set the threshold T;

[0102] (3) With the pixel p as the center, select 16 pixel points on the adjacent circle with a radius of 3;

[0103] (4) If there are N consecutive points in the adjacent circle whose brightness exceeds the threshold T, then the pixel p is a feature point;

[0104] (5) Traverse all pixels in the image and repeat the above steps;

[0105] (6) Use non-maximum suppression in the area of the feature point set to avoid clustering of feature points.

[0106] The scale information of the feature points is obtained by constructing an image pyramid. The input picture is downsampled level by level to obtain an image pyramid. The bottom layer of the pyramid is the original image, and the image resolution decreases by 1 / 1.2 for each increased layer. The number of pyramid layers set in the present invention is 8 layers. When performing feature matching, images on different layers are matched, thereby achieving scale invariance.

[0107] The rotation information of the Oriented FAST corner is obtained by the connection line between the gray centroid and the geometric center of the image block around the feature point. The calculation process is as follows:

[0108] (1) Define the moment of the image block as:

[0109]

[0110] Among them, B is the image patch near the feature point, x and y are the horizontal and vertical coordinates of the pixel point, and I(x, y) is the gray value of the pixel point.

[0111] (2) Calculate the centroid of the image patch as:

[0112]

[0113] (3) Connect the geometric center O of the image patch with the centroid C to obtain the direction vector The direction of the feature point is:

[0114] θ = arctan(m 01 / m 10 )

[0115] Through the above method, we obtain the OrientedFAST corner points with scale and rotation descriptions.

[0116] BRIEF is a binary descriptor, which consists of multiple 0s and 1s, is simple and convenient to store, and has less computational complexity. Its description and matching process are as follows:

[0117] (1) Take N pairs of points in the area of S×S around the feature point according to the Gaussian distribution, and perform Gaussian smoothing on these 2N points;

[0118] (2) Compare the pixel sizes of the first point and the second point in each pair of points respectively. If the pixel value of the first point is greater than that of the second point, take 1; otherwise, take 0, and obtain an N-dimensional vector composed of 0s and 1s.

[0119] (3) Use the Hamming distance for matching, that is, if the number of bits with the same value in the descriptors of two feature points is greater than the threshold, they are matched.

[0120] Step 3: Eliminate the dynamic feature points of the fused image based on the dual-threshold elimination method of the IMU and the depth camera.

[0121] Fuse the fused image with the IMU data, use object detection algorithms such as YOLOv8 to detect potential dynamic objects as dynamic targets, and use the dual-threshold elimination method of the IMU and the depth camera to eliminate the dynamic feature points of the dynamic targets.

[0122] Using object detection algorithms such as YOLOv8 can only detect objects that may move, and the true dynamicity of the targets still needs to be further judged, as Figure 2 shown. As Figure 4 shown, the dual-threshold elimination method of the IMU and the depth camera includes:

[0123] When the target P is a dynamic feature point, the observed 3D target points are the same feature on the same dynamic target, but have different coordinates in the world coordinate system. Given the camera coordinate system coordinates (u2, v2) of the feature point p2 in the current frame, the RGB-D camera can be used to calculate its coordinates P2 in the camera coordinate system I2 cam2 :

[0124]

[0125] where d2 represents the depth obtained by the depth camera.

[0126] Similarly, the coordinates P1 of the feature point p1 matched in the previous frame in the camera coordinate system I1 cam1 :

[0127]

[0128] Substituting the joint calibration parameters of the camera and IMU and the IMU pre-integration quantities into the kinematic formula, the rotation and translation matrices (R 21 , t 21 ) from the previous frame to this frame can be obtained, and the coordinates of the previous frame pixel in the camera coordinate system I2 can be calculated

[0129] P1 cam2 = R 21 P1 cam1 + t 21

[0130] Calculate the Chebyshev distance between the two 3D points in the camera coordinate system I2:

[0131]

[0132] When the distance is greater than the threshold, the feature point is determined to be a dynamic feature point.

[0133] When the number of dynamic feature points in a target is greater than the threshold, the target is determined to be a dynamic target, and all feature points in the target are removed. An example is shown Figure 3 as shown. The green points are normal feature points, and the red points are the removed feature points.

[0134] During the process of fusing the fused image with IMU data, since the sampling frequency of the IMU is much higher than the acquisition frequency of the image, a large amount of IMU data will accumulate between every two frames of images. When performing iterative optimization of data fusion, integrating the IMU data again is required every time the historical state is updated, which undoubtedly increases the computational complexity. To solve this problem, the present invention adopts the IMU pre-integration technology, and forms a single pose transformation value by combining the IMU data between consecutive image frames. In this way, only the result of this pre-integration needs to be considered during each optimization, without relying on the data of the previous frame, thereby significantly reducing the computational burden and improving the efficiency of data processing.

[0135] Based on the position P, velocity V, and attitude quaternion Q at time i, the P, V, and Q at time j can be obtained. Converting the traditional IMU integration model to a pre-integration model is as follows:

[0136]

[0137] Among them, P wbj represents the position at time j from the body coordinate system (body) to the world coordinate system (world), that is, the position in the world coordinate system; P wbi represents the same position at time i; represents the velocity in the world coordinate system at time i; represents the velocity in the world coordinate system at time j; Δt represents the time difference between time i and time j; g w represents the gravitational acceleration in the world coordinate system; q wbi represents the attitude quaternion from the body coordinate system to the world coordinate system at time i; q wbj represents the same attitude quaternion at time j; represents the pre-integrated quantity of attitude change; represents the acceleration in the body coordinate system at time t; represents the angular velocity in the body coordinate system; represents the multiplication of quaternions.

[0138] The pre-integrated quantity is only related to the IMU measurement values. The pre-integrated quantities of displacement change, velocity change, and attitude change are:

[0139]

[0140] Among them, represents the pre-integrated quantity (increment) of displacement change; represents the pre-integrated quantity (increment) of velocity change.

[0141] Step 4: Estimate the camera pose for the remaining feature points using the PnP method.

[0142] Since an RGB-D camera is used, the 3D position of the feature points can be determined from the depth map, so the motion can be directly solved by the non-linear optimization Perspective-n-Point (PnP) method.

[0143] For n three-dimensional space points in an image, we hope to calculate the pose of the camera (including rotation R and translation t), whose Lie group representation is T. For a spatial point coordinate P = [X, Y, Z] T , the projection in the pixel coordinate system of the camera is u = [u, v] T , and the internal parameters of the camera are where f x , f y are the focal lengths of the camera in the x and y directions respectively, and c x , c y are the optical center translations of the camera in the x and y directions respectively. Since an RGB-D camera is used, the 3D point position can be obtained by the following formula:

[0144]

[0145] where d is the actual distance from the object corresponding to the pixel point given by the depth camera to the camera.

[0146] The relationship between the pixel position and the spatial point position is as follows:

[0147]

[0148] where s is the scale, that is, the distance from the spatial point to the camera plane.

[0149] Due to the unknown camera pose and the existence of observation noise, there is an error e in the above equation. We sum the errors of n spatial points in an image to construct a least squares problem to optimize the camera pose:

[0150]

[0151] where T * represents the optimized Lie group, used to distinguish from the original Lie group T. s i represents the scale of the i-th feature point, that is, the distance from the feature point to the camera plane.

[0152] As Figure 5 shown, through ORB feature matching, it can be known that p1 and p2 belong to the projections of the spatial point P in two adjacent images. However, due to the unknown camera pose, there is an error between the initially estimated and the actual p2. After constructing the above least squares problem, the Lie group T of the camera pose can be solved by optimization algorithms such as the Gauss-Newton method, and then the actual motion of the camera can be obtained.

[0153] Step 5: Optimize the pose again through BA (Bundle Adjustment), and finally obtain a pose estimation result with higher accuracy. This step is the same as the processing method of ORB-SLAM3.

[0154] Embodiment 2

[0155] This application also provides an infrared / visible light fusion all-day autonomous positioning system guided by semantic information in a dynamic scenario, which is implemented based on the above method. The system includes

[0156] A fused image module, which is used to send a visible light image and an infrared image into a trained fusion model guided by semantic information to obtain a fused image;

[0157] A feature point extraction module, which is used to extract feature points from the fused image;

[0158] A dynamic feature point removal module, which is used to fuse the fused image with IMU data again, and remove the dynamic feature points of the fused image based on the dual-threshold removal method of IMU and RGB-D cameras;

[0159] A camera pose estimation module, which is used to estimate the camera pose using the remaining feature points;

[0160] An optimization module, which is used to optimize the estimated camera pose.

[0161] This application can also provide a computer device, including: at least one processor, a memory, at least one network interface, and a user interface. Each component in the device is coupled together through a bus system. It can be understood that the bus system is used to achieve the connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus, and a status signal bus.

[0162] Among them, the user interface can include a display, a keyboard, or a pointing device. For example, a mouse, a trackball, a touchpad, or a touch screen, etc.

[0163] It can be understood that the memory in the disclosed embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0164] In some embodiments, the memory stores the following elements, executable modules or data structures, or subsets or supersets thereof: an operating system and application programs.

[0165] Among them, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., and is used to implement various basic services and process hardware-based tasks. The application programs include various application programs, such as a media player and a browser, etc., and are used to implement various application services. The program for implementing the method of the disclosed embodiments of the present application can be included in the application programs.

[0166] In the above embodiments, by calling the programs or instructions stored in the memory, specifically, the programs or instructions stored in the application programs, the processor is configured to:

[0167] Execute the steps of the above method.

[0168] The above method can be applied to or implemented by a processor. The processor may be an integrated circuit chip with the ability to process signals. During implementation, the steps of the above method can be completed by the integrated logic circuit of the hardware in the processor or instructions in the form of software. The above-mentioned processor may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed above. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Combining the steps of the above-disclosed method can be directly embodied as being executed and completed by a hardware decoding processor, or by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0169] It can be understood that these embodiments described in the present application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0170] For software implementation, the technology of the present application can be implemented by executing the functional modules of the present application (such as procedures, functions, etc.). The software code can be stored in the memory and executed by the processor. The memory can be implemented inside or outside the processor.

[0171] The present application may also provide a non-volatile storage medium for storing a computer program. When the computer program is executed by a processor, each step in the above method embodiments can be implemented.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the embodiments, those of ordinary skill in the art should understand that any modification or equivalent replacement of the technical solutions of the present application does not depart from the spirit and scope of the technical solutions of the present application, and they should all be covered by the scope of the claims of the present application.

Claims

1. An all-day autonomous positioning method for infrared / visible light fusion guided by semantic information in a dynamic scenario, including: Step 1: Send the visible light image and the infrared image into a trained fusion model guided by semantic information to obtain a fused image; Step 2: Extract feature points from the fused image; Step 3: Further fuse the fused image with the IMU data, and eliminate the dynamic feature points of the fused image based on the double-threshold elimination method of the IMU and RGB-D cameras; Step 4: Estimate the camera pose using the remaining feature points; Step 5: Optimize the estimated camera pose; The further fusion of the fused image with the IMU data also includes pre-integrating the IMU data, and the pre-integration model is: Among them, P wbj represents the position of the IMU data at time j in the world coordinate system; P wbi represents the position of the IMU data at time i in the world coordinate system; represents the velocity of the IMU data at time i in the world coordinate system; represents the velocity of the IMU data at time j in the world coordinate system; Δt represents the time difference between time i and time j; g w represents the gravitational acceleration in the world coordinate system; q wbi represents the attitude quaternion for rotating the IMU data at time i from the body coordinate system to the world coordinate system; q wbj represents the attitude quaternion for rotating the IMU data at time j from the body coordinate system to the world coordinate system; represents the pre-integrated quantity of attitude change; represents the acceleration of the IMU data at time t in the body coordinate system; represents the angular velocity of the IMU data in the body coordinate system; represents the multiplication of quaternions; The pre-integrated quantities of displacement change, velocity change, and attitude change are: Among them, represents the pre-integrated component of displacement change; represents the pre-integrated component of velocity change; The double-threshold elimination method of the IMU and RGB-D cameras includes: Calculate the coordinates of the feature point p2 in the current frame in the camera coordinate system I2 using the data of the RGB-D camera where (u2, v2) represents the camera coordinate system coordinates of p2; d2 represents the depth obtained by the camera in the current frame; f x and f y represent the focal lengths of the camera in the x and y directions, respectively; The coordinates of the feature point p1 matched in the previous frame in the camera coordinate system I1 Among them, (u1, v1) represents the camera coordinate system coordinates of p1; d1 represents the depth obtained by the previous frame of the camera; Bring the joint calibration parameters of the camera and the IMU and the IMU pre-integrated quantities into the kinematic formula to obtain the rotation and translation matrices (R 21 , t 21 ) from the previous frame to this frame, and calculate the coordinates P1 of the feature point p1 in the previous frame in the camera coordinate system I2 cam2 : P1 cam2 = R 21 P1 cam1 + t 21 Calculate the Chebyshev distance ψ of two three-dimensional points in the camera coordinate system I2: Among them, represents the x - coordinate of the feature point p1 in the previous frame in the camera coordinate system I2; Y1 cam2 represents the y - coordinate of the feature point p1 in the previous frame in the camera coordinate system I2; represents the z - coordinate of the feature point p1 in the previous frame in the camera coordinate system I2; When the Chebyshev distance ψ is greater than the first set threshold, it is determined that the feature point is a dynamic feature point; When the number of dynamic feature points in a target is greater than the second set threshold, it is determined that the target is a dynamic target, and all feature points in the target are eliminated.

2. The infrared / visible light fusion all-day autonomous positioning method guided by semantic information in a dynamic scenario according to claim 1, wherein The fusion model is a generative adversarial network model.

3. The infrared / visible light fusion all-day autonomous positioning method guided by semantic information in a dynamic scene according to claim 2, wherein The training process of the fusion model includes: First, use the object detection algorithm to detect the objects and foreground targets that interfere with positioning in the infrared image, extract the foreground targets in the infrared image to obtain the foreground image, and delete the foreground targets in the visible light image, only retaining the background image after removing the foreground targets; secondly, connect the infrared image and the visible light image in channels, send them into the generator, and output the fused image; thirdly, separate the fused image into the fused foreground image and the fused background image; finally, send the fused foreground / background image and the foreground / background image before fusion into the foreground discriminator and the background discriminator respectively to judge the authenticity. If it is false, use the generator to regenerate.

4. The infrared / visible light fusion all-day autonomous positioning method guided by semantic information in a dynamic scene according to claim 2, wherein The loss function of the fusion model includes 1 generator loss function and 2 discriminator loss functions; The generator loss function L G is as follows: L G = L content + L edge + αL adversarial where L content represents content loss: Wherein, H and W respectively represent the height and width of the image; ||·|| F represents the Frobenius norm; ξ represents the weight coefficient; I ir represents the infrared image; I vi represents the visible light image; I f represents the fused image; represents the gradient of each pixel point in the fused image; represents the gradient of each pixel point in the visible light image; L edge Indicates edge loss: Among them, G canny represents the weight based on the Canny operator; L adversarial denotes that the adversarial loss is the loss function of the conventional FusionGAN model; α is the weight coefficient; The two discriminator loss functions include the foreground discriminator loss function $L$ front and the background discriminator loss function $L$ back ; Foreground discriminator loss function L front is as follows: Among them, N represents the number of fused images; a and b respectively represent the labels of the fused image I f and the visible light image I vi ; D front (I f ) and D front (I ir ) respectively represent the discrimination results of the foreground discriminator for the fused image and the visible light image; Background discriminator loss function L back is as follows: Among them, c and d respectively represent the labels of the fused image I f and the visible light image I vi ; D back (I f ) and D back (I vi ) respectively represent the discrimination results of the background discriminator on the fused image and the visible light image.

5. The infrared / visible light fusion all-day autonomous positioning method guided by semantic information in a dynamic scenario according to claim 1, characterized in that, For the extraction of feature points from the fused image, the ORB image feature detection and description method is used for feature point extraction.

6. The method for all-day autonomous positioning by infrared / visible light fusion guided by semantic information in a dynamic scenario according to claim 1, wherein The estimation of the camera pose using the remaining feature points is specifically to use the non-linear optimization PnP method for camera pose estimation.

7. The infrared / visible light fusion all-day autonomous positioning method guided by semantic information in a dynamic scenario according to claim 1, characterized in that The optimization of the estimated camera pose is specifically the ORB-SLAM3 processing method.

8. An infrared / visible light fusion all-weather autonomous positioning system guided by semantic information in a dynamic scenario, implemented based on the method described in any one of claims 1-7, characterized in that The system includes: A fused image module for sending the visible light image and the infrared image into a trained fusion model guided by semantic information to obtain a fused image; A feature point extraction module for extracting feature points from the fused image; A dynamic feature point elimination module for further fusing the fused image with the IMU data and eliminating the dynamic feature points of the fused image based on the double-threshold elimination method of the IMU and RGB-D cameras; A camera pose estimation module for estimating the camera pose using the remaining feature points; and An optimization module for optimizing the estimated camera pose.

Images