Semantic information guided infrared / visible light fused all-time autonomous positioning method and system in dynamic scene

Abstract

The invention provides an infrared / visible light fusion all-time autonomous positioning method and system guided by semantic information in a dynamic scene, and the method comprises the steps: transmitting a visible light image and an infrared image into a trained fusion model guided by the semantic information, and obtaining a fusion image; performing feature point extraction on the fused image; fusing the fused image with the IMU data, and eliminating dynamic feature points of the fused image based on a dual-threshold elimination method of the IMU and an RGB-D camera; estimating the pose of the camera by using the remaining feature points; and optimizing the estimated camera pose. Compared with the prior art, the method has the advantages that on the basis of a traditional visual odometer, the generative adversarial network guided by semantic information is used for fusing the infrared and visible light images, and the dual-threshold dynamic target feature point elimination strategy based on the IMU and the RGB-D camera is used, so that the influence of a dynamic object on the visual positioning precision is effectively avoided, and the all-day and all-time positioning accuracy is improved. And especially, the accuracy of self-positioning in a night environment is improved.

Description

Technical Field

[0001] The present application belongs to the field of navigation technology, and specifically relates to an infrared / visible light fusion all-day autonomous positioning method and system guided by semantic information in dynamic scenes. Background Art

[0002] Autonomous positioning technology determines the position and posture of an unmanned system through information observed by sensors in a completely unknown environment, providing a data basis for subsequent map construction and navigation tasks. Autonomous positioning is not only a core component of unmanned systems, but also the key to ensuring their stable operation in complex environments.

[0003] Traditional unmanned systems often use a fusion of satellite navigation and inertial navigation. However, satellite positioning systems are greatly affected by the environment. When working in valleys, buildings, and indoors, satellite signals are easily blocked or multipath effects occur; in wartime, signals are easily shielded or interfered. When satellite positioning fails, pure inertial navigation is easily affected by sensor errors, system noise, and integral cumulative errors. After long-term use, position and velocity information drifts, and additional time is required to correct the accumulated errors, resulting in instability and inaccuracy of the navigation system. Pure visual positioning was proposed by Stanford University as early as the 1980s. However, due to the limitations of sensor accuracy and processor performance, it was only in the experimental research stage for a long time. With the advancement of electronic information technology in the 21st century, vision-based positioning has made great progress. At present, the front-end image processing of visual positioning technology can be divided into feature point method and direct method, and the back-end state estimation can be divided into filtering-based algorithm and optimization-based algorithm.

[0004] Although pure visual positioning algorithms have made great progress, the accuracy and stability of position and attitude estimation based on image information still cannot meet the accuracy requirements. Multi-sensor fusion is an important development direction for visual positioning. Visual sensors and inertial measurement units (IMUs) are highly complementary and both have relatively low prices. The visual / inertial fusion method has a wide range of applications. The earliest visual / inertial system was proposed by Anastasios Mourikis and others at the University of California, Riverside in 2007, and a series of classic algorithms such as VI-ORB, VINS-Mono, and ORB-SLAM3 have emerged since then.

[0005] The above technologies are used almost exclusively in situations where there is sufficient light. Autonomous positioning technology for unmanned systems based on vision in extreme lighting conditions such as low light and complete darkness is still in the exploratory stage. At present, the mainstream method is to use infrared or infrared fusion visible light strategy. However, research at home and abroad is still in the theoretical research stage, and no all-day autonomous positioning system that can meet the requirements of real-time and stable operation has been implemented.

[0006] In 2016, Borges, Vidas, and others proposed an infrared visual odometer based on optical flow tracking to detect known objects, thereby constraining the scale of monocular navigation. However, since the algorithm does not optimize the map, the positioning error is large. In 2017, Chen et al. proposed a visual odometer calculation method RGB-T that integrates infrared images and visible light images. Infrared images are used during the day to improve the accuracy of visible light. Infrared features can work independently at night, and then FAST corner points (“Features from Accelerated Segment Test” corner point detection algorithm) are extracted from the image to track and realize the fusion of heterogeneous image observation information. In 2020, Zhao et al. proposed an infrared feature extraction network ThermalPoint based on the existing visible light feature extraction network Superpoint, which effectively solves the problems of low texture and luminosity changes in infrared images. However, the algorithm cannot solve the cumulative error caused by the long-term operation of the odometer, and the author did not give the test results of outdoor operation. In 2023, Wang et al. proposed the first edge-based monocular thermal inertial odometry (ETIO), which uses edge-extracted binary images instead of original images for pose estimation to overcome the poor quality of thermal infrared images. It combines IMU pre-integration with the reprojection error of all edge feature observations to optimize the pose graph and perform real-time estimation on the sliding window of the most recent state. However, the amount of edge feature data is large, difficult to optimize and manage for a long time, and can only perform local matching. When the camera motion direction and the edge direction are approximately parallel, the edge depth cannot be initialized.

[0007] In addition, the existing mainstream visual SLAM algorithms assume that objects in the environment are stationary or have low motion, but in actual applications, most scenes are complex and dynamic. Methods such as the RANSAC algorithm (Random Sample Consensus) can solve the problems caused by dynamic targets that occupy a small area of ​​the screen and are small in number to a certain extent. However, when the number of feature points on the dynamic target is large, this method will fail. The SLAM system cannot distinguish the movement of the carrier and the object, resulting in inaccurate positioning and serious deviations in mapping, and the robustness and accuracy of the system are greatly reduced.

[0008] In 2018, Yu et al. from Tsinghua University proposed DS-SLAM, which combines the semantic segmentation network SegNet and the mobile consistency detection method to filter out the dynamic parts in the scene. Compared with ORB-SLAM2, DS-SLAM adds semantic segmentation and dense semantic octree map construction threads. The problems with DS-SLAM are: 1) The semantic segmentation time overhead is 37.6ms, which is roughly equivalent to the tracking thread, and real-time performance is difficult to guarantee; 2) The types of objects that can be identified in the semantic segmentation network are limited, which limits its application scope; 3) The external point detection method constrained by the polar line cannot find all external points. When the object moves along the polar line, this method will fail. DynaSLAM, proposed in 2018, is also based on ORB-SLAM2, adding the ability of dynamic object detection and background repair. When using RGB-D cameras, it uses a combination of multi-view geometry and deep learning to detect moving objects, and when using monocular or binocular cameras, it only uses deep learning methods. The disadvantages of this method are: 1) removing all potential dynamic points may result in too few remaining static feature points and affect the SLAM effect; 2) DynaSLAM is not real-time, and the time-consuming process is mainly due to the Region Growing algorithm. RDS SLAM proposed in 2020 is based on ORB-SLAM3, adding a semantic thread and a semantic-based optimization thread, using movement probability to update and propagate semantic information, and using a data association algorithm to remove outliers in tracking. RDS SLAM solves the real-time problem of semantic-based methods, but only removes outliers for key frames, and has limitations in removing disturbances for large and violently moving dynamic targets. Summary of the invention

[0009] The purpose of this application is to overcome the defect of failure of visible light vision / inertial fusion navigation technology in scenes with many dynamic objects under extreme lighting conditions such as dark light environments without GNSS signals or complete darkness.

[0010] In order to achieve the above objectives, this application proposes an infrared / visible light fusion all-day autonomous positioning method guided by semantic information in dynamic scenes, including:

[0011] Step 1: Send the visible light image and infrared image into the trained fusion model guided by semantic information to obtain the fused image;

[0012] Step 2: Extract feature points from the fused image;

[0013] Step 3: Fuse the fused image with the IMU data again, and remove the dynamic feature points of the fused image based on the double threshold elimination method of IMU and RGB-D camera;

[0014] Step 4: Estimate the camera pose using the remaining feature points;

[0015] Step 5: Optimize the estimated camera pose.

[0016] As an improvement of the above method, the fusion model is a generative adversarial network model.

[0017] As an improvement of the above method, the training process of the fusion model includes:

[0018] Firstly, the infrared image is used to detect the objects and foreground targets that interfere with positioning by using the target detection algorithm. The foreground target in the infrared image is taken out to obtain the foreground image, and the foreground target is deleted in the visible light image, leaving only the background image after removing the foreground target. Secondly, the infrared image and the visible light image are connected on the channel and sent to the generator to output the fused image. Thirdly, the fused image is separated into the fused foreground image and the fused background image. Finally, the fused foreground / background image and the foreground / background image before fusion are respectively sent to the foreground discriminator and the background discriminator to judge the authenticity. If they are false, they are regenerated using the generator.

[0019] As an improvement of the above method, the loss function of the fusion model includes 1 generator loss function and 2 discriminator loss functions;

[0020] The generator loss function L G for:

[0021] L G =L content +L edge +αL adversarial

[0022] Among them, L content Indicates content loss:

[0023]

[0024] Where H and W represent the image height and width respectively; ||·|| F represents the Frobenius normal form; ξ represents the weight coefficient; I ir Indicates infrared image; I vi Represents a visible light image; I f represents the fused image; Represents the gradient of each pixel in the fused image; Represents the gradient of each pixel in the visible light image;

[0025] L edge Represents the marginal loss:

[0026]

[0027] Among them, G canny Represents the weight based on the canny operator;

[0028] L adversarial Indicates that the adversarial loss is the loss function of the conventional FusionGAN model; α is the weight coefficient;

[0029] The 2 discriminator loss functions include the foreground discriminator loss function L front and background discriminator loss function L back ;

[0030] Foreground discriminator loss function L front for:

[0031]

[0032] Where N represents the number of fused images; a and b represent the foreground discriminator’s f and visible light image I vi Label; D front (I f ) and D front (I ir ) represent the discrimination results of the foreground discriminator on the fusion image and the visible light image respectively;

[0033] Background discriminator loss function L back for:

[0034]

[0035] Among them, c and d represent the background discriminator for the fusion image I f and visible light image I vi Label; D back (I f ) and D back (I vi ) represent the discrimination results of the background discriminator on the fusion image and the visible light image respectively.

[0036] As an improvement of the above method, feature points are extracted from the fused image using an ORB image feature detection and description method.

[0037] As an improvement of the above method, the fusion of the fused image and the IMU data further includes pre-integration processing of the IMU data, and the pre-integration model is:

[0038]

[0039] Among them, P wbj represents the position of the IMU data at time j in the world coordinate system; Pwbi Indicates the position of the IMU data at time i in the world coordinate system; Indicates the speed of the IMU data at time i in the world coordinate system; represents the speed of the IMU data at time j in the world coordinate system; Δt represents the time difference between time i and time j; g w Represents the gravitational acceleration in the world coordinate system; q wbi Represents the attitude quaternion of the IMU data at time i from the body coordinate system to the world coordinate system; q wbj Represents the attitude quaternion of the IMU data at time j from the body coordinate system to the world coordinate system; Pre-integrated component representing attitude change; Represents the acceleration of the IMU data in the body coordinate system at time t; Indicates the angular velocity of the IMU data in the body coordinate system; Represents the multiplication of quaternions;

[0040] The pre-integrated components of displacement change, velocity change, and attitude change are:

[0041]

[0042] in, Represents the pre-integrated quantity of displacement change; Indicates the pre-integrated amount of speed change.

[0043] As an improvement of the above method, the dual threshold rejection method of the IMU and RGB-D camera includes:

[0044] Use the data from the RGB-D camera to calculate the coordinates of the feature point p2 in the current frame in the camera coordinate system I2

[0045]

[0046] Among them, (u2, v2) represents the camera coordinate system coordinate of p2; d2 represents the depth obtained by the camera in the current frame; f x 、f y Represents the focal length of the camera in the x and y directions respectively;

[0047] The coordinates P1 of the feature point p1 matched in the previous frame in the camera coordinate system I1 cam1 (X1 cam1 ,Y1 cam1 ,Z1 cam1 ):

[0048]

[0049] Among them, (u1, v1) represents the camera coordinate system coordinate of p1; d1 represents the depth obtained by the camera in the previous frame;

[0050] The joint calibration parameters of the camera and IMU and the IMU pre-integrated components are introduced into the kinematic formula to obtain the rotation and translation matrix (R 21 ,t 21 ), calculate the coordinates P1 of the feature point p1 of the previous frame in the camera coordinate system I2 cam2 :

[0051] P1 cam2 =R 21 P1 cam1 +t 21

[0052] Calculate the Chebyshev distance ψ between two 3D points in the camera coordinate system I2:

[0053] ψ=max(|X1 cam2 -X2 cam2 |,|Y1 cam2 -Y2 cam2 |,|Z1 cam2 -Z2 cam2 |)

[0054] Among them, X1 cam2 Indicates the x coordinate of the feature point p1 in the previous frame in the camera coordinate system I2; Y1 cam2 Indicates the y coordinate of the feature point p1 in the previous frame in the camera coordinate system I2; Z1 cam2 Indicates the z coordinate of the feature point p1 of the previous frame in the camera coordinate system I2;

[0055] When the Chebyshev distance ψ is greater than the first set threshold, the feature point is judged to be a dynamic feature point;

[0056] When the number of dynamic feature points in a target is greater than a second set threshold, the target is determined to be a dynamic target, and all feature points in the target are removed.

[0057] As an improvement of the above method, the camera pose is estimated by using the remaining feature points, specifically by using a nonlinear optimization PnP method to estimate the camera pose.

[0058] As an improvement to the above method, the estimated camera pose is optimized, specifically an ORB-SLAM3 processing method.

[0059] The present application also provides an infrared / visible light fusion all-day autonomous positioning system guided by semantic information in dynamic scenes, which is implemented based on the above method, and the system includes:

[0060] The fusion image module is used to feed the visible light image and the infrared image into the trained fusion model guided by semantic information to obtain a fused image;

[0061] A feature point extraction module is used to extract feature points from the fused image;

[0062] The dynamic feature point elimination module is used to fuse the fused image with the IMU data and eliminate the dynamic feature points of the fused image based on the dual threshold elimination method of the IMU and RGB-D camera;

[0063] A camera pose estimation module is used to estimate the camera pose using the remaining feature points;

[0064] The optimization module is used to optimize the estimated camera pose.

[0065] Compared with the prior art, the advantages of this application are:

[0066] 1. Infrared images and visible light images often contain various complementary information. Especially at night, infrared images can filter out the influence of lights and accurately highlight targets such as pedestrians and vehicles whose surface temperature is greatly different from the ambient temperature. At the same time, visible light images also contain more texture information. Traditional fusion methods usually calibrate fusion parameters based on the observations of algorithm designers, while the present invention pre-marks objects that need to be focused on in the training set, and uses semantic information to guide the fusion of infrared and visible light, which can make the fusion more purposeful, and the fused image can better detect potential dynamic targets.

[0067] 2. Traditional SLAM algorithms based on feature points often have poor accuracy in dynamic scenes. This paper proposes a dual-threshold feature point elimination method based on IMU and RGB-D camera. The real dynamics of the object are judged by calculating the Chebyshev distance of the three-dimensional coordinates of the matching feature points in the unified coordinate system and the number of dynamic feature points of the same target. All feature points of dynamic targets are eliminated to avoid the influence of feature points with mismatches and unclear moving ranges on the accuracy of dynamic targets.

[0068] 3. When the external lighting conditions change, a single sensor cannot handle all working scenarios. The present invention uses a semantic information-guided generative adversarial network to fuse infrared and visible light images, which can better detect external targets, not only avoid the influence of dynamic targets, but also enable visual autonomous positioning technology to work all day long.

[0069] 4. Based on the traditional visual odometer, the present invention uses a generative adversarial network guided by semantic information to fuse infrared and visible light images, and uses a dual-threshold dynamic target feature point elimination strategy based on IMU and RGB-D camera to effectively avoid the influence of dynamic objects on the visual positioning accuracy, and improve the accuracy of autonomous positioning throughout the day, especially in nighttime environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 The figure shows the schematic diagram of the infrared and visible light fusion image training process;

[0071] Figure 2 Shown is a schematic diagram of dynamic feature points;

[0072] Figure 3 The figure shows a schematic diagram of eliminating objects in motion;

[0073] Figure 4 The figure shows the flow chart of double threshold dynamic object feature point elimination;

[0074] Figure 5 Shown is a schematic diagram of reprojection error;

[0075] Figure 6 Shown is a flow chart of the infrared / visible light fusion all-day autonomous positioning method guided by semantic information in dynamic scenes. DETAILED DESCRIPTION

[0076] The technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0077] In order to solve the problem of failure of visible light vision / inertial fusion navigation technology in scenes with many dynamic objects under extreme lighting conditions such as dark light environments without GNSS signals or complete darkness, this application introduces visual navigation based on infrared features into the traditional visual inertial fusion navigation system, and uses the semantic information of dynamic objects to guide the fusion of infrared and visible light. The image fusion problem is modeled as an adversarial game problem between the generator and the discriminator using a framework based on generative adversarial networks (GANs). The discriminator is used to force the generator to generate infrared enhanced images whose probability distribution of the fusion results is consistent with the target distribution. Based on the fused image features, the feature point method is used for feature tracking, and the pose state quantity is optimally estimated by combining IMU data with the extracted image features.

[0078] The present invention adopts a visual mileage calculation framework with tight coupling of RGB-D camera, infrared camera and IMU. First, a model of infrared and visible light fusion guided by semantic information is trained, and the visible light image and infrared image are sent to the model to obtain a fused image. The fused image contains the features of infrared and visible light images, which is more convenient for the target recognition algorithm to detect potential dynamic objects. The front end of the visual odometer extracts feature points, and then uses target detection algorithms such as YOLOv8 to detect targets with potential motion, uses geometric constraints to judge the true dynamics of the target, removes feature points on dynamic targets, and fuses the remaining feature points with IMU information to estimate the rough camera motion. The back end is consistent with ORB-SLAM3, and the posture is optimized again through BA (Bundle Adjustment, bundle adjustment), and finally obtains a posture estimation result with higher accuracy.

[0079] Example 1

[0080] like Figure 6 As shown, the infrared / visible light fusion all-day autonomous positioning method guided by semantic information in dynamic scenes provided by this application includes:

[0081] Step 1: Send the visible light image and infrared image into the trained fusion model guided by semantic information to obtain the fused image.

[0082] The training process of the infrared and visible light image fusion model is as follows: Figure 1 As shown in Figure 2, the fusion model is a generative adversarial network model. First, the infrared image I ir YOLOv8 and other target detection algorithms are used to detect objects such as people and cars that interfere with positioning, as well as foreground targets. The foreground targets detected in the infrared image are taken out to obtain the foreground image. vi Delete the target accordingly, and only keep the background image after removing the target; then, connect the infrared image and the visible light image on the channel, send them to the generator, and output the fused image; again, separate the fused image into the fused foreground image and the fused background image using the above method; finally, send the fused foreground / background image and the foreground / background image before fusion to the foreground discriminator and the background discriminator to judge the authenticity. If it is false, let the generator regenerate it. In the repeated generation confrontation, the generated image is consistent with the image we need in probability distribution.

[0083] The loss function of the fusion model is divided into the generator loss function and two discriminator loss functions. The generator loss function is defined as:

[0084] L G =L content +L edge +αL adversarial

[0085] Among them, the content loss L content Make the pixel intensity of the fused image similar to that of the infrared image, and the gradient change similar to that of the visible light image. edge Preserve the edge information of the fused image. Adversarial loss L adversarial The definition is related to the discriminator. It is the loss function of the conventional FusionGAN model, which helps the training generator to "fool" the discriminator. α is the weight coefficient.

[0086] The content loss is defined as follows:

[0087]

[0088] Where H and W represent the image height and width, respectively, ||·|| F represents the Frobenius normal form, ξ represents the weight coefficient, I f represents the fused image, Represents the gradient of each pixel in the fused image, Represents the gradient of each pixel in the visible light image. The first term aims to preserve the infrared information in the fused image, and the second term aims to preserve the gradient information contained in the visible light image.

[0089] The marginal loss is defined as:

[0090]

[0091] The definition of edge loss is based on the first term of content loss, which retains infrared information and uses the weight G based on the canny operator. canny , making the training more focused on the edge information based on infrared images.

[0092] The discriminator is designed to distinguish the fused image from the foreground / background images based on the features extracted from the fused image and the visible background image / infrared foreground image.

[0093] The loss functions of the foreground and background discriminators are defined as:

[0094]

[0095] Where N represents the number of fused images, a, c and b, d represent the discriminator’s f and visible light image I vi Label, D · (I f ) and D · (I vi ) represent the discrimination results of the current discriminator on the fusion image and the visible light image respectively.

[0096] In this application, "semantic information guidance" refers to the use of target detection algorithms to detect objects such as people and cars as semantic information, that is, "possibly moving objects". In generative adversarial fusion, the discriminator focuses on whether the fused image can detect the target well.

[0097] Step 2: Extract feature points from the fused image.

[0098] Feature point extraction can be performed using methods such as ORB. The first step of ORB (Oriented Fast and Rotated Brief) feature detection uses the Oriented FAST corner detection algorithm to find key points in the image, and then uses the improved BRIEF descriptor to describe the image area around the feature point (key point).

[0099] Oriented FAST corner points add scale and rotation descriptions based on FAST corner points by constructing image pyramids and grayscale centroid method. The FAST corner point detection process is as follows:

[0100] (1) Select pixel p in the image, whose brightness is I p ;

[0101] (2) Setting the threshold T;

[0102] (3) Taking pixel p as the center, select 16 pixels on a neighboring circle with a radius of 3;

[0103] (4) If the brightness of N consecutive points in the adjacent circle exceeds the threshold T, then the pixel p is a feature point;

[0104] (5) Traverse all pixels in the image and repeat the above steps;

[0105] (6) Use non-maximum suppression in areas where feature points are concentrated to avoid clustering of feature points.

[0106] The feature point scale information is obtained by constructing an image pyramid. The input image is downsampled step by step to obtain an image pyramid, the bottom layer of the pyramid is the original image, and the image resolution decreases by 1 / 1.2 with each additional layer. The number of pyramid layers set by the present invention is 8. When matching features, images on different layers are matched, thereby achieving scale invariance.

[0107] The rotation information of the Oriented FAST corner point is obtained by connecting the grayscale centroid and the geometric center of the image block around the feature point. The calculation process is as follows:

[0108] (1) Define the moment of the image block as:

[0109]

[0110] Among them, B is the image block near the feature point, x, y are the horizontal and vertical coordinates of the pixel point, and I (x, y) is the gray value of the pixel point.

[0111] (2) Calculate the centroid of the image block as:

[0112]

[0113] (3) Connect the geometric center O of the image block and the center of mass C to obtain the direction vector The direction of the feature point is:

[0114] θ=arctan(m 01 / m 10 )

[0115] Through the above method, we obtain OrientedFAST corner points with scale and rotation description.

[0116] BRIEF is a binary descriptor, which consists of multiple 0s and 1s. It is easy to store and requires little computation. The description and matching process is as follows:

[0117] (1) Select N pairs of points in an area S×S around the feature point according to Gaussian distribution, and perform Gaussian smoothing on these 2N points;

[0118] (2) Compare the pixel sizes of the previous point and the next point in each pair of points. If the pixel value of the previous point is greater than that of the next point, it is taken as 1, otherwise it is taken as 0, thus obtaining an N-dimensional vector consisting of 0 and 1.

[0119] (3) Use Hamming distance for matching, that is, if the number of identical values ​​of each bit in the descriptors of two feature points is greater than the threshold, they are matched.

[0120] Step 3: The dynamic feature points of the fused image are removed by using the dual threshold removal method based on IMU and depth camera.

[0121] The fused image is fused with the IMU data again, and potential dynamic objects are detected using target detection algorithms such as YOLOv8. The dynamic feature points of dynamic targets are eliminated using the dual threshold elimination method of IMU and depth camera.

[0122] Using target detection algorithms such as YOLOv8 can only detect objects that may be moving, and the true dynamics of the target still needs further judgment, such as Figure 2 As shown. Figure 4 As shown, the dual threshold rejection method of IMU and depth camera includes:

[0123] When the target P is a dynamic feature point, the observed 3D target point is the same feature on the same dynamic target, but with different coordinates in the world coordinate system. Given the camera coordinate system coordinates (u2, v2) of the feature point p2 in the current frame, the RGB-D camera can be used to calculate its coordinates P2 in the camera coordinate system I2. cam2 :

[0124]

[0125] Among them, d2 represents the depth obtained by the depth camera.

[0126] Similarly, the coordinates of the feature point p1 matched in the previous frame in the camera coordinate system I1 are P1 cam1 :

[0127]

[0128] By bringing the joint calibration parameters of the camera and IMU and the IMU pre-integrated components into the kinematic formula, we can get the rotation and translation matrices (R 21 ,t 21 ), the coordinates of the previous frame pixel in the camera coordinate system I2 can be calculated

[0129] P1 cam2 =R 21 P1 cam1 +t 21

[0130] Calculate the Chebyshev distance between two 3D points in the camera coordinate system I2:

[0131]

[0132] When the distance is greater than the threshold, the feature point is judged to be a dynamic feature point.

[0133] When the number of dynamic feature points in a target is greater than the threshold, the target is judged as a dynamic target and all feature points in the target are removed. Figure 3 As shown, the green points are normal feature points, and the red points are removed feature points.

[0134] In the process of fusing the fused image with the IMU data, since the sampling frequency of the IMU is much higher than the acquisition frequency of the image, a large amount of IMU data will be accumulated between every two frames of images. When performing iterative optimization of data fusion, the IMU data needs to be re-integrated each time the historical state is updated, which undoubtedly increases the complexity of the calculation. In order to solve this problem, the present invention adopts IMU pre-integration technology to merge the IMU data between consecutive image frames to form a single posture transformation value. In this way, only the result of this pre-integration needs to be considered each time optimization is performed, without relying on the data of the previous frame, thereby significantly reducing the computational burden and improving the efficiency of data processing.

[0135] According to the position P, velocity V, and attitude quaternion Q at time i, P, V, and Q at time j can be obtained, and the traditional IMU integration model can be converted into a pre-integration model as follows:

[0136]

[0137] Among them, P wbj P represents the position at time j from the body coordinate system (body) to the world coordinate system (world), that is, the position in the world coordinate system; wbi Indicates the same position as above at time i; Represents the velocity in the world coordinate system at time i; represents the speed in the world coordinate system at time j; Δt represents the time difference between time i and time j; g w Represents the gravitational acceleration in the world coordinate system; q wbi The quaternion representing the attitude of the body coordinate system transferred to the world coordinate system at time i; q wbj Represents the same attitude quaternion as above at time j; Pre-integrated component representing attitude change; represents the acceleration in the body coordinate system at time t; represents the angular velocity in the body coordinate system; Represents quaternion multiplication.

[0138] The pre-integrated quantity is only related to the IMU measurement value. The pre-integrated quantity of displacement change, velocity change, and attitude change is:

[0139]

[0140] in, Indicates the pre-integrated amount (increment) of displacement change; Indicates the pre-integrated amount (increment) of speed change.

[0141] Step 4: Estimate the camera pose using the PnP method for the remaining feature points.

[0142] Since an RGB-D camera is used, the 3D position of the feature points can be determined by the depth map, so the motion can be solved directly by the nonlinear optimization PnP (Perspective-n-Point) method.

[0143] For n three-dimensional space points in an image, we want to calculate the camera's position (including rotation R and translation t), whose Lie group is represented by T. For a space point coordinate P = [X, Y, Z] T , the projection in the camera's pixel coordinate system is u=[u,v] T , the camera’s internal parameters are where f x ,f y are the focal lengths of the camera in the x and y directions, c x ,c y is the optical center translation of the camera in the x and y directions respectively. Since an RGB-D camera is used, the 3D point position can be calculated by the following formula:

[0144]

[0145] Among them, d is the actual distance from the pixel point given by the depth camera to the object corresponding to the camera.

[0146] The relationship between pixel position and spatial point position is as follows:

[0147]

[0148] Among them, s is the scale, that is, the distance from the spatial point to the camera plane.

[0149] Since the camera pose is unknown and there is observation noise, the above equation has an error e. We sum the errors of n spatial points in an image, construct a least squares problem, and optimize the camera pose:

[0150]

[0151] Among them, T * Represents the optimized Lie group, which is used to distinguish it from the original Lie group T. i Represents the scale of the i-th feature point, that is, the distance from the feature point to the camera plane.

[0152] like Figure 5 As shown in the figure, through ORB feature matching, we can know that p1 and p2 belong to the projection of spatial point P in two adjacent images, but because the camera position is unknown, the estimated initial There is an error with the actual p2. After constructing the above least squares problem, the Lie group T of the camera pose can be solved by optimization algorithms such as the Gauss-Newton method, and then the actual movement of the camera can be obtained.

[0153] Step 5: Optimize the pose again through BA (Bundle Adjustment) to finally obtain a pose estimation result with higher accuracy. This step is the same as the processing method of ORB-SLAM3.

[0154] Example 2

[0155] The present application also provides an infrared / visible light fusion all-day autonomous positioning system guided by semantic information in dynamic scenes, which is implemented based on the above method. The system includes

[0156] The fusion image module is used to feed the visible light image and the infrared image into the trained fusion model guided by semantic information to obtain a fused image;

[0157] A feature point extraction module is used to extract feature points from the fused image;

[0158] The dynamic feature point elimination module is used to fuse the fused image with the IMU data and eliminate the dynamic feature points of the fused image based on the dual threshold elimination method of the IMU and RGB-D camera;

[0159] A camera pose estimation module is used to estimate the camera pose using the remaining feature points;

[0160] The optimization module is used to optimize the estimated camera pose.

[0161] The present application may also provide a computer device, comprising: at least one processor, a memory, at least one network interface and a user interface. The various components in the device are coupled together through a bus system. It is understood that the bus system is used to achieve connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.

[0162] The user interface may include a display, a keyboard or a pointing device, such as a mouse, a trackball, a touch pad or a touch screen.

[0163] It is understood that the memory in the disclosed embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0164] In some embodiments, the memory stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system and applications.

[0165] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present disclosure can be included in the application.

[0166] In the above embodiment, the processor may also call a program or instruction stored in the memory, specifically, a program or instruction stored in an application program, and is used to:

[0167] Execute the steps of the above method.

[0168] The above method can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The above-disclosed methods, steps and logic block diagrams can be implemented or executed. The general processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the above-disclosed method can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0169] It is understood that the embodiments described in the present application can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.

[0170] For software implementation, the technology of the present application can be implemented by executing the functional modules (such as procedures, functions, etc.) of the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0171] The present application may also provide a non-volatile storage medium for storing a computer program. When the computer program is executed by a processor, each step in the above method embodiment can be implemented.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.

Claims

1. An infrared / visible light fusion all-day autonomous positioning method guided by semantic information in dynamic scenes, comprising: Step 1: Send the visible light image and infrared image into the trained fusion model guided by semantic information to obtain the fused image; Step 2: Extract feature points from the fused image; Step 3: Fuse the fused image with the IMU data again, and remove the dynamic feature points of the fused image based on the double threshold elimination method of IMU and RGB-D camera; Step 4: Estimate the camera pose using the remaining feature points; Step 5: Optimize the estimated camera pose.

2. The infrared / visible light fusion all-day autonomous positioning method guided by semantic information in dynamic scenes according to claim 1 is characterized in that: The fusion model is a generative adversarial network model.

3. The infrared / visible light fusion all-day autonomous positioning method guided by semantic information in dynamic scenes according to claim 2 is characterized in that: The training process of the fusion model includes: Firstly, the infrared image is used to detect the objects and foreground targets that interfere with positioning by using the target detection algorithm. The foreground target in the infrared image is taken out to obtain the foreground image, and the foreground target is deleted in the visible light image, leaving only the background image after removing the foreground target. Secondly, the infrared image and the visible light image are connected on the channel and sent to the generator to output the fused image. Thirdly, the fused image is separated into the fused foreground image and the fused background image. Finally, the fused foreground / background image and the foreground / background image before fusion are respectively sent to the foreground discriminator and the background discriminator to judge the authenticity. If they are false, they are regenerated using the generator.

4. The all-day autonomous positioning method of infrared / visible light fusion guided by semantic information in dynamic scenes according to claim 2, characterized in that: The loss function of the fusion model includes 1 generator loss function and 2 discriminator loss functions; The generator loss function L G for: L G =L content +L edge +αL adversarial Among them, L content Indicates content loss: Where H and W represent the image height and width respectively; ||·|| F represents the Frobenius normal form; ξ represents the weight coefficient; I ir Indicates infrared image; I vi Represents a visible light image; I f represents the fused image; Represents the gradient of each pixel in the fused image; Represents the gradient of each pixel in the visible light image; L edge Represents the marginal loss: Among them, G canny Represents the weight based on the canny operator; L adversarial Indicates that the adversarial loss is the loss function of the conventional FusionGAN model; α is the weight coefficient; The 2 discriminator loss functions include the foreground discriminator loss function L front and background discriminator loss function L back ; Foreground discriminator loss function L front for: Where N represents the number of fused images; a and b represent the foreground discriminator’s f and visible light image I vi Label; D front (I f ) and D front (I ir ) represent the discrimination results of the foreground discriminator on the fusion image and the visible light image respectively; Background discriminator loss function L back for: Among them, c and d represent the background discriminator for the fusion image I f and visible light image I vi Label; D back (I f ) and D back (I vi ) represent the discrimination results of the background discriminator on the fusion image and the visible light image respectively.

5. The infrared / visible light fusion all-day autonomous positioning method guided by semantic information in dynamic scenes according to claim 1, characterized in that: The feature points of the fused image are extracted by using an ORB image feature detection and description method.

6. The all-day autonomous positioning method of infrared / visible light fusion guided by semantic information in dynamic scenes according to claim 1, characterized in that: The fusion of the fused image and the IMU data further includes pre-integration processing of the IMU data, and the pre-integration model is: Among them, P wbj represents the position of the IMU data at time j in the world coordinate system; P wbi Indicates the position of the IMU data at time i in the world coordinate system; Indicates the speed of the IMU data at time i in the world coordinate system; represents the speed of the IMU data at time j in the world coordinate system; Δt represents the time difference between time i and time j; g w Represents the gravitational acceleration in the world coordinate system; q wbi Represents the attitude quaternion of the IMU data at time i from the body coordinate system to the world coordinate system; q wbj Represents the attitude quaternion of the IMU data at time j from the body coordinate system to the world coordinate system; Pre-integrated component representing attitude change; Represents the acceleration of the IMU data in the body coordinate system at time t; Indicates the angular velocity of the IMU data in the body coordinate system; Represents the multiplication of quaternions; The pre-integrated components of displacement change, velocity change, and attitude change are: in, Represents the pre-integrated quantity of displacement change; Indicates the pre-integrated amount of speed change.

7. The all-day autonomous positioning method of infrared / visible light fusion guided by semantic information in dynamic scenes according to claim 6, characterized in that: The dual threshold rejection method of the IMU and RGB-D camera includes: Use the data from the RGB-D camera to calculate the coordinates of the feature point p2 in the current frame in the camera coordinate system I2 Among them, (u2, v2) represents the camera coordinate system coordinate of p2; d2 represents the depth obtained by the camera in the current frame; f x 、f y Represents the focal length of the camera in the x and y directions respectively; The coordinates of the feature point p1 matched in the previous frame in the camera coordinate system I1 Among them, (u1, v1) represents the camera coordinate system coordinate of p1; d1 represents the depth obtained by the camera in the previous frame; The joint calibration parameters of the camera and IMU and the IMU pre-integrated components are introduced into the kinematic formula to obtain the rotation and translation matrix (R 21 ,t 21 ), calculate the coordinates P1 of the feature point p1 of the previous frame in the camera coordinate system I2 cam2 : P1 cam2 =R 21 P1 cam1 +t 21 Calculate the Chebyshev distance ψ between two 3D points in the camera coordinate system I2: in, Indicates the x coordinate of the feature point p1 in the previous frame in the camera coordinate system I2; Y1 cam2 Indicates the y coordinate of the feature point p1 of the previous frame in the camera coordinate system I2; Indicates the z coordinate of the feature point p1 of the previous frame in the camera coordinate system I2; When the Chebyshev distance ψ is greater than the first set threshold, the feature point is judged to be a dynamic feature point; When the number of dynamic feature points in a target is greater than a second set threshold, the target is determined to be a dynamic target, and all feature points in the target are removed.

8. The all-day autonomous positioning method of infrared / visible light fusion guided by semantic information in dynamic scenes according to claim 1, characterized in that: The method of estimating the camera pose by using the remaining feature points specifically employs a nonlinear optimization PnP method to estimate the camera pose.

9. The infrared / visible light fusion all-day autonomous positioning method guided by semantic information in dynamic scenes according to claim 1, characterized in that: The estimated camera pose is optimized, specifically an ORB-SLAM3 processing method.

10. An infrared / visible light fusion all-day autonomous positioning system guided by semantic information in dynamic scenes, implemented based on the method described in any one of claims 1 to 9, characterized in that: The system comprises: The fusion image module is used to feed the visible light image and the infrared image into the trained fusion model guided by semantic information to obtain a fused image; A feature point extraction module is used to extract feature points from the fused image; The dynamic feature point elimination module is used to fuse the fused image with the IMU data and eliminate the dynamic feature points of the fused image based on the dual threshold elimination method of the IMU and RGB-D camera; A camera pose estimation module, for estimating the camera pose using the remaining feature points; and The optimization module is used to optimize the estimated camera pose.

Images