A method and system for obtaining fused images in SLAM

Abstract

This application relates to the field of driverless positioning, and provides a method and system for obtaining a fused image in SLAM. By obtaining real-time visible light camera sensor data and thermal infrared camera sensor data; inputting the real-time visible light camera sensor data and the thermal infrared camera sensor data into a pre-established contrastive learning neural network model to obtain a fused image. The method and system for obtaining a fused image in SLAM provided by this application can achieve the acquisition of a fused image with high texture and color features, and the fused image can ensure the reliability and low latency of SLAM.

Description

Technical Field

[0001] This application relates to the field of driverless positioning, and particularly to a method and system for acquiring fused images in SLAM. Background Art

[0002] With the development of driverless systems and artificial intelligence, a fast, reliable, and efficient positioning system, as a prerequisite for subsequent tasks such as obstacle avoidance or decision-making, has become an important part of all driver assistance systems or autonomous navigation systems. Among them, Simultaneous Localization And Mapping (SLAM), as a key technology in driverless systems, can assist robots in performing tasks such as path planning, autonomous exploration, and navigation. In a dark environment, a robot starts moving from an unknown position, locates itself based on its position and the map during the movement, and at the same time builds an incremental map on the basis of its own positioning to achieve the robot's autonomous positioning and navigation.

[0003] SLAM mainly includes: sensor data, visual odometry design, backend, mapping, and loop detection. Among them, sensor data is particularly crucial. Sensor data mainly refers to various types of raw data collected in the actual environment, including laser scan data, video image data, point cloud data, etc.

[0004] In the process of acquiring sensor data, a thermal infrared camera is usually combined with a visible light camera to obtain environmental information. Among them, as a type of visual sensor, the thermal infrared camera provides rich temperature information and is less affected by changing illuminance or background clutter, making it a good complement to the visible light camera. To make full use of the characteristics of infrared that can sense temperature and form images, many scholars have conducted a series of explorations: Tarek and Beauvisage did similar work. They started from the architecture of the SLAM system itself and used the method of multi-modal stereo matching to find the correspondence between visible light images and long-wave thermal infrared images, realizing a visual odometry (VO) for pose estimation; Poujol et al. used the traditional discrete wavelet transform method to fuse infrared and visible light images and applied it to monocular visual odometry; in 2020, Beauvisage et al. proposed a new multi-modal monocular visual odometry method, which tracks features of both modalities simultaneously, but only uses the camera that provides the best tracking quality to estimate motion, and can achieve accurate monocular visual odometry without parameter tuning. On the other hand, some scholars use deep learning methods to process images of both modalities. In 2016, Choi et al. introduced a convolutional neural network to enhance low-resolution thermal infrared images and used it to achieve tasks such as pedestrian detection, visual odometry measurement, and image registration; while Sun et al. proposed the FuseSeg network to fuse visible light and thermal infrared images, and the network can output pixel-level semantic labels.

[0005] The above research work only focuses on links such as feature matching and key frame extraction in the SLAM system process, and does not pay attention to the quality of infrared and visible light images themselves; or only focuses on one index of the quality of the fused image, and the improvement of the quality of this image itself may not be helpful for SLAM. For example, reliability and low latency are important indexes to be considered in the SLAM process, and the enhanced images used for SLAM must meet high texture and color characteristics. Summary of the Invention

[0006] In order to obtain a fused image for ensuring the reliability and low latency of SLAM, the present application provides a method and system for obtaining a fused image in SLAM.

[0007] In the first aspect of the present application, a method for obtaining a fused image in SLAM is provided, including:

[0008] Obtain real-time visible light camera sensor data and thermal infrared camera sensor data.

[0009] Input the real-time visible light camera sensor data and thermal infrared camera sensor data into a pre-established contrastive learning neural network model to obtain a fused image.

[0010] The contrastive learning neural network model is obtained through the following steps:

[0011] Obtain historical thermal infrared images, historical visible light images, and historical fusion images.

[0012] Encode the historical thermal infrared images and historical visible light images into feature vectors.

[0013] Use the image patches at the same positions in the historical thermal infrared images and historical visible light images as positive samples, and the image patches at different positions as negative samples.

[0014] According to the positive and negative samples, compare the adversarial loss and contrastive loss at the levels of the positive and negative samples, and retain the positive and negative samples with low contrastive loss and adversarial loss.

[0015] Use the positive and negative samples with low contrastive loss and adversarial loss as the input of the preliminary contrastive learning neural network model, and the corresponding historical fusion image as the output of the preliminary contrastive learning neural network model, and train the preliminary contrastive learning neural network model to obtain the contrastive learning neural network model.

[0016] Optionally, in the step of encoding the historical thermal infrared images and historical visible light images into feature vectors, it is implemented through the following model:

[0017]

[0018] X is the historical thermal infrared image, Y is the historical thermal infrared image, p(x, y) is the joint probability distribution function of X and Y, p(x) is the marginal probability distribution function of X, p(y) is the marginal probability distribution function of Y, and MI(X, Y) is the historical fusion image.

[0019] Optionally, the adversarial loss is obtained through the following model:

[0020] L GAN (G, D, X, Y) = E y～Y logD(y) +

[0021] E x～X log(1 - D(G(x)))

[0022] L GAN is the adversarial loss, G is the generator of the network, D represents the discriminator, x is the image sample in the visible light domain, and y is the image sample in the infrared light domain.

[0023] Optionally, the contrastive loss is obtained through the following model:

[0024] L cyc (G, F) = E y～Y ||G(F(y)) - y||1 +

[0025] E x～X ||F(G(x)) - x|| 1

[0026] L cyc is the contrast loss, E is the expectation, G is the generator that generates the y domain from the image domain x, and F is the generator that generates the x domain from the image domain y.

[0027] The second aspect of this application provides a system for obtaining fused images in SLAM, including: an acquisition system and a processing system.

[0028] The acquisition system is used to acquire real-time visible light camera sensor data and thermal infrared camera sensor data.

[0029] The processing system is used to input the real-time visible light camera sensor data and thermal infrared camera sensor data into a pre-established contrast learning neural network model to obtain a fused image.

[0030] The contrast learning neural network model is obtained through the following steps:

[0031] Acquire historical thermal infrared images, historical visible light images, and historical fused images.

[0032] Encode the historical thermal infrared images and historical visible light images into feature vectors.

[0033] Take the image patches at the same positions in the historical thermal infrared images and historical visible light images as positive samples, and the image patches at different positions as negative samples.

[0034] According to the positive samples and negative samples, compare the adversarial loss and contrast loss at the levels of the positive samples and negative samples, and retain the positive samples and negative samples with low contrast loss and adversarial loss.

[0035] Take the positive samples and negative samples with low contrast loss and adversarial loss as the input of the preliminary contrast learning neural network model, and the corresponding historical fused image as the output of the preliminary contrast learning neural network model, and train the preliminary contrast learning neural network model to obtain the contrast learning neural network model.

[0036] Optionally, in the step of encoding the historical thermal infrared images and historical visible light images into feature vectors, it is implemented through the following model:

[0037]

[0038] X is a historical thermal infrared image, Y is a historical thermal infrared image, p(x, y) is the joint probability distribution function of X and Y, p(x) is the marginal probability distribution function of X, p(y) is the marginal probability distribution function of Y, and MI(X,Y) is the historical fusion image.

[0039] Optionally, the adversarial loss is obtained through the following model:

[0040] L GAN (G,D,X,Y) = E y～Y logD(y)+

[0041] E x～X log(1 - D(G(x)))

[0042] L GAN is the adversarial loss, G is the generator of the network, D represents the discriminator, x is the image sample in the visible light domain, and y is the image sample in the infrared light domain.

[0043] Optionally, the contrastive loss is obtained through the following model:

[0044] L cyc (G,F) = E y～Y ||G(F(y)) - y|| 1 +

[0045] E x～X ||F(G(x)) - x|| 1

[0046] L cyc is the contrastive loss, E is the expectation, G is the generator that generates the y domain from the image domain x, and F is the generator that generates the x domain from the image domain y.

[0047] As can be seen from the above technical solutions, the present application provides a method and system for obtaining a fused image in SLAM. By acquiring real-time visible light camera sensor data and thermal infrared camera sensor data; inputting the real-time visible light camera sensor data and thermal infrared camera sensor data into a pre-established contrastive learning neural network model to obtain a fused image. The contrastive learning neural network model is obtained through the following steps: acquiring historical thermal infrared images, historical visible light images, and historical fused images; encoding the historical thermal infrared images and historical visible light images into feature vectors; using the image patches at the same positions in the historical thermal infrared images and historical visible light images as positive samples, and the image patches at different positions as negative samples; according to the positive samples and negative samples, comparing the adversarial loss and contrastive loss at the levels of the positive samples and negative samples, and retaining the positive samples and negative samples with low contrastive loss and adversarial loss; using the positive samples and negative samples with low contrastive loss and adversarial loss as the input of the preliminary contrastive learning neural network model, and the corresponding historical fused image as the output of the preliminary contrastive learning neural network model, and training the preliminary contrastive learning neural network model to obtain a contrastive learning neural network model. The method and system for obtaining a fused image in SLAM provided by the present application can realize the acquisition of a fused image with high texture and color features, and the fused image can ensure the reliability and low latency of SLAM. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0049] Figure 1 It is a flowchart for obtaining a contrastive learning neural network model provided by an embodiment of the present application;

[0050] Figure 2 It is a flowchart for a method of obtaining a fused image in SLAM provided by an embodiment of the present application;

[0051] Figure 3 It is a comparison diagram of the method provided by an embodiment of the present application and the real trajectory;

[0052] Figure 4 It is a schematic structural diagram of a system for obtaining a fused image in SLAM provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] The embodiments will be described in detail below, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following examples do not represent all embodiments consistent with the present application. They are merely examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0054] See Figure 1 , which is a flowchart for obtaining a contrastive learning neural network model provided by an embodiment of the present application.

[0055] See Figure 2 , which is a flowchart for a method of obtaining a fused image in SLAM provided by an embodiment of the present application.

[0056] In a first aspect of the embodiments of the present application, a method for obtaining a fused image in SLAM is provided, including:

[0057] S1. Obtain real-time visible light camera sensor data and thermal infrared camera sensor data.

[0058] The obtained visible light camera sensor data and thermal infrared camera sensor data are obtained under low illumination conditions.

[0059] S2. Input the real-time visible light camera sensor data and thermal infrared camera sensor data into a pre-established contrastive learning neural network model to obtain a fused image.

[0060] The obtained fused image is a pseudo-visible light image. After obtaining the pseudo-visible light image, the image is input into the SLAM system, ORB features are extracted, key frames are determined, pose estimation is performed, and finally local map construction and loop detection are completed through backend optimization.

[0061] The contrastive learning neural network model is obtained through the following steps:

[0062] S101. Obtain historical thermal infrared images, historical visible light images, and historical fused images.

[0063] S102. Encode the historical thermal infrared image and the historical visible light image into feature vectors.

[0064] In the embodiments of the present application, the mutual information theory method is adopted. At the same time, an adversarial loss function and a contrastive loss function are designed in the network structure. The method uses the principle of mutual information for image generation, which can improve the similarity between the generated image and the original image.

[0065] In the step of encoding the historical thermal infrared image and the historical visible light image into feature vectors, it is implemented through the following model:

[0066]

[0067] Let X be the historical thermal infrared image, Y be the historical thermal infrared image, p(x, y) be the joint probability distribution function of X and Y, p(x) be the marginal probability distribution function of X, p(y) be the marginal probability distribution function of Y, and MI(X, Y) be the historical fusion image.

[0068] Among them, the adversarial loss is used to make the generated pseudo-visible light image visually similar to the real visible light image as much as possible.

[0069] S103: Use the image patches at the same positions in the historical thermal infrared image and the historical visible light image as positive samples, and the image patches at different positions as negative samples.

[0070] S104: According to the positive samples and negative samples, compare the adversarial loss and the contrast loss at the levels of the positive samples and negative samples, and retain the positive samples and negative samples with low contrast loss and adversarial loss.

[0071] The adversarial loss is obtained through the following model:

[0072] L GAN (G, D, X, Y) = E y～Y logD(y) +

[0073] E x～X log(1 - D(G(x)))

[0074] L GAN is the adversarial loss, G is the generator of the network, D represents the discriminator, x is the image sample in the visible light domain, and y is the image sample in the infrared light domain.

[0075] The contrast loss is obtained through the following model:

[0076] L cyc (G, F) = E y～Y ||G(F(y)) - y|| 1 +

[0077] E x～X ||F(G(x)) - x|| 1

[0078] L cyc is the contrast loss, E is the expectation, G is the generator that generates the y domain from the image domain x, and F is the generator that generates the x domain from the image domain y.

[0079] S105. Use the positive and negative samples with low contrast loss and adversarial loss as the input of the preliminary contrast learning neural network model, and the corresponding historical fused images as the output of the preliminary contrast learning neural network model, and train the preliminary contrast learning neural network model to obtain the contrast learning neural network model.

[0080] The following uses specific embodiments to illustrate the effects of the method provided by the embodiments of the present application in actual applications.

[0081] As shown in Table 1, Visible SLAM is the original nighttime visible light image, Thermal SLAM is the original infrared image, and DVT-SLAM is the pseudo-visible light image provided by the embodiments of the present application, and the effects after inputting the three groups of images into the SLAM system are compared.

[0082] In the embodiments of the present application, the absolute trajectory errors of the original nighttime visible light image, the original infrared image, and the pseudo-visible light image provided by the embodiments of the present application are calculated respectively, and the Root Mean Squared Error (RMSE) is used to statistically analyze the ATE. The first column in the table is the result of SLAM using the original nighttime visible light image. Because the original visible light image has insufficient brightness at night, problems such as insufficient number of feature points, incorrect matching, and pose drift occur during the process of extracting feature points, and a complete trajectory cannot be obtained, resulting in SLAM failure; the second column is the result of SLAM using the nighttime infrared image. The thermal infrared camera can still image at night, and complete trajectories can be obtained in sequences 2, 3, and 4, but there are still relatively large positioning and navigation errors. In the last column, using the pseudo-visible light image generated by the embodiments of the present application as the input, the positioning error has been significantly reduced compared to before, proving the effectiveness of the method proposed in this article.

[0083] Table - 1 - Image comparison experiment

[0084]

[0085] A trajectory experiment was carried out for nighttime positioning, and the experimental results show that the positioning effect is good. As Figure 3 shown, it is a comparison diagram of the method provided by the embodiments of the present application and the real trajectory. The solid line in the figure is the trajectory of the method provided by the embodiments of the present application, and the dashed line is the real trajectory.

[0086] The above results show that the fused image supplements richer RGB texture features on the basis of the edge features of the infrared image, the pose and trajectory estimation are closer to the true value, lower ATEs are obtained in all four sequences, and better SLAM positioning effects can be achieved.

[0087] As can be seen from the above technical solutions, a method for obtaining a fused image in SLAM provided by an embodiment of the present application includes obtaining real-time visible light camera sensor data and thermal infrared camera sensor data; inputting the real-time visible light camera sensor data and the thermal infrared camera sensor data into a pre-established contrastive learning neural network model to obtain a fused image. The contrastive learning neural network model is obtained through the following steps: obtaining historical thermal infrared images, historical visible light images, and historical fused images; encoding the historical thermal infrared images and the historical visible light images into feature vectors; using the image patches at the same positions in the historical thermal infrared images and the historical visible light images as positive samples, and the image patches at different positions as negative samples; according to the positive samples and the negative samples, comparing the adversarial loss and the contrastive loss at the levels of the positive samples and the negative samples, and retaining the positive samples and the negative samples with low contrastive loss and adversarial loss; using the positive samples and the negative samples with low contrastive loss and adversarial loss as the input of a preliminary contrastive learning neural network model, and the corresponding historical fused image as the output of the preliminary contrastive learning neural network model, and training the preliminary contrastive learning neural network model to obtain a contrastive learning neural network model. The method for obtaining a fused image in SLAM provided by an embodiment of the present application can achieve the acquisition of a fused image with high texture and color features, and the fused image can ensure the reliability and low latency of SLAM.

[0088] See Figure 4 , which is a schematic structural diagram of a system for obtaining a fused image in SLAM provided by an embodiment of the present application.

[0089] In a second aspect of the embodiments of the present application, a system for obtaining a fused image in SLAM is provided, including: an acquisition system and a processing system.

[0090] The acquisition system is used to obtain real-time visible light camera sensor data and thermal infrared camera sensor data.

[0091] The processing system is used to input the real-time visible light camera sensor data and the thermal infrared camera sensor data into a pre-established contrastive learning neural network model to obtain a fused image.

[0092] The contrastive learning neural network model is obtained through the following steps:

[0093] Obtain historical thermal infrared images, historical visible light images, and historical fused images.

[0094] Encode the historical thermal infrared images and the historical visible light images into feature vectors.

[0095] Use the image patches at the same positions in the historical thermal infrared images and the historical visible light images as positive samples, and the image patches at different positions as negative samples.

[0096] Based on the positive samples and negative samples, compare the adversarial loss and contrastive loss at the levels of positive samples and negative samples, and retain the positive samples and negative samples with low contrastive loss and adversarial loss.

[0097] Use the positive samples and negative samples with low contrastive loss and adversarial loss as the input of the preliminary contrastive learning neural network model, and the corresponding historical fusion images as the output of the preliminary contrastive learning neural network model, and train the preliminary contrastive learning neural network model to obtain the contrastive learning neural network model.

[0098] Input the real-time visible light camera sensor data and thermal infrared camera sensor data into the acquisition system. The acquisition system transmits the real-time visible light camera sensor data and thermal infrared camera sensor data to the processing system. The processing system inputs the real-time visible light camera sensor data and thermal infrared camera sensor data into a pre-established contrastive learning neural network model to obtain a fusion image.

[0099] As can be seen from the above technical solutions, the embodiments of the present application provide an acquisition system for fusion images in SLAM. By acquiring real-time visible light camera sensor data and thermal infrared camera sensor data; inputting the real-time visible light camera sensor data and thermal infrared camera sensor data into a pre-established contrastive learning neural network model to obtain a fusion image. The contrastive learning neural network model is obtained through the following steps: acquiring historical thermal infrared images, historical visible light images, and historical fusion images; encoding the historical thermal infrared images and historical visible light images into feature vectors; using the image patches at the same positions in the historical thermal infrared images and historical visible light images as positive samples, and the image patches at different positions as negative samples; based on the positive samples and negative samples, compare the adversarial loss and contrastive loss at the levels of positive samples and negative samples, and retain the positive samples and negative samples with low contrastive loss and adversarial loss; use the positive samples and negative samples with low contrastive loss and adversarial loss as the input of the preliminary contrastive learning neural network model, and the corresponding historical fusion images as the output of the preliminary contrastive learning neural network model, and train the preliminary contrastive learning neural network model to obtain the contrastive learning neural network model. The acquisition system for fusion images in SLAM provided by the embodiments of the present application can achieve the acquisition of fusion images with high texture and color features, and the fusion images can ensure the reliability and low latency of SLAM.

[0100] For the similar parts between the embodiments provided in the present application, reference can be made to each other. The specific embodiments provided above are only several examples under the general concept of the present application and do not constitute a limitation to the protection scope of the present application. For those skilled in the art, any other implementation manner extended based on the solution of the present application without creative efforts belongs to the protection scope of the present application.

Claims

1. A method for obtaining a fused image in SLAM, characterized in that, it includes: Obtain real-time visible light camera sensor data and thermal infrared camera sensor data; Input the real-time visible light camera sensor data and thermal infrared camera sensor data into a pre-established contrastive learning neural network model to obtain a fused image; The contrastive learning neural network model is obtained through the following steps: Obtain historical thermal infrared images, historical visible light images, and historical fused images; Encode the historical thermal infrared image and the historical visible light image into feature vectors, which is achieved through the following model: X is the historical thermal infrared image, Y is the historical thermal infrared image, p(x, y) is the joint probability distribution function of X and Y, p(x) is the marginal probability distribution function of X, p(y) is the marginal probability distribution function of Y, and MI(X,Y) is the historical fused image; Take the image patches at the same positions in the historical thermal infrared image and the historical visible light image as positive samples, and the image patches at different positions as negative samples; According to the positive and negative samples, compare the adversarial loss and contrastive loss at the level of positive and negative samples, and retain the positive and negative samples with low contrastive loss and adversarial loss; The adversarial loss is obtained through the following model: L GAN (G, D, X, Y) = E y～Y logD(y) + E x～X log(1 - D(G(x))) L GAN To counteract the loss, G is the generator of the network, D represents the discriminator, x is the image sample in the visible light domain, and y is the image sample in the infrared light domain; The contrastive loss is obtained through the following model: L cyc (G,F) = E y～Y ||G(F(y)) - y|| 1 + E x～X ||F(G(x)) - x|| 1 L cyc is the contrastive loss, E is the expectation, G is the generator that generates from the image domain x to the y domain, and F is the generator that generates from the image domain y to the x domain; Take the positive and negative samples with low contrastive loss and adversarial loss as the input of the preliminary contrastive learning neural network model, and the corresponding historical fused image as the output of the preliminary contrastive learning neural network model, and train the preliminary contrastive learning neural network model to obtain the contrastive learning neural network model.

2. A system for obtaining a fused image in SLAM, characterized in that, The system for obtaining a fused image in SLAM is used to execute the method for obtaining a fused image in SLAM according to claim 1, and includes: an acquisition system and a processing system; The acquisition system is used to obtain real-time visible light camera sensor data and thermal infrared camera sensor data; The processing system is used to input the real-time visible light camera sensor data and thermal infrared camera sensor data into a pre-established contrastive learning neural network model to obtain a fused image; The contrastive learning neural network model is obtained through the following steps: Obtain historical thermal infrared images, historical visible light images, and historical fused images; Encode the historical thermal infrared image and the historical visible light image into feature vectors, which is achieved through the following model: X is the historical thermal infrared image, Y is the historical thermal infrared image, p(x, y) is the joint probability distribution function of X and Y, p(x) is the marginal probability distribution function of X, p(y) is the marginal probability distribution function of Y, and MI(X,Y) is the historical fused image; Take the image patches at the same positions in the historical thermal infrared image and the historical visible light image as positive samples, and the image patches at different positions as negative samples; According to the positive and negative samples, compare the adversarial loss and contrastive loss at the level of positive and negative samples, and retain the positive and negative samples with low contrastive loss and adversarial loss; The adversarial loss is obtained through the following model: L GAN (G, D, X, Y) = E y～Y logD(y) + E x～X log(1 - D(G(x))) L GAN For the adversarial loss, G is the generator of the network, D represents the discriminator, x is the image sample in the visible light domain, and y is the image sample in the infrared light domain; The contrastive loss is obtained through the following model: L cyc (G,F) = E y～Y ||G(F(y)) - y|| 1 + E x～X ||F(G(x)) - x|| 1 L cyc is the contrastive loss, E is the expectation, G is the generator that generates from the image domain x to the y domain, and F is the generator that generates from the image domain y to the x domain; Use the positive and negative samples with low contrast loss and adversarial loss as the input of the preliminary contrast learning neural network model, and the corresponding historical fusion images as the output of the preliminary contrast learning neural network model, and train the preliminary contrast learning neural network model to obtain a contrast learning neural network model.

Images