Robot environment perception method based on multi-sensor feature fusion

Abstract

The application relates to a robot environment perception method based on multi-sensor feature fusion, and aims at the technical problems of asynchronous sampling of sensors in the prior art and how to realize feature fusion based on the difference of sensor reliability. The robot of the method performs environment perception by fusing the information of multiple sensors, optimizes the quality of sensor information, and extracts multi-level features during automatic navigation or in the process of reconnaissance on the scene, the reliability of the sensor and the hash code are multiplied, the features extracted by the hash code and the reliability of the sensor are taken as the input of a neural network, then the neural network is trained to obtain a result output, so that environment perception is realized; finally, the reliability of the sensor and the hash feature matrix are multiplied to realize feature level fusion, so that effective fusion of multiple sensors is realized, the reliability of the sensor and the hash code are multiplied, and the multiplied vector is input into the neural network to realize environment perception and obstacle identification.

Description

Technical Field

[0001] This invention relates to robot localization technology, specifically a robot environmental perception method based on multi-sensor feature fusion. Background Technology

[0002] During autonomous navigation or on-site reconnaissance, robots need to fuse information from multiple sensors for environmental perception, such as traditional cameras, infrared cameras, lidar, and temperature sensors. Based on this multi-sensor information, the data is fused to achieve global environmental perception. For example, Chinese patent application number 201910583355.0, published on September 17, 2019, entitled "Environmental Perception System and Method for Road Construction Robot Based on Multi-Source Sensors," addresses this issue. However, two problems need to be considered in this fusion process: First, the acquisition intervals of each sensor are inconsistent, leading to asynchronous sampling. How to optimally fuse the information collected by asynchronously sampled devices is a crucial problem. Second, the information collected by each sensor has discrepancies, meaning the reliability of the information varies. Finding a reliable algorithm to select reliable information is a prerequisite for achieving sensor feature fusion. Summary of the Invention

[0003] To overcome the above shortcomings, the purpose of this invention is to provide a robot environment perception method based on multi-sensor feature fusion, which mainly addresses the technical problems of asynchronous sampling by sensors in existing similar technologies, and how to achieve feature fusion based on differences in sensor reliability. This objective is achieved through the following technical solution.

[0004] A robot environment perception method based on multi-sensor feature fusion is proposed. In this method, the robot performs environmental perception by fusing information from multiple sensors during automatic navigation or reconnaissance. The reliability of the sensors and the hash code are multiplied by a dot product. The features extracted from the hash code and the reliability of the sensors are used as inputs to a neural network. After training the neural network, the output result is obtained, thereby realizing environmental perception. Finally, feature-level fusion is achieved, realizing the effective fusion of multiple sensors. The specific steps of this method are as follows: (1) Sensor information quality optimization, that is, the dimensionality reduction and reconstruction of information are achieved through variational autoencoders. The goal of variational autoencoders is to continuously optimize the loss function during the iteration process, realize data reconstruction, and obtain the transmitted data. (2) Use DenseNet to extract multi-level features from the reconstructed sensor samples; (3) Measure sensor reliability based on ensemble learning; (4) Solve the problem of asynchronous sampling based on semantic consistency among multiple sensors, that is, by finding the features with the highest semantic correlation among different sensors, use the method of co-factor matrix decomposition to learn the potential correlation semantics between multiple sensors to obtain sensor reliability and feature fusion of hash feature matrix; (5) Realize environmental perception and obstacle recognition based on sensor reliability and feature fusion of hash feature matrix; that is, multiply the sensor reliability with hash code, and input the vector of the multiplication into the neural network to realize environmental perception and obstacle recognition.

[0005] In step (1), it is assumed that the original data collected by the sensor is X. After being automatically encoded by the VAE variational autoencoder, the hidden variable Z of the original data is formed. Then, the generator realizes the reconstruction of the sample and optimizes the quality of sensor information. Through continuous iteration, the "distorted" information is corrected.

[0006] Step (2) reconstructs the sensor sample generated in step (1). DenseNet is used to extract multi-level features.

[0007] In step (3), the reliability of the sensor is measured by assessing the overall correlation between the multi-level features extracted by each sensor and the features extracted by the remaining sensors; that is, in obtaining the reliability... Based on this, the reliability is compared with a threshold, and sensors with lower reliability are eliminated.

[0008] The potential association semantic formula between multiple sensors in step (4) is as follows: ,in, This represents multi-level features, where B represents the hash code set. and It is a feature matrix extracted based on a hash code set.

[0009] The present invention features high sensor dimensionality consistency, reducing the computational complexity of feature dimensionality reduction. In particular, it fuses features from different sensors with varying reliability in asynchronous robot sampling, enabling environmental perception and obstacle recognition. It is suitable as a robot environmental perception method based on multi-sensor feature fusion, and as a technological improvement of similar methods. Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating the working principle of the dimensionality reduction and reconstruction processing of the VAE variational autoencoder of the present invention.

[0011] Figure 2 This is a block diagram illustrating the working principle of the multi-level feature extraction of DenseNet in this invention.

[0012] Figure 3 This is a block diagram illustrating the working principle of the reliability measurement of each sensor in this invention.

[0013] Figure 4 This is a block diagram illustrating the working principle of hash codes trained by multiple different sensor samples in this invention.

[0014] Figure 5 This is a block diagram illustrating the working principle of sensor reliability and hash feature matrix feature fusion in this invention. Detailed Implementation

[0015] The specific implementation steps of the present invention will now be described in further detail with reference to the accompanying drawings. Figures 1-5 As shown, the specific steps of this method are as follows:

[0016] (1) Sensor information quality optimization. Due to interference (such as video image distortion caused by high temperature in fire scenarios) or noise, the quality of information received by the sensor becomes unstable. Therefore, it is necessary to use a variational autoencoder (VAE) to reduce the dimensionality and reconstruct the information. The goal of the variational autoencoder is to continuously optimize the loss function during the iteration process to achieve data reconstruction.

[0017] Assuming the raw data acquired by the sensor is X, after automatic encoding by a VAE variational autoencoder, it forms the latent variable Z of the original data. Then, a generator reconstructs the sample. Reconstructing the sensor sample means optimizing the quality of the sensor information. Through continuous iteration, "distorted" information is corrected, such as... Figure 1 As shown.

[0018] (2) DenseNet is used to implement multi-level feature extraction of the sensor reconstruction samples. That is, the sensor reconstruction samples generated above... This paper employs DenseNet (Dense Convolutional Network) to extract multi-level features. This method differs fundamentally from traditional CNNs (Convolutional Neural Networks). It not only mitigates the vanishing gradient problem inherent in CNNs but also enhances feature transfer, thus more effectively extracting multi-level features. Extraction, such as Figure 2 As shown.

[0019] (3) Measuring sensor reliability based on ensemble learning. The environment is perceived through the feature information of each sensor. For example, if a robot uses a traditional camera to identify an obstacle as an "obstacle," it's worth considering whether other sensors can also accurately identify the obstacle. This method measures the overall correlation between the multi-level features extracted by each sensor and the features extracted by the other sensors to determine the sensor's reliability. To achieve this reliability measurement, this method uses ensemble learning to determine the reliability of each sensor for the overall result; that is, in obtaining reliability... Based on this, the reliability is compared with a threshold, and sensors with lower reliability are eliminated, such as... Figure 3 As shown.

[0020] (4) Solving the problem of asynchronous sampling based on semantic consistency among multiple sensors. Since the sampling frequency of each sensor is different, it is difficult to achieve automatic feature fusion between sensors by using synchronous sampling. This method finds the features with the highest semantic correlation among different sensors and uses the method of cooperative matrix factorization to learn the potential semantic correlation between multiple sensors:

[0021] (Formula 1)

[0022] in, This represents multi-level features, where B represents the hash code set. and It is a feature matrix extracted based on a hash code set. Through continuous iteration, each sensor will find the optimal order of information for the current time slot (one detection cycle), such as... Figure 4 As shown.

[0023] (5) Environmental perception and obstacle recognition are achieved through feature fusion based on sensor reliability and hash feature matrix. This involves multiplying the sensor reliability by the hash code and inputting the resulting vector into a neural network to achieve environmental perception and obstacle recognition, such as... Figure 5 As shown.

[0024] The sensor's reliability is multiplied by its hash code. The features extracted from the hash code and the sensor's reliability are then used as input to a neural network. After training the neural network, the output is obtained, thus achieving environmental perception. Ultimately, feature-level fusion is achieved, enabling effective fusion of multiple sensors, and thus environmental perception and obstacle recognition. Figure 5 As shown.

[0025] The innovations of this invention are as follows: (1) This invention proposes a robot environment perception method based on multi-sensor feature fusion. The algorithm of this method solves two problems: the technical problem of asynchronous sampling of sensors; and the technical problem of how to achieve feature fusion based on the different reliability of sensors. (2) Based on the multi-level feature extraction of DenseNet, the ensemble learning method is used to measure the reliability of sensors. (3) Based on the semantic consistency between multiple sensors, the semantic association of multi-sensor data is realized, solving the problem of time asynchrony caused by asynchronous sampling. (4) The hash feature matrix method is used to achieve feature fusion, solving the problem of inconsistent dimensions of various sensors and reducing the computational complexity of feature dimensionality reduction.

Claims

1. A robot environment perception method based on multi-sensor feature fusion, wherein the robot perceives the environment by fusing information from multiple sensors during autonomous navigation or reconnaissance. The reliability of the sensors is multiplied by their hash codes, and the features extracted from the hash codes and the sensor reliability are used as input to a neural network. After training the neural network, the output is obtained, thereby achieving environment perception. Ultimately, feature-level fusion is achieved, realizing effective fusion of multiple sensors; characterized in that… The specific steps of this method are as follows: (1) Sensor information quality optimization, that is, using variational autoencoders to achieve dimensionality reduction and reconstruction of information. The goal of variational autoencoders is to continuously optimize the loss function during the iteration process to achieve data reconstruction and obtain sensor reconstruction samples; (2) Using DenseNet to achieve multi-level feature extraction of sensor reconstruction samples; (3) Using ensemble learning to measure sensor reliability; (4) Solving the problem of asynchronous sampling based on semantic consistency among multiple sensors, that is, by finding the features with the highest semantic correlation among different sensors, using the method of co-factor matrix decomposition to learn the potential correlation semantics among multiple sensors to obtain sensor reliability and feature fusion of hash feature matrix; (5) Using sensor reliability and feature fusion of hash feature matrix to achieve environmental perception and obstacle recognition; that is, multiplying the sensor reliability with the hash code, and extracting the hash code The reliability of features and sensors is used as input to the neural network, and then the neural network is trained to obtain the output result, thereby realizing environmental perception and obstacle recognition; the reliability of sensors in step (3) is measured by measuring the overall correlation between the multi-level features extracted by each sensor and the features extracted by the other sensors. That is, the multi-level features of each sensor are respectively obtained by the neural network to obtain the result prediction. The result prediction is collected into the final result and the accuracy of each sensor is used to obtain the reliability of each sensor based on multiple iterations. The reliability is to judge the reliability of the sensor by comprehensively obtaining the accuracy of each sensor in each iteration; in step (4), multiple visual photos and original multimodal samples constitute multiple lidar images. Two intersection matrices and latent semantic representations are obtained through collaborative matrix decomposition, and the training data hash code is obtained through threshold exchange.

2. The robot environment perception method based on multi-sensor feature fusion according to claim 1, characterized in that... In step (1), it is assumed that the original data collected by the sensor is X. After being automatically encoded by the VAE variational autoencoder, the hidden variable Z of the original data is formed. Then, the generator realizes the reconstruction of the sample and optimizes the quality of sensor information. Through continuous iteration, the "distorted" information is corrected.

3. The robot environment perception method based on multi-sensor feature fusion according to claim 1, characterized in that... Step (2) reconstructs the sensor sample generated in step (1). DenseNet is used to extract multi-level features.

4. The robot environment perception method based on multi-sensor feature fusion according to claim 1, characterized in that... The potential association semantic formula between multiple sensors in step (4) is as follows: ,in, This represents multi-level features, where B represents the hash code set. and It is a feature matrix extracted based on a hash code set.

Images