Multi-modal fusion obstacle detection method and device based on artificial intelligence blind guiding

Abstract

The invention discloses a multi-modal fusion obstacle detection method based on artificial intelligence blind guiding. The method comprises the following steps: acquiring an infrared image and a color image of a scene through an infrared camera and a color camera respectively; sending acquired infrared and color bimodal images to a convolutional neural network Q1 and a convolutional neural network Q2, wherein the convolutional neural network Q1 and the convolutional neural network Q2 convert the images into a first multi-channel feature map and a second multi-channel feature map for later flattening as vectors; carrying out vectorization representation on the first multi-channel feature map and the second multi-channel feature map, and carrying out feature vector coding on a sequence of the first multi-channel feature map and the second multi-channel feature map to generate a plurality of prediction vectors; classfiying mutiple prediction vectors and making position prediction. A Transform structure is introduced in the obstacle detection process, multi-modal fusion is more effectively achieved, Transform-block is introduced, the features of infrared and color images are fully fused, and the obstacle detection precision under the low-illumination scene is improved.

Description

technical field [0001] The invention relates to the technical field of natural image processing, in particular to a multi-modal fusion obstacle detection method and device based on artificial intelligence guiding blindness. Background technique [0002] According to statistics from the China Disabled Persons' Federation, there are currently at least 5 million blind people in my country, and as the population ages, the number of blind people is also increasing year by year. "Guiding the blind" has always been a hot research topic. Before the rise of artificial intelligence, intelligent blind guide has been the solution for blind guide pursued by researchers. With the explosion of artificial intelligence in this century, this pursuit is gradually becoming a reality. The emergence of deep learning and convolutional neural networks has enabled the application of computer vision in blind guides to gradually subvert traditional blind guide technologies that rely on ultrasonic an...

AI Technical Summary

Problems solved by technology

[0005] (1) Most of the traditional non-vision only use ultrasonic and infrared sensors, and the judgment of obstacles is limited to the azimuth and distance, and the accuracy is low;

[0006] (2) Traditional machine vision mainly uses pre-written algorithms to perform feature recognition on targets in images. This method has poor migration ability and is not intelligent;

[0007] (3) The machine vision method based on deep learning learns the characteristics of images through data set training, can recognize images of various scenes, and perform target detection, and the detection effect is also very good, but in dark scenes, color images can obtain object information Few, difficult to effectively detect obstacles

[0008] (4) CNN-based multi-modal obstacle detection method can extract infrared and color dual-modal image features for fusion to better detect obstacles, but cannot fully integrate features

Images