A mobile monitoring platform based on thermal imaging cameras

Abstract

The application discloses a kind of mobile monitoring platform based on thermal imaging camera, including mobile robot mechanical chassis, host computer, multi-line laser radar and thermal imaging camera.It is suitable for mobile robot field.It solves the technical problems that it is difficult to apply monitoring platform on mobile robot, secondly, it is difficult to plan path and navigate obstacle avoidance for robot, then, it is difficult to complete target object identification and temperature extraction, finally, it is difficult to complete target object identification work using thermal imaging camera in industrial scene.The application adopts modular structure, which is convenient for disassembly and maintenance in later period, and is easy to install and use.The application adopts pine forest four-wheel chassis, which has high mechanical strength and is convenient for crossing speed bump and other road obstacles.The application adopts Leshen multi-line laser radar, which can simulate real three-dimensional environment.The application measures ambient light and temperature field using Hikvision thermal imaging camera, and can complete temperature extraction of target object using multi-modal detection method.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of mobile robots, and specifically relates to a mobile monitoring platform based on a thermal imaging camera. BACKGROUND

[0002] Mobile robots, as outstanding representatives of modern technological development, have rich technical backgrounds and application prospects behind them. Mobile robots have high autonomy and flexibility, and can perform autonomous navigation, path planning, target recognition and other operations according to different environmental and task requirements. At the same time, with the continuous development of artificial intelligence technology, mobile robots also have the ability to learn, adapt and evolve, and can continuously improve their performance and intelligent level. Thermal imaging temperature measurement technology has the advantages of non-contact temperature measurement, fast temperature measurement, high-precision temperature measurement and long-distance temperature measurement compared with traditional temperature measurement technology. In industrial production, thermal imaging temperature measurement technology can be used to detect the running state and temperature distribution of equipment, and improve the efficiency and safety of equipment operation. Multi-modal detection technology usually includes data preprocessing, feature extraction, multi-modal fusion and classification recognition steps. At present, some multi-modal large models can process multiple types of information such as text, pictures, audio and video at the same time, and exhibit higher levels of intelligence and flexibility.

[0003] The existing technology has the following problems: first, it is difficult to apply a monitoring platform on a mobile robot; second, it is difficult to plan a path for the robot and navigate to avoid obstacles; third, it is difficult to complete target object recognition and temperature extraction; and finally, it is difficult to use a thermal imaging camera to complete target object recognition in an industrial scene. SUMMARY

[0004] The present application aims to at least solve one of the technical problems existing in the prior art; for this purpose, the present application proposes a mobile monitoring platform based on a thermal imaging camera, which is used to solve the following technical problems:

[0005] First, it is difficult to apply a monitoring platform on a mobile robot; second, it is difficult to plan a path for the robot and navigate to avoid obstacles; third, it is difficult to complete target object recognition and temperature extraction; and finally, it is difficult to use a thermal imaging camera to complete target object recognition in an industrial scene.

[0006] To solve the above problems, the first aspect of the present application provides a mobile monitoring platform based on a thermal imaging camera, comprising: a mobile robot mechanical chassis, an upper computer, a multi-line laser radar and a thermal imaging camera; the mobile robot mechanical chassis, the multi-line laser radar and the thermal imaging camera are all connected to the upper computer in a wired manner; the upper computer serves as a central processing unit and is responsible for the system scheduling of the entire monitoring platform; the control of the robot mechanical chassis and the processing of thermal imaging data are completed according to the data signals uploaded by the sensors; the motion planning, obstacle avoidance and target recognition of the mobile robot are completed according to the processing results, including the following modules:

[0007] A data acquisition module: acquires thermal imaging images, visible light images and multi-line laser radar data according to sensors;

[0008] A data processing module: pre-processes the thermal imaging images and the visible light images, including denoising, image enhancement and correction; pre-processes the multi-line laser radar data, including filtering denoising, data cleaning and standardization;

[0009] A feature extraction module: extracts features from the processed thermal imaging images, visible light images and multi-line laser radar data; generates a multi-modal feature vector according to the feature extraction results;

[0010] A data analysis module: adjusts the motion state of the robot to control the mechanical chassis through a control algorithm; constructs a three-dimensional model of the environment according to the multi-line laser radar and analyzes the best path of the robot for navigation and obstacle avoidance; adopts a multi-modal detection method to identify and detect the target objects in the temperature region.

[0011] As a further scheme of the present application: according to the features extracted from the processed thermal imaging images, visible light images and multi-line laser radar data, the following steps are included:

[0012] Features are extracted from the thermal imaging images, including temperature distribution features, hot spot features and edge features; features are extracted from the visible light images, including color features, texture features and shape features; features are extracted from the multi-line laser radar data, including three-dimensional shape features, distance features and reflection intensity features.

[0013] As a further scheme of the present application: according to the feature extraction results, a multi-modal feature vector is generated, including the following steps:

[0014] The features extracted from the thermal imaging images, visible light images and multi-line laser radar data are pre-processed by normalization;

[0015] According to the pre-processed features, feature-level fusion is performed, the features extracted from the thermal imaging images, visible light images and multi-line laser radar data are spliced to generate a long feature vector;

[0016] The long feature vector is reduced in dimension by a principal component analysis method to generate a multi-modal feature vector.

[0017] As a further scheme of the present application: a three-dimensional model of the environment is constructed according to the multi-line laser radar, and the optimal path of the robot is analyzed to navigate and avoid obstacles, comprising the following steps:

[0018] After the robot is started, the multi-line laser radar is used for navigation and obstacle avoidance, and the data of the multi-line laser radar is converted into point cloud data, wherein each point represents the intersection of a laser beam and an obstacle; the point cloud data includes the position, shape and size of the obstacle;

[0019] Before the robot moves, the global map is simulated to construct a three-dimensional model of the environment; the ground point cloud and the non-ground point cloud are extracted by processing the point cloud data; a two-dimensional grid map is obtained by mapping the three-dimensional point cloud map;

[0020] According to the extracted ground point cloud and non-ground point cloud, the path of the robot is planned; a path planning algorithm is used to search for the optimal path from the current position to the target position while avoiding obstacles in the two-dimensional grid map;

[0021] According to the two-dimensional grid map mapped by the three-dimensional point cloud map, the obstacle avoidance effect in the three-dimensional environment is simulated, and all point clouds within the height range of the robot will be regarded as obstacles to form the forward movement channel of the robot; in the two-dimensional grid map, all point clouds within the height range of the robot are marked as obstacles;

[0022] Through the analysis formula: Y = {(x, y) | h(x, y) ≤ H}

[0023] Obstacle point set Y is obtained; wherein H represents the height of the robot, and h(x, y) represents the height of each point (x, y) in the two-dimensional grid map.

[0024] As a further scheme of the present application: the sensors used all support operation in a high-temperature environment.

[0025] As a further scheme of the present application: the internal wiring of the platform adopts a USB-CAN wiring method.

[0026] As a further scheme of the present application: a multi-modal detection method is used to identify and detect target objects in a temperature region, comprising the following steps:

[0027] According to the visible light image after collection and processing, a target detection model is used to identify or classify the target objects in the temperature region;

[0028] The target object in the visible light image is labeled using the COCO data set, and the label includes the category of the target object and the X coordinate and Y coordinate of the target object label box; the labeled data set is input into the target detection model for training;

[0029] By real-time acquisition and processing of the visible light image, the real-time visible light image is input into the trained target detection model, the target object in the real-time visible light image is recognized or classified in real time, and the X coordinate and Y coordinate of the target object label box in the visible light image are output; after the target object is recognized and detected in the visible light channel, the X coordinate and Y coordinate of the target object label box are synchronized to the thermal imaging channel; by synchronizing the labels of the visible light and thermal imaging channels, the target object is recognized in the thermal imaging channel;

[0030] According to the recognition result of the target object, the temperature value of the target object is predicted by using the deep learning model;

[0031] The multi-modal feature vector is labeled, and the labeled multi-modal feature vector is input into the deep learning model for training;

[0032] According to the trained deep learning model, the real-time multi-modal feature vector is extracted by real-time acquisition and processing of the thermal imaging image, the visible light image and the multi-line laser radar data; the real-time multi-modal feature vector is input into the trained deep learning model to predict the temperature value of the target object and output the result;

[0033] According to the recognition result of the target object, the actual temperature value of the target object is read in real time by the thermal imaging camera; the actual temperature value is compared and corrected with the predicted temperature value to obtain the temperature value of the target object.

[0034] As a further scheme of the present application, it comprises: a mobile monitoring platform whose mechanical structure is directly powered by a direct current power supply to power each sensor.

[0035] Compared with the prior art, the present application has the following advantages:

[0036] The present application adopts a modular structure, which is convenient for disassembly and maintenance in the later period, and is convenient to install and use. The pine four-wheel chassis has high mechanical strength and is convenient for crossing roadblocks such as speed reduction belts. The Laiseen multi-line laser radar can simulate a real three-dimensional environment. The Hikvision thermal imaging camera measures the ambient light and temperature field, and the multi-modal detection method with the same resolution can complete the temperature extraction of the target object. The USB-CAN wiring method avoids errors caused by various electromagnetic interferences in industrial scenes. The mobile power supply avoids the complex application procedures caused by the factory area wiring, and can also flexibly select the measurement position, and has certain improvement in safety. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only aim to some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0038] Fig. 1 The mobile monitoring flowchart of the present application;

[0039] Fig. 2 The structural schematic diagram of the present application;

[0040] Fig. 3 The monitoring result example diagram of the present application. DETAILED DESCRIPTION

[0041] The technical solutions of the present application will be described clearly and completely in combination with the embodiments. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort belong to the protection scope of the present application.

[0042] Please refer to Figs. 1-3 The first aspect embodiment of the present application provides a mobile monitoring platform based on a thermal imaging camera, which comprises a mobile robot mechanical chassis, an upper computer, a multi-line laser radar and a thermal imaging camera. The mobile robot mechanical chassis, the multi-line laser radar and the thermal imaging camera are all connected to the upper computer in a wired manner. The upper computer serves as a central processing unit and is responsible for the system scheduling of the whole monitoring platform. The control of the robot mechanical chassis and the processing of the thermal imaging data are completed according to the data signals uploaded by the sensors. The motion planning, obstacle avoidance and target recognition of the mobile robot are completed according to the processing results, which comprises the following modules.

[0043] Data acquisition module: thermal imaging images, visible light images and multi-line laser radar data are collected according to the sensors;

[0044] Data processing module: the thermal imaging images and the visible light images are preprocessed, including denoising, image enhancement and correction. The multi-line laser radar data are preprocessed, including filter denoising, data cleaning and standardization.

[0045] Feature extraction module: features are extracted from the processed thermal imaging images, visible light images and multi-line laser radar data. Multi-modal feature vectors are generated according to the feature extraction results.

[0046] The data analysis module: controls the motion state of the robot through the control algorithm to control the mechanical chassis; constructs a three-dimensional model of the environment according to the multi-line laser radar and analyzes the best path of the robot for navigation and obstacle avoidance; adopts a multi-modal detection method to identify and detect the target object in the temperature area.

[0047] Specifically, the power of the mobile robot pine forest four-wheel chassis, the host computer, the Laishen multi-line laser radar and the Hikvision thermal imaging camera is turned on, the host computer is started, and the monitoring platform is loaded and run. Set the target point in the host computer. Configure the working parameters of the thermal imaging camera, visible light camera and multi-line laser radar. The multi-line laser radar starts to scan the surrounding environment to generate a three-dimensional model of the environment. The host computer pre-processes the laser radar data through the data processing module, including filtering and denoising, data cleaning and standardization. According to the three-dimensional model, the data analysis module analyzes and calculates the best path from the current position to the target point. The host computer adjusts the motion state of the robot through the control algorithm to control the mechanical chassis to move forward according to the planned path. In the process of moving forward, the environment model is constantly updated through the laser radar data, and the path is adjusted to avoid new obstacles. When the robot reaches the target point, the host computer receives the position confirmation signal. The host computer controls the gimbal of the camera to lift up and adjusts the angle of the camera to cover the target area. Start the data acquisition function of the environment light and temperature field dual channel. The thermal imaging camera and the visible light camera start collecting image data at the same time. The data acquisition module uploads the collected thermal imaging image, visible light image and multi-line laser radar data to the host computer. The data processing module performs preprocessing operations such as denoising, image enhancement and correction on the image data. The laser radar data is further filtered and denoised and cleaned. The feature extraction module extracts features from the processed thermal imaging image, visible light image and multi-line laser radar data and generates a multi-modal feature vector. The data analysis module uses a multi-modal detection method to identify and detect the target object in the temperature area. Use deep learning algorithm or image processing algorithm for target detection. After completing the target detection, the camera reading is completed, the host computer can plan the return path, control the robot to return to the initial target point and save the monitoring data to the folder of the current date.

[0048] In one embodiment of the present application, the features extracted from the processed thermal imaging image, visible light image and multi-line laser radar data include the following steps:

[0049] The features extracted from the thermal imaging image include temperature distribution features, hot spot features and edge features; the features extracted from the visible light image include color features, texture features and shape features; the features extracted from the multi-line laser radar data include three-dimensional shape features, distance features and reflection intensity features.

[0050] Specifically, features are extracted from the thermal images, including but not limited to: temperature distribution features, hot spot features, and edge features; by using image processing software or algorithms, the thermal images are analyzed for temperature field, and the temperature distribution features of different regions in the images are obtained through color mapping or numerical representation. By using segmentation algorithms in image processing, hot spot regions are identified and extracted. Edges are the boundary regions of temperature changes in thermal images, and edge detection algorithms are used to extract edge features in the images. Features are extracted from visible light images, including but not limited to: color features, texture features, and shape features; the visible light images are converted from RGB color space to a color space more suitable for color feature extraction, including: HSV or Lab color space. Color histogram, color moment, or color aggregation vector methods are used to extract color features in the images. Texture analysis algorithms, including: gray level co-occurrence matrix, local binary pattern, etc., are used to extract texture features in the images. Shape analysis algorithms, including: contour extraction, shape matching, or shape descriptor, etc., are used to extract shape features in the images. Features are extracted from multi-line laser radar data, including but not limited to: three-dimensional shape features, distance features, and reflection intensity features. Point cloud data of the laser radar is used to construct a three-dimensional model or a grid. Three-dimensional shape analysis algorithms, including: three-dimensional reconstruction, surface fitting, or shape matching, etc., are used to extract three-dimensional shape features. Laser radar can directly measure the distance between the object and the sensor, and by processing the point cloud data of the laser radar, distance features between different points are extracted. The reflection intensity of the laser radar is related to the surface material and roughness of the object, and by processing the point cloud data of the laser radar, the reflection intensity features of each point are extracted.

[0051] In one embodiment of the present application, a multi-modal feature vector is generated according to the feature extraction results, including the following steps:

[0052] The features extracted from the thermal images, visible light images, and multi-line laser radar data are normalized for preprocessing;

[0053] According to the preprocessed features, feature-level fusion is performed to splice the features extracted from the thermal images, visible light images, and multi-line laser radar data, generating a long feature vector;

[0054] By principal component analysis method, the long feature vector is reduced in dimension to generate a multi-modal feature vector.

[0055] Specifically, the extracted features are normalized to eliminate the dimensional differences and numerical range differences between different features. The normalized thermal imaging image features, visible light image features, and multi-line laser radar data features are spliced to generate a long feature vector. The spliced long feature vector is used as the input data set of principal component analysis. The data set is subjected to principal component analysis to find the main components or feature directions in the data. The number of retained principal components is determined by calculating the variance contribution rate of each principal component. Principal components with higher variance contribution rates can be selected to retain most of the information of the original data. According to the determined number of retained principal components, the long feature vector is subjected to dimension reduction processing to generate a multi-modal feature vector.

[0056] In one embodiment of the present application, a three-dimensional model of the environment is constructed based on the multi-line laser radar, and the optimal path of the robot is analyzed for navigation and obstacle avoidance, including the following steps:

[0057] After the robot is started, the multi-line laser radar is used for navigation and obstacle avoidance, and the data of the multi-line laser radar is converted into point cloud data, wherein each point represents the intersection of a laser beam and an obstacle; the point cloud data includes the position, shape, and size of the obstacle;

[0058] Before the robot moves, the global map is simulated to construct a three-dimensional model of the environment; ground point clouds and non-ground point clouds are extracted by processing the point cloud data; a two-dimensional grid map is obtained by mapping the three-dimensional point cloud map;

[0059] According to the extracted ground point clouds and non-ground point clouds, the path of the robot is planned; a path planning algorithm is used to search for the best path in the two-dimensional grid map from the current position to the target position while avoiding obstacles;

[0060] According to the two-dimensional grid map mapped by the three-dimensional point cloud map, the obstacle avoidance effect in the three-dimensional environment is simulated, and all point clouds within the height range of the robot will be regarded as obstacles to form the forward movement channel of the robot; in the two-dimensional grid map, all point clouds within the height range of the robot are marked as obstacles;

[0061] The formula Y = {(x, y) | h(x, y) ≤ H} is analyzed

[0062] Obstacle point set Y is obtained; wherein H represents the height of the robot, and h(x, y) represents the height of each point (x, y) in the two-dimensional grid map.

[0063] Specifically, before the robot moves, the global map needs to be built, the SLAM part adopts the open source Lego Loam algorithm, before the algorithm is compiled and run, the laser radar and the inertial navigation need to be jointly calibrated, here, the li dar_align is used for external parameter calibration. After the Lego Loam mapping, the ground point cloud and the non-ground point cloud can be clearly extracted; the path planning part adopts the DWA path planning algorithm combined with the NDT positioning algorithm, according to the two-dimensional grid map mapped by the three-dimensional point cloud map, the obstacle avoidance effect in the simulated three-dimensional environment can be achieved, in the mapping process, the octmap function package is used to straight-through filter the height of the robot, the point cloud higher than the robot will be filtered out, the remaining point cloud is mapped downward, the point cloud within the height range of the robot will be regarded as an obstacle, so that the forward motion channel of the robot is formed. The NDT algorithm as the positioning algorithm in the forward movement of the robot can have more position information than the two-dimensional positioning algorithm.

[0064] In one embodiment of the present application, the sensors used all support high-temperature operation.

[0065] Specifically, the sensors used all support high-temperature operation, and can adapt to most factory environments.

[0066] In one embodiment of the present application, the platform internal wiring adopts a USB-CAN wiring method.

[0067] Specifically, the USB-CAN wiring method is adopted, which avoids errors caused by various electromagnetic interferences in industrial scenarios.

[0068] In one embodiment of the present application, a multi-modal detection method is used to identify and detect target objects in the temperature region, including the following steps:

[0069] According to the visible light image after collection and processing, the target detection model is used to identify or classify the target objects in the temperature region;

[0070] The COCO data set is used to label the target objects in the visible light image, and the labeling includes the category of the target object and the X and Y coordinates of the target object labeling box; the labeled data set is input into the target detection model for training;

[0071] The visible light image collected and processed in real time is input into the trained target detection model, the target in the visible light image is recognized or classified in real time, and the X coordinate and Y coordinate of the target marking box in the visible light image are output; after the target in the visible light channel is recognized and detected, the X coordinate and Y coordinate of the target marking box are synchronized to the thermal imaging channel; the target in the thermal imaging channel is recognized through the synchronization of the marking of the visible light and thermal imaging channels;

[0072] According to the recognition result of the target, the temperature value of the target is predicted by using a deep learning model;

[0073] The multi-modal feature vector is labeled, and the labeled multi-modal feature vector is input into the deep learning model for training;

[0074] According to the trained deep learning model, the real-time multi-modal feature vector is extracted by collecting and processing the thermal imaging image, the visible light image and the multi-line laser radar data in real time; the real-time multi-modal feature vector is input into the trained deep learning model to predict the temperature value of the target and output the result;

[0075] According to the recognition result of the target, the actual temperature value of the target is read in real time by the thermal imaging camera; the actual temperature value is compared and corrected with the predicted temperature value to obtain the temperature value of the target.

[0076] Specifically, the target in the visible light image is labeled by using a COCO data set through the visible light image collected and processed, and the labeling includes the category of the target and the X coordinate and Y coordinate of the target marking box. A lightweight framework Darknet and a Yo l oV3 detection algorithm are used to train the labeled data set, and a target detection model is obtained. The visible light image is preprocessed, such as denoising and enhancement, to improve the target detection effect. The real-time collected visible light image is input into the trained target detection model to recognize and classify the target. The category of the target and the X coordinate and Y coordinate of the marking box are output. The X coordinate and Y coordinate of the target marking box recognized in the visible light channel are synchronized to the thermal imaging channel to realize the recognition of the target in the thermal imaging image. According to the recognition result of the target and the multi-modal feature vector, the temperature value of the target is predicted by using a deep learning model. At the same time, the actual temperature value of the target is read in real time by using a half-sphere thermal imaging camera of Hikvision. The predicted temperature value is compared and corrected with the actual temperature value to improve the accuracy of temperature measurement.

[0077] In one embodiment of the present application, the mobile monitoring platform includes a direct current power supply provided by the mechanical structure thereof, which directly supplies power to each sensor.

[0078] Specifically, the mobile monitoring platform has a direct current power supply, which can directly power each sensor, avoiding a pull line, and improving safety and convenience.

[0079] The above embodiments are only used to illustrate the technical method of the present application but not limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical method of the present application.

Claims

1. A thermal camera based mobile monitoring platform, comprising: The mobile robot mechanical chassis, the host computer, the multi-line laser radar and the thermal imaging camera; the mobile robot mechanical chassis, the multi-line laser radar and the thermal imaging camera are connected to the host computer in a wired manner; the host computer serves as a central processing unit and is responsible for system scheduling of the entire monitoring platform; the control of the robot mechanical chassis and the processing of the thermal imaging data are completed according to the data signals uploaded by the sensors; the motion planning, obstacle avoidance and target object recognition of the mobile robot are completed according to the processing results, characterized by comprising the following modules: A data acquisition module: collecting thermal imaging images, visible light images and multi-line laser radar data according to sensors; A data processing module: pre-processing the thermal imaging images and the visible light images, including denoising, image enhancement and correction; pre-processing the multi-line laser radar data, including filtering denoising, data cleaning and standardization; A feature extraction module: extracting features from the processed thermal imaging images, visible light images and multi-line laser radar data; generating a multi-modal feature vector according to the feature extraction results; A data analysis module: adjusting the motion state of the robot to control the mechanical chassis through a control algorithm; constructing a three-dimensional model of the environment according to the multi-line laser radar and analyzing the best path of the robot for navigation and obstacle avoidance; using a multi-modal detection method to identify and detect target objects in the temperature region; A multi-modal detection method is used to identify and detect target objects in the temperature region, including the following steps: According to the collected and processed visible light images, a target detection model is used to identify or classify target objects in the temperature region; Using the COCO dataset, the target objects in the visible light images are labeled, including the category of the target object and the X and Y coordinates of the target object bounding box; the labeled dataset is input into the target detection model for training; Through real-time collection and processing of the visible light images, the real-time visible light images are input into the trained target detection model to identify or classify the target objects in the visible light images, and the X and Y coordinates of the target object bounding box in the visible light images are output; after identifying and detecting the target object in the visible light channel, the X and Y coordinates of the target object bounding box are synchronized to the thermal imaging channel; by synchronizing the labels of the visible light and thermal imaging channels, the target object is identified in the thermal imaging channel; According to the identification result of the target object, a deep learning model is used to predict the temperature value of the target object; Label the multi-modal feature vector and input the labeled multi-modal feature vector into the deep learning model for training; According to the trained deep learning model, real-time multi-modal feature vectors are extracted by real-time collection and processing of thermal imaging images, visible light images and multi-line laser radar data; the real-time multi-modal feature vectors are input into the trained deep learning model to predict the temperature value of the target object and output the result; According to the identification result of the target object, the actual temperature value of the target object is read in real time by the thermal imaging camera; the actual temperature value is compared and corrected with the predicted temperature value to obtain the temperature value of the target object.

2. The mobile monitoring platform based on thermal camera according to claim 1, characterized in that, According to the processed thermal imaging image, visible light image and multi-line laser radar data, the features are extracted, including the following steps: From the thermal imaging image, the features are extracted, including: temperature distribution feature, hot spot feature and edge feature; From the visible light image, the features are extracted, including: color feature, texture feature and shape feature; From the multi-line laser radar data, the features are extracted, including: three-dimensional shape feature, distance feature and reflection intensity feature.

3. The mobile monitoring platform based on thermal camera according to claim 2, characterized in that, According to the feature extraction result, a multi-modal feature vector is generated, including the following steps: The features extracted from the thermal imaging image, visible light image and multi-line laser radar data are normalized for preprocessing; According to the preprocessed features, feature-level fusion is performed, and the features extracted from the thermal imaging image, visible light image and multi-line laser radar data are spliced to generate a long feature vector; Through principal component analysis method, the long feature vector is reduced in dimension to generate a multi-modal feature vector.

4. The mobile monitoring platform based on thermal camera according to claim 1, characterized in that, According to the multi-line laser radar, a three-dimensional model of the environment is constructed and the best path of the robot is analyzed for navigation and obstacle avoidance, including the following steps: After the robot is started, navigation and obstacle avoidance are performed through the multi-line laser radar, and the data of the multi-line laser radar is converted into point cloud data, wherein each point represents an intersection of a laser beam and an obstacle; The point cloud data includes: the position, shape and size of the obstacle; Before the robot moves, the global map is simulated to construct a three-dimensional model of the environment; By processing the point cloud data, ground point cloud and non-ground point cloud are extracted; According to the three-dimensional point cloud map, a two-dimensional grid map is obtained; According to the extracted ground point cloud and non-ground point cloud, the path of the robot is planned; Using the path planning algorithm, the best path from the current position to the target position while avoiding obstacles in the two-dimensional grid map is searched; According to the two-dimensional grid map mapped by the three-dimensional point cloud map, the obstacle avoidance effect in the three-dimensional environment is simulated, and all point clouds within the height range of the robot will be regarded as obstacles to form the forward movement channel of the robot; In the two-dimensional grid map, all point clouds within the height range of the robot are marked as obstacles. By analyzing the formula: Obtain the set of obstacle points Where H represents the height of the robot, Represents each point in a two-dimensional raster map The height.

5. The thermal camera based mobile monitoring platform of claim 1, wherein, Including: The sensors used all support operation in high temperature environment.

6. The thermal camera based mobile monitoring platform of claim 1, wherein, Including: The internal wiring of the platform adopts USB-CAN wiring method.

7. The thermal camera based mobile monitoring platform of claim 1, wherein, Including: The mobile monitoring platform has a direct current power supply in its mechanical structure, which directly powers each sensor.

Images