Exploration and acquisition integrated robot based on multi-modal information perception

Abstract

The invention belongs to the technical field of mining equipment, and particularly relates to an exploration and acquisition integrated robot based on multi-modal information perception. Comprising an operation bin, symmetrical crawler wheel moving mechanisms are arranged on the two sides of the operation bin, symmetrical supporting rod pieces are arranged at the tail end of the operation bin and used for stabilizing operation of equipment, and a double-fusion type mechanical arm is arranged at the upper end of the operation bin and used for grabbing and collecting ore samples; a machine vision module, a laser radar, a signal amplifier and a blower device are arranged at the front end of the operation bin, and exploration of the mine environment and construction of a three-dimensional scene are achieved. According to the invention, the temperature, humidity and gas environment in the mine are detected to the greatest extent, the operation stability can be improved through the supporting frame at the rear end of the operation bin, and the overall operation stability is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of mining equipment, and in particular relates to an integrated exploration and collection robot based on multimodal information perception. Background Art

[0002] Coal is a solid combustible mineral that is gradually formed by ancient plants buried underground and undergoing complex biochemical and physicochemical changes. The supply of coal is also related to the stability of the development of my country's industry and even the entire society. The safety of coal supply is also the most important part of my country's energy security. Although there are many mining machines at present, accidents still often occur in underground coal mine construction areas. There are dangers in many underground coal mining areas, such as gas and high-density dust, which will directly or indirectly affect the life safety of construction workers. In addition, in many underground construction areas, the environment is sometimes small and sometimes the tunnel surface is rugged, which is not suitable for the operation of large coal mining machines. Therefore, manual mining is often used, which has high labor costs and low efficiency. If you dig rashly without conducting in-depth exploration of the mine, it will endanger the life safety of construction workers to a great extent, and it may also cause a great waste of resources. It can be seen that it is of vital importance to explore the mine in advance and collect and analyze the materials in the mine. Summary of the invention

[0003] The purpose of the present invention is to provide a remote control robot that can be used for mine exploration and ore sampling, effectively protecting the life and health of personnel. At the same time, it can replace humans to work in complex environments such as small spaces and rugged ground, and provide an integrated exploration and collection robot based on multimodal information perception.

[0004] The present invention adopts the following technical scheme: an integrated exploration and collection robot based on multimodal information perception, including an operating compartment, symmetrical track wheel moving mechanisms are arranged on both sides of the operating compartment, symmetrical support rods are arranged at the end of the operating compartment to stabilize the operation of the equipment, and a double fusion mechanical arm is arranged at the upper end of the operating compartment to realize the grasping and collection of mineral samples; a machine vision module, a laser radar, a signal amplifier and a blowing device are arranged at the front end of the operating compartment to realize the exploration of the mine environment and the construction of a three-dimensional scene.

[0005] In some embodiments, a Beidou dual-mode module, a gas detection sensor, a temperature and humidity detection module, a gravity detection sensor, an air pump drive module and an air pump are installed inside the operation warehouse. The Beidou dual-mode module and the temperature and humidity detection module are installed on one side of the operation warehouse, and the gas detection sensor, the air pump drive module and the air pump are installed on one side. A gravity detection sensor and a loading box are installed at the end of the operation warehouse.

[0006] In some embodiments, the support rod includes a support frame, a stretching mechanism and a magnetic device. The magnetic device is installed on the lower surface of the stretching mechanism and the magnetic device is installed on the top of the support frame. The support frame is located inside the stretching mechanism. The stretching mechanism and the support frame are connected and retracted by a hydraulic device.

[0007] In some embodiments, the dual fusion robotic arm includes: Robotic arm base; A third mechanical arm, wherein the lower end of the third mechanical arm is mounted on the mechanical arm base and can rotate and swing around the mechanical arm base; A first mechanical arm, the first mechanical arm is mounted on the upper end of the third mechanical arm and can rotate around the third mechanical arm, and a clamping device and a punching device are provided at the front end of the first mechanical arm; The second mechanical arm is installed on the upper end of the third mechanical arm and can rotate around the third mechanical arm. A sampling device is arranged at the front end of the second mechanical arm.

[0008] In some embodiments, the clamping device and the punching device include: A first front-end server connecting plate, wherein the first front-end server connecting plate is installed at the front end of the first mechanical arm, and a fifth servo motor is installed in the mounting groove of the first front-end server connecting plate; A second standard connecting disc, the second standard connecting disc is installed at the output end of the fifth servo motor, the second standard connecting disc is connected to the right-angle fixing plates symmetrical on both sides, the part main board is installed in the gap between the two right-angle fixing plates, and the fifth hexagonal linkage wheel symmetrical on both sides is installed at the bottom of the part main board; A gear arm, wherein the gear arm is mounted on the fifth hexagonal linkage wheel, a clamping piece symmetrical on both sides is mounted on the upper end of the gear arm, and a connecting rod is hingedly connected between the main plate of the part and the clamping piece; A punching rod is installed in the opening at the front end of the component mainboard. A telescopic spring is installed in the opening of the component mainboard. A blocking device is provided and connected to the punching rod.

[0009] In some embodiments, the sampling device comprises: A second front-end server connecting plate, wherein the second front-end server connecting plate is mounted at the front end of the second mechanical arm, and a seventh servo motor is mounted in a mounting groove of the second front-end server connecting plate; The first standard connecting disc is connected to the output end of the seventh servo motor; The sampling nozzle is installed on the first standard connecting disc.

[0010] The method for using the integrated exploration and collection robot based on multimodal information perception includes: S1: Drive the integrated exploration and collection robot based on multimodal information perception into the mine and detect the gas and humidity in the mine; S2: LiDAR scans and collects point cloud data inside the mine to build a three-dimensional surface model inside the mine; S3: The operator observes the surrounding situation of the integrated exploration and collection robot based on multimodal information perception through the machine vision module. The machine vision module performs feasibility analysis on the environment and connects to an external display screen through a signal amplifier to display the environment inside the mine. S4: The dual fusion robot arm collects mineral samples. When the machine vision module detects the mineral sample in front, the drilling rod mines the mineral sample to realize the grabbing and collection of the mineral sample; the air pump driving module drives the air pump to work, and the powdered mineral sample is sucked through the sampling nozzle and transported to the loading box through the air pump pipe to complete the collection of the powdered mineral sample; S5: The gravity detection sensor measures the weight of the material in the loading box, and when it exceeds the set value, an alarm is issued to prompt.

[0011] In some embodiments, step S1 includes: S11: Beidou positioning real-time monitoring of the movement trajectory of the exploration and collection integrated robot based on multi-modal information perception; S12: The gas detection sensor detects the contents of four gases, O2, H2S, CO, and CH4, in the mine cave, and analyzes the safety of the mine cave. When the contents of the four gases reach the set values, different alarm sounds are issued to prompt; S13: The temperature and humidity detection module detects the temperature and humidity in the mine and displays the detected values.

[0012] In some embodiments, step S2 includes: Determine whether the collection site is in a high dust environment; If the collection site is not in a high dust environment, the LiDAR scans and collects point cloud data to build a three-dimensional surface model; If the collection site is in a high dust environment, turn on the blower to deal with the dust, then use the lidar to scan and collect point cloud data, and use the 3D modeling optimization method to construct a 3D surface model.

[0013] In some embodiments, the 3D modeling optimization method includes: Establish a three-dimensional space coordinate system and determine the origin, coordinate axis direction and unit length of the coordinate system; Discretize the three-dimensional space and divide it into small units for storing and processing the data collected by the sensor and subsequent modeling calculations; The fusion algorithm based on Bayesian theory fuses the data collected by the lidar at different times to model the target; Use the deep learning-based convolutional neural network denoising algorithm to filter out the noise in the working scene; Use Kalman filter algorithm and particle filter algorithm to filter out interfering sound waves; Using polygonal mesh modeling method, according to the target surface point cloud data obtained after fusion, the point cloud is connected into triangular patches to construct the target surface model; The probability density function is used to describe the distribution of dust in three-dimensional space, and finally the three-dimensional model is constructed.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention can grasp and collect mineral samples through the motor-driven dual-fusion robotic arm, realize the exploration of mine caves and three-dimensional reconstruction of terrain by using machine vision modules and lidar, adopt multiple algorithms to work together, locate its own position and posture by repeatedly observing environmental features, and simultaneously couple multiple sensors to detect the temperature, humidity and gas environment in the mine cave to the greatest extent, and improve the operation stability through the rear end support frame of the operation warehouse to ensure the overall smooth operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a structural schematic diagram of the present invention; Figure 2 It is a left side view of the present invention; Figure 3 It is a right side view of the present invention; Figure 4 It is a rear view of the present invention; Figure 5 It is a structural schematic diagram of the crawler of the present invention; Figure 6 It is a structural schematic diagram of the dual fusion type mechanical arm of the present invention; Figure 7 It is a left view of the double fusion type mechanical arm of the present invention; Figure 8 It is a right side view of the double fusion type mechanical arm of the present invention; The markings in the attached drawings are: 1-operation compartment; 2-first robotic arm; 3-end effector of the first robotic arm (grasping); 4-second robotic arm; 5-end effector of the second robotic arm (sampling); 6-moving mechanism; 7-support rod; 8-machine vision module; 9-ranging radar; 10-wheel hub; 11-connecting shaft; 12-support frame; 13-Beidou dual-mode module; 14-temperature and humidity detection module; 15-blowing device 16-gravity detection sensor; 17-charging box; 18-gas detection sensor; 19-air pump drive module; 20-air pump; 21-stretching mechanism; 22-hydraulic device; 23-magnetic device; 24-rolling bearing; 25-fixed shaft; 26-rolling bearing; 27-large gear; 28-small gear; 29-shock absorber; 30-first server fixing plate; 31-second server fixing plate; 32-third server fixing plate; 33-first servo motor; 34-first hexagonal linkage wheel; 35-server main support plate; 36-server upper base; 37-fourth server fixing Plate; 38-second servo motor; 39-second hexagonal linkage wheel; 40-arm No. 1 L; 41-arm No. 1 R; 42-support arm No. 1; 43-arm No. 2 L; 44-arm No. 2 R; 45-arm No. 3 L; 46-arm No. 3 R; 47-lower end of arm No. 4 L; 48-arm No. 4 R; 49-arm No. 5 L; 50-arm No. 5 R; 51-third hexagonal linkage wheel; 52-third servo motor; 53-support arm No. 2; 54-fourth servo motor; 55-first front end server connection plate; 56-first Five servo motors; 57-parts main board; 58-fifth hexagonal linkage wheel; 59-clamping piece; 60-connecting rod; 61-punching rod; 62-support arm No. 3; 63-sixth servo motor; 64-second front-end servo connection plate; 65-seventh servo motor; 66-first standard connection disc; 67-sampling nozzle; 68-third hexagonal linkage wheel; 69-fourth hexagonal linkage wheel; 70-second standard connection disc; 71-right-angle fixing plate; 72-gear arm; 73-sixth hexagonal linkage wheel. DETAILED DESCRIPTION

[0016] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0017] In the description of the present invention, unless otherwise specified, the terms "upper", "lower", "left", "right", "inner", "outer", "front end", "rear end", "head", "tail" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", "third" and the like are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0018] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0019] A robot for integrated exploration and collection based on multimodal information perception comprises an operation compartment 1, symmetrical crawler wheel moving mechanisms 6 are arranged on both sides of the operation compartment 1, symmetrical support rods 7 are arranged at the end of the operation compartment 1 for stabilizing the operation of the equipment, a double fusion mechanical arm is arranged at the upper end of the operation compartment 1 for grasping and collecting mineral samples; a machine vision module 8, a laser radar 9, a signal amplifier and a blower 15 are arranged at the front end of the operation compartment 1 for realizing the exploration of the mine environment and the construction of a three-dimensional scene.

[0020] Specifically, the crawler wheel moving mechanism 6 includes a wheel hub 10 and a connecting shaft 11, etc. One side of the connecting shaft 11 is connected to the wheel hub 10, and the other side is connected to the fixed shaft 25 by a rolling bearing 24. The step shaft is set up by the rolling bearing 24 to ensure the stability of the robot when moving. The rolling bearing 26 is installed at the end of the connecting shaft 11, and a large gear 27 is installed on the outside. The large gear 27 on the outside is meshed with a small gear 28, and the small gear 28 is connected to a high-power drive motor. Through meshing, the small gear 28 drives the large gear 27 to achieve the effect of deceleration and torque increase, ensuring the stability of movement. At the same time, the drive motor outputs a relatively small power to drive the mechanism to operate, realizing the driving of the robot as a whole. The front and rear crawler wheels 10 are equipped with separate high-power motors, which can be used as driving wheels to achieve peak-shifting drive and direction adjustment. One end of the connecting shaft 11 is connected to a shock absorber 29 to reduce vibration during operation and ensure stable operation.

[0021] A Beidou dual-mode module 13, a gas detection sensor 18, a temperature and humidity detection module 14, a gravity detection sensor 16, an air pump drive module 19 and an air pump 20 are installed inside the operation warehouse 1. The Beidou dual-mode module 13 and the temperature and humidity detection module 14 are installed on one side of the operation warehouse 1, and the gas detection sensor 18, the air pump drive module 19 and the air pump 20 are installed on the other side. A gravity detection sensor 16 and a loading box 17 are installed at the end of the operation warehouse 1.

[0022] Specifically, symmetrical laser radars 9 are arranged on both sides of the front end of the operation warehouse 1. The laser radars 9 are connected to the industrial controller through wiring. The laser radars 9 transmit the detected signals to the industrial controller. The industrial controller processes the input signals and transmits the processed results to the external display screen. The three-dimensional model in the mine cave can be observed through the external display screen. A machine vision module 8 is arranged in the middle of the front end of the operation warehouse 1. The machine vision module 8 is connected to the industrial controller through wiring. The machine vision module 8 transmits the detected signals to the industrial controller. The industrial controller processes the input signals and transmits the processed results to the external display screen. The scene in the mine cave can be observed through the external display screen. A signal amplifier is arranged at the lower part of the front end of the operation warehouse 1. The signal amplifier is connected to the industrial controller through wiring. The signal required for external connection is amplified to facilitate receiving the transmitted signals and avoid signal interruption due to interference. A blower 15 is arranged at the lower part of the front end of the operation warehouse 1. The blower 15 is connected to the power supply through wiring.

[0023] The support rod 7 includes a support frame 12, a stretching mechanism 21 and a magnetic device 23. The magnetic device 23 is installed on the lower surface of the stretching mechanism 21, and the magnetic device 23 is installed on the top of the support frame 12. The support frame 12 is located inside the stretching mechanism 21. The stretching mechanism 21 and the support frame are connected and retracted through a hydraulic device 22. When the hydraulic device 22 is started, the stretching mechanism 21 starts to be lowered, and the support frame 12 is placed on the ground. When the hydraulic device 22 is turned off, the stretching mechanism 21 starts to rise, and the support frame 12 is retracted. At the same time, the support frame 12 is fastened by the magnetic device 24 to prevent the support frame 12 from automatically lowering due to gravity.

[0024] The dual fusion robot arm includes: Robotic arm base; A third mechanical arm, wherein the lower end of the third mechanical arm is mounted on the mechanical arm base and can rotate and swing around the mechanical arm base; A first mechanical arm, the first mechanical arm is mounted on the upper end of the third mechanical arm and can rotate around the third mechanical arm, and a clamping device and a punching device are provided at the front end of the first mechanical arm; The second mechanical arm is installed on the upper end of the third mechanical arm and can rotate around the third mechanical arm. A sampling device is arranged at the front end of the second mechanical arm.

[0025] Specifically, the robot arm base includes a base, a first server fixing plate 30 is provided on the base, a second server fixing plate 31 is installed on the upper end, a third server fixing plate 32 is installed on the upper end of the second server fixing plate 31, and a first servo motor 33 is installed in the mounting groove of the third server fixing plate 32 to control the rotation of the base. The first servo motor 33 is connected to the server main support plate 35 through the first hexagonal linkage wheel 34, and the server main support plate 35 is connected to the server upper base 36.

[0026] Specifically, the third mechanical arm includes a fourth servo fixing plate 37 symmetrically mounted on both sides of the servo upper base 36, and a second servo motor 38 is mounted in the mounting grooves of the fourth servo fixing plates 37 on both sides to control the rotation of the third mechanical arm. The second servo motor 38 is connected to the lower end of the No. 1 arm L40 and the lower end of the No. 1 arm R41 through a second hexagonal linkage wheel 39, and the No. 1 support arm 42 is mounted in the corresponding bayonet between the No. 1 arm L40 and the No. 1 arm R41.

[0027] Specifically, the first robot arm includes: the lower end of arm No. 2 L43 hingedly connected to the upper end of arm No. 1 L40 through the third hexagonal linkage wheel 68, and the lower end of arm No. 2 R44 hingedly connected to the upper end of arm No. 1 R41 through the third hexagonal linkage wheel 51, and a third servo motor 52 is installed in the mounting groove. Support arm No. 2 53 is installed in the corresponding bayonet of arm No. 2 L43 and arm No. 2 R44, and a fourth servo motor 54 is installed in the mounting groove of the upper end of arm No. 2 L43 and the upper end of arm No. 2 R44 for controlling the rotation of arm No. 4, which is connected to the lower end 47 of arm No. 4 L and the lower end of arm No. 4 R48 on both sides through the fourth hexagonal linkage wheel 69, and a clamping device and a punching device are installed in the corresponding bayonet of the upper end 47 of arm No. 4 L and the upper end of arm No. 4 R48.

[0028] Specifically, the second robot arm includes: the lower end of arm No. 3 L45 hingedly connected to the upper end of arm No. 1 L40 through the third hexagonal linkage wheel 68, and the lower end of arm No. 3 R46 hingedly connected to the upper end of arm No. 1 R41 through the third hexagonal linkage wheel 51. Support arm No. 3 62 is installed in the corresponding bayonet of arm No. 3 L45 and arm No. 3 R46, and the sixth servo motor 63 is installed in the mounting groove of the upper end of arm No. 3 L45 and the upper end of arm No. 3 R46 to control the rotation of arm No. 5, which is connected to the lower end 49 of arm No. 5 L and the lower end of arm No. 5 R50 on both sides through the sixth hexagonal linkage wheel 73, and the sampling device is installed in the corresponding bayonet of the upper end of arm No. 5 L49 and the upper end of arm No. 5 R50.

[0029] The clamping device and the punching device include: A first front-end server connecting plate 55, wherein the first front-end server connecting plate 55 is installed at the front end of the first mechanical arm, and a fifth servo motor 56 is installed in a mounting groove of the first front-end server connecting plate 55; A second standard connecting disc 70, which is mounted at the output end of the fifth servo motor 56, and is connected to a right-angle fixing plate 71 symmetrically disposed on both sides, a component main plate 57 is mounted in the gap between the two right-angle fixing plates 71, and a fifth hexagonal linkage wheel 58 symmetrically disposed on both sides is mounted at the bottom of the component main plate 57; A gear arm 72, the gear arm 72 is mounted on the fifth hexagonal linkage wheel 58, a bilaterally symmetrical clamping piece 59 is mounted on the upper end of the gear arm 72, and a connecting rod 60 is hingedly connected between the component main board 57 and the clamping piece 59; The punching rod 61 is installed in the opening at the front end of the component main board 57 . A telescopic spring is installed in the opening of the component main board 57 . A blocking device is provided and connected to the punching rod 61 .

[0030] The sampling device includes: A second front-end server connecting plate 64, wherein the second front-end server connecting plate 64 is installed at the front end of the second robot arm, and a seventh servo motor 65 is installed in the mounting groove of the second front-end server connecting plate 64; The first standard connection disc is connected to 66, and the first standard connection disc is connected to 66 to install the output end of the seventh servo motor 65; The sampling nozzle 67 is installed on the first standard connecting disc 66.

[0031] A method for using an integrated exploration and collection robot based on multimodal information perception, comprising: S1: Drive the integrated exploration and collection robot based on multimodal information perception into the mine and detect the gas and humidity in the mine; S2: LiDAR 9 scans and collects point cloud data in the mine cave, and constructs a three-dimensional surface model in the mine cave; S3: The operator observes the surroundings of the integrated exploration and collection robot based on multimodal information perception through the machine vision module. The machine vision module 8 performs a feasibility analysis on the environment and displays the environment in the mine cave by connecting to an external display screen through a signal amplifier. S4: The dual fusion robot arm collects mineral samples. When the machine vision module 8 detects the mineral sample in front, the punching rod 61 mines the mineral sample to realize the grabbing and collection of the mineral sample; the air pump driving module 19 drives the air pump 20 to work, and the powdered mineral sample is sucked by the sampling nozzle 67 and transported to the charging box 17 through the air pump pipe to complete the collection of the powdered mineral sample; S5: The gravity detection sensor 16 measures the weight of the material in the charging box, and when it exceeds the set value, an alarm is sounded to prompt.

[0032] Step S1 includes: S11: Beidou positioning real-time monitoring of the movement trajectory of the exploration and collection integrated robot based on multi-modal information perception; S12: The gas detection sensor 18 detects the contents of four gases, namely O2, H2S, CO and CH4, in the mine cave, and analyzes the safety of the mine cave. When the contents of the four gases reach the set values ​​respectively, different alarm sounds are issued to prompt; S13: The temperature and humidity detection module 14 detects the temperature and humidity in the mine cave, and displays the detected values.

[0033] Step S2 includes: Determine whether the collection site is in a high dust environment; If the collection site is not in a high dust environment, the laser radar 9 scans and collects point cloud data to construct a three-dimensional surface model; If the collection site is in a high dust environment, the blower 15 is turned on to process the dust, and then the laser radar 9 scans and collects point cloud data, and a three-dimensional surface model is constructed using a three-dimensional modeling optimization method.

[0034] 3D modeling optimization methods include: 1) Establish a three-dimensional space coordinate system and determine the origin, coordinate axis direction and unit length of the coordinate system; 2) Discretize the three-dimensional space and divide it into small units for storing and processing the data collected by the sensor and subsequent modeling calculations; 3) A fusion algorithm based on Bayesian theory fuses the data collected by the laser radar 9 at different times to model the target; 4) Use the deep learning-based convolutional neural network denoising algorithm to filter out the noise in the working scene; The data processing is divided into 5 steps. The first is the convolution layer, which is one of the core components of CNN and is used to extract the features of images or data. For the input data X (which can be an image data matrix or some representation of LiDAR point cloud data), assuming that the convolution kernel (filter) is, the convolution operation can be expressed as: Among them, Y is the output feature map of the convolution layer, (i, j) represents the position coordinates in the output feature map, and (m, n) represents the position coordinates in the convolution kernel.

[0035] The second is the activation function, which is used to introduce nonlinear factors so that the neural network can learn complex Under high dust interference, RELU is used as the activation function, and the data is constant, which can be expressed as: Then comes the pooling layer. The pooling layer reduces the amount of data while retaining the main features and compressing the data. During the operation of the robot, the maximum pooling is often used for compression, which can be expressed as: Among them, P is the output of the pooling layer, and (i, j) represents the position coordinates in the output pooling map.

[0036] The fourth step is the fully connected layer, which is used to integrate the previously extracted features and perform the final classification or regression tasks. Assuming that the input of the fully connected layer is Z (the feature vector after processing by the previous layers), the weight matrix is ​​W, and the bias vector is b, the output O of the fully connected layer can be expressed as: The last step is the loss function. When training the CNN model, a loss function needs to be defined to measure the gap between the model output and the real noise-free data, so as to adjust the network parameters through the back-propagation algorithm.

[0037] During operation, the loss function can be expressed as: Where N is the number of samples, is the i-th sample of the real noise-free data, It is the denoised data corresponding to the i-th sample output by the model.

[0038] The integration of the above algorithms further improves the quality and reliability of the overall sensor data, allowing the robot to operate, work, collect and process various information more accurately in high-dust environments.

[0039] 5) Use Kalman filter algorithm and particle filter algorithm to filter out interfering sound waves; Kalman filter algorithm, Kalman filter has advantages when the state equation and measurement equation of the system are linear and Gaussian distribution. In a high dust environment, the laser radar 9 and the machine vision module 8 use linear state equations and measurement equations to update the current state estimate based on the state estimate at the previous moment and the current measurement value. In the reconstruction of the three-dimensional model, the state variables and observation variables in the three-dimensional model reconstruction are first determined, and then the initial estimate and covariance matrix of the state variables are given. According to the state equation, the state estimate and control input at the previous moment are used to predict the state at the current moment. When new observation data is obtained, the residual between the observed value and the predicted value and the covariance matrix of the residual are calculated according to the observation equation. Then, the predicted value and the observed value are weighted and fused through the Kalman gain to obtain a more accurate state estimate at the current moment, and the covariance matrix of the state estimate is updated. The prediction and update steps are repeated continuously to achieve accurate reconstruction of the three-dimensional model.

[0040] Kalman filter algorithm: Prediction equation: Update equation: in, It is the prediction of the state at time k based on the state estimation at time k-1. is the state transition matrix, is the optimal state estimate at time k-1, is the forecast covariance matrix, is the process noise covariance matrix, is the Kalman gain, is the measurement matrix, is the measured value at time k, is the measurement noise covariance matrix, is the optimal state estimate at time k, is the updated covariance matrix.

[0041] When the robot performs drilling and collecting operations on ore samples in the mine, a large amount of dust is generated, which not only interferes with the recognition function of the machine vision module, but also affects the collection of acoustic wave signals. For the data fusion processing of the machine vision module 8 and the laser radar 9, the fusion algorithm of the Bayesian theory is used to continuously optimize the data integration effect; for the acoustic wave signal processing, the particle filter algorithm is used, and the laser radar 9 is combined to filter the acoustic waves generated during the collection process. At this time, the uncertainty of the sensor measurement increases and may present a complex distribution. The particle filter uses a large number of particles to represent the state distribution of the system. These particles propagate in the state space according to the dynamic model and are resampled according to the measured values ​​to increase the data accuracy.

[0042] Particle filter algorithm has advantages when the state equation and measurement equation of the system are nonlinear and non-Gaussian. In high dust conditions, the uncertainty of sensor measurement increases and may present a complex distribution. Particle filtering uses a large number of particles to represent the state distribution of the system. These particles propagate in the state space according to the dynamic model and are resampled according to the measured values ​​to increase the accuracy of the data. In the reconstruction of the three-dimensional model, the state transfer function and observation function in the three-dimensional model reconstruction are first determined, and a group of particles are randomly generated in the state space. Each particle represents a possible state of the three-dimensional model. According to the state transfer function, each particle is propagated to obtain the predicted particle state. Then, according to the observation function and the actual observation data, the importance weight of each particle is calculated, and the particle is resampled according to the weight. The estimated value of the three-dimensional model state is obtained by weighted averaging the resampled particles. As the iteration proceeds, the particles will gradually converge to the high-weight area, thereby obtaining a more accurate three-dimensional model reconstruction result.

[0043] 6) Using polygonal mesh modeling methods, based on the target surface point cloud data obtained after fusion, the point cloud is connected into triangular patches through algorithms such as triangulation to construct the target surface model; 7) Use probability density function to describe the distribution of dust in three-dimensional space and finally complete the construction of three-dimensional model.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An integrated exploration and collection robot based on multimodal information perception, characterized in that: The device comprises an operation chamber (1), wherein symmetrical crawler wheel moving mechanisms (6) are arranged on both sides of the operation chamber (1), and symmetrical support rods (7) are arranged at the end of the operation chamber (1) for stabilizing the operation of the device. A double fusion type mechanical arm is arranged at the upper end of the operation chamber (1) for grasping and collecting mineral samples; and a machine vision module (8), a laser radar (9), a signal amplifier and an air blowing device (15) are arranged at the front end of the operation chamber (1) for realizing the exploration of the mine environment and the construction of a three-dimensional scene.

2. The integrated exploration and collection robot based on multimodal information perception according to claim 1, characterized in that: The operation chamber (1) is provided with a Beidou dual-mode module (13), a gas detection sensor (18), a temperature and humidity detection module (14), a gravity detection sensor (16), an air pump drive module (19) and an air pump (20) inside. The Beidou dual-mode module (13) and the temperature and humidity detection module (14) are installed on one side of the operation chamber (1), and the gas detection sensor (18), the air pump drive module (19) and the air pump (20) are installed on the other side. The gravity detection sensor (16) and the charging box (17) are installed at the end of the operation chamber (1).

3. The integrated exploration and collection robot based on multimodal information perception according to claim 1, characterized in that: The support rod (7) comprises a support frame (12), a stretching mechanism (21) and a magnetic device (23); the magnetic device (23) is installed on the lower surface of the stretching mechanism (21); the magnetic device (23) is installed on the top of the support frame (12); the support frame (12) is located inside the stretching mechanism (21); and the stretching mechanism (21) and the support frame are connected and retracted via a hydraulic device (22).

4. The integrated exploration and collection robot based on multimodal information perception according to claim 3 is characterized in that: The dual fusion type robotic arm comprises: Robotic arm base; A third mechanical arm, wherein the lower end of the third mechanical arm is mounted on the mechanical arm base and can rotate and swing around the mechanical arm base; A first mechanical arm, the first mechanical arm is mounted on the upper end of the third mechanical arm and can rotate around the third mechanical arm, and a clamping device and a punching device are provided at the front end of the first mechanical arm; The second mechanical arm is installed on the upper end of the third mechanical arm and can rotate around the third mechanical arm. A sampling device is arranged at the front end of the second mechanical arm.

5. The integrated exploration and collection robot based on multimodal information perception according to claim 4, characterized in that: The clamping device and the punching device include: A first front-end server connection plate (55), the first front-end server connection plate (55) being mounted at the front end of the first mechanical arm, and a fifth servo motor (56) being mounted in a mounting groove of the first front-end server connection plate (55); A second standard connecting disc (70), the second standard connecting disc (70) being mounted on the output end of the fifth servo motor (56), the second standard connecting disc (70) being connected to a right-angle fixing plate (71) symmetrical on both sides, a component main plate (57) being mounted in the gap between the two right-angle fixing plates (71), and a fifth hexagonal linkage wheel (58) being symmetrical on both sides being mounted on the bottom of the component main plate (57); A gear arm (72), the gear arm (72) being mounted on the fifth hexagonal linkage wheel (58), a clamping piece (59) symmetrically mounted on the upper end of the gear arm (72), and a connecting rod (60) being hingedly connected between the component main plate (57) and the clamping piece (59); A punching rod (61) is installed in an opening at the front end of the component main board (57). A telescopic spring is installed in the opening of the component main board (57). A blocking device is provided and connected to the punching rod (61).

6. The integrated exploration and collection robot based on multimodal information perception according to claim 4, characterized in that: The sampling device comprises: A second front-end server connection plate (64), the second front-end server connection plate (64) being mounted at the front end of the second mechanical arm, and a seventh servo motor (65) being mounted in a mounting groove of the second front-end server connection plate (64); The first standard connecting disc is connected to (66), and the first standard connecting disc is connected to (66) an output end of a seventh servo motor (65) is installed; A sampling nozzle (67), wherein the sampling nozzle (67) is mounted on the first standard connecting disc (66).

7. The method for using the exploration and collection integrated robot based on multimodal information perception according to claim 4 is characterized in that: include: S1: Drive the integrated exploration and collection robot based on multimodal information perception into the mine and detect the gas and humidity in the mine; S2: The laser radar (9) scans and collects point cloud data in the mine cave, and constructs a three-dimensional surface model of the mine cave; S3: The operator observes the surroundings of the integrated exploration and collection robot based on multimodal information perception through the machine vision module. The machine vision module (8) performs a feasibility analysis on the environment and displays the environment in the mine cave by connecting to an external display screen through a signal amplifier. S4: The dual fusion robot arm collects mineral samples. When the machine vision module (8) detects the mineral sample in front, the drilling rod (61) mines the mineral sample to achieve the grabbing and collection of the mineral sample. The air pump driving module (19) drives the air pump (20) to work, and the powdered mineral sample is sucked through the sampling nozzle (67) and transported to the loading box (17) through the air pump pipe, thereby completing the collection of the powdered mineral sample. S5: The gravity detection sensor (16) measures the weight of the material in the loading box and sounds an alarm when the weight exceeds a set value.

8. The method for using the exploration and collection integrated robot based on multimodal information perception according to claim 7 is characterized in that: The step S1 comprises: S11: Beidou positioning real-time monitoring of the movement trajectory of the exploration and collection integrated robot based on multi-modal information perception; S12: The gas detection sensor (18) detects the contents of four gases, namely O2, H2S, CO and CH4, in the mine cave, and analyzes the safety of the mine cave. When the contents of the four gases reach the set values ​​respectively, different alarm sounds are issued to give a prompt; S13: The temperature and humidity detection module (14) detects the temperature and humidity in the mine and displays the detected values.

9. The method for using the exploration and collection integrated robot based on multimodal information perception according to claim 7, characterized in that: The step S2 comprises: Determine whether the collection site is in a high dust environment; If the collection site is not in a high dust environment, the laser radar (9) scans and collects point cloud data to construct a three-dimensional surface model; If the collection site is in a high dust environment, the blower (15) is turned on to process the dust, and then the laser radar (9) scans and collects point cloud data, and a three-dimensional surface model is constructed using a three-dimensional modeling optimization method.

10. The method for using the exploration and collection integrated robot based on multimodal information perception according to claim 9, characterized in that: The three-dimensional modeling optimization method comprises: Establish a three-dimensional space coordinate system and determine the origin, coordinate axis direction and unit length of the coordinate system; Discretize the three-dimensional space and divide it into small units for storing and processing the data collected by the sensor and subsequent modeling calculations; The fusion algorithm based on Bayesian theory fuses the data collected by the laser radar (9) at different times to model the target; Use the deep learning-based convolutional neural network denoising algorithm to filter out the noise in the working scene; Use Kalman filter algorithm or particle filter algorithm to filter out interfering sound waves; Using polygonal mesh modeling method, according to the target surface point cloud data obtained after fusion, the point cloud is connected into triangular patches to construct the target surface model; The probability density function is used to describe the distribution of dust in three-dimensional space, and finally the three-dimensional model is constructed.

Images