A method for detecting relative pose of a support-drifting anchor robot based on cross-modal information fusion

Abstract

A kind of relative pose detection method and system of cross-modal information fusion based on tunneling anchor robot, it is related to the field of heading machine.Solve the problem that fixed target is frequently moved in existing visual positioning technology, calibration process is complex.The method comprises: multi-view thermal imaging camera is connected and installed at the end of cantilever heading machine by mechanical arm, and thermal imaging target image, visible light image and depth information installed at the front end of step-by-step anchor support equipment are dynamically collected during the movement of heading machine;And image pre-processing is carried out;Dual-branch network is used to fuse the pre-processed thermal imaging target image and visible light image;The geometric features of fused image are extracted using M-UNet segmentation algorithm improved UNet segmentation algorithm, the three-dimensional coordinates of spherical heat source geometric center are obtained by extracting the geometric center three-dimensional coordinates of bounding box;The relative pose relationship between cantilever heading machine and step-by-step anchor support equipment is calculated using the pose solution model of heading anchor combined unit.

Description

Technical Field

[0001] This invention relates to the field of tunneling machine technology, specifically to a method and system for relative pose detection of a tunneling support robot based on cross-modal information fusion. Background Technology

[0002] Intelligent coal mining is a key step in achieving high-quality development of the coal industry. As an important mining equipment in underground coal mines, the level of intelligence of tunneling machines is crucial to the realization of intelligent coal mining. Therefore, achieving accurate detection of the position and posture of tunneling machines has become an urgent issue on the road to intelligent coal mining development.

[0003] The working environment of the tunneling face is complex. The tunneling machine is easily affected by harsh environments such as high dust and fog, strong noise, low light, and strong magnetic fields. Conventional position and posture detection technologies, such as inertial navigation technology, total station technology, IGPS technology, and ultra-wideband technology, are difficult to achieve accurate position and posture detection of the tunneling machine.

[0004] Existing pose detection technologies have the following problems: inertial navigation technology suffers from cumulative measurement errors; total station technology has difficulty measuring multiple targets simultaneously; iGPS technology is easily affected by the high dust environment in underground coal mines; ultra-wideband technology has problems with base station setup and stability; and ordinary machine vision technology is easily affected by low light and high dust environments in tunnels. Summary of the Invention

[0005] This invention addresses the challenges of frequent target relocation and complex calibration procedures in existing visual positioning technologies. To solve these problems, the invention employs the following technical solution:

[0006] Option 1: This invention proposes a relative pose detection method for a tunneling and anchoring robot based on cross-modal information fusion. The relative pose detection method includes the following steps:

[0007] Step 1: A multi-view thermal imaging camera is connected to and installed at the end of the cantilever tunneling machine via a robotic arm, and dynamically acquires thermal imaging target images, visible light images and depth information installed at the front end of the stepping anchor support equipment during the movement of the tunneling machine;

[0008] Step 2: Perform image preprocessing on the thermal imaging target images and visible light images acquired by the multi-view thermal imaging camera;

[0009] Step 3: Use a dual-branch network to fuse the preprocessed thermal imaging target image and the visible light image;

[0010] Step 4: Based on the fused thermal imaging image and visible light image, the M-UNet segmentation algorithm, which is an improved UNet segmentation algorithm, is used to extract the geometric features of the fused image. By filtering the point cloud information of the depth camera around the spherical heat source target, the bounding box of the point cloud surrounding the spherical heat source is obtained and fused with the thermal imaging target image and the visible light image. The three-dimensional coordinates of the geometric center of the spherical heat source are obtained by extracting the three-dimensional coordinates of the geometric center of the bounding box.

[0011] Step 5: Establish the coordinate system of the multi-view thermal imaging camera, the coordinate system of the thermal imaging target, the coordinate system of the cantilever tunneling machine, and the coordinate system of the stepping anchor support equipment. Use the pose calculation model of the tunneling-support-anchor combined unit to calculate the relative pose relationship between the cantilever tunneling machine and the stepping anchor support equipment. The pose relationship is the three attitude information of roll angle, pitch angle, and yaw angle, and the three position information of x, y, and z axes.

[0012] Furthermore, in a preferred embodiment, the excavator and anchor robot further includes a wireless communication module, and the multi-view thermal imaging camera mentioned in step 1 includes an infrared thermal imaging camera, a visible light camera, and a depth camera.

[0013] Furthermore, a preferred embodiment is provided, wherein step 2, which involves image preprocessing of the thermal imaging target image and the visible light image acquired by the multi-view thermal imaging camera, includes distortion correction, image registration, and image enhancement of the thermal imaging target image and the visible light image.

[0014] Furthermore, a preferred embodiment is provided, wherein the dual-branch network in step 3 includes an encoder and a decoder. The encoder is mainly composed of a first encoder and a second encoder. The first encoder is used to extract the basic features of the input image, while the second encoder is used to capture more refined image feature information. The decoder generates a fused image.

[0015] Furthermore, a preferred embodiment is provided, wherein the method for calculating the relative pose relationship between the cantilever tunneling machine and the stepping anchor support equipment using the pose calculation model of the tunneling and anchoring robot in step 5 is as follows:

[0016]

[0017] In the above formula, This is the transformation matrix between the tunneling machine's body coordinate system and the stepping anchor support equipment's body coordinate system; This is the transformation matrix between the tunneling machine's body coordinate system and the infrared multi-view thermal imaging camera coordinate system; This is the transformation matrix between the coordinate system of the multi-view thermal imaging camera and the coordinate system of the heat source target; This is the transformation matrix between the coordinate system of the heat source target and the coordinate system of the stepping anchor support equipment;

[0018] The tunneling machine body and the infrared multi-view thermal imaging camera are rigidly connected, maintaining a constant relative position. The transformation matrix is ​​obtained through rigid body transformation measurement and calculation. ;

[0019] ;

[0020] The heat source target and the stepping anchor support equipment are rigidly connected, and their relative positions remain unchanged. The transformation matrix is ​​obtained through rigid body transformation measurement and calculation. .

[0021] Furthermore, a preferred embodiment is provided, wherein the transformation matrix between the tunneling machine's body coordinate system and the stepping anchor support equipment's body coordinate system is... The rotation matrix R is converted into Euler angles using the quaternion method to obtain three attitude information: roll angle, pitch angle, and yaw angle. The x, y, and z axis position information is obtained from the translation vector t.

[0022] Option 2: A relative pose detection system for a tunneling and anchoring robot based on cross-modal information fusion, the relative pose detection system comprising:

[0023] The image acquisition module is used to connect a multi-view thermal imaging camera to the end of a cantilever tunneling machine via a robotic arm, and dynamically acquire thermal imaging target images, visible light images, and depth information installed at the front end of the stepping anchor support equipment during the movement of the tunneling machine;

[0024] The preprocessing module is used to preprocess the thermal imaging target images and visible light images acquired by the multi-view thermal imaging camera.

[0025] The fusion module is used to fuse the preprocessed thermal imaging target image and the visible light image using a dual-branch network.

[0026] The 3D coordinate acquisition module is used to extract the geometric features of the fused image based on the fused thermal imaging image and visible light image using the M-UNet segmentation algorithm, which is an improved UNet segmentation algorithm. By filtering the point cloud information of the depth camera around the spherical heat source target, the bounding box of the point cloud surrounding the spherical heat source is obtained and fused with the thermal imaging target image and the visible light image. The 3D coordinates of the geometric center of the spherical heat source are obtained by extracting the 3D coordinates of the geometric center of the bounding box.

[0027] The relative pose detection module is used to establish the coordinate system of the multi-view thermal imaging camera, the coordinate system of the thermal imaging target, the coordinate system of the cantilever tunneling machine, and the coordinate system of the stepping anchor support equipment. The relative pose relationship between the cantilever tunneling machine and the stepping anchor support equipment is calculated using the pose calculation model of the tunneling-support-anchor combined unit. The pose relationship is three attitude information: roll angle, pitch angle, and yaw angle, and three position information of x, y, and z axes.

[0028] Option 3: A computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method described in Option 1.

[0029] Option 4: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in Option 1.

[0030] The advantages of this invention are:

[0031] The relative pose detection method for a tunneling and anchoring robot based on cross-modal information fusion described in this invention has the advantages of non-contact visual measurement, high precision, low cost, and no cumulative error.

[0032] This invention utilizes the characteristic of positioning targets following movement in the collaborative operation process of tunneling and anchoring robots, solving the problems of frequent relocation of fixed targets and complex calibration procedures in traditional visual positioning technology.

[0033] The relative pose detection method for tunneling and anchoring robots based on cross-modal information fusion described in this invention highly integrates thermal imaging images, visible images, and depth information, enabling precise positioning of target objects in harsh environments such as high dust, low light, and strong noise in underground coal mines.

[0034] The present invention describes a method for relative pose detection of a tunneling and anchoring robot based on cross-modal information fusion, which integrates thermal imaging images, visible light images, and depth information. The method uses a multi-view thermal imaging camera to dynamically acquire thermal imaging target images and visible light images, and achieves image feature fusion through a dual-branch network. Then, it is fused with the synchronously acquired depth information to achieve cross-modal fusion. This method can accurately locate the three-dimensional coordinates of the geometric center of a spherical heat source in harsh environments such as high dust, low illumination, and strong noise in coal mines.

[0035] The relative pose detection method for a tunneling and anchoring robot based on cross-modal information fusion described in this invention adopts an improved M-UNet segmentation algorithm: the improved M-UNet algorithm adds a multi-scale attention module after the encoder-decoder connection layer, which reduces the amount of computation without reducing the spatial dimension, and can efficiently aggregate multi-scale spatial information.

[0036] The pose calculation model of the tunneling-support-anchor combined unit constructed in this invention establishes coordinate systems for a multi-view thermal imaging camera, an infrared target, the tunneling machine, and the anchor support equipment. Through rigid body transformation, SVD decomposition, and chain operations of the transformation matrix, it calculates the roll angle, pitch angle, yaw angle, and three-axis position information. This achieves efficient calculation of the roll angle, pitch angle, and yaw angle, as well as the x, y, and z-axis position information, between the cantilever tunneling machine and the stepping anchor support equipment in the tunneling-support-anchor robot.

[0037] This invention is based on the relative pose detection technology of extracting target feature information through cross-modal information fusion. It leverages the advantages of infrared thermal imaging in resisting dust, fog, and darkness to provide an effective positioning reference for the excavation and anchoring robot.

[0038] This invention is also applicable to the field of positioning reference for tunneling and anchoring robots. Attached Figure Description

[0039] Figure 1 This is a flowchart of the relative pose detection method for the tunneling and anchoring robot based on cross-modal information fusion as described in Implementation Method 1.

[0040] Figure 2 This is a schematic diagram of the structure of the tunneling and anchoring combined unit in the tunneling and anchoring robot relative pose detection method based on cross-modal information fusion as described in Implementation Method 1.

[0041] Among them, stepping anchor support equipment 1, cantilever tunneling machine 2, thermal imaging target 3, and multi-view thermal imaging camera 4.

[0042] Figure 3 This is a schematic diagram of the improved M-UNet segmentation algorithm network framework for the relative pose detection method of the tunneling and anchoring robot based on cross-modal information fusion as described in Implementation Method 1.

[0043] Figure 4 This is a schematic diagram of the EMA module structure of the relative pose detection method for the tunneling and anchoring robot based on cross-modal information fusion as described in Implementation Method Eleven. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0045] Implementation Method 1: This implementation method proposes a relative pose detection method for a tunneling and anchoring robot based on cross-modal information fusion. The relative pose detection method includes the following steps:

[0046] Step 1: A multi-view thermal imaging camera is connected to and installed at the end of the cantilever tunneling machine via a robotic arm, and dynamically acquires thermal imaging target images, visible light images and depth information installed at the front end of the stepping anchor support equipment during the movement of the tunneling machine;

[0047] Step 2: Perform image preprocessing on the thermal imaging target images and visible light images acquired by the multi-view thermal imaging camera;

[0048] Step 3: Use a dual-branch network to fuse the preprocessed thermal imaging target image and the visible light image;

[0049] Step 4: Based on the fused thermal imaging image and visible light image, the M-UNet segmentation algorithm, which is an improved UNet segmentation algorithm, is used to extract the geometric features of the fused image. By filtering the point cloud information of the depth camera around the spherical heat source target, the bounding box of the point cloud surrounding the spherical heat source is obtained and fused with the thermal imaging target image and the visible light image. The three-dimensional coordinates of the geometric center of the spherical heat source are obtained by extracting the three-dimensional coordinates of the geometric center of the bounding box.

[0050] Step 5: Establish the coordinate system of the multi-view thermal imaging camera, the coordinate system of the thermal imaging target, the coordinate system of the cantilever tunneling machine, and the coordinate system of the stepping anchor support equipment. Use the pose calculation model of the tunneling-support-anchor combined unit to calculate the relative pose relationship between the cantilever tunneling machine and the stepping anchor support equipment. The pose relationship is the three attitude information of roll angle, pitch angle, and yaw angle, and the three position information of x, y, and z axes.

[0051] Implementation Method 2: This implementation method further defines the relative pose detection method for the tunneling and anchoring robot based on cross-modal information fusion described in Implementation Method 1. The multi-view thermal imaging camera mentioned in step 1 includes an infrared thermal imaging camera, a visible light camera, and a depth camera.

[0052] Implementation Method 3: This implementation method further defines the relative pose detection method for the tunneling and anchoring robot based on cross-modal information fusion described in Implementation Method 1. Step 2, which performs image preprocessing on the thermal imaging target image and visible light image acquired by the multi-view thermal imaging camera, includes distortion correction, image registration, and image enhancement of the thermal imaging target image and the visible light image.

[0053] Implementation Method 4: This implementation method further defines the relative pose detection method for the tunneling and anchoring robot based on cross-modal information fusion described in Implementation Method 1. In step 3, the dual-branch network includes an encoder and a decoder. The encoder is mainly composed of a first encoder and a second encoder. The first encoder is used to extract the basic features of the input image, while the second encoder is used to capture more refined image feature information. The decoder generates the fused image.

[0054] Implementation Method 5: This implementation method further defines the relative pose detection method for the tunneling and anchoring robot based on cross-modal information fusion described in Implementation Method 4. Step 4 also includes adding a multi-scale attention module after the connection layer between the encoder and decoder in the improved M-UNet segmentation algorithm.

[0055] Implementation Method Six: This implementation method further defines the relative pose detection method for the tunneling and anchoring robot based on cross-modal information fusion described in Implementation Method One. The method for calculating the relative pose relationship between the cantilever tunneling machine and the stepping anchoring support equipment using the tunneling and anchoring robot pose calculation model in step 5 is as follows:

[0056]

[0057] In the above formula, This is the transformation matrix between the tunneling machine's body coordinate system and the stepping anchor support equipment's body coordinate system; This is the transformation matrix between the tunneling machine's body coordinate system and the infrared multi-view thermal imaging camera coordinate system; This is the transformation matrix between the coordinate system of the multi-view thermal imaging camera and the coordinate system of the heat source target; This is the transformation matrix between the coordinate system of the heat source target and the coordinate system of the stepping anchor support equipment;

[0058] The tunneling machine body and the infrared multi-view thermal imaging camera are rigidly connected, maintaining a constant relative position. The transformation matrix is ​​obtained through rigid body transformation measurement and calculation. ;

[0059] ;

[0060] The heat source target and the stepping anchor support equipment are rigidly connected, and their relative positions remain unchanged. The transformation matrix is ​​obtained through rigid body transformation measurement and calculation. .

[0061] Implementation Method Seven: This implementation method further defines the relative pose detection method for the tunneling machine anchor support robot based on cross-modal information fusion described in Implementation Method Six. The transformation matrix between the tunneling machine's body coordinate system and the stepping anchor support equipment's body coordinate system is... The rotation matrix R is converted into Euler angles using the quaternion method to obtain three attitude information: roll angle, pitch angle, and yaw angle. The x, y, and z axis position information can be obtained from the translation vector t.

[0062] Implementation Method 8: This implementation method proposes a relative pose detection system for a tunneling and anchoring robot based on cross-modal information fusion. The relative pose detection system includes:

[0063] The image acquisition module is used to connect a multi-view thermal imaging camera to the end of a cantilever tunneling machine via a robotic arm, and dynamically acquire thermal imaging target images, visible light images, and depth information installed at the front end of the stepping anchor support equipment during the movement of the tunneling machine;

[0064] The preprocessing module is used to preprocess the thermal imaging target images and visible light images acquired by the multi-view thermal imaging camera.

[0065] The fusion module is used to fuse the preprocessed thermal imaging target image and the visible light image using a dual-branch network.

[0066] The 3D coordinate acquisition module is used to extract the geometric features of the fused image based on the fused thermal imaging image and visible light image using the M-UNet segmentation algorithm, which is an improved UNet segmentation algorithm. By filtering the point cloud information of the depth camera around the spherical heat source target, the bounding box of the point cloud surrounding the spherical heat source is obtained and fused with the thermal imaging target image and the visible light image. The 3D coordinates of the geometric center of the spherical heat source are obtained by extracting the 3D coordinates of the geometric center of the bounding box.

[0067] The relative pose detection module is used to establish the coordinate system of the multi-view thermal imaging camera, the coordinate system of the thermal imaging target, the coordinate system of the cantilever tunneling machine, and the coordinate system of the stepping anchor support equipment. The relative pose relationship between the cantilever tunneling machine and the stepping anchor support equipment is calculated using the pose calculation model of the tunneling-support-anchor combined unit. The pose relationship is three attitude information: roll angle, pitch angle, and yaw angle, and three position information of x, y, and z axes.

[0068] Implementation Method Nine: This implementation method provides a computer device, including a memory and a processor. The memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes the methods described in Implementation Methods One to Seven.

[0069] Implementation Method 10: This implementation method provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the methods described in Implementation Methods 1 to 7.

[0070] Implementation Method Eleven: This implementation method provides an example, which is used to explain the above-described implementation methods one through eight. The specific example is as follows:

[0071] See Figures 1 to 4 This embodiment describes a method for relative pose detection of a tunneling and anchoring robot based on cross-modal information fusion, which includes the following steps:

[0072] (1) A multi-view thermal imaging camera is connected to and installed at the end of the cantilever tunneling machine via a robotic arm, and dynamically acquires thermal imaging target images, visible light images and depth information installed at the front end of the stepping anchor support equipment during the movement of the tunneling machine; the multi-view thermal imaging camera consists of an infrared thermal imaging camera, a visible light camera and a depth camera.

[0073] (2) Perform image preprocessing on the thermal imaging images and visible light images acquired by the multi-view thermal imaging camera. The image preprocessing includes distortion correction, image registration and image enhancement of the thermal imaging images and visible light images.

[0074] (3) A dual-branch network is used to fuse the preprocessed thermal imaging image and the visible light image. The dual-branch network mainly consists of an encoder and a decoder. The encoder mainly consists of a basic encoder and a detail encoder. The basic encoder is used to extract the basic features of the input image, while the detail encoder is used to capture more refined image feature information. The decoder generates the fused image.

[0075] (4) Based on the fusion image of thermal imaging image and visible light image, the M-UNet segmentation algorithm, which further improves the UNet segmentation algorithm, extracts the geometric features of the fusion image. By filtering the point cloud information around the spherical heat source target from the depth camera, the bounding box of the point cloud surrounding the spherical heat source is obtained and fused with the thermal imaging image and the visible light image. The three-dimensional coordinates of the geometric center of the spherical heat source are obtained by extracting the three-dimensional coordinates of the geometric center of the bounding box. Compared with the traditional binocular ranging method, this processing method can effectively avoid mis-measuring the center of the outer surface of the spherical heat source as its geometric center three-dimensional coordinates, directly measure the three-dimensional coordinates of the geometric center of the spherical heat source, reduce measurement errors, and improve the positioning accuracy of the spherical heat source.

[0076] The harsh environment of underground coal mine tunneling faces, characterized by high dust, low illumination, and strong noise, affects the accuracy of image segmentation and consequently the relative pose detection accuracy of the tunneling and anchoring robot. This invention introduces a multi-scale attention mechanism (EMA) module into the image segmentation algorithm to eliminate interference factors in the environment, enhance the extraction of geometric features from the fused image, and enable the network to adaptively focus on the target segmentation region, thereby improving detection accuracy.

[0077] The image segmentation algorithm described is the M-UNet segmentation algorithm, which is an improvement on UNet.

[0078] Traditional UNet segmentation algorithm first uses multiple sets of convolutional blocks stacked in the encoder to gradually extract deep features of the fused image when processing images, and then uses max pooling to downsample the multi-level feature maps. This processing method is difficult to efficiently aggregate multi-scale spatial information. The improved M-UNet segmentation algorithm adds a multi-scale attention module after the encoder-decoder connection layer.

[0079] Specifically, the input image is processed through two 3×3 convolution operations to obtain a fused image of size 512×512. Before the max pooling operation, two 1×1 convolution branches and one 3×3 convolution branch are added in parallel, and cross-space learning is performed. This reduces the computational load without reducing the spatial dimension, and can efficiently aggregate multi-scale spatial information. Then, max pooling is performed to downsample and reduce the size of the input image. Next, three downsampling operations are performed to reduce the feature map resolution while expanding the receptive field of the model and enhancing the feature representation ability. Finally, the deconvolution operation in the decoder is used to upsample the multi-level feature maps to gradually restore the size of the feature maps.

[0080] (5) Establish the coordinate system of the multi-view thermal imaging camera, the coordinate system of the thermal imaging target, the coordinate system of the cantilever tunneling machine, and the coordinate system of the stepping anchor support equipment. Calculate the relative pose relationship between the cantilever tunneling machine and the stepping anchor support equipment using the pose calculation model of the tunneling and anchor support robot. The pose relationship consists of three attitude information points: roll angle, pitch angle, and yaw angle, and three position information points: x, y, and z. Specifically:

[0081] Based on the positional relationships between the coordinate systems of the multi-view thermal imaging camera, the thermal imaging target, the cantilever tunneling machine, and the stepping anchor support equipment, a pose calculation model for the tunneling and anchor support robot is established to calculate the relative pose calculation model between the cantilever tunneling machine and the stepping anchor support equipment.

[0082]

[0083] In the above formula, This is the transformation matrix between the tunneling machine's body coordinate system and the stepping anchor support equipment's body coordinate system; This is the transformation matrix between the tunneling machine's body coordinate system and the infrared multi-view thermal imaging camera coordinate system; This is the transformation matrix between the coordinate system of the multi-view thermal imaging camera and the coordinate system of the heat source target; This is the transformation matrix between the coordinate system of the heat source target and the coordinate system of the stepping anchor support equipment;

[0084] The tunnel boring machine body and the infrared multi-view thermal imaging camera are rigidly connected, and their relative positions remain unchanged. The transformation matrix is ​​obtained by measuring and calculating through rigid body transformation. ;

[0085] Next, the transformation matrix between the coordinate system of the multi-view thermal imaging camera and the coordinate system of the heat source target is solved. :

[0086] The point set {q}, consisting of the three-dimensional coordinates of the geometric center of the spherical heat source in the camera coordinate system, is obtained from (4). i Given the three-dimensional coordinates of the geometric center of the spherical heat source in the target coordinate system, the point set {p} constitutes this set. iThe rigid body transformation relationship between point sets is as follows:

[0087]

[0088] In the above formula, R is the rotation matrix from the multi-view thermal imaging camera to the target, t is the translation vector from the multi-view thermal imaging camera to the target, and i = 1, 2, 3, ..., n

[0089] Further calculation of the centroids of the target point set and the camera point set:

[0090] ,

[0091] In the above formula, Let the centroid of the target point set be... The centroid of the camera point set

[0092] The centralized point is:

[0093] ,

[0094] To further solve for R using the SVD decomposition method, firstly, construct the covariance matrix of the centered point set:

[0095]

[0096] In the above formula, H The covariance matrix of the centered point set

[0097] Further H Perform SVD decomposition:

[0098] H =UΣVT

[0099] In the above formula, U is a 3×3 orthogonal matrix whose column vectors are the left singular vectors of H; Σ is a 3×3 diagonal matrix whose diagonal elements are singular values ​​arranged in descending order; V is a 3×3 orthogonal matrix whose column vectors are the right singular vectors of H.

[0100] Rotation matrix R from multi-view thermal imaging camera to target:

[0101] R=VUT

[0102] Translation vector t from the multi-view thermal imaging camera to the target:

[0103]

[0104] Furthermore, the transformation matrix between the coordinate system of the multi-view thermal imaging camera and the coordinate system of the heat source target can be obtained. :

[0105]

[0106] The heat source target and the stepping anchor support equipment are rigidly connected, and their relative positions remain unchanged. The transformation matrix is ​​obtained through rigid body transformation measurement and calculation. ;

[0107] Further, the transformation matrix between the tunneling machine's body coordinate system and the stepping anchor support equipment's body coordinate system will be adjusted. The rotation matrix R is converted into Euler angles using the quaternion method to obtain three attitude information: roll angle, pitch angle, and yaw angle. The x, y, and z axis position information can be obtained from the translation vector t.

[0108] Thermal imaging images are mainly formed by capturing infrared radiation emitted by objects, and are resistant to dust and darkness. Visible light images have rich color information and can clearly display the texture, shape and details of objects. This invention not only fuses thermal imaging images and visible light images in a dual-modal manner, but also innovatively fuses depth information with them in the form of a point cloud stereo bounding box, achieving cross-modal information fusion that combines the advantages of all three. Visual measurement is characterized by non-contact, high precision, low cost and no cumulative error, and high positioning accuracy.

[0109] Those skilled in the art will understand that the above description is merely a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. This is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0110] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A method for detecting relative pose of a branch-and-anchor robot based on cross-modal information fusion, characterized in that, The relative pose detection method comprises the following steps: Step 1, the multi-view thermal imaging camera is connected and installed at the end of the boom-type tunneling machine through a mechanical arm, and thermal imaging target images, visible light images and depth information installed at the front end of the step-type anchor support equipment are dynamically collected during the movement of the tunneling machine; Step 2, the thermal imaging target images and visible light images collected by the multi-view thermal imaging camera are preprocessed; Step 3, the preprocessed thermal imaging target images and visible light images are fused by using a double-branch network; Step 4, based on the fused thermal imaging images and visible light images, the geometric features of the fused images are extracted by using an improved UNet segmentation algorithm M-UNet segmentation algorithm, the point cloud surrounding box of the spherical heat source is obtained by screening the point cloud information of the depth camera around the spherical heat source target, and is fused with the thermal imaging target images and visible light images, and the three-dimensional coordinates of the geometric center of the spherical heat source are obtained by extracting the three-dimensional coordinates of the geometric center of the surrounding box; Step 5, a multi-view thermal imaging camera coordinate system, a thermal imaging target coordinate system, a boom-type tunneling machine coordinate system and a step-type anchor support equipment coordinate system are established, and a tunneling and supporting anchor robot pose solving model is used to calculate the relative pose relationship between the boom-type tunneling machine and the step-type anchor support equipment, the pose relationship being three attitude information of roll angle, pitch angle and heading angle and three-axis position information of x, y and z; The multi-view thermal imaging camera of step 1 comprises an infrared thermal imaging camera, a visible light camera and a depth camera; In step 4, a multi-scale attention module is added after the connection layer of the encoder and the decoder of the improved M-UNet segmentation algorithm.

2. The cross-modal information fusion-based relative pose detection method for a branch-and-anchor robot according to claim 1, characterized in that, In step 2, the image preprocessing of the thermal imaging target images and visible light images collected by the multi-view thermal imaging camera comprises the steps of distortion correction, image registration and image enhancement of the thermal imaging target images and visible light images.

3. The cross-modal information fusion-based relative pose detection method for a branch-and-anchor robot according to claim 1, characterized in that, In step 3, the double-branch network comprises an encoder and a decoder, the encoder mainly comprises a first encoder and a second encoder, the first encoder is used to extract basic features of the input image, and the second encoder is used to capture finer image feature information, and the decoder generates a fused image.

4. The cross-modal information fusion-based relative pose detection method for a branch-and-anchor robot according to claim 1, characterized in that, In step 5, the method for calculating the relative pose relationship between the boom-type tunneling machine and the step-type anchor support equipment by using the tunneling and supporting anchor robot pose solving model is as follows: In the above formula, is the conversion matrix between the tunneling machine body coordinate system and the step-by-step anchor support equipment body coordinate system; is the conversion matrix between the tunneling machine body coordinate system and the infrared multi-view thermal imaging camera coordinate system; is the conversion matrix between the multi-view thermal imaging camera coordinate system and the heat source target coordinate system; is the conversion matrix between the heat source target coordinate system and the step-by-step anchor support equipment body coordinate system; Wherein, the tunneling machine body and the infrared multi-view thermal imaging camera are connected through a rigid body, the relative position remains unchanged, and the conversion matrix is obtained through rigid body transformation measurement calculation ; ； The heat source target and the walking anchor support equipment body are connected through a rigid body, the relative position is kept unchanged, and a conversion matrix is obtained through rigid body transformation measurement calculation .

5. The cross-modal information fusion-based relative pose detection method for a branch-and-anchor robot according to claim 4, characterized in that, The conversion matrix between the machine body coordinate system of the heading machine and the machine body coordinate system of the step-by-step anchor support equipment The rotation matrix R is converted into Euler angles by the quaternion method to obtain three kinds of attitude information of roll angle, pitch angle and heading angle, and the translation vector t is used to obtain three-axis position information of x, y and z.

6. A relative pose detection system for a cross-measure information fusion-based branch-and-anchor robot, comprising: The relative pose detection system comprises: An image acquisition module is used for connecting and installing the multi-view thermal imaging camera at the end of the boom-type tunneling machine through a mechanical arm, and dynamically collecting thermal imaging target images, visible light images and depth information installed at the front end of the step-type anchor support equipment during the movement of the tunneling machine; A preprocessing module is used for preprocessing the thermal imaging target images and visible light images collected by the multi-view thermal imaging camera; A fusion module is used for fusing the preprocessed thermal imaging target images and visible light images by using a double-branch network; The three-dimensional coordinate acquisition module is configured to extract geometric features of the fused image by using an improved UNet segmentation algorithm, i.e., an M-UNet segmentation algorithm, based on the fused thermal imaging image and the visible light image, to obtain a point cloud surrounding box of the spherical heat source by screening point cloud information of the depth camera around the spherical heat source target, and to fuse the point cloud surrounding box with the thermal imaging target image and the visible light image, so as to obtain the three-dimensional coordinates of the geometric center of the spherical heat source by extracting the three-dimensional coordinates of the geometric center of the surrounding box. The relative pose detection module is configured to establish a multi-view thermal imaging camera coordinate system, a thermal imaging target coordinate system, a boom-type tunneling machine coordinate system and a step-type anchor support equipment coordinate system, and to calculate the relative pose relationship between the boom-type tunneling machine and the step-type anchor support equipment by using a tunnel-anchor robot pose solution model, wherein the relative pose relationship includes three attitude information, i.e., roll angle, pitch angle and heading angle, and three-axis position information, i.e., x, y and z. The multi-view thermal imaging camera in the image acquisition module includes an infrared thermal imaging camera, a visible light camera and a depth camera. The three-dimensional coordinate acquisition module further includes a step of adding a multi-scale attention module after a connection layer of an encoder and a decoder in the improved M-UNet segmentation algorithm.

7. Computer device comprising a memory and a processor, characterized in that The memory stores a computer program, and when the processor executes the computer program stored in the memory, the processor executes the method of any one of claims 1-5.

8. A computer readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the processor executes the computer program, the steps of the method of any one of claims 1-5 are implemented.

Images