Robot perception fusion method based on multi-modal attention mechanism and improved YOLOv8

Abstract

A robot perception fusion method for a multi-modal attention mechanism and improved YOLOv8 relates to the technical field of industrial robots, and comprises the following steps: respectively acquiring a visible light image and an infrared image by carrying a visible light camera and an infrared thermal imager, inputting the visible light image into a UNet encoder, extracting edge and structural features, inputting the infrared image into the UNet encoder, and carrying out image fusion on the infrared image and the visible light image; extracting the edge and structure features, extracting the thermal distribution features, inputting the edge and structure features and the thermal distribution features into a cross attention mechanism to obtain fusion features, optimizing a YOLOv8 detection framework, training a data set by using the optimized YOLOv8 detection framework, and outputting a target information position; according to the method, the target detection and feature recognition capability of the inspection robot in a complex industrial environment is improved, memory access is effectively reduced by using a partial convolution Pconv module, an original CIoU / GIoU loss function is replaced by Shape-IoU, the shape overlapping relation between a target frame and a real boundary can be measured more accurately, and bounding box regression is optimized.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of industrial robots, in particular to a robot perception fusion method combining a multi-modal attention mechanism and an improved YOLOv8. BACKGROUND

[0002] At present, the demand for unmanned inspection robots is increasing in the current industrial field, such as petrochemical industry, power inspection, and urban gas pipeline network. The existing robot inspection system generally relies on a single visual mode, such as a visible light camera. However, it is limited by factors such as environmental light changes, smoke interference, and temperature difference insensitivity, resulting in low detection accuracy. At the same time, although the traditional YOLO series detection algorithm has fast inference speed, its performance decreases significantly in the face of small targets and multiple target scenes, and the model parameters are complex, making it difficult to deploy on edge computing devices.

[0003] In addition, most existing multi-modal perception methods only perform simple feature-level splicing, lack explicit guidance and attention enhancement mechanisms between cross-modal, and are difficult to fully exploit the complementary characteristics of infrared and visible light images in structure, thermal, and edge levels. Therefore, there is an urgent need for a lightweight and efficient cross-modal attention mechanism combined with detection model structure optimization to achieve high-robustness perception of targets in complex scenes. SUMMARY

[0004] The purpose of the application is to overcome the shortcomings of the prior art and provide a robot perception fusion method combining a multi-modal attention mechanism and an improved YOLOv8.

[0005] The technical solution adopted by the application is: A robot perception fusion method combining a multi-modal attention mechanism and an improved YOLOv8, specifically comprising the following steps: S1, mounting a visible light camera and an infrared thermal imager to respectively collect visible light images and infrared images ; S2, using a double-flow decomposition-recombination structure, inputting the visible light images in step S1 to a UNet encoder to extract edge and structure features ; inputting the infrared images in step S1 to a UNet encoder to extract thermal distribution features ; wherein, H is the height of the feature matrix, W is the width of the feature matrix, C is the number of channels of the feature matrix; S3, realizing cross-modal feature alignment and complementary enhancement based on a cross-attention mechanism Cross-Attention Mechanism, combining the edge and structure features in step S2 and thermal distribution characteristics input into the cross attention mechanism to obtain fused features ; S4, the output fused features of the previous step are taken as the input of the optimized YOLOv8 detection framework, and the YOLOv8 detection framework is optimized, and the specific structure of the optimized YOLOv8 detection framework is that all 3x3Conv modules of C2f modules of the backbone network in the original YOLOv8 are replaced with partial convolution PConv modules; Shape-IoU Loss is used to replace the original CIoU / GIoU loss function; S5, using the optimized YOLOv8 detection framework to train the data set, and outputting the target information position.

[0006] The multi-modal attention mechanism and the robot perception fusion method of the improved YOLOv8, the calculation process of the cross attention mechanism is: taking the visible light feature as the query vector , the infrared feature as the key and value ; the formula is defined as: ; wherein is the feature dimension, K T is the transpose of the key vector.

[0007] The multi-modal attention mechanism and the robot perception fusion method of the improved YOLOv8, the PConv module replaces the conventional convolution layer, only the first quarter of the input channel is convolved, and the remaining channels are not processed, the FLOPs of the Pconv module is reduced to 1 / 16 of the standard convolution, the formula is: ; wherein represents the first quarter of the input channel, represents the remaining three quarters of the channel, represents channel splicing.

[0008] The multi-modal attention mechanism and the robot perception fusion method of the improved YOLOv8, Shape-IoU Loss is defined as: ; wherein, is the shape perception distance penalty, is the shape penalty, which is used to measure the difference between two boundary boxes in width, height or aspect ratio.

[0009] The multimodal attention mechanism and the improved YOLOv8 robot perception fusion method, in step S5, specifically involve training the optimized YOLOv8 detection framework as follows: Using the LLVIP (Low-Light Vision Infrared-Paired) multimodal dataset, visible light images are... and infrared images The input is fed into the Unet encoder to extract visible light and infrared features. The visible light and infrared features are then fed into the cross-attention mechanism to obtain fused features. The fused features are then fed into the improved YOLOv8 detection framework for training, and finally a neural network model is obtained.

[0010] Due to the adoption of the technical solution described above, the present invention has the following advantages: The multimodal attention mechanism and improved YOLOv8 robot perception fusion method described in this invention use visible light and infrared images as input for multimodal image processing. A two-stream decomposition-reconstruction architecture is employed to extract features from each image, and a cross-attention mechanism is fused to achieve feature-level cross-modal complementary enhancement. By combining this with the optimization of the improved YOLOv8 detection framework, the target detection and feature recognition capabilities of inspection robots in complex industrial environments are improved. The use of a partially convolutional Pconv module effectively reduces memory access, and Shape-IoU replaces the original CIoU / GIoU loss function, enabling more accurate measurement of the shape overlap relationship between the target box and the ground truth boundary, thus optimizing bounding box regression. Attached Figure Description

[0011] Figure 1 This is the overall structural diagram of the present invention. Detailed Implementation

[0012] The present invention will be further explained and described below with reference to the accompanying drawings and embodiments. However, this should not be construed as limiting the scope of protection of the present invention. The purpose of disclosing the present invention is to protect all technical improvements within the scope of the present invention.

[0013] Combined with appendix Figure 1 The aforementioned multimodal attention mechanism and improved YOLOv8 robot perception fusion method specifically include the following steps: S1: Equipped with a visible light camera and an infrared thermal imager, respectively acquiring visible light images. and infrared images ; S2: Using a dual-stream decomposition-reconstruction structure, the visible light image described in step S1 is... The input is fed into the UNet encoder to extract edge and structural features. The infrared image described in step S1 Input to UNet encoder to extract thermal distribution features ;in, H The height of the feature matrix, W The width of the feature matrix. C The number of channels in the feature matrix; The dual-stream decomposition-reconstruction structure is designed to address the differences in perceived content and distribution characteristics between visible light and infrared images. The decomposition stage encodes features for both modalities separately, while the reconstruction stage fuses semantic and thermal information from different modalities to achieve cross-modal perceptual complementarity. Specifically, the visible light image input passes through a UNet encoder to extract edge features. Infrared images are input into a UNet encoder to extract thermal features. It exhibits a stronger response, especially to high-temperature and high-heat targets, such as heating pipes and leaking steam. S3: Achieve cross-modal feature alignment and complementary enhancement based on the cross-attention mechanism, combining the edge and structural features described in step S2. and thermal distribution characteristics Input is fed into a cross-attention mechanism to obtain fused features. ; The computational process of the cross-attention mechanism is as follows: using visible light features as query vector Infrared features As a key Sum The formula is defined as follows: ; in K is the feature dimension. T This is the transpose of the key vector.

[0014] The cross-attention mechanism uses a query-key-value attention structure, with visible light features as the main query and infrared features as key-value pairs.

[0015] S4: To adapt to the deployment requirements of edge computing platforms and improve target detection performance, the fused features output from the previous step are used as input to optimize the YOLOv8 detection framework. The specific structure of the optimized YOLOv8 detection framework is as follows: This includes optimization of the backbone network and improvement of the loss function; The optimization of the backbone network refers to replacing all 3x3Conv modules of the C2f module in the backbone network of the original YOLOv8 detection framework with PConv modules; The Pconv module achieves lightweight computation through a dynamic parameter sharing mechanism, reducing computation by approximately 30% compared to traditional convolutions. It performs convolution only on the first quarter of the input channels, leaving the remaining channels unprocessed. The Pconv module reduces FLOPs to 1 / 16 of standard convolutions, while effectively reducing memory access and significantly lowering model load, making it suitable for deployment on edge platforms such as Jetson, AGX, and Xavier. The Pconv module replaces conventional convolutional layers, as shown in the formula: ; in This represents the first quarter of the input channel. This indicates the remaining three-quarters of the channel. Indicates channel splicing; Shape-IoU Loss is defined as: ; in For shape-perceived distance penalty, Shape penalty is used to measure the difference between two bounding boxes in width, height, or aspect ratio.

[0016] The improvement of the loss function refers to replacing the original CIoU / GIoU loss function with Shape-IoU Loss. Shape-IoU can more accurately measure the shape overlap relationship between the target box and the ground truth boundary, and it has better performance, especially in polygon anomaly detection. S5. Train the dataset using the optimized YOLOv8 detection framework and output the target information location; the specific steps for training the optimized YOLOv8 detection framework are as follows: Using the LLVIP (Low-Light Vision Infrared-Paired) multimodal dataset, visible light images are... and infrared images The input is fed into the Unet encoder to extract visible light and infrared features. The visible light and infrared features are then fed into the cross-attention mechanism to obtain fused features. The fused features are then fed into the improved YOLOv8 detection framework for training, and finally a neural network model is obtained.

[0017] In use, the optimized YOLOv8 detection framework is deployed on a mobile inspection robot platform, equipped with a 2-megapixel visible light camera and a 640×512 resolution infrared thermal imager to collect multimodal image information. Combined with an IMU, LiDAR, and GNSS chip, it achieves centimeter-level positioning accuracy, ensuring that the inspection robot can accurately locate targets. The computing unit uses an NVIDIA Jetson AGX Xavier chip, which enables the invention to have high-precision perception capabilities, excellent real-time performance, and embedded deployment adaptability, making it suitable for autonomous inspection tasks in high-risk scenarios such as power, petroleum, and chemical industries.

[0018] The parts of this invention not described in detail are prior art.

[0019] The embodiments selected herein for the purpose of disclosing the inventive objectives are currently considered suitable; however, it should be understood that the invention is intended to include all variations and modifications of the embodiments that fall within the scope of this concept and invention.

Claims

1. A robot perception fusion method based on multimodal attention mechanism and improved YOLOv8; characterized in that: Specifically, the following steps are included: S1, equipped with a visible light camera and an infrared thermal imager, respectively acquires visible light images. and infrared images ; S2. Using a dual-stream decomposition-reconstruction structure, the visible light image described in step S1 is processed. The input is fed into the UNet encoder to extract edge and structural features. The infrared image described in step S1 Input to UNet encoder to extract thermal distribution features ;in, H The height of the feature matrix, W The width of the feature matrix. C The number of channels in the feature matrix; S3. Based on the Cross-Attention Mechanism, cross-modal feature alignment and complementary enhancement are achieved, combining the edge and structural features described in step S2. and thermal distribution characteristics Input is fed into a cross-attention mechanism to obtain fused features. ; S4. Use the fused features output from the previous step as input to optimize the YOLOv8 detection framework, and optimize the YOLOv8 detection framework. The specific structure of the optimized YOLOv8 detection framework is as follows: replace all 3x3Conv modules of the C2f module in the backbone network of the original YOLOv8 with partially convolutional PConv modules; and use Shape-IoU Loss to replace the original CIoU / GIoU loss function. S5. Train the dataset using the optimized YOLOv8 detection framework and output the target information location.

2. The robot perception fusion method based on multimodal attention mechanism and improved YOLOv8 according to claim 1, characterized in that: The computational process of the cross-attention mechanism is as follows: using visible light features as query vector Infrared features As a key Sum The formula is defined as follows: ； in K is the feature dimension. T This is the transpose of the key vector.

3. The robot perception fusion method based on multimodal attention mechanism and improved YOLOv8 according to claim 1, characterized in that: The PConv module replaces the conventional convolutional layer, performing convolution only on the first quarter of the input channels, leaving the remaining channels unprocessed. The FLOPs of the Pconv module are reduced to 1 / 16 of those of a standard convolution, as shown in the formula: ； in This represents the first quarter of the input channel. This indicates the remaining three-quarters of the channel. This indicates channel splicing.

4. The robot perception fusion method based on multimodal attention mechanism and improved YOLOv8 according to claim 1, characterized in that: The Shape-IoU Loss mentioned in step S4 is defined as follows: ； in, For shape-perceived distance penalty, Shape penalty is used to measure the difference between two bounding boxes in width, height, or aspect ratio.

5. The robot perception fusion method based on multimodal attention mechanism and improved YOLOv8 according to claim 1, characterized in that: In step S5, the specific steps for training the optimized YOLOv8 detection framework are as follows: Using the LLVIP (Low-Light Vision Infrared-Paired) multimodal dataset, visible light images are... and infrared images The input is fed into the Unet encoder to extract visible light and infrared features. The visible light and infrared features are then fed into the cross-attention mechanism to obtain fused features. The fused features are then fed into the improved YOLOv8 detection framework for training, and finally a neural network model is obtained.

Images