A road scene target detection method, system, device and medium based on a Transformer and cross-modal

Abstract

A road scene target detection method, system, device and medium based on a Transformer and cross-modal, the method comprising: constructing a road scene target detection network model based on a Transformer and cross-modal; using an RGB and infrared dual-modal road scene target detection dataset for model training, and using the trained model to realize road scene target detection; the system, device and medium are used to realize the method; the application constructs three key modules of cross-modal feature fusion, direction structure enhancement and scale perception attention, systematically improves the feature representation capability of the model from three aspects of modal complementation, structure expression and scale modeling, and the three modules are mutually coordinated, so that the overall network can still maintain stable and accurate perception performance in a complex environment, and a technical solution with high robustness and generalization capability is provided for multi-modal target detection and scene understanding.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and further relates to deep learning and digital image processing technology. Specifically, it is a method, system, device and medium for road scene target detection based on Transformer and cross-modal. This invention can be used for road scene target detection. Background Technology

[0002] With the rapid development of intelligent transportation systems, autonomous driving technology, and intelligent monitoring systems, target detection technology in complex environments has gradually become an important research direction in the field of computer vision. Among them, pedestrian detection, as a key task in environmental perception systems, is of great significance in application scenarios such as intelligent driving assistance systems, drone monitoring, and smart cities. Traditional pedestrian detection methods usually rely on single-modal data (such as RGB visible light images) for target recognition. Although RGB images can provide rich texture and scene semantic information, their imaging quality is easily affected by factors such as changes in illumination, shadow interference, occlusion, and inclement weather. Significant performance degradation often occurs at night or in low-visibility environments, thus limiting the stability and reliability of the system in complex environments.

[0003] To overcome the limitations of single-modal perception, multimodal perception technology has gained widespread attention in recent years. By fusing multi-source information from different sensors, such as visible light images, infrared thermal imaging, LiDAR, and event cameras, more comprehensive and stable target information can be obtained under various environmental conditions. Infrared images can capture the thermal radiation characteristics of targets, are insensitive to changes in illumination, and still maintain good target saliency in nighttime or low-light environments; while RGB images have significant advantages in representing texture details and structural information. Therefore, through multimodal information fusion, the complementary characteristics between different modalities can be fully utilized, thereby improving the perception capability and robustness of target detection systems in complex environments.

[0004] However, existing multimodal target detection methods still face numerous technical challenges. On the one hand, different modal data exhibit significant differences in imaging mechanisms, information distribution, and feature representation, making effective alignment and deep fusion of multimodal features a persistent research challenge. On the other hand, traditional fusion methods often employ simple feature splicing or weighted fusion strategies, failing to fully exploit the deep semantic relationships and complementary information between multimodal data. Furthermore, in complex real-world scenarios, small-scale targets, background interference, and occlusion can significantly impact detection accuracy. Therefore, constructing an efficient fusion method that fully leverages the advantages of multimodal information to improve the accuracy and robustness of target detection in complex environments has become a crucial technical problem urgently needing to be solved in the field of multimodal perception.

[0005] Patent application CN116453014A discloses a multimodal road scene target detection method based on images and events. The method incorporates an efficient attention mechanism module to fuse information from images at adjacent time points, improving detection accuracy while maintaining almost no change in model parameters and computational cost. A lightweight feature extraction network is used to extract features from events, reducing model memory size and aligning with the high temporal resolution of event data. To reuse the temporal information of event data, a long short-term memory (LSTM) mechanism module is introduced, fusing different event features from historical time points by controlling the state of internal modules. During image and event feature fusion, a feature fusion module is used to complete information interaction between the two modalities, improving target detection accuracy. However, this invention still has shortcomings. The method fails to address the difficulties in fusion between the two modalities in terms of temporal synchronization, spatial alignment, and feature representation, resulting in relatively low detection accuracy.

[0006] Patent application CN120408372A discloses a road scene recognition method and system based on modal information evaluation. The method's implementation steps include constructing a multi-task scene recognition model based on extracted multimodal features; the multimodal features include audio features and video image frame features; the multi-task scene recognition model includes a cooperative network and a backbone classification network, with the cooperative network extracting VGG16 visual features. Places365 is used as the modal quality assessment network, and the BRISQUE algorithm based on locally normalized brightness coefficients is used as the teacher model to provide learning reference labels for the modal quality assessment network. The backbone classification network uses a multimodal DBN network to perform unsupervised joint representation of visual and audio scene information. However, this invention still has shortcomings. The method mainly relies on the traditional VGG16-places365 and DBN network structure, which is insufficient in its ability to model deep semantic relationships and temporal dynamic features between multimodal data. Therefore, the feature fusion effect and recognition accuracy in complex road environments are still limited.

[0007] Defects and shortcomings of existing technology: 1. Existing technologies still fail to effectively address the differences in time synchronization, spatial alignment, and feature representation between different modalities during multimodal fusion, resulting in limited cross-modal information collaborative utilization. Consequently, the stability and accuracy of detection results may still be affected under complex scene conditions.

[0008] 2. Existing methods still mainly rely on traditional deep network architectures in model structure design. They are relatively insufficient in modeling deep semantic relationships between multimodal data and dynamic temporal features in complex environments, making it difficult to fully explore the complementary information between multimodal data. Therefore, there is still room for improvement in feature fusion effect and scene recognition accuracy in complex road environments. Summary of the Invention

[0009] To overcome the shortcomings of existing technologies, this invention aims to provide a method, system, device, and medium for road scene target detection based on Transformer and cross-modal approaches. It introduces a Cross Modality Transformer Fusion (CMTF) module to achieve deep feature fusion and global context modeling using cross-modal self-attention. A Directional Feature Enhancement (DFEM) module is introduced to enhance the directional perception of target features through directional convolutional branches. A Scale-aware Multi-level Attention (SMAM) module is introduced to adaptively fuse multi-layer and multi-scale features, combining channel and spatial attention with global Transformer encoding to improve the perception of small and multi-scale targets. The fused features are then fed into an anchorless detection head for target classification and localization. The overall framework, through cross-modal complementarity, directional enhancement, and multi-scale modeling, achieves robust and high-precision target detection in complex road environments. Actual testing demonstrates that this invention can achieve high-precision road scene target detection.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: A road scene target detection method based on Transformer and cross-modal approaches includes the following steps: Step 1: Construct a road scene object detection network model based on Transformer and cross-modal architecture, including two backbone networks, a cross-modal Transformer fusion module CMTF, a soft attention mechanism module SCAM, a directional feature enhancement module DFEM, a scale-aware multi-layer attention module SMAM, and four detection heads; where: The backbone network is used to extract features from RGB and infrared images respectively, and inputs the extracted features into the cross-modal Transformer fusion module (CMTF). The CMTF utilizes cross-modal self-attention to achieve deep feature fusion and global context modeling, and its output is input into the soft attention mechanism module (SCAM). The SCAM constructs attention weights based on global context to adaptively weight the input features, modeling the long-distance dependency between small targets and complex backgrounds, and its output is input into the directional feature enhancement module (DFEM). The DFEM enhances features through directional convolutional branches. The system enhances the perception of target features by incorporating directional awareness. The outputs of the cross-modal Transformer fusion module (CMTF), the soft attention mechanism module (SCAM), and the directional feature enhancement module (DFEM) are all input into the scale-aware multi-layer attention module (SMAM). The SMAM combines channel and spatial attention with global Transformer encoding to improve the perception of small and multi-scale targets. The outputs of these modules are then input into the four detection heads for target classification and localization. Step 2: Train the model using the RGB and infrared dual-modal road scene target detection dataset, and use the trained model to detect road scene targets.

[0011] The backbone network is the MobileNet V2 network.

[0012] The cross-modal Transformer fusion module (CMTF) includes two 1*1 convolutional modules (Conv1*1), two Flatten modules, a multi-head self-attention mechanism (MHSA), a layer normalization layer (LN), a feedforward neural network (FFN), and three summation operation modules; wherein: Given output features from the backbone network, including RGB image features and infrared image features RGB image features serve as input features for the cross-modal Transformer fusion module CMTF. The features are obtained by sequentially passing the first 1x1 convolution Conv1*1 and the first Flatten module. Infrared image features The features are obtained by sequentially passing the second 1x1 convolution Conv1*1 and the second Flatten module. ;feature and characteristics Features were obtained through the multi-head self-attention mechanism MHSA. ; Use the first summation operation module to extract features and characteristics By fusing, features are obtained. The second summation module is used to extract the features. and characteristics By fusing, features are obtained. ;feature After layer normalization LN, the features are obtained ;feature Features are obtained through a feedforward neural network (FFN). Finally, the third summation operation is used to sum the features. and characteristics The features are fused to obtain the output features. .

[0013] The Directional Feature Enhancement (DFEM) module includes a 1*3 convolution Conv1*3, a 3*1 convolution Conv3*1, a 1*1 convolution Conv1*1, two 3*3 convolution Conv3*3, a SiLU activation function, a Flatten module, a Multi-Head Self-Attention (MHSA) mechanism, a Concat operation module, a dot product operation module, and a summation operation module; wherein: Given features from the soft attention mechanism module SCAM The features are obtained by performing a 1x3 convolution (Conv1*3), a 3x1 convolution (Conv3*1), and a first 3x3 convolution (Conv3*3). , and Use the Concat operation module to extract features , and By fusing, features are obtained. ;feature Features are obtained by performing a 1x1 convolution Conv1*1. ;feature The features are obtained by sequentially passing the second 3x3 convolution Conv3*3 and the SiLU activation function. ; Use the dot product operation module to transform features and characteristics By fusing, features are obtained. ;feature Features are obtained sequentially through the Flatten module and the Multi-Head Self-Attention (MHSA) module. Finally, the summation module is used to sum the features. and characteristics By fusing, features are obtained. .

[0014] The scale-aware multilayer attention module (SMAM) includes three downsampling modules (Downsample), three global average pooling modules (GAP), three fully connected modules (FC), six SiLU activation functions, six dot product operation modules, three average pooling modules (AvgPool), three max pooling modules (MaxPool), three 7*7 convolutional modules (Conv7*7), one flatten module, one multi-head self-attention module (MHSA), and one summation operation module; wherein: Given the output features from the cross-modal Transformer fusion module CMTF, the soft attention mechanism module SCAM, and the orientation feature enhancement module DFEM , and Features serve as input features for the Scale-Aware Multilayer Attention Module (SMAM); Features are obtained through the first downsampling module, Downsample. ;feature The features are obtained by sequentially passing the first global average pooling module (GAP), the first fully connected module (FC), and the first SiLU activation function. ;feature Features are obtained through the second downsampling module, Downsample. ;feature The features are obtained by sequentially passing the second global average pooling module (GAP), the second fully connected module (FC), and the second SiLU activation function. ;feature Features are obtained through the third downsampling module, Downsample. ;feature The features are obtained by sequentially passing the third global average pooling module (GAP), the third fully connected module (FC), and the third SiLU activation function. ; Use the first dot product operation module to transform the features and characteristics By fusing, features are obtained. ; Use the second dot product operation module to transform the features and characteristics By fusing, features are obtained. ; Use the third dot product operation module to transform the features and characteristics By fusing, features are obtained. ; Features The features are obtained by first performing average pooling (AvgPool) and first performing max pooling (MaxPool), and then by sequentially passing the two outputs through a first 7x7 convolution (Conv7x7) and a fourth SiLU activation function. ; Features The features are obtained by performing a second average pooling (AvgPool) and a second max pooling (MaxPool) on the outputs, followed by a second 7x7 convolution (Conv7x7) and a fifth SiLU activation function. ; Features The features are obtained by passing the data through a third average pooling (AvgPool) and a third max pooling (MaxPool), followed by passing both outputs through a third 7x7 convolution (Conv7x7) and a sixth SiLU activation function. The fourth dot product operation module is used to transform the features. and By fusing, features are obtained. ;Use the fifth dot product operation module to transform features and By fusing, features are obtained. ;Use the sixth dot product operation module to transform features and By fusing, features are obtained. ; Features , and Features are obtained by sequentially passing through the Flatten module and the Multi-Head Self-Attention (MHSA) module. Finally, the summation module is used to sum the features. , , and The features are fused to obtain the output features. .

[0015] A road scene target detection system based on Transformer and cross-modal approaches, used to implement the above method, includes: Network building blocks: These are used to construct a road scene object detection network model based on Transformer and cross-modal approaches. The model includes two backbone networks, a cross-modal Transformer fusion module (CMTF), a soft attention mechanism module (SCAM), a directional feature enhancement module (DFEM), a scale-aware multi-layer attention module (SMAM), and four detection heads. The backbone network is used to extract features from RGB and infrared images respectively, and inputs the extracted features into the cross-modal Transformer fusion module (CMTF). The CMTF utilizes cross-modal self-attention to achieve deep feature fusion and global context modeling, and its output is input into the soft attention mechanism module (SCAM). The SCAM constructs attention weights based on global context to adaptively weight the input features, modeling the long-distance dependency between small targets and complex backgrounds, and its output is input into the directional feature enhancement module (DFEM). The DFEM enhances features through directional convolutional branches. The system enhances the perception of target features by incorporating directional awareness. The outputs of the cross-modal Transformer fusion module (CMTF), the soft attention mechanism module (SCAM), and the directional feature enhancement module (DFEM) are all input into the scale-aware multi-layer attention module (SMAM). The SMAM combines channel and spatial attention with global Transformer encoding to improve the perception of small and multi-scale targets. The outputs of these modules are then input into the four detection heads for target classification and localization. Network training and object detection module: used to train the model using RGB and infrared dual-modal road scene object detection datasets, and to use the trained model to realize road scene object detection.

[0016] The present invention also provides a road scene target detection device based on Transformer and cross-modal, comprising: The memory stores the computer program of the road scene target detection method based on Transformer and cross-modal, and is a computer-readable device. A processor is used to implement the aforementioned road scene target detection method based on Transformer and cross-modal when executing the computer program.

[0017] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the aforementioned road scene target detection method based on Transformer and cross-modal approaches.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention designs a cross-modal Transformer fusion module (CMTF), whose input comes from the output features of the backbone network (MobileNet V2). This module achieves deep fusion between RGB image features and thermal infrared image features through a cross-modal self-attention mechanism. In this structure, features of different modalities are first mapped to a unified embedding space through 1×1 convolution, thereby reducing the problem of inconsistent feature distribution caused by modal differences. Subsequently, the correlation between cross-modal features is modeled using a multi-head self-attention mechanism (MHSA), enabling the network to capture important contextual dependencies globally and further enhance feature representation capabilities with a feedforward neural network. In this way, the rich texture and color information in the RGB modality can complement the stable target contour information in the thermal infrared modality, thereby achieving information synergy and semantic enhancement between heterogeneous modalities. This not only effectively alleviates the perception limitations of a single modality in complex environments but also significantly improves the target representation capability and robustness of the model under conditions of illumination changes, occlusion, and complex backgrounds.

[0019] (2) This invention designs a directional feature enhancement module DFEM, whose input comes from the output features of the soft attention mechanism module SCAM. This module enhances the modeling of the directional structural information of the target by introducing a multi-branch directional convolutional structure and combining local convolutional attention and global Transformer attention mechanisms. Specifically, the module extracts directionally sensitive structural features through multi-directional branches such as 1×3 convolution, 3×1 convolution, and 3×3 convolution to capture the geometric shape of the target in the horizontal, vertical, and diagonal directions; then, feature fusion is achieved through feature concatenation and 1×1 convolution, and nonlinear activation is combined to enhance the expressive power, and then feature adaptive weighting is achieved through dot product operation; at the same time, a global modeling branch based on a multi-head self-attention mechanism is introduced to model and fuse long-distance contextual information. Through this multi-level feature enhancement strategy, the model can more accurately characterize the target structure with obvious directional characteristics. In scenes with occlusion, cluttered backgrounds, or complex target shapes, it effectively improves the recognition and expressive power of the target structure, thereby improving detection and recognition performance.

[0020] (3) This invention designs a scale-aware multi-layer attention module (SMAM), whose input comes from the output features of the cross-modal Transformer fusion module (CMTF), the soft attention mechanism module (SCAM), and the orientation feature enhancement module (DFEM), to achieve adaptive fusion of multi-level and multi-scale features. This module first obtains feature representations at different scales through a multi-branch downsampling structure, and combines global average pooling with fully connected layers to generate channel attention weights, achieving dynamic modeling of the importance of features at different levels. Simultaneously, it constructs spatial attention branches by combining average pooling and max pooling with 7×7 convolutions to enhance the perception of target spatial location information. Furthermore, it introduces a Transformer-based cross-scale context modeling mechanism to model the global dependencies of the fused multi-scale features, thereby achieving effective interaction and supplementation between information at different scales. Through the above design, the model significantly enhances its perception of small targets and targets with varying scales while maintaining global semantic understanding capabilities. Especially in complex scenes, it effectively alleviates the problem of insufficient feature representation caused by differences in target size, thereby improving the overall stability and accuracy of detection.

[0021] In summary, this invention systematically enhances the feature representation capability of the model from three levels: modal complementarity, structural representation, and scale modeling, by constructing three key modules: cross-modal feature fusion, directional structure enhancement, and scale-aware attention. Specifically, the cross-modal fusion module achieves deep synergy between RGB and thermal infrared information, the directional feature enhancement module strengthens the structural and morphological representation of the target, and the scale-aware multi-level attention module further improves the model's adaptability to multi-scale targets. The synergy of these three modules enables the overall network to maintain stable and accurate perception performance even in complex environments, providing a highly robust and generalizable technical solution for multimodal target detection and scene understanding. Attached Figure Description

[0022] Figure 1 This is a network structure diagram of an embodiment of the present invention.

[0023] Figure 2 This is a structural diagram of the CMTF module according to an embodiment of the present invention.

[0024] Figure 3 This is a structural diagram of the DFEM module according to an embodiment of the present invention.

[0025] Figure 4 This is a structural diagram of the SMAM module according to an embodiment of the present invention.

[0026] Figure 5 This is a simulation visualization result diagram of the present invention. Detailed Implementation

[0027] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0028] A road scene target detection method based on Transformer and cross-modal approaches includes the following steps: Step 1: As Figure 1 As shown, a road scene object detection network model based on Transformer and cross-modal approaches is constructed, including two backbone networks, a cross-modal Transformer Fusion (CMTF) module, a soft attention mechanism (SCAM) module, a directional feature enhancement module (DFEM), a scale-aware multi-level attention module (SMAM), and four detection heads; wherein: The backbone network is a MobileNet V2 network, used to extract features from RGB and infrared images respectively, and input the extracted features into the Cross Modality Transformer Fusion (CMTF) module. The CMTF module utilizes cross-modal self-attention to achieve deep feature fusion and global context modeling, and its output is input into the Soft Attention Mechanism (SCAM) module. The SCAM module adaptively weights the input features by constructing attention weights based on global context, modeling the long-distance dependency between small targets and complex backgrounds, and its output is input into the Directional Feature Enhancement Module (DFEM) module. The DFEM module enhances the directional perception capability of target features through directional convolution branches. The CMTF module and the Soft Attention Mechanism module are then combined. The outputs of both the SCAM (Structured Computing Module) and the Directional Feature Enhancement Module (DFEM) are input to the Scale-aware Multi-level Attention Module (SMAM). The Scale-aware Multi-level Attention Module (SMAM) combines channel and spatial attention with global Transformer encoding to improve the perception of small and multi-scale targets.The outputs of the Cross Modality Transformer Fusion (CMTF), Soft Attention Mechanism (SCAM), Directional Feature Enhancement Module (DFEM), and Scale-aware Multi-level Attention Module (SMAM) are respectively input into the anchorless detection head for target classification and localization. Step 101: As Figure 2 As shown, the cross-modal Transformer fusion module (CMTF) includes two 1x1 convolutional modules (Conv1*1), two Flatten modules, a multi-head self-attention mechanism (MHSA), a layer normalization layer (LN), a feedforward neural network (FFN), and three summation operations. ;in: Given output features from the backbone network (MobileNet V2), including RGB image features and infrared image features RGB image features serve as input features for the cross-modal Transformer fusion module CMTF. The features are obtained by sequentially passing the first 1x1 convolution Conv1*1 and the first Flatten module. Infrared image features The features are obtained by sequentially passing the second 1x1 convolution Conv1*1 and the second Flatten module. ;feature and characteristics Features were obtained through the multi-head self-attention mechanism MHSA. ; Using the first summation operation Features and characteristics By fusing, features are obtained. ; Using the second summation operation Features and characteristics By fusing, features are obtained. ;feature After layer normalization LN, the features are obtained ;feature Features are obtained through a feedforward neural network (FFN). Finally, use the third summation operation. Features and characteristics The features are fused to obtain the output features. ; The Cross-Modal Transformer Fusion Module (CMTF) is represented by the following formula: .

[0029] Step 102: As Figure 3 As shown, the Directional Feature Enhancement (DFEM) module includes a 1x3 convolution Conv1*3, a 3x1 convolution Conv3*1, a 1x1 convolution Conv1*1, two 3x3 convolutions Conv3*3, a SiLU activation function, a Flatten module, a Multi-Head Self-Attention (MHSA) mechanism, a Concat operation, and a dot product operation. and a summation operation ;in: Given features from the soft attention mechanism module SCAM The features are obtained by performing a 1x3 convolution (Conv1*3), a 3x1 convolution (Conv3*1), and a first 3x3 convolution (Conv3*3). , and Use the Concat operation to combine features , and By fusing, features are obtained. ;feature Features are obtained by performing a 1x1 convolution Conv1*1. ;feature The features are obtained by sequentially passing the second 3x3 convolution Conv3*3 and the SiLU activation function. Using dot product operation Features and characteristics By fusing, features are obtained. ;feature Features are obtained sequentially through the Flatten module and the Multi-Head Self-Attention (MHSA) module. Finally, use the summation operation. Features and characteristics By fusing, features are obtained. ; The Directional Feature Enhancement Module (DFEM) formula is expressed as follows: .

[0030] Step 103: As Figure 4 As shown, the Scale-Aware Multilayer Attention Module (SMAM) includes three downsampling modules (Downsample), three global average pooling modules (GAP), three fully connected modules (FC), six SiLU activation functions, and six dot product operations. Three average pooling (AvgPool), three max pooling (MaxPool), three 7x7 convolutional (Conv7x7), one Flatten module, one multi-head self-attention (MHSA) module, and one summation operation. ;in: Given the output features from the cross-modal Transformer fusion module CMTF, the soft attention mechanism module SCAM, and the orientation feature enhancement module DFEM , and Features serve as input features for the Scale-Aware Multilayer Attention Module (SMAM); Features are obtained through the first downsampling module, Downsample. ;feature The features are obtained by sequentially passing the first global average pooling module (GAP), the first fully connected module (FC), and the first SiLU activation function. ;feature Features are obtained through the second downsampling module, Downsample. ;feature The features are obtained by sequentially passing the second global average pooling module (GAP), the second fully connected module (FC), and the second SiLU activation function. ;feature Features are obtained through the third downsampling module, Downsample. ;feature The features are obtained by sequentially passing the third global average pooling module (GAP), the third fully connected module (FC), and the third SiLU activation function. ; Using the first dot product operation Features and characteristics By fusing, features are obtained. ; Using the second dot product operation Features and characteristics By fusing, features are obtained. ; Using the third dot product operation Features and characteristics By fusing, features are obtained. ; Features The features are obtained by first performing average pooling (AvgPool) and first performing max pooling (MaxPool), and then by sequentially passing the two outputs through a first 7x7 convolution (Conv7x7) and a fourth SiLU activation function. ; Features The features are obtained by performing a second average pooling (AvgPool) and a second max pooling (MaxPool) on the outputs, followed by a second 7x7 convolution (Conv7x7) and a fifth SiLU activation function. ; Features The features are obtained by passing the data through a third average pooling (AvgPool) and a third max pooling (MaxPool), followed by passing both outputs through a third 7x7 convolution (Conv7x7) and a sixth SiLU activation function. ; Using the fourth dot product operation Features and By fusing, features are obtained. ; Using the fifth dot product operation Features and By fusing, features are obtained. ; Using the sixth dot product operation Features and By fusing, features are obtained. ; Features , and Features are obtained by sequentially passing through the Flatten module and the Multi-Head Self-Attention (MHSA) module. Finally, use the summation operation. Features , , and The features are fused to obtain the output features. .

[0031] The formula for the Scale-Aware Multilayer Attention Module (SMAM) is as follows: in, This indicates the downsampling module, Downsample.

[0032] Step 2: Set the number of training rounds to ≥200, the batch size to ≥24, and train the Transformer and cross-modal road scene object detection network model built in Step 1 using the training set in the RGB and infrared dual-modal road scene object detection public dataset. Output the training weight file CMT-Det.pt. Step 3: With a batch size ≥ 8, use the test set from the RGB and infrared dual-modal road scene object detection public dataset and the weight file CMT-Det.pt obtained in Step 2 to perform road scene object detection on the road scene object detection network model based on Transformer and cross-modal modes constructed in Step 1, and obtain the object detection results.

[0033] This invention also provides a road scene target detection system based on Transformer and cross-modal approaches, comprising: Network building blocks: These are used to construct a road scene object detection network model based on Transformer and cross-modal architectures. They include two backbone networks, a Cross Modality Transformer Fusion (CMTF) module, a Soft Attention Mechanism (SCAM) module, a Directional Feature Enhancement Module (DFEM) module, a Scale-aware Multi-level Attention Module (SMAM) module, and four detection heads. The backbone network is a MobileNet V2 network, used to extract features from RGB and infrared images respectively, and input the extracted features into the Cross Modality Transformer Fusion (CMTF) module. The CMTF module utilizes cross-modal self-attention to achieve deep feature fusion and global context modeling, and its output is input into the Soft Attention Mechanism (SCAM) module. The SCAM module adaptively weights the input features by constructing attention weights based on global context, modeling the long-distance dependency between small targets and complex backgrounds, and its output is input into the Directional Feature Enhancement Module (DFEM) module. The DFEM module enhances the directional perception capability of target features through directional convolution branches. The CMTF module and the Soft Attention Mechanism module are then combined. The outputs of both the SCAM (Structured Computing Module) and the Directional Feature Enhancement Module (DFEM) are input to the Scale-aware Multi-level Attention Module (SMAM). The SMAM combines channel and spatial attention with global Transformer encoding to improve the perception of small and multi-scale targets.The outputs of the Cross Modality Transformer Fusion (CMTF), Soft Attention Mechanism (SCAM), Directional Feature Enhancement Module (DFEM), and Scale-aware Multi-level Attention Module (SMAM) are respectively input into the anchorless detection head for target classification and localization. Network training module: Set the number of training rounds to ≥200, the batch size to ≥24, and use the training set in the RGB and infrared dual-modal road scene object detection public dataset to train the road scene object detection network model based on Transformer and cross-modal constructed in step one, and output the training weight file CMT-Det.pt; Object detection module: Batch size ≥ 8. The test set in the RGB and infrared dual-modal road scene object detection public dataset and the weight file CMT-Det.pt obtained in step 2 are used to perform road scene object detection on the road scene object detection network model based on Transformer and cross-modality constructed in step 1, and the object detection results are obtained.

[0034] The present invention also provides a road scene target detection device based on Transformer and cross-modal, comprising: The memory stores the computer program of the road scene target detection method based on Transformer and cross-modal, and is a computer-readable device. A processor is used to implement the aforementioned road scene target detection method based on Transformer and cross-modal when executing the computer program.

[0035] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the aforementioned road scene target detection method based on Transformer and cross-modal approaches.

[0036] The effects of the present invention will be further illustrated below with simulation experiments: 1. Simulation Experiment Conditions The hardware platform for the simulation experiment of this invention is: Intel i9-13900K processor with a main frequency of 3.0GHz and 64G RAM.

[0037] The software platform of the simulation experiment platform of this invention is: Windows 11 operating system and PyCharm, PyTorch 2.1.0, CUDA 12.1.

[0038] 2. Simulation Steps The training set of a publicly available dataset for RGB and infrared dual-modal road scene object detection is input into a road scene object detection network model based on Transformer and cross-modality for optimization training. The training process is as follows: the input image is processed by a feature extraction network to extract features, resulting in feature maps of different scales. These feature maps are then reconstructed to obtain more refined feature maps. Based on this, the loss is calculated to complete the road scene detection based on this invention. The number of iterations (epochs) is set to 200.

[0039] 3. Simulation content and result analysis The following is combined Figure 5 The simulation effects of the present invention will be further described.

[0040] Figure 5 This is a visualization of the road scene detection of this invention. Columns 1 and 3 are RGB images, and columns 2 and 4 are infrared images. Figure 5 As shown, although RGB images provide rich appearance information and contextual semantic cues, their performance is easily affected by complex lighting conditions and occlusion factors, resulting in significant degradation. Infrared images, on the other hand, can highlight the thermal radiation characteristics of pedestrians and maintain good stability in low-light or low-illumination environments, but their structural detail information is relatively scarce. By effectively fusing complementary feature information from the two modalities, this invention can achieve accurate pedestrian localization under various complex scene conditions, including low-visibility environments, cluttered backgrounds, severe occlusion, and small-scale targets. Experimental results show that this invention exhibits good robustness and stability in real RGB-T road scenes, providing a reliable and effective solution for pedestrian detection tasks in complex environments.

Claims

1. A road scene target detection method based on Transformer and cross-modal methods, characterized in that, Includes the following steps: Step 1: Construct a road scene object detection network model based on Transformer and cross-modal architecture, including two backbone networks, a cross-modal Transformer fusion module CMTF, a soft attention mechanism module SCAM, a directional feature enhancement module DFEM, a scale-aware multi-layer attention module SMAM, and four detection heads; where: The backbone network is used to extract features from RGB and infrared images respectively, and inputs the extracted features into the cross-modal Transformer fusion module CMTF; the cross-modal Transformer fusion module CMTF uses cross-modal self-attention to achieve deep feature fusion and global context modeling, and inputs its output into the soft attention mechanism module SCAM; The soft attention mechanism module SCAM adaptively weights the input features by constructing attention weights based on global context, models the long-distance dependency between small targets and complex backgrounds, and inputs its output to the directional feature enhancement module DFEM. The directional feature enhancement module DFEM strengthens the directional perception capability of target features through directional convolution branches. The outputs of the cross-modal Transformer fusion module CMTF, the soft attention mechanism module SCAM, and the directional feature enhancement module DFEM are all input to the scale-aware multi-layer attention module SMAM. The scale-aware multi-layer attention module SMAM combines channel and spatial attention as well as global Transformer encoding to improve the perception capability of small targets and multi-scale targets. The outputs of the cross-modal Transformer fusion module CMTF, the soft attention mechanism module SCAM, the directional feature enhancement module DFEM, and the scale-aware multi-layer attention module SMAM are respectively input to the four detection heads for target classification and localization. Step 2: Train the model using the RGB and infrared dual-modal road scene target detection dataset, and use the trained model to detect road scene targets.

2. The road scene target detection method based on Transformer and cross-modal as described in claim 1, characterized in that, The backbone network is the MobileNet V2 network.

3. The road scene target detection method based on Transformer and cross-modal as described in claim 1, characterized in that, The cross-modal Transformer fusion module (CMTF) includes two 1*1 convolutional modules (Conv1*1), two Flatten modules, a multi-head self-attention mechanism (MHSA), a layer normalization layer (LN), a feedforward neural network (FFN), and three summation operation modules; wherein: Given output features from the backbone network, including RGB image features and infrared image features RGB image features serve as input features for the cross-modal Transformer fusion module CMTF. The features are obtained by sequentially passing the first 1x1 convolution Conv1*1 and the first Flatten module. Infrared image features The features are obtained by sequentially passing the second 1x1 convolution Conv1*1 and the second Flatten module. ;feature and characteristics Features were obtained through the multi-head self-attention mechanism MHSA. ; Use the first summation operation module to extract features and characteristics By fusing, features are obtained. The second summation module is used to extract the features. and characteristics By fusing, features are obtained. ;feature After layer normalization LN, the features are obtained ;feature Features are obtained through a feedforward neural network (FFN). Finally, the third summation operation is used to sum the features. and characteristics The features are fused to obtain the output features. .

4. The road scene target detection method based on Transformer and cross-modal as described in claim 1, characterized in that, The Directional Feature Enhancement (DFEM) module includes a 1*3 convolution Conv1*3, a 3*1 convolution Conv3*1, a 1*1 convolution Conv1*1, two 3*3 convolution Conv3*3, a SiLU activation function, a Flatten module, a Multi-Head Self-Attention (MHSA) mechanism, a Concat operation module, a dot product operation module, and a summation operation module; wherein: Given features from the soft attention mechanism module SCAM The features are obtained by performing a 1x3 convolution (Conv1*3), a 3x1 convolution (Conv3*1), and a first 3x3 convolution (Conv3*3). , and Use the Concat operation module to extract features , and By fusing, features are obtained. ;feature Features are obtained by performing a 1x1 convolution Conv1*1. ;feature The features are obtained by sequentially passing the second 3x3 convolution Conv3*3 and the SiLU activation function. ; Use the dot product operation module to transform features and characteristics By fusing, features are obtained. ;feature Features are obtained sequentially through the Flatten module and the Multi-Head Self-Attention (MHSA) module. Finally, the summation module is used to sum the features. and characteristics By fusing, features are obtained. .

5. The road scene target detection method based on Transformer and cross-modal as described in claim 1, characterized in that, The scale-aware multilayer attention module (SMAM) includes three downsampling modules (Downsample), three global average pooling modules (GAP), three fully connected modules (FC), six SiLU activation functions, six dot product operation modules, three average pooling modules (AvgPool), three max pooling modules (MaxPool), three 7*7 convolutional modules (Conv7*7), one flatten module, one multi-head self-attention module (MHSA), and one summation operation module; wherein: Given the output features from the cross-modal Transformer fusion module CMTF, the soft attention mechanism module SCAM, and the orientation feature enhancement module DFEM , and Features serve as input features for the Scale-Aware Multilayer Attention Module (SMAM); Features are obtained through the first downsampling module, Downsample. ;feature The features are obtained by sequentially passing the first global average pooling module (GAP), the first fully connected module (FC), and the first SiLU activation function. ;feature Features are obtained through the second downsampling module, Downsample. ;feature The features are obtained by sequentially passing the second global average pooling module (GAP), the second fully connected module (FC), and the second SiLU activation function. ;feature Features are obtained through the third downsampling module, Downsample. ;feature The features are obtained by sequentially passing the third global average pooling module (GAP), the third fully connected module (FC), and the third SiLU activation function. ; Use the first dot product operation module to transform the features and characteristics By fusing, features are obtained. ; Use the second dot product operation module to transform the features and characteristics By fusing, features are obtained. ; Use the third dot product operation module to transform the features and characteristics By fusing, features are obtained. ; Features The features are obtained by first performing average pooling (AvgPool) and first performing max pooling (MaxPool), and then by sequentially passing the two outputs through a first 7x7 convolution (Conv7x7) and a fourth SiLU activation function. ; Features The features are obtained by performing a second average pooling (AvgPool) and a second max pooling (MaxPool) on the outputs, followed by a second 7x7 convolution (Conv7x7) and a fifth SiLU activation function. ; Features The features are obtained by passing the data through a third average pooling (AvgPool) and a third max pooling (MaxPool), followed by passing both outputs through a third 7x7 convolution (Conv7x7) and a sixth SiLU activation function. The fourth dot product operation module is used to transform the features. and By fusing, features are obtained. ;Use the fifth dot product operation module to transform features and By fusing, features are obtained. ;Use the sixth dot product operation module to transform features and By fusing, features are obtained. ; Features , and Features are obtained by sequentially passing through the Flatten module and the Multi-Head Self-Attention (MHSA) module. Finally, the summation module is used to sum the features. , , and The features are fused to obtain the output features. .

6. A road scene target detection system based on Transformer and cross-modal methods, used to implement the method of claim 1, characterized in that, include: Network building blocks: These are used to construct a road scene object detection network model based on Transformer and cross-modal approaches. The model includes two backbone networks, a cross-modal Transformer fusion module (CMTF), a soft attention mechanism module (SCAM), a directional feature enhancement module (DFEM), a scale-aware multi-layer attention module (SMAM), and four detection heads. The backbone network is used to extract features from RGB and infrared images respectively, and inputs the extracted features into the cross-modal Transformer fusion module CMTF; the cross-modal Transformer fusion module CMTF uses cross-modal self-attention to achieve deep feature fusion and global context modeling, and inputs its output into the soft attention mechanism module SCAM; The soft attention mechanism module SCAM adaptively weights the input features by constructing attention weights based on global context, models the long-distance dependency between small targets and complex backgrounds, and inputs its output to the directional feature enhancement module DFEM. The directional feature enhancement module DFEM strengthens the directional perception capability of target features through directional convolution branches. The outputs of the cross-modal Transformer fusion module CMTF, the soft attention mechanism module SCAM, and the directional feature enhancement module DFEM are all input to the scale-aware multi-layer attention module SMAM. The scale-aware multi-layer attention module SMAM combines channel and spatial attention as well as global Transformer encoding to improve the perception capability of small targets and multi-scale targets. The outputs of the cross-modal Transformer fusion module CMTF, the soft attention mechanism module SCAM, the directional feature enhancement module DFEM, and the scale-aware multi-layer attention module SMAM are respectively input to the four detection heads for target classification and localization. Network training and object detection module: used to train the model using RGB and infrared dual-modal road scene object detection datasets, and to use the trained model to realize road scene object detection.

7. A road scene target detection device based on Transformer and cross-modal methods, characterized in that, include: The memory stores the computer program of the road scene target detection method based on Transformer and cross-modal as described in any one of claims 1-5, and is a computer-readable device; A processor, configured to execute the computer program to implement the road scene target detection method based on Transformer and cross-modal as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the road scene target detection method based on Transformer and cross-modal as described in any one of claims 1-5.

Images