Visible light and infrared cross-modal target detection method based on selective state space

Abstract

The invention provides a visible light and infrared cross-modal target detection method based on a selective state space. According to the method, a novel simple module is provided, and the module carries out linear combination on visible light channels to generate a grayscale image with random weight. By dynamically adjusting the channel weight, the gray value distribution range of the visible light image is expanded, so that the statistical characteristics of the visible light image are better matched with the target characteristics of the infrared image. Meanwhile, a selective state space feature scanning mechanism of a Mama framework is integrated into a YOLO detection framework in a clue mining module form, the efficient calculation characteristic and long sequence modeling capability of linear complexity of the mechanism are brought into full play, and aiming at the characteristics of low contrast and fuzzy contour features of an infrared image, feature weights are dynamically distributed, so that the detection accuracy of the infrared image is improved. According to the method, the edge and geometric structure of the target are accurately focused, the spatial continuity and semantic relevance of shallow features are enhanced, key information such as the contour of the infrared target is quickly captured and enhanced, and the infrared target recognition accuracy of the model in a complex scene is improved.

Description

Technical Field

[0001] This invention relates to a method for cross-modal target detection in visible light and infrared based on selective state space, belonging to the field of infrared guided imaging target detection. Background Technology

[0002] This invention focuses on cross-modal target detection, specifically by using visible light modal training to improve infrared modal detection performance, similar to domain generalization tasks. The core challenge stems from the varying difficulty of acquiring data from different modalities: visible light data is relatively easy to acquire, primarily capturing color and texture information of objects; while infrared data is more difficult to acquire, especially under specific conditions such as changes in the thermal state of objects (e.g., moving versus stationary vehicles). The difficulty in acquiring infrared data highlights the necessity of exploring the generalization capabilities of visible light data for infrared detection tasks. Acquiring infrared images is limited by equipment cost, scene thermal radiation conditions (e.g., dynamic changes in the thermal state of objects), and environmental interference (e.g., smoke, darkness), resulting in scarce labeled data. Visible light images, on the other hand, have advantages such as low acquisition cost and wide scene coverage. Therefore, using visible light modal training to achieve cross-modal generalization of infrared detection becomes a key path to overcome the infrared data bottleneck. This task essentially belongs to domain generalization without pre-trained models, and its core challenge lies in the fact that visible light relies on color and texture features, while infrared images represent thermal radiation differences using grayscale values. Traditional visible light detection networks struggle to bridge the feature gap between modalities. To date, many papers have conducted research on domain-generalized object detection without pre-trained models: Reference 1: Liu, Hong, Pinhao Song, and RunweiDing. "WQT and DG-YOLO: Towards domain generalization in underwater object detection." arXiv preprint arXiv:2004.06333 (2020). This paper proposes a water quality migration data augmentation method, WQT, to increase the domain diversity of the dataset. It integrates a domain-invariant module (DIM) into YOLOv3 to adversarially learn and extract water quality-related features, and uses an invariant risk minimization penalty (IRM Penalty) to mine domain-invariant semantic information from underwater images, thus achieving domain generalization for underwater object detection. However, its design is not intended for underwater water quality change scenarios and lacks targeted optimization for the "nonlinear distribution of thermal radiation grayscale" and "contour-dominant" characteristics of infrared modalities. It does not explicitly enhance the extraction of edge abrupt changes caused by temperature differences in thermal infrared images, resulting in limitations in adaptability when processing shallow contour cues of infrared targets.Reference 2: Chen, Yang, et al. "Achieving domain generalization for underwater object detection by domain mixup and contrastive learning." Neurocomputing528 (2023): 20-34. This paper proposes a method combining conditional bilateral style transfer and feature-level domain mixing. Conditional bilateral style transfer transforms underwater images from one source domain to another to generate stylized samples. Feature-level domain mixing interpolates features from different domains in the feature space to synthesize new domain samples, thus enriching the domain diversity of the training data. This scheme achieves inter-domain style transfer through the linear transformation matrix of conditional bilateral style transfer, initially bridging the visual differences between different underwater environments. Feature-level domain mixing expands the coverage of domain distribution through feature interpolation. However, the stylized images generated by CBST still retain texture details from the original domain and do not completely strip away domain-specific features. Furthermore, feature-based linear interpolation, assuming a linear manifold structure, struggles to cover the nonlinear distribution of domain differences in underwater environments. Reference 3: Li, Deng, et al. "Prompt-driven dynamic object-centric learning for single domain generalization." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024. This paper introduces an object-centric gating module based on cue-based learning. Scene cues (such as environmental description text) guide the model to focus on core object features, suppressing background interference. This module leverages the cross-modal representation capabilities of the CLIP pre-trained model to enhance the semantic consistency of feature representations, significantly optimizing object localization accuracy in single-domain generalized object detection tasks. However, the module design relies on a predefined scene cue semantic space. When the semantic distribution of the target scene exceeds the coverage of the training cue, the model's ability to semantically associate object boundaries decreases. Furthermore, the visual-linguistic features fused through Slot-Attention, with semantic texture as the core representation, suffer from insufficient contour feature extraction for targets without obvious texture features, resulting in limited robustness under complex lighting conditions.Reference 4: Liu, Yue, et al. "Vmamba: Visual state space model." Advances in neural information processing systems 37 (2024): 103031-103063. This paper proposes a hierarchical visual backbone network that integrates Mamba modules, capturing long-distance spatial dependencies through a multi-directional scanning state space model. This architecture demonstrates advantages in global feature extraction for tasks such as image classification. However, as a general visual backbone, it lacks specific optimization for modal differences in cross-modal detection: on the one hand, it does not design a modal alignment mechanism to handle the feature gap between visible light and infrared images, leading to semantic incompatibility during cross-modal detection; on the other hand, it lacks an adaptive extraction module for thermal radiation contours in infrared images, making it difficult to adapt to the characteristics of infrared targets with "low texture and high grayscale abrupt changes."

[0003] The shortcomings of the aforementioned technologies are as follows: existing cross-modal target detection schemes face three major technical bottlenecks: insufficient infrared target contour feature extraction capability, over-reliance on specific domain distribution assumptions, and lack of universal cross-modal generalization mechanism. To address these issues, this study proposes a simplified module to guide the network to focus on the geometric contour features of the target. At the same time, it constructs a clue mining module based on the SS2D scanning mechanism. By strengthening the spatial continuity representation of the infrared target contour, it achieves efficient extraction and transfer of common features from visible light to infrared modes. Furthermore, it constructs a generalization framework that does not rely on scene priors, fundamentally breaking through the inherent limitations of traditional networks in cross-domain detection application scenarios. Summary of the Invention

[0004] The technical problem solved by this invention is to overcome the problems of weak infrared contour feature extraction capability and lack of cross-domain generalization mechanism in existing cross-modal target detection methods. It provides a visible light and infrared cross-modal target detection method based on selective state space, which can achieve high-precision detection of infrared targets in complex scenes when trained only by visible light data. It breaks the deterministic correlation of texture information in visible light images, and deconstructs the texture features that originally depended on fixed color distribution due to the randomness of channel weights, which significantly improves the generalization performance of the model from visible light to infrared modality.

[0005] The technical solution of this invention is: Firstly, a method for cross-modal target detection in visible light and infrared based on selective state space is provided, including: A cross-modal detection model is constructed, which consists of the following components connected in sequence: a simplified backbone module, an improved YOLOv8 backbone network, a multi-scale feature fusion processing unit, and a detection head. A visible light image is input into the cross-modal detection model, and after processing by the simple drying module, a single-channel grayscale image is output. The improved YOLOv8 backbone network receives single-channel grayscale images, performs state space modeling and feature extraction, and outputs the features to the multi-scale feature fusion processing unit. The multi-scale feature fusion processing unit performs fusion processing on the features and outputs the result to the detection head, which then outputs the target detection result.

[0006] Preferably, when the simplified module processes visible light images: Randomly generated coefficients The coefficients satisfy: , ; The generation coefficients are used to perform weighted convolution on the three channels of visible light to generate a single-channel grayscale image.

[0007] Where R, G, and B are the weights of the three visible light channels, respectively.

[0008] Preferred, improved YOLOv8 backbone network comprises, in sequence: two GSSBlock and Conv combinations, and two C2f and Conv combinations.

[0009] Preferably, each GSSBlock comprises, in sequence: a 3×3 depthwise separable convolution, SS2D, layer normalization, and a gating block.

[0010] Preferably, SS2D includes: linear layers, depthwise separable convolutions, activation functions, scanning methods, and layer normalization; The SS2D scanning method involves converting a two-dimensional feature map into a one-dimensional sequence using four preset scanning methods, extracting features through a state-space model, and then restoring it to a two-dimensional feature map. The four preset scanning methods include: scanning from left to right starting from the first row, scanning from top to bottom starting from the first column, scanning from right to left starting from the last row, and scanning from bottom to top starting from the last column.

[0011] Preferably, the gate block includes convolution, depthwise separable convolution, and activation function; used for feature selective enhancement, focusing on the contour and edge features of infrared targets by dynamically adjusting weights.

[0012] Preferably, the processing flow of the multi-scale feature fusion processing unit includes: PAFPN neck processing: A top-down approach is adopted, using 2× upsampling to pass high-level features; a bottom-up approach is adopted, using 3×3 convolution downsampling to aggregate low-level features; Feature fusion: Integrating features at different scales through splicing operations.

[0013] Preferably, the detection head includes a regression head and a classification head. The regression head outputs the coordinates of the oriented bounding box, and the classification head outputs the category of the detected target.

[0014] Secondly, a visible light and infrared cross-modal target detection system based on selective state space is provided, characterized by comprising: a simplified module, an improved YOLOv8 backbone network, a multi-scale feature fusion processing unit, and a detection head; wherein: The simplified backbone module performs weighted convolution on the visible light from the input system to generate a single-channel grayscale image, which is then output to the improved YOLOv8 backbone network. An improved YOLOv8 backbone network performs state space modeling and feature extraction, and outputs the features to a multi-scale feature fusion processing unit; it includes two GSSBlock and Conv combination connected in sequence, and two C2f and Conv combination connected in sequence; each GSSBlock includes a 3×3 depthwise separable convolution, SS2D, layer normalization and gate block connected in sequence. The multi-scale feature fusion processing unit performs feature fusion processing and outputs the result to the detection head. The processing flow includes: PAFPN neck processing and feature fusion. Among them, PAFPN neck processing adopts a top-down path, which transmits high-level features through 2× upsampling, and adopts a bottom-up path, which aggregates low-level features through 3×3 convolution downsampling. During feature fusion, features of different scales are integrated through a concatenation operation. The detection head, including a regression head and a classification head, is used to output the target detection results. The regression head outputs the coordinates of the oriented bounding box, and the classification head outputs the detected target category.

[0015] Preferably, the SS2D in GSSBlock includes: linear layers, depthwise separable convolutions, activation functions, scanning methods, and layer normalization; The SS2D scanning method involves converting a two-dimensional feature map into a one-dimensional sequence using four preset scanning methods, extracting features through a state-space model, and then restoring it to a two-dimensional feature map. The four preset scanning methods include: scanning from left to right starting from the first row, scanning from top to bottom starting from the first column, scanning from right to left starting from the last row, and scanning from bottom to top starting from the last column.

[0016] Compared with the prior art, the present invention has the following advantages: (1) This invention proposes a grayscale image generation method based on random linear combination weighting of RGB three channels. By applying random coefficients to the RGB channels through a simplified module and performing linear combination, the visible light image is converted into a single-channel grayscale image. This process breaks the deterministic correlation of texture information, so that the texture features that originally depended on a fixed color distribution are deconstructed due to the randomness of the channel weights, thereby guiding the model to focus on the target contour features and better match the grayscale characteristics of infrared images.

[0017] (2) Regarding the optimization of the detection framework, this invention integrates the selective state space mechanism into YOLOv8, which significantly improves the ability to capture shallow features such as target contours and effectively bridges the performance gap between visible light and infrared cross-modal detection. At the same time, by utilizing the selective feature filtering capability of the state space model, general visual cues such as contours are strengthened during the cross-modal feature mapping process, ultimately constructing a lightweight and efficient detection framework that is adapted to infrared modalities. Attached Figure Description

[0018] Figure 1 This is an overall flowchart of a visible-infrared cross-modal target detection method based on selective state space according to an embodiment of the present invention; Figure 2 This is a simplified modular structure diagram of a visible-infrared cross-modal target detection method based on selective state space according to an embodiment of the present invention; Figure 3 This is a weighted convolution structure diagram of a visible-infrared cross-modal target detection method based on selective state space according to an embodiment of the present invention; Figure 4 This is a flowchart of an improved detection backbone network for a visible-infrared cross-modal target detection method based on selective state space, according to an embodiment of the present invention. Figure 5 This is a structural diagram of the GSSBlock cue mining module of a visible-infrared cross-modal target detection method based on selective state space according to an embodiment of the present invention. Figure 6 This is an SS2D structural diagram of a visible-infrared cross-modal target detection method based on selective state space according to an embodiment of the present invention; Figure 7 This is a diagram illustrating the SS2D scanning method of a visible-infrared cross-modal target detection method based on selective state space according to an embodiment of the present invention. Figure 8 This is a structural diagram of a gating module for a visible-infrared cross-modal target detection method based on selective state space, according to an embodiment of the present invention. Detailed Implementation

[0019] To address the shortcomings of existing cross-modal target detection methods, such as weak infrared contour feature extraction capabilities and lack of cross-domain generalization mechanisms, this invention provides a visible-infrared cross-modal target detection method based on selective state space. This method achieves high-precision detection of infrared targets in complex scenes when trained solely on visible light data. It breaks the deterministic correlation of texture information in visible light images, deconstructing texture features that originally depended on fixed color distributions due to the randomness of channel weights, and significantly improving the model's generalization performance from visible light to infrared modalities.

[0020] The specific implementation method includes the following steps: The input visible light image channels are linearly combined using a simplified module to generate a single-channel grayscale image with random weights. The grayscale image is then input into the feature extraction part and state space modeling is performed to focus on the target edges and geometric structures. After multi-scale feature fusion, the detection result is finally output through the detection head.

[0021] The specific processing flow of the simplified module is as follows: linear combination weighted convolution is performed on the RGB channels of the visible light image to generate a single-channel grayscale image. The weighted convolution satisfies Output=α×R+β×G+γ×B, and α+β+γ=1, where α, β, and γ are random coefficients with a value range of [0,1]. The random coefficients are dynamically updated each time the processing is performed. The state space modeling is achieved through a cue mining module, which is integrated into the backbone network and contains two consecutive GSSBlocks and a standard convolutional combination. The backbone network is also known as the core network.

[0022] The specific processing flow of the clue mining module is as follows: The feature map is input into each GSSBlock and 3×3 depthwise separable convolution, SS2D operation, layer normalization and gate block processing are performed in sequence.

[0023] SS2D includes linear layers, depthwise separable convolutions, activation functions, scanning methods, and layer normalization.

[0024] The SS2D scanning method involves converting a two-dimensional feature map into a one-dimensional sequence using four preset scanning methods, extracting features through a state-space model, and then restoring it to a two-dimensional feature map. The four scanning methods are: scanning from left to right starting from the first row, scanning from top to bottom starting from the first column, scanning from right to left starting from the last row, and scanning from bottom to top starting from the last column.

[0025] The gating block consists of convolutions, depthwise separable convolutions, and activation functions. The gating block's function is to enhance feature selectivity by dynamically adjusting weights to focus on key shallow features such as the contours and edges of infrared targets.

[0026] The visible light image is input into the simple module and linearly combined to be converted into a single-channel grayscale image. The grayscale image is then input into the feature extraction part and the state space is modeled through the cue mining module. After multi-scale feature fusion, the detection result is finally output through the detection head.

[0027] The RGB channels of a visible light image are subjected to weighted convolution to generate a single-channel grayscale image. The weighted convolution satisfies Output=α×R+β×G+γ×B, and α+β+γ=1, where α, β, and γ are random coefficients with values ​​ranging from [0,1]. The random coefficients are dynamically updated each time the image is processed. State space modeling is achieved through a cue mining module, which is integrated into the backbone network and consists of two consecutive GSSBlocks and a standard convolution.

[0028] The integrated clue mining module specifically operates as follows: The processing flow of the integrated clue mining module is as follows: The feature map is input into each GSSBlock, and 3×3 depthwise separable convolution, SS2D operation, layer normalization, and gating block processing are executed sequentially. The SS2D scanning method converts the two-dimensional feature map into a one-dimensional sequence according to four preset scanning methods. After extracting features through a state-space model, the two-dimensional feature map is restored. The gating block consists of convolution, depthwise separable convolution, and activation functions. The function of the gating block is to enhance feature selectivity by dynamically adjusting weights to focus on key shallow features such as the contours and edges of infrared targets.

[0029] The features output from the backbone network are input into multi-scale feature fusion. The specific processing flow of multi-scale feature fusion is as follows: PAFPN neck processing: top-down path: high-level features are passed through 2× upsampling; bottom-up path: low-level features are aggregated through 3×3 convolution downsampling; feature fusion: features of different scales are integrated through concatenation operation.

[0030] The specific processing flow of the detection head is as follows: the regression head outputs the coordinates of the oriented bounding box, and the classification head outputs the category of the detected target.

[0031] An RGB three-channel visible light image with input dimensions [H,W,3] is processed by a simplified module to output a single-channel grayscale image with dimensions [H,W,1].

[0032] The simplified module specifically works as follows: In each processing step, coefficients α, β, γ ∈ [0, 1] that satisfy α + β + γ = 1 are randomly generated. Then, the generated coefficients are used to perform a weighted convolution of the visible light three channels: a linear combination is performed on the RGB channels: Output = α×R + β×G + γ×B, generating a single-channel grayscale image output.

[0033] The input is a [H,W,1] grayscale image output from the simplified backbone module. After passing through the improved YOLOv8 backbone network, multi-scale features are extracted, and the feature map with enhanced contours and geometric structures is output.

[0034] The improved YOLOv8 backbone network specifically includes a cue mining module that mines clues about infrared targets in the input feature map. This module consists of two consecutive GSSBlocks and a standard convolutional combination.

[0035] Specifically, the GSSBlock performs 3×3 depthwise separable convolution, SS2D operation, layer normalization, and gating block processing sequentially.

[0036] Specifically, SS2D includes linear layers, depthwise separable convolutions, activation functions, scanning methods, and layer normalization. The SS2D scanning method involves converting the two-dimensional feature map into a one-dimensional sequence using four preset scanning methods, extracting features through a state-space model, and then restoring the two-dimensional feature map.

[0037] The gating block processing specifically involves a gating block composed of convolutions, depthwise separable convolutions, and activation functions. The gating block enhances feature selectivity by dynamically adjusting weights to focus on key shallow features such as the contours and edges of infrared targets.

[0038] The enhanced contour and geometric structure feature map output from the backbone network is input, and multi-scale fused features are output after multi-scale feature fusion.

[0039] Input multi-scale fusion features, and after regression and classification by the detection head, output the target category prediction result and the target orientation detection box coordinates.

[0040] This invention is based on a grayscale image generation method using a random linear combination weighted RGB three-channel algorithm. By applying random coefficients to the RGB channels through a simplified module, a visible light image is converted into a single-channel grayscale image. This process breaks the deterministic correlation of texture information, deconstructing texture features that originally depended on fixed color distributions due to the randomness of channel weights. This guides the model to focus on target contour features, better matching the grayscale characteristics of infrared images. Regarding detection framework optimization, this invention integrates a selective state-space mechanism into YOLOv8, significantly improving the ability to capture shallow features such as target contours and effectively bridging the performance gap between visible light and infrared cross-modal detection. Simultaneously, by utilizing the selective feature filtering capability of the state-space model, general visual cues such as contours are strengthened during cross-modal feature mapping, ultimately constructing a lightweight and efficient detection framework adapted to infrared modalities.

[0041] Example: This embodiment is implemented based on the technical solution of the present invention, and provides detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiment.

[0042] The overall flowchart of a visible-infrared cross-modal target detection method based on selective state space is as follows: Figure 1 As shown, the input image first passes through a simplified backbone module, then is input into an improved backbone network, followed by a multi-scale feature fusion module, and finally input into a detection head to output the category.

[0043] Simplified module structure diagram as follows Figure 2 As shown, it consists of a weighted convolution and a regular convolution.

[0044] Weighted convolution structure diagram as follows Figure 3 As shown, the input is three visible light channels, and coefficients α, β, γ ∈ [0, 1] are randomly generated to satisfy α + β + γ = 1. Then, the generated coefficients are used to perform a weighted convolution of the three visible light channels: a linear combination is performed on the RGB channels: Output = α×R + β×G + γ×B, generating a single-channel grayscale image output.

[0045] The improved backbone detection network flowchart is as follows: Figure 4 As shown, the output of the simplified module passes through GSSBlock, then through Conv, then through GSSBlock and Conv, then through C2f and Conv, and finally through C2f and Conv to output the feature.

[0046] GSSBlock structure diagram as follows Figure 5 As shown, the process includes 2D convolution and batch normalization. One branch is passed through depthwise separable convolution, then through SS2D and added to the residual. Finally, the branch is passed through layer normalization and a gating module and added to the residual before being output.

[0047] SS2D structure diagram as follows Figure 6 As shown, the process includes passing through linear layers, depthwise separable convolutions, scanning methods, and layer normalization followed by another linear layer.

[0048] SS2D scanning method Figure 7 As shown, the input image feature map is read and calculated in the encoding stage using four scanning methods, and then the features are recovered in the decoding stage.

[0049] The structure diagram of the gate control module is as follows: Figure 8 As shown, after the input batch normalization layer, it goes through three branches. Branch one uses convolution, branch two uses convolution and depthwise separable convolution. Branch one multiplies with branch two, and after convolution, it is added to the residual of branch three.

[0050] Existing cross-modal target detection schemes face three core technical bottlenecks: insufficient infrared target contour feature extraction capability, over-reliance on domain-specific distribution assumptions, and a lack of universal cross-modal generalization mechanisms. To address these issues, this study proposes a simplified module to guide the network to focus on target geometric contour features. Simultaneously, a cue mining module based on the SS2D scanning mechanism is constructed. By strengthening the spatial continuity representation of infrared target contours, efficient common feature extraction and transfer from visible light to infrared modalities are achieved. Furthermore, a generalization framework independent of scene priors is built, fundamentally overcoming the inherent limitations of traditional networks in cross-domain detection applications. This invention utilizes a grayscale image generation method based on a random linear combination weighted RGB three-channel combination. The simplified module applies random coefficients to the RGB channels for linear combination, converting visible light images into single-channel grayscale images. This process breaks the deterministic correlation of texture information, deconstructing texture features that originally depended on fixed color distributions due to the randomness of channel weights. This guides the model to focus on target contour features, better matching the grayscale characteristics of infrared images. Regarding the optimization of the detection framework, this invention integrates a selective state-space mechanism into YOLOv8, significantly improving the ability to capture shallow features such as target contours and effectively bridging the performance gap between visible light and infrared cross-modal detection. Simultaneously, by utilizing the selective feature filtering capability of the state-space model, general visual cues such as contours are enhanced during cross-modal feature mapping, ultimately constructing a lightweight and efficient detection framework adapted to infrared modalities.

[0051] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

[0052] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A selective state space based visible and infrared cross-modal object detection method, characterized in that The application relates to a cross-modal detection model, and the cross-modal detection model comprises, in sequence, a simple stem module, an improved YOLOv8 backbone network, a multi-scale feature fusion processing unit and a detection head. The cross-modal detection model is input with a visible light image, and a single-channel gray image is output after the simple stem module processes the visible light image. The improved YOLOv8 backbone network receives the single-channel gray image, performs state space modeling and feature extraction, and outputs the features to the multi-scale feature fusion processing unit. The multi-scale feature fusion processing unit outputs the features after fusion processing to the detection head, and the detection head outputs a target detection result. When the simple stem module processes the visible light image:

2. The method of claim 1, wherein: The three channels of the visible light are subjected to weighted convolution by using generation coefficients to generate a single-channel gray image output Output: Randomly generated coefficients , the coefficients satisfy: , ; Wherein R, G and B are weights of the three channels of the visible light. The improved YOLOv8 backbone network comprises, in sequence, two GSSBlock and Conv combinations and two C2f and Conv combinations.

3. The method of claim 1, wherein: Each GSSBlock comprises, in sequence, a 3*3 depth separable convolution, an SS2D, layer normalization and a gating block.

4. The method of claim 3, wherein: The SS2D comprises a linear layer, a depth separable convolution, an activation function, a scanning mode and layer normalization.

5. The selective state space based visible and infrared cross-modality target detection method according to claim 4, characterized in that: The scanning mode of the SS2D is to convert a two-dimensional feature map into a one-dimensional sequence according to a preset four scanning modes, restore the two-dimensional feature map after feature extraction by a state space model, and output the two-dimensional feature map. The preset four scanning modes comprise scanning from the first row to the right, scanning from the first column from top to bottom, scanning from the last row from right to left and scanning from the last column from bottom to top. The gating block comprises a convolution, a depth separable convolution and an activation function, is used for selective enhancement of features, dynamically adjusts weights, and focuses on contour and edge features of an infrared target.

6. The selective state space based visible and infrared cross-modality target detection method according to claim 4, wherein: The processing procedure of the multi-scale feature fusion processing unit comprises:

7. The method of claim 1, wherein: PAFPN neck processing: a top-down path is adopted, high-level features are transmitted through 2* upsampling, a bottom-up path is adopted, and low-level features are aggregated through 3*3 convolution downsampling; Feature fusion: different scale features are integrated through splicing operation. The detection head comprises a regression head and a classification head, the regression head outputs coordinates of a directional bounding box, and the classification head outputs a detection target category.

8. The selective state space based visible and infrared cross-modality target detection method according to claim 1, wherein: The application relates to a cross-modal detection model, and the cross-modal detection model comprises, in sequence, a simple stem module, an improved YOLOv8 backbone network, a multi-scale feature fusion processing unit and a detection head.

9. A selective state space based visible and infrared cross-modal target detection system, characterized in that The simple stem module performs weighted convolution on the visible light input into the system, and outputs a single-channel gray image to the improved YOLOv8 backbone network; The improved YOLOv8 backbone network performs state space modeling and feature extraction, and outputs the features to the multi-scale feature fusion processing unit; the improved YOLOv8 backbone network comprises, in sequence, two GSSBlock and Conv combinations and two C2f and Conv combinations; each GSSBlock comprises, in sequence, a 3*3 depth separable convolution, an SS2D, layer normalization and a gating block. ​ ​ The multi-scale feature fusion processing unit outputs the features to the detection head after performing fusion processing on the features; the processing procedure comprises: PAFPN neck processing and feature fusion; wherein: the PAFPN neck processing adopts a top-down path, and high-level features are transmitted through 2× upsampling; a bottom-up path is adopted: low-level features are aggregated through 3×3 convolution downsampling; when the features are fused, different scale features are integrated through a splicing operation; The detection head comprises a regression head and a classification head, and is used for outputting a target detection result, wherein: the regression head outputs coordinates of a directional bounding box, and the classification head outputs a detection target category.

10. The selective state space based visible and infrared cross-modality target detection system of claim 9, wherein: The SS2D in the GSSBlock comprises: a linear layer, a depth separable convolution, an activation function, a scanning mode, and layer normalization; The scanning mode of the SS2D is to convert a two-dimensional feature map into a one-dimensional sequence according to four preset scanning modes, restore the two-dimensional feature map after extracting features through a state space model, and output the two-dimensional feature map. The four preset scanning modes comprise: scanning from left to right starting from the first row according to a row, scanning from top to bottom starting from the first column according to a column, scanning from right to left starting from the last row according to a row, and scanning from bottom to top starting from the last column according to a column.

Images