A method and device for detecting a marine target based on dual-mode perception

Abstract

This invention relates to a method and apparatus for detecting maritime targets based on dual-modal perception, comprising the following steps: acquiring visible light and infrared imaging images of the same scene and inputting them together into a YOLOv8-based dual-stream backbone network; after five stages of processing, obtaining RGB local feature maps of the first to fifth layers of the visible light imaging image and IR feature maps of the first to fifth layers of the infrared imaging image; inputting the RGB local feature maps of the second to fourth layers and the IR feature maps into a BCFM module composed of RFB components to obtain three corresponding dual-modal fused features; then inputting the three dual-modal fused features and the second-layer IR feature map together into the network neck of the dual-stream backbone network to output the fused features; and finally inputting the fused features into the network head of the dual-stream backbone network to output the detection result of the maritime target detection. This invention is particularly capable of accurately detecting small maritime targets.

Description

Technical Field

[0001] This invention relates to the field of machine vision and maritime safety related technologies, and in particular to a method and device for detecting maritime targets based on dual-modal perception. Background Technology

[0002] In recent years, CNNs and Transformers (ViT) have been the mainstream architectures in computer vision, each with its own significant advantages and limitations. CNNs are limited by their local receptive fields, which greatly restricts their ability to model long-range dependencies. In contrast, ViT uses a self-attention mechanism to obtain a global receptive field, but faces the burden of quadratic computational complexity. Recently, S4 has shown stronger ability to capture long-term dependencies in sequential data while maintaining linear time complexity. In order to adaptively select information based on the current input content, S6 has achieved a major breakthrough in reducing inference latency and improving overall performance by adopting selective state space and hardware-aware algorithms, leading to the proposal of the Mamba Block. As a core component of Mamba, S6 is essentially designed for processing one-dimensional sequential data. However, visual data is not a one-dimensional sequence; it is non-sequential and contains rich spatial information, which makes S6 face significant challenges in adapting to visual data. To address this, Zhu et al. proposed the Vision Mamba (Vim), which combines bidirectional S6 for global visual context modeling of data dependencies with location embedding for location-aware visual understanding. Liu et al. proposed the 2D Selective Scan (SS2D) module. The input image patch is first unfolded into a sequence along four different paths. Each sequence is processed in parallel by independent S6 blocks, and then reconstructed into a two-dimensional feature map as the final output through a fusion operation, effectively bridging the gap between S6 and visual data. However, existing methods often rely on pre-defined prior knowledge when processing the mapping from two-dimensional images to one-dimensional sequences. The non-causal nature of two-dimensional spatial information in images differs fundamentally from the sequential dependence of token sequences in language modeling, posing a challenge to traditional sequential scanning strategies. Cao et al. proposed a new paradigm for processing random shuffling scan sequences. Compared to the local inductive bias of convolutional operations, while self-attention mechanisms have achieved a global receptive field, this method, through an unbiased global information sampling strategy, allows the network to completely escape the constraints of fixed local priors.

[0003] Traditional convolutional neural networks (CNNs) are typically trained and detected using visible light images of targets as input. However, this approach has not yet yielded ideal results in small-target detection missions at sea. Infrared images, generated by capturing the thermal radiation information emitted by objects, are more stable in harsh environments (such as fog, rain, and snow) compared to visible light images. Therefore, there is a clear complementarity between infrared and visible light images, making multispectral image detection an effective solution. Unlike traditional detection networks that only support a single modality input, multispectral detection requires processing data from both infrared and visible light modalities simultaneously. This makes a two-stream convolutional neural network (CNN) an ideal choice for its backbone network.

[0004] However, existing multispectral detection methods typically only perform simple pixel-level addition operations on visible light and infrared images, or perform shallow fusion of features from the two modalities, or target-level fusion with insufficient information coordination.

[0005] Therefore, a method for deep fusion of features from the two modalities is needed to improve the accuracy of maritime target detection. Summary of the Invention

[0006] The purpose of this invention is to at least address one of the shortcomings of the prior art and to provide a method and apparatus for detecting maritime targets based on dual-modal sensing.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: Specifically, a method for detecting maritime targets based on dual-modal perception is proposed, including the following: Visible light imaging images and infrared imaging images of the same scene are acquired and input into a dual-stream backbone network based on YOLOv8. After five stages of processing, the RGB local feature maps of the first to fifth layers of the corresponding visible light imaging image and the IR feature maps of the first to fifth layers of the corresponding infrared imaging image are obtained respectively. The RGB local feature maps and IR feature maps of the second to fourth layers are input into the BCFM module composed of RFB components to obtain three bimodal fusion features corresponding to the second, third and fourth layers; Then, the three dual-modal features and the IR features of the second layer are fused together. Figure 1 The same input is fed into the network neck of the dual-stream backbone network, and the fused features are output. The fused features are fed into the network header of the dual-stream backbone network, and the detection results of the maritime target detection are output.

[0008] Furthermore, specifically, the data processing procedure of the BCFM module, which consists of RFB components, includes: and It will be divided into fixed-size, non-overlapping patches, among which and These are the RGB local feature map and IR feature map of the i-th layer, respectively. Then, each patch is mapped to a one-dimensional vector token and embedded with positional encoding, thus obtaining a single-modal token sequence. First, we need to perform layer normalization on both token sequences, using the following formula: , Where E[] represents the mean and Var[] represents the variance. It is a constant. It is a scaling parameter. The shift parameter, after performing layer-by-layer normalization operations, is used to process the token sequence using a random shuffling strategy. Using the same random seed, the originally ordered tokens are shuffled to obtain the shuffled sequence. Subsequently, on A linear transformation and dimensionality increase are performed. Then, depthwise separable convolution is used to fuse the local features of the single-modal sequence. After the local information fusion is completed, the SiLU activation function is used as a gating regulator to obtain the processing result. The processing result is input into the Selective State Space Model (SSM) module to associate tokens at different positions in the sequence, thereby obtaining the associated features. Then, sequentially compare it with the gated selection after SiLU activation function. , Element-wise multiplication yields , , , An anti-random shuffling strategy is employed to restore the sequence after the above operations to the original token sequence, and the dimension is reduced to the same level as the original. Same, obtained , , , Then, by adding these features together, we obtain... , Then, the next RFB layer is input, and after repeated operations, the output is restored to the original feature dimensions. The original feature map is then connected through residuals to obtain the fused feature map. and Then, each obtained fused feature map is summed element by element. ,but That is, targeting and The dual-modal fusion feature.

[0009] This invention also proposes a maritime target detection device based on dual-modal sensing, comprising the following: The data acquisition module is used to acquire visible light imaging images and infrared imaging images of the same scene and input them into a dual-stream backbone network based on YOLOv8. After five stages of processing, the RGB local feature maps of the first to fifth layers of the corresponding visible light imaging image and the IR feature maps of the first to fifth layers of the corresponding infrared imaging image are obtained respectively. The dual-modal fusion module is used to input the RGB local feature maps and IR feature maps of the second to fourth layers into the BCFM module composed of RFB components to obtain three dual-modal fusion features corresponding to the second, third and fourth layers; The second fusion module is used to subsequently fuse the three dual-modal features and the IR features of the second layer. Figure 1 The same input is fed into the network neck of the dual-stream backbone network, and the fused features are output. The detection result output module is used to input the fused features into the network head of the dual-stream backbone network and output the detection results of the maritime target detection.

[0010] The beneficial effects of this invention are as follows: This invention proposes a method and device for detecting maritime targets based on dual-modal sensing. By applying the proposed BCFM module to a YOLOv8-based dual-stream backbone network, visible light and infrared images are input into the dual-stream backbone network, generating multi-scale hierarchical feature maps. The visible light feature maps are as follows: The infrared feature maps are as follows: Subsequently, At the feature level, the BCFM module is used to perform deep fusion of the dual-modal features, and the fused feature map is residually connected to the original feature map of the corresponding level. The result of the residual connection is downsampled to obtain the corresponding... Dual-modal feature fusion across three feature levels. Considering the clear outlines of infrared images, and the sparse feature representation, weak target signals, and small pixel sizes of small targets at sea, larger-sized targets in the infrared image stream are fused together. Feature map, and the features after deep fusion as described above Figure 1 The neck module of the input network is used to carry out further feature aggregation and target detection. Finally, the detection head completes the target detection task, which can obtain accurate target detection results, especially for small targets at sea. Attached Figure Description

[0011] The above and other features of this disclosure will become more apparent from the detailed description of the embodiments illustrated in conjunction with the accompanying drawings. In the accompanying drawings, the same reference numerals denote the same or similar elements. Obviously, the drawings described below are merely some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without any creative effort. In the drawings: Figure 1 The diagram shown is a structural block diagram of the state-space model involved in this invention; Figure 2 The diagram shown is a schematic of the S6-based Mamba module structure involved in this invention. Figure 3 The diagram shown is a schematic of the dual-modal feature cross-fusion module (BCFM) proposed in this invention. Figure 4 The diagram shows the structure of the RFB (Random Sequence Fusion Module) based on a random shuffling strategy proposed in this invention. Figure 5 The diagram shows a schematic of the dual-modal image detection network structure based on a dual-stream backbone and an RFB module proposed in this invention. Figure 6 The diagram shows a flowchart of a maritime target detection method based on dual-modal sensing proposed in this invention. Detailed Implementation

[0012] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The same reference numerals used throughout the accompanying drawings indicate the same or similar parts.

[0013] The feasibility analysis of the proposed solution is as follows: State space models (SSMs) are mathematical models used to describe the evolution of the state of a dynamic system over time, consisting of two parts: state equations and observation equations [1]. As shown in Figure 1, the state equations describe the dynamic process of the evolution of the internal state of the system over time, while the observation equations establish the mapping relationship between the internal state of the system and the external observable output. In deep learning SSMs, skip connections are usually not used, and the equations are simplified to: (1) (2) in, Is The state variable at time t, yes Input variables at time, yes Output variables at time 10:00 It is the state transition matrix. It is a control input matrix. It is the control output matrix.

[0014] In the continuous-time domain, the dynamic behavior of a system is typically described by differential equations. When the system is subjected to discrete inputs in addition to continuous inputs, it needs to be discretized to transform the continuous-time system into a discrete-time system. This process usually relies on zero-order hold techniques to reconstruct and sample the continuous signal, thereby converting the continuous state-space model into a discrete form. Discretization transforms the original model from a mapping of functions to functions. Transform into a sequence-to-sequence mapping After discretization, the model can be computed using linear recursion and global convolution. The discretized equation is: (3) (4) In this process, continuous parameters are transformed into discrete parameters using a fixed formula. , .

[0015] In SSMs, matrix A is responsible for capturing information from previous states to construct new states. To enhance its ability to model long-range dependencies, discretized SSMs use the HiPPO matrix instead of the randomly initialized matrix A. HiPPO compresses the input into a set of compact coefficients by projecting the historical input signal onto a set of orthogonal basis functions, thereby constructing a hidden state that can effectively summarize historical information. This method significantly improves the model's ability to process long-range dependency sequences with limited storage capacity. To further reduce the computational burden brought by the HiPPO matrix, the model introduces normal plus low rank (NPLR) decomposition to reparameterize the HiPPO matrix A, as detailed in formula (5), and finally forms structured state space sequence models (S4) [2].

[0016] (5) In the above SSMs, the system matrices A, B, C, and Δ are all static parameters and will not be dynamically adjusted with the change of the input sequence. Although this fixed parameter mechanism allows SSMs to achieve efficient parallel training by converting to the standard convolution form when the time-invariant assumption is satisfied, it also limits the model's ability to make dynamic inferences based on different input contents, making it difficult to flexibly filter information or perceive context. To overcome this limitation, Gu et al. proposed Selective State Space Models (S6) [3]. S6 no longer compresses all historical information indiscriminately, but instead dynamically parameters the core parameters B, C, and time scale parameter Δ of SSMs based on input, so that the model can adaptively select information based on the current input content, thereby achieving context perception and enhancing the ability to filter information from the input sequence. However, the dynamic parameters that depend on the input will transform S6 from a linear time-invariant system into a linear time-varying system, thereby destroying the advantage of SSMs being able to be efficiently computed through convolution, resulting in a decrease in the efficiency of parallel computation during training. To address this issue, hardware-aware optimization strategies were further incorporated, employing techniques such as kernel fusion, parallel scanning, and recomputation to improve the computational efficiency of the S6 layer while maintaining the model's expressive power.

[0017] like Figure 2 As shown, the Mamba Block is a highly efficient sequence modeling unit built on the H3 architecture by integrating a selective state space layer (S6 layer) and a gating mechanism. Compared to the H3 module, the Mamba Block replaces the original multiplication gating with a simpler nonlinear activation function (such as SiLU or Swish), and adds an S6 layer to the main branch propagation path to enhance the ability to capture long-range dependencies.

[0018] S6, as a core component of Mamba, is essentially designed for processing one-dimensional sequence data. However, visual data is non-sequential and contains rich spatial information, which makes S6 face significant challenges in adapting to visual data. To this end, Zhu et al. proposed Vision Mamba (Vim), which combines bidirectional S6 for global visual context modeling of data dependencies and location embedding for visual understanding of location awareness [4]; Liu et al. proposed the 2D Selective Scan (SS2D) module [5]. The input image block is first expanded into a sequence along four different paths. Each sequence is processed in parallel by independent S6 blocks, and then reshaped into a two-dimensional feature map as the final output through a fusion operation, which effectively bridges the gap between S6 and visual data. However, existing methods often rely on artificially preset prior knowledge when processing the mapping of two-dimensional images to one-dimensional sequences. The non-causal characteristics of two-dimensional spatial information of images are fundamentally different from the sequential dependency of token sequences in language modeling, which poses a challenge to the traditional sequential scanning strategy. Cao et al. proposed a new paradigm for processing random shuffling scan sequences. Compared to the local inductive bias of convolution operations, the self-attention mechanism has achieved a global receptive field, but this method completely frees the network from the constraints of fixed local priors through an unbiased global information sampling strategy [6].

[0019] In the field of multimodal object detection, existing research mainly focuses on three major directions: pixel-level fusion, feature-level fusion, and object-level fusion. All three methods effectively improve the performance of single-modal detection models by leveraging the complementarity between multimodal information. Specifically, pixel-level fusion is usually performed in the data preprocessing stage, directly merging or superimposing the original data from different modalities to form a fused image before inputting it into the detection network, thereby preserving the detailed information of the original data to the maximum extent. Feature-level fusion is implemented in the feature extraction stage of the detection process. Data from different modalities are first processed independently to extract their respective feature representations, and then fused at the feature layer. Object-level fusion is a late-stage decision fusion; each single-modal model first generates detection results independently, and then these predictions are integrated to form a unified detection output.

[0020] Infrared images reveal the clear thermal structure of objects regardless of brightness, but lack textural details. In contrast, visible light images capture rich object texture and scene information, but lighting conditions significantly affect image quality. Multimodal fusion fully leverages the complementarity between different modalities, integrating information extracted from different modalities into a stable multimodal representation. Therefore, bimodal sensing technology, by integrating complementary information from visible light and infrared cameras, provides the detection network with richer sensing dimensions, thereby improving the system's reliability, sensing capabilities, and environmental adaptability.

[0021] [1].Rudolph Emil Kalman. “A New Approach to Linear Filtering andPrediction Problems”. In: (1960). [2].Albert Gu, Karan Goel, and Christopher Ré. “Efficiently ModelingLong Sequences with Structured State Spaces”. In: The InternationalConference on Learning Representations (ICLR). 2022. [3].Gu A, Dao T. Mamba: Linear-time sequence modeling with selectivestate spaces[J]. arxiv preprint arxiv:2312.00752, 2023. [4].Zhu L, Liao B, Zhang Q, et al. Vision mamba: Efficient visualrepresentation learning with bidirectional state space model[J]. arxivpreprint arxiv:2401.09417, 2024. [5].Liu Y, Tian Y, Zhao Y, et al. Vmamba: Visual state space model[J]. Advances in neural information processing systems, 2024, 37: 103031-1030,63. [[ID=][6].Cao K, He X, Hu T, et al. Shuffle mamba: State space models withrandom shuffle for multi-modal image fusion[J]. arxiv preprint arxiv:2409.01728, 2024. After combining with the analysis of related problems in the background art, Example 1, refer to Figure 5 and Figure 6 It should be noted that there seems to be an error in the original text where "1030,63" in item [8] should probably be "103063". The translation is based on the provided text as accurately as possible.This invention proposes a method for detecting maritime targets based on dual-modal perception. The method utilizes a dual-modal image detection network structure based on a dual-stream backbone and an RFB module for maritime target detection. The method includes the following: Visible light imaging images and infrared imaging images of the same scene are acquired and input into a dual-stream backbone network based on YOLOv8. After five stages of processing, the RGB local feature maps of the first to fifth layers of the corresponding visible light imaging image and the IR feature maps of the first to fifth layers of the corresponding infrared imaging image are obtained respectively. The RGB local feature maps and IR feature maps of the second to fourth layers are input into the BCFM module composed of RFB components to obtain three bimodal fusion features corresponding to the second, third and fourth layers; Then, the three dual-modal features and the IR features of the second layer are fused together. Figure 1 The same input is fed into the network neck of the dual-stream backbone network, and the fused features are output. The fused features are fed into the network header of the dual-stream backbone network, and the detection results of the maritime target detection are output.

[0022] In this embodiment 1, by applying the proposed BCFM module to a YOLOv8-based dual-stream backbone network, after the visible light image and infrared image are respectively input into the dual-stream backbone network, multi-scale hierarchical feature maps are generated. The visible light feature maps are as follows: The infrared feature maps are as follows: Subsequently, At the feature level, the BCFM module is used to perform deep fusion of the dual-modal features, and the fused feature map is residually connected to the original feature map of the corresponding level. The result of the residual connection is downsampled to obtain the corresponding... Dual-modal feature fusion across three feature levels. Considering the clear outlines of infrared images, and the sparse feature representation, weak target signals, and small pixel sizes of small targets at sea, larger-sized targets in the infrared image stream are fused together. Feature map, and the features after deep fusion as described above Figure 1 The neck module of the input network is used to carry out further feature aggregation and target detection. Finally, the detection head completes the target detection task, which can obtain accurate target detection results, especially for small targets at sea.

[0023] As a preferred embodiment of the present invention, specifically, the data processing process of the BCFM module composed of RFB components includes, and It will be divided into fixed-size, non-overlapping patches, among which and These are the RGB local feature map and IR feature map of the i-th layer, respectively. Then, each patch is mapped to a one-dimensional vector token and embedded with positional encoding, thus obtaining a single-modal token sequence. First, we need to perform layer normalization on both token sequences, using the following formula: , After completing the layer-by-layer normalization operation, a random shuffling strategy is used to process the token sequence. Using the same random seed, the originally ordered token order is shuffled to obtain the shuffled sequence. Subsequently, on A linear transformation and dimensionality increase are performed. Then, depthwise separable convolution is used to fuse the local features of the single-modal sequence. After the local information fusion is completed, the SiLU activation function is used as a gating regulator to obtain the processing result. The processing result is input into the Selective State Space Model (SSM) module to associate tokens at different positions in the sequence, thereby obtaining the associated features. Then, sequentially compare it with the gated selection after SiLU activation function. , Element-wise multiplication yields , , , An anti-random shuffling strategy is employed to restore the sequence after the above operations to the original token sequence, and the dimension is reduced to the same level as the original. Same, obtained , , , Then, by adding these features together, we obtain... , Then, the next RFB layer is input, and after repeated operations, the output is restored to the original feature dimensions. The original feature map is then connected through residuals to obtain the fused feature map. and Then, each obtained fused feature map is summed element by element. ,but That is, targeting and The dual-modal fusion feature.

[0024] In a preferred embodiment of the present invention, such as Figure 3As shown, this invention employs a Bimodal Cross-Fusion Module (BCFM) as the fusion component for bimodal image features. BCFM consists of two layers of Random Fusion Blocks (RFBs) used to model the correlation between cross-modal objects. The input feature map... and After the first layer of dual RFB processing, sequences with preliminary cross-modal feature association and complementarity are obtained. Furthermore, since the left branch of the RFB significantly influences the right branch, the output of each RFB is more biased towards the input of the left branch. Therefore, in the second layer, this invention uses sequence crossover for correcting and fusion, ultimately obtaining... , .

[0025] The RFB (Recursive Freeform Buffer) is a key component, divided into left and right branches. For each input unimodal token sequence, the features are first normalized using layer normalization to stabilize their distribution. Then, a random sorting operation shuffles the original token order, introducing sequence diversity. Next, a linear layer performs dimensionality upscaling on the shuffled features. Depthwise separable convolutions effectively capture local feature patterns while reducing computational overhead. Using the SiLU activation function as a gating regulator adaptively adjusts the information transmission process, preserving more important information while suppressing less important information. Selective SSM further processes the feature sequence to capture long-range dependencies. Then, the intermediate branch extending from the left branch, after passing through the gating regulator, performs element-wise multiplication with each unimodal sequence, learning the key information relevant to different modalities and suppressing information that needs to be ignored. Finally, an inverse random sorting operation restores the tokens to their original order, followed by dimensionality reduction through a linear layer, resulting in a fused output.

[0026] In practical applications, such as Figure 4 As shown, the visible light image (RGB) And infrared images (thermal) These inputs are fed separately into a two-stream feature extraction network. After five stages of processing, RGB local feature maps are obtained. and IR feature map .in and The inputs are fed separately into the BCFM module to associate cross-modal features in the selected state space, reducing the differences between cross-modal features and enhancing the consistency of the fused feature representation. Specifically, and It will be divided into fixed-size, non-overlapping patches. Each patch will then be mapped to a one-dimensional vector token and embedded with positional encoding, resulting in a unimodal token sequence. As shown in Figure 4, the first step is to perform layer normalization on both token sequences, using the following formula: , Where E[] represents the mean and Var[] represents the variance. It is a constant. It is a scaling parameter. This refers to the translation parameter. After layer normalization, a random shuffling strategy is used to process the token sequence. Using the same random seed, the originally ordered tokens are shuffled to obtain the shuffled sequence. Subsequently, regarding A linear transformation and dimensionality increase are performed to provide a richer information representation space. Next, depthwise separable convolution is used to fuse the local features of the single-modal sequence. After local information fusion, the SiLU activation function is used as a gating regulator. The SiLU function adaptively adjusts information transmission, preserving more important information while suppressing less important information. After this series of processing steps, the results are input into the Selective State Space Model (Selective SSM) module to associate tokens at different positions in the sequence, obtaining the associated features. Then, sequentially compare it with the gated selection after SiLU activation function. , Perform element-wise multiplication. , , , In this way, the feature sequences of visible light and infrared images can interact, enabling them to mutually acquire key components of interest in different modalities, achieving information transfer and focusing on key areas across modal features. An anti-random shuffling strategy is employed to restore the sequence after the above operations to the original token sequence, and the dimensionality is reduced to the same level as the original. Same, obtained , , , Then, by adding these features together, we obtain... , Then, input the next layer of RFB. After repeating the operation, the output is restored to the original feature dimensions, and the original feature map is connected through residuals to obtain the fused feature map. and Then, each obtained fused feature map is added element-wise (i.e., ... ), and then together with the second feature map of the infrared branch. The inputs are fed into the neck of the network, which has four inputs (an additional input compared to YOLOv8n). Finally, the fused features from the neck output are fed into the network head to obtain the detection results.

[0027] Example 2: The present invention also proposes a maritime target detection device based on dual-modal sensing, comprising the following: The data acquisition module is used to acquire visible light imaging images and infrared imaging images of the same scene and input them into a dual-stream backbone network based on YOLOv8. After five stages of processing, the RGB local feature maps of the first to fifth layers of the corresponding visible light imaging image and the IR feature maps of the first to fifth layers of the corresponding infrared imaging image are obtained respectively. The dual-modal fusion module is used to input the RGB local feature maps and IR feature maps of the second to fourth layers into the BCFM module composed of RFB components to obtain three dual-modal fusion features corresponding to the second, third and fourth layers; The second fusion module is used to subsequently fuse the three dual-modal features and the IR features of the second layer. Figure 1 The same input is fed into the network neck of the dual-stream backbone network, and the fused features are output. The detection result output module is used to input the fused features into the network head of the dual-stream backbone network and output the detection results of the maritime target detection.

[0028] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0029] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or system capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0030] Although the description of the invention has been quite detailed and particularly of several described embodiments, it is not intended to limit it to any of these details or embodiments or any particular embodiment, but should be considered as providing a broad possible interpretation of the claims by referring to the appended claims and taking into account the prior art, thereby effectively covering the intended scope of the invention. Furthermore, the invention has been described above with respect to embodiments foreseeable by the inventors in order to provide a useful description, and non-substantial modifications to the invention that have not yet been foreseen may still represent equivalent modifications.

[0031] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. Any embodiment that achieves the technical effects of the present invention using the same means should fall within the protection scope of the present invention. Within the protection scope of the present invention, various modifications and variations can be made to the technical solutions and / or implementation methods.

Claims

1. A method for detecting maritime targets based on dual-modal sensing, characterized in that, Including the following: Visible light imaging images and infrared imaging images of the same scene are acquired and input into a dual-stream backbone network based on YOLOv8. After five stages of processing, the RGB local feature maps of the first to fifth layers of the corresponding visible light imaging image and the IR feature maps of the first to fifth layers of the corresponding infrared imaging image are obtained respectively. The RGB local feature maps and IR feature maps of the second to fourth layers are input into the BCFM module composed of RFB components to obtain three bimodal fusion features corresponding to the second, third and fourth layers; Then, the three dual-modal fused features and the IR feature map of the second layer are input together into the network neck of the dual-stream backbone network, and the fused features are output. The fused features are fed into the network header of the dual-stream backbone network, and the detection results of the maritime target detection are output.

2. The maritime target detection method based on dual-modal sensing according to claim 1, characterized in that, Specifically, the data processing procedure of the BCFM module, which consists of RFB components, includes: and It will be divided into fixed-size, non-overlapping patches, among which and These are the RGB local feature map and IR feature map of the i-th layer, respectively. Then, each patch is mapped to a one-dimensional vector token and embedded with positional encoding, thus obtaining a single-modal token sequence. First, we need to perform layer normalization on both token sequences, using the following formula: , Where E[] represents the mean and Var[] represents the variance. It is a constant. It is a scaling parameter. The shift parameter, after performing layer-by-layer normalization operations, is used to process the token sequence using a random shuffling strategy. Using the same random seed, the originally ordered tokens are shuffled to obtain the shuffled sequence. Subsequently, on A linear transformation and dimensionality increase are performed. Then, depthwise separable convolution is used to fuse the local features of the single-modal sequence. After the local information fusion is completed, the SiLU activation function is used as a gating regulator to obtain the processing result. The processing result is input into the Selective State Space Model (SSM) module to associate tokens at different positions in the sequence, thereby obtaining the associated features. Then, sequentially compare it with the gated selection after SiLU activation function. , Element-wise multiplication yields , , , An anti-random shuffling strategy is employed to restore the sequence after the above operations to the original token sequence, and the dimension is reduced to the same level as the original. Same, obtained , , , Then, by adding these features together, we obtain... , Then, the next RFB layer is input, and after repeated operations, the output is restored to the original feature dimensions. The original feature map is then connected through residuals to obtain the fused feature map. and Then, each obtained fused feature map is summed element by element. ,but That is, targeting and The dual-modal fusion feature.

3. A maritime target detection device based on dual-modal sensing, characterized in that, Including the following: The data acquisition module is used to acquire visible light imaging images and infrared imaging images of the same scene and input them into a dual-stream backbone network based on YOLOv8. After five stages of processing, the RGB local feature maps of the first to fifth layers of the corresponding visible light imaging image and the IR feature maps of the first to fifth layers of the corresponding infrared imaging image are obtained respectively. The dual-modal fusion module is used to input the RGB local feature maps and IR feature maps of the second to fourth layers into the BCFM module composed of RFB components to obtain three dual-modal fusion features corresponding to the second, third and fourth layers; The second fusion module is used to input the three dual-modal fused features and the IR feature map of the second layer into the network neck of the dual-stream backbone network and output the fused features. The detection result output module is used to input the fused features into the network head of the dual-stream backbone network and output the detection results of the maritime target detection.

Images