A SAR and optical cooperative detection method and device for port and marine facilities

Abstract

The embodiment of the present disclosure provides a SAR and optical collaborative detection method and device for port and offshore facilities, relates to the technical field of remote sensing target detection, and comprises the following steps: acquiring a remote sensing image target detection data set, wherein samples in the data set include paired SAR remote sensing images and optical remote sensing images of port and offshore facilities and corresponding target labels; a multi-modal fusion target detection network is constructed, wherein the network includes a double-flow encoder, a multi-modal fusion module and a detection head; the double-flow encoder includes two branches, which are respectively used for extracting SAR features and optical features; the multi-modal fusion module sequentially includes a deformable alignment unit, a bidirectional cross attention unit, a gate fusion unit and a lightweight global context enhancement unit, and is used for fusing the features; the detection head is used for target detection on the fused features; the network is trained by using the data set to obtain a SAR and optical collaborative detection model, and thereafter, the model is used for target detection, so that the detection effect is improved.

Description

Technical Field

[0001] This disclosure relates to the field of remote sensing target detection technology, and in particular to a SAR and optical co-detection method and apparatus for ports and marine facilities. Background Technology

[0002] Remote sensing monitoring of ports and offshore facilities (such as wharves, berthed ships, drilling platforms, offshore wind power facilities, buoys, etc.) is a core technology for realizing intelligent waterway management, marine resource development, maritime safety monitoring, and disaster emergency response. These scenarios involve diverse target types, dense distribution, and complex backgrounds (such as sea waves, wharf walls, and mixed onshore facilities), placing extremely high demands on high-precision target detection around the clock and in all weather conditions.

[0003] Currently, remote sensing target detection for ports and offshore facilities mainly relies on single-mode data, namely synthetic aperture radar (SAR) remote sensing imagery or optical remote sensing imagery. SAR imagery possesses the ability to penetrate clouds and rain, operates day and night, and can stably capture the scattering characteristics of targets such as metal structures and ships. However, it suffers from significant speckle noise, especially in areas with sea clutter and dense infrastructure, resulting in weak semantic interpretability of targets and extremely high manual annotation costs. Optical remote sensing imagery offers high resolution, intuitive texture and color information, and strong identification capabilities for ship types and facility details. However, it is highly susceptible to interference from clouds and fog, changes in lighting, darkness at night, and sea surface flares and shadows, making it difficult to ensure continuous observation during critical periods such as inclement weather and nighttime. Therefore, fusing SAR and optical remote sensing imagery for collaborative detection has become a key direction for improving system robustness.

[0004] Although existing research has explored multimodal fusion strategies (such as early fusion, late fusion, and mid-term fusion), current technologies still face the following four core shortcomings in SAR and optical collaborative inspection tasks for ports and offshore facilities: Geometric misalignment: Port and offshore facilities are mostly artificial rigid targets. Due to differences in imaging mechanisms between SAR and optical remote sensing images, the same berthed vessel or pier edge will exhibit irregular and nonlinear geometric deformations and spatial position shifts. Especially in densely packed berth areas, where vessels are close together and at pier shoreline bends, traditional methods based on fixed convolution or rigid registration cannot effectively model this local geometric distortion, leading to cross-modal feature space misalignment. This directly causes vessel target box drift, missed detection of berthed vessels, and misjudgment of facility boundaries.

[0005] Semantic complementarity is challenging: In port and offshore facility scenarios, SAR remote sensing images highlight the strong scattering centers of ships and the metal structures of platforms, but are accompanied by severe speckle noise and sea surface clutter. Optical remote sensing images provide details of ship hull color, deck texture, and dock appearance, but are affected by shadows, water surface reflections, and cloud cover. Existing methods often employ simple stitching or summation, failing to achieve bidirectional semantic interaction and filtering. This not only makes it difficult to extract deep complementary information from the two modalities but also easily amplifies SAR speckle noise or introduces optical shadow interference, resulting in feature contamination from densely berthed ships and failure to identify small-sized facilities.

[0006] Imbalanced modal contributions: Port and maritime weather is highly variable. Common operating conditions such as sea fog, rain, and nighttime operations can cause optical modes to suddenly fail, highlighting the all-weather capability of SAR; the advantage of optical detail is obvious during clear days. Existing methods use fixed or heuristic fusion weights and cannot autonomously sense changes in modal quality. When clouds and fog suddenly appear or light levels drop sharply, the forced fusion of low-quality optical features can easily lead to a collapse in detection performance, making it difficult to meet the all-weather robustness requirements for continuous port monitoring and maritime emergency response.

[0007] Insufficient global context modeling capabilities: Port and maritime targets span a wide range of scales (from large wind turbine installation platforms to small lifeboats and buoys), are densely distributed, and often obscure each other. Meanwhile, complex background interference such as sea waves and dock textures easily confuse with real targets. Existing methods mostly focus on local feature alignment and fusion, lacking effective modeling of global context relationships. This makes it difficult to distinguish targets from similar backgrounds and capture large-scale dependencies, thus affecting the accurate detection of small buoys and obscuring vessels, limiting practical operational applications. Summary of the Invention

[0008] In a first aspect, embodiments of this disclosure provide a SAR and optical co-detection method for ports and offshore facilities, the method comprising: Obtain a remote sensing image target detection dataset, wherein each sample in the remote sensing image target detection dataset includes paired SAR remote sensing images and optical remote sensing images of ports and offshore facilities and their corresponding target labels; A multimodal fusion target detection network is constructed, comprising a dual-stream encoder, a multimodal fusion module, and a detection head. The dual-stream encoder includes two independent branches: the first branch is a SAR remote sensing image encoder, used to extract the structural and scattering features of SAR remote sensing images to obtain SAR features; the second branch is an optical remote sensing image encoder, used to extract the global appearance features of optical remote sensing images to obtain optical features. The multimodal fusion module sequentially includes a deformable alignment unit, a bidirectional cross-attention unit, a gated fusion unit, and a lightweight global context enhancement unit, used to fuse SAR features and optical features to obtain multi-scale fused features. The detection head adopts a path aggregation network-feature pyramid network (PAN-FPN) structure to perform target detection on the multi-scale fused features to obtain the target detection result. The multimodal fusion target detection network was trained using a remote sensing image target detection dataset to obtain a SAR and optical co-detection model. The target detection results were obtained by using a SAR and optical co-detection model to perform target detection on paired SAR and optical remote sensing images of the target port and offshore facilities.

[0009] In some possible implementations of the first aspect, the deformable alignment unit is used for: Predict the spatial offset field of SAR and optical features using an offset prediction network; The spatial positions of SAR and optical features are adjusted based on the spatial offset field to achieve geometric alignment between the adjusted SAR and optical features.

[0010] In some possible implementations of the first aspect, the bidirectional cross-attention unit is used for: The aligned SAR features are projected as the first Query, and the aligned optical features are projected as the first Key and the first Value. The first scaled dot product attention from optical mode to SAR mode is calculated based on the first Query, the first Key, and the first Value. Multi-head stitching and linear projection are performed based on the first scaled dot product attention, and the results are residually connected and layer normalized with the aligned SAR features to obtain the enhanced SAR features. The aligned optical features are projected as the second Query, and the aligned SAR features are projected as the second Key and the second Value. The second scaled dot product attention from SAR mode to optical mode is calculated based on the second Query, the second Key, and the second Value. Multi-head stitching and linear projection are performed based on the second scaled dot product attention, and the results are residually connected and layer normalized with the aligned optical features to obtain the enhanced optical features.

[0011] In some possible implementations of the first aspect, the gating fusion unit is used for: The enhanced SAR features and enhanced optical features are concatenated by channel stitching and then fused using 1×1 convolution to obtain the first fused feature; The first fused feature is subjected to global average pooling, and the adaptive fusion weights corresponding to the enhanced SAR feature and the enhanced optical feature are calculated through a gated weight network. The enhanced SAR features and enhanced optical features are weighted and fused using adaptive fusion weights corresponding to the enhanced SAR features and enhanced optical features to obtain the second fused feature. Based on this, a third fused feature is obtained by 1×1 convolution with residual connections.

[0012] In some possible implementations of the first aspect, the lightweight global context enhancement unit employs a residual Transformer structure for: The third fusion feature is layer normalized and input into a multi-head self-attention module. The result is then residually connected with the third fusion feature to obtain the fourth fusion feature. The fourth fusion feature is layer normalized and input into the feedforward network. The result is then residually connected with the fourth fusion feature to obtain the fifth fusion feature, which is then used as the multi-scale fusion feature.

[0013] In some possible implementations of the first aspect, a multimodal fusion target detection network is trained using a remote sensing image target detection dataset to obtain a SAR and optical co-detection model, including: The samples in the remote sensing image target detection dataset are uniformly adjusted to a fixed size; The adjusted remote sensing image target detection dataset is divided into a training dataset and a validation dataset according to a preset ratio. The multimodal fusion target detection network was trained multiple times using the training dataset, and validated using the validation dataset after each training round. Its performance index was calculated, and the network parameters with the optimal performance index were saved. The multimodal fusion target detection network with the optimal performance index was used as the SAR and optical co-detection model.

[0014] Among the possible implementations of the first aspect, the multimodal fusion object detection network is trained multiple times using a training dataset, including: The training dataset is augmented, and the augmented training dataset is used to train the multimodal fusion object detection network for multiple rounds.

[0015] Secondly, embodiments of this disclosure provide a SAR and optical co-detection device for ports and offshore facilities, the device comprising: The acquisition module is used to acquire the remote sensing image target detection dataset, wherein each sample in the remote sensing image target detection dataset includes paired SAR remote sensing images and optical remote sensing images of ports and marine facilities and their corresponding target labels; The construction module is used to build a multimodal fusion target detection network, which includes a dual-stream encoder, a multimodal fusion module, and a detection head. The dual-stream encoder includes two independent branches: the first branch is a SAR remote sensing image encoder, used to extract the structural and scattering features of SAR remote sensing images to obtain SAR features; the second branch is an optical remote sensing image encoder, used to extract the global appearance features of optical remote sensing images to obtain optical features. The multimodal fusion module includes a deformable alignment unit, a bidirectional cross-attention unit, a gated fusion unit, and a lightweight global context enhancement unit, used to fuse SAR features and optical features to obtain multi-scale fused features. The detection head adopts a PAN-FPN structure to perform target detection on the multi-scale fused features to obtain the target detection result. The training module is used to train a multimodal fusion target detection network using a remote sensing image target detection dataset to obtain a SAR and optical co-detection model. The detection module is used to perform target detection on paired SAR and optical remote sensing images of target ports and marine facilities using a SAR and optical joint detection model, and obtain target detection results.

[0016] Thirdly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0017] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0018] In summary, the embodiments of this disclosure acquire a remote sensing image target detection dataset containing paired SAR and optical remote sensing images of ports and offshore facilities, along with their corresponding target labels. A multimodal fusion target detection network consisting of a dual-stream encoder, a multimodal fusion module, and a detection head is constructed. This network is then trained using the aforementioned dataset to obtain a SAR and optical collaborative detection model. This model is then used to detect targets in paired SAR and optical remote sensing images of ports and offshore facilities, achieving high-precision, all-weather, and 24 / 7 detection of port and offshore facility targets.

[0019] Specifically, this embodiment ensures cross-modal pairing consistency from the data source by using a dual-stream encoder to extract the structural and scattering features of SAR remote sensing images and the global appearance features of optical remote sensing images, fully preserving SAR and light features. Based on this, the multimodal fusion module sequentially executes a deformable alignment unit, a bidirectional cross-attention unit, a gated fusion unit, and a lightweight global context enhancement unit, progressively overcoming four core challenges: geometric misalignment, semantic complementarity difficulties, unbalanced modal contributions, and insufficient global context modeling. The deformable alignment unit adaptively corrects irregular nonlinear geometric offsets between SAR and optical features in the feature space, achieving pixel-level spatial alignment; the bidirectional cross-attention unit establishes bidirectional semantic interaction, finely filters complementary information, and suppresses SAR speckle and optical shadow interference; the gated fusion unit dynamically senses modal quality and adaptively allocates contribution, ensuring stable fusion under various imaging conditions; the lightweight global context enhancement unit models long-range dependencies with extremely low computational overhead, enhancing the discrimination of multi-scale targets and complex backgrounds; and the PAN-FPN structured detection head fully reuses multi-scale fusion features, significantly improving the detection accuracy of ports and offshore facilities at different scales. The entire method is optimized end-to-end, and can obtain reliable target detection results in complex scenarios such as day-night cycles, cloud cover, and severe sea conditions, providing solid technical support for intelligent waterway management, marine resource development, maritime safety monitoring, and disaster emergency response.

[0020] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A flowchart of a SAR and optical co-detection method for ports and offshore facilities provided in this disclosure is shown. Figure 2 A structural diagram of a multimodal fusion target detection network provided in an embodiment of this disclosure is shown; Figure 3 A structural diagram of a multimodal fusion module provided in an embodiment of this disclosure is shown; Figure 4 This diagram illustrates a structural diagram of a SAR and optical co-detection device for ports and offshore facilities provided in an embodiment of this disclosure. Figure 5A structural diagram of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0023] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0024] To address the technical problems mentioned in the background art, this disclosure provides a SAR and optical collaborative detection method and apparatus for ports and offshore facilities. It acquires a remote sensing image target detection dataset containing paired SAR and optical remote sensing images of ports and offshore facilities, along with their corresponding target labels. A multimodal fusion target detection network is constructed, consisting of a dual-stream encoder, a multimodal fusion module, and a detection head. This network is then trained using the dataset to obtain a SAR and optical collaborative detection model. This model is then used to detect targets in paired SAR and optical remote sensing images of ports and offshore facilities, achieving high-precision, all-weather, and 24 / 7 detection of port and offshore facility targets.

[0025] The following detailed description, in conjunction with the accompanying drawings, of a SAR and optical collaborative detection method and apparatus for ports and marine facilities provided by this disclosure, through specific embodiments, will be provided in detail.

[0026] Figure 1 A flowchart of a SAR and optical co-detection method for ports and offshore facilities provided in this disclosure is shown, as illustrated in the embodiments below. Figure 1 As shown, method 100 may include the following steps: S110, Obtain the target detection dataset from remote sensing images.

[0027] The remote sensing image target detection dataset includes multiple samples. Each sample consists of paired SAR and optical remote sensing images of ports and maritime facilities, along with their corresponding target labels, such as target category and location information. Furthermore, the pairing here refers to spatiotemporal matching.

[0028] S120, construct a multimodal fusion target detection network.

[0029] Among them, multimodal fusion target detection networks can, for example, Figure 2 As shown, it includes a dual-stream encoder, a multimodal fusion module, and a detection head.

[0030] Furthermore, the dual-stream encoder includes two independent branches, namely the first branch and the second branch. Specifically, the first branch is a SAR remote sensing image encoder, which is composed of a standard convolutional neural network and is used to extract the structural and scattering features of SAR remote sensing images to obtain SAR features. The second branch is an optical remote sensing image encoder, composed of a standard convolutional neural network, used to extract global appearance features of optical remote sensing images to obtain optical features. .

[0031] The multimodal fusion module can be like Figure 3 As shown, the system comprises a deformable alignment unit, a bidirectional cross-attention unit, a gated fusion unit, and a lightweight global context enhancement unit. These components are used to perform multi-level deep fusion of SAR and optical features, from geometric to semantic and ultimately global context, to obtain multi-scale fused features. The functions of the deformable alignment unit, bidirectional cross-attention unit, gated fusion unit, and lightweight global context enhancement unit are described in detail below: (1) Deformable Alignment Unit The deformable alignment unit is used to achieve cross-modal spatial alignment with Deformable Convolution v2 via offset prediction. Its execution process is as follows: Using offset prediction network Predicting SAR features and optical characteristics Spatial offset field : in, This indicates channel splicing.

[0032] According to the spatial offset field SAR features and optical characteristics The spatial location is adjusted to make the adjusted SAR features With adjusted optical characteristics Geometric alignment between them.

[0033] Specifically, using the spatial offset field SAR features and optical characteristics Perform deformable sampling. For SAR features and optical characteristics a certain position in Its deformable sample value is: in, Indicate SAR characteristics a certain position in The corresponding deformable sampled values; Indicates optical characteristics a certain position in The corresponding deformable sampled values; Indicates the total number of sampling points; Indicates the first The weight of each sampling point; Indicates the first Preset spatial offset of each sampling point; Indicates the first Each sampling point in the spatial offset field Spatial offset in; Indicates SAR features In the middle, position Eigenvalues ​​at; Indicating in optical characteristics In the middle, position The eigenvalue at that location.

[0034] SAR features and optical characteristics Deformable sampling results and Perform convolution operations to obtain adjusted SAR features. With adjusted optical characteristics : =Conv( ) =Conv( ) Among them, Conv( () indicates convolution. Adjusted SAR features With adjusted optical characteristics Also known as aligned SAR features and aligned optical features.

[0035] (2) Bidirectional cross-attention unit Bidirectional cross-attention units are used for semantic-level information interaction and enhancement between multimodal modes. Their execution process is as follows: Attention in the direction from optical mode to SAR mode: Aligned SAR features Projecting the first query will align the optical features. The projections are the first Key and the first Value, denoted as follows: , , : in, , , These represent the Query projection weight matrix, Key projection weight matrix, and Value projection weight matrix, respectively.

[0036] according to , , Calculate the first scaled dot product attention from optical mode to SAR mode : in, This represents the normalized exponential function; express The vector dimension.

[0037] Based on the first scaled dot product attention Perform multi-head stitching and linear projection, and then compare the results with the aligned SAR features. Residual connectivity and layer normalization are performed to obtain enhanced SAR features. : in, Indicates attention based on the first scaled dot product. The result obtained by multi-head stitching and linear projection; Indicates multi-head splicing; The number of heads indicating multi-head attention; Indicates the first The first scaled dot product attention is obtained by calculating the attention head; Represents the linear projection weight matrix; Representation layer normalization.

[0038] Attention in the SAR mode-to-optical mode direction: Aligned optical features Projecting the second query will align the SAR features. The projection is the second key and the second value, denoted as follows: , , : in, , , These represent the Query projection weight matrix, Key projection weight matrix, and Value projection weight matrix, respectively.

[0039] according to , , Calculate the second scaled dot product attention from SAR mode to optical mode: in, This represents the normalized exponential function; express The vector dimension.

[0040] Based on the second scaled dot product attention Perform multi-head stitching and linear projection, and then compare the results with the aligned optical features. Residual connectivity and layer normalization are performed to obtain enhanced optical features. : in, Based on the second scaled dot product attention The result obtained by multi-head stitching and linear projection; Indicates multi-head splicing; The number of heads indicating multi-head attention; Indicates the first The second scaled dot product attention is obtained by calculating the attention head; Represents the linear projection weight matrix; Representation layer normalization.

[0041] (3) Gated fusion unit The gated fusion unit is used to achieve adaptive fusion between modalities based on the bidirectionally enhanced features. Its execution process is as follows: Enhanced SAR characteristics and enhanced optical features Channel concatenation is performed, and feature fusion is achieved through 1×1 convolution to obtain the first fused feature. : in, Represents a 1×1 convolution; This indicates channel splicing.

[0042] For the first fusion feature Global average pooling is performed, and enhanced SAR features are calculated through a gated weight network. and enhanced optical features Corresponding adaptive fusion weights: in, , These represent enhanced SAR features. and enhanced optical features Corresponding adaptive fusion weights; This represents the normalized exponential function; Indicates global average pooling; and These are the gating parameters.

[0043] use , Enhanced SAR features and enhanced optical features Weighted fusion is performed to obtain the second fusion feature. : According to the second fusion feature The third fused feature is obtained through a 1×1 convolution with residual connections. : (4) Lightweight global context enhancement unit The lightweight global context enhancement unit adopts a residual Transformer structure to improve the global modeling capability of fused features. Its execution process is as follows: Third fusion feature Layer normalization is performed and input into a multi-head self-attention module. The result is then combined with a third fusion feature. Perform residual connections to obtain the fourth fusion feature. : in, Indicates the third fusion feature The result of layer normalization; Representation layer normalization; This represents the output of the multi-head self-attention module; This indicates a multi-head self-attention module; This represents the first learnable residual scaling factor.

[0044] Fourth fusion feature Layer normalization is performed and input into the feedforward network. The result is then combined with the fourth fused feature. Perform residual connections to obtain the fifth fusion feature. The fifth fusion feature As a feature of multi-scale fusion: in, Indicates the fourth fusion feature The result of layer normalization; Representation layer normalization; This represents the output of the feedforward network; Indicates a feedforward network; This represents the learnable second residual scaling factor.

[0045] Furthermore, multi-head self-attention module Feedforward network The calculations are approximated as follows: in, The number of heads indicating multi-head attention; Indicates the first A query for each attention head; Indicates the first A key to attention; Indicates the first Value of each attention head; express The vector dimension; This represents the normalized exponential function; Represents feedforward network The weight matrix of the first layer; Indicates the activation function; Represents feedforward network The weight matrix of the first layer.

[0046] It is worth noting that the detection head here adopts a PAN-FPN structure for target detection based on multi-scale fused features, such as target classification and bounding box regression, to obtain target detection results, such as target category and location information. The PAN-FPN structure can fully utilize semantic and spatial features at different levels to achieve high-precision target detection in SAR-optical multimodal remote sensing images.

[0047] S130 uses a remote sensing image target detection dataset to train a multimodal fusion target detection network, resulting in a SAR and optical co-detection model.

[0048] In some embodiments, the samples in the remote sensing image target detection dataset are uniformly adjusted to a fixed size.

[0049] The adjusted remote sensing image target detection dataset is divided into a training dataset and a validation dataset according to a preset ratio (e.g., 4:1).

[0050] The multimodal fusion target detection network is trained in multiple rounds using the training dataset. Specifically, the training dataset can be augmented, and the augmented training dataset can be used to train the multimodal fusion target detection network in multiple rounds. After each round of training, the multimodal fusion target detection network is validated using the validation dataset, its performance index is calculated, and the network parameters when the performance index is optimal are saved. The multimodal fusion target detection network loaded with the network parameters when the performance index is optimal is used as the SAR and optical co-detection model.

[0051] S140 uses a SAR and optical co-detection model to detect targets in paired SAR and optical remote sensing images of the target port and marine facilities, and obtains the target detection results.

[0052] Among them, the target ports and offshore facilities are the ports and offshore facilities that need to be subject to target detection.

[0053] In summary, the embodiments disclosed herein achieve at least the following technical effects compared to the prior art: (1) Accurately correct cross-modal geometric offsets of ports and offshore facilities to achieve structural-level spatial alignment. Addressing the complex nonlinear deformations of targets such as wharf shorelines, moored vessels, and offshore platforms in SAR and optical features, this embodiment employs deformable alignment units to adaptively compensate for local geometric mismatches within the feature space. Without external explicit registration, pixel-level precise alignment can be achieved at the feature level, significantly eliminating typical problems such as target frame drift of densely packed berthed vessels and wharf edge misalignment, greatly improving the positioning accuracy of berthed vessels and coastal facilities, and providing a precise spatial foundation for subsequent fusion.

[0054] (2) Bidirectional selective semantic refinement enhances facility structure and suppresses sea state interference. To address unique interferences such as port surface reflections, wave clutter, and fog shadows, this embodiment utilizes bidirectional cross-attention units to establish a cross-modal feature interaction channel. SAR features, guided by optical details, refine missing texture and boundary information; optical features, guided by SAR stable structures, enhance scattering centers and geometric contours. This mechanism bidirectionally filters complementary features that play a crucial role in port facility and vessel identification, while adaptively suppressing SAR sea clutter speckle, optical shadows, and flares, ensuring that the fused features possess both structural integrity and visual recognizability, significantly improving the detection confidence of densely moored vessels and dock facilities under complex lighting and sea conditions.

[0055] (3) Dynamic adaptive fusion weights ensure robust operation around the clock. Addressing common port conditions such as sea fog, rain, and darkness at night that cause optical mode failure, the gated fusion unit designed in this embodiment can predict the confidence level of each mode in real time based on the input feature quality and adaptively adjust the contribution weights. During clear daytime conditions, it automatically emphasizes optical details, while dynamically enhancing SAR contributions and suppressing optical interference during periods of cloud cover, nighttime, or severe surge noise. This mechanism fundamentally avoids amplifying erroneous information, ensuring the model maintains stable output in drastically changing marine environments, truly leveraging the synergistic advantages of all-weather SAR and high-resolution optical capabilities to meet the continuous needs of dynamic channel monitoring and maritime search and rescue.

[0056] (4) Lightweight global context enhancement improves the ability to distinguish between dense targets at multiple scales and complex backgrounds. Addressing challenges such as drastic scale changes in ports and offshore facilities, the easy confusion of small targets with waves, and mutual occlusion between ships in densely packed berths, this embodiment introduces a lightweight global context enhancement unit to efficiently model long-range global spatial dependencies with minimal computational increment. The network can perceive contextual information such as a large area of ​​sea surface and wharf shoreline, effectively distinguishing small buoys, lifeboats, and wave clutter, improving feature completion and detection recall of occluded ships. While maintaining high inference efficiency and real-time performance, it significantly improves the accuracy of multi-scale target detection, strongly supporting practical operations such as port facility inspection and maritime traffic monitoring.

[0057] To facilitate further understanding, the method 100 provided in this disclosure will be described in detail below with reference to a specific embodiment, as shown in the following figure: (1) Obtain the target detection dataset from remote sensing images .

[0058] The remote sensing image target detection dataset includes multiple samples, each of which includes paired SAR and optical remote sensing images of ports and marine facilities and their corresponding target labels.

[0059] (2) The remote sensing image target detection dataset The samples in the dataset were uniformly adjusted to a fixed size to obtain the adjusted remote sensing image target detection dataset. .

[0060] (3) The adjusted remote sensing image target detection dataset is divided into two parts at a ratio of 4:1. Divided into training dataset and validation dataset .

[0061] (4) Construct the multimodal fusion target detection network as described above.

[0062] (5) For the training dataset Perform data augmentation and use the augmented training dataset. The multimodal fusion object detection network is trained in multiple rounds, and a validation dataset is used after each round of training. The multimodal fusion target detection network was validated, its performance indicators were calculated, and the network parameters when the performance indicators were optimal were saved.

[0063] (6) The multimodal fusion target detection network with the network parameters at the optimal loading performance index is used as the SAR and optical collaborative detection model. The model is used to perform target detection on the paired SAR remote sensing images and optical remote sensing images of the target port and marine facilities to obtain the target detection results.

[0064] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0065] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0066] Figure 4 This disclosure provides a structural diagram of a SAR and optical co-detection device for ports and offshore facilities, as shown in the embodiments below. Figure 4 As shown, device 400 may include: The acquisition module 410 is used to acquire a remote sensing image target detection dataset, wherein each sample in the remote sensing image target detection dataset includes paired SAR remote sensing images and optical remote sensing images of ports and marine facilities and their corresponding target labels.

[0067] Module 420 is used to construct a multimodal fusion target detection network, which includes a dual-stream encoder, a multimodal fusion module, and a detection head. The dual-stream encoder includes two independent branches: the first branch is a SAR remote sensing image encoder, used to extract the structural and scattering features of SAR remote sensing images to obtain SAR features; the second branch is an optical remote sensing image encoder, used to extract the global appearance features of optical remote sensing images to obtain optical features. The multimodal fusion module includes a deformable alignment unit, a bidirectional cross-attention unit, a gated fusion unit, and a lightweight global context enhancement unit, used to fuse SAR features and optical features to obtain multi-scale fused features. The detection head adopts a PAN-FPN structure to perform target detection on the multi-scale fused features to obtain the target detection result.

[0068] Training module 430 is used to train a multimodal fusion target detection network using a remote sensing image target detection dataset to obtain a SAR and optical co-detection model.

[0069] The detection module 440 is used to perform target detection on paired SAR and optical remote sensing images of target ports and marine facilities using a SAR and optical joint detection model, and obtain target detection results.

[0070] Understandable Figure 4 Each module / unit in the illustrated device 400 has the ability to implement Figure 1 The functions of each step in method 100 shown, and their corresponding technical effects, will not be elaborated here for the sake of brevity.

[0071] Figure 5 A structural diagram of an electronic device provided in an embodiment of this disclosure is shown. Electronic device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 500 may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0072] like Figure 5 As shown, the electronic device 500 may include a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0073] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0074] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as method 100. For example, in some embodiments, method 100 may be implemented as a computer program product, including a computer program tangibly contained in a computer-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

[0075] The various embodiments described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), payload programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0076] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0077] In the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0078] It should be noted that this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions. These computer instructions are used to cause a computer to execute method 100 and achieve the corresponding technical effects achieved by executing the method in the embodiments of this disclosure; for the sake of brevity, they will not be elaborated further here.

[0079] In addition, this disclosure also provides a computer program product including a computer program that implements method 100 when executed by a processor.

[0080] To provide interaction with a user, the embodiments described above can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0081] The embodiments described above can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with the implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0082] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0083] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired results of this disclosure are achieved, and this is not limited herein.

[0084] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A SAR and optical collaborative detection method for ports and offshore facilities, characterized in that, The method includes: Obtain a remote sensing image target detection dataset, wherein each sample in the remote sensing image target detection dataset includes paired SAR remote sensing images and optical remote sensing images of ports and marine facilities and their corresponding target labels; A multimodal fusion target detection network is constructed, comprising a dual-stream encoder, a multimodal fusion module, and a detection head. The dual-stream encoder includes two independent branches: a first branch is a SAR remote sensing image encoder, used to extract the structural and scattering features of SAR remote sensing images to obtain SAR features; the second branch is an optical remote sensing image encoder, used to extract the global appearance features of optical remote sensing images to obtain optical features. The multimodal fusion module sequentially includes a deformable alignment unit, a bidirectional cross-attention unit, a gated fusion unit, and a lightweight global context enhancement unit, used to fuse the SAR features and the optical features to obtain multi-scale fused features. The detection head adopts a PAN-FPN structure, used to perform target detection on the multi-scale fused features to obtain target detection results. The multimodal fusion target detection network is trained using the remote sensing image target detection dataset to obtain a SAR and optical co-detection model; The SAR and optical co-detection model is used to detect targets in paired SAR and optical remote sensing images of the target port and offshore facilities, and the target detection results are obtained.

2. The method according to claim 1, characterized in that, The deformable alignment unit is used for: The spatial offset fields of the SAR features and the optical features are predicted by an offset prediction network; The spatial positions of the SAR feature and the optical feature are adjusted according to the spatial offset field so that the adjusted SAR feature and the adjusted optical feature are geometrically aligned.

3. The method according to claim 2, characterized in that, The bidirectional cross-attention unit is used for: The aligned SAR features are projected as the first Query, and the aligned optical features are projected as the first Key and the first Value. The first scaled dot product attention from optical mode to SAR mode is calculated based on the first Query, the first Key, and the first Value. Multi-head stitching and linear projection are performed based on the first scaled dot product attention, and the results are residually connected and layer normalized with the aligned SAR features to obtain the enhanced SAR features. The aligned optical features are projected as the second Query, and the aligned SAR features are projected as the second Key and the second Value. The second scaled dot product attention from SAR mode to optical mode is calculated based on the second Query, the second Key, and the second Value. Multi-head stitching and linear projection are performed based on the second scaled dot product attention, and the results are residually connected and layer normalized with the aligned optical features to obtain the enhanced optical features.

4. The method according to claim 3, characterized in that, The gated fusion unit is used for: The enhanced SAR features and enhanced optical features are concatenated by channel stitching and then fused using 1×1 convolution to obtain the first fused feature; The first fused feature is subjected to global average pooling, and the adaptive fusion weights corresponding to the enhanced SAR feature and the enhanced optical feature are calculated through a gated weight network. The enhanced SAR features and enhanced optical features are weighted and fused using adaptive fusion weights corresponding to the enhanced SAR features and enhanced optical features to obtain the second fused feature. Based on this, a third fused feature is obtained by 1×1 convolution with residual connections.

5. The method according to claim 4, characterized in that, The lightweight global context enhancement unit adopts a residual Transformer structure and is used for: The third fusion feature is layer normalized and input into a multi-head self-attention module. The result is then residually connected with the third fusion feature to obtain the fourth fusion feature. The fourth fusion feature is layer normalized and input into a feedforward network. The result is then residually connected with the fourth fusion feature to obtain the fifth fusion feature, which is then used as the multi-scale fusion feature.

6. The method according to claim 5, characterized in that, The step of training the multimodal fusion target detection network using the remote sensing image target detection dataset to obtain the SAR and optical joint detection model includes: The samples in the remote sensing image target detection dataset are uniformly adjusted to a fixed size; The adjusted remote sensing image target detection dataset is divided into a training dataset and a validation dataset according to a preset ratio. The multimodal fusion target detection network is trained multiple times using the training dataset, and the multimodal fusion target detection network is validated using the validation dataset after each training round. Its performance index is calculated, and the network parameters when the performance index is optimal are saved. The multimodal fusion target detection network with the network parameters when the performance index is optimal is used as the SAR and optical co-detection model.

7. The method according to claim 6, characterized in that, The step of training the multimodal fusion object detection network using the training dataset in multiple rounds includes: The training dataset is augmented, and the augmented training dataset is used to train the multimodal fusion object detection network for multiple rounds.

8. A SAR and optical collaborative detection device for ports and offshore facilities, characterized in that, The device includes: The acquisition module is used to acquire a remote sensing image target detection dataset, wherein each sample in the remote sensing image target detection dataset includes paired SAR remote sensing images and optical remote sensing images of ports and marine facilities and their corresponding target labels. A construction module is used to build a multimodal fusion target detection network, wherein the multimodal fusion target detection network includes a dual-stream encoder, a multimodal fusion module, and a detection head; the dual-stream encoder includes two independent branches, specifically, the first branch is a SAR remote sensing image encoder, used to extract the structural and scattering features of SAR remote sensing images to obtain SAR features, and the second branch is an optical remote sensing image encoder, used to extract the global appearance features of optical remote sensing images to obtain optical features; the multimodal fusion module sequentially includes a deformable alignment unit, a bidirectional cross-attention unit, a gated fusion unit, and a lightweight global context enhancement unit, used to fuse the SAR features and the optical features to obtain multi-scale fusion features; the detection head adopts a PAN-FPN structure, used to perform target detection on the multi-scale fusion features to obtain target detection results; The training module is used to train the multimodal fusion target detection network using the remote sensing image target detection dataset to obtain a SAR and optical co-detection model. The detection module is used to perform target detection on the paired SAR and optical remote sensing images of the target port and marine facilities using the SAR and optical joint detection model, and obtain the target detection results.

9. An electronic device, characterized in that, The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method described in any one of claims 1-7.

Images