Optical-SAR (Synthetic Aperture Radar) fusion target detection method under cloud and mist conditions

Abstract

The invention relates to an optical-SAR (Synthetic Aperture Radar) fusion target detection method under a cloud and mist condition. The method comprises the following steps: constructing an optical-SAR fusion target detection data set under the cloud and mist condition; an optical-SAR fusion target detection model is constructed; the model is composed of a double-flow backbone network based on YOLOv5 expansion and a continuous dual-polymerization Transform module embedded in the middle of the double-flow backbone network. Training the detection model based on the data set; and performing fusion target detection by using the trained detection model. According to the method, information of two dimensions of a channel and a space in the Transform can be fused, feature aggregation of cross-space and channel dimensions is realized through dual modes of a dual aggregation Transform block between modules and in the modules, and deep fusion of features in the modules is performed by means of an adaptive interaction module and a space door feedforward network. The multi-modal feature expression can be integrated more efficiently, and the performance of optical-SAR fusion target detection is improved comprehensively.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and target detection technology, and in particular to an optical-SAR fusion target detection method under foggy conditions. Background Art

[0002] Synthetic Aperture Radar (SAR) offers the advantages of all-day, all-weather coverage and the ability to penetrate clouds and fog. However, its image interpretation is often hampered by issues such as speckle noise, making it difficult to interpret. In contrast, optical imagery clearly captures both the spectral and spatial information of objects, making interpretation relatively simple. However, this often results in information loss in foggy conditions. To improve target detection, the fusion of optical and SAR imagery data has become an effective solution.

[0003] In recent years, SAR data has been recognized as a novel auxiliary data source for recovering optical images obscured by clouds, primarily because microwave signals penetrate clouds better than optical signals. SAR systems acquire surface information by transmitting microwave signals and receiving the reflected echoes. Because clouds attenuate microwave signals less, SAR images can, to a certain extent, reveal surface features obscured by clouds.

[0004] Combining SAR data with optical remote sensing data can fully leverage their complementary nature and improve the accuracy and reliability of obtaining information about the ground beneath clouds. For example, SAR data can help identify cloud areas, filling in data gaps in optical images caused by cloud obstruction. Data fusion can also improve the accuracy of cloud edge detection and cloud type identification. Furthermore, with the continuous advancement of remote sensing technology, the resolution and accuracy of SAR data are also continuously improving, providing more possibilities for the detailed extraction of ground beneath clouds. Therefore, using SAR data as an auxiliary data source and fusing it with optical remote sensing data has become an important approach to addressing cloud obstruction and improving the utilization of remote sensing data. This fusion not only enhances the efficiency of data analysis but also improves the accuracy of overall target detection, providing more reliable support for various applications.

[0005] However, current fusion methods primarily rely on simple feature addition or splicing strategies, which fail to fully explore and exploit the deep complementary relationship between optical and SAR imagery. This approach can limit fusion results and fail to fully leverage the strengths of both data sources. Therefore, exploring more advanced and effective fusion techniques to fully integrate information from different image types, thereby improving the accuracy of subcloud surface information extraction and target detection, has become a key research issue. Summary of the Invention

[0006] In order to solve the technical problems existing in the above-mentioned prior art, the purpose of the present invention is to provide an optical-SAR fusion target detection method under cloud and fog conditions, which can effectively solve the shortcomings of traditional rule-based or finite state machine-based behavior rule modeling, and aims at sharing and transmitting complex data between various behavior elements in spacecraft system confrontation simulation, as well as the construction requirements of intelligent behavior rules. The behavior tree is modified, and a behavior rule modeling solution for spacecraft system confrontation is designed. A graphical behavior rule modeling method is provided to improve the automation level and confidence of spacecraft system confrontation simulation.

[0007] To achieve the above-mentioned object of the invention, the present invention provides an optical-SAR fusion target detection method under foggy conditions, comprising the following steps:

[0008] Step S1: Perform simulated cloud and fog addition processing on the preset optical image to construct an optical-SAR fusion target detection dataset under cloud and fog conditions;

[0009] Step S2: constructing an optical-SAR fusion target detection model; the optical-SAR fusion target detection basic model includes an optical feature detector and a SAR feature detector, and a dual-aggregation Transformer module embedded between the optical feature detector and the SAR feature detector;

[0010] Step S3: training the optical-SAR fusion target detection model based on the data set constructed in step S1 to obtain a trained optical-SAR fusion target detection model;

[0011] Step S4: Utilize the trained optical-SAR fusion target detection model to perform fusion target detection.

[0012] According to a technical solution of the present invention, in step S1, it specifically includes:

[0013] Construct target category system and sample category specification based on optical remote sensing image data and expert experience knowledge;

[0014] Based on the target category system and the sample category specification, horizontal box annotation is performed on the target position and category in the remote sensing image data;

[0015] Slicing the annotated optical remote sensing image data, and performing cloud and fog occlusion of different proportions on the sliced ​​optical remote sensing image data to form an optical-SAR fusion target detection dataset under the cloud and fog conditions;

[0016] The optical-SAR fusion target detection dataset under cloud and fog conditions is divided into a training set, a validation set and a test set.

[0017] According to a technical solution of the present invention, in step S2, the optical feature detector and the SAR feature detector are the same, both being YOLOv5 networks, including a backbone network, a feature pyramid network, and a detection network;

[0018] The backbone network is used to extract a multi-scale feature map of the input image;

[0019] The feature pyramid network is used to fuse the high-level semantic information and low-level positioning information extracted by the backbone network;

[0020] The detection network is used to detect the fused multi-scale feature map to obtain the target fine-grained classification and positioning results of the pixel points on each feature map;

[0021] The dual-aggregation Transformer module includes a spatial Transformer module and a channel Transformer module connected in series;

[0022] The original input residual features input to the spatial Transformer module are divided into two paths. One path is normalized in series and then subjected to an adaptive spatial self-attention module, and then added element-by-element to the other path of original input residual features. The resulting original residual features are normalized in series and subjected to an SGFN layer, and then added element-by-element to the other path of original residual features to form the spatial Transformer module output features for input into the channel Transformer module.

[0023] The original input residual features of the channel Transformer module are divided into two paths. One path is normalized in series and then passes through the adaptive channel self-attention module, and then is added element-by-element to the other path of original input residual features. The resulting original residual features are normalized in series and passed through the SGFN layer, and then added element-by-element to the other path of original residual features to form the output features of the channel Transformer module.

[0024] The input features of the adaptive spatial self-attention module are divided into two paths, one of which is generated by the spatial self-attention module to generate the self-attention feature Y s , the other path generates convolution feature Y through the deep convolution module w1 , feature self-attention Y s and convolutional features Y w1 After adaptive adjustment by the first adaptive interaction module, the output features of the spatial window self-attention module are generated;

[0025] The input features of the adaptive channel self-attention module are divided into two paths, one of which is generated by the spatial self-attention module to generate the self-attention feature Y c , the other path generates convolution feature Y through the deep convolution modulew2 , feature self-attention Y c and convolutional features Y w2 After adaptive adjustment by the second adaptive interaction module, the output features of the channel self-attention module are generated;

[0026] The first adaptive interaction module performs self-attention feature Y s Perform spatial interaction processing and convolution feature Y w1 After channel interaction processing, the two are fused; the second adaptive interaction module is used to process the self-attention feature Y c Perform channel interaction processing and convolution feature Y w2 After performing spatial interaction processing, the two are fused; the spatial interaction processing is to generate a spatial attention map to perform spatial weighting on the other branch, and the channel interaction processing is to generate a channel attention map through global average pooling to perform channel reweighting;

[0027] The SGFN layer captures nonlinear spatial information by dividing the input features into two parts along the channel dimension, one part is convolved, and the other part is multiplied with the convolution output through a skip connection.

[0028] According to a technical solution of the present invention, in step S3, the following steps are specifically included:

[0029] The training set in step S1 is used to train the optical-SAR fusion target detection network, and the validation set in step S1 is used to observe the performance of optical-SAR fusion target detection under foggy conditions. A domain positive and negative sample allocation strategy based on aspect ratio is adopted, a cross entropy loss function is used as the fine-grained classification loss function, CIoU is used as the positioning loss function, a stochastic gradient descent optimization algorithm is used, a learning rate decay strategy is set, and the parameters of the optical-SAR fusion target detection network are updated until the network performance converges, thereby obtaining an optical-SAR fusion target detection model under foggy conditions.

[0030] According to a technical solution of the present invention, in step S4, the steps specifically include:

[0031] The test set in step S1 is input into the trained optical-SAR fusion target detection model to obtain numerical and visual detection results on the test set.

[0032] According to one aspect of the present invention, an electronic device includes: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device performs the above-mentioned optical-SAR fusion target detection method under cloud and fog conditions.

[0033] According to one aspect of the present invention, a computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the above-mentioned optical-SAR fusion target detection method under cloud and fog conditions.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] This paper proposes a method for optical-SAR fusion target detection in foggy conditions. By simulating a large number of random clouds, it helps build a robust model. This method uses a dual spatial-channel aggregation approach to perform feature fusion, aggregating spatial and channel features both within and between blocks. This results in a robust feature representation and comprehensively improves the performance of optical-SAR fusion target detection.

[0036] The present invention, by constructing a continuous dual-aggregation Transformer block, effectively solves the problem in the prior art that the Transformer module adopts the traditional multi-head attention mechanism to fuse features, cannot fully combine the spatial and channel attention mechanisms, and has weak feature expression ability of the basic network; by constructing a continuous dual-aggregation Transformer module, the features are alternately applied with spatial self-attention and channel self-attention in continuous Transformer blocks, thereby realizing feature aggregation between blocks and within blocks, improving the feature expression ability of the network and enhancing the deep feature extraction ability of the network.

[0037] This paper introduces the ability to fuse local convolutional information with global features in the traditional Transformer module through the design of SGFN and DATB. The SGFN module improves the ability to capture spatial information and handle channel redundancy, while the DATB module achieves stronger feature modeling capabilities in both spatial and channel dimensions through AS-SA and AC-SA. While maintaining computational efficiency, it enhances the model's expressiveness and generalization performance, comprehensively improving the performance of optical-SAR fusion target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0039] Figure 1 A flowchart of an optical-SAR fusion target detection method under foggy conditions according to one embodiment of the present invention is schematically shown;

[0040] Figure 2 A schematic diagram schematically illustrates an optical-SAR fusion target detection network provided according to one embodiment of the present invention;

[0041] Figure 3 Schematically showing a schematic diagram of a dual-aggregation Transformer module provided according to one embodiment of the present invention;

[0042] Figure 4 Schematically shows a schematic diagram of an adaptive interaction module provided in one embodiment of the present invention;

[0043] Figure 5 The figure schematically shows a schematic diagram of a spatial gated feedforward network module provided according to one embodiment of the present invention. DETAILED DESCRIPTION

[0044] The description of the embodiments in this specification should be combined with the corresponding drawings, which should be considered a complete part of this specification. In the drawings, the shapes and thicknesses of the embodiments may be exaggerated and indicated for simplicity or convenience. Furthermore, the various structural components in the drawings will be described separately. It is worth noting that components not shown in the drawings or not described in words are known to those of ordinary skill in the art.

[0045] The description of the embodiments herein and any references to directions and orientations are for ease of description only and are not to be construed as limiting the scope of the present invention. The following description of the preferred embodiments may involve combinations of features, which may exist independently or in combination. The present invention is not specifically limited to the preferred embodiments. The scope of the present invention is defined by the claims.

[0046] like Figure 1 As shown, the optical-SAR fusion target detection method under foggy conditions of the present invention includes the following steps:

[0047] Step S1: Perform simulated cloud and fog addition processing on the preset optical image to construct an optical-SAR fusion target detection dataset under cloud and fog conditions;

[0048] Step S2: constructing an optical-SAR fusion target detection model; the optical-SAR fusion target detection model includes an optical feature detector, a SAR feature detector, and a dual-aggregate Transformer module embedded between the optical feature detector and the SAR feature detector;

[0049] Step S3: training the optical-SAR fusion target detection model based on the data set constructed in step S1 to obtain a trained optical-SAR fusion target detection model;

[0050] Step S4: Utilize the trained optical-SAR fusion target detection model to perform fusion target detection.

[0051] In one embodiment of the present invention, step S1 specifically includes:

[0052] Construct target category system and sample category specification based on optical remote sensing image data and expert experience knowledge;

[0053] Based on the target category system and sample category specifications, the target location and category in the remote sensing image data are marked with horizontal boxes;

[0054] The annotated optical remote sensing image data is sliced ​​and the sliced ​​optical remote sensing image data is subjected to cloud and fog occlusion at different proportions to form an optical-SAR fusion target detection dataset under fog conditions; the slice size can be set to 256*256, and the overlap coefficient is set to 20%.

[0055] The optical-SAR fusion target detection dataset under foggy conditions is divided into training, validation, and test sets. The ratio of training, validation, and test sets can be set to 7:1:2.

[0056] According to one embodiment of the present invention, Figure 2 As shown, in step S2, the optical feature detector and the SAR feature detector are the same, both of which are YOLOv5 networks, including a backbone network, a feature pyramid network, and a detection network;

[0057] The backbone network is used to extract multi-scale feature maps of the input image. It consists of a Conv module, a C3 module, and an SPPF module. The Conv module mainly consists of a convolutional layer, a batch normalization layer, and an activation function. The C3 module contains three standard convolutional layers and multiple Bottleneck blocks. The Bottleneck block is a standard bottleneck structure used to improve model performance. The SPPF module is a spatial pyramid pooling module that generates multiple fixed-length feature vectors by performing pooling operations on the input feature map at different scales, thereby capturing contextual information at different scales.

[0058] The Feature Pyramid Network (FPN) is used to fuse high-level semantic information extracted by the backbone network with low-level positioning information. Through C3, upsample, and concatenation operations, the FPN fuses high-level feature maps with rich semantic information and low-level feature maps with positioning information, enhancing the model's multi-scale representation capabilities.

[0059] The detection network is used to detect three feature maps of different scales and obtain the target fine-grained classification and positioning results of the pixels on each feature map;

[0060] The dual-aggregate Transformer module consists of a spatial Transformer module and a channel Transformer module connected in series.

[0061] The original input residual features of the input spatial Transformer module are added element-by-element to the original input residual features of the spatial Transformer module after passing through the series normalization layer and adaptive spatial self-attention module (AS-SA). The original residual features of the addition result are added element-by-element to the other original residual features after passing through the series normalization layer and SGFN layer to form the output features of the spatial Transformer module;

[0062] The original input residual features of the input channel Transformer module are added element-by-element to the original input residual features of the channel Transformer module after passing through the series normalization layer and adaptive channel self-attention module (AC-SA). The original residual features of the addition result are added element-by-element to the other original residual feature output after passing through the series normalization layer and SGFN layer to form the output features of the channel Transformer module.

[0063] The input features of the adaptive spatial self-attention module (AS-SA) are divided into two paths. One path is passed through the spatial self-attention module (SW-SA) to generate the self-attention feature Y s , the other path generates convolution feature Y through the deep convolution module (DW-Conv) w1 , self-attention feature Y s and convolutional features Y w1 After adaptive adjustment by the first adaptive interaction module, the output features of the spatial window self-attention module are generated;

[0064] The input features of the adaptive channel self-attention module (AC-SA) are divided into two paths, one of which is generated by the spatial self-attention module (CW-SA) to generate the self-attention feature Y c , the other path generates convolution feature Y through the deep convolution module (DW-Conv) w2 , self-attention feature Y c and convolutional features Y w2 After adaptive adjustment by the second adaptive interaction module, the output features of the channel self-attention module are generated;

[0065] The first adaptive interaction module is used to self-attention feature Y s Perform spatial interaction processing and convolution feature Y w1 After channel interaction processing, the two are fused; the second adaptive interaction module processes the self-attention feature Y c Perform channel interaction processing and convolution feature Y w2After performing spatial interaction processing, the two are fused. Spatial interaction processing generates a spatial attention map to spatially weight the other branch, while channel interaction processing generates a channel attention map through global average pooling and performs channel reweighting. By fusing the features after channel interaction and spatial interaction processing, the effective combination of local and global features is achieved.

[0066] The implementation details of the SW-SA module are as follows:

[0067] For the input characteristics of SW-SA module The query matrix Q, key matrix K and value matrix V are generated by linear mapping. The formula is:

[0068] Q=XW Q ,K=XW K ,V=XW V , (1)

[0069] Among them, W Q ,W K , is the linear projection matrix.

[0070] Then the query matrix Q, key matrix K and value matrix V are divided into several non-overlapping local windows, each containing N w pixels, and flatten these windows. The flattened matrix is ​​Q s , K s and V s , whose dimensions are

[0071] Then, these matrices are divided into h attention heads: The dimension of each attention head is

[0072] Figure 3 The case where h = 1 is shown in the figure. The output of the i1th attention head Defined as:

[0073]

[0074] Where D represents the relative position code.

[0075] Finally, the final feature is obtained by reshaping and concatenating the outputs of all attention heads

[0076]

[0077] in is the linear projection matrix used to fuse all features.

[0078] In addition, to capture more spatial information, the design of the Swin Transformer is referenced and a sliding window operation is used as the default setting. This sliding window mechanism helps improve the model's spatial information modeling capabilities.

[0079] For the CW-SA module, the implementation details are:

[0080] Given an input X, generate a query matrix Q by linear projection c , key matrix K c Sum matrix V c , and reshape these matrices to have a size of The projection vector of .

[0081] Then, these projection vectors are divided into h attention heads, and each attention head performs self-attention calculation independently. To simplify the representation, Figure 3 Part (b) shows the case where h = 1. For the i2th attention head, the calculation process is:

[0082]

[0083] in, is the output of the i2th head, and α is a learnable temperature parameter used to adjust the scale of the inner product before softmax. Finally, the final channel self-attention feature is obtained by concatenating and reshaping the outputs of all heads This process is similar to Formula (3) in spatial window self-attention SW-SA, which ensures the effective aggregation of all channel information.

[0084] The implementation details of the adaptive interaction module AIM are as follows:

[0085] First, the value matrix V in the self-attention module is applied with parallel depth convolution (DW-Conv) to establish a direct connection between self-attention and convolution. The output after convolution is expressed as Then, AIM’s adaptive adjustment: Spatial Interaction (SI) and Channel Interaction (CI). SI calculates the spatial attention map S-Map of input B, with a size of CI is achieved by global average pooling H GP Calculate the channel attention map C-Map, the size is The calculation formulas for these two operations are as follows:

[0086]

[0087] Among them, f(·) is the Sigmoid function, σ(·) is the GELU function, W1, W2, W3, W4 are the weight matrices of point-by-point convolution, which are used to adjust the scaling of the channel dimension, and r1 and r2 are the scaling ratios of the convolution.

[0088] Two interactions are achieved through element-wise multiplication:

[0089]

[0090] The calculation formulas for the AS-SA module and AC-SA module in the dual-aggregation Transformer module are:

[0091]

[0092] where Y s ,Y c ,Y w Represent the outputs of SW-SA, CW-SA and DW-Conv respectively.

[0093] AS-SA (Adaptive Spatial Self-Attention) introduces channel-spatial interaction based on SW-SA to enhance the fusion between convolution and attention; AC-SA (Adaptive Channel Self-Attention) introduces spatial-channel interaction based on CW-SA to improve the output performance of the convolution branch.

[0094] The implementation details of the SGFN module are as follows:

[0095]

[0096] Among them, W d are the learnable parameters of the depthwise convolution, and The two parts after the feature is split along the channel are of size where C′ is the hidden dimension in SGFN.

[0097] Spatial window self-attention calculates attention within a local window, generates query, key, and value matrices, and extracts high-level features;

[0098] Channel self-attention performs self-attention calculation along the channel dimension, divides the input into multiple heads, applies the attention mechanism independently, and obtains the output of channel features;

[0099] The adaptive interaction module introduces locality between the self-attention module and the convolution branch, and improves the fusion effect of global and local features by adaptively adjusting features;

[0100] The spatial gate feedforward network introduces a spatial gate mechanism, combines deep convolution with element-by-element multiplication, captures nonlinear spatial information, reduces channel redundancy, and improves feature expression capabilities;

[0101] By alternately organizing dual-space Transformer blocks and dual-channel Transformer blocks, feature aggregation across spatial and channel dimensions is achieved; further, by combining dual-aggregation Transformer blocks with adaptive interaction modules, multi-level features are fused to obtain fused features.

[0102] According to one embodiment of the present invention, step S3 specifically includes:

[0103] The optical-SAR fusion target detection network was trained using the training set from step S1. The performance of the optical-SAR fusion target detection under foggy conditions was observed using the validation set from step S1. Based on the set positive and negative sample allocation strategy and localization loss function, the parameters of the optical-SAR fusion target detection network were updated using a stochastic gradient descent optimization algorithm with a learning rate decay strategy until network performance converged, resulting in the optical-SAR fusion target detection model under foggy conditions. The cross-entropy loss function was used as the fine-grained classification loss function, and the CIoU loss function was used as the localization loss function. The backbone network was initialized using the CSPDarknet-53 pre-trained on the COCO dataset. Stochastic gradient descent was used for 200 epochs with a batch size of 64 and an initial learning rate of 0.01. The parameters of the optical-SAR fusion target detection network under foggy conditions were updated until network performance converged, resulting in the optical-SAR fusion target detection model under foggy conditions. The intersection-over-union ratio threshold between target boxes in the post-processing algorithm was set to 0.5 to filter out redundant boxes.

[0104] According to one embodiment of the present invention, in step S4, it specifically includes:

[0105] The test set from step S1 is fed into the trained optical-SAR fusion target detection model to obtain numerical and visual detection results on the test set. The mean average precision (mAP) is used as the evaluation metric to compare the numerical results of the base network with the dual-aggregated Transformer-based optical-SAR target detection method. The visual results of the base network with the dual-aggregated Transformer-based optical-SAR target detection method are also compared.

[0106] This paper, through the design of SGFN and DATB, integrates the ability to fuse local convolutional information with global features within the traditional Transformer. SGFN improves spatial information capture and handles channel redundancy, while DATB achieves stronger feature modeling capabilities in both spatial and channel dimensions through AS-SA and AC-SA. This design enhances the model's expressiveness and generalization while maintaining computational efficiency, comprehensively improving the performance of optical-SAR fusion target detection.

[0107] According to one aspect of the present invention, an electronic device is provided, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory; when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device performs an optical-SAR fusion target detection method under cloud and fog conditions as described in any of the above technical solutions.

[0108] The processor may be a central processing unit (CPU), other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0109] According to one aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions. When the computer instructions are executed by a processor, an optical-SAR fusion target detection method under cloud and fog conditions is implemented as described in any one of the above technical solutions.

[0110] Computer-readable storage media may include any medium capable of storing or transmitting information. Examples of computer-readable storage media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and the like. The code segments may be downloaded via a computer network such as the Internet, an intranet, and the like.

[0111] Furthermore, it should be noted that the present invention may be provided as a method, apparatus, or computer program product. Thus, embodiments of the present invention may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0112] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0114] It should also be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal device comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal device comprising the element.

[0115] Finally, it should be noted that the above is a preferred embodiment of the present invention. It should be noted that although the preferred embodiment of the present invention has been described, it is clear that those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles of the present invention. Such improvements and modifications should also be considered as within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. An optical-SAR fusion target detection method under foggy conditions, characterized in that: The following steps are involved: Step S1: Perform simulated cloud and fog addition processing on the preset optical image to construct an optical-SAR fusion target detection dataset under cloud and fog conditions; Step S2: constructing an optical-SAR fusion target detection basic model; the optical-SAR fusion target detection basic model includes an optical feature detector and a SAR feature detector, and a dual-aggregation Transformer module embedded between the optical feature detector and the SAR feature detector; Step S3: training the optical-SAR fusion target detection model based on the data set constructed in step S1 to obtain a trained optical-SAR fusion target detection model; Step S4: Utilize the trained optical-SAR fusion target detection model to perform fusion target detection.

2. The optical-SAR fusion target detection method under foggy conditions according to claim 1, characterized in that: In the step S1, it specifically includes: Construct target category system and sample category specification based on optical remote sensing image data and expert experience knowledge; Based on the target category system and the sample category specification, horizontal box annotation is performed on the target position and category in the remote sensing image data; Slicing the annotated optical remote sensing image data, and performing cloud and fog occlusion of different proportions on the sliced ​​optical remote sensing image data to form an optical-SAR fusion target detection dataset under the cloud and fog conditions; The optical-SAR fusion target detection dataset under cloud and fog conditions is divided into a training set, a validation set and a test set.

3. The optical-SAR fusion target detection method under foggy conditions according to claim 2, characterized in that: In step S2, the optical feature detector and the SAR feature detector are the same, both being YOLOv5 networks, including a backbone network, a feature pyramid network, and a detection network; The backbone network is used to extract a multi-scale feature map of the input image; The feature pyramid network is used to fuse the high-level semantic information and low-level positioning information extracted by the backbone network; The detection network is used to detect the fused multi-scale feature map to obtain the target fine-grained classification and positioning results of the pixel points on each feature map; The dual-aggregation Transformer module includes a spatial Transformer module and a channel Transformer module connected in series; The original input residual features input to the spatial Transformer module are divided into two paths. One path is normalized in series and then subjected to an adaptive spatial self-attention module, and then added element-by-element to the other path of original input residual features. The resulting original residual features are normalized in series and subjected to an SGFN layer, and then added element-by-element to the other path of original residual features to form the spatial Transformer module output features for input into the channel Transformer module. The original input residual features of the channel Transformer module are divided into two paths. One path is normalized in series and then passes through the adaptive channel self-attention module, and then is added element-by-element to the other path of original input residual features. The resulting original residual features are normalized in series and passed through the SGFN layer, and then added element-by-element to the other path of original residual features to form the output features of the channel Transformer module. The input features of the adaptive spatial self-attention module are divided into two paths, one of which is generated by the spatial self-attention module to generate the self-attention feature Y s , the other path generates convolution feature Y through the deep convolution module w1 , self-attention feature Y s and convolutional features Y w1 After adaptive adjustment by the first adaptive interaction module, the output features of the spatial window self-attention module are generated; The input features of the adaptive channel self-attention module are divided into two paths, one of which is generated by the spatial self-attention module to generate the self-attention feature Y c , the other path generates convolution feature Y through the deep convolution module w2 , self-attention feature Y c and convolutional features Y w2 After adaptive adjustment by the second adaptive interaction module, the output features of the channel self-attention module are generated; The first adaptive interaction module performs self-attention feature Y s Perform spatial interaction processing and convolution feature Y w1 After channel interaction processing, the two are fused; the second adaptive interaction module is used to process the self-attention feature Y c Perform channel interaction processing and convolution feature Y w2 After performing spatial interaction processing, the two are fused; the spatial interaction processing is to generate a spatial attention map to perform spatial weighting on the other branch, and the channel interaction processing is to generate a channel attention map through global average pooling to perform channel reweighting; The SGFN layer captures nonlinear spatial information by dividing the input features into two parts along the channel dimension, one part is convolved, and the other part is multiplied with the convolution output through a skip connection.

4. The optical-SAR fusion target detection method under foggy conditions according to claim 3, characterized in that: In the step S3, it specifically includes: The training set in step S1 is used to train the optical-SAR fusion target detection network, and the validation set in step S1 is used to observe the performance of optical-SAR fusion target detection under foggy conditions. A domain positive and negative sample allocation strategy based on aspect ratio is adopted, a cross entropy loss function is used as the fine-grained classification loss function, CIoU is used as the positioning loss function, a stochastic gradient descent optimization algorithm is used, a learning rate decay strategy is set, and the parameters of the optical-SAR fusion target detection network are updated until the network performance converges, thereby obtaining an optical-SAR fusion target detection model under foggy conditions.

5. The optical-SAR fusion target detection method under foggy conditions according to claim 4, characterized in that: In the step S4, it specifically includes: The test set in step S1 is input into the trained optical-SAR fusion target detection model to obtain numerical and visual detection results on the test set.

6. An electronic device, characterized in that: include: One or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device performs the optical-SAR fusion target detection method under cloud and fog conditions as described in any one of claims 1 to 6.

7. A computer-readable storage medium, characterized in that Used to store computer instructions, which, when executed by a processor, implement the optical-SAR fusion target detection method under cloud and fog conditions as described in any one of claims 1 to 6.

Images