SAR image-oriented frozen optical basis model dual-domain residual compensation method and system

Abstract

The application discloses a frozen optical base model double-domain residual compensation method and system for SAR images, and relates to the technical field of cross-modal computer vision. The method freezes the input parameters of the SAR image and extracts the features of the optical base model, and obtains a to-be-adapted feature map at a compensation position; the to-be-adapted feature map is subjected to frequency decomposition, the low-frequency components are kept unchanged and are subjected to selective modulation for reconstruction to obtain a reconstructed feature; a spatial gating map corresponding to the spatial position of the to-be-adapted feature map is generated according to the reconstructed feature, and then spatial modulation is performed to form a noise domain residual, and a purified feature map is obtained through residual mixing; the purified feature map is subjected to linear transformation, and then position-by-position modulation is performed to form a semantic domain residual, and an adapted feature map is obtained through residual mixing and is transmitted to a subsequent feature extraction unit. The application does not need to modify the frozen backbone parameters, can compensate for the noise domain deviation and semantic domain deviation of the SAR features under spatial cooperation, has low adaptation cost, and can be reused for different SAR downstream tasks.

Description

Technical Field

[0001] This application relates to the fields of computer vision and radar image feature processing technology, and in particular to a dual-domain residual compensation method and system for frozen optical fundamental models of SAR images. Background Technology

[0002] The optical foundation model, pre-trained on large-scale optical images, can form a feature space with strong geometric representation and semantic expression capabilities. Transferring this type of model to Synthetic Aperture Radar (SAR) image processing tasks is beneficial for reusing existing visual priors and reducing the cost of redesigning and training large-scale backbone networks for non-optical modalities.

[0003] However, SAR images are formed by active microwave imaging, and their pixel intensity and spatial texture are affected by factors such as coherent imaging, target scattering mechanisms, observation angle, and multiplicative speckle, resulting in significant differences in appearance statistics compared to optical images formed based on reflected light. When SAR images are directly input into the optical fundamental model with frozen parameters, the features within the model exhibit systematic deviations relative to the features of the optical image, leading to a decrease in the quality of the transferred features and consequently affecting the performance of downstream SAR image tasks.

[0004] To address the aforementioned biases, existing technologies primarily employ the following approaches: First, preprocessing the SAR image at the input end, such as speckle removal. However, this processing only applies to the pixel domain and struggles to correct internal feature biases already formed in each layer of the frozen optical foundation model. Second, designing task-specific backbone networks, prediction heads, or loss functions. However, this approach requires altering the downstream model structure and cannot create a cross-modal adaptation mechanism for the optical foundation model applicable to different SAR tasks. Third, reducing the differences between the optical and SAR domains from the perspectives of data generation, global distribution alignment, or cue learning. However, this approach is insufficient in handling biases with spatial locality within features, resulting in background and effective scattering regions in the features receiving approximately identical corrections, thus limiting adaptation accuracy. Fourth, fine-tuning all parameters of the frozen backbone. However, this increases storage and training costs and may impair the universality of pre-trained features.

[0005] Therefore, how to effectively compensate for the deviations caused by modal differences in the internal features of the frozen optical basic model without modifying the original parameters of the model is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] The purpose of this application is to provide a dual-domain residual compensation method and system for frozen optical basic models of SAR images. It achieves compensation for noise domain bias and semantic domain bias of SAR features under spatial coordination without modifying the frozen backbone parameters. It has low adaptation cost and can be reused for different downstream SAR tasks.

[0007] To achieve the above objectives, this application provides the following solution:

[0008] Firstly, this application provides a dual-domain residual compensation method for a frozen optical fundamental model of SAR images, including:

[0009] Acquire synthetic aperture radar (SAR) images, input the SAR images into the feature extraction backbone of the optical basic model, keep the parameters of the feature extraction backbone frozen, and obtain the feature map to be adapted at at least one compensation position of the feature extraction backbone.

[0010] The feature map to be adapted is decomposed by frequency to obtain low-frequency components and multiple high-frequency components. The low-frequency components are kept unchanged, and the multiple high-frequency components are selectively modulated and reconstructed to obtain the reconstructed features.

[0011] A spatial gating map corresponding to the spatial position of the feature map to be adapted is generated based on the reconstructed features. The reconstructed features are spatially modulated using the spatial gating map to form a noise domain residual. The noise domain residual is mixed with the feature map to be adapted to obtain a cleaned feature map.

[0012] The purification feature map is expanded into a feature label sequence according to its spatial location and linearly transformed. The feature labels after linear transformation are modulated position by position using the spatial gating map to form a semantic domain residual. The semantic domain residual is mixed with the feature label sequence to obtain the fitting feature map.

[0013] The adapted feature map is passed as the output of the corresponding compensation position to the subsequent feature extraction unit of the feature extraction backbone.

[0014] In one embodiment, the SAR image is a single-channel SAR image, and the single-channel SAR image is copied by channel and then input into the feature extraction backbone;

[0015] The compensation location is provided with a dual-domain residual compensation unit, which is located after one or more feature extraction blocks of the feature extraction backbone. The dual-domain residual compensation unit includes a frequency-selective cleanup adaptation branch for forming the noise domain residual and the spatial gating graph, and a gated semantic residual modulation branch for reusing the spatial gating graph and forming the semantic domain residual.

[0016] In one embodiment, frequency decomposition is performed on the feature map to be adapted, including:

[0017] While keeping the spatial dimensions unchanged, the feature map to be adapted is projected onto the dimension-reduced channel space, and a single-layer discrete wavelet transform is performed on the obtained dimension-reduced features to obtain low-frequency sub-bands as the low-frequency components and multiple high-frequency detail sub-bands as the multiple high-frequency components.

[0018] The multiple high-frequency detail subbands are spliced ​​together along the channel dimension in a preset direction to form a high-frequency joint feature, and the high-frequency joint feature is modulated in the joint space formed by the multiple high-frequency detail subbands and the feature channel;

[0019] The modulation result is restored to the corresponding high-frequency detail subband, and the restored high-frequency detail subband and the unchanged low-frequency subband are used as inputs to perform inverse discrete wavelet transform to obtain the reconstructed features.

[0020] In one embodiment, modulating the high-frequency joint features includes:

[0021] Local feature extraction is performed on the high-frequency joint features to obtain high-frequency responses that integrate information from different high-frequency detail subbands and different feature channels. Channel modulation weights are generated based on the global average pooling and global max pooling results of the high-frequency responses. Channel-weighted summation is then performed on the high-frequency responses based on the channel modulation weights to obtain the modulated high-frequency joint features.

[0022] In one embodiment, generating a spatial gating map based on the reconstructed features includes:

[0023] Spatial feature extraction is performed on the reconstructed features to obtain gated intermediate features. The channel mean, channel maximum, and channel mean of the reconstructed features are calculated to obtain three single-channel spatial summaries. The three single-channel spatial summaries are fused and mapped to the spatial gating map.

[0024] Each gate value in the spatial gating diagram corresponds one-to-one with each spatial position in the feature diagram to be adapted, and all of them are not less than a preset lower limit, which is greater than zero.

[0025] In one embodiment, the reconstructed features are modulated position-by-position using the spatial gating map, and the modulation result is projected onto the same channel dimension as the feature map to be adapted, forming the noise domain residual;

[0026] Based on a trainable first residual scale parameter, the residual mixing is performed on the noise domain residual and the feature map to be adapted. The initial value of the first residual scale parameter makes the frequency selective cleanup adaptation branch close to an identity mapping in the early stage of training.

[0027] In one embodiment, the linear transformation is achieved through amplitude-direction projection mapping, which uses a fixed identity matrix as the identity mapping, forms a low-rank update of the identity mapping through a trainable low-rank factor, performs row-by-row normalization on the updated mapping to determine the direction, and controls the amplitude of each output channel through a trainable positive amplitude vector.

[0028] In one embodiment, the spatial gating map is unfolded in the same spatial unfolding order as the sequence of generated feature markers, and the unfolding result is channel-projected to obtain a position modulation factor that varies with a value of 1 as the center. Each position modulation factor corresponds one-to-one with the corresponding feature marker, and the position modulation factor is used to perform the position-by-position modulation on the linearly transformed feature marker.

[0029] Based on a trainable second residual scale parameter, the semantic domain residual and the feature label sequence are subjected to residual mixing. The initial value of the second residual scale parameter makes the gated semantic residual modulation branch approach an identity mapping in the early stage of training.

[0030] In one embodiment, the original parameters of the feature extraction backbone are kept unchanged during training, and the parameters of the dual-domain residual compensation unit are updated based on the training loss of the downstream task of SAR image.

[0031] Initial values ​​are set for the low-rank factor used to form low-rank updates, the first residual scale parameter used for noise domain residual mixing, and the second residual scale parameter used for semantic domain residual mixing, so that the dual-domain residual compensation unit approaches an identity mapping in the early stage of training.

[0032] Secondly, this application provides a dual-domain residual compensation system for a frozen optical fundamental model of SAR images, comprising:

[0033] The frozen feature acquisition module is used to input the SAR image into the feature extraction backbone of the optical basic model, keep the parameters of the feature extraction backbone frozen, and obtain the feature map to be adapted at at least one compensation position of the feature extraction backbone.

[0034] The noise domain residual compensation module is used to perform frequency decomposition on the feature map to be adapted, to obtain low-frequency components and multiple high-frequency components, keep the low-frequency components unchanged, selectively modulate the multiple high-frequency components and reconstruct them, generate a spatial gating map corresponding to the spatial position of the feature map to be adapted based on the obtained reconstructed features, use the spatial gating map to spatially modulate the reconstructed features to form a noise domain residual, and perform residual mixing between the noise domain residual and the feature map to be adapted to obtain a cleaned feature map;

[0035] The semantic domain residual compensation module is used to expand the purified feature map into a feature label sequence according to the spatial location and perform a linear transformation, use the spatial gating map to modulate the linearly transformed feature labels position by position to form a semantic domain residual, and perform residual mixing between the semantic domain residual and the feature label sequence to obtain an adapted feature map.

[0036] The adaptation feature transfer module is used to transfer the adaptation feature map as the output of the corresponding compensation position to the subsequent feature extraction unit of the feature extraction backbone.

[0037] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0038] This application provides a dual-domain residual compensation method and system for frozen optical fundamental models of SAR images. By keeping low-frequency components unchanged after frequency decomposition and selectively modulating only high-frequency components, the compensation acts on local high-frequency perturbation components in the features without destroying the still transferable coarse-grained spatial structure in the optical fundamental model. This achieves targeted correction of feature deviations caused by SAR speckle and clutter. By generating spatial gating maps from the frequency-reconstructed features, the gating values ​​correspond one-to-one with the spatial locations of the feature maps to be adapted. This concentrates the noise domain residuals at the spatial locations requiring correction, preventing the background region and the effective scattering region from receiving consistent correction, thus improving the limited adaptability. The efficiency of capacity utilization is improved. By using the same spatial gating map for position-wise modulation multiplexing of semantic domain residuals, noise domain compensation and semantic domain compensation are kept consistent in spatial location, avoiding inconsistent spatial weights learned by the two types of compensation. This allows for the simultaneous correction of spatial offsets in deep semantic responses while purifying local high-frequency disturbances. By superimposing both noise domain residuals and semantic domain residuals onto the original features in a residual mixing manner, the compensation process does not modify the original parameters of the frozen optical basic model. The adapted feature map can be directly passed to the subsequent feature extraction unit, resulting in low training and storage costs. It also retains the universality of pre-trained features, and the obtained cross-modal adapted features can be reused for different downstream tasks of SAR images. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is an application environment diagram of the dual-domain residual compensation method for the frozen optical basic model of SAR images in the embodiments of this application;

[0041] Figure 2This is a flowchart illustrating the dual-domain residual compensation method for the frozen optical fundamental model of SAR images in the embodiments of this application.

[0042] Figure 3 This is a schematic diagram showing the connection relationship between the dual-domain residual compensation unit and the frozen optical basic model in the embodiments of this application;

[0043] Figure 4 This is a schematic diagram of the structure of the frequency-selective wavelet purification adaptation branch in the embodiments of this application;

[0044] Figure 5 This is a schematic diagram of the spatial gating map generation process in an embodiment of this application;

[0045] Figure 6 This is a schematic diagram of the structure of the gated guided semantic residual modulation branch in the embodiments of this application;

[0046] Figure 7 This is a block diagram of the dual-domain residual compensation system for the frozen optical basic model of SAR images in the embodiments of this application. Detailed Implementation

[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] This application provides a dual-domain residual compensation method for a frozen optical fundamental model of SAR images, which can be applied to, for example... Figure 1 The application environment is illustrated. The terminal is used to acquire or receive SAR images, such as a satellite ground station terminal equipped with a SAR payload, a UAV ground control terminal, or a computer with SAR image processing capabilities. The terminal transmits the SAR images to a server via a network. The server deploys a parameter-frozen optical fundamental model and a dual-domain residual compensation unit to perform dual-domain residual compensation on the SAR images, obtaining cross-modal adaptation features. Further downstream tasks based on these cross-modal adaptation features can be performed. The data storage system stores the SAR images, optical fundamental model parameters, dual-domain residual compensation unit parameters, and cross-modal adaptation features. The data flow is as follows: the terminal sends the SAR images to the server via the network; after completing dual-domain residual compensation, the server returns the cross-modal adaptation features or downstream task results to the terminal, or writes the relevant data to the data storage system.

[0049] In other embodiments, the above-described dual-domain residual compensation method can also be executed entirely locally on the terminal, and this application does not limit this.

[0050] Example 1

[0051] like Figure 2 As shown, a dual-domain residual compensation method for frozen optical fundamental models of SAR images is provided. This method is executed by a computer device and includes the following steps:

[0052] S1, freeze the optical backbone to extract SAR features to be adapted;

[0053] S2, wavelet decomposition, high-frequency joint modulation and feature reconstruction;

[0054] S3, spatial gated guided noise domain residual compensation;

[0055] S4, Purification Feature Expansion and Amplitude-Direction Projection;

[0056] S5, shared space gating guided semantic domain residual compensation;

[0057] S6 transmits and outputs cross-modal adaptation features.

[0058] The following detailed explanation will illustrate the implementation process of the above steps.

[0059] In the specific implementation of step S1 above, a SAR image is acquired, the SAR image is input into the feature extraction backbone of the optical basic model, the parameters of the feature extraction backbone are kept frozen, and the feature map to be adapted is obtained at at least one compensation position of the feature extraction backbone.

[0060] In this embodiment, the SAR image is a single-channel SAR amplitude image. The same channel of the single-channel SAR amplitude image is copied three times to form a three-channel input, which is then input into the feature extraction backbone of the optical fundamental model to match the input interface of the optical fundamental model. The above channel copying process does not introduce additional trainable input transformation branches, and cross-modal adaptation is mainly completed in the internal feature space of the optical fundamental model.

[0061] like Figure 3 As shown, in this embodiment, the optical basic model feature extraction backbone uses a pre-trained DINOv3-ConvNeXt-Base, which includes four stages. The number of feature extraction blocks in each stage is 3, 3, 27, and 3 respectively, and the original parameters of the backbone remain unchanged during training. Dual-domain residual compensation units are placed after the last feature extraction block in the first, second, and fourth stages, and one unit is placed every three feature extraction blocks in the third stage, for a total of 12 dual-domain residual compensation units. This arrangement ensures that the compensation positions cover features from shallow to deep layers, while configuring more compensation positions in the third stage, which has a larger number of feature extraction blocks and higher semantic level.

[0062] Optionally, the compensation position and number can be adjusted according to the number of stages of the optical basic model, the scale of the feature map, and the computational resources, but it should be ensured that the dual-domain residual compensation unit receives the two-dimensional feature map to be adapted and returns the adapted feature map to the corresponding feature stream.

[0063] A dual-domain residual compensation unit is provided at the compensation location. This unit includes a frequency-selective purification adaptation branch and a gated-guided semantic residual modulation branch. The frequency-selective purification adaptation branch is used to form the noise domain residual and the spatial gating map, while the gated-guided semantic residual modulation branch is used to multiplex the spatial gating map and form the semantic domain residual. In this embodiment, the frequency-selective purification adaptation branch implements frequency decomposition and high-frequency selective modulation based on discrete wavelet transform, hereinafter referred to as the frequency-selective wavelet purification adaptation branch.

[0064] Let X be the feature map to be adapted received by any dual-domain residual compensation unit. X belongs to a C×H×W dimensional feature space, where C represents the number of channels, and H and W represent the height and width of the feature map, respectively.

[0065] The frequency-selective wavelet purification and adaptation branch receives X and outputs a purified feature map and a spatially gated map G. The gated semantic residual modulation branch receives the purified feature map and the spatially gated map G and outputs an adaptation feature map.

[0066] At any compensation location, the overall compensation relationship of the dual-domain residual compensation unit satisfies:

[0067] ;

[0068] In the formula, To adapt the feature map, For the feature map to be adapted, For noise domain residuals, For spatial gating diagrams, This represents the semantic domain residual. In the following implementation, X represents the feature map F to be adapted, and F represents the feature map F to be adapted. GSRM Represents the adaptation feature map .

[0069] In the specific implementation of step S2 above, frequency decomposition is performed on the feature map to be adapted to obtain low-frequency components and multiple high-frequency components. The low-frequency components are kept unchanged, and the multiple high-frequency components are selectively modulated and reconstructed to obtain the reconstructed features. Among them, the low-frequency components are low-frequency sub-bands obtained by discrete wavelet transform, and the multiple high-frequency components are multiple high-frequency detail sub-bands obtained by discrete wavelet transform. The selective modulation is performed in the joint space composed of multiple high-frequency detail sub-bands and feature channels.

[0070] like Figure 4 As shown, step S2 specifically includes steps S21 to S25:

[0071] S21, while keeping the spatial dimensions unchanged, project the feature map to be adapted onto the dimension-reduced channel space to obtain the dimension-reduced features.

[0072] Specifically, the feature map X to be adapted is sequentially subjected to 1×1 convolution, batch normalization, and SiLU activation, projecting X into a dimensionality-reduced feature Z. Let the number of channels of Z be R, and in this embodiment, R=C / 4, thereby reducing the computational cost of the frequency-selective wavelet purification adaptation branch while keeping the H×W spatial grid unchanged.

[0073] S22 performs a single-level discrete wavelet transform on the dimension-reduced features to obtain low-frequency subbands and multiple high-frequency detail subbands.

[0074] Specifically, the discrete wavelet transform uses a single-layer Haar discrete wavelet transform, satisfying:

[0075] ;

[0076] In the formula, Z LL Z represents the low-frequency subband. LH Z HL and Z HH These represent the high-frequency detail subbands in three directions, DWT Haar This represents the Haar discrete wavelet transform.

[0077] Each sub-band has a spatial dimension of H / 2 × W / 2 and a channel count of R. The low-frequency sub-band Z... LL It retains a relatively coarse-grained scene layout, which remains unchanged during high-frequency detail modulation.

[0078] S23, multiple high-frequency detail subbands are spliced ​​together along the channel dimension in a preset direction to form a high-frequency joint feature.

[0079] Specifically, the high-frequency joint feature Z H satisfy:

[0080] ;

[0081] In the formula, Concat() represents the concatenation operation along the channel dimension. H It belongs to the 3R×H / 2×W / 2-dimensional feature space, and its directional sub-band dimension and the reduced feature channel dimension together form the sub-band-channel joint space.

[0082] S24, perform local feature extraction, generate channel modulation weights and channel weighting on the high-frequency joint features in sequence to obtain the modulated high-frequency joint features.

[0083] Specifically, for the high-frequency joint feature Z H Local feature extraction is performed to obtain the high-frequency response F, which integrates information from different high-frequency detail subbands and different feature channels.H ,satisfy:

[0084] ;

[0085] In the formula, DSConv(·) represents depthwise separable convolution, BN represents batch normalization, and SiLU represents the SiLU activation function.

[0086] Then, based on the high-frequency response F H The global average pooling result and the global max pooling result generate the channel modulation weight A. freq ,satisfy:

[0087] ;

[0088] In the formula, GAP(·) represents global average pooling, GMP(·) represents global max pooling, MLP(·) represents multilayer perceptron, sigmoid(·) represents sigmoid mapping, and [·;·] represents concatenation operation.

[0089] In this embodiment, the multilayer perceptron is implemented by projecting 6R input channels to hidden channels, performing GELU activation, and projecting back to 3R channels.

[0090] Finally, based on the channel modulation weight A freq For high frequency response F H Channel weighting is performed to obtain the modulated high-frequency joint features. ,satisfy:

[0091] ;

[0092] In the formula, ⊙ represents element-wise multiplication.

[0093] S25, the modulated high-frequency joint features are restored into three R-channel high-frequency detail subbands according to the preset direction order before splicing, and the restored high-frequency detail subbands and the unchanged low-frequency subbands are used as input to perform inverse discrete wavelet transform to obtain the reconstructed features.

[0094] Specifically, reconstructing feature Z recon satisfy:

[0095] ;

[0096] In the formula, IDWT Haar This represents the Haar inverse discrete wavelet transform.

[0097] Single-layer Haar wavelet transform can separate coarse-grained layout from local high-frequency variations with low computational cost; it preserves the low-frequency subband Z. LLRemaining unchanged can reduce disturbances to the transferable spatial structure; modulating only the high-frequency detail subbands allows the frequency-selective wavelet purification adaptation branch to perform residual correction on the original features, rather than replacing the original features entirely.

[0098] In the specific implementation of step S3 above, based on the reconstructed feature Z... recon A spatial gating map corresponding to the spatial location of the feature map X to be adapted is generated. The reconstructed features are spatially modulated using the spatial gating map to form a noise domain residual. The noise domain residual is mixed with the residual of the feature map to be adapted to obtain a cleaned feature map.

[0099] Step S3 specifically includes steps S31 to S34:

[0100] S31, spatial feature extraction is performed on the reconstructed features to obtain gated intermediate features (such as...). Figure 5 (As shown).

[0101] Specifically, the gated intermediate feature Y satisfies:

[0102] ;

[0103] In the formula, Conv 1×1 DWConv represents a 1×1 convolution. d=2 This represents a depthwise convolution with an inflation rate of 2.

[0104] S32, calculate the channel mean of the gated intermediate feature, the channel maximum value of the gated intermediate feature, and the channel mean of the reconstructed feature respectively to obtain three single-channel spatial summaries. The three single-channel spatial summaries are then fused and mapped into a spatial gating graph.

[0105] like Figure 5 As shown, specifically, the spatial gating graph G satisfies:

[0106] ;

[0107] In the formula, Avg c This indicates that the mean is calculated along the channel dimension, Max. c Conv represents taking the maximum value along the channel dimension. 7×7 This represents a 7×7 convolution, where clamp() denotes numerical truncation, and ε is a preset lower bound.

[0108] Each gating value in the spatial gating map G corresponds one-to-one with each spatial position in the feature map X to be adapted, and all values ​​are not less than a preset lower limit ε, which is greater than zero. In this embodiment, ε is set to 0.1, meaning that the values ​​of the spatial gating map G are restricted to the range of 0.1 to 1.

[0109] The preset lower bound can prevent the spatial gating graph from decaying completely to zero, thereby preserving a weak but effective scattering response and maintaining the gradient that is passed to the subsequent gating-guided semantic residual modulation branch.

[0110] Since the spatial gating map G is generated after wavelet reconstruction back to the original H×W spatial grid, each position of the spatial gating map G corresponds to the corresponding spatial position of the feature map X to be adapted.

[0111] S33, using a spatial gating map, modulates the reconstructed features position-by-position, and projects the modulation result onto the same channel dimension as the feature map to be adapted, forming a noise domain residual (e.g. Figure 4 (As shown).

[0112] Specifically, the noise domain residual Δ noise(X) satisfy:

[0113] ;

[0114] S34. Based on the trainable first residual scale parameter, residual mixing is performed on the noise domain residual and the feature map to be adapted to obtain the purified feature map output by the frequency-selective wavelet purification adaptation branch.

[0115] Specifically, purification feature map F (FS-WPA)(X) satisfy:

[0116] ;

[0117] In the formula, This represents the first residual scale parameter of the l-th dual-domain residual compensation unit.

[0118] In this embodiment, Initialize to make The near-zero value allows the frequency-selective wavelet purification adaptation branch to approach an identity mapping in the early stages of training, and gradually learn the required noise domain correction strength during the training process.

[0119] In the specific implementation of step S4 above, the purification feature map is expanded into a feature label sequence according to its spatial location and then subjected to a linear transformation. In this embodiment, the linear transformation is achieved through amplitude-direction projection mapping.

[0120] Specifically, such as Figure 6 As shown, the purification feature map F FS-WPA Expanded into a feature label sequence X according to a fixed order corresponding to the spatial positions of H×W. tok ,satisfy:

[0121] ;

[0122] In the formula, T represents the operation of unfolding a two-dimensional feature map into a sequence according to a fixed spatial order, and X...tok It belongs to an N×C dimensional feature space, where N is the number of feature labels.

[0123] The gated semantic residual modulation branch adds a label-by-label linear mapping to the feature label sequence without modifying the convolution kernel in the frozen backbone.

[0124] The amplitude-direction projection mapping uses a fixed identity matrix as the identity mapping. A low-rank update of the identity mapping is formed through a trainable low-rank factor. The updated mapping is then normalized row-by-row to determine the direction. The amplitude of each output channel is controlled by a trainable positive amplitude vector, satisfying the following conditions:

[0125] ;

[0126] ;

[0127] ;

[0128] ;

[0129] ;

[0130] In the formula, V0 is a fixed identity mapping, and I C Let A be a C-order identity matrix; B and A are trainable low-rank factors, where B belongs to a C×r dimensional parameter space and A belongs to an r×C dimensional parameter space, where r is the low-rank order and a is the low-rank update scaling factor; ΔV is the low-rank update, and V is the updated mapping; V hat For direction mapping, ||V|| row This indicates that the L2 norm is calculated for each row of V; m is a positive magnitude vector, m raw To train the original magnitude vector, the softplus mapping keeps m positive, with an upper limit of 5.0 to suppress excessive channel amplification; W MDP The weights are for the magnitude-direction projection mapping.

[0131] When the low-rank update approaches zero, the orientation mapping remains close to the fixed identity mapping, thereby reducing the initial perturbation to the frozen optical features.

[0132] In the specific implementation of step S5 above, the spatial gating graph is used to modulate the feature labels after linear transformation position by position to form semantic domain residuals. The semantic domain residuals are then mixed with the feature label sequence to obtain the fitting feature map.

[0133] Specifically, the spatial gating map is unfolded in the same spatial unfolding order as the generated feature label sequence, and the unfolding result is channel-projected to obtain the position modulation factor that varies with the value 1 as the center. Each position modulation factor corresponds one-to-one with the corresponding feature label.

[0134] The single-channel spatial gating map G is arranged according to the feature label sequence X. tok The same spatial unfolding order is converted into a gating mark G. tok ,satisfy:

[0135] ;

[0136] In the formula, T g G represents the same spatial unfolding order as T. tok It belongs to an N×1 dimensional feature space.

[0137] Gating mark G tok Perform channel projection and generate a position modulation factor M that varies around the value 1 using the hyperbolic tangent function, satisfying:

[0138] ;

[0139] In the formula, tanh(·) represents the hyperbolic tangent function, Linear(·) represents the linear mapping, and γ is the position modulation amplitude coefficient. In this embodiment, γ=0.1, so the elements of the position modulation factor M are between 0.9 and 1.1.

[0140] The output of the gated projection is close to zero during initialization, thus gradually introducing position-selective modulation during training. Because T g With the generation of feature label sequence X tok The same spatial unfolding order is used, and each row in M ​​is related to X. tok The corresponding feature labels in the data maintain a one-to-one spatial correspondence.

[0141] Using weight W MDP For the feature-labeled sequence X tok Perform a label-by-label linear transformation to obtain the linearly transformed feature label X. tilde And the semantic domain residual Δ is formed using the position modulation factor M. sem ,satisfy:

[0142] ;

[0143] ;

[0144] Based on the trainable second residual scale parameter, the semantic domain residual Δ sem and feature label sequence X tok Perform residual mixing and restore the C×H×W dimensional adaptation feature map F according to the original spatial order. GSRM ,satisfy:

[0145] ;

[0146] In the formula, βl Represents the second residual scale parameter of the l-th dual-domain residual compensation unit, and Reshape() represents the operation of restoring the sequence to a two-dimensional feature map according to the original spatial order.

[0147] In this embodiment, β l Initialize to make sigmoid(β) l The value close to zero makes the gated semantic residual modulation branch approach an identity mapping in the early stages of training. When the low-rank update is close to zero, the position modulation factor is close to 1, and the second residual scale parameter is close to zero, the gated semantic residual modulation branch approaches an identity mapping.

[0148] The shared spatial gating graph G enables the two types of residual compensation to maintain synergy in spatial location: the frequency-selective wavelet purification adaptation branch determines the spatial location to be suppressed or retained based on the reconstructed local response, and the gating guides the semantic residual modulation branch to adjust the semantic transformation intensity at the same location, thereby avoiding inconsistent spatial weights learned by noise domain processing and semantic domain processing.

[0149] In the specific implementation of step S6 above, the feature map F will be adapted. GSRM Return the feature flow of the frozen optical fundamental model; for dual-domain residual compensation units located at the end of a stage, output cross-modal adaptation features from the corresponding stage; for dual-domain residual compensation units located within a stage, continue to input the adaptation feature map into the subsequent feature extraction block of the same stage.

[0150] Optionally, one or more levels of output from the feature extraction backbone can be used for cross-modal adaptation features for downstream SAR image tasks such as target detection.

[0151] Example 2

[0152] This embodiment also provides a training method for the method described in Embodiment 1 above. During training, the original parameters of the optical basic model feature extraction backbone remain unchanged. The parameters of the dual-domain residual compensation unit are updated based on the training loss from the downstream task of the SAR image. The updated parameters include frequency domain projection parameters, high-frequency detail modulation parameters, spatial gating generation parameters, low-rank factors B and A, and the original amplitude vector m. raw Gated projection parameters and first residual scale parameters Second residual scaling parameter .

[0153] Initial values ​​are set for the low-rank factor used to form low-rank updates, the first residual scale parameter used for noise domain residual mixing, and the second residual scale parameter used for semantic domain residual mixing, respectively, so that the dual-domain residual compensation unit approaches an identity mapping in the early stages of training. The structure and loss function of the downstream task module do not constitute necessary limitations on the dual-domain residual compensation mechanism of this application.

[0154] Example 3

[0155] To verify the cross-modal adaptation effect of this application, directional target detection can be used as a downstream verification task of SAR with dense spatial prediction requirements. The frozen optical base model and the model with added dual-domain residual compensation units can be compared under the same data partitioning, downstream network, and evaluation protocol. This verification method keeps the downstream modules unchanged, ensuring that performance changes primarily reflect the effect of feature adaptation within the frozen backbone.

[0156] In one verification implementation, the RSAR dataset was used for component comparison. The mAP50 of the frozen optical fundamental model baseline was 84.3%, 86.8% with only the amplitude-direction projection branch set, 90.1% with only the frequency-selective wavelet cleansing adaptation branch set, 92.3% with both branches combined but without sharing the spatial gating map, and 92.8% with the complete dual-domain residual compensation method. These results indicate that noise domain compensation and semantic domain compensation have complementary effects, and sharing the spatial gating map can further improve spatial selectivity.

[0157] In another verification implementation, on a FAIR-CSAR subset containing 22 categories, the mAP50 of the frozen optical baseline model was 33.7%, and after adding the full dual-domain residual compensation method, it was 53.3%. The above values ​​are only used to illustrate that implementing this application under the frozen backbone condition can improve the SAR feature adaptation effect, and do not limit the downstream tasks, datasets or evaluation metrics to which this application is applicable.

[0158] Example 4

[0159] This embodiment also provides a dual-domain residual compensation system for a frozen optical fundamental model of SAR images, such as... Figure 7 As shown, the system includes a frozen feature acquisition module, a noise domain residual compensation module, a semantic domain residual compensation module, and an adaptive feature transfer module. This system can be implemented through software modules, hardware modules, or a combination of both.

[0160] The frozen feature acquisition module is used to perform step S1, which involves inputting the SAR image into the feature extraction backbone of the optical basic model, keeping the parameters of the feature extraction backbone frozen, and obtaining the feature map to be adapted at at least one compensation position of the feature extraction backbone.

[0161] The noise domain residual compensation module is used to execute steps S2 to S3, namely, to perform frequency decomposition on the feature map to be adapted, to obtain low-frequency components and multiple high-frequency components, to keep the low-frequency components unchanged, to selectively modulate and reconstruct the multiple high-frequency components, to generate a spatial gating map corresponding to the spatial position of the feature map to be adapted based on the obtained reconstructed features, to use the spatial gating map to perform spatial modulation on the reconstructed features to form a noise domain residual, and to mix the noise domain residual with the feature map to be adapted to obtain a cleaned feature map.

[0162] The semantic domain residual compensation module is used to execute steps S4 to S5, namely, expanding the purified feature map into a feature label sequence according to the spatial location and performing a linear transformation, using a spatial gating map to modulate the linearly transformed feature labels position by position to form a semantic domain residual, and mixing the semantic domain residual with the feature label sequence to obtain the adapted feature map.

[0163] The adaptation feature transfer module is used to perform step S6, which involves transferring the adaptation feature map as the output of the corresponding compensation location to the subsequent feature extraction unit of the feature extraction backbone. Optionally, the adaptation feature transfer module is also used to output cross-modal adaptation features from one or more levels of the feature extraction backbone for downstream tasks of SAR images.

[0164] Furthermore, without altering the fundamental relationship of dual-domain residual compensation, other frozen optical fundamental models or reversible discrete wavelet bases capable of outputting two-dimensional feature maps can be employed. The ratio of dimensionality reduction channels, the number of compensation positions, the lower rank order, the lower gate limit, and the position modulation amplitude can be set according to computational resources and the characteristics of SAR data. Among these, single-layer Haar wavelet, R=C / 4, ε=0.1, and γ=0.1 are the preferred settings in this embodiment.

[0165] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A dual-domain residual compensation method for a frozen optical fundamental model of SAR images, characterized in that, include: Acquire synthetic aperture radar (SAR) images, input the SAR images into the feature extraction backbone of the optical basic model, keep the parameters of the feature extraction backbone frozen, and obtain the feature map to be adapted at at least one compensation position of the feature extraction backbone. The feature map to be adapted is decomposed by frequency to obtain low-frequency components and multiple high-frequency components. The low-frequency components are kept unchanged, and the multiple high-frequency components are selectively modulated and reconstructed to obtain the reconstructed features. A spatial gating map corresponding to the spatial position of the feature map to be adapted is generated based on the reconstructed features. The reconstructed features are spatially modulated using the spatial gating map to form a noise domain residual. The noise domain residual is mixed with the feature map to be adapted to obtain a cleaned feature map. The purification feature map is expanded into a feature label sequence according to its spatial location and linearly transformed. The feature labels after linear transformation are modulated position by position using the spatial gating map to form a semantic domain residual. The semantic domain residual is mixed with the feature label sequence to obtain the fitting feature map. The adapted feature map is passed as the output of the corresponding compensation position to the subsequent feature extraction unit of the feature extraction backbone.

2. The dual-domain residual compensation method for the frozen optical fundamental model of SAR images according to claim 1, characterized in that, The SAR image is a single-channel SAR image. The single-channel SAR image is copied by channel and then input into the feature extraction backbone. The compensation location is provided with a dual-domain residual compensation unit, which is located after one or more feature extraction blocks of the feature extraction backbone. The dual-domain residual compensation unit includes a frequency-selective cleanup adaptation branch for forming the noise domain residual and the spatial gating graph, and a gated semantic residual modulation branch for reusing the spatial gating graph and forming the semantic domain residual.

3. The dual-domain residual compensation method for the frozen optical fundamental model of SAR images according to claim 1, characterized in that, The frequency decomposition of the feature map to be adapted includes: While keeping the spatial dimensions unchanged, the feature map to be adapted is projected onto the dimension-reduced channel space, and a single-layer discrete wavelet transform is performed on the obtained dimension-reduced features to obtain low-frequency sub-bands as the low-frequency components and multiple high-frequency detail sub-bands as the multiple high-frequency components. The multiple high-frequency detail subbands are spliced ​​together along the channel dimension in a preset direction to form a high-frequency joint feature, and the high-frequency joint feature is modulated in the joint space formed by the multiple high-frequency detail subbands and the feature channel; The modulation result is restored to the corresponding high-frequency detail subband, and the restored high-frequency detail subband and the unchanged low-frequency subband are used as inputs to perform inverse discrete wavelet transform to obtain the reconstructed features.

4. The dual-domain residual compensation method for the frozen optical fundamental model of SAR images according to claim 3, characterized in that, Modulating the high-frequency joint features includes: Local feature extraction is performed on the high-frequency joint features to obtain high-frequency responses that integrate information from different high-frequency detail subbands and different feature channels. Channel modulation weights are generated based on the global average pooling and global max pooling results of the high-frequency responses. Channel-weighted summation is then performed on the high-frequency responses based on the channel modulation weights to obtain the modulated high-frequency joint features.

5. The dual-domain residual compensation method for the frozen optical fundamental model of SAR images according to claim 1, characterized in that, Generate a spatial gating map based on the reconstructed features, including: Spatial feature extraction is performed on the reconstructed features to obtain gated intermediate features. The channel mean, channel maximum, and channel mean of the reconstructed features are calculated to obtain three single-channel spatial summaries. The three single-channel spatial summaries are fused and mapped to the spatial gating map. Each gate value in the spatial gating diagram corresponds one-to-one with each spatial position in the feature diagram to be adapted, and all of them are not less than a preset lower limit, which is greater than zero.

6. The dual-domain residual compensation method for the frozen optical fundamental model of SAR images according to claim 5, characterized in that, The reconstructed features are modulated position-by-position using the spatial gating map, and the modulation result is projected onto the same channel dimension as the feature map to be adapted, forming the noise domain residual. Based on a trainable first residual scale parameter, the residual mixing is performed on the noise domain residual and the feature map to be adapted. The initial value of the first residual scale parameter makes the frequency selective cleanup adaptation branch close to an identity mapping in the early stage of training.

7. The dual-domain residual compensation method for the frozen optical fundamental model of SAR images according to claim 1, characterized in that, The linear transformation is achieved through amplitude-direction projection mapping, which uses a fixed identity matrix as the identity mapping. A low-rank update of the identity mapping is formed by a trainable low-rank factor. The updated mapping is normalized row by row to determine the direction, and the amplitude of each output channel is controlled by a trainable positive amplitude vector.

8. The dual-domain residual compensation method for the frozen optical fundamental model of SAR images according to claim 7, characterized in that, The spatial gating graph is unfolded in the same spatial unfolding order as the generated feature label sequence, and the unfolding result is channel-projected to obtain a position modulation factor that varies with the value 1 as the center. Each position modulation factor corresponds one-to-one with the corresponding feature label. The position modulation factor is used to perform the position-by-position modulation on the linearly transformed feature label. Based on a trainable second residual scale parameter, the semantic domain residual and the feature label sequence are subjected to residual mixing. The initial value of the second residual scale parameter makes the gated semantic residual modulation branch approach an identity mapping in the early stage of training.

9. The dual-domain residual compensation method for frozen optical fundamental models of SAR images according to any one of claims 1 to 8, characterized in that, During training, the original parameters of the feature extraction backbone are kept unchanged, and the parameters of the dual-domain residual compensation unit are updated based on the training loss of the downstream task of SAR image. Initial values ​​are set for the low-rank factor used to form low-rank updates, the first residual scale parameter used for noise domain residual mixing, and the second residual scale parameter used for semantic domain residual mixing, so that the dual-domain residual compensation unit approaches an identity mapping in the early stage of training.

10. A dual-domain residual compensation system for a frozen optical fundamental model of SAR images, characterized in that, include: The frozen feature acquisition module is used to input the SAR image into the feature extraction backbone of the optical basic model, keep the parameters of the feature extraction backbone frozen, and obtain the feature map to be adapted at at least one compensation position of the feature extraction backbone. The noise domain residual compensation module is used to perform frequency decomposition on the feature map to be adapted, to obtain low-frequency components and multiple high-frequency components, keep the low-frequency components unchanged, selectively modulate the multiple high-frequency components and reconstruct them, generate a spatial gating map corresponding to the spatial position of the feature map to be adapted based on the obtained reconstructed features, use the spatial gating map to spatially modulate the reconstructed features to form a noise domain residual, and perform residual mixing between the noise domain residual and the feature map to be adapted to obtain a cleaned feature map; The semantic domain residual compensation module is used to expand the purified feature map into a feature label sequence according to the spatial location and perform a linear transformation, use the spatial gating map to modulate the linearly transformed feature labels position by position to form a semantic domain residual, and perform residual mixing between the semantic domain residual and the feature label sequence to obtain an adapted feature map. The adaptation feature transfer module is used to transfer the adaptation feature map as the output of the corresponding compensation position to the subsequent feature extraction unit of the feature extraction backbone.

Images