Training method and device of remote sensing cross-domain target detection model and detection method

Abstract

The application discloses a training method and device of a remote sensing cross-domain target detection model and a detection method, and belongs to the technical field of remote sensing image detection; the method comprises the following steps: obtaining a target domain remote sensing image without labeling; inputting the target domain remote sensing image into a teacher model and a student detection model respectively to obtain teacher spatial token features and student spatial token features; performing spatial layout alignment on the teacher spatial token features and the student spatial token features, so that the tokens at corresponding spatial positions are one-to-one corresponding; determining a teacher similarity matrix and a student similarity matrix respectively based on the similarity between the tokens at each spatial position in the aligned teacher spatial token features and student spatial token features; applying a positive odd-order polynomial kernel with an order greater than 1 to the teacher similarity matrix and the student similarity matrix element by element to obtain a teacher kernel matrix and a student kernel matrix; determining a relationship distillation loss according to the difference between the teacher kernel matrix and the student kernel matrix; and updating the parameters of the student detection model according to a target loss containing the relationship distillation loss.

Description

Technical Field

[0001] This application relates to the field of remote sensing image detection technology, specifically to a training method, apparatus, and detection method for a remote sensing cross-domain target detection model. Background Technology

[0002] The performance of remote sensing target detection models degrades significantly when deployed across domains. The core bottleneck is that background areas such as roads, water surfaces, and buildings occupy the vast majority of pixels in remote sensing images, while foreground targets such as aircraft, ships, and vehicles are extremely sparse. This results in the alignment loss in the unlabeled target domain being dominated by background statistical information, making it difficult to effectively transfer the semantic relationship structure between foreground targets.

[0003] To alleviate this problem, related technical approaches include region-level domain alignment, detection knowledge distillation, frozen base model feature reuse, semantic segmentation domain adaptation, and foreground-aware weighted alignment. While these approaches can guide the model to focus on foreground regions or transfer teacher network knowledge from different perspectives, their distillation or alignment objects are still concentrated on single-point features, region responses, or pixel-level predictions. This results in insufficient transfer of semantic relationship structures between foreground targets during unlabeled training in the target domain, limiting the improvement in target domain detection accuracy. Summary of the Invention

[0004] This application provides a training method, apparatus, and detection method for a remote sensing cross-domain target detection model, which addresses the problem of insufficient transfer of semantic relationship structure between foreground targets and limited improvement in target domain detection accuracy during unlabeled training in the target domain.

[0005] To achieve the above objectives, the embodiments of this application disclose the following technical solutions: In a first aspect, embodiments of this application provide a training method for a remote sensing cross-domain target detection model, comprising the following steps: Acquire unlabeled remote sensing images of the target domain; Input the remote sensing image of the target domain into the teacher model to obtain the teacher spatial token features; Input the remote sensing image of the target domain into the student detection model to obtain the student spatial token features; Spatial layout alignment is performed on the teacher space token features and student space token features to ensure that space tokens in corresponding spatial locations correspond one-to-one. Based on the similarity between all pairs of spatial location tokens in the teacher spatial token features after spatial layout alignment, determine the teacher similarity matrix corresponding to the remote sensing image of the target domain. Based on the similarity between all pairs of spatial location tokens in the student spatial token features after spatial layout alignment, determine the student similarity matrix corresponding to the target domain remote sensing image; Apply a positive odd-order polynomial kernel of order greater than 1 to each element of the teacher similarity matrix and the student similarity matrix to obtain the teacher kernel matrix and the student kernel matrix, respectively. The relationship distillation loss is determined based on the difference between the student kernel matrix and the teacher kernel matrix; The parameters of the student detection model are updated based on the target loss, which includes relation distillation loss.

[0006] In some possible implementations of the first aspect, the step of determining the relation distillation loss based on the difference between the student kernel matrix and the teacher kernel matrix includes: Determine the relationship gap between the student kernel matrix and the teacher kernel matrix; Dynamic weights are generated based on the relationship gap; The relation distillation loss is determined based on the dynamic weights, student kernel matrix, and teacher kernel matrix.

[0007] In some possible implementations of the first aspect, the step of determining the gap between the student kernel matrix and the teacher kernel matrix includes: Vectorize the student kernel matrix and the teacher kernel matrix respectively to obtain the student kernel vector and the teacher kernel vector; The cosine similarity of the kernel relation is determined based on the student kernel vector and the teacher kernel vector. The relationship gap is determined based on the cosine similarity of the nuclear relationship.

[0008] In some possible implementations of the first aspect, the kernel relation cosine similarity is determined by the following formula: In the formula, For the first Kernel cosine similarity of remote sensing image samples in a target domain; For the first The student kernel matrix corresponding to each sample; For the first The teacher kernel matrix corresponding to each sample; Vectorization; This represents the L2 norm.

[0009] In some possible implementations of the first aspect, vectorization is performed using only the off-diagonal elements or upper triangular off-diagonal elements of the student kernel matrix and the teacher kernel matrix.

[0010] In some possible implementations of the first aspect, the relationship gap is determined by the following formula: In the formula, For the first The relationship gaps between remote sensing image samples of each target domain; This indicates the cosine similarity of the kernel relation. Crop to range .

[0011] In some possible implementations of the first aspect, the dynamic weights are determined by the following formula: In the formula, For the first Dynamic weights corresponding to remote sensing image samples of the target domain; For the relationship gap; It is a dynamic weighted index.

[0012] In some possible implementations of the first aspect, the step of determining the relation distillation loss based on the dynamic weights, the student kernel matrix, and the teacher kernel matrix includes: Based on the student kernel matrix and the teacher kernel matrix, determine the basic kernel relation differences; The relation distillation loss is determined based on the dynamic weights and basic kernel relation differences corresponding to the remote sensing image samples of each target domain.

[0013] In some possible implementations of the first aspect, the basic kernel relation difference term is determined by the following formula: In the formula, For the first Student kernel matrix corresponding to each remote sensing image sample in the target domain; For the first Teacher kernel matrix corresponding to each remote sensing image sample in the target domain; For the first The fundamental kernel relation difference terms corresponding to each remote sensing image sample in the target domain; This indicates that the gradient operation has been stopped. Denotes the Frobenius norm; This represents the number of spatial tokens in a single remote sensing image of the target domain.

[0014] In some possible implementations of the first aspect, the relational distillation loss is determined by the following formula: In the formula, For the first Dynamic weights corresponding to remote sensing image samples of the target domain; For the relationship of distillation loss; The number of remote sensing image samples for the target domain within the batch; For the first The fundamental kernel relation difference terms corresponding to each remote sensing image sample in the target domain.

[0015] In some possible implementations of the first aspect, the dynamic weights, after stopping gradient processing, are used as adjustment coefficients for the relational distillation loss when participating in the calculation of relational distillation loss.

[0016] In some possible implementations of the first aspect, the step of applying a positive odd-order polynomial kernel of order greater than 1 to each element of the teacher similarity matrix and the student similarity matrix includes: For each pair of spatial location tokens in the teacher similarity matrix and student similarity matrix, a positive odd-order power transformation is performed to superlinearly amplify the similarity relationship of highly similar foreground token pairs, suppress the similarity relationship of low-similarity background token pairs, and retain the sign of negative correlation.

[0017] In some possible implementations of the first aspect, the positive odd-order power transformation is determined by the following formula: In the formula, and The first and second kernels in the student kernel matrix and teacher kernel matrix, respectively. The spatial location token and the first Each spatial location token corresponds to an element; and These are the first two numbers in the student similarity matrix and the teacher similarity matrix, respectively. The spatial location token and the first Similarity between tokens in spatial locations; Let be the order of the kernel of a positive odd-order polynomial. It is a positive odd number greater than 1.

[0018] In some possible implementations of the first aspect, the order of the positive odd-order polynomial kernel The value is 3.

[0019] In some possible implementations of the first aspect, the remote sensing image of the target domain is input into the teacher model and the student detection model after being processed by fixed weak enhancement; wherein, the fixed weak enhancement process includes deterministic scaling, filling, cropping and normalization, and does not employ random strong geometric transformations, and does not perform large-angle rotation, random perspective or CutMix operations that would change the spatial token correspondence.

[0020] In some possible implementations of the first aspect, the parameters of the teacher model are kept frozen during training; wherein, the teacher space token features are obtained by online extraction or offline pre-extraction; when the online extraction method is used, the target domain fixed weak augmentation view is input into the teacher model for forward propagation to obtain the initial teacher token features, and the initial teacher token features are subjected to stopping gradient processing, and the initial teacher token features after stopping gradient processing are used as the teacher space token features. When using the offline pre-extraction method, the teacher token features of the pre-stored target domain remote sensing image are read as teacher spatial token features. The pre-stored teacher token features are features extracted and stored from the target domain remote sensing image by the teacher model before training.

[0021] In some possible implementations of the first aspect, the step of determining the teacher similarity matrix corresponding to the target domain remote sensing image based on the pairwise similarity between all spatial location tokens in the teacher spatial token features after spatial layout alignment includes: Flatten the teacher space token features into a teacher token sequence; Subtract the spatial mean of all spatial location tokens in the corresponding image from the teacher token sequence, and then perform L2 normalization to obtain the normalized teacher token sequence. Based on the pairwise similarity between all spatial location tokens in the normalized teacher token sequence, the teacher similarity matrix corresponding to the remote sensing image of the target domain is determined.

[0022] In some possible implementations of the first aspect, the teacher token sequence is determined by the following formula: In the formula, This is the normalized sequence of teacher tokens; The token characteristics of the teacher's space after spatial layout alignment; Indicates the flattening operation; ; This indicates L2 normalization.

[0023] In some possible implementations of the first aspect, the step of determining the student similarity matrix corresponding to the target domain remote sensing image based on the pairwise similarity between all spatial location tokens in the spatially aligned student spatial token features includes: Flatten the student space token features into a student token sequence; Subtract the spatial mean of all spatial location tokens within the corresponding image from the student token sequence, and then perform L2 normalization to obtain the normalized student token sequence. Based on the pairwise similarity between all spatial location tokens in the normalized student token sequence, the student similarity matrix corresponding to the remote sensing image of the target domain is determined.

[0024] In some possible implementations of the first aspect, the normalized student token sequence is determined by the following formula: In the formula, This is the normalized sequence of student tokens; The token characteristics of the student space after spatial layout alignment; Indicates the flattening operation; ; This indicates L2 normalization.

[0025] In some possible implementations of the first aspect, when the spatial dimensions of the teacher space token feature and the student space token feature are different, spatial layout alignment is achieved by at least one of bilinear interpolation, region average pooling, or token sampling.

[0026] In some possible implementations of the first aspect, the target loss further includes a detection supervision loss and a domain adaptation loss, which are determined by the following steps: Acquire labeled source domain remote sensing images, which include bounding box annotations and category annotations; Input the source domain remote sensing image into the student detection model to obtain the source domain detection prediction result; Based on the source domain detection prediction results and the annotation of the source domain remote sensing image, the detection supervision loss is determined; The source domain features extracted from the source domain remote sensing image and the target domain features extracted from the target domain remote sensing image by the student detection model are used to determine the domain adaptation loss through a domain discriminator.

[0027] In some possible implementations of the first aspect, the target loss is determined by the following formula: ; In the formula, Loss to the target; To detect and monitor losses; For domain adaptation loss weights; Domain adaptation loss; Weights for distillation loss related to the relationship; This refers to the loss during distillation.

[0028] Secondly, embodiments of this application provide a training apparatus for a remote sensing cross-domain target detection model, comprising: The image acquisition module is used to acquire unlabeled remote sensing images of the target domain; The teacher feature extraction module is used to input the remote sensing image of the target domain into the teacher model to obtain the teacher spatial token features; The student feature extraction module is used to input the remote sensing image of the target domain into the student detection model to obtain the student spatial token features; The spatial alignment module is used to align the spatial layout of teacher spatial token features and student spatial token features, so that spatial tokens at corresponding spatial locations correspond one-to-one. The first matrix determination module is used to determine the teacher similarity matrix corresponding to the target domain remote sensing image based on the similarity between all pairs of spatial location tokens in the teacher spatial token features after spatial layout alignment. The second matrix determination module is used to determine the student similarity matrix corresponding to the target domain remote sensing image based on the similarity between all pairs of spatial location tokens in the student spatial token features after spatial layout alignment. The kernel matrix determination module is used to apply a positive odd-order polynomial kernel of order greater than 1 to each element of the teacher similarity matrix and the student similarity matrix to obtain the teacher kernel matrix and the student kernel matrix. The distillation loss determination module is used to determine the relational distillation loss based on the difference between the student kernel matrix and the teacher kernel matrix. The parameter update module is used to update the parameters of the student detection model based on the target loss, which includes relation distillation loss.

[0029] Thirdly, embodiments of this application provide a cross-domain remote sensing image detection method, comprising the following steps: Acquire the remote sensing image to be detected; The remote sensing image to be detected is input into the student detection model to obtain the target detection result; wherein the student detection model is trained according to the training method described in any one of the embodiments of the first aspect.

[0030] In some implementations of the third aspect, during the inference phase, only the main path of the student detection model is retained for target detection. The teacher model, the student weak enhancement branch, as well as token flattening and normalization operations, pairwise similarity matrix calculation, multinomial kernel matrix calculation, relation gap and dynamic weight calculation, and relation distillation loss calculation are not performed during the inference phase.

[0031] Fourthly, embodiments of this application provide an electronic device, including: One or more processing devices; A storage device on which one or more programs are stored; When one or more programs are executed by one or more processing devices, the one or more processing devices implement the method described in any of the first or third aspects above.

[0032] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processing device, implements the method described in any one of the first or third aspects described above.

[0033] One or more technical solutions provided in the first to fifth aspects of the embodiments of this application have at least the following technical effects: 1. By converting the teacher space token features and student space token features into similarity matrices between pairs of spatial location tokens in the image, and applying a positive odd-order polynomial kernel of order greater than 1 to each element of the teacher similarity matrix and student similarity matrix, the superlinear response property of the polynomial kernel can be utilized to maintain a large response in the kernel matrix for the originally high similarity between semantically similar foreground space tokens, while the originally close-to-zero similarity between weakly correlated background space tokens or between foreground and background space tokens is quickly suppressed.

[0034] Based on this, the relation distillation loss is determined according to the difference between the student kernel matrix and the teacher kernel matrix. This allows the difference in relations between highly similar foreground token pairs to dominate the relation distillation loss, thereby effectively suppressing the difference in relations between low-similarity background token pairs. In this way, background statistics no longer dominate the alignment loss, and the semantic relation structure between foreground tokens can be preferentially transferred to the student detection model. This solves the problem of insufficient transfer of foreground semantic relation structure caused by the dominant role of background statistics in the alignment loss, even when the target domain is unlabeled.

[0035] 2. By determining the relationship gap between the student kernel matrix and the teacher kernel matrix, generating dynamic weights based on this gap, and then determining the relation distillation loss based on these dynamic weights, the strength of the relation distillation loss can be positively correlated with the current degree of recovery of the teacher model's relation structure by the student detection model. When the relationship gap between the student kernel matrix and the teacher kernel matrix is ​​large in the early stages of training, the dynamic weights increase accordingly, and the relation distillation loss provides a strong structural transfer constraint. Conversely, as the student kernel matrix gradually approaches the teacher kernel matrix and the relationship gap decreases in the later stages of training, the dynamic weights automatically decrease, and the constraint strength of the relation distillation loss weakens accordingly. In this way, the transfer of foreground semantic relation structure can be enhanced in the early stages of training, while the distillation constraint is automatically weakened in the later stages, reducing the interference of the relation distillation loss on the classification and localization optimization of the main detection task.

[0036] 3. By performing fixed weak augmentation on the target domain remote sensing image and then inputting it into the teacher model and student detection model respectively, the teacher model and student detection model can process fixed weak augmented views of the target domain with the same spatial content and consistent layout. Based on this, the student detection model extracts student spatial token features through the student weak augmentation branch. Utilizing the characteristic that the student weak augmentation branch and the student detection model share feature extraction parameters, the relation distillation gradient can be backpropagated to the shared parameters of the student detection model during the training phase via the student weak augmentation branch, without the need to introduce an additional independent student auxiliary model. In this way, the student weak augmentation branch can be avoided as a trainable network parallel to the teacher model and student detection model.

[0037] 4. By freezing the parameters of the teacher model during training and using either online extraction followed by stopping gradient processing or offline pre-extraction to obtain teacher space token features, it can be ensured that the teacher model is not affected by the parameter updates of the student detection model during target domain training and is not contaminated by early prediction noise in the target domain. This allows the extracted teacher space token features to provide a more stable semantic relationship structure. Consequently, the teacher model can consistently provide stable external structural anchors for unlabeled samples in the target domain, reducing the dependence of foreground semantic relationship structure transfer on the prediction quality of the target domain detector itself.

[0038] 5. By omitting the execution of teacher model, student weak enhancement branch, token flattening and normalization operations, pairwise similarity matrix calculation, multinomial kernel matrix calculation, relation gap and dynamic weight calculation, and relation distillation loss calculation during the inference phase, and only retaining the main path of the student detection model for target detection, the deployment phase can avoid additional computational overhead and increase the complexity of the model structure. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0040] Figure 1 This is a flowchart illustrating a training method for a remote sensing cross-domain target detection model, provided for some embodiments of this application.

[0041] Figure 2 This is a schematic diagram of the structure of a training device for a remote sensing cross-domain target detection model, provided for some embodiments of this application.

[0042] Figure 3This is a schematic diagram of the structure of an electronic device suitable for implementing a training method for a remote sensing cross-domain target detection model, provided for some embodiments of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] In the embodiments of this application, the terms "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0045] In the description of the embodiments of this application, the term "multiple" means two or more.

[0046] In the description of the embodiments of this application, the term "and / or" refers to and covers any and all possible combinations of one or more of the associated listed items. The term "and / or" describes an association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this application generally indicates that the preceding and following related objects have an "or" relationship.

[0047] In the description of embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0048] Background Overview: Remote sensing target detection is widely used in scenarios such as traffic flow monitoring, parking lot vehicle statistics, airport aircraft identification, port vessel supervision, power facility inspection, post-disaster building assessment, border and coastal defense patrols, and large-scale land and resources management. In actual deployment, detection models often need to be migrated from existing labeled data source domain platforms to new satellite, UAV, regional, seasonal, or imaging conditions. Due to the large coverage area of ​​remote sensing images, complex background types, and the low proportion of targets in the image area, foreground targets such as aircraft, ships, vehicles, and storage tanks typically occupy only a very small portion of the total image pixels. Conventional domain alignment or global distillation loss functions are easily dominated by background areas such as roads, sea surfaces, farmland, and building rooftops, while the semantic relationships of the foreground targets that are truly needed are not adequately preserved.

[0049] In related technologies, some schemes guide the domain classifier to focus on potential foreground regions through region proposals, but their alignment objects are still the domain discrimination results of the detector's internal features or region proposals. Other schemes use knowledge distillation to transfer teacher model features to the student detector, but their distillation or alignment objects still focus on single-point features, region responses, or pixel-level predictions, without specifically transferring the pairwise relationship structure between foreground tokens within the image. Therefore, in unlabeled training of the target domain, background statistics play a dominant role in the alignment loss, leading to insufficient transfer of semantic relationship structures between foreground targets and limiting the improvement of target domain detection accuracy.

[0050] Another approach utilizes detector self-prediction or centrality estimation to distinguish foreground from background. However, when target domain prediction is unstable or remotely sensed targets are extremely small and sparse, foreground estimation is susceptible to false label noise. Furthermore, existing distillation losses often use fixed weights, continuing to impose strong constraints even when students have approached the teacher relationship structure in the later stages of training, which can easily interfere with the classification and localization optimization of the main detection task.

[0051] To address the aforementioned issues, this application provides a training method for a remote sensing cross-domain target detection model. This method can be executed by an execution entity such as a processor, server, or electronic device. Please refer to [link to relevant documentation]. Figure 1 and Figure 2 The training method for the remote sensing cross-domain target detection model includes the following steps:

[0052] S101, acquire unlabeled remote sensing images of the target domain.

[0053] The target domain remote sensing image refers to unlabeled remote sensing images acquired from different satellite platforms, UAV platforms, sensors, regions, seasons, or imaging conditions than the source domain remote sensing images. For example, it might be adapting a detection model trained on satellite platform A to satellite platform B, adapting a vehicle detection model trained on satellite imagery to UAV imagery, or adapting a port vessel detection model from one region to a sea scene in another region. The source domain remote sensing image refers to a remote sensing image with target bounding boxes and category labels, used to provide supervisory signals for the detection task. The bounding box label can be the coordinates of the bounding rectangle for each foreground target instance; the category label can be the semantic category label to which each foreground target instance belongs, such as aircraft, ship, vehicle, or storage tank.

[0054] The target domain remote sensing images do not contain bounding box and category labels. During training, they are only used to construct teacher-student relationship distillation branches and do not participate in the calculation of the detection supervision loss. For example, the target domain remote sensing images can be read in batches from the target domain partition of the training dataset, with each batch containing... Zhang target area remote sensing image.

[0055] S102, Input the remote sensing image of the target domain into the teacher model to obtain the teacher spatial token features.

[0056] The teacher model (also known as the remote sensing base model) refers to a pre-trained visual base model whose parameters are frozen during training and which does not participate in backpropagation. It is used to provide stable spatial semantic relationship structure anchors for remote sensing images in the target domain. For example, the teacher model is preferably a visual base model pre-trained on large-scale remote sensing images, satellite images, or remote sensing-natural image hybrid data, which can provide relatively stable spatial semantic relationships. The teacher spatial token feature refers to the feature vector representation extracted by the teacher model from the remote sensing image in the target domain, corresponding to each spatial location. Each spatial location corresponds to one token feature vector.

[0057] Furthermore, in some embodiments, teacher space token features can be obtained using online extraction or offline pre-extraction. When using online extraction, a fixed weakly augmented view of the target domain is input into the teacher model for forward propagation to obtain initial teacher token features. Gradient stopping processing is then performed on the initial teacher token features, and the initial teacher token features after gradient stopping processing are used as teacher space token features. Gradient stopping processing refers to separating the initial teacher token features from the gradient propagation path in the computation graph, so that the gradient does not flow through the teacher model during backpropagation, thereby ensuring that the parameters of the teacher model remain unchanged during training.

[0058] When using the offline pre-extraction method, the teacher token features of the pre-stored target domain remote sensing image are read as teacher spatial token features. The pre-stored teacher token features are features extracted and stored from the target domain remote sensing image by the teacher model before training.

[0059] S103, Input the remote sensing image of the target domain into the student detection model to obtain the student spatial token features.

[0060] Here, the student detection model refers to the actual target detection network that needs to be adapted and deployed. Its parameters are updated during training, and it is the only detection model used for inference after training. For example, the student detection model can be a convolutional-transformer hybrid detector, a DETR-type detector, or other detection networks with spatial feature maps; the student spatial token feature refers to the feature vector representation extracted by the student detection model from the remote sensing image of the target domain, corresponding to each spatial location. It can be understood that the student detection model is the target detection model to be trained.

[0061] Specifically, in some embodiments, the target domain remote sensing image is input into the teacher model and the student detection model after being processed by fixed weak enhancement; wherein, the fixed weak enhancement process includes deterministic scaling, filling, cropping and normalization, and does not use random strong geometric transformations, and does not perform large-angle rotation, random perspective or CutMix operations that would change the spatial token correspondence.

[0062] For example, deterministic scaling can uniformly scale a remote sensing image of a target domain to a preset fixed size (e.g., Pixels or Instead of using random scaling, the images are padded (pixels) to ensure that the teacher and student detection models see a completely consistent spatial layout within the same batch. For example, the padding can use constant padding or edge mirroring to complete a non-square image into a square before inputting it into the model.

[0063] The target domain remote sensing image after fixed weak enhancement processing can be called the target domain fixed weak enhancement view. This means that the target domain fixed weak enhancement view is input into the teacher model and the student detection model to obtain teacher spatial token features and student spatial token features. For example, the target domain fixed weak enhancement view can be input into the student weak enhancement branch of the student detection model, and the output of the student weak enhancement branch at a selected intermediate feature layer can be extracted as the student spatial token feature.

[0064] Among them, the student weak augmentation branch refers to an auxiliary forward propagation path executed by the student detection model on the target domain with a fixed weak augmentation view during the training phase. It shares the backbone network and intermediate feature extraction parameters with the main training path of the student detection model and is not a second detection network independent of the student detection model. The selected intermediate feature layer refers to the intermediate feature layer in the student detection model that matches the resolution of the teacher space token features and has semantic context.

[0065] For example, if the student detection model includes a batch normalization layer, the statistics (mean and variance) of the batch normalization layer can be temporarily fixed when the student weak enhancement branch performs forward propagation, in order to prevent the auxiliary forward propagation of this branch from changing the batch normalization statistics of the main detection path.

[0066] S104, Spatial layout alignment of teacher space token features and student space token features, so that space tokens in corresponding spatial positions correspond one-to-one.

[0067] Spatial layout alignment refers to adjusting the spatial token features of teachers and students to the same or one-to-one spatial resolution, ensuring that teacher tokens and student tokens in the same spatial location can be matched, thus providing a spatial correspondence basis for the subsequent construction of pairwise relation matrices.

[0068] As an example, alignment of teacher spatial token features with student spatial token features can be achieved by adjusting the spatial resolution of one or both. When the spatial dimensions of teacher spatial token features and student spatial token features are different, spatial layout alignment is achieved through at least one of bilinear interpolation, region average pooling, or token sampling.

[0069] Specifically, when the resolution of the teacher spatial token feature is higher than that of the student spatial token feature, region average pooling can be performed on the teacher spatial token feature, for example, by... The teacher feature map is divided into... For the corresponding grid, the average value of the token features at all spatial locations within each grid is taken to obtain the aligned teacher token features with the same resolution as the student spatial token features.

[0070] When the resolution of the teacher space token feature is lower than that of the student space token feature, bilinear interpolation upsampling can be performed on the teacher space token feature, or region average pooling downsampling can be performed on the student space token feature to make their resolutions consistent.

[0071] In addition, in student detection models based on the Transformer architecture, spatial layout alignment can also be achieved through token sampling. For example, a subset with the same number of teacher tokens can be uniformly sampled from the student token sequence according to spatial location, or the teacher token sequence can be interpolated and expanded to match the number of student tokens.

[0072] S105, Based on the similarity between all pairs of spatial location tokens in the teacher spatial token features after spatial layout alignment, determine the teacher similarity matrix corresponding to the remote sensing image of the target domain.

[0073] The teacher similarity matrix refers to a square matrix whose elements are the similarity between any two spatial location tokens in the teacher spatial token features. The rows and columns of the matrix correspond to the spatial locations in the target domain remote sensing image. Line 1 The elements of the column represent the first... The spatial location token and the first The similarity between spatial location tokens. The teacher similarity matrix characterizes the overall structural representation of the semantic relationships between spatial locations in the target domain remote sensing image by the teacher model.

[0074] Specifically, in some embodiments, the teacher similarity matrix corresponding to the target domain remote sensing image can be determined by the following steps based on the pairwise similarity between all spatial location tokens in the teacher spatial token features after spatial layout alignment: The first step is to flatten the teacher space token features into a teacher token sequence; For example, the flattening operation will flatten the shape into Teacher space token features Reshape into shape The teacher token sequence, in which The total number of spatial locations, Number of teacher-specific channels.

[0075] The second step is to subtract the spatial mean of all spatial location tokens in the corresponding image from the flattened teacher token sequence and then perform L2 normalization to obtain the normalized teacher token sequence. As an example, the normalized teacher token sequence can be determined by the following formula: In the formula: This is the normalized sequence of teacher tokens; The token characteristics of the teacher's space after spatial layout alignment; Indicates the flattening operation; This indicates spatial centralization, subtracting the mean; This indicates L2 normalization.

[0076] Spatial centralization is achieved by taking the teacher token sequence for each image along the spatial dimension (i.e., ...). The mean of the features of each spatial location token is calculated, and then the mean is subtracted from each spatial location token to eliminate global activation bias between different images or different models. L2 normalization is to divide the feature vector of each spatial location token by its L2 norm so that its magnitude is 1, so that the inner product of the next two spatial location tokens is equivalent to their cosine similarity.

[0077] The third step is to determine the teacher similarity matrix corresponding to the remote sensing image of the target domain based on the pairwise similarity between all spatial location tokens in the normalized teacher token sequence.

[0078] S106. Based on the similarity between all pairs of spatial location tokens in the student spatial token features after spatial layout alignment, determine the student similarity matrix corresponding to the remote sensing image of the target domain.

[0079] Among them, the student similarity matrix refers to a square matrix with the similarity between any two spatial location tokens in the student spatial token features as its elements. Its construction method corresponds to that of the teacher similarity matrix, which describes the overall structural representation of the semantic relationship between various spatial locations in the target domain remote sensing image by the student detection model.

[0080] Specifically, in some embodiments, the step of determining the student similarity matrix corresponding to the target domain remote sensing image based on the pairwise similarity between all spatial location tokens in the spatially aligned student spatial token features includes: The first step is to flatten the student space token features into a student token sequence; The second step is to subtract the spatial mean of all spatial location tokens in the image from the flattened student token sequence and then perform L2 normalization to obtain the normalized student token sequence. As an example, the normalized student token sequence can be determined by the following formula: In the formula: This is the normalized sequence of student tokens; The token characteristics of the student space after spatial layout alignment; Indicates the flattening operation; This indicates spatial centralization, subtracting the mean; This indicates L2 normalization.

[0081] Additionally, it's important to note that the flattening, spatial centering, and L2 normalization operations for the student token sequence are consistent with those for the teacher token sequence, and neither involves learnable parameters. Furthermore, the student and teacher token sequences can have different numbers of channels. The number of channels for the student space token feature. The number of channels represents the token features in the teacher space. Since the subsequent distillation operation targets the relationship matrix between tokens rather than the original channel features, it is not required that the number of channels for teachers and students be the same.

[0082] The third step is to determine the student similarity matrix corresponding to the remote sensing image of the target domain based on the pairwise similarity between all spatial location tokens in the normalized student token sequence.

[0083] Furthermore, based on the aforementioned normalized teacher token sequence and student token sequence Teacher similarity matrix Student similarity matrix They can be determined using the following formulas respectively: In the formula: The teacher similarity matrix; Student similarity matrix; This is the normalized sequence of teacher tokens; This is the normalized sequence of student tokens. Teacher similarity matrix. The Middle Line 1 Column elements For the first The teacher token and the first The normalized inner product of each teacher's token is equivalent to the cosine similarity between them; the student similarity matrix... Similarly, by calculating the pairwise similarity matrix using the inner product form described above, "single-point feature distillation" is transformed into "image-internal relational structure distillation," thereby avoiding forcing the student detection model to copy the feature values ​​of the teacher model channel by channel.

[0084] S107. Apply a positive odd-order polynomial kernel of order greater than 1 to each element of the teacher similarity matrix and the student similarity matrix to obtain the teacher kernel matrix and the student kernel matrix.

[0085] The polynomial kernel refers to the power transformation operation performed on the similarity values ​​of each pair of spatial location tokens in the similarity matrix, i.e., an element-wise exponentiation operation with the similarity value as the base and a preset order as the exponent; the positive odd order refers to the order of the polynomial kernel being a positive odd number greater than 1. Furthermore, the teacher kernel matrix is ​​the matrix obtained by transforming the teacher similarity matrix element-wise with the polynomial kernel; the student kernel matrix is ​​the matrix obtained by transforming the student similarity matrix element-wise with the polynomial kernel.

[0086] Specifically, in some embodiments, the step of applying a positive odd-order polynomial kernel of order greater than 1 to each element of the teacher similarity matrix and the student similarity matrix includes: For each pair of spatial location tokens in the teacher similarity matrix and student similarity matrix, a positive odd-order power transformation is performed to superlinearly amplify the similarity relationship of highly similar foreground token pairs, suppress the similarity relationship of low-similar background token pairs, and retain the sign of negative correlations to avoid incorrectly converting mutually exclusive relationships into positive correlations.

[0087] Among them, positive odd-order power transformation refers to element-wise power operation with each element in the similarity matrix as the base and a positive odd number greater than 1 as the exponent; superlinear amplification refers to the fact that high similarity elements with absolute values ​​close to 1 (usually corresponding to foreground-foreground token pairs) still maintain a high response value after odd-order power transformation, thus occupying a larger weight in the subsequent distillation loss; suppressing low similarity background token pairs refers to the fact that low similarity elements with absolute values ​​close to 0 (usually corresponding to background-background or foreground-background token pairs) have their response values ​​significantly reduced after odd-order power transformation, thus their contribution to the subsequent distillation loss is automatically weakened.

[0088] Preferably, in some embodiments, the positive odd-order power transform can be determined by the following formula: In the formula: and The first and second kernels in the student kernel matrix and teacher kernel matrix, respectively. The spatial location token and the first Each spatial location token corresponds to an element; and These are the first two numbers in the student similarity matrix and the teacher similarity matrix, respectively. The spatial location token and the first Similarity between tokens in spatial locations; Let be the order of the kernel of a positive odd-order polynomial. A positive odd number greater than 1 is used to control the degree of nonlinear discrimination of similarity relationships by the kernel function. For example, the order of a positive odd-order polynomial kernel... The value is 3.

[0089] when When the similarity between two foreground tokens (e.g., two spatial locations within the same vehicle cluster or the same ship region) in the teacher model is 1, After three nuclear transformations, it becomes It still maintains a high response rate; if the similarity between two background tokens or foreground-background mixed token pairs is 1... After three nuclear transformations, it becomes Its contribution is significantly suppressed. If two tokens are negatively correlated, the negative sign is preserved, which avoids incorrectly converting a mutually exclusive relationship into a positive correlation.

[0090] Of course, this application is not limited thereto. In other embodiments, the order of the positive odd-order polynomial kernel... It can also take the value 5 or other positive odd numbers greater than 1; when When the value is larger, the amplification effect on highly similar foreground relationships and the suppression effect on low similar background relationships are stronger. The specific value can be selected according to the scale of the target domain image, the sparsity of the target, and the training stability.

[0091] S108, determine the relational distillation loss based on the difference between the student kernel matrix and the teacher kernel matrix.

[0092] Among them, relation distillation loss refers to the loss function term constructed based on the difference between the student kernel matrix and the teacher kernel matrix. It is used to constrain the student detection model to learn the spatial relation structure of the target domain remote sensing image during the training process, so that the student detection model can obtain the semantic relation structure knowledge of the target domain foreground without copying the single-point feature channel values ​​of the teacher model.

[0093] Specifically, in some embodiments, the step of determining the relation distillation loss based on the difference between the student kernel matrix and the teacher kernel matrix includes: The first step is to determine the gap between the student kernel matrix and the teacher kernel matrix.

[0094] Among them, the relationship gap refers to the quantitative measure of the overall structural consistency between the student kernel matrix and the teacher kernel matrix. The larger the gap, the greater the difference between the current spatial relationship structure of the student detection model and the teacher model, and the more powerful the distillation constraint needs to be applied.

[0095] As an example, the gap between the student kernel matrix and the teacher kernel matrix can be determined through the following steps: The first sub-step involves vectorizing the student kernel matrix and the teacher kernel matrix respectively, resulting in the student kernel vector and the teacher kernel vector.

[0096] Vectorization refers to transforming the shape into... The kernel matrix is ​​expanded into a length of A one-dimensional vector is used to calculate the overall directional similarity between two kernel matrices.

[0097] For example, vectorization is performed using only the off-diagonal elements or upper triangular off-diagonal elements of the student kernel matrix and teacher kernel matrix. Specifically, the diagonal elements of the kernel matrix correspond to the similarity between each spatial location token and itself. After centering and L2 normalization, the self-similarity is always 1 or close to 1, contributing very little to the discriminative power of relational differences. Retaining the diagonal elements would dilute the cross-spatial relational information contained in the off-diagonal elements. Therefore, vectorization is performed using only the off-diagonal elements or upper triangular off-diagonal elements of the student kernel matrix and teacher kernel matrix.

[0098] When vectorization uses the upper triangular off-diagonal elements of the student kernel matrix and the teacher kernel matrix, the following effect can be achieved: because the similarity matrix is ​​symmetric (i.e., The kernel matrix after polynomial kernel transformation also maintains symmetry. The off-diagonal elements of the upper triangle contain all the information on spatial positional relationships, thus avoiding redundant calculations.

[0099] The second sub-step involves determining the cosine similarity of the kernel relation based on the student kernel vector and the teacher kernel vector.

[0100] Among them, the kernel relation cosine similarity refers to the cosine value of the angle between the student kernel vector and the teacher kernel vector. It is used to measure the degree of consistency between the student kernel relation pattern and the teacher kernel relation pattern in the vector space. The theoretical value range is [-1, 1]. The closer the value is to 1, the more consistent the relation structure of the two is. The lower the value, the greater the difference between the two.

[0101] As an example, the kernel relation cosine similarity can be determined by the following formula: In the formula: For the first Kernel cosine similarity of remote sensing image samples in a target domain; For the first The student kernel matrix corresponding to each sample; For the first The teacher kernel matrix corresponding to each sample; Vectorization; This represents the L2 norm.

[0102] For example, when When the value is close to 1, it indicates that the vector directions of the student kernel matrix and the teacher kernel matrix are highly consistent, meaning that the student detection model has recovered the spatial relationship structure of the target domain image well.

[0103] The third sub-step is to determine the relation gap based on the cosine similarity of the kernel relation.

[0104] Preferably, the relationship gap can be determined by the following formula: In the formula: For the first The relationship gaps between remote sensing image samples of each target domain; This indicates the cosine similarity of the kernel relation. Crop to range .

[0105] For example, the cropping operation will Negative values ​​are truncated to 0, and values ​​greater than 1 are truncated to 1, thus reducing the relationship gap. It is a non-negative value, and its range is [value range missing]. .when At a higher level (student relationship structure is close to teacher relationship structure), The smaller the value, the smaller the subsequent dynamic weight; when At a lower level (when the student relationship structure differs significantly from the teacher relationship structure), When the value is close to 1, the subsequent dynamic weight is larger, and the distillation constraint is strengthened.

[0106] Of course, this application is not limited to this. In other embodiments, the relationship gap can also be determined by the following formula:

[0107] ; The second step is to generate dynamic weights based on the relationship gap.

[0108] Among them, dynamic weight refers to the sample-level loss adjustment coefficient obtained by mapping the relation gap as input through a preset function, which is used to adaptively adjust the distillation intensity according to the current alignment state of each target domain remote sensing image sample.

[0109] Preferably, the dynamic weights can be determined by the following formula: In the formula: For the first Dynamic weights corresponding to remote sensing image samples of the target domain; For the relationship gap; It is a dynamic weighted index.

[0110] For example, dynamic weight index The shape of the decay curve used to control the dynamic weights as the relationship gap changes can range from 1.0 to 3.0, preferably 1.5. When At that time, the dynamic weights and the relationship gap show a linear relationship.

[0111] The third step is to determine the relation distillation loss based on the dynamic weights, student kernel matrix, and teacher kernel matrix.

[0112] Preferably, the relation distillation loss can be determined by the following steps based on the dynamic weights, student kernel matrix, and teacher kernel matrix: The first sub-step involves determining the basic kernel relation differences based on the student kernel matrix and the teacher kernel matrix. For example, the basic kernel relation differences can be determined using the following formula: In the formula: For the first Student kernel matrix corresponding to each remote sensing image sample in the target domain; For the first Teacher kernel matrix corresponding to each remote sensing image sample in the target domain; For the first The fundamental kernel relation difference terms corresponding to each remote sensing image sample in the target domain; This indicates that the gradient operation has been stopped. Denotes the Frobenius norm; This represents the number of spatial tokens in a single remote sensing image of the target domain.

[0113] Additionally, it's necessary to note that the Frobenius norm is used to measure the element-wise overall difference between the student kernel matrix and the teacher kernel matrix. Divided by The purpose of normalization is to eliminate scale differences caused by different spatial resolutions, and to reduce the number of spatial tokens in an image. As the feature map resolution changes Normalization makes the differences in the basic kernel relation comparable for images with different spatial resolutions. The gradient stopping operation is applied to the teacher kernel matrix. This ensures that the teacher kernel matrix is ​​used only as a relational structure objective in loss calculation, and its gradient is not propagated back towards the teacher model.

[0114] The second sub-step involves determining the relation distillation loss based on the dynamic weights and fundamental kernel relation differences corresponding to the remote sensing image samples of each target domain. Here, relation distillation loss refers to the average loss of the samples before multiplication by the global relation distillation loss weights. As an example, the relation distillation loss can be determined by the following formula:

[0115] In the formula: For the first Dynamic weights corresponding to remote sensing image samples of the target domain; For the relationship of distillation loss; The number of remote sensing image samples for the target domain within the batch; For the first The fundamental kernel relation difference terms corresponding to each remote sensing image sample in the target domain.

[0116] Preferably, when the dynamic weights are used in the calculation of relational distillation loss, they are treated with a stopping gradient and used as adjustment coefficients for relational distillation loss, rather than participating in optimization as independently learnable parameters.

[0117] Additionally, it is worth noting that in other embodiments, the relation distillation loss can also be a power of the relation difference.

[0118] S109, Update the parameters of the student detection model based on the target loss, where the target loss includes relation distillation loss.

[0119] Through the above steps, this embodiment of the application can transfer relatively stable foreground semantic relationships from the teacher model (frozen remote sensing base model) to the student detector under the condition that the target domain is unlabeled. Compared with directly matching single-point features, this embodiment of the application matches the relational structure between spatial tokens within the image, thus not forcing the student detection model to replicate the channel representation of the teacher model, and can retain the feature degrees of freedom required for the detector to complete classification and localization. Through the superlinear response of the polynomial kernel, highly similar foreground token pairs occupy a higher weight in the distillation loss, and the influence of low-similarity background token pairs is suppressed, thereby alleviating the problem that the number of background regions in remote sensing images is much greater than that of foreground targets.

[0120] Specifically, in some embodiments, the target loss further includes detection supervision loss and domain adaptation loss. The detection supervision loss and domain adaptation loss are determined through the following steps:

[0121] The first step is to acquire a labeled source domain remote sensing image, which includes bounding box annotations and category annotations.

[0122] The second step is to input the source domain remote sensing image into the student detection model to obtain the source domain detection prediction results.

[0123] The third step is to determine the detection supervision loss based on the source domain detection prediction results and the annotation of the source domain remote sensing image.

[0124] The detection supervision loss refers to a loss function constructed based on the difference between the student detection model's predicted output of the source domain image and the ground truth annotation. It is used to constrain the student detection model to maintain the classification and localization capabilities required for object detection. For example, the detection supervision loss may include classification loss (e.g., cross-entropy loss or focus loss) and bounding box regression loss (e.g., L1 loss, Smooth L1 loss, or GIoU loss).

[0125] The fourth step involves using a domain discriminator to determine the domain adaptation loss by combining the source domain features extracted from the source domain remote sensing image and the target domain features extracted from the target domain remote sensing image by the student detection model.

[0126] Specifically, in some embodiments, the target loss can be determined by the following formula: In the formula: Loss to the target; To detect and monitor losses; For domain adaptation loss weights; Domain adaptation loss; Weights for distillation loss related to the relationship; This refers to the loss during distillation.

[0127] For example, relational distillation loss weights The relative importance of relational distillation loss, detection supervision loss, and domain adaptation loss is used to balance the relative importance of these losses, preferably set to 1.0. Domain adaptation loss weights The value is determined by the default settings of the underlying domain adaptation framework.

[0128] Additionally, it should be noted that in embodiments without a domain adaptation module, the domain adaptation loss can be omitted, thus allowing... In this case, the target loss consists of two parts: detection supervision loss and relation distillation loss. In another embodiment, The possible value is 1.

[0129] In each training iteration, the gradients of the relevant parameters of the student detection model are calculated based on the target loss, and then updated using an optimizer. Specifically, the set of parameters to be updated can be denoted as:

[0130] ; In the formula, The set of parameters to be updated; The shared feature extraction parameters of the student detection model include learnable parameters in the student detection model backbone network, feature projection layer, feature fusion layer, or encoder that are located before the selected intermediate feature layer and in the layer in which they are located; The main path parameters of the student detection model include learnable parameters in the detection decoder, classification prediction head, bounding box regression head, and overlap prediction head. These are optional parameters for the domain adaptation module, including learnable parameters in the domain discriminator, consistency constraint branch, or pseudo-label training branch.

[0131] Additionally, it is worth noting that if a student token adaptation layer or projection layer is set up, the parameters of the adaptation layer or projection layer are also part of the parameter set to be updated and are updated by relation distillation loss.

[0132] The optimizer can be a stochastic gradient descent, Adam, or AdamW optimizer, and can be updated using the following formula: In the formula: For the first The parameters of the student detection model in the next iteration; For the first The parameters of the student detection model are updated in the next iteration; Loss to target Regarding student detection model parameters The gradient; This indicates an optimizer update operation.

[0133] The parameters of the teacher model remain frozen during training and are not part of the parameter set to be updated. Specifically, the parameters of the teacher model are denoted as teacher parameters, which remain unchanged during training and can be determined by the following formula:

[0134] In the formula: For the first The parameters of the teacher model in the next iteration; For the first The parameters of the teacher model in the next iteration.

[0135] Finally, it is necessary to explain the order of the polynomial kernel. Dynamic weighting of the index Relationship between distillation loss weights Domain Adaptation Loss Weights These are preset hyperparameters or training scheduling parameters.

[0136] In summary, based on Figure 1 The method for training a cross-domain target detection model in remote sensing, as shown in this application, has at least the following technical effects: 1. By converting the teacher space token features and student space token features into similarity matrices between pairs of spatial location tokens in the image, and applying a positive odd-order polynomial kernel of order greater than 1 to each element of the teacher similarity matrix and student similarity matrix, the superlinear response property of the polynomial kernel can be utilized to maintain a large response in the kernel matrix for the originally high similarity between semantically similar foreground space tokens, while the originally close-to-zero similarity between weakly correlated background space tokens or between foreground and background space tokens is quickly suppressed.

[0137] Based on this, the relation distillation loss is determined according to the difference between the student kernel matrix and the teacher kernel matrix. This allows the difference in relations between highly similar foreground token pairs to dominate the relation distillation loss, thereby effectively suppressing the difference in relations between low-similarity background token pairs. In this way, background statistics no longer dominate the alignment loss, and the semantic relation structure between foreground tokens can be preferentially transferred to the student detection model. This solves the problem of insufficient transfer of foreground semantic relation structure caused by the dominant role of background statistics in the alignment loss, even when the target domain is unlabeled.

[0138] 2. By determining the relationship gap between the student kernel matrix and the teacher kernel matrix, generating dynamic weights based on this gap, and then determining the relation distillation loss based on these dynamic weights, the strength of the relation distillation loss can be positively correlated with the current degree of recovery of the teacher model's relation structure by the student detection model. When the relationship gap between the student kernel matrix and the teacher kernel matrix is ​​large in the early stages of training, the dynamic weights increase accordingly, and the relation distillation loss provides a strong structural transfer constraint. Conversely, as the student kernel matrix gradually approaches the teacher kernel matrix and the relationship gap decreases in the later stages of training, the dynamic weights automatically decrease, and the constraint strength of the relation distillation loss weakens accordingly. In this way, the transfer of foreground semantic relation structure can be enhanced in the early stages of training, while the distillation constraint is automatically weakened in the later stages, reducing the interference of the relation distillation loss on the classification and localization optimization of the main detection task.

[0139] 3. By performing fixed weak augmentation on the target domain remote sensing image and then inputting it into the teacher model and student detection model respectively, the teacher model and student detection model can process fixed weak augmented views of the target domain with the same spatial content and consistent layout. Based on this, the student detection model extracts student spatial token features through the student weak augmentation branch. Utilizing the characteristic that the student weak augmentation branch and the student detection model share feature extraction parameters, the relation distillation gradient can be backpropagated to the shared parameters of the student detection model during the training phase via the student weak augmentation branch, without the need to introduce an additional independent student auxiliary model. In this way, the student weak augmentation branch can be avoided as a trainable network parallel to the teacher model and student detection model.

[0140] 4. By freezing the parameters of the teacher model during training and using either online extraction followed by stopping gradient processing or offline pre-extraction to obtain teacher space token features, it can be ensured that the teacher model is not affected by the parameter updates of the student detection model during target domain training and is not contaminated by early prediction noise in the target domain. This allows the extracted teacher space token features to provide a more stable semantic relationship structure. Consequently, the teacher model can consistently provide stable external structural anchors for unlabeled samples in the target domain, reducing the dependence of foreground semantic relationship structure transfer on the prediction quality of the target domain detector itself.

[0141] The technical solutions of the embodiments of this application will be further described below with reference to specific examples.

[0142] Example 1: Cross-satellite remote sensing target detection training Taking cross-satellite remote sensing target detection as an example, the source domain image is a labeled image from satellite platform A, and the target domain remote sensing image is an unlabeled image from satellite platform B. Before training, a frozen remote sensing base model is used to extract teacher spatial token features from the target domain remote sensing images, and these features are saved according to the target domain image number. The teacher model is not updated throughout the training process and is used to provide a stable semantic relationship structure for the target domain.

[0143] When training the student detection model, a fixed weakly augmented view is generated for remote sensing images of the same target domain, for example, uniformly scaled to a preset size. Pixels are then normalized. This fixed weak enhancement view... Figure 1 On one hand, the model takes the teacher as input or reads the corresponding offline teacher features; on the other hand, the weak enhancement branch of the student detection model is input. Student spatial token features are extracted in the intermediate feature layer corresponding to the teacher token resolution. After spatial layout alignment, the teacher spatial token features and student spatial token features are flattened into token sequences, and then spatial mean centering and L2 normalization are performed.

[0144] Subsequently, the teacher similarity matrix and student similarity matrix are calculated separately, and a cubic polynomial kernel is applied to obtain the teacher kernel matrix and student kernel matrix. For example, the similarity between two vehicle foreground tokens in the teacher model is... After three nuclear transformations, it becomes It maintains a strong influence even in losses; the similarity of a certain background token pair is... After three nuclear transformations, it becomes Its impact is significantly reduced. Finally, the cosine similarity of the kernel relation between the student kernel matrix and the teacher kernel matrix is ​​calculated. According to the relationship gap Generate dynamic weights Combined with the basic kernel relation difference term Obtain the relationship distillation loss The student detection model parameters are updated only by backpropagating along with the source domain detection supervision loss. During the inference phase, only the main path of the student detection model is used; the teacher model, the student weak enhancement branch, and all kernel matrix calculation modules are not executed.

[0145] Example 2: Satellite to UAV Scene Migration In satellite-to-drone scene migration, the target scale, texture sharpness, and shooting angle of the target domain drone imagery may differ significantly from those of the source domain satellite imagery. If global feature alignment is performed directly, large areas of roads, vegetation, sea surfaces, or building backgrounds will account for the major contribution to the loss.

[0146] In this embodiment, the frozen remote sensing base model is used to extract the spatial token relationship structure in the target domain remote sensing image. Since similar vehicle, ship, or aircraft regions typically have high similarity in the teacher feature space, the multinomial kernel preferentially amplifies these highly similar foreground relationships; weak correlations between background regions or between the background and foreground are suppressed. Thus, the student detection model can still obtain foreground relationship structure guidance even without target domain annotation.

[0147] In this embodiment, the student weak enhancement branch can select intermediate feature layers before and after the detector encoder input to balance spatial details and semantic context. If the teacher token resolution is higher than the student feature layer resolution, region average pooling can be performed on the teacher spatial token features; if the teacher token resolution is lower than the student feature layer resolution, pooling can be performed on the student features or interpolation can be performed on the teacher features. A similarity matrix is ​​constructed after ensuring that the number of aligned spatial tokens remains consistent. Dynamic weight exponent. Can be set to Relationship between distillation loss weights Can be set to The parameters described above are merely illustrative embodiments and do not constitute a limitation on the scope of protection. They can be adjusted according to the target scale, image resolution, and training stability.

[0148] Based on the same inventive concept as the training method for a remote sensing cross-domain target detection model in the foregoing embodiments, please refer to... Figure 2 This application provides a training device for a remote sensing cross-domain target detection model, comprising: Image acquisition module 201 is used to acquire unlabeled remote sensing images of the target domain; Teacher feature extraction module 202 is used to input the target domain remote sensing image into the teacher model to obtain teacher spatial token features; Student feature extraction module 203 is used to input the target domain remote sensing image into the student detection model to obtain student spatial token features; The spatial alignment module 204 is used to align the spatial layout of the teacher's spatial token features and the student's spatial token features so that the spatial tokens at corresponding spatial positions correspond one-to-one. The first matrix determination module 205 is used to determine the teacher similarity matrix corresponding to the target domain remote sensing image based on the similarity between all pairs of spatial location tokens in the teacher spatial token features after spatial layout alignment. The second matrix determination module 206 is used to determine the student similarity matrix corresponding to the target domain remote sensing image based on the similarity between all pairs of spatial location tokens in the student spatial token features after spatial layout alignment. The kernel matrix determination module 207 is used to apply a positive odd-order polynomial kernel of order greater than 1 to each element of the teacher similarity matrix and the student similarity matrix to obtain the teacher kernel matrix and the student kernel matrix. The distillation loss determination module 208 is used to determine the relational distillation loss based on the difference between the student kernel matrix and the teacher kernel matrix. The parameter update module 209 is used to update the parameters of the student detection model according to the target loss, wherein the target loss includes relation distillation loss.

[0149] In some embodiments, the distillation loss determination module 208 is specifically used for: Determine the relationship gap between the student kernel matrix and the teacher kernel matrix; Dynamic weights are generated based on the relationship gap; The relation distillation loss is determined based on the dynamic weights, student kernel matrix, and teacher kernel matrix.

[0150] In some embodiments, the distillation loss determination module 208 is further configured to: Vectorize the student kernel matrix and the teacher kernel matrix respectively to obtain the student kernel vector and the teacher kernel vector; The cosine similarity of the kernel relation is determined based on the student kernel vector and the teacher kernel vector. The relationship gap is determined based on the cosine similarity of the nuclear relationship.

[0151] In some embodiments, the kernel relation cosine similarity is determined by the following formula: In the formula, For the first Kernel cosine similarity of remote sensing image samples in a target domain; For the first The student kernel matrix corresponding to each sample; For the first The teacher kernel matrix corresponding to each sample; Vectorization; This represents the L2 norm.

[0152] In some embodiments, vectorization is performed using only the off-diagonal elements or upper triangular off-diagonal elements of the student kernel matrix and the teacher kernel matrix.

[0153] In some embodiments, the relationship gap is determined by the following formula: In the formula, For the first The relationship gaps between remote sensing image samples of each target domain; This indicates the cosine similarity of the kernel relation. Crop to range .

[0154] In some embodiments, the dynamic weights are determined by the following formula: In the formula, For the first Dynamic weights corresponding to remote sensing image samples of the target domain; For the relationship gap; It is a dynamic weighted index.

[0155] In some embodiments, the distillation loss determination module 208 is further configured to: Based on the student kernel matrix and the teacher kernel matrix, determine the basic kernel relation differences; The relation distillation loss is determined based on the dynamic weights and basic kernel relation differences corresponding to the remote sensing image samples of each target domain.

[0156] In some embodiments, the basic kernel relation difference term is determined by the following formula: In the formula, For the first Student kernel matrix corresponding to each remote sensing image sample in the target domain; For the first Teacher kernel matrix corresponding to each remote sensing image sample in the target domain; For the first The fundamental kernel relation difference terms corresponding to each remote sensing image sample in the target domain; This indicates that the gradient operation has been stopped. Denotes the Frobenius norm; This represents the number of spatial tokens in a single remote sensing image of the target domain.

[0157] In some embodiments, the relational distillation loss is determined by the following formula: In the formula, For the first Dynamic weights corresponding to remote sensing image samples of the target domain; For the relationship of distillation loss; The number of remote sensing image samples for the target domain within the batch; For the first The fundamental kernel relation difference terms corresponding to each remote sensing image sample in the target domain.

[0158] In some embodiments, the dynamic weights, after stopping gradient processing, are used as adjustment coefficients for the relational distillation loss when participating in the calculation of relational distillation loss.

[0159] In some embodiments, the kernel matrix determination module 207 is specifically used for: For each pair of spatial location tokens in the teacher similarity matrix and student similarity matrix, a positive odd-order power transformation is performed to superlinearly amplify the similarity relationship of highly similar foreground token pairs, suppress the similarity relationship of low-similarity background token pairs, and retain the sign of negative correlation.

[0160] In some embodiments, the positive odd-order power transform is determined by the following formula: In the formula, and The first and second kernels in the student kernel matrix and teacher kernel matrix, respectively. The spatial location token and the first Each spatial location token corresponds to an element; and These are the first two numbers in the student similarity matrix and the teacher similarity matrix, respectively. The spatial location token and the first Similarity between tokens in spatial locations; Let be the order of the kernel of a positive odd-order polynomial. It is a positive odd number greater than 1.

[0161] In some embodiments, the order of the positive odd-order polynomial kernel The value is 3.

[0162] In some embodiments, the target domain remote sensing image is input into the teacher model and the student detection model after being processed by fixed weak enhancement; wherein, the fixed weak enhancement process includes deterministic scaling, filling, cropping and normalization, and does not employ random strong geometric transformations, and does not perform large-angle rotation, random perspective or CutMix operations that would change the spatial token correspondence.

[0163] In some embodiments, the parameters of the teacher model are kept frozen during training; wherein, the teacher space token features are obtained by online extraction or offline pre-extraction; when the online extraction method is used, the target domain fixed weak augmentation view is input into the teacher model for forward propagation to obtain the initial teacher token features, and the initial teacher token features are subjected to stopping gradient processing, and the initial teacher token features after stopping gradient processing are used as the teacher space token features. When using the offline pre-extraction method, the teacher token features of the pre-stored target domain remote sensing image are read as teacher spatial token features. The pre-stored teacher token features are features extracted and stored from the target domain remote sensing image by the teacher model before training.

[0164] In some embodiments, the first matrix determining module 205 is specifically used for: Flatten the teacher space token features into a teacher token sequence; Subtract the spatial mean of all spatial location tokens in the corresponding image from the teacher token sequence, and then perform L2 normalization to obtain the normalized teacher token sequence. Based on the pairwise similarity between all spatial location tokens in the normalized teacher token sequence, the teacher similarity matrix corresponding to the remote sensing image of the target domain is determined.

[0165] In some embodiments, the teacher token sequence is determined by the following formula: In the formula, This is the normalized sequence of teacher tokens; The token characteristics of the teacher's space after spatial layout alignment; Indicates the flattening operation; This indicates spatial centralization, subtracting the mean; This indicates L2 normalization.

[0166] In some embodiments, the second matrix determining module 206 is specifically used for: Flatten the student space token features into a student token sequence; Subtract the spatial mean of all spatial location tokens within the corresponding image from the student token sequence, and then perform L2 normalization to obtain the normalized student token sequence. Based on the pairwise similarity between all spatial location tokens in the normalized student token sequence, the student similarity matrix corresponding to the remote sensing image of the target domain is determined.

[0167] In some embodiments, the normalized student token sequence is determined by the following formula: In the formula, This is the normalized sequence of student tokens; The token characteristics of the student space after spatial layout alignment; Indicates the flattening operation; ; This indicates L2 normalization.

[0168] In some embodiments, when the spatial dimensions of the teacher space token feature and the student space token feature are different, spatial layout alignment is achieved by at least one of bilinear interpolation, region average pooling, or token sampling.

[0169] In some embodiments, the target loss further includes detection supervision loss and domain adaptation loss; the parameter update module 209 is further configured to: Acquire labeled source domain remote sensing images, which include bounding box annotations and category annotations; Input the source domain remote sensing image into the student detection model to obtain the source domain detection prediction result; Based on the source domain detection prediction results and the annotation of the source domain remote sensing image, the detection supervision loss is determined; The source domain features extracted from the source domain remote sensing image and the target domain features extracted from the target domain remote sensing image by the student detection model are used to determine the domain adaptation loss through a domain discriminator.

[0170] In some embodiments, the target loss is determined by the following formula: ; In the formula, Loss to the target; To detect and monitor losses; For domain adaptation loss weights; Domain adaptation loss; Weights for distillation loss related to the relationship; This refers to the loss during distillation.

[0171] It is understandable that the modules recorded in the training device for the remote sensing cross-domain target detection model are similar to... Figure 1 The steps described correspond to those in the training method for the remote sensing cross-domain target detection model. Therefore, the operations, features, and beneficial effects described above also apply to the training apparatus and its modules for the remote sensing cross-domain target detection model, and will not be repeated here.

[0172] The modules described in some embodiments of this application can be implemented in software or hardware. The described modules can also be housed in a processing device. Furthermore, the names of these modules do not necessarily limit the module itself; for example, the image acquisition module 201 can also be described as an "image receiving module".

[0173] Based on the same inventive concept as the training method for a cross-domain target detection model in the foregoing embodiments, this application provides a cross-domain remote sensing image detection method, including the following steps: The first step is to acquire the remote sensing image to be detected; The second step is to input the remote sensing image to be detected into the student detection model to obtain the target detection result; wherein the student detection model is trained according to the training method described in any of the above embodiments.

[0174] In some embodiments, during the inference phase, only the main path of the student detection model is retained for object detection. The teacher model, the student weak enhancement branch, as well as token flattening and normalization operations, pairwise similarity matrix calculation, multinomial kernel matrix calculation, relation gap and dynamic weight calculation, and relation distillation loss calculation are not performed during the inference phase; for example, they can be deleted or turned off.

[0175] Specifically, during inference, the remote sensing image of the target domain is input into the student detection model after conventional image preprocessing. It is then processed sequentially through the backbone network, encoder or feature fusion module, detection decoder and detection head, and the target bounding box, target category and confidence score are directly output.

[0176] This setup, by omitting the execution of teacher model, student weak enhancement branch, token flattening and normalization operations, pairwise similarity matrix calculation, multinomial kernel matrix calculation, relation gap and dynamic weight calculation, and relation distillation loss calculation during the inference phase, and retaining only the main path of the student detection model for target detection, can prevent additional computational overhead during the deployment phase and avoid increasing the complexity of the model structure.

[0177] Please see Figure 3Based on the same inventive concept as the training method for a cross-domain target detection model or the cross-domain remote sensing image detection method in the foregoing embodiments, this application provides an electronic device. This electronic device may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), etc., and fixed terminals such as digital TVs, desktop computers, etc. The electronic device includes a processing unit 301 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM 302 (read-only memory) or a program loaded from storage device 308 into RAM 303 (random access memory). RAM 303 also stores various programs and data required for the operation of the electronic device. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output interface (i.e., I / O interface 305) is also connected to the bus 304.

[0178] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touch screens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data.

[0179] In particular, embodiments of this application also provide a computer-readable storage medium storing a computer program, which is implemented when executed by the processing device 301. Figure 1 The method shown is for training a remote sensing cross-domain target detection model or for cross-domain remote sensing image detection.

[0180] The computer-readable medium described in some embodiments of this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device or apparatus, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CDROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0181] In some embodiments of this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be executed by instructions, used by a device or apparatus, or used in conjunction with it. In some embodiments of this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with instructions, devices or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0182] Additionally, it is worth noting that the aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently without being assembled into the electronic device.

[0183] Finally, it is necessary to note that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of the methods according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using dedicated hardware-based implementations that perform the specified functions or operations, or can be implemented using a combination of dedicated hardware and computer instructions.

[0184] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A training method for a remote sensing cross-domain target detection model, characterized in that, Includes the following steps: Acquire unlabeled remote sensing images of the target domain; The target domain remote sensing image is input into the teacher model to obtain the teacher spatial token features; The target domain remote sensing image is input into the student detection model to obtain student spatial token features; The teacher space token feature and the student space token feature are spatially aligned to ensure that the space tokens in corresponding spatial locations correspond one-to-one. Based on the similarity between all pairs of spatial location tokens in the teacher spatial token features after spatial layout alignment, determine the teacher similarity matrix corresponding to the target domain remote sensing image; Based on the similarity between all pairs of spatial location tokens in the student spatial token features after spatial layout alignment, determine the student similarity matrix corresponding to the target domain remote sensing image; Apply a positive odd-order polynomial kernel of order greater than 1 to each element of the teacher similarity matrix and the student similarity matrix to obtain the teacher kernel matrix and the student kernel matrix, respectively. The relationship distillation loss is determined based on the difference between the student kernel matrix and the teacher kernel matrix; The parameters of the student detection model are updated based on the target loss, wherein the target loss includes the relation distillation loss.

2. The training method for a remote sensing cross-domain target detection model according to claim 1, characterized in that, The step of determining the relation distillation loss based on the difference between the student kernel matrix and the teacher kernel matrix includes: Determine the relationship gap between the student kernel matrix and the teacher kernel matrix; Dynamic weights are generated based on the relationship gaps; The relation distillation loss is determined based on the dynamic weights, the student kernel matrix, and the teacher kernel matrix.

3. The training method for a remote sensing cross-domain target detection model according to claim 2, characterized in that, The step of determining the gap between the student kernel matrix and the teacher kernel matrix includes: The student kernel matrix and the teacher kernel matrix are vectorized respectively to obtain the student kernel vector and the teacher kernel vector; The cosine similarity of the kernel relationship is determined based on the student kernel vector and the teacher kernel vector. The relationship gap is determined based on the cosine similarity of the kernel relationship.

4. The training method for a remote sensing cross-domain target detection model according to claim 3, characterized in that, The vectorization is performed using only the off-diagonal elements or upper triangular off-diagonal elements of the student kernel matrix and the teacher kernel matrix.

5. The training method for a remote sensing cross-domain target detection model according to claim 2, characterized in that, The distillation loss is determined by the following formula: In the formula, For the first The dynamic weights corresponding to each remote sensing image sample in the target domain; The relationship is the distillation loss; The number of remote sensing image samples for the target domain within the batch; For the first The fundamental kernel relation difference terms corresponding to each remote sensing image sample in the target domain.

6. The training method for a remote sensing cross-domain target detection model according to claim 1, characterized in that, The target loss also includes detection supervision loss and domain adaptation loss, which are determined through the following steps: Acquire a source domain remote sensing image with annotations, the source domain remote sensing image including detection box annotations and category annotations; The source domain remote sensing image is input into the student detection model to obtain the source domain detection prediction result; Based on the source domain detection prediction results and the annotation of the source domain remote sensing image, the detection supervision loss is determined; The source domain features extracted from the source domain remote sensing image and the target domain features extracted from the target domain remote sensing image by the student detection model are used to determine the domain adaptation loss through a domain discriminator.

7. A training device for a remote sensing cross-domain target detection model, characterized in that, include: The image acquisition module is used to acquire unlabeled remote sensing images of the target domain; The teacher feature extraction module is used to input the target domain remote sensing image into the teacher model to obtain teacher spatial token features; The student feature extraction module is used to input the target domain remote sensing image into the student detection model to obtain student spatial token features; The spatial alignment module is used to perform spatial layout alignment on the teacher space token feature and the student space token feature, so that the space tokens at corresponding spatial positions correspond one-to-one. The first matrix determination module is used to determine the teacher similarity matrix corresponding to the target domain remote sensing image based on the similarity between all pairs of spatial location tokens in the teacher spatial token features after spatial layout alignment. The second matrix determination module is used to determine the student similarity matrix corresponding to the target domain remote sensing image based on the similarity between all pairs of spatial location tokens in the student spatial token features after spatial layout alignment. The kernel matrix determination module is used to apply a positive odd-order polynomial kernel of order greater than 1 to each element of the teacher similarity matrix and the student similarity matrix to obtain the teacher kernel matrix and the student kernel matrix. The distillation loss determination module is used to determine the relationship distillation loss based on the difference between the student kernel matrix and the teacher kernel matrix. The parameter update module is used to update the parameters of the student detection model according to the target loss, wherein the target loss includes the relation distillation loss.

8. A cross-domain remote sensing image detection method, characterized in that, Includes the following steps: Acquire the remote sensing image to be detected; The remote sensing image to be detected is input into the student detection model to obtain the target detection result; wherein the student detection model is trained by the training method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, include: One or more processing devices; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processing devices, the one or more processing devices implement the method as claimed in claim 8 or any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processing device, implements the method of claim 8 or any one of claims 1 to 6.

Images