Remote sensing multi-task target detection method and device based on multimodality

Abstract

The present application relates to a multi-modal remote sensing multi-task target detection method and device. The method comprises: constructing a remote sensing multi-task target detection model; the remote sensing multi-task target detection model comprises a parameter-sharing image encoder, a multi-task head network and a dynamic adjustment module; the parameter-sharing image encoder comprises a network layer, a convolutional layer and an image encoder with a grid-level sparse hybrid expert network; inputting remote sensing images of multiple modalities into the parameter-sharing image encoder for image encoding to obtain output features; inputting the output features into the multi-task head network for target detection, calculating the loss of each task in the target detection process, and dynamically adjusting the learning rate of each task head in the dynamic adjustment module with the loss of the task as the target to optimize the convergence speed and directional consistency of the remote sensing multi-task target detection model, and using the optimized remote sensing multi-task target detection model to realize multi-task target detection. The use of this method can improve the efficiency of multi-target detection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-modal remote sensing multi-task target detection method and device. Background Art

[0002] Remote sensing target detection typically involves multiple sensors employing different imaging mechanisms, resulting in diverse data modalities. Traditional detection models are typically developed for specific datasets associated with a single modality and predefined format for detection tasks. Such traditional approaches, such as Yimian Dai's "Asymmetric contextual modulation for infrared small target detection," ignore the value of joint knowledge inherent in unified remote sensing environments. Furthermore, aerial platforms such as drones and satellites are often equipped with multiple sensors, making it crucial to simultaneously process images from various modalities. Multi-source target detection methods, such as Jinming Liu's "Multi-source remote sensing image fusion for ship target detection and recognition" and Hongkang Zhang's "Hgr correlation pooling fusion framework for recognition and classification in multimodal remote sensing data," primarily rely on scarce, impractical, and inflexible spatially aligned paired images and spatial alignment algorithms. These methods are also limited to performing detection tasks in a single format. Therefore, developing a unified model that does not rely on spatially aligned image pairs and can handle detection tasks in multiple modalities and formats (hereafter referred to as "multi-task") is highly desirable, but this area has remained understudied. To fill this research gap, we propose a new task, called Multimodal Dataset and Multitask Object Detection, which aims to detect objects in any given image, regardless of its modality and whether its predefined detection task is horizontal boxes or oriented boxes.

[0003] This task is closely related to two key research areas: multi-dataset object detection and multi-task learning. However, its task has unique challenges. In traditional multi-dataset object detection, such as Georgios Kapidis's "Multi-dataset, multitask learning of egocentric vision tasks" and Ren'e Ranftl's "Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer", although images may have different properties (such as natural images and paintings), they often share similar underlying concepts (optical concepts). Simple joint training methods are often effective, and the performance of a single model trained on the combined dataset is often better than that of a model trained on a single dataset. Xudong Wang adopts specialized attention mechanism layers in "Toward universal object detection by domain attention" that serve as domain-specific attention mechanisms for each dataset. Hang Xu's "Universal-rcnn: Universal object detector via transferable graph r-cnn" introduces a partitioned detector trained across multiple datasets, integrating features through a graph-based cross-dataset attention module. Xingyi Zhou’s “Simple multi-dataset detection” advances this concept by proposing a unified label space and emphasizing the importance of batch sampling strategies.

[0004] In contrast, multimodal datasets in remote sensing (e.g., visible light, synthetic aperture radar, infrared imagery) exhibit fundamentally different notions of modality. While these modalities may share some common knowledge, significant differences in data representation lead to a large modality gap, which complicates the integration of cross-modal information. In addition, remote sensing datasets typically contain multiple annotation types, such as horizontal boxes and oriented boxes, further increasing the complexity of model learning. These challenges may hinder the learning and optimization of traditional models in the following ways: 1) Representation constraints: A dense model that shares the same parameters across multiple tasks and modalities may be limited in representation capabilities, as a single set of parameters may have difficulty effectively fitting the diverse distribution inherent in each dataset. 2) Optimization inconsistency: Differences in learning difficulty between different modalities and tasks may lead to asynchronous optimization rates or optimization directions for various components of the model. This inconsistency may lead to conflicting optimization results, affecting the model's ability to achieve different loss targets. Summary of the Invention

[0005] Based on this, it is necessary to provide a multi-modal remote sensing multi-task target detection method and device that can improve the efficiency of multi-target detection in order to address the above technical problems.

[0006] A multi-task target detection method for remote sensing based on multimodality, the method comprising:

[0007] Acquire a multimodal dataset; the multimodal dataset includes remote sensing images of multiple modalities; construct a remote sensing multi-task target detection model; the remote sensing multi-task target detection model includes a parameter-shared image encoder, a multi-task head network, and a dynamic adjustment module; the parameter-shared image encoder includes a network layer, a convolutional layer, and an image encoder with a grid-level sparse mixture of experts network;

[0008] The remote sensing image input parameters of multiple modalities are shared by the image encoder to perform image encoding and obtain output features;

[0009] The output features are input into the multi-task head network for target detection. The loss of each task in the target detection process is calculated. In the dynamic adjustment module, the learning rate of each task head is dynamically adjusted with the loss of the task as the target to optimize the convergence speed and direction consistency of the remote sensing multi-task target detection model. The optimized remote sensing multi-task target detection model is used to realize multi-task target detection.

[0010] A multi-modal remote sensing multi-task target detection device, comprising:

[0011] Build a detection model module to acquire a multimodal dataset; the multimodal dataset includes remote sensing images of multiple modalities; build a remote sensing multi-task target detection model; the remote sensing multi-task target detection model includes a parameter-shared image encoder, a multi-task head network, and a dynamic adjustment module; the parameter-shared image encoder includes a network layer, a convolutional layer, and an image encoder with a grid-level sparse mixture of experts network;

[0012] An image encoding module is used to encode the input parameters of remote sensing images of multiple modalities using a shared image encoder to obtain output features;

[0013] The model optimization and target detection module is used to input the output features into the multi-task head network for target detection, calculate the loss of each task in the target detection process, and dynamically adjust the learning rate of each task head based on the task loss in the dynamic adjustment module to optimize the convergence speed and direction consistency of the remote sensing multi-task target detection model. The optimized remote sensing multi-task target detection model is used to realize multi-task target detection.

[0014] The above-mentioned multi-task target detection method and device for remote sensing based on multimodality, this application constructs a unified architecture including a parameter-sharing image encoder, a multi-task head network and a dynamic adjustment module. The grid-level sparse hybrid expert network in the parameter-sharing image encoder calls different expert networks based on the image region features, accurately extracts the image features of each modality, bridges the modal difference gap, solves the problem of large differences in multimodal data representation and obvious modal gaps, and reduces the computational burden. The multi-task head network processes tasks of different annotation types separately, and combines the dynamic adjustment module to dynamically adjust the learning rate according to the task loss, optimizes the convergence speed and direction consistency between different tasks, overcomes the optimization inconsistency problem of traditional models in multi-task learning, and enables the model to accurately learn the detection method of different labeled targets to cope with the complexity brought by multiple annotation types of remote sensing data. A single model is achieved for efficient processing of multimodality and multi-task. On the one hand, it has strong representation capabilities and efficient computation, overcoming the difficulties of integrating multimodal data and waste of resources faced by traditional methods; on the other hand, by optimizing the task convergence characteristics, it ensures stable optimization of each task, solves the dilemma that traditional models cannot take into account multimodality and multitasking, reduces the complexity of model deployment, improves practicality and versatility, and meets the needs of multi-task target detection in multimodal remote sensing images in practical applications, filling the research gap in related fields and providing innovative solutions for the development of remote sensing target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 1 is a flow chart of a multi-task target detection method for remote sensing based on multi-modality in one embodiment;

[0016] Figure 2 Schematic diagram of a remote sensing multi-task target detection model in one embodiment;

[0017] Figure 3 Schematic diagram showing a comparison between a grid-level sparse hybrid expert network and an image-level sparse hybrid expert network in one embodiment;

[0018] Figure 4 A schematic diagram illustrating routing visualization of a grid-level sparse hybrid expert network for image inputs of different modalities in another embodiment;

[0019] Figure 5 The figure is a structural block diagram of a multi-modal remote sensing multi-task target detection device in one embodiment. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0021] In one embodiment, Figure 1 As shown, a multi-modal remote sensing multi-task target detection method is provided, which includes the following steps:

[0022] Step 102, obtaining a multimodal dataset; the multimodal dataset includes remote sensing images of multiple modalities; constructing a remote sensing multi-task target detection model; the remote sensing multi-task target detection model includes a parameter-sharing image encoder, a multi-task head network and a dynamic adjustment module; the parameter-sharing image encoder includes a network layer, a convolutional layer and an image encoder with a grid-level sparse hybrid expert network.

[0023] In view of the significant differences in the essentially different mode concepts and data representations between remote sensing multimodal datasets (such as visible light, synthetic aperture radar, and infrared images), this application constructs the following Figure 2 The remote sensing multi-task target detection model shown in Figure 1 plays a key role in this. A grid-level sparse hybrid expert network in the parameter-sharing image encoder plays a key role. Through a dynamic routing mechanism, only a small number of experts relevant to the input features are activated, enabling cross-modal knowledge sharing and efficient extraction of modality-specific features. The image is divided into grid regions, and different expert networks are invoked to process different regions based on their characteristics. This effectively captures the unique features of each modality image in different regions, thus overcoming the feature extraction barriers caused by modality differences. For example, texture features in synthetic aperture radar images and color and shape features in visible light images can be accurately extracted using appropriate expert networks. This enables the model to obtain more representative and discriminative features when faced with data from different modalities, providing strong support for subsequent target detection. The grid-level sparse hybrid expert network improves the model's representational capabilities by specifically extracting multimodal image features, enabling it to better handle the feature diversity and complexity introduced by modality differences. At the same time, its sparse calling of expert networks avoids processing all features to the same extent, reduces the computational burden, and enables the model to run efficiently when processing complex multimodal data. It overcomes the difficulties in information integration and waste of computing resources caused by modal differences in traditional methods when facing multimodal data.

[0024] Step 104 : The remote sensing image input parameters of multiple modalities are shared by an image encoder for image encoding to obtain output features.

[0025] In step 106, the output features are input into the multi-task head network for target detection, the loss of each task in the target detection process is calculated, and the learning rate of each task head is dynamically adjusted with the task loss as the target in the dynamic adjustment module to optimize the convergence speed and direction consistency of the remote sensing multi-task target detection model. The optimized remote sensing multi-task target detection model is used to realize multi-task target detection.

[0026] Considering that remote sensing datasets typically contain multiple annotation types (such as horizontal and oriented boxes) and that learning difficulty varies between tasks, the multi-task head network processes different detection tasks (such as horizontal and oriented box detection) separately. During the object detection process, it calculates the loss for each task and passes these loss values ​​to the dynamic adjustment module. The dynamic adjustment module dynamically adjusts the learning rate of each task head based on the task loss, ensuring that each task head can adaptively optimize learning based on the characteristics and difficulty of its own task. For example, when the target task of detecting horizontal box annotations is difficult to learn, the dynamic adjustment module will appropriately increase the learning rate of this task head to accelerate its convergence so that it can keep up with the learning progress of other tasks, avoiding conflicting optimization results caused by inconsistent learning progress between tasks and ensuring that the model can accurately learn the characteristics and detection methods of targets with different annotation types. The dynamic adjustment module's dynamic adjustment of the learning rate of the multi-task head network significantly optimizes the convergence speed and directional consistency between tasks. By dynamically adjusting the learning rate according to the task loss, each task head can be optimized at a more appropriate speed and direction, avoiding the optimization inconsistency problem that occurs in traditional models in multi-task learning due to differences in the learning difficulty of different tasks. This ensures that the model can simultaneously and efficiently learn detection tasks for targets of different labeled types, improving the model's learning and optimization effects on complex remote sensing datasets.

[0027] In addition, the unified model designed in this application changes the limitations of traditional methods in developing models for single modalities and predefined format detection tasks. A single model can simultaneously process multimodal image inputs with high efficiency and performance, and support multi-task detection at the same time. This not only improves the versatility and practicality of the model, but also reduces the complexity of model deployment. Compared with the traditional method of building and deploying multiple models for different modalities and tasks, only one unified model needs to be deployed to meet the needs of multiple remote sensing image modalities and multiple target detection tasks, reducing application costs and system complexity, and effectively solving the problem that traditional models cannot handle multiple modalities and multiple tasks at the same time.

[0028] In the above-mentioned multi-task target detection method based on multimodality, the above-mentioned multi-task target detection method and device based on multimodality, this application constructs a unified architecture including a parameter-sharing image encoder, a multi-task head network and a dynamic adjustment module. The grid-level sparse hybrid expert network in the parameter-sharing image encoder calls different expert networks based on the image region features, accurately extracts the image features of each modality, bridges the modality difference gap, solves the problem of large differences in multimodal data representation and obvious modality gaps, and reduces the computational burden. The multi-task head network processes tasks of different annotation types separately, and combines the dynamic adjustment module to dynamically adjust the learning rate according to the task loss, optimizes the convergence speed and direction consistency between different tasks, overcomes the optimization inconsistency problem of traditional models in multi-task learning, and enables the model to accurately learn the detection method of different labeled targets to cope with the complexity brought by multiple annotation types of remote sensing data. A single model is achieved for efficient processing of multimodality and multi-tasks. On the one hand, it has strong representation capabilities and efficient computation, overcoming the difficulties of integrating multimodal data and waste of resources faced by traditional methods; on the other hand, by optimizing the task convergence characteristics, it ensures stable optimization of each task, solves the dilemma that traditional models cannot take into account multimodality and multitasking, reduces the complexity of model deployment, improves practicality and versatility, and meets the needs of multi-task target detection in multimodal remote sensing images in practical applications, filling the research gap in related fields and providing innovative solutions for the development of remote sensing target detection.

[0029] In one embodiment, remote sensing image input parameters of multiple modalities are shared by an image encoder for image encoding to obtain output features, including:

[0030] Encode the remote sensing image according to the network layer and convolution layer in the parameter sharing image encoder to obtain deep image features;

[0031] The deep image features are fed into an image encoder with a grid-level sparse mixture of experts network to obtain output features.

[0032] In a specific embodiment, grid-level features are dynamically routed to expert network modules dedicated to their specific features for input features deep within the network. Features passing through the deep layers of the neural network are weighted by the outputs of multiple expert network modules. The grid-level routing network dynamically determines and selects the most relevant expert for each local feature. A grid-level sparse hybrid expert network creates a sparser feature space within the image encoding module. By focusing on local patterns, the model can independently learn multiple modalities and local target patterns. This design effectively addresses the challenges of a crowded feature space and enhances the model's expressive power. Figure 3 Schematic diagram comparing a grid-level sparse hybrid expert network and an image-level sparse hybrid expert network in one embodiment.

[0033] In one embodiment, deep image features are input into an image encoder with a grid-level sparse mixture of experts network to obtain output features including:

[0034] The deep image features are input into the image encoder with the grid-level sparse mixture of experts network, and the output features are

[0035]

[0036] in:

[0037]

[0038] N is the total number of experts, G is the gating function, represents the nth 1×1 convolution expert, each expert has an embedding representation in the matrix E, x ij Represents the local spatial input feature of the i-th row and j-th column in the depth image feature, TOP k Indicates the selection of k experts with the highest probability.

[0039] In a specific embodiment, the input feature x is first transformed by the matrix W, and then W is transformed into x and compare the embedding of each expert in E to calculate the similarity. This comparison is done by W x The similarity is normalized by multiplying the norm of ε and E to ensure that the similarity is insensitive to scale changes.

[0040] The similarity scores are processed by the Softmax function and converted into a probability distribution. This means that the gating function assigns a probability to each expert to indicate its relevance to the input feature x. Finally, TOP k The operation selects the k experts with the highest probability and reweights only these experts, resetting the weights of the unselected experts to zero. This step reduces computational complexity by focusing on a small number of experts and sparsely sparsely modeling, while enhancing the model's expressiveness in handling diverse tasks and modalities.

[0041] In summary, f MoE (x ij ) is a weighted sum of the outputs from the first k experts, with the weights determined by a gating function G that dynamically selects the most relevant experts. This creates a sparser feature space in the backbone model. By focusing on local patterns, the model can independently learn multimodal and local target patterns, effectively addressing the challenge of a crowded feature space and improving the model's expressiveness. In the actual implementation, to fully utilize the pre-trained backbone network weights, the weights of the newly added experts are initialized by copying the weights of the corresponding pre-trained 1×1 convolutional layer. This ensures that all experts are evenly selected when downstream model fine-tuning begins. Figure 4The routing visualization of the grid-level sparse mixture of experts network is given for image inputs of different modalities.

[0042] In one embodiment, calculating the loss of each task in the object detection process includes:

[0043] The training loss of the i-th iteration of task head t is expressed as The loss of each task maintains an exponential moving average as a smoothed historical statistic, expressed as Right now Here, α represents the smoothing factor, which is between 0 and 1.

[0044] In one embodiment, the dynamic adjustment module dynamically adjusts the learning rate of each task head based on the task loss as a target, including:

[0045] Calculate the inverse of the convergence rate of each task head according to the loss of the task, and use the inverse of the convergence rate to calculate the reweighting factor of the learning rate of the task head;

[0046] Calculate the consistency score of the task loss and smoothed historical statistics, and design a strategy for dynamically reweighting the shared weight backbone network based on the consistency score;

[0047] The learning rate of each task head is dynamically adjusted according to the reweighting factor and the strategy of dynamically reweighting the shared weight backbone network.

[0048] In a specific embodiment, the image encoding after the grid-level sparse hybrid expert network image encoder is sent to the detection task head of the network for result prediction. In multimodal, multi-dataset and multi-task object detection tasks, a major challenge is the different learning difficulties between modalities and tasks. This change may lead to unsynchronized optimization rates and inconsistent optimization directions, resulting in objective conflicts between different loss functions. To address this issue, a new dynamic learning rate adjustment method is provided to manage the different learning difficulties between tasks and modalities. The dynamic learning rate adjustment method uses the loss of each task head as an indicator to determine the current convergence rate of each task and the overall optimization direction of the network, and adjusts the learning rate accordingly. Specifically, one strategy is used for the learning rate of each task head (non-shared network) to balance the relative convergence rate of each task, and the other strategy is used for the backbone (network with shared weights) to ensure consistency in the optimization direction.

[0049] This dynamic learning rate adjustment method proposed in this application balances the learning complexity across different tasks and modalities, balancing relative convergence speeds and ensuring consistency in optimization direction. Unlike traditional techniques that primarily modify loss weights or gradients, which often lack precise control over specific network submodules or suffer from inefficiencies, this application provides fine-grained control while maintaining optimization efficiency.

[0050] In one embodiment, calculating the inverse of the convergence rate of each task head according to the loss of the task includes:

[0051] For the learning rate adjustment of the head, the inverse of the convergence rate of each task head is calculated according to the loss of the task:

[0052]

[0053] in, represents the training loss of the i-th iteration of task t, represents the smoothed historical statistics of the i-th iteration of task t.

[0054] In one embodiment, the Softmax of temperature θ is then used to reweight the learning rate of the corresponding network task head, aiming to balance the convergence speed of each task. The reweighting factor of the learning rate of the task head is calculated using the inverse of the convergence rate, including:

[0055] The reweighting factor of the learning rate of the task head is calculated using the inverse of the convergence rate:

[0056]

[0057] in, The inverse of the convergence rate of the i-th iteration of task t, where T is the total number of tasks and θ is the temperature.

[0058] In a specific embodiment, a relatively large value This indicates that task t converges faster, resulting in a smaller Therefore the reweighting factor Lower to prevent too rapid convergence. Smaller values ​​of result in larger This strategy ensures that the convergence rate of each task remains balanced throughout the training process.

[0059] In a specific embodiment, for the learning rate adjustment of the backbone, the reweighting is based on the historical consistency of each loss. In order to measure the consistency of training convergence, a consistency score C based on his_L and cur_L is defined. Specifically, his_L and cur_L are first converted to probability distributions using the function P, which uses a simple Softmax function:

[0060] P(L)=Softmax(L)

[0061] Next, the consistency scores of the task losses and smoothed historical statistics are calculated to evaluate whether the current losses of each task remain stable and consistent with their historical values.

[0062] In one embodiment, calculating the consistency score of the task loss and the smoothed historical statistics includes:

[0063] Compute the consistency score of the task loss and smoothed historical statistics as

[0064]

[0065] Where cur_L represents the training loss, his_L represents the smoothed historical statistics, P(L) = Softmax(L), T is the total number of tasks, represents the training loss of the i-th iteration of task t, represents the smoothed historical statistics of the i-th iteration of task t.

[0066] Larger values ​​of c indicate that the relative values ​​of the current iteration's losses are similar to their historical values, suggesting that the current batch of examples has stabilized the network updates. In this case, the learning rate should be increased to enable the network to converge faster. Conversely, lower values ​​of c indicate instability, suggesting that the current example makes some tasks more difficult to learn than the previous average, while other tasks are easier to learn. If the network updates shared weights too aggressively in this situation, the network will be optimized towards the more difficult tasks in the current iteration, potentially harming easier tasks. Therefore, the network should be updated cautiously with a lower learning rate.

[0067] To balance this, the following strategy is proposed to dynamically reweight the shared weight backbone network.

[0068] In one embodiment, a strategy for dynamically re-weighting a shared weight backbone network based on a consistency score is designed, including:

[0069] Design a strategy to dynamically reweight the shared weight backbone network based on the consistency score:

[0070]

[0071] Among them, the scalar factor 2 ensures that the reweighted values ​​after the sigmoid function are in the range (0, 2), b is a hyperparameter, which is interpreted as the threshold for reweighting, that is, when c is b, the reweighting is 1, τ is the temperature parameter for value sensitivity adjustment, and C is the consistency score.

[0072] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0073] In one embodiment, Figure 5 As shown, a multi-modal remote sensing multi-task target detection device is provided, comprising: a detection model construction module 502, an image encoding module 504 and a model optimization and target detection module 506, wherein:

[0074] Constructing a detection model module 502 for acquiring a multimodal dataset; the multimodal dataset includes remote sensing images of multiple modalities; constructing a remote sensing multi-task target detection model; the remote sensing multi-task target detection model includes a parameter-sharing image encoder, a multi-task head network, and a dynamic adjustment module; the parameter-sharing image encoder includes a network layer, a convolutional layer, and an image encoder with a grid-level sparse mixture of experts network;

[0075] An image encoding module 504 is configured to encode the input parameters of remote sensing images of multiple modalities using a shared image encoder to obtain output features;

[0076] The model optimization and target detection module 506 is used to input the output features into the multi-task head network for target detection, calculate the loss of each task in the target detection process, and dynamically adjust the learning rate of each task head based on the loss of the task in the dynamic adjustment module to optimize the convergence speed and direction consistency of the remote sensing multi-task target detection model, and use the optimized remote sensing multi-task target detection model to realize multi-task target detection.

[0077] Regarding the specific definition of the multi-modal remote sensing multi-task target detection device, please refer to the definition of the multi-modal remote sensing multi-task target detection method above, which will not be repeated here. The various modules in the above-mentioned multi-modal remote sensing multi-task target detection device can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0078] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and such modifications and improvements are intended to fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A multi-task target detection method for remote sensing based on multimodality, characterized in that: The method comprises: Acquire a multimodal dataset; the multimodal dataset includes remote sensing images of multiple modalities; construct a remote sensing multi-task target detection model; the remote sensing multi-task target detection model includes a parameter-sharing image encoder, a multi-task head network, and a dynamic adjustment module; the parameter-sharing image encoder includes a network layer, a convolutional layer, and an image encoder with a grid-level sparse hybrid expert network; Inputting the remote sensing images of the multiple modalities into the parameter shared image encoder for image encoding to obtain output features; Inputting the output features into the multi-task head network for target detection, calculating the loss of each task in the target detection process, dynamically adjusting the learning rate of each task head based on the loss of the task in the dynamic adjustment module to optimize the convergence speed and direction consistency of the remote sensing multi-task target detection model, and implementing multi-task target detection using the optimized remote sensing multi-task target detection model; The dynamic adjustment module dynamically adjusts the learning rate of each task head based on the loss of the task, including: Calculating the inverse of the convergence rate of each task head according to the loss of the task, and calculating the reweighting factor of the learning rate of the task head using the inverse of the convergence rate; Calculating a consistency score between the loss and smoothed historical statistics of the task, and designing a strategy for dynamically reweighting the shared weight backbone network based on the consistency score; The learning rate of each task head is dynamically adjusted according to the reweighting factor and the strategy of the dynamic reweighting shared weight backbone network.

2. The method according to claim 1, characterized in that Inputting the remote sensing images of the multiple modalities into the parameter shared image encoder for image encoding to obtain output features, including: Encoding the remote sensing image according to the network layer and the convolutional layer in the parameter sharing image encoder to obtain deep image features; The deep image features are input into an image encoder with a grid-level sparse mixture of experts network to obtain output features.

3. The method according to claim 2, characterized in that The deep image features are input into an image encoder with a grid-level sparse mixture of experts network to obtain output features including: The deep image features are input into the image encoder with a grid-level sparse hybrid expert network, and the output features are in: N is the total number of experts, G is the gating function, represents the nth 1×1 convolution expert, each expert has an embedding representation in the matrix E, x ij Represents the local spatial input feature of the i-th row and j-th column in the depth image feature, TOP k Indicates the selection of k experts with the highest probability.

4. The method according to claim 1, wherein Calculate the loss of each task in the object detection process, including: The training loss of the i-th iteration of task head t is expressed as The loss of each task maintains an exponential moving average as a smoothed historical statistic, expressed as Right now Here, α represents the smoothing factor, which is between 0 and 1.

5. The method according to claim 1, wherein The inverse of the convergence rate of each task head is calculated based on the loss of the task, including: The inverse of the convergence rate of each task head is calculated based on the loss of the task in, represents the training loss of the i-th iteration of task t, represents the smoothed historical statistics of the i-th iteration of task t.

6. The method according to claim 1, wherein The reweighting factor of the learning rate of the task head is calculated using the inverse of the convergence rate, including: The reweighting factor of the learning rate of the task head is calculated using the inverse of the convergence rate: in, The inverse of the convergence rate of the i-th iteration of task t, where T is the total number of tasks and θ is the temperature.

7. The method according to claim 1, characterized in that Compute the consistency score of the loss and smoothed historical statistics for the task, including: Compute the consistency score of the loss and smoothed historical statistics for the task as Where cur_L represents the training loss, his_L represents the smoothed historical statistics, P(L) = Softmax(L), T is the total number of tasks, represents the training loss of the i-th iteration of task t, represents the smoothed historical statistics of the i-th iteration of task t.

8. The method according to claim 1, characterized in that Designing a strategy for dynamically re-weighting the shared weight backbone network based on the consistency score, including: Design a strategy to dynamically re-weight the shared weight backbone network based on the consistency score: Among them, the scalar factor 2 ensures that the reweighted values ​​after the sigmoid function are in the range (0, 2), b is a hyperparameter, which is interpreted as the threshold for reweighting, that is, when c is b, the reweighting is 1, τ is the temperature parameter for value sensitivity adjustment, and C is the consistency score.

9. A multi-modal remote sensing multi-task target detection device, characterized in that: The device comprises: Constructing a detection model module for acquiring a multimodal dataset; the multimodal dataset includes remote sensing images of multiple modalities; constructing a remote sensing multi-task target detection model; the remote sensing multi-task target detection model includes a parameter-sharing image encoder, a multi-task head network, and a dynamic adjustment module; the parameter-sharing image encoder includes a network layer, a convolutional layer, and an image encoder with a grid-level sparse hybrid expert network; An image encoding module, configured to input the remote sensing images of the multiple modalities into the parameter shared image encoder for image encoding to obtain output features; The model optimization and target detection module is used to input the output features into the multi-task head network for target detection, calculate the loss of each task in the target detection process, dynamically adjust the learning rate of each task head with the loss of the task as the target in the dynamic adjustment module to optimize the convergence speed and direction consistency of the remote sensing multi-task target detection model, and implement multi-task target detection using the optimized remote sensing multi-task target detection model; the dynamic adjustment module dynamically adjusts the learning rate of each task head with the loss of the task as the target, including: Calculating the inverse of the convergence rate of each task head according to the loss of the task, and calculating the reweighting factor of the learning rate of the task head using the inverse of the convergence rate; Calculating a consistency score between the loss and smoothed historical statistics of the task, and designing a strategy for dynamically reweighting the shared weight backbone network based on the consistency score; The learning rate of each task head is dynamically adjusted according to the reweighting factor and the strategy of the dynamic reweighting shared weight backbone network.

Images