Multi-mode-based remote sensing multi-task target detection method and device

Abstract

The invention relates to a multi-mode-based remote sensing multi-task target detection method and device. The method comprises the following steps: constructing a remote sensing multi-task target detection model; the remote sensing multi-task target detection model comprises a parameter sharing image encoder, a multi-task head network and a dynamic adjustment module; the parameter sharing image encoder comprises a network layer, a convolutional layer and an image encoder with a grid-level sparse hybrid expert network; inputting the multi-modal remote sensing images into a parameter sharing image encoder for image encoding to obtain output features; the output features are input into a multi-task head network for target detection, the loss of each task in the target detection process is calculated, and the learning rate of each task head is dynamically adjusted in a dynamic adjustment module with the loss of the task as the target to achieve optimization of the convergence speed and direction consistency of a remote sensing multi-task target detection model. And realizing multi-task target detection by using the optimized remote sensing multi-task target detection model. By adopting the method, the multi-target detection efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-modal remote sensing multi-task target detection method and device. Background Art

[0002] Remote sensing target detection usually involves multiple sensors that employ different imaging mechanisms, resulting in diverse data modalities. Traditional detection models are usually developed for specific datasets related to a single modality and predefined format detection task. Such traditional approaches, such as Yimian Dai in “Asymmetric contextual modulation for infrared small target detection”, ignore the value of joint knowledge inherent in a unified remote sensing environment. In addition, aerial platforms such as drones and satellites are often equipped with multiple sensors, so it is crucial to process images from various modalities simultaneously. Multi-source target detection methods in “Multi-source remote sensing image fusion for ship target detection and recognition.” by Jinming Liu and “Hgr correlation pooling fusion framework for recognition and classification in multimodal remote sensing data” by Hongkang Zhang mainly rely on scarce, impractical, and inflexible spatially aligned paired images and spatial alignment algorithms. These methods are also limited to performing detection tasks in a single format. Therefore, there is an urgent need to develop a unified model that does not rely on spatially aligned image pairs and can handle multiple modalities and perform multiple format detection tasks (hereafter referred to as “multi-task”), but this area has not been fully explored. To fill this research gap, we propose a new task, called Multimodal Dataset and Multi-Task Object Detection, which aims to detect objects in any given image, regardless of its modality and whether its predefined detection task is horizontal or oriented boxes.

[0003] This task is closely related to two key research areas: multi-dataset object detection and multi-task learning. However, its task has unique challenges. In traditional multi-dataset object detection, such as Georgios Kapidis's "Multi-dataset, multitask learning of egocentric vision tasks" and Ren'e Ranftl's "Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer", although images may have different properties (such as natural images and paintings), they usually share similar underlying concepts (optical concepts). Simple joint training methods are often effective, and the performance of a single model trained on the combined dataset is usually better than that of a model trained on a single dataset. Xudong Wang in "Toward universal object detection by domain attention" adopts specialized attention mechanism layers that serve as domain-specific attention mechanisms for each dataset. Hang Xu's "Universal-rcnn: Universal object detector via transferable graph r-cnn" introduces a partitioned detector trained across multiple datasets, integrating features through a graph-based cross-dataset attention module. Xingyi Zhou’s “Simple multi-dataset detection” advances this concept by proposing a unified label space and emphasizing the importance of batch sampling strategies.

[0004] In contrast, multimodal datasets in remote sensing (e.g., visible light, synthetic aperture radar, infrared imagery) exhibit fundamentally different notions of modality. While these modalities may share certain common knowledge, significant differences in data representation lead to a large modality gap, which complicates the integration of cross-modal information. In addition, remote sensing datasets often contain multiple annotation types, such as horizontal boxes and oriented boxes, further increasing the complexity of model learning. These challenges may hinder the learning and optimization of traditional models in the following aspects: 1) Representation constraints: A dense model that shares the same parameters across multiple tasks and modalities may be limited in representation capabilities, as a single set of parameters may have difficulty effectively fitting the diverse distributions inherent in each dataset. 2) Optimization inconsistency: Differences in learning difficulty between different modalities and tasks may lead to asynchronous optimization rates or optimization directions of various components of the model. This inconsistency may lead to conflicting optimization results, affecting the model's ability to achieve different loss targets. Summary of the invention

[0005] Based on this, it is necessary to provide a multi-modal remote sensing multi-task target detection method and device that can improve the efficiency of multi-target detection in order to address the above technical problems.

[0006] A multi-task target detection method for remote sensing based on multi-modality, the method comprising:

[0007] Acquire a multimodal dataset; the multimodal dataset includes remote sensing images of multiple modalities; construct a remote sensing multi-task target detection model; the remote sensing multi-task target detection model includes a parameter-sharing image encoder, a multi-task head network, and a dynamic adjustment module; the parameter-sharing image encoder includes a network layer, a convolutional layer, and an image encoder with a grid-level sparse hybrid expert network;

[0008] The remote sensing image input parameters of multiple modalities are shared by the image encoder to perform image encoding and obtain output features;

[0009] The output features are input into the multi-task head network for target detection. The loss of each task in the target detection process is calculated. In the dynamic adjustment module, the learning rate of each task head is dynamically adjusted with the loss of the task as the target to optimize the convergence speed and direction consistency of the remote sensing multi-task target detection model. The optimized remote sensing multi-task target detection model is used to realize multi-task target detection.

[0010] A multi-modal remote sensing multi-task target detection device, the device comprising:

[0011] Construct a detection model module for acquiring a multimodal dataset; the multimodal dataset includes remote sensing images of multiple modalities; construct a remote sensing multi-task target detection model; the remote sensing multi-task target detection model includes a parameter-sharing image encoder, a multi-task head network, and a dynamic adjustment module; the parameter-sharing image encoder includes a network layer, a convolutional layer, and an image encoder with a grid-level sparse hybrid expert network;

[0012] An image encoding module is used to encode the input parameters of remote sensing images of multiple modalities using a shared image encoder to obtain output features;

[0013] The model optimization and target detection module is used to input the output features into the multi-task head network for target detection, calculate the loss of each task in the target detection process, and dynamically adjust the learning rate of each task head based on the loss of the task in the dynamic adjustment module to optimize the convergence speed and direction consistency of the remote sensing multi-task target detection model. The optimized remote sensing multi-task target detection model is used to realize multi-task target detection.

[0014] The above-mentioned multi-task target detection method and device based on multi-modality in remote sensing is constructed by constructing a unified architecture including a parameter-sharing image encoder, a multi-task head network and a dynamic adjustment module. The grid-level sparse hybrid expert network in the parameter-sharing image encoder calls different expert networks based on the image region features, accurately extracts the image features of each modality, crosses the modality difference gap, solves the problem of large differences in multi-modal data representation and obvious modality gap, and reduces the computational burden. The multi-task head network processes tasks of different annotation types separately, and combines the dynamic adjustment module to dynamically adjust the learning rate according to the task loss, optimizes the convergence speed and direction consistency between different tasks, and overcomes the optimization inconsistency problem of traditional models in multi-task learning, so that the model can accurately learn the detection methods of different annotated targets and cope with the complexity brought by multiple annotation types of remote sensing data. A single model is achieved for efficient processing of multi-modality and multi-task. On the one hand, it has strong representation capabilities and efficient computation, overcoming the difficulties and resource waste faced by traditional methods in integrating multimodal data. On the other hand, by optimizing the task convergence characteristics and ensuring stable optimization of each task, it solves the dilemma that traditional models cannot take into account both multimodality and multitask, reduces the complexity of model deployment, improves practicality and versatility, and meets the needs of multi-task target detection in multimodal remote sensing images in practical applications, filling the research gap in related fields and providing innovative solutions for the development of remote sensing target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of a flow chart of a multi-task target detection method for remote sensing based on multi-modality in an embodiment;

[0016] Figure 2 A schematic diagram of a remote sensing multi-task target detection model in an embodiment;

[0017] Figure 3 A schematic diagram showing a comparison between a grid-level sparse hybrid expert network and an image-level sparse hybrid expert network in one embodiment;

[0018] Figure 4 A schematic diagram of routing visualization of a grid-level sparse hybrid expert network for image inputs of different modalities in another embodiment;

[0019] Figure 5 The present invention is a structural block diagram of a multi-modal remote sensing multi-task target detection device in one embodiment. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0021] In one embodiment, Figure 1 As shown, a multi-modal remote sensing multi-task target detection method is provided, comprising the following steps:

[0022] Step 102, obtaining a multimodal dataset; the multimodal dataset includes remote sensing images of multiple modalities; constructing a remote sensing multi-task target detection model; the remote sensing multi-task target detection model includes a parameter shared image encoder, a multi-task head network and a dynamic adjustment module; the parameter shared image encoder includes a network layer, a convolutional layer and an image encoder with a grid-level sparse hybrid expert network.

[0023] Aiming at the significant differences in the essentially different mode concepts and data representations between remote sensing multimodal datasets (such as visible light, synthetic aperture radar, infrared images), this application constructs Figure 2 The remote sensing multi-task target detection model shown in Figure 1 plays a key role in the grid-level sparse hybrid expert network in the parameter-sharing image encoder. Through the dynamic routing mechanism, only a small number of experts related to the input features are activated to achieve efficient extraction of cross-modal shared knowledge and modality-specific features. The image is divided into grid regions, and different expert networks are called to process different regions according to their features. The unique features of each modality image in different regions can be effectively captured, thus overcoming the feature extraction barriers caused by modality differences. For example, texture features in synthetic aperture radar images and color and shape features in visible light images can be accurately extracted through appropriate expert networks, so that the model can obtain more representative and discriminative features when facing different modal data, providing strong support for subsequent target detection. The grid-level sparse hybrid expert network improves the representation ability of the model by extracting multimodal image features in a targeted manner, and can better deal with the feature diversity and complexity problems caused by modality differences. At the same time, its sparse calling of expert networks avoids processing all features to the same extent, reduces the computational burden, and enables the model to run efficiently when processing complex multimodal data. It overcomes the difficulties in information integration and waste of computing resources caused by modal differences in traditional methods when facing multimodal data.

[0024] Step 104 , the remote sensing image input parameters of multiple modalities are shared by an image encoder for image encoding to obtain output features.

[0025] In step 106, the output features are input into the multi-task head network for target detection, the loss of each task in the target detection process is calculated, and the learning rate of each task head is dynamically adjusted in the dynamic adjustment module with the loss of the task as the target to optimize the convergence speed and direction consistency of the remote sensing multi-task target detection model, and the multi-task target detection is realized using the optimized remote sensing multi-task target detection model.

[0026] Considering that remote sensing datasets usually contain multiple annotation types (such as horizontal boxes and oriented boxes) and the differences in learning difficulty between different tasks, the multi-task head network processes different detection tasks (horizontal box detection, oriented box detection, etc.) separately, calculates the loss of each task during the target detection process, and passes these loss values ​​to the dynamic adjustment module. The dynamic adjustment module dynamically adjusts the learning rate of each task head based on the loss of the task, ensuring that each task head can adaptively optimize learning according to the characteristics and difficulty of its own task. For example, when the learning difficulty of the target task of detecting horizontal box annotations is high, the dynamic adjustment module will appropriately increase the learning rate of the task head to speed up its convergence speed so that it can keep up with the learning progress of other tasks, avoid conflicting optimization results caused by inconsistent learning progress between tasks, and ensure that the model can accurately learn the characteristics and detection methods of targets with different annotation types. The dynamic adjustment module dynamically adjusts the learning rate of the multi-task head network, which significantly optimizes the convergence speed and direction consistency between tasks. By dynamically adjusting the learning rate according to the task loss, each task head can be optimized at a more appropriate speed and direction, avoiding the optimization inconsistency problem of traditional models in multi-task learning due to differences in learning difficulty of different tasks, ensuring that the model can simultaneously and efficiently learn detection tasks of targets of different labeled types, and improving the learning and optimization effects of the model on complex remote sensing datasets.

[0027] In addition, the unified model designed in this application changes the limitations of traditional methods in developing models for single modality and predefined format detection tasks. A single model can simultaneously process multi-modal image inputs with high efficiency and performance, and support multi-task detection at the same time. This not only improves the versatility and practicality of the model, but also reduces the complexity of model deployment. Compared with the traditional method of building and deploying multiple models for different modalities and tasks, only one unified model needs to be deployed to meet the needs of multiple remote sensing image modalities and multiple target detection tasks, reducing application costs and system complexity, and effectively solving the problem that traditional models cannot handle multi-modality and multi-task at the same time.

[0028] In the above-mentioned multi-task target detection method based on multimodality, the above-mentioned multi-task target detection method and device based on multimodality, this application constructs a unified architecture including a parameter-sharing image encoder, a multi-task head network and a dynamic adjustment module. The grid-level sparse hybrid expert network in the parameter-sharing image encoder calls different expert networks according to the image region features, accurately extracts the image features of each modality, crosses the modality difference gap, solves the problem of large differences in multimodal data representation and obvious modality gap, and reduces the computational burden. The multi-task head network processes tasks of different annotation types separately, and dynamically adjusts the learning rate according to the task loss in combination with the dynamic adjustment module, optimizes the convergence speed and direction consistency between different tasks, and overcomes the optimization inconsistency problem of traditional models in multi-task learning, so that the model can accurately learn the detection method of different annotated targets and cope with the complexity brought by multiple annotation types of remote sensing data. A single model is realized for efficient processing of multimodality and multi-task. On the one hand, it has strong representation capabilities and efficient computation, overcoming the difficulties and resource waste faced by traditional methods in integrating multimodal data. On the other hand, by optimizing the task convergence characteristics and ensuring stable optimization of each task, it solves the dilemma that traditional models cannot take into account both multimodality and multitask, reduces the complexity of model deployment, improves practicality and versatility, and meets the needs of multi-task target detection in multimodal remote sensing images in practical applications, filling the research gap in related fields and providing innovative solutions for the development of remote sensing target detection.

[0029] In one embodiment, remote sensing image input parameters of multiple modalities are shared by an image encoder for image encoding to obtain output features, including:

[0030] Encode the remote sensing image according to the network layer and the convolution layer in the parameter sharing image encoder to obtain the deep image features;

[0031] The deep image features are fed into an image encoder with a grid-level sparse mixture of experts network to obtain output features.

[0032] In a specific embodiment, for input features in the deep layer of the network, the grid-level features are dynamically routed to the expert network module dedicated to its features. Among them, the features passed through the deep layer of the neural network are weighted by the results output by multiple expert network modules. The grid-level routing network determines the most relevant experts for each local feature dynamically. The grid-level sparse hybrid expert network creates a sparser feature space in the image encoding module. By focusing on local patterns, the model can independently learn multiple modalities and local target patterns. The design effectively solves the challenges of crowded feature space and enhances the expressiveness of the model. Figure 3 Schematic diagram of the comparison between a grid-level sparse hybrid expert network and an image-level sparse hybrid expert network in one embodiment.

[0033] In one embodiment, the deep image features are input into an image encoder with a grid-level sparse hybrid expert network to obtain output features, including:

[0034] The deep image features are input into the image encoder with a grid-level sparse mixture of experts network, and the output features are

[0035]

[0036] in:

[0037]

[0038] N is the total number of experts, G is the gating function, represents the nth 1×1 convolutional expert, each of which has an embedding representation in the matrix E, x ij Represents the local spatial input feature of the i-th row and j-th column in the deep image feature, TOP k Indicates the k experts with the highest probability of selection.

[0039] In a specific embodiment, the input feature x is first transformed by the matrix W, and then W x and compare them with the embedding of each expert in E to calculate the similarity. This comparison is done through W x The similarity is normalized by multiplying it by the norm of E to ensure that it is insensitive to scale changes.

[0040] The similarity scores are processed by the Softmax function and converted into a probability distribution. This means that the gating function assigns a probability to each expert to indicate its relevance to the input feature x. Finally, TOP k The operation selects the k experts with the highest probability, and only re-weights these experts, and the weights of the unselected experts are reset to zero. This step reduces the computational complexity by focusing on a small number of experts and sparsely calculating the model, while enhancing the model's expressiveness in handling diverse tasks and modalities.

[0041] In summary, f MoE (x ij ) is the weighted sum of the outputs from the first k experts, and the weights are determined by the gating function G that dynamically selects the most relevant experts. It creates a sparser feature space in the backbone model. By focusing on local patterns, the model can independently learn multimodal and local target patterns, thereby effectively addressing the challenges of crowded feature space and improving the expressiveness of the model. In the actual implementation, in order to make full use of the pre-trained backbone network weights, the weights of the newly added experts are initialized by copying the corresponding pre-trained 1×1 convolutional layer weights, which ensures that all experts are evenly selected when the downstream model fine-tuning begins. Figure 4The routing visualization of the grid-level sparse hybrid expert network is given for image inputs of different modalities.

[0042] In one embodiment, calculating the loss of each task in the object detection process includes:

[0043] The training loss of the i-th iteration of task head t is expressed as The loss of each task maintains an exponential moving average as a smoothed historical statistic, expressed as Right now Here, α represents the smoothing factor, which is between 0 and 1.

[0044] In one embodiment, the dynamic adjustment module dynamically adjusts the learning rate of each task head based on the loss of the task, including:

[0045] The inverse of the convergence rate of each task head is calculated according to the loss of the task, and the reweighting factor of the learning rate of the task head is calculated using the inverse of the convergence rate;

[0046] Calculate the consistency score of the task loss and smoothed historical statistics, and design a strategy for dynamically re-weighting the shared weight backbone network based on the consistency score;

[0047] The learning rate of each task head is dynamically adjusted according to the reweighting factor and the strategy of dynamically reweighting the shared weight backbone network.

[0048] In a specific embodiment, the image encoding after the grid-level sparse hybrid expert network image encoder is sent to the detection task head of the network for result prediction. In multimodal, multi-dataset, and multi-task object detection tasks, a major challenge is the different learning difficulties between modalities and tasks. This change may lead to unsynchronized optimization rates and inconsistent optimization directions, resulting in objective conflicts between different loss functions. To address this issue, a new dynamic learning rate adjustment method is provided to manage the different learning difficulties between tasks and modalities. The dynamic learning rate adjustment method uses the loss of each task head as an indicator to determine the current convergence rate of each task and the overall optimization direction of the network, and adjusts the learning rate accordingly. Specifically, one strategy is used for the learning rate of each task head (non-shared network) to balance the relative convergence rate of each task, and another strategy is used for the backbone (network with shared weights) to ensure consistency in the optimization direction.

[0049] The dynamic learning rate adjustment method proposed in this application balances the learning complexity between different tasks and modalities, balances the relative convergence speed and ensures the consistency of the optimization direction. Unlike traditional techniques that mainly modify loss weights or gradients, which usually lack precise control over specific network submodules or have inefficiencies, this application provides fine-grained control while maintaining optimization efficiency.

[0050] In one embodiment, calculating the inverse of the convergence rate of each task head according to the loss of the task includes:

[0051] For the learning rate adjustment of the head, the inverse of the convergence rate of each task head is calculated according to the loss of the task:

[0052]

[0053] in, represents the training loss of the i-th iteration of task t, Represents the smoothed historical statistics of the i-th iteration of task t.

[0054] In one embodiment, the Softmax of temperature θ is then used to reweight the learning rate of the corresponding network task head, aiming to balance the convergence speed of each task. The reweighting factor of the learning rate of the task head is calculated using the inverse of the convergence rate, including:

[0055] The reweighting factor of the learning rate of the task head is calculated using the inverse of the convergence rate:

[0056]

[0057] in, The inverse of the convergence rate of the i-th iteration of task t, where T is the total number of tasks and θ is the temperature.

[0058] In a specific embodiment, a relatively large value This indicates that task t converges faster, resulting in a smaller So the reweighting factor lower to prevent too rapid convergence. Smaller values ​​of result in larger This strategy ensures that the convergence rate of each task remains balanced throughout the training process.

[0059] In a specific embodiment, for the learning rate adjustment of the backbone, the reweighting is based on the historical consistency of each loss. In order to measure the consistency of training convergence, a consistency score C based on his_L and cur_L is defined. Specifically, his_L and cur_L are first converted to probability distributions using the function P, which uses a simple Softmax function:

[0060] P(L)=Softmax(L)

[0061] Next, the consistency scores of the task's losses and smoothed historical statistics are calculated to evaluate whether the current losses of each task remain stable and consistent with their historical values.

[0062] In one embodiment, calculating the consistency score of the loss of the task and the smoothed historical statistics includes:

[0063] Compute the consistency score of the task loss and the smoothed historical statistics as

[0064]

[0065] Where cur_L represents the training loss, his_L represents the smoothed historical statistics, P(L) = Softmax(L), T is the total number of tasks, represents the training loss of the i-th iteration of task t, Represents the smoothed historical statistics of the i-th iteration of task t.

[0066] Larger c values ​​indicate that the relative values ​​of the current iteration losses are similar to their historical values, which indicates that the current batch of samples stabilizes the network updates. In this case, the learning rate needs to be increased to make the network converge faster. Conversely, lower c values ​​indicate instability, indicating that the current sample makes some tasks harder to learn than the previous average, while other tasks are easier to learn. If the network updates the shared weights too aggressively in this case, the network will be optimized in the direction of the harder tasks in the current iteration, which may hurt the easier tasks. Therefore, the network should be updated cautiously to reduce the learning rate.

[0067] To balance this, the following strategy of dynamically re-weighting the shared weight backbone network is proposed.

[0068] In one embodiment, a strategy for dynamically re-weighting a shared weight backbone network based on a consistency score is designed, including:

[0069] Design a strategy to dynamically re-weight the shared weight backbone network based on the consistency score:

[0070]

[0071] Among them, the scalar factor 2 ensures that the reweighted value after the sigmoid function is in the range (0,2), b is a hyperparameter, which is interpreted as the threshold of reweighting, that is, when c is b, the reweighting is 1, τ is the temperature parameter for value sensitivity adjustment, and C is the consistency score.

[0072] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0073] In one embodiment, Figure 5 As shown, a multi-modal remote sensing multi-task target detection device is provided, comprising: a detection model building module 502, an image encoding module 504 and a model optimization and target detection module 506, wherein:

[0074] Constructing a detection model module 502, for acquiring a multimodal data set; the multimodal data set includes remote sensing images of multiple modalities; constructing a remote sensing multi-task target detection model; the remote sensing multi-task target detection model includes a parameter-sharing image encoder, a multi-task head network, and a dynamic adjustment module; the parameter-sharing image encoder includes a network layer, a convolutional layer, and an image encoder with a grid-level sparse hybrid expert network;

[0075] An image encoding module 504 is used to encode the remote sensing image input parameters of multiple modalities using a shared image encoder to obtain output features;

[0076] The model optimization and target detection module 506 is used to input the output features into the multi-task head network for target detection, calculate the loss of each task in the target detection process, and dynamically adjust the learning rate of each task head in the dynamic adjustment module with the loss of the task as the target to optimize the convergence speed and direction consistency of the remote sensing multi-task target detection model, and use the optimized remote sensing multi-task target detection model to realize multi-task target detection.

[0077] For the specific definition of the multi-modal remote sensing multi-task target detection device, please refer to the definition of the multi-modal remote sensing multi-task target detection method above, which will not be repeated here. Each module in the above-mentioned multi-modal remote sensing multi-task target detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0078] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-task target detection method for remote sensing based on multi-modality, characterized in that: The method comprises: Acquire a multimodal dataset; the multimodal dataset includes remote sensing images of multiple modalities; construct a remote sensing multi-task target detection model; the remote sensing multi-task target detection model includes a parameter-sharing image encoder, a multi-task head network and a dynamic adjustment module; the parameter-sharing image encoder includes a network layer, a convolutional layer and an image encoder with a grid-level sparse hybrid expert network; Inputting the remote sensing images of the multiple modalities into the parameter sharing image encoder for image encoding to obtain output features; The output features are input into the multi-task head network for target detection, the loss of each task in the target detection process is calculated, and the learning rate of each task head is dynamically adjusted in the dynamic adjustment module with the loss of the task as the target to optimize the convergence speed and direction consistency of the remote sensing multi-task target detection model, and multi-task target detection is realized using the optimized remote sensing multi-task target detection model.

2. The method according to claim 1, characterized in that Inputting the remote sensing images of the multiple modalities into the parameter sharing image encoder for image encoding to obtain output features, including: Encoding the remote sensing image according to the network layer and the convolution layer in the parameter sharing image encoder to obtain deep image features; The deep image features are input into an image encoder with a grid-level sparse hybrid expert network to obtain output features.

3. The method according to claim 1, characterized in that The deep image features are input into an image encoder with a grid-level sparse hybrid expert network to obtain output features, including: The deep image features are input into an image encoder with a grid-level sparse hybrid expert network, and the output features are in: N is the total number of experts, G is the gating function, represents the nth 1×1 convolutional expert, each of which has an embedding representation in the matrix E, x ij Represents the local spatial input feature of the i-th row and j-th column in the deep image feature, TOP k Indicates the k experts with the highest probability of selection.

4. The method according to claim 1, characterized in that: Calculate the loss of each task in the target detection process, including: The training loss of the i-th iteration of task head t is expressed as The loss of each task maintains an exponential moving average as a smoothed historical statistic, expressed as Right now Here, α represents the smoothing factor, which is between 0 and 1.

5. The method according to claim 1, characterized in that The dynamic adjustment module dynamically adjusts the learning rate of each task head based on the loss of the task, including: Calculating the inverse of the convergence rate of each task head according to the loss of the task, and calculating the reweighting factor of the learning rate of the task head using the inverse of the convergence rate; Calculating a consistency score between the loss and the smoothed historical statistics of the task, and designing a strategy for dynamically re-weighting the shared weight backbone network based on the consistency score; The learning rate of each task head is dynamically adjusted according to the reweighting factor and the strategy of dynamically reweighting the shared weight backbone network.

6. The method according to claim 5, characterized in that The inverse of the convergence rate of each task head is calculated based on the loss of the task, including: The inverse of the convergence rate of each task head is calculated based on the loss of the task: in, represents the training loss of the i-th iteration of task t, Represents the smoothed historical statistics of the i-th iteration of task t.

7. The method according to claim 5, characterized in that The re-weighting factor of the learning rate of the task head is calculated using the inverse of the convergence rate, including: The reweighting factor of the learning rate of the task head is calculated using the inverse of the convergence rate: in, The inverse of the convergence rate of the i-th iteration of task t, where T is the total number of tasks and θ is the temperature.

8. The method according to claim 5, characterized in that Compute the consistency score of the loss and smoothed historical statistics for the task, including: Compute the consistency score of the loss and smoothed historical statistics for the task as Where cur_L represents the training loss, his_L represents the smoothed historical statistics, P(L) = Softmax(L), T is the total number of tasks, represents the training loss of the i-th iteration of task t, Represents the smoothed historical statistics of the i-th iteration of task t.

9. The method according to claim 5, characterized in that Designing a strategy for dynamically re-weighting the shared weight backbone network based on the consistency score, including: Design a strategy to dynamically re-weight the shared weight backbone network based on the consistency score: Among them, the scalar factor 2 ensures that the reweighted value after the sigmoid function is in the range (0,2), b is a hyperparameter, which is interpreted as the threshold of reweighting, that is, when c is b, the reweighting is 1, τ is the temperature parameter for value sensitivity adjustment, and C is the consistency score.

10. A multi-task target detection device based on multi-modality remote sensing, characterized in that: The device comprises: Constructing a detection model module for acquiring a multimodal data set; the multimodal data set includes remote sensing images of multiple modalities; constructing a remote sensing multi-task target detection model; the remote sensing multi-task target detection model includes a parameter-sharing image encoder, a multi-task head network, and a dynamic adjustment module; the parameter-sharing image encoder includes a network layer, a convolutional layer, and an image encoder with a grid-level sparse hybrid expert network; An image encoding module, used for inputting the remote sensing images of the multiple modalities into the parameter sharing image encoder for image encoding to obtain output features; The model optimization and target detection module is used to input the output features into the multi-task head network for target detection, calculate the loss of each task in the target detection process, and dynamically adjust the learning rate of each task head with the loss of the task as the target in the dynamic adjustment module to optimize the convergence speed and direction consistency of the remote sensing multi-task target detection model, and use the optimized remote sensing multi-task target detection model to realize multi-task target detection.

Images