Remote sensing small target detection network based on visible light-infrared bimodal

Abstract

The invention relates to the technical field of remote sensing small target detection, and discloses a remote sensing small target detection network based on visible light-infrared bimodal, and provides a Multi-Order Gateted Aggregation Inverted Residual (MIR) module, the MIR ingeniously utilizes the design idea of manifolds interest, and the problem of information loss of a small-scale target is relieved to a great extent. Besides, on the basis of MIR, the advantage of multi-modal information complementation is utilized, a cross-stage multi-modal fusion network architecture is designed, and the detection precision is remarkably improved. Meanwhile, in order to effectively solve the problems of complexity and diversity of a remote sensing scene, the invention provides a GViL module (Vision LSTM Based on Graphic Path), and the GViL achieves a remarkable effect by utilizing the high efficiency of gradient path analysis and the modeling capability of a Vision LSTM and ViL on the context. According to the method, the performance of the model is verified on a multi-mode remote sensing data set VEDAI and Dronevehile, and an excellent result is obtained.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing small target detection, and in particular to a remote sensing small target detection network based on visible light-infrared dual mode. Background Art

[0002] With the rapid development of modern science and technology, remote sensing technology has become one of the important means of obtaining surface information. Remote sensing images have shown great potential in many fields, including intelligent transportation, land planning, environmental monitoring, and military target positioning and identification, because they can provide large-scale, high-resolution views of the surface. These images usually include satellite and aerial images, which can capture subtle changes in the surface and play an irreplaceable role in monitoring and analysis.

[0003] Remote sensing images provide a bird's-eye view of the Earth's surface and include satellite and aerial images. Figure 1 As shown in the figure, most objects in remote sensing images may be very small in scale and difficult to identify based on appearance alone. In addition, effective information is easily lost in the network feedforward process, resulting in inappropriate gradient updates, which affects the accuracy of target detection. On the contrary, to identify these objects, we need to rely on their environment, that is, their contextual information, because the surrounding environment can provide valuable clues about their shape, direction, and other features. At the same time, we also face problems such as large scale changes and complex backgrounds in multi-scale scenes.

[0004] Existing remote sensing target detection technology mainly faces the following challenges:

[0005] 1. Small-scale targets are prone to losing key information during network forward propagation, resulting in reduced detection accuracy;

[0006] 2. The background of remote sensing images is complex. For example, the scale, shape, posture and direction of remote sensing targets vary. Image features will change significantly under different lighting, weather and seasonal conditions. Moreover, remote sensing images may come from different imaging heights and viewing angles, which will affect the shape and appearance of the target.

[0007] 3. The extreme imbalance between the target and the background causes the model to focus on the background rather than the target;

[0008] 4. Some methods to reduce information loss, such as reversible architectures or deep supervision, may increase inference costs or error accumulation;

[0009] 5. Multi-scale feature fusion may bring redundant information and affect the performance of the model.

[0010] In order to solve the above problems, researchers have proposed a variety of methods, including developing new network backbones or attention modules, designing multi-scale feature fusion modules, and detail design (such as rotation boxes, loss function design, label assignment, etc.). Although these methods have improved the detection performance to a certain extent, they still have limitations and need further research and improvement. Summary of the invention

[0011] In order to solve the technical problem of information loss of small-scale targets in remote sensing scenes during forward propagation, the present invention provides a remote sensing small target detection network based on visible light-infrared dual modality. The present invention proposes a multi-order gated aggregation inverse residual module (Multi-order Gated Aggregation-Inverted Residual Module, MIR), which regards the output manifold of the entire module as "manifolds of interest", releases low-dimensional information in the form of decompression, and filters it with lightweight deep convolution, and the features are then projected back to low-dimensional representation through linear convolution. MIR includes two parts: Information Aggregation Block (IA Block) and Inverted Residual Block, wherein IA Block includes a gating branch and a local-global semantic information aggregation branch, wherein the gating branch is based on the SiLU mechanism and can adaptively extract the target context semantic information; the semantic information aggregation branch uses DepthwiseConv with different sizes of convolution kernels to extract low-order, medium-order, and high-order information respectively, which can effectively adapt to targets of different scales. In order to effectively enhance the representation ability of the model, the Inverted Residual Block first decouples the output manifold of the IA Block, then uses DepthwiseConv to integrate high-dimensional features, and finally projects back to low dimensions through linear convolution. In many scenarios, since multiple modalities can achieve information complementarity, they can effectively solve the problem of remote sensing target detection in scenes such as occlusion, shadow, and darkness. This paper combines the MIR module with the single-modal detector YOLOv8s to design a multi-stage feature fusion detection architecture Dual-Modality Framework.

[0012] In the present invention, a method for improving the global understanding ability of images is designed to effectively cope with the complexity and diversity of remote sensing scenes. Based on the gradient path design scheme, the present invention utilizes the advantages of long short-term memory of Vision-LSTM (ViL) to design a GViL module. Combined with the PAN-FPN structure, it can widely aggregate contextual information and enhance the understanding ability of the model. In addition, in order to solve the imbalance problem between the target and the background in remote sensing small target detection, the present invention uses Adaptive Threshold Focal Loss (ATFL) to replace the traditional BCE loss function. At the same time, the normalized Wasserstein distance and CIoU are integrated to construct a new IoU loss function.

[0013] To achieve the purpose of the present invention, the present invention is implemented through the following scheme.

[0014] The remote sensing small target detection network based on visible light-infrared dual mode provided by the present invention includes:

[0015] Multi-order Gated Aggregation-Inverted Residual Module (MIR), which is used to effectively aggregate semantic information and alleviate the information loss of small-scale targets in forward propagation;

[0016] The cross-stage multimodal fusion network architecture is used to integrate the information of visible light and infrared modalities, realize the information complementarity between visible light and infrared modalities, and improve the detection accuracy;

[0017] The gradient-based visual long short-term memory network module, or GViL module, is used to enhance the model’s global understanding and address the complexity and diversity of remote sensing scenes.

[0018] A new IoU loss function is constructed by combining the adaptive threshold focal loss (ATFL) with the normalized Wasserstein distance (NWD) CIoU to solve the problem of imbalance between target and background and the extreme sensitivity of bounding box regression to small remote sensing targets.

[0019] Preferably, the MIR module comprises:

[0020] Information Aggregation Block (IA Block), which includes a gating branch and a local-global semantic information aggregation branch. It mainly aggregates local perception and global context information through a gating mechanism.

[0021] The inverted residual block is used to integrate the effective information in the output manifold of the IA Block to obtain the “manifolds of interest”.

[0022] Preferably, the operation of integrating the valid information in the information aggregated by the IA Block to obtain the "manifolds of interest" is:

[0023] First, perform Expansion Convolution on the output manifold of IA Block, where the value of ExpansionRatio is 4. This operation uses Conv 1×1 After dimensional expansion, the depthwise convolution is used to extract fine-grained features of the overall features, and then the dimensionality reduction operation is used to integrate effective information, and finally the manifolds of interest are obtained.

[0024] Preferably, the GViL module comprises:

[0025] Vision LSTM (ViL), used to model context;

[0026] Gradient path analysis is designed to enhance the global information modeling capabilities of ViL.

[0027] Preferably, the network also includes an AFBR Loss function, and the construction process of the function is as follows:

[0028] The normalized Gaussian distance (NWD) is used to measure the bounding box IoU loss, and the balance factor α is used to achieve a balance with the default CIoU of YOLOv8 to form a new NWD-CIoU bounding box regression loss function, which is the AFBR Loss loss function.

[0029] Preferably, the YOLOv8 loss function comprises two parts: classification loss and bounding box regression loss.

[0030] Compared with the related art, the remote sensing small target detection network based on visible light-infrared dual mode provided by the present invention has the following beneficial effects:

[0031] 1. Based on the multi-order interaction theory of game theory and combined with the idea of ​​"manifolds of interest", this paper proposes a Multi-order Gated Aggregation-Inverted Residual Module (MIR). MIR significantly alleviates the information loss in the forward propagation of small-scale targets and greatly improves the detection accuracy.

[0032] 2. The present invention adopts YOLOv8s as the basic model, and further designs a dual-modality remote sensing target detection architecture Dual-modality Framework based on multi-stage feature fusion, which effectively integrates multimodal information to significantly improve the detection performance.

[0033] 3. Taking advantage of the gradient path analysis, the present invention designs a GViL module to enhance the global information modeling capability of ViL. Experimental results confirm the effectiveness of the module.

[0034] 4. This invention introduces methods such as Adaptive Threshold Focal Loss and Normalized Wasserstein Distance, which can greatly improve the performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Images of small-scale target instances from a bird’s-eye view in the VEDAI dataset: (a) reveals the similarities in physical features between different targets; (b) shows targets under occlusion and shadow, emphasizing the necessity of multimodal data fusion; (c) shows targets that are easily misidentified against complex backgrounds;

[0036] Figure 2 This is the MIR-YOLO architecture diagram proposed in the present invention;

[0037] Figure 3 The following are the architecture diagrams of the IA block and the GViL block in the present invention: (a) shows the design details of the IA block structure; (b) describes the specific architecture of the GViL block;

[0038] Figure 4 It is a schematic diagram of the inverted Residual Block structure;

[0039] Figure 5 Visualization of YOLOv8+MIR heat map and related comparison: The red ellipse circled part indicates that the network model failed to correctly identify and pay attention to; the cyan ellipse area indicates that the model obviously misallocated attention;

[0040] Figure 6 The structure is designed for two fusion modes: early fusion and mid-term fusion;

[0041] Figure 7 Schematic diagram for comparing different models: Figures ad represent various samples in the VEDAI dataset. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0043] 1. The remote sensing small target detection network based on visible light-infrared dual modality mainly includes the following aspects.

[0044] 1.1. MIR and dual-modality MIR framework:

[0045] Inspired by the relevant research on game-theoretic multi-order interactions between pixels in images, the present invention re-examines the application of ConvNets, ViTs, and CNN-ViT series network architectures in the field of remote sensing target detection. According to the three important concepts proposed by the game-theoretic multi-order interaction theory in deep neural networks: "simple (local) concepts (low-order)", "middle-complex concepts (middle-order)", and "complex (global) concepts (high-order)". Moreover, existing theories believe that ConvNets pay more attention to "simple (local) concepts" and ViTs pay more attention to "complex (global) concepts". It is worth noting that the theory has proved through experiments that "middle-complex concepts" are more robust feature representations. The current CNN-ViT series architecture is exploring "middle-complex concepts", but it still faces challenges. At the same time, it benefits from the multi-order gated aggregation network proposed by Li et al. based on this theory.

[0046] Figure 2 As shown in Figure 1, MIR-YOLO consists of two main parts: the backbone network and the head. The backbone network consists of two feature extraction branches that process two different information modalities. The head is responsible for multi-scale feature fusion and detection tasks. Specifically, the MIR module is designed to effectively aggregate semantic information, while the GViL block enhances the global understanding ability of the model.

[0047] Figure 3 (a) shows the design details of the IA block structure, with special attention paid to the lightweight multi-order gating aggregation part, which mainly consists of the gating branch and the information aggregation branch. Figure 3 (b) Describes the specific architecture of the GViL block, a component that effectively utilizes the global modeling capabilities of visual LSTM.

[0048] (multi-order gated aggregation network, MogaNet), the present invention rethinks the design of the network architecture of remote sensing target detection tasks. In remote sensing small target detection, in order to overcome the challenges brought by small-scale targets in remote sensing images and reduce the effective information loss caused by small targets in the forward propagation process, the present invention proposes a MIR (Moga inverted Residual) module based on the research conclusions of multi-order interactions in game theory. The backbone of the network of the present invention is mainly composed of MIR stacking. The MIR module consists of two parts: Information Aggregation Block (IABlock) and inverted Residual Block. Among them, the IA Block mainly aggregates local perception and global context information through a gating mechanism, and the present invention regards the entire MIR module as a large activation layer, and regards its output as a "manifold of interest". Researchers have long believed that "manifolds of interest" in neural networks can be embedded in low-dimensional subspaces. Specifically, such as Figure 2 The structure shown in the MIR Block in the figure, the present invention believes that the information contained in all channels of the output manifold of IABlock is more than the information contained in the manifold of interest, that is, the output manifold of IABlock contains some redundant information. Therefore, the present invention connects IA Block and inverted residual Block in series. The inverted residual Block mainly expands the "output manifold" of IA Block to the middle layer first, decouples the information of the narrow layer, and uses the depth-separable convolution to fully integrate the effective information of the middle layer. Finally, it is projected back to the low-dimensional representation through linear convolution, which proves that such operation can effectively retain effective information. In general, the MIR module can increase the network depth and obtain deeper feature representations while reducing information redundancy, fully retaining multi-order gated aggregation information, and reducing information loss, thereby improving the accuracy of remote sensing small target detection. The heat map visualization comparison results of the overall MIR module and other advanced backbone networks are shown in the figure. Figure 5 As shown, Figure 5 YOLOv8+MIR heat map visualization and related comparison: The red ellipse circle indicates that the network model fails to correctly identify and focus on the key areas related to the task, which may lead to a decrease in prediction performance. The cyan ellipse area indicates that the model obviously misallocates attention, paying more attention to the background and irrelevant details. The comparison with MobileNetv4 and YOLOv8s+C2f fully verifies the effectiveness of the MIR module.

[0049] IA Block makes full use of the above ideas. In order to fully aggregate local perception and global context information and stimulate more robust and expressive mid-order interaction strength, the present invention focuses on designing the LightWeight Multi-order Gated Aggregation (LightWeight Moga) lightweight multi-order gated aggregation part. Figure 3 As shown in (a), first, in the multi-order information aggregation branch R(.) of this part, the present invention uses a multi-branch structure, cooperates with the large kernel channel-by-channel convolution Large Kernel DepthwiseConv, and uses the expansion operation (dilation∈{1,2,3}), which can realize the adjustment of model receptive fields of different sizes. Such an operation enables the present invention to obtain low-order, medium-order, and high-order information in the multi-order interaction theory of game theory respectively, and then use Concatenate to perform multi-branch information fusion. In the left gating branch L(.) of IA Block, the present invention uses Partial Conv and ChannelShuffle operations to reduce the amount of floating-point operations and enhance the information flow between channels. In the gating branch L(.), SiLU can effectively and adaptively select the local perception and context information aggregated by R(.) in the multi-order information aggregation branch on the right. The specific expression of the IA Block part is:

[0050] Y=Conv 1×1 (Norm(X in )) (1.1)

[0051] Z=GELU(Y+(GAP(Y)-Y)a⊙γ) (1.2)

[0052] Among them, X in represents the input feature, Y represents the intermediate tensor. Due to the inductive bias of the convolution operation, Y has static local low-order features. GAP(.) can extract global complex shape information. Z represents the feature map after GELU activation, and γ represents the scale factor.

[0053] Y out =LightWeightMoga(Z) (1.3)

[0054] where Y out Represents the output manifold of IA Block. LightWeightMoga efficiently aggregates local perception and context information while achieving lightweight. Figure 5 As shown in ., LightWeightMoga contains two branches, and the specific expression of its left gating branch is as follows.

[0055] L(X in )=SiLU(Conv 1×1 (CS(PConv(X in ))) (1.4)

[0056] L(.) represents the left gated branch of LightWeightMoga. CS represents the ChannelShuffle operation, and PConv represents Partial Conv. Both of them effectively enhance the flow of information. It is worth noting that Figure 3 (a), the information aggregation branch on the right contains multiple sub-branches, and its specific expression is as follows:

[0057] part1,part2,part3=Split(DWconv 5×5 (X in )) (1.5)

[0058] Z = Concatenate(DWconv 5×5 (part1),part2,DWConv 7×7 (part3)) (1.6)

[0059] R(X in )=L(Z) (1.7)

[0060] R(.) represents the right information aggregation branch. Assuming that the number of channels before channel splitting is C, the number of channels corresponding to part1 is divided into Corresponding DWConv 5×5 , dilation value is 2, which can appropriately expand the model receptive field and is mainly responsible for the extraction of mid-order features. The number of channels corresponding to part3 is divided into Corresponding DWConv 7×7 The dilation of is set to 3 to fully obtain the global context information, and part 1 mainly captures the low-order feature representation. From the perspective of multi-order interactions in game theory, the final Concatenate operation fully integrates low-order, mid-order, and high-order information, improving the robustness and expressiveness of the model.

[0061] Finally, the output X of the entire module MIR out It can be expressed as:

[0062] X out =Conv 1×1 (CS(PConv((L(X in )⊙R(X in ))))) (1.8)

[0063] The main function of the inverted residual block is to integrate the effective information in the information aggregated by the IA Block to obtain the "manifolds of interest". The main operation of the present invention is to first perform an expansion convolution operation on the output manifold of the IA Block, where the expansion ratio is 4. This operation uses Conv 1×1 After dimensional expansion, the depthwise convolution is used to extract fine-grained features of the overall image, and then the dimension reduction operation is used to integrate effective information, and finally the manifolds of interest are obtained. The specific process is expressed as:

[0064] Y out =ProjectConv(DWConv(ExpansionConv)) (1.9)

[0065] The final output is Y out Represents manifolds of interest, the main implementation method of ExpansionConv is Conv 1×1 , the purpose is to increase the dimension according to the ratio. On the contrary, the main purpose of ProjectConv is to achieve dimensionality reduction. The number of dimensions before and after dimensionality increase is the same. DWConv is mainly to achieve information interaction after dimensionality increase, but does not introduce additional computational effort.

[0066] Based on the remarkable performance of the MIR module, the present invention combines the multi-stage feature level fusion method to achieve information complementarity between the visible light and infrared modalities. The present invention designs a dual-modality MIR framework for target detection. The specific structure is as follows: Figure 2 shown.

[0067] 1.2, GViL Block:

[0068] Considering the complexity and diversity of remote sensing scenes and the variability of spatiotemporal resolution, the ability to understand the global image can provide more accurate semantic information for remote sensing scene detection. The generalized efficient layer aggregation network structure (GeneralizeEfficient Layer Aggregation Network) showed excellent results in YOLOv9, proving the effectiveness of the gradient path analysis design. In the present invention, the present invention proposes GViL Block based on the network structure design of gradient path analysis. The network structure design of gradient path analysis can enhance the expression ability of the network and is an efficient and high-quality network architecture. The present invention utilizes the inherent characteristics of the Vison LSTM model, and GViL can effectively capture the global dependencies in remote sensing images, thereby generating rich semantic information. In the remote sensing small target detection task, multi-scale features are of great significance for encoding targets with scale differences. The common strategy for multi-scale feature extraction is to adopt the classic top-down and bottom-up feature pyramid networks. The present invention combines GViL Block and YOLOv8 feature pyramid to fine-grainedly integrate the physical information of shallow networks and the rich semantic information of deep networks, thereby improving the accuracy of remote sensing small target detection.

[0069] GViL Block is highly efficient and lightweight. Figure 3 As shown in (b), first, a Conv1×1 operation is performed to extract features, and then a Cross Stage Partial Operation is performed, which proves that such an operation can maximize the gradient source and gradient timestamp and improve the reasoning speed. Then, the present invention performs a stacking operation of ViL Block on Part2, which can utilize the inherent characteristics of ViL Block's long short-term memory to achieve global range perception. It is worth noting that the processing of the input of ViL Block is similar to that of the input of ViL Transformer. The input image is first divided into small blocks and linearly projected, and then the position code is embedded to promote the visual perception understanding of the precise position, and a series of patch tokens are generated. The sequence is then processed by alternating mLSTM blocks, where the odd blocks process the patches row by row from the upper left to the lower right, and the even blocks process from the lower right to the upper left. This simple alternating design allows ViL to effectively process non-sequential inputs, such as images, without introducing additional calculations. Finally, the results of the multi-branch structure are merged, and then a Conv1×1 convolution operation is performed to achieve interaction and integration of information between channels. The overall expression can be expressed as:

[0070] part1,part2=Split(Conv 1×1 (X in)) (1.10)

[0071] Among them, X in Represents the input feature, part1 and part2 are the results of equal division of the input tensor.

[0072] Y=ViLBlock(part2) (1.11)

[0073] Y out =Conv 1×1 (Concat(part1,part2,Y,ViLBlock(Y))) (1.12)

[0074] Y is an intermediate tensor used to store intermediate results. out It is the output feature of the entire module, containing extensive high-level semantic information.

[0075] 1.3 AFBR Loss:

[0076] In remote sensing small target detection, due to the extreme imbalance between the target and the background, the model pays more attention to the background features rather than the target features, which is one of the reasons why small target detection is considered a challenging task. To solve this problem, the present invention introduces a new Adaptive Threshold FocalLoss (ATFL) function, which decouples the target and the background, and uses an adaptive mechanism to adjust the loss weight, forcing the model to allocate more attention to the target features. Secondly, the bounding box regression is extremely sensitive to small remote sensing targets. To overcome this problem, the present invention uses the normalized Gaussian distance (NWD) to measure the bounding box IoU loss, and achieves a balance with the default CIoU of YOLOv8 through the balance factor α, forming a new NWD-CIoU bounding box regression loss. The new loss function is called AFBR Loss, and its expression is:

[0077]

[0078] Where LossFunction represents the standard default loss function of YOLOv8. The YOLOv8 loss function mainly consists of two parts: classification loss and bounding box regression loss. pos Indicates the total number of cells where the target exists.

[0079] ATFL represents the adaptive threshold focal loss function, specifically, when When the label value is positive, p t =p, otherwise, p t =1-p, where p refers to the predicted probability value. Represents the predicted probability value of the next epoch. λ is a hyperparameter that needs to be manually adjusted.

[0080]

[0081]

[0082] NWD stands for Normalized Gaussian Wasserstein Distance, Denote two Gaussian distributions N a , N b The second-order Wasserstein distance, C is a hyperparameter, usually set to the average size of the corresponding dataset, and α represents the balance coefficient, which is used to balance the relative size of CIoU and NWD.

[0083] 2. Experiment:

[0084] In order to verify the effectiveness and robustness of the MIR-YOLO method proposed in the present invention in remote sensing small target detection, the present invention has been fully verified on public multimodal remote sensing target detection datasets such as VEDAI and DroneVehicle, and compared with current advanced methods.

[0085] 2.1、Dataset:

[0086] VEDAI is a dataset for detecting tiny vehicles in aerial images. It has two modalities, infrared and visible light, and consists of 1,246 images. VEDAI is a benchmark for automatic target detection algorithms in unconstrained environments. In addition to their small size, the vehicles included in this dataset also exhibit different variability, such as multiple directions, lighting / shadow changes, specular reflections, or occlusions. In addition, each image has two types of resolution: 1024×1024 and 512×512.

[0087] DroneVehicle is a large-scale RGB-infrared vehicle remote sensing target detection dataset based on the perspective of high-altitude drones. The dataset aims to overcome a series of challenges when the resolution of target objects is low, especially in low-light conditions. DroneVehicle collects 28,439 pairs of RGB-infrared images from drones, covering vehicle targets in scenes such as urban roads, residential areas, and parking lots, from day to night. Due to the large gap between RGB images and infrared images, cross-modal images provide extremely effective information.

[0088] 2.2 Experimental details:

[0089] The network architecture of the present invention is implemented by the public deep learning framework pytorch. The experiments of the present invention are all carried out on the NVIDIA A6000 48GB video memory graphics card. The present invention uses the VEDAI dataset as the benchmark dataset for the ablation experiment of the present invention, and the VEDAI, DroneVehicle and other datasets are used as the benchmark datasets for the comparative experiments of the present invention. It is worth noting that in VEDAI, due to the small number of plane class instances and the significant difference between its shape and other types of instances from a bird's eye view, in order to enhance the robustness of the network structure, the present invention ignores this class. Due to the limitation of computing resources, the present invention only trains half of the DroneVehicle dataset and names the data as DroneVehicle-Half. The hyperparameter settings of the present invention are mainly as follows: the number of training rounds is set to 250 epochs, the AdamW optimizer is used, the batchsize is set to 8, and other parameters are consistent with the default parameters of YOLOv8.

[0090] 2.3 Accuracy evaluation:

[0091] Accuracy evaluation is to measure the consistency and difference between the detection results and the labels. This paper mainly uses recall, precision and mean average precision (mAP) 50 and mAP 50:95 ) and parameter quantities (params) are used to evaluate the methods to be compared and to quantify the performance of the model proposed in the present invention.

[0092] 2.4 Ablation study:

[0093] In order to effectively illustrate the effectiveness of the MIR-YOLO model architecture proposed in the present invention, the present invention uses ablation experiments to verify the three proposed methods. The ablation experiments of the present invention are all conducted on the VEDAI dataset, and the benchmark model is selected to use YOLOv8s.

[0094] A. Verification of baseline model: As shown in Table 1, the present invention compares the latest YOLO series on the VEDAI dataset. The present invention evaluates the size and reasoning ability of the model from performance indicators such as Layers, Params, and GFLOPs. mAP is used to 50 To measure the performance of the model. It is not difficult to find that although YOLOv8n has the lowest number of parameters, its accuracy index mAP 50 On the contrary, YOLOv9 is the SOTA method in the field of object detection, with accuracy indicators such as mAP. 50As high as 66.0%, but it is also accompanied by a huge amount of parameters. In order to balance the detection accuracy and model size, the present invention fully compares the YOLOv8s and YOLOv10 series architectures, and finally the present invention chooses to use the YOLOv8s model as the basic model (Baseline) of the present invention.

[0095] Table 1. The baseline ablation experiments all use the RGB modality in the VEDAI dataset to compare the model size and reasoning ability under different baseline YOLO frameworks.

[0096]

[0097] B. Verify the multi-stage feature fusion method: In order to verify the excellence of multi-stage feature fusion, Figure 6 As shown in Table 2, the present invention designs two common fusion methods for the basic model: early fusion fusion-early and mid-term fusion fusion-mid. As shown in Table 2, the fusion-mid method shows a general effect. Although fusion-early has the highest accuracy index, it ignores the size of the number of parameters. Multi-stage fusion has a lower accuracy index mAP. 50 While reaching 67.0%, the number of parameters is also limited to 16M. It can be seen that multi-stage fusion is a better multi-modal fusion method.

[0098] Table 2. Verification of the impact of different modalities on the VEDAI dataset

[0099]

[0100]

[0101] C. Verification of MIR module: The present invention compares different multi-scale feature extraction modules (C3, C2f, GELAN), as shown in Table 3. The experimental results show that the MIR module designed by the present invention has achieved significant advantages from a comprehensive perspective, among which mAP 50 It reaches 64.8%, which is 0.9% more than the basic model, but uses fewer parameters.

[0102] Table 3. Comparison of multi-scale feature extraction modules on the RGB modality of the VEDAI Dataset

[0103]

[0104] D. Verify the GViL module: In order to verify the effect of the GViL module, it is worth noting that the basic model I chose to use in this experiment is the MIR+Dual-Modality Framework (MDMF) network structure. The specific model structure can be referred to Figure 2 . Based on the gradient planning design, the present invention combines SHViT, the Mamba-Like Linear Attention (MLLA) module of Demystify Mamba, and the Dilated Reparam Block (DRB) of UniRepLKNet. SHViT proposes a unique single-head self-attention mechanism; the MLLA module can capture contextual relationships extensively, and has only a linear computational complexity relationship; the Dilated Reparam Block (DRB) combines the idea of ​​reparameterization and converts the entire module into a large kernel convolution equivalently. As shown in Table 4, the advantages of the GViL module designed by the present invention are fully demonstrated. A good balance is achieved between the precision and recall parameters, and the mAP 50 The indicator is as high as 68.7%, and the more stringent indicator mAP 50:95 It also reached 40.2%.

[0105] Table 4. Comparison of the performance of various advanced modules in capturing global information

[0106]

[0107] E. Verify AFBR Loss: Based on the original model of YOLOv8s, the present invention introduces the advanced Slideloss, which can adaptively learn the positive sample threshold parameters and the negative sample threshold parameters. FocalerLoss pays more attention to the impact of difficult and easy samples on the bounding box regression results, focusing on different regression samples to improve the effect of the detector. MPDIoULoss optimizes the situation where the predicted box and the actual box have the same aspect ratio, but the height and width of the two are different. As shown in Table 5, under the same conditions of other parameters, MPDIoULoss is better than the AFBR Loss cited in the present invention in mAP. 50 The accuracy index is 0.4% higher, but AFBR Loss has a higher requirement for mAP. 50:95 The accuracy index is 0.3% higher.

[0108] Table 5. Performance comparison of different loss functions on basicmodel

[0109]

[0110] F. Overall ablation experiment: The overall ablation experiment of the present invention is conducted on the VEDAI dataset. It is worth noting that the single-modal network architecture uses the RGB visible light modality for experiments. The experimental results in Table 6 show that compared with the basic model (YOLOv8s), the final accuracy index mAP50 of the model proposed by the present invention reaches 69.7% (an increase of 5.8%), and mAP50 reaches 41.2% (an increase of 3.3%). Since the network designed by the present invention is a dual-modal network, the number of parameters increases by 4.96M compared to the original.

[0111] Table 6. Ablation test results of the designed module as a whole

[0112]

[0113] In this section, the present invention makes a full comparison of the latest algorithms and the proposed method on the VEDAI dataset. These methods mainly come from the latest YOLO series, such as YOLOv9, YOLOv10m; VisionTransformer series, such as SHViT, EfficientViT, and large kernel convolutional network series, such as the recent LSKNet. The experimental results are shown in Table 7. EfficientViT has the lowest number of parameters, and YOLOv9 has the lowest mAP in the detection of Trackers category. 50 The index is the highest, but a large number of parameters are generated. YOLOrs can achieve the highest accuracy in the Pickup and Vans categories, but it is obviously insufficient in the detection of the Boats category, mainly due to the low recall rate. The model proposed in the present invention can balance the size of the parameters while significantly higher than other advanced models in the recognition accuracy of car, pickup, truck, others and boats categories, and the comprehensive index mAP50 reaches 69.7%, which is much higher than other detectors.

[0114] 2.5. Comparative experiment:

[0115] Table 7. Comparative experiments on the VEDAI dataset with other advanced models

[0116]

[0117]

[0118] In order to fully verify the generalization of the model, in this section, the present invention is fully verified on the Droneviewicle visible light-infrared dataset. It is worth noting that due to the limitation of the graphics card computing power, the dataset used by the present invention only uses half of the Droneviewicle dataset. The present invention divides the training set of the Droneviewicle dataset into two parts, and only trains the first part, with a total of 8995 training set pictures and a total of 1469 test set pictures. In this section, the present invention chooses to compare the latest models, such as YOLOv8s of the YOLO series, which can achieve a good balance between accuracy and parameter quantity; YOLOv9 reduces the information loss of the target during forward propagation through auxiliary reversible branches, greatly improving the target detection accuracy. RepViT can learn global representations excellently, and through structural reorganization (Structural Re-parameterization), it introduces a multi-branch topology structure, which greatly improves the performance of its model. VanillaNet shows excellent performance due to its deep training strategy. As shown in Table 8, the detection accuracy of the model of the present invention is higher than that of other advanced models in each type, mAP 50 Achieved 50.2%, mAP 50:95 It reaches 31.9%, however, the model of the present invention shows a slightly higher number of parameters.

[0119] Table 8. Comparative experiments with advanced models on the Dronevehivle dataset

[0120]

[0121] Figure 7 The following is a comparative visualization diagram of different models: Figures ad represent various samples in the VEDAI dataset. The area surrounded by the red ellipse is misclassified even though it has regressed to the position of the positive sample; the area surrounded by the yellow ellipse indicates an instance where the model mistakenly identifies the positive sample as a negative sample; the area surrounded by the dark brown ellipse shows where the model mistakenly classifies the negative sample as a positive sample.

[0122] 3. Conclusion:

[0123] In the present invention, a new network architecture, Dual-Modality MIR-YOLO, is introduced to improve the accuracy of remote sensing small target detection. The model takes advantage of the complementary information between multi-modalities and significantly improves the accuracy of target detection. In the model structure, the present invention proposes two core modules, MIR Block and GViL Block. MIRBlock can effectively alleviate the information loss problem of small-scale targets in the feedforward process. The experimental results show that the model accuracy of YOLOv8s combined with MIR Block is 0.9% higher than the accuracy of the Baseline model, and the dual-modal architecture is 1.6% higher than the dual-modal architecture based on the Baseline model, and 4.7% higher than the simple baseline model. The present invention innovatively uses Vision LSTM and proposes a GViL module, which effectively solves the complexity and variability of remote sensing background. GViL can model the global range well and obtain rich semantic information. By combining the GViL module with the PAN-FPN structure, coordinating the dual-modal structure designed by the present invention, and the introduced loss function, the accuracy of the final model reaches 69.7%, exceeding the accuracy of many advanced models.

[0124] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A remote sensing small target detection network based on visible light-infrared dual mode, characterized in that: include: Multi-order Gated Aggregation-Inverted Residual Module (MIR), which is used to effectively aggregate semantic information and alleviate the information loss of small-scale targets in forward propagation; The cross-stage multimodal fusion network architecture is used to integrate the information of visible light and infrared modalities, realize the information complementarity between visible light and infrared modalities, and improve the detection accuracy; The gradient-based visual long short-term memory network module, or GViL module, is used to enhance the model’s global understanding and address the complexity and diversity of remote sensing scenes. A new IoU loss function is constructed by combining the adaptive threshold focal loss (ATFL) with the normalized Wasserstein distance (NWD) CIoU to solve the problem of imbalance between target and background and the extreme sensitivity of bounding box regression to small remote sensing targets.

2. The remote sensing small target detection network based on visible light-infrared dual mode as claimed in claim 1, characterized in that: The MIR module includes: Information Aggregation Block (IA Block), which includes a gating branch and a local-global semantic information aggregation branch. It mainly aggregates local perception and global context information through a gating mechanism. The inverted residual block is used to integrate the effective information in the output manifold of the IA Block to obtain the "manifolds of interest".

3. The remote sensing small target detection network based on visible light-infrared dual mode as claimed in claim 2 is characterized in that: The operation of integrating the valid information in the information aggregated by IA Block to obtain the "manifolds of interest" is: First, perform Expansion Convolution on the output manifold of IA Block, where the value of ExpansionRatio is 4. This operation uses Conv 1×1 After dimensional expansion, the depthwise convolution is used to extract fine-grained features of the overall features, and then the dimensionality reduction operation is used to integrate effective information, and finally the manifolds of interest are obtained.

4. The remote sensing small target detection network based on visible light-infrared dual mode as claimed in claim 1, characterized in that: The GViL module includes: Visual Long Short-Term Memory (VisionLSTM, ViL), used to model context; Gradient path analysis is designed to enhance the global information modeling capabilities of ViL.

5. The remote sensing small target detection network based on visible light-infrared dual mode as claimed in claim 1, characterized in that: Including the AFBRLoss loss function, its construction process is: The normalized Gaussian distance (NWD) is used to measure the bounding box IoU loss, and the balance factor α is used to achieve a balance with the default CIoU of YOLOv8 to form a new NWD-CIoU bounding box regression loss function, which is the AFBRLoss loss function.

6. The remote sensing small target detection network based on visible light-infrared dual mode as claimed in claim 5, characterized in that: The YOLOv8 loss function consists of two parts: classification loss and bounding box regression loss.

Images