Target detection method and device based on adaptive modal selection, equipment and medium

Abstract

The present application relates to the technical field of artificial intelligence, and provides a target detection method and device based on adaptive modal selection, equipment and medium, which can extract multi-scale visible light features and thermal infrared features respectively by using independent lightweight backbone networks, so as to reduce the overhead; cross-modal attention fusion is performed on the visible light features and the thermal infrared features of the finest scale, which can further reduce the parameter quantity and the calculation quantity; adaptive modal selection is performed according to the cross-modal attention fusion features and the thermal infrared features of the finest scale, so as to dynamically adjust the fusion weight and effectively improve the generalization ability; the adaptive modal selection fusion features, the intermediate scale fusion features and the coarsest scale fusion features are input into a detection head to obtain a target detection result, and more accurate target detection based on multi-modal and multi-scale features is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a target detection method, apparatus, device, and medium based on adaptive modality selection. Background Technology

[0002] In safety-critical applications such as autonomous driving, drone inspection, and security monitoring, target detection systems need to maintain stable detection performance under various lighting conditions (such as daytime, nighttime, dusk, rain, and fog). Visible light (Red, Green, Blue, RGB) cameras can provide rich color and texture information under good lighting conditions, but their performance degrades sharply in low light, at night, or in inclement weather. Thermal infrared (IR) cameras sense thermal radiation signals and are unaffected by ambient light, remaining effective at night and in low light conditions, but they lack the fine-grained texture and edge information found in RGB images, and their detection performance is poor when the target and background temperatures are similar. Therefore, fusing the complementary modes of RGB and IR is an effective way to achieve robust all-weather detection.

[0003] However, existing RGB-IR fusion methods typically introduce a large number of additional parameters and computational overhead. In addition, most existing methods adopt a uniform fusion strategy for all spatial locations of the entire image, ignoring the differences in modal dependence of different regions (e.g., shadow areas are more dependent on IR, while bright areas are more dependent on RGB), and also lack the adaptive ability to dynamically adjust modal weights according to the overall scene illumination.

[0004] Therefore, there is an urgent need for a lightweight, adaptive RGB-IR fusion target detection method. Summary of the Invention

[0005] In view of the above, it is necessary to provide a target detection method, device, equipment and medium based on adaptive mode selection, which aims to solve the problems of high cost and lack of adaptive capability in the target detection process.

[0006] A target detection method based on adaptive modality selection, the target detection method based on adaptive modality selection includes: In response to a target detection command triggered in the target scene, the visible light image and thermal infrared image of the target in the target scene are acquired; The target visible light image is subjected to multi-scale feature extraction using a first lightweight backbone network to obtain visible light features at multiple scales, and the target thermal infrared image is subjected to multi-scale feature extraction using a second lightweight backbone network independent of the first lightweight backbone network to obtain thermal infrared features at multiple scales. Cross-modal attention fusion is performed on the most detailed visible light features and thermal infrared features to obtain cross-modal attention fusion features; Adaptive mode selection is performed based on the cross-modal attention fusion features and the finest-scale thermal infrared features to obtain adaptive mode selection fusion features; The visible light features and thermal infrared features at the intermediate and coarsest scales are fused separately to obtain the intermediate-scale fused features and the coarsest-scale fused features. The adaptive modality selection fusion feature, the intermediate scale fusion feature, and the coarsest scale fusion feature are input into the detection head to obtain the target detection result.

[0007] A target detection device based on adaptive mode selection, the target detection device based on adaptive mode selection includes: The acquisition unit is used to acquire visible light images and thermal infrared images of the target in the target scene in response to a target detection command triggered in the target scene. The extraction unit is used to extract multiple scales of visible light features from the target visible light image using a first lightweight backbone network, and to extract multiple scales of thermal infrared features from the target thermal infrared image using a second lightweight backbone network independent of the first lightweight backbone network. The fusion unit is used to perform cross-modal attention fusion of the most detailed visible light features and thermal infrared features to obtain cross-modal attention fused features; The selection unit is used to perform adaptive mode selection based on the cross-modal attention fusion features and the finest-scale thermal infrared features to obtain adaptive mode selection fusion features; The fusion unit is also used to fuse the visible light features and thermal infrared features at the intermediate scale and the coarsest scale respectively to obtain the intermediate scale fused features and the coarsest scale fused features. The input unit is used to input the adaptive modality selection fusion feature, the intermediate scale fusion feature and the coarsest scale fusion feature into the detection head to obtain the target detection result.

[0008] A computer device, the computer device comprising: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the target detection method based on adaptive modality selection.

[0009] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the target detection method based on adaptive modality selection.

[0010] As can be seen from the above technical solutions, this invention can extract multi-scale visible light features and thermal infrared features using independent lightweight backbone networks to reduce overhead; cross-modal attention fusion of the finest-scale visible light features and thermal infrared features can further reduce the number of parameters and computational load; adaptive modality selection is performed based on the cross-modal attention fusion features and the finest-scale thermal infrared features to dynamically adjust the fusion weights, which can effectively improve the generalization ability; the adaptive modality selection fusion features, intermediate-scale fusion features and coarsest-scale fusion features are input into the detection head to obtain the target detection result, thus realizing more accurate target detection based on multi-modal and multi-scale features. Attached Figure Description

[0011] Figure 1 This is a flowchart of a preferred embodiment of the target detection method based on adaptive modality selection of the present invention.

[0012] Figure 2 This is a functional block diagram of a preferred embodiment of the target detection device based on adaptive mode selection of the present invention.

[0013] Figure 3 This is a schematic diagram of the structure of a computer device that implements a preferred embodiment of the target detection method based on adaptive modality selection according to the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the target detection method based on adaptive modality selection according to the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0016] The target detection method based on adaptive modality selection is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0017] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), interactive network television (IPTV), smart wearable device, etc.

[0018] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0019] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0020] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0021] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0022] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0023] This embodiment can be applied to an all-weather target detection system for low-altitude UAV inspection. The target detection system adopts an edge-cloud collaborative architecture of "edge-side acquisition, edge-side inference, and cloud-side optimization." Specifically, it includes: End-side: Visible light (Red, Green, Blue, RGB) cameras and thermal infrared (IR) cameras mounted on drones (they can also be mounted on inspection robots or fixed monitoring platforms) have similar fields of view and are triggered synchronously to ensure that the image pairs are aligned in space and time. The drone flies along a preset route and triggers the dual cameras to take pictures simultaneously at designated waypoints to obtain RGB and IR image pairs of the same scene.

[0024] Edge computing: Deployed on the airborne edge computing device of the UAV, integrating various functional units and modules. The edge device receives image pairs in real time to complete preprocessing, feature extraction, fusion and target detection, and finally outputs the detection results.

[0025] Cloud side: The cloud server includes functional modules such as data storage, model training, and feedback optimization. The edge device uploads object detection results, detection confidence, spatial weight map, etc. to the cloud. The cloud uses the accumulated data to perform incremental model training and parameter optimization. The updated model and parameters can be distributed to the edge device after verification.

[0026] The above architecture can form a closed loop of "collection-detection-interpretation-feedback-optimization".

[0027] In this embodiment, the target detection method based on adaptive modality selection specifically includes: S10, in response to a target detection command triggered in the target scene, acquires a visible light image and a thermal infrared image of the target in the target scene.

[0028] In this embodiment, the target scene can be a scene where target detection needs to be performed.

[0029] In this embodiment, the target detection command can be triggered according to actual detection needs.

[0030] In this embodiment, acquiring the target visible light image and the target thermal infrared image of the target scene includes: The visible light image and the thermal infrared image of the target are acquired simultaneously using a co-located or near-co-located visible light camera and a thermal infrared camera.

[0031] For example, after the drone arrives at the inspection point, it simultaneously triggers the RGB and IR cameras to acquire RGB and IR images with a resolution of 640×640. The acquired images are then transmitted in real time to an edge computing device, where preprocessing operations such as size normalization (already 640×640), median filtering for noise reduction, and brightness normalization (stretching the brightness of RGB and IR images to the [0,1] range) are performed to obtain the preprocessed visible light image of the target. and target thermal infrared image .

[0032] S11, a first lightweight backbone network is used to extract multiple scale features from the target visible light image to obtain multiple scale visible light features, and a second lightweight backbone network independent of the first lightweight backbone network is used to extract multiple scale features from the target thermal infrared image to obtain multiple scale thermal infrared features.

[0033] In this embodiment, the first lightweight backbone network and the second lightweight backbone network can be two independent networks that do not share parameters, so as to adapt to the different physical characteristics of RGB and IR respectively. Specifically, the lightweight backbone network can include lightweight convolutional neural networks such as YOLOv8n (You Only Look Once version 8 nano), MobileNet, or ShuffleNet.

[0034] For example, the first lightweight backbone network and the second lightweight backbone network can be two independent YOLOv8n backbone networks. The first lightweight backbone network can be initialized with weights pre-trained on the COCO (Common Objects in Context) dataset, and the second lightweight backbone network can be initialized with weights pre-trained on the target dataset from an IR-only (Infrared-Only) detector.

[0035] In this embodiment, the multiple scales can be the following three scales: P3 (80×80×64), P4 (40×40×128), and P5 (20×20×256). Among them, P3 is the highest spatial resolution, and P5 is the lowest spatial resolution. The P3 layer retains rich spatial details and is most critical for small targets and cross-modal complementarity.

[0036] S12 performs cross-modal attention fusion on the most detailed visible light features and thermal infrared features to obtain cross-modal attention fusion features.

[0037] In this embodiment, the finest scale corresponds to the highest spatial resolution.

[0038] In this embodiment, the cross-modal attention fusion of the most minute visible light features and thermal infrared features to obtain cross-modal attention fusion features includes: The visible light features at the finest scale are enhanced with channel attention and spatial attention to obtain visible light attention features; The finest-scale thermal infrared features are subjected to channel attention enhancement and spatial attention enhancement to obtain thermal infrared attention features. The visible light attention features and the thermal infrared attention features are exchanged across modalities using depthwise separable convolution to obtain visible light cross features and thermal infrared cross features. The visible light attention features are weighted element-wise using the sigmoid activation values ​​of the thermal infrared cross features to obtain visible light weighted features; The thermal infrared attention features are weighted element-wise using the sigmoid activation values ​​of the visible light cross features to obtain the thermal infrared weighted features; The visible light weighted features and the thermal infrared weighted features are gated and fused to obtain the cross-modal attention fusion features.

[0039] Among them, channel attention uses an adaptive kernel-size 1D convolution (ECA (Efficient Channel Attention) mechanism) to capture cross-channel dependencies.

[0040] Specifically, channel attention enhancement of the most fine-scale visible light features includes: performing global average pooling on the most fine-scale visible light features to obtain a 1×1×C vector, generating channel weights through a one-dimensional convolution with an adaptive kernel size (the kernel size is automatically determined according to the number of channels) and a sigmoid function, and then multiplying it with the original most fine-scale visible light features.

[0041] The spatial attention enhancement of the finest-scale visible light features includes: performing max pooling and average pooling on the finest-scale visible light features in the channel dimension to obtain two H×W×1 feature maps, concatenating them, and generating a spatial attention weight map through 7×7 convolution and sigmoid activation, and then multiplying it element-wise with the original finest-scale visible light features.

[0042] Specifically, the cross-modal feature exchange between the visible light attention features and the thermal infrared attention features using depthwise separable convolution includes: employing a 3×3 depthwise separable convolution (number of groups = number of input channels) for cross-modal feature exchange, thereby achieving information exchange between modalities while reducing the number of parameters to 1 / C of that of conventional convolution. in C in Indicates the number of input channels.

[0043] Specifically, the visible light attention features are weighted element-wise using the sigmoid activation value of the thermal infrared cross features, and the thermal infrared attention features are weighted element-wise using the sigmoid activation value of the visible light cross features. This allows the IR features to suppress noise regions (such as shadows and overexposure) in the RGB features, while the RGB features can highlight discriminative textures in the IR features.

[0044] The step of gating and fusing the visible light weighted features and the thermal infrared weighted features to obtain the cross-modal attention fusion features includes: The finest visible light features are sequentially input into a global average pooling layer and a 1*1 convolutional layer, and then activated by sigmoid to obtain a channel-level visible light gating vector. The finest-scale thermal infrared features are sequentially input into the global average pooling layer and the 1*1 convolutional layer, and then activated by sigmoid to obtain a channel-level thermal infrared gating vector. The first feature is obtained by calculating the product of the visible light weighted feature and the visible light gate vector; The second feature is obtained by calculating the product of the thermal infrared weighted feature and the thermal infrared gated vector; The first feature and the second feature are concatenated along the channel dimension to obtain the first concatenated feature; The first spliced ​​feature is projected through a 1*1 convolution to obtain the cross-modal attention fusion feature.

[0045] Taking P3 (80×80×64) as the finest scale as an example, the shape of the cross-modal attention fusion feature is 80×80×64.

[0046] In the above embodiments, depthwise separable convolution and 1D channel attention are used to achieve cross-modal attention fusion, which significantly reduces the number of parameters and computational cost compared to traditional Transformer-based fusion methods. Specifically, only about 0.42M parameters are added to achieve deep cross-modal interaction, with a total model parameter count of about 5M. The inference speed on an edge GPU (Graphics Processing Unit) can reach 238 FPS (Frames Per Second), meeting the requirements of real-time applications.

[0047] S13, perform adaptive mode selection based on the cross-modal attention fusion features and the finest-scale thermal infrared features to obtain adaptive mode selection fusion features.

[0048] In this embodiment, the step of performing adaptive mode selection based on the cross-modal attention fusion features and the finest-scale thermal infrared features to obtain adaptive mode selection fusion features includes: The cross-modal attention fusion feature and the finest-scale thermal infrared feature are spliced ​​together along the channel dimension to obtain the second spliced ​​feature; The second concatenated feature is subjected to global average pooling to obtain a global feature vector. The global feature vector is input into the scene analysis network to obtain the global scene modality preference vector; The second stitched feature is input into the spatial weight generation network to obtain a pixel-level modality preference spatial weight map; The fusion weights are calculated based on the global scene modal preference vector and the pixel-level modal preference space weight map. The cross-modal attention fusion feature and the finest-scale thermal infrared feature are weighted and fused according to the fusion weight to obtain the adaptive modality selection fusion feature.

[0049] For example, the shape of the cross-modal attention fusion feature is 80×80×64, the shape of the finest-scale thermal infrared feature is 80×80×64, and after being stitched together in the channel dimension, the second stitched feature is a tensor of 80×80×128.

[0050] The scene analysis network may include two 1×1 convolutions (with ReLU activation in the middle layer and 8 channels in the middle), outputting a 2D vector. This vector is then processed by softmax under the control of a learnable temperature parameter τ (e.g., a value of 2) to obtain the global scene modality preference vector. The scene analysis network utilizes a lightweight network to aggregate global statistical information to complete scene type estimation.

[0051] For example, the scene analysis network may include a first 1×1 convolutional layer, a ReLU activation layer and a second 1×1 convolutional layer connected in sequence, outputting a global scene modality preference vector with a dimension of 2, and controlling the softness and hardness through a learnable temperature parameter.

[0052] The spatial weight generation network comprises two 1x1 convolutions (with ReLU activation in the middle layer and 16 channels in the middle), outputting a 2×80×80 tensor. This tensor is then processed by softmax with the same temperature parameter τ to obtain the pixel-level modality preference spatial weight map. The spatial weight generation network generates the pixel-level modality preference map through a parallel network.

[0053] For example, the spatial weight generation network may include a third 1×1 convolutional layer, a ReLU activation layer and a fourth 1×1 convolutional layer connected in sequence, outputting a spatial weight map with a dimension of 2, and performing softmax normalization through the learnable temperature parameter.

[0054] The step of calculating the fusion weights based on the global scene modal preference vector and the pixel-level modal preference space weight map includes: Obtain the overall preference weights of the visible light mode and the thermal infrared mode under global illumination from the global scene modal preference vector; Obtain the visible light single-channel pixel weight and the thermal infrared single-channel pixel weight from the pixel-level modal preference space weight map; The first value is obtained by multiplying the overall preference weight of the visible light mode with the pixel weight of the visible light single channel. The second value is obtained by multiplying the overall preference weight of the thermal infrared mode by the pixel weight of the thermal infrared single channel. The third value is obtained by summing the first value, the second value, and the minimum constant. The quotient of the first value and the third value is calculated to obtain the fusion weight.

[0055] The minimum constant can prevent the denominator from being 0.

[0056] The step of weighted fusion of the cross-modal attention fusion feature and the finest-scale thermal infrared feature according to the fusion weight includes: multiplying the fusion weight by the cross-modal attention fusion feature element by element, and multiplying (1-fusion weight) by the finest-scale thermal infrared feature element by element, and concatenating the two features obtained after element-wise multiplication to obtain the adaptive modality selection fusion feature.

[0057] Specifically, the cross-modal attention fusion feature and the finest-scale thermal infrared feature are weighted and fused according to the fusion weight to fuse image spatial details. Based on global illumination statistics, bimodal weights are dynamically assigned to each pixel, and finally, a physically enhanced fusion feature map is output as the input for subsequent target detection and recognition tasks.

[0058] Through the aforementioned fusion mechanism, the model can automatically favor reliable modes under different lighting conditions (e.g., favoring RGB during the day and IR at night), while spatial weights can interpretably demonstrate mode selection preferences. Specifically, through global scene analysis and spatial weight generation, the model can dynamically adjust the fusion weights of RGB and IR based on lighting conditions and local region characteristics, automatically favoring RGB during the day and IR at night, without explicit time or weather labeling, and exhibiting strong generalization ability.

[0059] In this embodiment, the pixel-level modal preference spatial weight map can be visualized to intuitively show the model's modal preferences under different lighting and spatial locations, which helps maintenance personnel understand the model's decision-making basis and improves the system's reliability.

[0060] S14, the visible light features and thermal infrared features at the intermediate scale and the coarsest scale are fused respectively to obtain the intermediate scale fused features and the coarsest scale fused features.

[0061] For example, for P4 (intermediate scale) and P5 (coarsest scale), the corresponding RGB and IR features are concatenated along the channel dimension and then projected back to the original number of channels via a 1*1 convolution to obtain the corresponding fused features.

[0062] S15, the adaptive modality selection fusion feature, the intermediate scale fusion feature and the coarsest scale fusion feature are input into the detection head to obtain the target detection result.

[0063] For example, the detection head can be a YOLOv8n detection head (including classification and regression branches), and the obtained target detection results can include the target's bounding box coordinates, type, and detection confidence.

[0064] The detection head can adopt an anchor-free CenterNet style to predict center point heatmaps, dimensions, and offsets. Alternatively, the detection head can adopt a YOLO-style decoupling head.

[0065] In this embodiment, after obtaining the target detection result, the method further includes: Obtain the detection confidence level and target type from the target detection results; When the detection confidence is less than the confidence threshold, or when the target type is a preset type, the visible light component is extracted from the pixel-level modality preference space weight map; Normalization is performed on the visible light component to obtain the normalized visible light component; The normalized visible light component is superimposed on the target visible light image to obtain a mode selection contribution heatmap.

[0066] For example, the confidence threshold can be 0.5, and the preset type can be pedestrian.

[0067] The modality selection contribution heatmap is used to characterize the dependence of different spatial locations on RGB or IR modes. The modality selection contribution heatmap may be labeled with information such as illumination intensity (e.g., B (Blue) value) and fusion weights, the latter being used to quantify modality preference.

[0068] For example, Gradient-weighted Class Activation Mapping (Grad-CAM) can be used to visualize the decision-making process of a lightweight, interpretable object detection neural network model. Specifically, the output of a target convolutional layer (usually a deep feature layer, such as feature layer (BATCHSIZE, 464, 64, 80)) in the encoder backbone of the lightweight recognition model can be selected as the input feature map A. A fused feature map containing a photovoltaic panel image is input into the network and forward propagated to obtain the network's predicted score for a specific defect category. Through backpropagation, the gradient of the target category prediction score relative to the input feature map A is calculated. This gradient reflects the importance of each location in the feature map to the final classification decision. This gradient is then globally averaged and pooled across the spatial dimensions (height and width) to obtain the weight of each feature channel k, which represents the overall importance of the k-th feature channel to the recognized category. The calculated channel weights are used to perform a weighted summation on each channel of the original feature map A to obtain an initial class activation map, which initially displays the spatial regions related to the identified category. The ReLU activation function is then applied to the initial class activation map to filter out regions that negatively impact the identified category, retaining only positively contributing regions, thus making the heatmap more focused. Finally, the ReLU-activated heatmap is upsampled using methods such as bilinear interpolation to make its size consistent with the original input image, generating the final visualization result, namely the modality selection contribution heatmap.

[0069] Taking the original visible light (RGB) and thermal infrared (IR) images under different lighting conditions and their corresponding gradient-weighted class activation mapping (Grad-CAM) heatmaps as an example, this study compares and analyzes the visible light images, infrared images, and corresponding Grad-CAM feature contribution maps of the same UAV target under three typical scenarios: bright lighting (B=121), medium lighting (B=78-83), and low lighting (B=53). The comparison reveals significant differences in the model's attention distribution under different lighting conditions: In bright scenes, the Grad-CAM activation of the RGB backbone features is highly concentrated in the target area, and the cross-modal attention fusion layer also exhibits a clear hotspot response. The model fully utilizes visible light texture information for accurate discrimination when the RGB modality is reliable. In low-light scenes, the Grad-CAM activation intensity of the RGB backbone is significantly reduced and dispersed, while the activation of the IR backbone is relatively enhanced. The cross-modal attention fusion layer shows a more balanced spatial attention distribution, intuitively verifying the effectiveness of the adaptive modality selection module in automatically increasing the infrared modality contribution when the lighting degrades. The fusion weights of the annotations quantify the modal preferences, namely, the RGB weight is as high as 0.90 in bright scenes, while the RGB weight drops to 0.88 and the IR weight increases accordingly in low-light scenes, which can clearly show the key spectral and spatial information on which the model performs all-weather target detection.

[0070] Through the above embodiments, interpretability analysis can be performed. Operation and maintenance personnel can observe whether the model reasonably biases IR (such as shadow areas) or RGB (such as bright areas) in low confidence areas to help determine whether the detection results are reliable.

[0071] In this embodiment, after obtaining the target detection result, the method further includes: At preset time intervals, all target detection results obtained at the edge can be packaged and uploaded to the cloud (including target bounding box coordinates, type, and detection confidence) and the corresponding modality selection contribution heatmap (only low-confidence samples). The cloud aggregates data uploaded by multiple drones and combines it with manual review and annotation to construct an incremental training set. The model is fine-tuned regularly (weekly): the backbone network is fixed, and only the gating parameters in the gating fusion process and the temperature parameter τ in the adaptive modality selection process are updated to adapt the model to seasonal and lighting changes. After the updated parameters are tested on the validation set (mAP (mean Average Precision) and FPS) and meet the standards, they are distributed to each edge device.

[0072] In the above embodiments, the robustness in cross-seasonal and cross-regional scenarios can be continuously improved through cloud feedback optimization.

[0073] This embodiment, through its lightweight design, significantly improves all-weather target detection accuracy while ensuring real-time inference speed. It can also visualize modality selection behavior through spatial weight graphs, making it suitable for deployment on resource-constrained edge devices. It can be applied to scenarios requiring robust all-weather perception, such as autonomous driving, drone inspection, and security monitoring.

[0074] As can be seen from the above technical solutions, this invention can extract multi-scale visible light features and thermal infrared features using independent lightweight backbone networks to reduce overhead; cross-modal attention fusion of the finest-scale visible light features and thermal infrared features can further reduce the number of parameters and computational load; adaptive modality selection is performed based on the cross-modal attention fusion features and the finest-scale thermal infrared features to dynamically adjust the fusion weights, which can effectively improve the generalization ability; the adaptive modality selection fusion features, intermediate-scale fusion features and coarsest-scale fusion features are input into the detection head to obtain the target detection result, thus realizing more accurate target detection based on multi-modal and multi-scale features.

[0075] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the target detection device based on adaptive modality selection of the present invention. The target detection device 11 based on adaptive modality selection includes a data acquisition unit 110, an extraction unit 111, a fusion unit 112, a selection unit 113, and an input unit 114. The module / unit referred to in this invention refers to a series of computer program segments that can be executed by a processor and perform a fixed function, and are stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0076] The acquisition unit 110 is used to acquire the visible light image and thermal infrared image of the target in the target scene in response to a target detection command triggered in the target scene. The extraction unit 111 is used to extract multiple scales of visible light features from the target visible light image using a first lightweight backbone network, and to extract multiple scales of thermal infrared features from the target thermal infrared image using a second lightweight backbone network independent of the first lightweight backbone network. The fusion unit 112 is used to perform cross-modal attention fusion on the most detailed visible light features and thermal infrared features to obtain cross-modal attention fusion features; The selection unit 113 is used to perform adaptive mode selection based on the cross-modal attention fusion features and the finest-scale thermal infrared features to obtain adaptive mode selection fusion features; The fusion unit 112 is also used to fuse the visible light features and thermal infrared features at the intermediate scale and the coarsest scale respectively to obtain the intermediate scale fused features and the coarsest scale fused features. The input unit 114 is used to input the adaptive modality selection fusion feature, the intermediate scale fusion feature and the coarsest scale fusion feature into the detection head to obtain the target detection result.

[0077] As can be seen from the above technical solutions, this invention can extract multi-scale visible light features and thermal infrared features using independent lightweight backbone networks to reduce overhead; cross-modal attention fusion of the finest-scale visible light features and thermal infrared features can further reduce the number of parameters and computational load; adaptive modality selection is performed based on the cross-modal attention fusion features and the finest-scale thermal infrared features to dynamically adjust the fusion weights, which can effectively improve the generalization ability; the adaptive modality selection fusion features, intermediate-scale fusion features and coarsest-scale fusion features are input into the detection head to obtain the target detection result, thus realizing more accurate target detection based on multi-modal and multi-scale features.

[0078] like Figure 3 The diagram shown is a schematic representation of the structure of a computer device that implements a target detection method based on adaptive modality selection according to a preferred embodiment of the present invention.

[0079] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a target detection program based on adaptive modality selection.

[0080] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0081] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0082] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code of a target detection program based on adaptive modality selection, but also to temporarily store data that has been output or will be output.

[0083] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing a target detection program based on adaptive modality selection) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0084] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes these applications to implement the steps in the various embodiments of the target detection method based on adaptive modality selection described above, for example... Figure 1 The steps are shown.

[0085] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a data acquisition unit 110, an extraction unit 111, a fusion unit 112, a selection unit 113, and an input unit 114.

[0086] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, a computer device, or a network device, etc.) or processor to execute portions of the target detection method based on adaptive modality selection described in the various embodiments of the present invention.

[0087] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0088] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0089] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0090] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0091] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0092] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0093] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish a communication connection between the computer device 1 and other computer devices.

[0094] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0095] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0096] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0097] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a target detection method based on adaptive modality selection, and the processor 13 can execute the multiple instructions to achieve the following: In response to a target detection command triggered in the target scene, the visible light image and thermal infrared image of the target in the target scene are acquired; The target visible light image is subjected to multi-scale feature extraction using a first lightweight backbone network to obtain visible light features at multiple scales, and the target thermal infrared image is subjected to multi-scale feature extraction using a second lightweight backbone network independent of the first lightweight backbone network to obtain thermal infrared features at multiple scales. Cross-modal attention fusion is performed on the most detailed visible light features and thermal infrared features to obtain cross-modal attention fusion features; Adaptive mode selection is performed based on the cross-modal attention fusion features and the finest-scale thermal infrared features to obtain adaptive mode selection fusion features; The visible light features and thermal infrared features at the intermediate and coarsest scales are fused separately to obtain the intermediate-scale fused features and the coarsest-scale fused features. The adaptive modality selection fusion feature, the intermediate scale fusion feature, and the coarsest scale fusion feature are input into the detection head to obtain the target detection result.

[0098] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0099] It should be noted that all the data involved in this case was legally obtained.

[0100] If any AI models, software tools, or components not belonging to this company appear in the embodiments of this invention, they are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this invention has been obtained by an entity authorized (with the knowledge and consent) or fully authorized by all parties through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0101] Unless otherwise specified, the network models used in this embodiment are all general models with a general structure, and are trained and fine-tuned using data related to this embodiment, so as to be applicable to this embodiment.

[0102] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0103] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0104] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0105] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0106] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0107] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0108] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A target detection method based on adaptive modality selection, characterized in that, The target detection method based on adaptive modality selection includes: In response to a target detection command triggered in the target scene, the visible light image and thermal infrared image of the target in the target scene are acquired; The target visible light image is subjected to multi-scale feature extraction using a first lightweight backbone network to obtain visible light features at multiple scales, and the target thermal infrared image is subjected to multi-scale feature extraction using a second lightweight backbone network independent of the first lightweight backbone network to obtain thermal infrared features at multiple scales. Cross-modal attention fusion is performed on the most detailed visible light features and thermal infrared features to obtain cross-modal attention fusion features; Adaptive mode selection is performed based on the cross-modal attention fusion features and the finest-scale thermal infrared features to obtain adaptive mode selection fusion features; The visible light features and thermal infrared features at the intermediate and coarsest scales are fused separately to obtain the intermediate-scale fused features and the coarsest-scale fused features. The adaptive modality selection fusion feature, the intermediate scale fusion feature, and the coarsest scale fusion feature are input into the detection head to obtain the target detection result.

2. The target detection method based on adaptive modality selection as described in claim 1, characterized in that, The acquisition of the target visible light image and the target thermal infrared image of the target scene includes: The visible light image and the thermal infrared image of the target are acquired simultaneously using a co-located or near-co-located visible light camera and a thermal infrared camera.

3. The target detection method based on adaptive modality selection as described in claim 1, characterized in that, The cross-modal attention fusion of the most minute visible light features and thermal infrared features to obtain cross-modal attention fusion features includes: The visible light features at the finest scale are enhanced with channel attention and spatial attention to obtain visible light attention features; The finest-scale thermal infrared features are subjected to channel attention enhancement and spatial attention enhancement to obtain thermal infrared attention features. The visible light attention features and the thermal infrared attention features are exchanged across modalities using depthwise separable convolution to obtain visible light cross features and thermal infrared cross features. The visible light attention features are weighted element-wise using the sigmoid activation values ​​of the thermal infrared cross features to obtain visible light weighted features; The thermal infrared attention features are weighted element-wise using the sigmoid activation values ​​of the visible light cross features to obtain the thermal infrared weighted features; The visible light weighted features and the thermal infrared weighted features are gated and fused to obtain the cross-modal attention fusion features.

4. The target detection method based on adaptive modality selection as described in claim 3, characterized in that, The gating fusion of the visible light weighted features and the thermal infrared weighted features to obtain the cross-modal attention fusion features includes: The finest visible light features are sequentially input into a global average pooling layer and a 1*1 convolutional layer, and then activated by sigmoid to obtain a channel-level visible light gating vector. The finest-scale thermal infrared features are sequentially input into the global average pooling layer and the 1*1 convolutional layer, and then activated by sigmoid to obtain a channel-level thermal infrared gating vector. The first feature is obtained by calculating the product of the visible light weighted feature and the visible light gate vector; The second feature is obtained by calculating the product of the thermal infrared weighted feature and the thermal infrared gated vector; The first feature and the second feature are concatenated along the channel dimension to obtain the first concatenated feature; The first spliced ​​feature is projected through a 1*1 convolution to obtain the cross-modal attention fusion feature.

5. The target detection method based on adaptive modality selection as described in claim 1, characterized in that, The adaptive mode selection process, performed based on the cross-modal attention fusion features and the finest-scale thermal infrared features, yields the adaptive mode selection fusion features, including: The cross-modal attention fusion feature and the finest-scale thermal infrared feature are spliced ​​together along the channel dimension to obtain the second spliced ​​feature; The second concatenated feature is subjected to global average pooling to obtain a global feature vector. The global feature vector is input into the scene analysis network to obtain the global scene modality preference vector; The second stitched feature is input into the spatial weight generation network to obtain a pixel-level modality preference spatial weight map; The fusion weights are calculated based on the global scene modal preference vector and the pixel-level modal preference space weight map. The cross-modal attention fusion feature and the finest-scale thermal infrared feature are weighted and fused according to the fusion weight to obtain the adaptive modality selection fusion feature.

6. The target detection method based on adaptive modality selection as described in claim 5, characterized in that, The calculation of fusion weights based on the global scene modal preference vector and the pixel-level modal preference space weight map includes: Obtain the overall preference weights of the visible light mode and the thermal infrared mode under global illumination from the global scene modal preference vector; Obtain the visible light single-channel pixel weight and the thermal infrared single-channel pixel weight from the pixel-level modal preference space weight map; The first value is obtained by multiplying the overall preference weight of the visible light mode with the pixel weight of the visible light single channel. The second value is obtained by multiplying the overall preference weight of the thermal infrared mode by the pixel weight of the thermal infrared single channel. The third value is obtained by summing the first value, the second value, and the minimum constant. The quotient of the first value and the third value is calculated to obtain the fusion weight.

7. The target detection method based on adaptive modality selection as described in claim 5, characterized in that, After obtaining the target detection result, the method further includes: Obtain the detection confidence level and target type from the target detection results; When the detection confidence is less than the confidence threshold, or when the target type is a preset type, the visible light component is extracted from the pixel-level modality preference space weight map; Normalization is performed on the visible light component to obtain the normalized visible light component; The normalized visible light component is superimposed on the target visible light image to obtain a mode selection contribution heatmap.

8. A target detection device based on adaptive mode selection, characterized in that, The target detection device based on adaptive mode selection includes: The acquisition unit is used to acquire visible light images and thermal infrared images of the target in the target scene in response to a target detection command triggered in the target scene. The extraction unit is used to extract multiple scales of visible light features from the target visible light image using a first lightweight backbone network, and to extract multiple scales of thermal infrared features from the target thermal infrared image using a second lightweight backbone network independent of the first lightweight backbone network. The fusion unit is used to perform cross-modal attention fusion of the most detailed visible light features and thermal infrared features to obtain cross-modal attention fused features; The selection unit is used to perform adaptive mode selection based on the cross-modal attention fusion features and the finest-scale thermal infrared features to obtain adaptive mode selection fusion features; The fusion unit is also used to fuse the visible light features and thermal infrared features at the intermediate scale and the coarsest scale respectively to obtain the intermediate scale fused features and the coarsest scale fused features. The input unit is used to input the adaptive modality selection fusion feature, the intermediate scale fusion feature and the coarsest scale fusion feature into the detection head to obtain the target detection result.

9. A computer device, characterized in that, The computer device includes: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the target detection method based on adaptive modality selection as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the target detection method based on adaptive modality selection as described in any one of claims 1 to 7.

Images