Night vehicle detection method and device based on unmanned aerial vehicle, and medium

Abstract

The invention provides a night vehicle detection method and device based on an unmanned aerial vehicle, and a medium. The method comprises the following steps: performing exposure correction on a night road image to obtain an exposure correction image; extracting visible light features and infrared features; adjusting the weight according to the illumination condition and the environment feature, and fusing the visible light feature and the infrared feature according to the weight to obtain a first fusion feature; performing feature alignment on the first fusion features and calculating attention features, performing thermal radiation feature analysis, visible light feature analysis and cross-modal feature analysis according to the attention features to obtain analysis features, dynamically allocating weights for the analysis features, performing fusion to obtain second fusion features, and detecting the vehicle; optimizing the image in the low-light environment through self-adaptive exposure correction; the cross-modal feature fusion ensures the detection stability in a complex environment; for different feature analysis tasks, a proper expert decision is selected through weight distribution, so that efficient detection performance is kept under different illumination conditions.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to a method, device, and medium for nighttime vehicle detection based on drones. Background Art

[0002] Using drones combined with image processing technology for road vehicle detection is convenient. However, for nighttime detection, the nighttime environment presents problems such as low light and dynamic lighting. Traditional image detection algorithms lack generalization capabilities in complex nighttime environments and are difficult to adapt to different types of target detection. Drone target detection often experiences significant performance degradation and is unable to improve the robustness of target detection. Secondly, in inspection tasks, multimodal data such as optical and infrared have significant heterogeneity due to different imaging mechanisms. This heterogeneity is mainly manifested in spectral characteristics, resolution differences, and uneven target salience. This in turn makes image detection algorithms ineffective in feature extraction, affecting subsequent target detection results. Summary of the Invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The purpose of this application is to solve one of the technical problems existing in the related art to at least a certain extent. The embodiments of this application provide a nighttime vehicle detection method, equipment and medium based on drones, which can improve the accuracy of vehicle detection.

[0005] An embodiment of the first aspect of the present application is a method for detecting vehicles at night based on a drone, comprising:

[0006] Obtaining a nighttime road image by shooting with a drone, and inputting the nighttime road image into a detection network, wherein the nighttime road image includes a visible light image and an infrared image;

[0007] performing exposure correction on the nighttime road image to obtain an exposure-corrected image;

[0008] Performing feature extraction on the exposure-corrected image to obtain visible light features corresponding to the visible light image and infrared features corresponding to the infrared image;

[0009] adjusting a first weight corresponding to a visible light feature and a second weight corresponding to an infrared feature according to lighting conditions and environmental characteristics of the image, and fusing the visible light feature and the infrared feature according to the first weight and the second weight to obtain a first fused feature;

[0010] Performing feature alignment on the first fused features to obtain an aligned feature, calculating an attention feature based on the aligned feature, performing thermal radiation feature analysis, visible light feature analysis, and cross-modal feature analysis based on the attention feature to obtain corresponding first, second, and third analytical features, respectively, dynamically assigning weights to the first, second, and third analytical features, and fusing them according to the assigned weights to obtain a second fused feature;

[0011] The vehicle is detected based on the second fusion feature.

[0012] According to an embodiment of the first aspect of the present application, performing exposure correction on the nighttime road image to obtain an exposure-corrected image includes:

[0013] Estimating a parameter map of an image filter based on potential features of the nighttime road image, and adjusting pixel values ​​of the nighttime road image based on the parameter map to generate a parameter map;

[0014] The exposure value of the nighttime road image is adjusted using multiple groups of image filters corresponding to the parameter map in sequence to obtain an exposure-corrected image; wherein the output of the previous group of image filters is used as the input of the current group of image filters, and each group of image filters includes a brightening filter and a sharpening filter.

[0015] According to an embodiment of the first aspect of the present application, the step of estimating a parameter mapping of an image filter based on potential features of the nighttime road image includes:

[0016] Performing residual convolution, downsampling, and maximum pooling on the nighttime road image to obtain residual coding features;

[0017] extracting latent features in a latent space from the residual encoding features;

[0018] The latent features are upsampled, and a parameter map of an image filter is generated according to the upsampling result.

[0019] According to an embodiment of the first aspect of the present application, the parameter mapping module includes a residual feature encoder, a latent feature extractor and a residual feature decoder;

[0020] The residual feature encoder includes a plurality of residual convolution blocks, in which the result of the input of the residual convolution block being processed by the first residual convolution branch and the result of the input being processed by the second residual convolution branch are added in a first addition layer, and the output end of the first addition layer is connected to the maximum pooling layer;

[0021] The potential feature extractor includes a plurality of Swin transformer blocks, wherein in the Swin transformer block, a result of processing an input of the Swin transformer block by a first-layer normalization module and a standard multi-head self-attention module is added to the input of the Swin transformer block to obtain a first addition result, and a result of processing the first addition result by a second-layer normalization module and a multi-layer perception module is added to the first addition result to obtain an output of the Swin transformer block;

[0022] The residual feature decoder includes multiple up-sampled residual convolution blocks. In the up-sampled residual convolution block, the up-sampled result of the input of the up-sampled residual convolution block is spliced ​​with the output of the first addition layer of the residual feature encoder. The spliced ​​result is processed by the third residual convolution branch and the result processed by the fourth residual convolution branch. The output of the up-sampled residual convolution block is added in the second addition layer.

[0023] According to an embodiment of the first aspect of the present application, performing feature alignment on the first fused features to obtain aligned features includes:

[0024] Performing feature alignment on the first fusion features through an alignment network based on a ResNet network to obtain an alignment feature;

[0025] Among them, the first three convolutional layers of the alignment network are inserted into the residual block of the cross-modal attention adapter, and the branches corresponding to the visible light features and the branches corresponding to the infrared features share parameters; the fourth convolutional layer is inserted into the deformable convolution.

[0026] According to an embodiment of the first aspect of the present application, calculating the attention feature based on the alignment feature includes:

[0027] A cross-attention calculation is performed on the visible light feature space and the infrared feature space according to the alignment features to generate a three-dimensional weight matrix, and an attention feature is calculated according to the three-dimensional weight matrix.

[0028] According to an embodiment of the first aspect of the present application, the first analytical feature is extracted by a first expert network, which adopts a depth-separable convolutional layer and a Mish activation function, and the depth-separable convolutional layer extracts the heat source distribution; the second analytical feature is extracted by a second expert network, which adopts a ResNeXt structure and a Swish activation function, and the ResNet structure enhances the edges of low-light images; the third analytical feature adopts a Transformer encoder for cross-modal association and a GELU activation function.

[0029] According to an embodiment of the first aspect of the present application, the method further includes:

[0030] Adjusting parameters of the detection network according to the loss function;

[0031] The loss function of the detection network includes a detection loss value, a prototype comparison loss value, and a lighting inconvenience loss value;

[0032] The detection loss value is expressed as: Where, is the detection loss value, N is the number of samples in the batch, C is the total number of target categories, and y i,c is the true label of the i-th sample in category c, p i,c is the category probability predicted by the network, and γ is the adjustment factor;

[0033] The prototype contrast loss value is expressed as: Where, is the prototype contrast loss value, M is the number of feature prototypes, φ v and φ i are the modal feature mapping functions of visible light and infrared modes respectively, and are the mth feature prototypes of visible light and infrared modes respectively, is the L2 norm distance;

[0034] The light inconvenience loss value is expressed as: Where, is the illumination inconvenience loss value, x is the night road image, X is the exposure correction image, G is the generator of the exposure correction network, and D is the discriminator of the exposure correction network. Calculate the gradient of the input.

[0035] An embodiment of the second aspect of the present application is an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the drone-based nighttime vehicle detection method as described in the embodiment of the first aspect of the present application is implemented.

[0036] An embodiment of the third aspect of the present application is a computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the drone-based nighttime vehicle detection method as described in the embodiment of the first aspect of the present application.

[0037] The above scheme has at least the following beneficial effects: through adaptive exposure correction, it can effectively optimize images in low-light environments, significantly improving the visibility and detail expression of the images; through dynamic cross-attention calculation of visible light and infrared image features, it can effectively fuse the information of the two modalities and improve the recognition rate of the target; cross-modal feature fusion not only enhances the model's ability to capture details, but also significantly reduces the semantic bias caused by inter-modal heterogeneity, ensuring the detection stability in complex environments; different expert networks are constructed for different feature parsing tasks to give full play to their respective advantages; through weight distribution, the most appropriate expert can be automatically selected for decision-making according to the ambient lighting conditions, thereby maintaining efficient detection performance under different lighting conditions; the system can still maintain high detection accuracy and low false alarm rate under low illumination and strong light interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0039] Figure 1 This is a step diagram of a nighttime vehicle detection method based on a drone;

[0040] Figure 2 A diagram showing the steps of performing exposure correction on a nighttime road image to obtain an exposure-corrected image;

[0041] Figure 3 It is a diagram of the steps for obtaining parameter mapping of an image filter based on the potential features of a nighttime road image;

[0042] Figure 4 is a schematic diagram of the exposure correction network;

[0043] Figure 5 It is a schematic diagram of the parameter mapping module;

[0044] Figure 6 It is a schematic diagram of the progressive filtering module;

[0045] Figure 7 It is a schematic diagram of the cross-modal knowledge transfer network. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0047] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and the like in the specification, claims, or accompanying drawings are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0048] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0049] The embodiments of the present application provide a nighttime vehicle detection method based on a drone.

[0050] Reference Figure 1 ,The night vehicle detection method includes ,the following steps:

[0051] Step S100: obtaining a nighttime road image by shooting with a drone, and inputting the nighttime road image into a detection network;

[0052] Step S200, performing exposure correction on the nighttime road image to obtain an exposure-corrected image;

[0053] Step S300 , performing feature extraction on the exposure-corrected image to obtain visible light features corresponding to the visible light image and infrared features corresponding to the infrared image;

[0054] Step S400, adjusting the first weight corresponding to the visible light feature and the second weight corresponding to the infrared feature according to the lighting conditions and environmental characteristics of the image, and fusing the visible light feature and the infrared feature according to the first weight and the second weight to obtain a first fused feature;

[0055] Step S500: Perform feature alignment on the first fused feature to obtain an aligned feature, calculate an attention feature based on the aligned feature, perform thermal radiation feature analysis, visible light feature analysis, and cross-modal feature analysis based on the attention feature to obtain corresponding first, second, and third analytical features, dynamically assign weights to the first, second, and third analytical features, and fuse them according to the assigned weights to obtain a second fused feature.

[0056] Step S600: Detect and obtain a vehicle based on the second fusion feature.

[0057] In step S100, a drone equipped with a visible light camera and an infrared camera flies along a road, capturing images of the road at night, i.e., nighttime road images. These nighttime road images include visible light images captured by the visible light camera and infrared images captured by the infrared camera. The nighttime road images are input into a detection network, which then detects vehicles traveling on the road in the nighttime road images.

[0058] Reference Figure 4 In step S200, the exposure correction network is used to perform exposure correction on the nighttime road image to obtain an exposure-corrected image. The exposure correction network includes a parameter mapping module and multiple progressive filtering modules. Each progressive filtering module is provided with a set of image filters, including a brightening filter and a sharpening filter.

[0059] Traditional exposure correction networks, with filters using adaptive parameters, typically assume uniform lighting conditions across the entire image. However, in real-world scenarios, these images encompass large scenes often captured under a variety of lighting conditions. The lighting conditions in different regions of drone-captured images vary spatially. The exposure correction network enables exposure correction for regions with varying lighting conditions.

[0060] Reference Figure 2 , performing exposure correction on a nighttime road image to obtain an exposure-corrected image, comprising the following steps:

[0061] Step S210, estimating a parameter map of an image filter based on potential features of the nighttime road image, and adjusting each pixel value of the nighttime road image based on the parameter map to generate a parameter map;

[0062] Step S220 , sequentially using multiple sets of image filters corresponding to the parameter map to adjust the exposure value of the nighttime road image to obtain an exposure-corrected image;

[0063] Among them, the output of the previous group of image filters serves as the input of the current group of image filters.

[0064] The parameter mapping module estimates the parameter mapping of the image filter based on the latent features of the nighttime road image. This parameter map is then adjusted to generate a parameter map. This parameter map is then fed into each progressive filtering module, which then uses multiple sets of image filters corresponding to the parameter map to adjust the exposure of the nighttime road image, performing progressive filtering.

[0065] Reference Figure 3 , the parameter mapping of the image filter is estimated based on the potential features of the nighttime road image, including the following steps:

[0066] Step S211, performing residual convolution, downsampling, and maximum pooling on the nighttime road image to obtain residual coding features;

[0067] Step S212, extracting potential features in the latent space from the residual coding features through the Swin transformer layer;

[0068] Step S213 , upsampling the potential features through the upsampling residual convolution block, and generating a parameter map of the image filter according to the upsampling result through the convolution layer.

[0069] Reference Figure 5 ,The parameter mapping module consists of three sub-modules: residual feature encoder, latent feature extractor and residual feature decoder.

[0070] The residual feature encoder includes multiple residual convolution blocks. In the residual convolution block, the result of the input of the residual convolution block being processed by the first residual convolution branch and the result of the processing by the second residual convolution branch are added in the first addition layer, and the output end of the first addition layer is connected to the maximum pooling layer; the first residual convolution branch includes two convolution layers with a size of 3x3 and a stride of 1, and the second residual convolution branch includes a convolution layer with a size of 1x1 and a stride of 1.

[0071] The potential feature extractor includes multiple Swin transformer blocks. In the Swin transformer block, the result of the input of the Swin transformer block being processed by a first-layer normalization module and a standard multi-head self-attention module is added to the input of the Swin transformer block to obtain a first addition result, and the result of the first addition result being processed by a second-layer normalization module and a multi-layer perception module is added to the first addition result to obtain the output of the Swin transformer block.

[0072] The residual feature decoder includes multiple upsampling residual convolution blocks. In the upsampling residual convolution block, the upsampling result of the input of the upsampling residual convolution block is spliced ​​with the output of the first addition layer of the residual feature encoder, and the splicing result is processed by the third residual convolution branch and the result of the processing by the fourth residual convolution branch is added in the second addition layer to obtain the output of the upsampling residual convolution block; the third residual convolution branch includes two 3x3 convolution layers with a stride of 1, and the fourth residual convolution branch includes a 1x1 convolution layer with a stride of 1.

[0073] For the residual feature encoder, the input image is defined as Where W, H, Cin are the width, height and channel of the input image respectively. Two residual convolution blocks (RCB modules) are used to extract features. We have: F = ε(I); where C is the number of feature channels. One RCB module extracts features to a size of (W / 2) × (H / 2) × C, and the next RCB module extracts features to a size of (W / 4) × (H / 4) × C. The convolutional layer is beneficial for visual processing, making the transformer model in the latent feature extractor more stable and converging faster. In addition, to improve coding efficiency, the original image I is downsampled through a convolution block of size 1x1 and stride 1, the two convolution results are added, and then two 2x2 size max pooling layers (Maxpooling) are used to process the added results to obtain F.

[0074] For the latent feature extractor, after mapping the input image to the high-dimensional latent space using the residual feature encoder, S Swin transformer blocks (STBs) are used, denoted as Extract features in the latent space from F have: In particular, the intermediate features {F0,F1,…,F S-1} and the output feature F′ is extracted by a series of cascaded STBs as: Each STB consists of a layer normalization module (LayerNorm), a standard multi-head self-attention (MSA) module, another layer normalization module, a two-layer multi-layer perception (MLP) module, and residual connections applied after each module. MSA allows the model to jointly extract information from different feature subspaces, which is beneficial for dealing with uneven illumination in images. The MLP transforms the input to a higher-dimensional space and then restores it to the original dimension as input.

[0075] For the residual feature decoder, two upsampled residual convolution blocks (URCBs) are used to upsample the latent features F′ to the original input size. Then, a convolutional layer (Conv) is used to generate N parameter maps, denoted as = P = D(F′); where C′ = N×N f N is the channel number of the parameter mapping. f is the number of image filters. Specifically, the size of the latent features output by the latent feature extractor is (W / 4)×(H / 4)×C. The latent features are input to a URCB module for processing, and the output features are (W / 4)×(H / 4)×C. They are then input to another URCB module for processing, and the output results are convolved to obtain parameter mapping. In the URCB module, the input features are upsampled (Upsampling) of 2x2 and then concatenated with the output features of the RCB module; the first RCB is skipped with the second URCB, and the second RCB is skipped with the first URCB. The concatenated results are passed through two 3x3 convolutional layers with a stride of 1, and then passed through a 1x1 convolutional layer with a stride of 1. The two convolutional results are added together to obtain the output features.

[0076] The parameter map is generated by adjusting each pixel value of the nighttime road image according to the parameter mapping.

[0077] Various image filters, such as gamma correction and image sharpening, are used to create more expressive and visually appealing images. However, manually adjusting the hyperparameters of these image filters based on visual perception and experience to adapt to a wide range of scenes is time-consuming. The progressive filtering module adaptively adjusts the hyperparameters of image filters.

[0078] Reference Figure 4 and Figure 6 , a coarse image is generated using filters with the first parameter map, and then the coarse image is gradually refined using filters with other parameter maps. Specifically, given a pixel value P in I in and pixel value P in the friendly image out , from P in to P out The pixel value adjustment of is a complex nonlinear mapping. To achieve more general adjustment, we gradually apply image filters with different parameter mappings, as follows: 0≤i≤N-1; where N represents the number of iterations, is a sharpening filter; (·) γ It is the level of the Gamma function used to control image exposure. N=4 is set as a trade-off between computational cost and performance gain.

[0079] To ensure the effectiveness of gradient-based DEDet optimization, the use of PFMs must adhere to the principle of differentiability. For gradient-based DEDet optimization, the progressive filtering modules must be differentiable with respect to their filter parameters. To train neural networks using gradient backpropagation, it is necessary to ensure that each module in the neural network is differentiable with respect to its parameters. Each progressive filtering module contains two differentiable image filters: a brightening filter and a sharpening filter.

[0080] The brightening filter maps the pixel value Pn of the night image to the pixel value Pbr of the brightening image through power conversion, thereby brightening the low-light image. The brightening process of the brightening filter is described as: Where γ is the parameter of the brightening filter, n and γ are differentiable.

[0081] The sharpening filter uses Gaussian filtering to achieve better detection performance by adding details to the image. The sharpening process of the sharpening filter is described as: Where λ is a positive scaling factor, the higher λ is, the more details are added to the image. The algorithm is differentiable with respect to Pbr and λ. The degree of image sharpening is adjusted by optimizing λ to achieve better detection performance. For the Gaussian filtering process of the sharpening filter, σ is set to 5 and the kernel size is set to 7. The kernel parameters are sampled from a two-dimensional Gaussian function and remain unchanged during training.

[0082] In step S300, feature extraction is performed on the exposure-corrected image to obtain visible light features corresponding to the visible light image and infrared features corresponding to the infrared image. Specifically, a modified ResNet50 architecture is used to extract features from the visible light and infrared images. The visible light image is processed through the first few layers of the convolutional network to extract edge and texture features, while the infrared image obtains thermal radiation information through a dedicated thermal feature extraction module. Specifically, two independent feature extraction networks are used, one for visible light and one for infrared, to ensure that features from both modalities are fully captured.

[0083] In step S400, the first weights corresponding to the visible light features and the second weights corresponding to the infrared features are adjusted based on the image's lighting conditions and environmental characteristics. The visible light features and infrared features are then fused together using the first and second weights to produce a first fused feature. This step aims to effectively integrate feature information from the visible and infrared modalities to enhance the accuracy and robustness of object detection. Specifically, a lightweight cross-modal attention adapter is employed to align the extracted features. This adapter dynamically calculates the weight mapping matrix for the visible light and infrared features, ensuring that key information from both modalities is effectively preserved during the feature fusion process. In the implementation, a channel-level parameter sharing mechanism is used to combine the visible light and infrared features to generate a fused feature representation. To further enhance the effectiveness of feature fusion, a dynamic weight allocation mechanism is used to automatically adjust the first weights for the visible light features and the second weights for the infrared features based on the input image's lighting conditions and environmental characteristics. This allows the system to flexibly select the optimal feature combination under varying lighting conditions, thereby improving the accuracy and stability of object detection. The first fused feature is then fused together using the first and second weights for the visible light and infrared features.

[0084] Through the steps of multimodal feature fusion and adaptation, the limitations of traditional single-modal methods in night target detection can be effectively solved, and the detection capability and robustness of the system in complex environments can be significantly improved.

[0085] Reference Figure 7 , step S500 is implemented through a cross-modal knowledge transfer network.

[0086] Performing feature alignment on the first fused features to obtain aligned features, comprising the following steps: performing feature alignment on the first fused features through an alignment network based on a ResNet network to obtain aligned features; wherein, the first three convolutional layers of the alignment network are inserted into the residual blocks of the cross-modal attention adapter, and parameters are shared between the branches corresponding to the visible light features and the branches corresponding to the infrared features; and the fourth convolutional layer is inserted into the deformable convolution.

[0087] Furthermore, the first three layers (conv1-conv3) are cross-modally adapted by inserting a lightweight cross-modal attention adapter with only 0.8% of the parameters of the original layer. Through the channel-level parameter sharing mechanism, the common edge features are retained in the conv1 stage, thereby enhancing the model's ability to capture details. The channel-level parameter sharing mechanism is described as: shared =α·W vis +(1-α)·W ir , α∈[0,1]; where α is a learnable modal balance factor, W vis and W ir represent the visible light and infrared branch parameters respectively.

[0088] In addition, the residual path is improved. A cross-modal attention adapter is added to achieve feature adaptation. The residual block structure is described as: in, is the original convolution operation, It is the adapter gating function, reflecting the incremental improvement of the feature adaptation path.

[0089] In addition, deformable convolution is introduced in the fourth convolutional layer (conv4) to dynamically adjust the receptive field, thereby improving the adaptability to night-time target deformation.

[0090] The shallow adapter preserves high-frequency details in the visible light modality, improving the gradient histogram contrast by 18.7%. Simultaneously, the thermal radiation signature of the infrared modality is enhanced, improving the signal-to-noise ratio by 23.4%. Test results on the FLIR dataset show that the bidirectional feature similarity reaches 0.812, and the feature dimension compression rate during the migration process is kept within 12%, ensuring feature consistency and effectiveness.

[0091] The attention feature is calculated based on the aligned features, including the following steps: performing cross-attention calculation on the visible light feature space and the infrared feature space based on the aligned features, generating a three-dimensional weight matrix, and calculating the attention feature based on the three-dimensional weight matrix. Furthermore, in terms of the bidirectional cross-attention mechanism, a dynamic weight calculation architecture is used. By generating a three-dimensional weight matrix, cross-attention calculation can be effectively performed on the visible light feature space and the infrared feature space. In terms of practical application effects, the test results on the FLIR-VID dataset show that the edge clarity of pedestrian targets is improved by 41%, the vehicle thermal feature artifacts are reduced by 67%, and the cross-modal feature alignment error is controlled within 3.2px, which significantly improves the accuracy of target detection.

[0092] Based on the attention features, thermal radiation feature analysis, visible light feature analysis, and cross-modal feature analysis are performed to obtain the corresponding first, second, and third analytical features, respectively. Weights are dynamically assigned to the first, second, and third analytical features, and the second fused features are fused based on the assigned weights. The first analytical feature is extracted by the first expert network, which uses a depthwise separable convolutional layer and Mish activation function to extract the heat source distribution. The second analytical feature is extracted by the second expert network, which uses a ResNet structure and Swish activation function to enhance the edges of low-light images. The third analytical feature uses a Transformer encoder for cross-modal association and a GELU activation function.

[0093] Specifically, by applying the MOE expert network and specialized application of the expert network, we constructed infrared experts (corresponding to the first expert network, Expert1), visible light experts (corresponding to the second expert network, Expert2), and fusion experts (corresponding to the third expert network, Expert3) for different feature analysis tasks. The infrared expert focuses on thermal radiation feature analysis, using a 3-layer DSConv structure and the Mish activation function (using deep separable convolution to extract heat source distribution); the visible light expert focuses on texture detail enhancement, using a 5-layer ResNeXt structure and the Swish activation function (using the ResNeXt structure to enhance low-light image edges); and the fusion expert utilizes a Transformer encoder for cross-modal association, with the GELU activation function.

[0094] Dynamic weight allocation during feature fusion is achieved through a dynamic gating network. In the dynamic gating network, dynamic selection of expert network weights is achieved through weight allocation, and weights of each expert network are dynamically allocated (for example, focusing on infrared experts in low illumination and balancing weights in mixed illumination). The actual operation results show that under low illumination conditions (<10lux), the weight of the visible light expert automatically drops below 0.2, while under strong heat source interference, the weight of the infrared expert increases to 0.65±0.12, and the decision delay is controlled within 2.3ms, ensuring real-time and accuracy. Finally, the outputs of each expert are integrated through a weighted voting mechanism. This design enables the system to adaptively adjust the modal fusion strategy, improve the accuracy of vehicle detection under complex illumination at night, and reduce the false detection rate.

[0095] During the model training process, the loss function is obtained based on the output of the network, and then the parameters of the detection network are adjusted according to the loss function until the model converges or the preset number of training times is reached;

[0096] Among them, the loss function of the detection network includes detection loss value, prototype comparison loss value and illumination inconvenience loss value;

[0097] The detection loss value is expressed as: Where, is the detection loss value, N is the number of samples in the batch, C is the total number of target categories, and y i,c is the true label of the i-th sample in category c, p i,c is the category probability predicted by the network, and γ is the adjustment factor;

[0098] The prototype contrast loss value is expressed as: Where, is the prototype contrast loss value, M is the number of feature prototypes, φ v and φ i are the modal feature mapping functions of visible light and infrared modes respectively, and are the mth feature prototypes of visible light and infrared modes respectively, is the L2 norm distance;

[0099] The light inconvenience loss value is expressed as: Where, is the illumination inconvenience loss value, x is the night road image, X is the exposure correction image, G is the generator of the exposure correction network, and D is the discriminator of the exposure correction network. Calculate the gradient of the input.

[0100] In general, the images collected by the drone are exposed and then fed into the first three layers of ResNet to embed a lightweight cross-modal attention adapter. This module dynamically calculates the weight mapping matrix of the infrared and visible light feature maps through a cross-attention mechanism, and uses cycle consistency loss to ensure distortion-free bidirectional migration. Secondly, a cross-modal semantic prototype vector is constructed for each target category, and the distance between prototypes of the same category and different modalities is minimized through prototype contrast loss. At the same time, an uncertainty calibration module is introduced to eliminate prototype offsets caused by lighting changes. Finally, a multi-expert hybrid model is used, with each expert network focusing on semantic parsing of a specific modality. The gated network dynamically assigns weights based on the input modality characteristics. Finally, the detection results are output through weighted voting, and staff make target judgments based on the detection results for further evaluation and processing.

[0101] First, the adaptive exposure correction network effectively optimizes images in low-light environments, significantly improving image visibility and detail. This approach not only avoids the global overexposure or underexposure issues that can occur with traditional exposure adjustment, but also ensures a gradual improvement in image quality through a step-by-step iterative process, thereby enhancing object detection accuracy. Second, the introduction of a hierarchical transfer mechanism makes the knowledge transfer from low-level local features to high-level global semantics more efficient. By dynamically calculating cross-attention on visible and infrared image features, the information from both modalities is effectively integrated, improving object recognition. This cross-modal feature fusion not only enhances the model's ability to capture detail but also significantly reduces semantic bias caused by inter-modal heterogeneity, ensuring detection stability in complex environments. Furthermore, the use of a multi-expert mixture model constructs infrared experts, visible light experts, and fusion experts for different feature parsing tasks, leveraging their respective strengths. Through weight allocation within a dynamic gating network, the most appropriate expert is automatically selected for decision-making based on ambient lighting conditions, maintaining efficient detection performance under varying lighting conditions. This flexibility enables the system to maintain high detection accuracy and low false alarm rates even in low illumination and strong light interference conditions. This effectively improves detection accuracy and reduces false alarm rates. This not only enhances drones' target detection capabilities in complex nighttime environments, but also provides more reliable technical support for applications in related fields.

[0102] An embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described drone-based nighttime vehicle detection method.

[0103] The electronic device may be any intelligent terminal including a computer.

[0104] In general, for the hardware structure of the electronic device, the processor can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0105] The memory can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory and is called by the processor to execute the methods of the embodiments of this application.

[0106] The input / output interface is used to realize information input and output.

[0107] The communication interface is used to realize the communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0108] The bus transmits information between the various components of the device (such as the processor, memory, input / output interface, and communication interface). The processor, memory, input / output interface, and communication interface communicate with each other within the device through the bus.

[0109] An embodiment of the present application provides a computer storage medium storing computer-executable instructions for executing the above-described drone-based nighttime vehicle detection method.

[0110] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium. In the above description of this specification, the reference terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" and the like are intended to mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0111] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0112] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0114] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0115] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. Although the embodiments of the present application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and purpose of the present application, and the scope of the present application is defined by the claims and their equivalents.

[0116] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present application, and these equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A nighttime vehicle detection method based on drones, characterized in that: include: Obtaining a nighttime road image by shooting with a drone, and inputting the nighttime road image into a detection network, wherein the nighttime road image includes a visible light image and an infrared image; performing exposure correction on the nighttime road image to obtain an exposure-corrected image; Performing feature extraction on the exposure-corrected image to obtain visible light features corresponding to the visible light image and infrared features corresponding to the infrared image; adjusting a first weight corresponding to a visible light feature and a second weight corresponding to an infrared feature according to lighting conditions and environmental characteristics of the image, and fusing the visible light feature and the infrared feature according to the first weight and the second weight to obtain a first fused feature; Performing feature alignment on the first fused features to obtain an aligned feature, calculating an attention feature based on the aligned feature, performing thermal radiation feature analysis, visible light feature analysis, and cross-modal feature analysis based on the attention feature to obtain corresponding first, second, and third analytical features, respectively, dynamically assigning weights to the first, second, and third analytical features, and fusing them according to the assigned weights to obtain a second fused feature; The vehicle is detected based on the second fusion feature.

2. The nighttime vehicle detection method based on drone according to claim 1, characterized in that: The step of performing exposure correction on the nighttime road image to obtain an exposure-corrected image includes: Estimating, by a parameter mapping module, a parameter mapping of an image filter based on potential features of the nighttime road image, and adjusting each pixel value of the nighttime road image based on the parameter mapping to generate a parameter map; Through a progressive filtering module, multiple groups of image filters corresponding to the parameter map are sequentially used to adjust the exposure value of the nighttime road image to obtain an exposure-corrected image; wherein the output of the previous group of image filters serves as the input of the current group of image filters, and each group of image filters includes a brightening filter and a sharpening filter.

3. The nighttime vehicle detection method based on drone according to claim 2, characterized in that: The step of estimating a parameter mapping of an image filter based on potential features of the nighttime road image comprises: Performing residual convolution, downsampling, and maximum pooling on the nighttime road image to obtain residual coding features; extracting latent features in a latent space from the residual encoding features; The potential features are upsampled, and convolution is performed based on the upsampling results to generate a parameter map of the image filter.

4. The nighttime vehicle detection method based on drone according to claim 2, characterized in that: The parameter mapping module includes a residual feature encoder, a latent feature extractor and a residual feature decoder; The residual feature encoder includes a plurality of residual convolution blocks, in which the result of the input of the residual convolution block being processed by the first residual convolution branch and the result of the input being processed by the second residual convolution branch are added in a first addition layer, and the output end of the first addition layer is connected to the maximum pooling layer; The potential feature extractor includes a plurality of Swin transformer blocks, wherein in the Swin transformer block, a result of processing an input of the Swin transformer block by a first-layer normalization module and a standard multi-head self-attention module is added to the input of the Swin transformer block to obtain a first addition result, and a result of processing the first addition result by a second-layer normalization module and a multi-layer perception module is added to the first addition result to obtain an output of the Swin transformer block; The residual feature decoder includes multiple up-sampled residual convolution blocks. In the up-sampled residual convolution block, the up-sampled result of the input of the up-sampled residual convolution block is spliced ​​with the output of the first addition layer of the residual feature encoder. The spliced ​​result is processed by the third residual convolution branch and the result processed by the fourth residual convolution branch. The output of the up-sampled residual convolution block is added in the second addition layer.

5. The nighttime vehicle detection method based on drone according to claim 1, characterized in that: The performing feature alignment on the first fused features to obtain aligned features includes: Performing feature alignment on the first fusion features through an alignment network based on a ResNet network to obtain an alignment feature; Among them, the first three convolutional layers of the alignment network are inserted into the residual block of the cross-modal attention adapter, and the branches corresponding to the visible light features and the branches corresponding to the infrared features share parameters; the fourth convolutional layer is inserted into the deformable convolution.

6. The nighttime vehicle detection method based on drone according to claim 1, characterized in that: The calculating the attention feature according to the alignment feature includes: A cross-attention calculation is performed on the visible light feature space and the infrared feature space according to the alignment features to generate a three-dimensional weight matrix, and an attention feature is calculated according to the three-dimensional weight matrix.

7. The nighttime vehicle detection method based on drone according to claim 1, characterized in that: The first analytical feature is extracted by a first expert network, which uses a depth-wise separable convolutional layer and a Mish activation function to extract heat source distribution; the second analytical feature is extracted by a second expert network, which uses a ResNeXt structure and a Swish activation function to enhance the edges of low-light images; the third analytical feature uses a Transformer encoder for cross-modal association and a GELU activation function.

8. The method for nighttime vehicle detection based on drone according to claim 1, characterized in that: The method further comprises: Adjusting parameters of the detection network according to the loss function; The loss function of the detection network includes a detection loss value, a prototype comparison loss value, and a lighting inconvenience loss value; The detection loss value is expressed as: Where, is the detection loss value, N is the number of samples in the batch, C is the total number of target categories, and y i,c is the true label of the i-th sample in category c, p i,c is the category probability predicted by the network, and γ is the adjustment factor; The prototype contrast loss value is expressed as: Where, is the prototype contrast loss value, M is the number of feature prototypes, φ v and φ i are the modal feature mapping functions of visible light and infrared modes respectively, and are the mth feature prototypes of visible light and infrared modes respectively, is the L2 norm distance; The light inconvenience loss value is expressed as: Where, is the illumination inconvenience loss value, x is the night road image, X is the exposure correction image, G is the generator of the exposure correction network, and D is the discriminator of the exposure correction network. Calculate the gradient of the input.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the drone-based nighttime vehicle detection method according to any one of claims 1 to 8 is implemented.

10. A computer storage medium, characterized in that Computer-executable instructions are stored, and the computer-executable instructions are used to execute the drone-based nighttime vehicle detection method according to any one of claims 1 to 8.

Images