Unmanned aerial vehicle based nighttime vehicle detection method, device and medium

Abstract

The application provides a UAV-based night vehicle detection method, device and medium, wherein the method comprises: performing exposure correction on a night road image to obtain an exposure-corrected image; extracting visible light features and infrared features; adjusting weights according to illumination conditions and environmental features, fusing the visible light features and the infrared features according to the weights to obtain first fused features; performing feature alignment on the first fused features and calculating attention features, respectively performing thermal radiation feature analysis, visible light feature analysis and cross-modal feature analysis according to the attention features to obtain analysis features, dynamically assigning weights to the analysis features, fusing to obtain second fused features and detecting vehicles; optimizing images in a low-light environment through adaptive exposure correction; cross-modal feature fusion ensures detection stability in complex environments; for different feature analysis tasks, appropriate expert decisions are selected through weight distribution, so that high-efficiency detection performance is maintained under different illumination conditions.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of image processing, and in particular to a night vehicle detection method based on a UAV, a device and a medium. BACKGROUND

[0002] Detecting vehicles on the road through a UAV combined with image processing technology is convenient. However, for night detection, there are problems such as low light and dynamic light in the night environment, and the generalization ability of traditional image detection algorithms is insufficient in complex night environments, making it difficult to adapt to different types of target detection. The performance of UAV target detection often decreases significantly, and the robustness of target detection cannot be improved. Secondly, in the inspection task, optical and infrared multi-modal data have significant heterogeneity due to different imaging mechanisms, which mainly manifests as spectral characteristics, resolution differences, and uneven target saliency, etc., thereby affecting the feature extraction of the image detection algorithm and the subsequent target detection result. SUMMARY

[0003] The following is a summary of the subject matter of the detailed description herein. This summary is not intended to limit the scope of the claims.

[0004] The present application aims to at least partially solve one of the technical problems in the related art, and provides a night vehicle detection method based on a UAV, a device and a medium, which can improve the vehicle detection accuracy.

[0005] Embodiments of the first aspect of the present application provide a night vehicle detection method based on a UAV, comprising:

[0006] Obtaining a night road image by photographing with a UAV, inputting the night road image into a detection network, wherein the night road image comprises a visible light image and an infrared image;

[0007] Performing exposure correction on the night road image to obtain an exposure corrected image;

[0008] Performing feature extraction on the exposure corrected image to extract visible light features corresponding to the visible light image and infrared features corresponding to the infrared image;

[0009] Adjusting a first weight corresponding to the visible light features and a second weight of the infrared features according to the light conditions and environmental features of the image, and fusing the visible light features and the infrared features according to the first weight and the second weight to obtain first fused features;

[0010] perform feature alignment on the first fusion feature to obtain an aligned feature, calculate an attention feature according to the aligned feature, and respectively perform thermal radiation feature analysis, visible light feature analysis and cross-modal feature analysis according to the attention feature to obtain corresponding first analysis feature, second analysis feature and third analysis feature respectively, dynamically assign weights to the first analysis feature, the second analysis feature and the third analysis feature, and fuse to obtain a second fusion feature according to the assigned weights;

[0011] detect a vehicle according to the second fusion feature.

[0012] According to the first aspect of the present application, the exposure correction of the night road image to obtain an exposure correction image includes:

[0013] According to the latent feature of the night road image, the parameter mapping of the image filter is estimated, and the parameter map of each pixel value of the night road image is generated according to the parameter mapping;

[0014] The exposure value of the night road image is adjusted in turn using a plurality of groups of image filters corresponding to the parameter map to obtain an exposure correction image; wherein the output of the last group of image filters is used as the input of the current group of image filters, and each group of image filters includes a brightening filter and a sharpening filter.

[0015] According to the first aspect of the present application, the parameter mapping of the image filter is estimated according to the latent feature of the night road image, including:

[0016] Residual convolution, down-sampling and maximum pooling are performed on the night road image to obtain residual coding features;

[0017] The latent feature in the latent space is extracted from the residual coding feature;

[0018] The latent feature is up-sampled, and the parameter mapping of the image filter is generated according to the up-sampling result.

[0019] According to the first aspect of the present application, the parameter mapping module includes a residual feature encoder, a latent feature extractor and a residual feature decoder;

[0020] The residual feature encoder includes a plurality of residual convolution blocks, in which the results of the first residual convolution branch processing and the second residual convolution branch processing of the residual convolution block input are added in the first addition layer, and the output end of the first addition layer is connected with the maximum pooling layer;

[0021] The potential feature extractor includes a plurality of Swin transformer blocks, in which the first layer normalization module and the standard multi-head self-attention module are used to process the input of the Swin transformer block, and the result is added to the input of the Swin transformer block to obtain a first addition result, the first addition result is processed by the second layer normalization module and the multi-layer perception module, and the result is added to the first addition result to obtain the output of the Swin transformer block.

[0022] The residual feature decoder includes a plurality of up-sampling residual convolution blocks, in which the input of the up-sampling residual convolution block is spliced with the output of the first addition layer of the residual feature encoder after up-sampling, the spliced result is processed by the third residual convolution branch and the fourth residual convolution branch, and the result is added in the second addition layer to obtain the output of the up-sampling residual convolution block.

[0023] According to the first aspect of the present application, the feature alignment of the first fusion feature includes:

[0024] The first fusion feature is aligned by an alignment network based on a ResNet network to obtain an aligned feature;

[0025] The first three convolution layers of the alignment network are inserted into the residual block of the cross-modal attention adapter, and the parameters of the branch corresponding to the visible light feature and the branch corresponding to the infrared feature are shared; the fourth convolution layer is inserted into the deformable convolution.

[0026] According to the first aspect of the present application, the attention feature is calculated according to the aligned feature, including:

[0027] According to the aligned feature, cross-attention calculation is performed on the visible light feature space and the infrared feature space to generate a three-dimensional weight matrix, and the attention feature is calculated according to the three-dimensional weight matrix.

[0028] According to the first aspect of the present application, the first analysis feature is extracted by a first expert network, the first expert network uses a depth separable convolution layer and a Mish activation function, and the depth separable convolution layer extracts a heat source distribution; the second analysis feature is extracted by a second expert network, the second expert network uses a ResNeXt structure and a Swish activation function, and the ResNet structure enhances the edge of the low-light image; the third analysis feature uses a Transformer encoder for cross-modal correlation and a GELU activation function.

[0029] According to the first aspect of the present application, the method further includes:

[0030] The parameters of the detection network are adjusted according to the loss function.

[0031] The loss function of the detection network includes detection loss value, prototype comparison loss value, and illumination inconvenience loss value.

[0032] The detection loss value is expressed as: In the formula, To detect the loss value, N is the number of samples in the batch, C is the total number of target categories, and y i,c Let p be the true label of the i-th sample in class c. i,c γ represents the class probability predicted by the network, and γ is a modulating factor.

[0033] The prototype contrast loss value is expressed as: In the formula, Here, φ represents the prototype contrast loss value, M is the number of feature prototypes, and φ is the prototype contrast loss value. v and φ i These are the modal characteristic mapping functions for the visible light and infrared modes, respectively. and These are the m-th feature prototypes of the visible light and infrared modes, respectively. The distance is the L2 norm.

[0034] The light inconvenience loss value is expressed as: In the formula, Let be the illumination inconvenience loss value, x be the nighttime road image, X be the exposure-corrected image, G be the generator of the exposure correction network, and D be the discriminator of the exposure correction network. This is for calculating the gradient of the input.

[0035] According to a second aspect of this application, an electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the UAV-based nighttime vehicle detection method as described in the first aspect of this application.

[0036] According to a third aspect of this application, a computer storage medium stores computer-executable instructions for performing a drone-based nighttime vehicle detection method as described in an embodiment of the first aspect of this application.

[0037] The above scheme has at least the following beneficial effects: through adaptive exposure correction, the image in a low-light environment can be effectively optimized, and the visibility and detail performance of the image are significantly improved; through dynamic cross-attention calculation of visible light and infrared image features, the information of the two modalities can be effectively fused, and the target recognition rate is improved; cross-modal feature fusion not only enhances the model's ability to capture details, but also significantly reduces semantic bias caused by inter-modal heterogeneity, ensuring detection stability in complex environments; different expert networks are constructed for different feature analysis tasks, and their respective advantages are fully utilized; through weight distribution, the most suitable expert can be automatically selected for decision-making according to the environmental light conditions, thereby maintaining efficient detection performance under different light conditions; the system can still maintain high detection accuracy and low false alarm rate in low light and strong light interference. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings are used to provide a further understanding of the technical scheme of the present application, and constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical scheme of the present application, and do not constitute a limitation on the technical scheme of the present application.

[0039] Figure 1 is a step diagram of a night vehicle detection method based on a UAV;

[0040] Figure 2 is a step diagram of exposure correction of a night road image to obtain an exposure correction image;

[0041] Figure 3 is a step diagram of estimating a parameter mapping of an image filter according to the latent features of the night road image;

[0042] Figure 4 is a schematic diagram of an exposure correction network;

[0043] Figure 5 is a schematic diagram of a parameter mapping module;

[0044] Figure 6 is a schematic diagram of a progressive filtering module;

[0045] Figure 7 is a schematic diagram of a cross-modal knowledge transfer network. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the present application.

[0047] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the description, claims or above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0048] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0049] The embodiments of the present application provide a night vehicle detection method based on a UAV.

[0050] With reference to Figure 1 , the night vehicle detection method comprises the following steps:

[0051] Step S100, obtaining a night road image by photographing through a UAV, and inputting the night road image into a detection network;

[0052] Step S200, performing exposure correction on the night road image to obtain an exposure corrected image;

[0053] Step S300, performing feature extraction on the exposure corrected image to extract visible light features corresponding to a visible light image and infrared features corresponding to an infrared image;

[0054] Step S400, adjusting a first weight corresponding to the visible light features and a second weight of the infrared features according to the illumination condition and environmental features of the image, and fusing the visible light features and the infrared features according to the first weight and the second weight to obtain first fused features;

[0055] Step S500, performing feature alignment on the first fused features to obtain aligned features, calculating attention features according to the aligned features, and respectively performing thermal radiation feature analysis, visible light feature analysis and cross-modal feature analysis according to the attention features to obtain corresponding first analysis features, second analysis features and third analysis features, respectively, dynamically assigning weights to the first analysis features, the second analysis features and the third analysis features, and fusing to obtain second fused features according to the assigned weights;

[0056] Step S600, detecting vehicles according to the second fused features.

[0057] For step S100, the UAV is equipped with a visible light camera and an infrared camera, and flies along the road to obtain an image of the road at night, i.e. a night road image. The night road image includes a visible light image taken by the visible light camera and an infrared image taken by the infrared camera. The night road image is input into the detection network, and the detection network is used to detect the night road image to detect vehicles walking on the road in the night road image.

[0058] Referring to Figure 4 For step S200, the night road image is exposed to the exposure correction network to obtain the exposure correction image. The exposure correction network includes a parameter mapping module and a plurality of progressive filtering modules. Each progressive filtering module is provided with a set of image filters, including a brightening filter and a sharpening filter.

[0059] The filter with adaptive parameters of the conventional exposure correction network usually assumes a uniform illumination condition for the entire image. However, in real scenes, due to the large scene contained in these images, which are usually taken under various lighting conditions, the lighting conditions of different regions of the image captured by the unmanned aerial vehicle differ in space. The exposure correction network can perform exposure correction on regions with different illumination conditions.

[0060] Referring to Figure 2 The exposure correction of the night road image obtains the exposure correction image, including the following steps:

[0061] Step S210, the parameter mapping of the image filter is estimated according to the latent feature of the night road image, and the parameter map is generated by adjusting the pixel value of the night road image according to the parameter mapping;

[0062] Step S220, the exposure value of the night road image is adjusted by using a plurality of sets of image filters corresponding to the parameter map in sequence to obtain the exposure correction image;

[0063] Wherein, the output of the previous set of image filters is taken as the input of the current set of image filters.

[0064] The parameter mapping module estimates the parameter mapping of the image filter according to the latent feature of the night road image, and generates the parameter map by adjusting the pixel value of the night road image according to the parameter mapping. The parameter map is input into each progressive filtering module, and a plurality of sets of image filters corresponding to the parameter map are used in sequence to adjust the exposure value of the night road image, and progressive filtering is performed.

[0065] Referring to Figure 3 The parameter mapping of the image filter is estimated according to the latent feature of the night road image, including the following steps:

[0066] Step S211, residual convolution, downsampling and maximum pooling are performed on the night road image to obtain residual coding features;

[0067] Step S212, latent features in the latent space are extracted from the residual coding features by the Swin transformer layer;

[0068] Step S213, the latent features are upsampled by the upsample residual convolution block, and the parameter mapping of the image filter is generated according to the upsampled results by the convolution layer.

[0069] Referring to Figure 5 The parameter mapping module includes three sub-modules: a residual feature encoder, a latent feature extractor, and a residual feature decoder.

[0070] The residual feature encoder includes a plurality of residual convolution blocks, in which the input of a residual convolution block is added to the output of a first residual convolution branch and the output of a second residual convolution branch in a first addition layer, and the output end of the first addition layer is connected with a max-pooling layer; the first residual convolution branch includes two convolution layers with a size of 3x3 and a stride of 1, and the second residual convolution branch includes a convolution layer with a size of 1x1 and a stride of 1.

[0071] The latent feature extractor includes a plurality of Swin transformer blocks, in which the input of a Swin transformer block is added to the output of a first layer normalization module and a standard multi-head self-attention module to obtain a first addition result, and the output of the first layer normalization module and a multi-layer perception module is added to the first addition result to obtain the output of the Swin transformer block.

[0072] The residual feature decoder includes a plurality of up-sampling residual convolution blocks, in which the input of an up-sampling residual convolution block is spliced with the output of the first addition layer of the residual feature encoder, and the spliced result is added to the output of a third residual convolution branch and the output of a fourth residual convolution branch in a second addition layer to obtain the output of the up-sampling residual convolution block; the third residual convolution branch includes two convolution layers with a size of 3x3 and a stride of 1, and the fourth residual convolution branch includes a convolution layer with a size of 1x1 and a stride of 1.

[0073] For the residual feature encoder, the input image is defined as where W, H, and Cin are the width, height, and channel of the input image, respectively. Two residual convolution blocks (RCB modules) are used to extract features There are: F = ε(I); where C is the number of feature channels. One RCB module extracts features to (W / 2) x (H / 2) x C size, and the next RCB module extracts features to (W / 4) x (H / 4) x C size. The convolution layer is beneficial to visual processing, making the transformer model in the latent feature extractor more stable and faster to converge. In addition, in order to improve the coding efficiency, the original image I is down-sampled by a 1x1 size, stride 1 convolution block, the two results after convolution are added, and then two 2x2 size max-pooling layers (Maxpooling) are used to process the added results to obtain F.

[0074] For the latent feature extractor, after the input image is mapped to a high-dimensional latent space with the residual feature encoder, S Swin Transformer Blocks (STB) are used, denoted as Features in the latent space are extracted from F There are: In particular, the intermediate features {F0, F1, …, F S-1} and the output feature F' are extracted by a series of cascaded STBs as: Each STB consists of a Layer Normalization module (LayerNorm), a standard multi-head self-attention (MSA) module, another Layer Normalization module, a double-layer multi-layer perception (MLP) module, and a residual connection applied after each module. MSA allows the model to jointly extract information in different feature subspaces, which is beneficial for handling the problem of uneven illumination in images. The MLP converts the input to a higher-dimensional space and then restores it to the original dimension as input.

[0075] For the residual feature decoder, two up-sampling residual convolution blocks (URCB) are used to up-sample the latent feature F' to the original input size. Then, a convolution layer (Conv) is used to generate N parameter maps, denoted as P = D(F'); where C' = N x N f is the channel number of the parameter map, N f is the number of image filters. Specifically, the latent feature output by the latent feature extractor has a size of (W / 4) x (H / 4) x C, the latent feature is input to a URCB module for processing, and an output feature with a size of (W / 4) x (H / 4) x C is output, then input to another URCB module for processing, and the output result is obtained by convolution. In the URCB module, the input feature is up-sampled by 2x2 size and then spliced with the output feature of the RCB module; where the first RCB is skip-connected with the second URCB, and the second RCB is skip-connected with the first URCB. The spliced result is passed through two convolution layers with a size of 3x3 and a stride of 1, and the spliced result is passed through a convolution layer with a size of 1x1 and a stride of 1, and the two convolution results are added to obtain the output feature.

[0076] A parameter map is generated by adjusting the pixel values of each pixel of the night road image according to the parameter map.

[0077] Various image filters, such as gamma correction and image sharpening, are used to create more expressive and visually appealing images. However, based on visual perception and experience, it takes a long time to manually adjust the hyperparameters of these image filters to adapt to a wide range of scenes. The hyperparameters of the image filters are adjusted adaptively by the progressive filtering module.

[0078] Referring to Figure 4 and Figure 6 , a coarse image is generated using a filter with the first parameter map, and then the coarse image is gradually refined using filters with other parameter maps. Specifically, given a pixel value P in and a pixel value P out in the friendly image, the pixel value adjustment from P in to P out is a complex non-linear mapping. To achieve a more general adjustment, image filters with different parameter mappings are applied step by step, with: 0≤i≤N-1; where N represents the number of iterations, is a sharpening filter; (·) γ is a Gamma function level for controlling image exposure, and N=4 is set as a trade-off considering the computational cost and performance gain.

[0079] To ensure the effectiveness of the gradient-based DEDet optimization, the use of PFM needs to adhere to the principle of differentiability. For gradient-based DEDet optimization, the progressive filtering module needs to be differentiable in terms of filter parameters. In order to train the neural network with gradient backpropagation, it is necessary to ensure that each module in the neural network is differentiable in terms of its parameters. Each progressive filtering module contains two differentiable image filters, namely the brightening filter and the sharpening filter.

[0080] The brightening filter maps the pixel value Pn of the night image to the pixel value Pbr of the brightened image through power transformation, so as to brighten the low-light image. The brightening process of the brightening filter is described as: where γ is the parameter of the brightening filter, and both P n and γ are differentiable.

[0081] The sharpening filter uses Gaussian filtering to add details to the image to obtain better detection performance. The sharpening process of the sharpening filter is described as: where λ is a positive proportion factor, and the higher λ is, the more details are added to the image. Moreover, Pbr and λ are differentiable. The degree of image sharpening is adjusted by optimizing λ to obtain better detection performance. For the Gaussian filtering process of the sharpening filter, its σ is set to 5 and the kernel size is set to 7. The parameters of the kernel are sampled from a two-dimensional Gaussian function and remain unchanged during the training process.

[0082] For step S300, feature extraction is performed on the exposure-corrected image, and visible light features corresponding to the visible light image and infrared features corresponding to the infrared image are extracted. Specifically, the improved ResNet50 architecture is used to extract features from the visible light and infrared images respectively. The visible light image extracts edge and texture features through the first few convolutional networks, while the infrared image obtains thermal radiation information through a specially used thermal feature extraction module. Specifically, two independent feature extraction networks are used, namely a visible light network and an infrared network, to ensure that the features of the two modalities can be fully captured.

[0083] For step S400, the first weight corresponding to the visible light feature and the second weight of the infrared feature are adjusted according to the illumination condition and environmental characteristics of the image, and the first fused feature is obtained by fusing the visible light feature and the infrared feature according to the first weight and the second weight. This step aims to effectively integrate feature information from visible light and infrared modalities to enhance the accuracy and robustness of target detection. Specifically, a lightweight cross-modal attention adapter is used to align the extracted features. This adapter dynamically calculates the weight mapping matrix of the visible light and infrared features to ensure that key information from both modalities can be effectively retained during feature fusion. In a specific implementation, the channel-level parameter sharing mechanism is used to combine the visible light and infrared features to generate a fused feature representation. To further improve the effect of feature fusion, a dynamic weight distribution mechanism is used to automatically adjust the first weight of the visible light feature and the second weight of the infrared feature according to the illumination condition and environmental characteristics of the input image. In this way, the system can flexibly select the optimal feature combination under different lighting conditions, thereby improving the accuracy and stability of target detection. The first fused feature is obtained by fusing the visible light feature and the infrared feature according to the first weight and the second weight.

[0084] Through the steps of multi-modal feature fusion and adaptation, the limitations of traditional single-modal methods in night target detection are effectively solved, and the detection ability and robustness of the system in complex environments are significantly improved.

[0085] Reference Figure 7 For step S500, cross-modal knowledge transfer network is implemented.

[0086] The first fused feature is aligned to obtain an aligned feature, including the following steps: the first fused feature is aligned by an alignment network based on a ResNet network to obtain an aligned feature; wherein the first three convolutional layers of the alignment network are inserted into the residual blocks of the cross-modal attention adapter, and the parameters of the branch corresponding to the visible light feature and the branch corresponding to the infrared feature are shared; the fourth convolutional layer is inserted into deformable convolution.

[0087] Further, the first 3 layers (conv1-conv3) are cross-modality adapted, with a lightweight cross-modality attention adapter inserted, which has a parameter quantity of only 0.8% of the original layer. Through a channel-level parameter sharing mechanism, the general edge features are retained at the conv1 stage, thereby enhancing the model's ability to capture details. The channel-level parameter sharing mechanism is described as: W shared = a W vis + (1-a) W ir , a e [0, 1]; where a is a learnable modality balance factor, W vis and W ir represent the visible light and infrared branch parameters, respectively.

[0088] In addition, the residual path is improved. A cross-modality attention adapter is added to achieve feature adaptation. The residual block structure is described as: wherein, is the original convolution operation, is the adapter gating function, which embodies the incremental improvement of the feature adaptation path.

[0089] In addition, a deformable convolution is introduced in the fourth layer convolution layer (conv4) to dynamically adjust the receptive field, thereby improving the adaptability to night target deformation.

[0090] Through the shallow adapter, the high-frequency detail retention of the visible light modality is achieved, and the gradient direction histogram contrast is improved by 18.7%; at the same time, the thermal radiation features of the infrared modality are strengthened, and the signal-to-noise ratio is improved by 23.4%. The test results on the FLIR dataset show that the bidirectional feature similarity reaches 0.812, and the feature dimension compression rate during the migration process is controlled within 12%, ensuring the consistency and effectiveness of the features.

[0091] The attention features are calculated based on the aligned features, including the following steps: cross-attention calculation is performed on the visible light feature space and the infrared feature space based on the aligned features, a three-dimensional weight matrix is generated, and the attention features are calculated based on the three-dimensional weight matrix. Further, in terms of bidirectional cross-attention mechanism, a dynamic weight calculation architecture is used. By generating a three-dimensional weight matrix, cross-attention calculation can be effectively performed on the visible light feature space and the infrared feature space. In terms of actual application effect, the test results on the FLIR-VID dataset show that the edge definition of the pedestrian target is improved by 41%, the vehicle thermal feature artifact is reduced by 67%, the cross-modality feature alignment error is controlled within 3.2px, and the accuracy of target detection is significantly improved.

[0092] The thermal radiation feature analysis, the visible light feature analysis and the cross-modal feature analysis are respectively performed according to the attention features, and first analysis features, second analysis features and third analysis features are respectively obtained, the first analysis features, the second analysis features and the third analysis features are dynamically assigned weights, and a second fusion feature is obtained by fusing according to the assigned weights. The first analysis features are extracted by a first expert network, the first expert network adopts a depth separable convolution layer and a Mish activation function, and the depth separable convolution layer extracts a heat source distribution; the second analysis features are extracted by a second expert network, the second expert network adopts a ResNet structure and a Swish activation function, and the ResNet structure enhances a low-light image edge; and the third analysis features adopt a Transformer encoder for cross-modal association and a GELU activation function.

[0093] Specifically, the MOE expert network is applied, different feature analysis tasks are respectively constructed by the professional application of the expert network, and an infrared expert (corresponding to the first expert network Expert1), a visible light expert (corresponding to the second expert network Expert2) and a fusion expert (corresponding to the third expert network Expert3) are constructed. The infrared expert focuses on thermal radiation feature analysis, adopts a 3-layer DSConv structure and a Mish activation function (a depth separable convolution is used to extract a heat source distribution); the visible light expert focuses on texture detail enhancement, adopts a 5-layer ResNeXt structure and a Swish activation function (a ResNeXt structure is used to enhance a low-light image edge); and the fusion expert uses a Transformer encoder for cross-modal association, and the activation function is GELU.

[0094] The dynamic allocation of weights in the feature fusion process is realized by the dynamic gating network. In the dynamic gating network, the dynamic selection of the expert network weight is realized by weight allocation, and the weights of the expert networks are dynamically allocated (such as focusing on the infrared expert in low light, and balancing the weights in mixed light), and the actual running effect shows that in the low light (<10 lux) condition, the weight of the visible light expert is automatically reduced to below 0.2, and in the strong heat source interference, the weight of the infrared expert is increased to 0.65±0.12, the decision delay is controlled within 2.3 ms, and the real-time and accuracy are ensured. Finally, the outputs of the experts are integrated by a weighted voting mechanism. The design enables the system to adaptively adjust the modal fusion strategy, reduces the false detection rate, and improves the accuracy of vehicle detection in night complex light.

[0095] In the model training process, a loss function is obtained according to the output result of the network, then the parameters of the detection network are adjusted according to the loss function, until the model converges or reaches a preset training number;

[0096] The loss function of the detection network includes a detection loss value, a prototype contrast loss value and a light inconvenience loss value.

[0097] The detection loss value is represented as: wherein, The detection loss value is represented as N, the number of samples in a batch, C, the total number of target classes, y i,c is the true label of the i-th sample in the c-th class, p i,c is the class probability predicted by the network, and γ is an adjustment factor.

[0098] The prototype contrast loss value is represented as: wherein, is the prototype contrast loss value, M is the number of feature prototypes, φ v and φ i are the modal feature mapping functions of the visible light and infrared modalities, respectively, and are the m-th feature prototypes of the visible light and infrared modalities, respectively, is the L2 norm distance.

[0099] The illumination inconvenience loss value is represented as: wherein, is the illumination inconvenience loss value, x is the night road image, X is the exposure correction image, G is the generator of the exposure correction network, D is the discriminator of the exposure correction network, is the gradient calculation for the input.

[0100] In summary, the image collected by the unmanned aerial vehicle is subjected to exposure correction, and then the picture is input to the first 3 layers of the ResNet embedding lightweight cross-modal attention adapter. This module dynamically calculates the weight mapping matrix of the infrared and visible light feature maps through the cross-attention mechanism, and adopts the cycle consistency loss to ensure that the bidirectional transfer is distortion-free. Secondly, cross-modal semantic prototype vectors are constructed for each target class, and the prototype contrast loss is used to minimize the distance between the prototypes of the same class in different modalities. At the same time, an uncertainty calibration module is introduced to eliminate the prototype shift caused by changes in illumination. Finally, a multi-expert hybrid model is used, each expert network focuses on the semantic analysis of a specific modality, and the gating network dynamically allocates weights according to the characteristics of the input modality. Finally, the detection result is output by weighted voting, and the target is judged by the staff according to the detection result, and further evaluation and processing are carried out.

[0101] First, through the adaptive exposure correction network, the image in low light environment can be effectively optimized, which significantly improves the visibility and detail performance of the image. This method not only avoids the global overexposure or underexposure problem that may occur in traditional exposure adjustment, but also ensures the gradual improvement of image quality through step-by-step iteration, thereby improving the accuracy of target detection. Second, the introduction of the hierarchical transfer mechanism makes the knowledge transfer from the bottom local feature to the high-level global semantic more efficient. Through dynamic cross-attention calculation of visible light and infrared image features, the information of the two modalities can be effectively fused, and the recognition rate of the target is improved. This cross-modal feature fusion not only enhances the model's ability to capture details, but also significantly reduces the semantic bias caused by the heterogeneity between modalities, ensuring the detection stability in complex environments. In addition, the use of multi-expert hybrid model, for different feature analysis tasks, respectively constructs infrared experts, visible light experts and fusion experts, fully utilizes their respective advantages. Through the weight distribution of the dynamic gating network, the most suitable expert can be automatically selected for decision-making according to the environmental lighting conditions, so as to maintain high detection performance under different lighting conditions. This flexibility enables the system to maintain high detection accuracy and low false alarm rate in low light and strong light interference conditions. Effectively improve the detection accuracy, reduce the false alarm rate. Not only improves the target detection capability of unmanned aerial vehicle in night complex environment, but also provides more reliable technical support for related fields.

[0102] Embodiments of the present application provide an electronic device. The electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the above-mentioned unmanned aerial vehicle-based night vehicle detection method when executing the computer program.

[0103] The electronic device can be any intelligent terminal including a computer.

[0104] In general, for the hardware structure of the electronic device, the processor can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present application.

[0105] The memory can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the relevant program codes are saved in the memory and are called and executed by the processor to perform the method of the embodiments of the present application.

[0106] The input / output interface is used to realize information input and output.

[0107] The communication interface is used to realize the communication interaction between the device and other devices, and the communication can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0108] The bus transmits information between various components (such as the processor, the memory, the input / output interface, and the communication interface) of the device. The processor, the memory, the input / output interface, and the communication interface realize the communication connection between each other inside the device through the bus.

[0109] The embodiments of the present application provide a computer storage medium. The computer storage medium stores computer executable instructions for executing the unmanned aerial vehicle based night vehicle detection method as described above.

[0110] Those of ordinary skill in the art will appreciate that all or some steps, systems of the above disclosed methods can be implemented as software, firmware, hardware, or appropriate combinations of them. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on computer readable media, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, it should be appreciated by those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. In the foregoing description of the present specification, reference has been made to descriptive terms such as "one embodiment / implementation", "another embodiment / implementation", or "some embodiments / implementations" etc. It is appreciated that such descriptive terms are not intended to necessarily limit or restrict the scope of the embodiments or examples to the described embodiments / examples, but are merely used to connect a feature described in connection with one embodiment or example with one or more other embodiments or examples. Furthermore, it is also understood that the described features are not necessarily limited to a single embodiment or example, but can be combined with each other or with other features of one or more embodiments or examples.

[0111] Those of ordinary skill in the art will appreciate that all or some steps, systems, functional modules / units in the above disclosed methods can be implemented as software, firmware, hardware, or appropriate combinations of them.

[0112] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected to achieve the purpose of the present embodiment scheme according to actual needs.

[0113] In addition, each of the functional units in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0114] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application essentially, or parts of the technical solutions contributing to the prior art, or all or a part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and various other media capable of storing programs.

[0115] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other ways. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms. Although the embodiments of the present application have been shown and described, those skilled in the art can understand that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the claims and their equivalents.

[0116] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A method for detecting vehicles at night based on a UAV, characterized in that, The method comprises the following steps: obtaining a night road image by taking a photograph by a UAV, inputting the night road image into a detection network, wherein the night road image comprises a visible light image and an infrared image; performing exposure correction on the night road image to obtain an exposure correction image; extracting visible light features corresponding to the visible light image and infrared features corresponding to the infrared image from the exposure correction image; adjusting a first weight corresponding to the visible light features and a second weight of the infrared features according to the illumination condition and environmental features of the image, and fusing the visible light features and the infrared features according to the first weight and the second weight to obtain first fused features; aligning the first fused features to obtain aligned features, calculating attention features according to the aligned features, and performing thermal radiation feature analysis, visible light feature analysis and cross-modal feature analysis according to the attention features to obtain corresponding first analysis features, second analysis features and third analysis features respectively, dynamically assigning weights to the first analysis features, the second analysis features and the third analysis features, and fusing to obtain second fused features according to the assigned weights; detecting a vehicle according to the second fused features. 2.The UAV-based nighttime vehicle detection method of claim 1, wherein, The method comprises the following steps: estimating a parameter mapping of an image filter according to the latent features of the night road image by a parameter mapping module, adjusting the pixel values of the night road image according to the parameter mapping to generate a parameter map; adjusting the exposure value of the night road image by using a plurality of groups of image filters corresponding to the parameter map in sequence to obtain an exposure correction image, wherein the output of the previous group of image filters is used as the input of the current group of image filters, and each group of image filters comprises a brightening filter and a sharpening filter. 3.The UAV-based nighttime vehicle detection method of claim 2, wherein, The method comprises the following steps: performing residual convolution, downsampling and maximum pooling on the night road image to obtain residual encoding features; extracting latent features in a latent space from the residual encoding features; performing upsampling on the latent features, and generating a parameter mapping of an image filter by convolution according to the upsampling result. 4.The UAV-based nighttime vehicle detection method of claim 2, wherein, The parameter mapping module comprises a residual feature encoder, a latent feature extractor and a residual feature decoder. The residual feature encoder comprises a plurality of residual convolution blocks, wherein the input of the residual convolution block is added in a first addition layer after being processed by a first residual convolution branch and a second residual convolution branch, and the output end of the first addition layer is connected with a maximum pooling layer. The latent feature extractor comprises a plurality of Swin transformer blocks, wherein the input of the Swin transformer block is added to the first addition result after being processed by a first layer normalization module and a standard multi-head self-attention module, and the processing result of a second layer normalization module and a multi-layer perception module is added to the first addition result to obtain the output of the Swin transformer block. The residual feature decoder comprises a plurality of up-sampling residual convolution blocks, in which the up-sampled result of the input of an up-sampling residual convolution block is spliced with the output of the first adding layer of the residual feature encoder, the spliced result is processed by the third residual convolution branch and the fourth residual convolution branch, and the results of the third residual convolution branch and the fourth residual convolution branch are added in the second adding layer to obtain the output of the up-sampling residual convolution block. 5.The UAV-based nighttime vehicle detection method of claim 1, wherein, The feature alignment of the first fusion feature comprises: The feature alignment of the first fusion feature comprises: The feature alignment of the first fusion feature comprises: 6.The UAV-based nighttime vehicle detection method of claim 1, wherein, The first three convolution layers of the alignment network are inserted into the residual block of the cross-modal attention adapter, and share parameters with the branch corresponding to the visible light feature and the branch corresponding to the infrared feature; the fourth convolution layer is inserted into deformable convolution. The attention feature calculated according to the alignment feature comprises: 7.The UAV-based nighttime vehicle detection method of claim 1, wherein, The cross-attention calculation of the visible light feature space and the infrared feature space is performed according to the alignment feature, a three-dimensional weight matrix is generated, and the attention feature is calculated according to the three-dimensional weight matrix. 8.The UAV-based nighttime vehicle detection method of claim 1, wherein, The first analytic feature is extracted by a first expert network, the first expert network adopts a depth separable convolution layer and a Mish activation function, the depth separable convolution layer extracts a heat source distribution; the second analytic feature is extracted by a second expert network, the second expert network adopts a ResNeXt structure and a Swish activation function, the ResNet structure enhances the edge of a low-light image; the third analytic feature adopts a Transformer encoder for cross-modal correlation and a GELU activation function. The method further comprises: Adjusting the parameters of the detection network according to a loss function; The detection loss value is represented as: In the formula, As the detection loss value, N is the number of samples in the batch, C is the total number of target categories, y i,c is the true label of the ith sample in the c category, p i,c is the category probability predicted by the network, and γ is an adjustment factor. The prototype contrast loss value is represented as: In the formula, is the prototype contrast loss value, M is the number of feature prototypes, φ v and φ i are the modal feature mapping functions of the visible light and infrared modalities, respectively, and are the mth feature prototypes of the visible light and infrared modalities, respectively, is the L2 norm distance; The inconvenience of illumination loss value is expressed as: In the formula, is the inconvenience of illumination loss value, x is the night road image, X is the exposure correction image, G is the generator of the exposure correction network, D is the discriminator of the exposure correction network, is the gradient calculation for input.

9. An electronic device, comprising: The loss function of the detection network comprises a detection loss value, a prototype contrast loss value and an inconvenience of light loss value. The method further comprises:

10. A computer storage medium, characterized in that, A memory, a processor and a computer program stored in the memory and executable on the processor, when the processor executes the computer program, the method for detecting night vehicles based on a UAV in any one of claims 1-8 is realized. Computer executable instructions are stored, and the computer executable instructions are used to execute the method for detecting night vehicles based on a UAV in any one of claims 1-8.

Images