Road vehicle detection method for high-resolution remote sensing image based on feature fusion

Abstract

The present application relates to the technical field of image processing, and more particularly to a high-resolution remote sensing image road vehicle detection method based on feature fusion, wherein an inverted residual self-attention module and a bidirectional multi-scale feature fusion module are added in YOLOv11; in the inverted residual self-attention module, an inverted residual structure is used to extract spatial features in combination with a depth separable convolution, and a multi-head attention mechanism is used to enhance features; in the bidirectional multi-scale feature fusion module, an adaptive pyramid is used to autonomously find the optimal feature down-sampling rate; the output features of the adaptive pyramid are controlled to transmit cross-layer information through a differentiable gating unit; and a double feature extraction unit is used to extract and fuse spatial information and channel information of the features. The detection method proposed in the present application achieves excellent performance on multiple public data sets, especially in small target and occluded target detection, and has high practical application value.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a high-resolution remote sensing image road vehicle detection method based on feature fusion. BACKGROUND

[0002] High-resolution remote sensing images are increasingly widely used in urban planning and intelligent driving, and among them, road vehicle detection is one of the key technologies to achieve these application scenarios. With the development of social economy and the progress of technology, road traffic flow is growing, and higher requirements are put forward for traffic management and monitoring. Traditional traffic monitoring systems mainly rely on ground-mounted cameras or sensors, but their coverage is limited, and it is difficult to meet the real-time monitoring needs in large-scale areas. In addition, in some special scenarios such as highways, remote areas, etc., the deployment of ground equipment not only has high cost, but also has the problem of difficult maintenance. Therefore, using high-resolution remote sensing images for vehicle detection has become a very promising method.

[0003] Early research mainly focused on manually designed feature extractors, such as SIFT (Scale-Invariant Feature Transform), HOG (Histogram of Oriented Gradients), etc. These methods capture significant features in images by carefully selecting and adjusting parameters, and although they have achieved some success in certain specific application scenarios, they lack adaptive learning ability and often have insufficient generalization performance, especially when dealing with complex backgrounds. In addition, such methods usually rely on a large amount of manually labeled data, making it difficult to deal with new situations or unknown types of targets.

[0004] With the continuous improvement of deep learning theory in the field of computer vision, Convolutional Neural Networks (CNN) have been widely applied to target detection tasks due to their strong feature representation ability and adaptive learning characteristics. CNN models can automatically learn effective feature representations from a large amount of labeled data and improve classification accuracy through multi-level abstraction. However, ordinary CNN models usually only focus on shallow features or deep features, failing to fully utilize the complementary relationship between different levels of features, resulting in more missed and false detections in complex backgrounds. SUMMARY

[0005] Therefore, the present application aims to provide a high-resolution remote sensing image road vehicle detection method based on feature fusion, which uses YOLOv11 as the basic framework, combines the inverted residual self-attention mechanism, multi-scale feature aggregation mechanism, and loss function optimization to improve the model's detection and processing capabilities for high-resolution remote sensing images.

[0006] To achieve the above purpose, the technical solution of the present application is as follows:

[0007] A high-resolution remote sensing image road vehicle detection method based on feature fusion, comprising:

[0008] S1: Obtain a road vehicle image dataset, and preprocess the road vehicle image dataset to obtain a training set;

[0009] S2: Construct a road vehicle detection network based on a YOLOv11 network, wherein an inverted residual self-attention module and a bidirectional multi-scale feature fusion module are added in the backbone network of YOLOv11; an inverted residual self-attention module is added in the neck network of YOLOv11; in the inverted residual self-attention module, an inverted residual structure is used to extract spatial features in combination with a depth separable convolution, and a multi-head attention mechanism is used for feature enhancement; in the bidirectional multi-scale feature fusion module, an adaptive pyramid is used to enable the network to autonomously discover the optimal feature down-sampling rate during training; the output features of the adaptive pyramid are controlled to transfer cross-layer information through a differentiable gating unit; a double feature extraction unit is used to extract spatial information and channel information from the output features of the differentiable gating unit, and the two kinds of information are fused to obtain output features;

[0010] S3: Train the road vehicle detection network constructed in step S2 using the training set obtained in step S1 to obtain a road vehicle detection model;

[0011] S4: Input the road vehicle image to be detected into the road vehicle detection model obtained in step S3 to obtain a road vehicle detection result.

[0012] Further, in the backbone network: after the input image is processed by two consecutive convolutions, it is input into the inverted residual self-attention module, and the output features of the inverted residual self-attention module are input into the inverted residual self-attention module again after being processed by two consecutive groups of convolutions and C3k2 operations and then being processed by a convolution operation; the features output by the inverted residual self-attention module are input into the neck network after being processed by an SPPF operation and a bidirectional multi-scale feature fusion module in turn.

[0013] Further, in the neck network: the output feature of the bidirectional multi-scale feature fusion module is up-sampled, and the up-sampled feature is fused with the output feature of the second C3k2 module in the backbone network along the data processing sequence; the fused feature is input into the inverted residual self-attention module for processing, and the processed feature is up-sampled; the up-sampled feature is fused with the output feature of the first C3k2 module in the backbone network along the data processing sequence; the fused feature is input into the first C3k2 module in the neck network for processing, and the processed feature is fused with the output feature of the inverted residual self-attention module after convolution operation; the fused feature is input into the second C3k2 module in the neck network for processing, and the processed feature is fused with the output feature of the inverted residual self-attention module after convolution operation; the fused feature is input into the third C3k2 module in the neck network for processing, and the output features of the three C3k2 modules are respectively input into the three detection heads in the head network of YOLOv11, and the road vehicle detection result is output.

[0014] Further, in the inverted residual self-attention module: the input feature is divided into 4 sub-features; 3 of the sub-features are linearly projected to obtain query tuples, key tuples and value tuples; according to the query tuples, the key tuples and the value tuples, the attention weight is obtained by the following formula:

[0015] ;

[0016] Where, Attention represents the attention weight, Q represents the query tuple, K represents the key tuple, V represents the value tuple, d represents the feature dimension, and B represents the relative position bias matrix; the value tuple is subjected to depth separable convolution, the obtained feature is added to the value tuple and the corresponding elements of the attention weight, and then the obtained feature is subjected to convolution operation and added to the corresponding elements of the remaining last sub-feature to obtain the output feature.

[0017] Further, the adaptive pyramid includes a plurality of adaptive pooling layers in cascade; wherein the input feature is down-sampled layer by layer through the plurality of adaptive pooling layers to obtain a first pooled feature; the first pooled feature is down-sampled layer by layer through the plurality of adaptive pooling layers along the reverse direction of its data processing to obtain a second pooled feature; the first pooled feature and the second pooled feature are input into the differentiable gating unit.

[0018] Further, in the differentiable gating unit, the first pooled feature and the second pooled feature are subjected to convolution processing to obtain gating weights; the first pooled feature and the second pooled feature are weighted summed according to the gating weights to obtain an output feature.

[0019] Further, in the double feature extraction unit: the input features are respectively subjected to channel attention mechanism processing and spatial attention mechanism processing, and channel weights and spatial weights are respectively obtained; the obtained channel weights and spatial weights are subjected to Hadmard product, and output features are obtained.

[0020] Further, the road vehicle detection network in step S2 further comprises a vehicle geometric constraint module and a shadow compensation module; wherein: the vehicle geometric constraint module constrains the output of the head network, and rejects the detection box whose length-width ratio is out of the range [0.8, 2.5]; the shadow compensation module predicts the vehicle shadow area through the generative adversarial network and corrects the detection box position; wherein, the shadow compensation module comprises a generator of U-Net structure and a discriminator of PatchGAN structure; the generator receives the original remote sensing image and the initial detection box, and generates the corresponding predicted vehicle shadow probability map; the predicted vehicle shadow probability map and the corresponding real vehicle shadow probability map are input into the discriminator, the gap score of the predicted vehicle shadow probability map and the real vehicle shadow probability map is calculated, and the generator is trained based on the gap score, so that the predicted vehicle shadow probability map is infinitely close to the real vehicle shadow probability map.

[0021] Further, in step S3, the road vehicle detection network is trained using a composite loss function; the composite loss function is:

[0022] ;

[0023] Wherein, L represents the composite loss function, represents the weighted intersection over union loss, represents the classification loss function, represents the edge perception loss, 、 and respectively represent the weights of the three losses.

[0024] Further, the weighted intersection over union loss is:

[0025] ;

[0026] Wherein, N represents the total number of targets used to calculate the loss function, ω i represents the dynamic weight coefficient corresponding to the detection of the i-th target, which varies nonlinearly with the prediction confidence, represents the prediction box of the i-th target, represents the real box of the i-th target, and ρ represents an exponential decay function of the distance between the center points of the prediction box and the real box.

[0027] A difficult sample mining mechanism is introduced on the basis of the standard classification loss function, and the classification loss function is obtained, that is, the classification loss function is:

[0028] ;

[0029] wherein, N hard represents the number of difficult samples in each training, represents the difficult sample set, represents the focus parameter, p j represents the probability of predicting correctly the jth sample in the difficult sample set, represents the adaptive modulation factor, represents the current training round, and represents the total training round;

[0030] The edge perception loss is:

[0031] ;

[0032] wherein, represents the gradient calculation of the mth order, E canny represents the structural edge obtained by Canny edge detection, E sobel represents the structural edge extracted by the Sobel operator, Spred represents the segmentation map of the image in the prediction frame, S gt represents the corresponding real label.

[0033] Compared with the prior art, the present application can achieve the following beneficial effects:

[0034] The feature fusion-based high-resolution remote sensing image road vehicle detection method provided by the present application takes YOLOv11 as a basic framework, introduces an inverted residual self-attention module, a bidirectional multi-scale feature fusion mechanism and a composite loss function optimization, and significantly improves the detection accuracy and robustness of road vehicles in high-resolution remote sensing images. First, the inverted residual self-attention module realizes the cooperative optimization of local detail preservation and global context perception through depth separable convolution combined with deformable spatial attention, effectively enhancing the feature extraction capability of the backbone network. Second, the bidirectional multi-scale feature fusion mechanism realizes cross-scale feature fusion through adaptive pooling and differentiable gating units, and enhances the feature response of occluded vehicles through a space-channel collaborative attention module. Finally, the composite loss function is composed of a weighted intersection over union loss, a classification focal loss and an edge perception loss, which respectively optimize the boundary box regression deviation, the class imbalance and the edge confusion problem, and improve the performance of the model in dense arrangement and complex scenes. BRIEF DESCRIPTION OF DRAWINGS

[0035] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application and of the related description are used to explain the application and are not intended to limit the application. In the drawings:

[0036] Figure 1A flowchart of a high-resolution remote sensing image road vehicle detection method based on feature fusion according to an embodiment of the present invention is shown in FIG. 1.

[0037] Figure 2 A schematic diagram of a road vehicle detection network according to an embodiment of the present invention is shown in FIG. 2.

[0038] Figure 3 A schematic diagram of a reverse residual self-attention module according to an embodiment of the present invention is shown in FIG. 3.

[0039] Figure 4 A schematic diagram of a bidirectional multi-scale feature fusion module according to an embodiment of the present invention is shown in FIG. 4.

[0040] Figure 5 A schematic diagram of a double feature extraction unit according to an embodiment of the present invention is shown in FIG. 5. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation on the present invention.

[0042] It should be noted that the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.

[0043] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present invention and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second" and the like are only used for description purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features limited by "first", "second" and the like can explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0044] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be connected inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0045] The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0046] As Figure 1 shown, the high-resolution remote sensing image road vehicle detection method based on feature fusion according to the embodiments of the present application comprises:

[0047] S1: Obtain a road vehicle image dataset and preprocess the road vehicle image dataset to obtain a training set. In the embodiments of the present application, the road vehicle image dataset used is the public dataset DLR 3K Vehicle, and the road vehicle image dataset is preprocessed by Mosaic enhancement, MixUp enhancement, geometric transformation and the like to complete data enhancement to obtain the training set.

[0048] S2: Construct a road vehicle detection network based on a YOLOv11 network.

[0049] The road vehicle detection network constructed by the embodiments of the present application is as Figure 2 shown, and also comprises a backbone network, a neck network and a head network. Among them, an inverted residual self-attention module and a bidirectional multi-scale feature fusion module are added in the backbone network; an inverted residual self-attention module is added in the neck network.

[0050] In the backbone network, after the input image is processed by two consecutive convolution operations (Conv), the input is input into the inverted residual self-attention (IRSA) module, and the output feature of the inverted residual self-attention module is processed by two consecutive convolution operations and C3k2 operations, and then is input into the inverted residual self-attention module after convolution operation. Further, one convolution operation and one C3k2 operation form a group, and the output feature of the inverted residual self-attention module is input into the inverted residual self-attention module after being processed by the above two consecutive groups of operations and then being processed by convolution operation. The feature output by the inverted residual self-attention module is sequentially input into the neck network after being processed by the SPPF operation and the bidirectional multi-scale feature fusion (BMFF) module. The C3k2 operation and the SPPF operation are both inherent module operations in the original YOLOv11 network, and no other changes are made in the present application except for adaptive adjustment of input and output. The present application introduces the inverted residual self-attention module into the backbone network of the original YOLOv11 network, enhances the feature extraction capability of the backbone network, and combines the depth separable convolution and the deformable spatial attention.

[0051] In the neck network, the output feature of the bidirectional multi-scale feature fusion module is upsampled, and the upsampled feature is fused with the output feature of the second C3k2 module in the backbone network along the data processing sequence. The fused feature is input into the inverted residual self-attention module for processing, and the processed feature is upsampled. The upsampled feature is fused with the output feature of the first C3k2 module in the backbone network along the data processing sequence. The fused feature is input into the first C3k2 module in the neck network for processing, and the processed feature is fused with the output feature of the inverted residual self-attention module after convolution operation. The fused feature is input into the second C3k2 module in the neck network for processing, and the processed feature is fused with the output feature of the inverted residual self-attention module after convolution operation. The fused feature is input into the third C3k2 module in the neck network for processing, and the output features of the three C3k2 modules are respectively input into the three detection heads (Detect) in the head network, and the road vehicle detection result is output.

[0052] The present method designs an inverted residual self-attention module for the characteristics of large scale difference and variable shape of vehicle targets in high-resolution remote sensing images. The inverted residual self-attention module uses an inverted residual structure, combines depth separable convolution for spatial feature extraction, and uses a multi-head attention mechanism for feature enhancement. Specifically, as shown in FIG. 2, the inverted residual self-attention module includes a depth separable convolution layer, a multi-head attention layer, and an inverted residual structure. Figure 3As shown, the input feature is divided into 4 sub-features; 3 of the sub-features are linearly projected to obtain a query tuple, a key tuple and a value tuple; according to the query tuple, the key tuple and the value tuple, the attention weight is obtained through the following formula:

[0053] ;

[0054] Wherein, Attention represents the attention weight, Q represents the query tuple, K represents the key tuple, V represents the value tuple, d represents the feature dimension, and B represents the relative position bias matrix; the value tuple is subjected to a depth separable convolution, the obtained feature is added to the value tuple and the corresponding elements of the attention weight, and then the obtained feature is subjected to a convolution operation and added to the corresponding elements of the remaining last sub-feature to obtain the output feature. In the embodiment of the application, specifically, the input feature is divided into 4 sub-features, 2 of which are query tuples and key tuples, 1 of which is subjected to a 1x1 convolution operation to obtain a value tuple, and the attention weight is obtained by the query tuple, the key tuple and the value tuple through the formula; the value tuple is subjected to a depth separable convolution with a scale of 3x3, the obtained feature is added to the value tuple and the corresponding elements of the attention weight, and then the obtained feature is subjected to a 1x1 convolution operation and added to the corresponding elements of the remaining last sub-feature to obtain the output feature.

[0055] The inverted residual self-attention module provided by the application realizes the collaborative optimization of local detail preservation and global context perception by combining depth separable convolution with deformable spatial attention, and effectively enhances the feature extraction capability of the backbone network. In addition, the depth separable convolution layer is introduced into the inverted residual self-attention module in the application, which is decomposed into a standard convolution and a point-by-point convolution, thereby reducing the amount of calculation.

[0056] The bidirectional multi-scale feature fusion module is as shown in Figure 4 The adaptive pyramid is used to enable the network to autonomously discover the optimal feature down-sampling rate during training; the output feature of the adaptive pyramid is controlled to complete cross-layer information transmission and fusion through the differentiable gating unit; the spatial information and channel information of the output feature of the differentiable gating unit are extracted by the double feature extraction unit, and the two kinds of information are fused to obtain the output feature.

[0057] The adaptive pyramid includes a plurality of cascaded adaptive pooling layers (Learnable Adaptive Pooling, LAP), and the adaptive pooling layer dynamically adjusts the pooling kernel size through a differentiable parameter θ, and the corresponding formula is:

[0058] ;

[0059] wherein, x represents an input of the adaptive pooling layer, k represents a pooling kernel of the adaptive pooling layer, s represents a feature map size of the input, and θ∈[0.25, 0.75]. The adaptive pyramid adopts a bidirectional cross-scale connection to cascade up-sampling of high-level semantic features and down-sampling of low-level detailed features. Specifically, in the adaptive pyramid, after the input features are sequentially down-sampled by a plurality of adaptive pooling layers, first pooled features are obtained; after the first pooled features are sequentially down-sampled by the plurality of adaptive pooling layers along an inverse direction of data processing of the first pooled features, second pooled features are obtained; and the first pooled features and the second pooled features are input into a differentiable gating unit. In an embodiment of the present application, the adaptive pyramid includes three adaptive pooling layers, that is, after the input features are sequentially down-sampled by the three adaptive pooling layers, the first pooled features F top are obtained. top After the first pooled features F bot are sequentially down-sampled by the three adaptive pooling layers along the inverse direction of data processing of the first pooled features F bot , the second pooled features F top are obtained. bot The first pooled features F top and the second pooled features F bot are input into the differentiable gating unit.

[0060] In the differentiable gating unit, convolution processing is performed on the first pooled features and the second pooled features to obtain gating weights; and the first pooled features and the second pooled features are weighted and summed according to the gating weights to obtain output features.

[0061] In an embodiment of the present application, the process of obtaining the gating weights is represented by the following formula:

[0062] G=σ(Conv 1×1 ([F top ,F bot ]));

[0063] wherein, G represents the gating weights, σ represents a Sigmoid function, and Conv 1×1 represents a convolution operation with a convolution kernel size of 1×1.

[0064] The process of obtaining the output features is represented by the following formula:

[0065] F out =G×F top +(1-G)×F bot ;

[0066] wherein, F out represents the output features.

[0067] In the dual-feature extraction unit, as Figure 5As shown, the input features are respectively subjected to channel attention mechanism processing and spatial attention mechanism processing, respectively obtaining channel weights and spatial weights; the obtained channel weights and spatial weights are subjected to Hadmard product, obtaining double feature weights; the double feature weights are assigned to the input features, obtaining output features. In the embodiment of the present application, in the process of channel attention mechanism processing of the input features, the input features are subjected to global average pooling, and the pooled features sequentially pass through a fully connected layer (FC)-ReLU activation layer-a fully connected layer (FC), obtaining channel weights; in the process of spatial attention mechanism processing of the input features, the input features are subjected to deformable convolution operation, obtaining spatial weights; the obtained channel weights and spatial weights are subjected to Hadmard product, obtaining output features.

[0068] The bidirectional multi-scale feature fusion module provided by the present application firstly performs adaptive maximum pooling on the features by using an adaptive feature pyramid, and adopts bidirectional cross-scale connection to cascade up-sampling of high-level semantic features and down-sampling of low-level detail features; then a differentiable feature selection gate is introduced, that is, a gating function is introduced during feature fusion to dynamically adjust the feature contribution weight, thereby suppressing redundant information; a deformable convolution kernel is introduced in the double feature extraction unit to complete spatial attention mechanism, thereby constructing a channel-spatial double-path feature weighting network module; the spatial attention path performs deformable convolution to obtain deformation features and spatial features; the channel attention path generates channel weights through a channel random drop strategy; the spatial features and the channel weights are combined, and finally the dynamic spatial modulation of the feature map is realized.

[0069] In the embodiment of the present application, the road vehicle detection network further comprises a vehicle geometric constraint module and a shadow compensation module; wherein: the vehicle geometric constraint module constrains the output of the head network, and rejects the detection box whose length-width ratio exceeds the range [0.8, 2.5]; the shadow compensation module predicts the vehicle shadow area and corrects the detection box position. Specifically, the shadow compensation module comprises a generator of U-Net structure and a discriminator of PatchGAN structure. The generator receives the original remote sensing image and the initial detection box, and generates the corresponding predicted vehicle shadow probability map; the predicted vehicle shadow probability map and the corresponding real vehicle shadow probability map are input into the discriminator, the gap score of the predicted vehicle shadow probability map and the real vehicle shadow probability map is calculated, and the generator is trained based on the gap score, so that the predicted vehicle shadow probability map is infinitely close to the real vehicle shadow probability map. In the embodiment of the present application, the generator in the shadow compensation module specifically adopts the existing lightweight MobileUNet, that is, the original remote sensing image and the initial detection box are received as input, and the vehicle shadow probability map is generated through the lightweight U-Net structure, and the discriminator of the shadow compensation module adopts PatchGAN, that is, the vehicle shadow probability map output by the generator and the real vehicle shadow probability map are input as input, and the real shadow area and the generated shadow area are distinguished through PatchGAN. Through training the whole shadow compensation module, the gap between the vehicle shadow probability map output by the generator and the real vehicle shadow probability map is continuously reduced, that is, through training, the vehicle shadow probability map output by the generator is infinitely close to the real vehicle shadow probability map. In actual use, the original remote sensing image and the initial detection box are directly input into the generator of the shadow compensation module to obtain the corresponding vehicle shadow probability map.

[0070] S3: training the road vehicle detection network constructed in step S2 using the training set obtained in step S1 to obtain a road vehicle detection model.

[0071] In step S3 of the embodiment of the present application, the road vehicle detection network is trained using a composite loss function; the composite loss function is:

[0072] ;

[0073] wherein L represents the composite loss function, represents the weighted intersection-over-union loss, represents the classification loss function, represents the edge-aware loss, 、 and Respectively represent the weight of three kinds of loss. The optimal weight of three kinds of loss is determined in the process of network model training, in the embodiment of the application, a curriculum learning strategy is adopted, that is, the classification loss is focused at the initial stage.

[0074] The application is to alleviate the bounding box regression deviation when vehicles are arranged densely, and a dynamic weighted intersection over union loss (WIoU Loss) function is proposed. The traditional IoU loss treats all samples equally, while the weighted intersection over union loss adaptively adjusts the penalty weight through the target size and the prediction confidence. The weighted intersection over union loss L WIoU As follows:

[0075] ;

[0076] Wherein, N represents the total number of targets for calculating the loss function, ω i represents the dynamic weight coefficient corresponding to the detection of the i-th target, which changes nonlinearly with the prediction confidence, represents the prediction box of the i-th target, represents the real box of the i-th target, and p represents an exponential decay function of the distance between the center points of the prediction box and the real box.

[0077] The application is to solve the problem of large proportion of vehicle and background pixels in remote sensing images. A difficult sample mining mechanism is introduced on the basis of the standard classification loss function, that is:

[0078] ;

[0079] Wherein, N hard represents the number of difficult samples, and in the embodiment of the application, the definition of difficult is that the proportion of vehicle and background pixels in remote sensing images is 1:1200, Omega represents the difficult sample set, gamma represents the focusing parameter, and p j represents the probability of predicting correctly the j-th sample in the difficult sample set Omega, tau represents an adaptive modulation factor, and specifically tau=2+sin(2pi t / T), wherein t represents the current training round, and T represents the total training round. The adaptive modulation factor tau with the sine function is used to realize periodic adjustment of the weight of difficult and easy samples. In the embodiment of the application, the difficult sample set Omega is adjusted through a dynamic screening strategy, that is, the first K gradient amplitude maximum negative samples are selected in each batch to form the difficult sample set Omega, and the value of K is 0.2 times the total number of samples.

[0080] The application is to solve the confusion problem of vehicle edges and background (such as road markings and shadows), and a multi-modal edge constraint term is designed to form an edge perception loss L edge , specifically:

[0081] ;

[0082] Wherein, denotes the gradient calculation of the m-th order, E canny denotes the structural edge obtained by Canny edge detection, E sobel denotes the structural edge extracted by Sobel operator, Spred denotes the segmentation map of the image in the prediction box, S gt denotes the corresponding real label.

[0083] The composite loss function provided by the application is composed of a weighted intersection-over-union loss, a classification focal loss and an edge-aware loss, and is respectively optimized for the bounding box regression deviation, the class imbalance and the edge confusion problem, thereby improving the performance of the model in dense arrangement and complex scenes.

[0084] In step S3 of the embodiment of the application, the model structure and parameters can also be further optimized according to the training condition, that is, the inference speed is improved and the model complexity is reduced by using model pruning and quantization technology. Specifically, model pruning reduces the number of parameters by removing unimportant connections or neurons, and quantization converts floating-point numbers to low-precision integers, thereby significantly reducing the amount of calculation and storage requirements.

[0085] S4: inputting a road vehicle image to be detected into the road vehicle detection model obtained in step S3, and predicting a road vehicle detection result.

[0086] In the implementation of the present application, the high-resolution remote sensing image road vehicle detection method based on feature fusion proposed in the present application is experimented. Specifically, in the 0.5 m high-resolution satellite image test, the boundary positioning error of the improved network for the vehicle target is reduced by 23.7%, and the missed detection rate under the tree shadow shielding scene is reduced by 18.4%. Through the ablation experiment of the inverse residual self-attention module, it is found that the depth separable convolution improves the detection recall rate of the multi-angle parked vehicle by 12.6%, and the multi-head self-attention mechanism reduces the false detection rate of the complex scene such as the overpass by 9.8%. The module parameter amount is only increased by 15.3%, and the inference speed is kept at 87 fps (1080ti), which meets the engineering deployment requirements. In the ablation experiment of the bidirectional multi-scale feature fusion module proposed in the present application, it is found that: in the experiment on the DOTA-Vehicle data set, the adaptive pyramid proposed in the present application improves the AP value of the micro-vehicle detection by 3.2% compared with the traditional fixed ratio pyramid (such as 1 / 8, 1 / 16 and 1 / 32 of FPN); the differentiable gating unit proposed in the present application reduces the vehicle overlap misjudgment rate from 14.7% to 9.3% in the overpass intersection scene. In addition, the embodiment of the present application compares the composite loss function proposed in the present application based on the DIOR-Vehicle data set, and finds that the composite loss function proposed in the present application improves the mAP by 12.4% (from 69.3% to 81.7%) compared with the single weighted intersection-over-union loss, the edge perception loss improves the detection stability in the shielding scene by 18.9%, the joint optimization strategy reduces the training shock and accelerates the convergence speed by 37%, that is, the epoch required to reach the same precision is reduced from 230 to 145. The embodiment of the present application also performs ablation experiment on the composite loss function: the weighted intersection-over-union loss improves the boundary box positioning accuracy by 7.3% in the dense scene where the vehicle spacing is less than 5 pixels, especially in the parking lot and other regular arrangement areas; the classification loss function with the difficulty sample mining mechanism improves the classification accuracy of small vehicles (<15 pixels) from 81.2% to 89.5% in the ISPRS Potsdam data set test, and reduces the false positive rate of mistaking tree crowns as vehicles by 14.8%; the edge perception loss with the multi-modal edge constraint term improves the vehicle contour intersection-over-union (Boundary IoU) from 72.1% to 83.6% in the WorldView-3 image test, especially improves the boundary fitting degree of vehicles with shadows (such as vehicles parked under trees).

[0087] It should be understood that the various forms of flow shown above can be reordered, steps added or removed. For example, the steps described in the present disclosure can be executed in parallel, in sequence, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which is not limited herein.

[0088] The above detailed description does not limit the scope of the application. Various modifications, combinations, sub-combinations and alternatives can be made to the detailed description. Any modification, equivalent replacement and improvement etc. made within the spirit and principle of the application shall be included in the scope of the application.

Claims

1. A method for road vehicle detection based on high-resolution remote sensing imagery using feature fusion, characterized in that, include: S1: Obtain a road vehicle image dataset and preprocess the road vehicle image dataset to obtain a training set; S2: Construct a road vehicle detection network based on the YOLOv11 network, wherein an inverse residual self-attention module and a bidirectional multi-scale feature fusion module are added to the backbone network of the YOLOv11; and the inverse residual self-attention module is added to the neck network of the YOLOv11. The inverted residual self-attention module utilizes an inverted residual structure combined with depthwise separable convolution for spatial feature extraction, while simultaneously employing a multi-head attention mechanism for feature enhancement. In the bidirectional multi-scale feature fusion module, an adaptive pyramid is used to enable the network to autonomously discover the optimal feature downsampling rate during training; the output features of the adaptive pyramid are controlled by a differentiable gating unit to transmit cross-layer information; a dual feature extraction unit is used to extract spatial and channel information from the output features of the differentiable gating unit, and the two types of information are fused to obtain the output features; S3: Use the training set obtained in step S1 to train the road vehicle detection network constructed in step S2 to obtain the road vehicle detection model; S4: Input the road vehicle image to be detected into the road vehicle detection model obtained in step S3 to predict the road vehicle detection result.

2. The method for road vehicle detection based on feature fusion in high-resolution remote sensing imagery according to claim 1, characterized in that, In the backbone network: After the input image is processed by two consecutive convolutions, it is input into the inverse residual self-attention module. The output features of the inverse residual self-attention module are then processed by two consecutive sets of convolutions and C3k2 operations, and then processed by another convolution operation before being input into the inverse residual self-attention module. The output features of the inverse residual self-attention module are then processed by SPPF operations and the bidirectional multi-scale feature fusion module before being input into the neck network.

3. The method for road vehicle detection based on feature fusion in high-resolution remote sensing imagery according to claim 1, characterized in that, In the neck network: The output features of the bidirectional multi-scale feature fusion module are upsampled, and the upsampled features are fused with the output features of the second C3k2 module in the backbone network along the data processing order. The fused features are input into the inverse residual self-attention module for processing, and the processed features are upsampled; the upsampled features are then fused with the output features of the first C3k2 module in the backbone network along the data processing order. The fused features are input into the first C3k2 module in the neck network for processing. The processed features are then convolved and fused with the output features of the inverted residual self-attention module. The fused features are input into the second C3k2 module in the neck network for processing. The processed features are then convolved and fused with the output features of the inverted residual self-attention module. The fused features are then input into the third C3k2 module in the neck network for processing. The output features of the three C3k2 modules are respectively input into the three detection heads in the YOLOv11 head network to output the road vehicle detection results.

4. The method for road vehicle detection based on feature fusion in high-resolution remote sensing imagery according to claim 1, characterized in that, In the inverted residual self-attention module: The input features are divided into four sub-features; linear projection is performed on three of these sub-features to obtain the query tuple, key tuple, and value tuple. Based on the query tuple, the key tuple, and the value tuple, the attention weight is derived using the following formula: ； Where Attention represents the attention weight, Q represents the query tuple, K represents the key tuple, V represents the value tuple, d represents the feature dimension, and B represents the relative position bias matrix; Perform a depthwise separable convolution on the tuple, add the resulting feature to the tuple and the corresponding element of the attention weight, then perform a convolution operation on the resulting feature and add it to the corresponding element of the last remaining sub-feature to obtain the output feature.

5. The method for road vehicle detection based on feature fusion in high-resolution remote sensing imagery according to claim 1, characterized in that, The adaptive pyramid includes multiple cascaded adaptive pooling layers; wherein, the input feature is downsampled layer by layer by multiple adaptive pooling layers to obtain a first pooling feature; the first pooling feature is downsampled layer by layer by multiple adaptive pooling layers in the reverse direction of its data processing to obtain a second pooling feature; the first pooling feature and the second pooling feature are input into the differentiable gating unit.

6. The method for road vehicle detection based on feature fusion in high-resolution remote sensing imagery according to claim 5, characterized in that, In the differentiable gating unit, the first pooling feature and the second pooling feature are convolved to obtain the gating weights; Based on the gating weights, the first pooling feature and the second pooling feature are weighted and summed to obtain the output feature.

7. The method for road vehicle detection based on feature fusion in high-resolution remote sensing imagery according to claim 1, characterized in that, In the dual feature extraction unit: The input features are processed by channel attention mechanism and spatial attention mechanism respectively to obtain channel weights and spatial weights; The obtained channel weights and spatial weights are multiplied by Hadmard to obtain the output features.

8. The method for road vehicle detection based on feature fusion in high-resolution remote sensing imagery according to claim 3, characterized in that, The road vehicle detection network in step S2 also includes a vehicle geometry constraint module and a shadow compensation module; wherein: The vehicle geometry constraint module constrains the output of the head network, rejecting detection boxes with aspect ratios exceeding the range of [0.8, 2.5]. The shadow compensation module predicts vehicle shadow regions and corrects the detection box positions. The shadow compensation module includes a generator with a U-Net structure and a discriminator with a PatchGAN structure. The generator receives the original remote sensing image and initial detection boxes, and generates a corresponding predicted vehicle shadow probability map. The predicted vehicle shadow probability map and the corresponding real vehicle shadow probability map are input into the discriminator to calculate the difference score between the predicted and real vehicle shadow probability maps. Based on the difference score, the generator is trained so that the predicted vehicle shadow probability map is infinitely close to the real vehicle shadow probability map.

9. The method for road vehicle detection based on feature fusion in high-resolution remote sensing imagery according to claim 1, characterized in that, In step S3, the road vehicle detection network is trained using a composite loss function; The composite loss function is: ； Where L represents the composite loss function, This indicates the weighted average loss. Represents the classification loss function. This represents edge perception loss. , and These represent the weights of the three types of losses, respectively.

10. The method for road vehicle detection based on feature fusion in high-resolution remote sensing imagery according to claim 9, characterized in that, The weighted intersection-union ratio loss is: ； Where N represents the total number of targets used to calculate the loss function, ω i This represents the dynamic weight coefficient corresponding to the detection of the i-th target, which changes non-linearly with the prediction confidence. This represents the predicted bounding box for the i-th target. Let ρ represent the ground truth bounding box of the i-th target, and let ρ represent the exponential decay function of the distance between the center point of the predicted box and the center point of the ground truth bounding box. By introducing a hard sample mining mechanism based on the standard classification loss function, the classification loss function is obtained, namely: ； Where, N hard Ω represents the number of hard samples in each training iteration, γ represents the hard sample set, and p represents the focusing parameter. j Let τ represent the probability of correctly predicting the j-th sample in the difficult sample set Ω, and let τ represent the adaptive modulation factor, τ=2+sin(2πt / T), where t represents the current training round and T represents the total number of training rounds; The edge-aware loss is: ； in, E represents the gradient calculation of the m-th order. canny E represents the structural edge obtained by Canny edge detection. sobel S represents the structural edges extracted by the Sobel operator. pred S represents the segmentation map of the image within the prediction bounding box. gt This indicates the corresponding actual annotation.

Images