Target detection method and system based on SIP and MFWF

Abstract

The invention provides a target detection method and system based on SIP and MFWF in the technical field of target detection, and the method comprises the steps: S1, obtaining a large number of historical aerial images, carrying out the preprocessing of the historical aerial images, constructing a data set, and dividing the data set into a training set, a verification set, and a test set; s2, creating a target detection model based on the backbone network, the neck network and the head network, and setting a loss function of the target detection model; s3, training the target detection model through the training set until the loss value of the loss function is smaller than a preset loss threshold value, verifying the trained target detection model through the verification set, and testing the target detection model passing verification through the test set; and S4, deploying the target detection model passing the test, and performing target detection through the deployed target detection model. The method has the advantage that the small target detection precision is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a target detection method and system based on SIP and MFWF. Background Art

[0002] In recent years, object detection models have been widely used in many fields, especially in drone scenarios, which can monitor and identify objects in different environments. When the shooting angle is high, many objects in the images taken by drones will appear smaller, blurry or difficult to distinguish, and traditional object detection models face a series of challenges and limitations in detecting small objects.

[0003] Spatial information perception technology is widely used to improve target attribute learning and spatial detail expression. It integrates multi-level context information and optimizes region proposals through global normalized attention mechanism or adaptive spatial aggregation technology to further enhance the accuracy of small target detection. By combining context information and pyramid pooling technology, detailed information can be retained more finely. However, these methods usually rely on simple background and foreground data and lack modeling of high-level semantic relationships in complex environments, resulting in insufficient performance of target detection models in occluded and low-resolution scenes.

[0004] Feature fusion technology is also used to improve target detection. For example, gated convolutional neural networks use "gate" structures to optimize feature fusion, and multi-scale feature learning networks focus on abstract semantic information to improve detection performance. However, when dealing with small targets, these feature fusion methods often cannot fully capture subtle features, resulting in a significant decrease in the detection performance of small targets. That is, simple weighted fusion methods cannot effectively balance the contributions of features at different levels, thereby further weakening the ability to identify small targets.

[0005] Therefore, how to provide a target detection method and system based on SIP and MFWF to improve the accuracy of small target detection has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a target detection method and system based on SIP and MFWF to improve the detection accuracy of small targets.

[0007] In a first aspect, the present invention provides a target detection method based on SIP and MFWF, comprising the following steps:

[0008] Step S1, obtaining a large number of historical aerial images containing different target types taken by drones, constructing a data set after preprocessing each of the historical aerial images, and dividing the data set into a training set, a validation set, and a test set;

[0009] Step S2: creating a target detection model based on the backbone network, the neck network and the head network, and setting a loss function of the target detection model;

[0010] The backbone network is constructed by a convolution module, a C2f module, a SIP module and an SPPF module; the convolution module is used to perform a convolution operation on the input aerial image to extract the underlying features including at least edges, textures and colors; the C2f module is used to perform feature conversion, branch processing and feature fusion on each underlying feature to obtain a representation map; the SIP module is used to extract fine-grained features of small targets and complex backgrounds from the representation map through space-to-depth conversion and a large selective convolution kernel mechanism; the SPPF module is used to perform a multi-scale pooling operation on each fine-grained feature to extract local features and global features;

[0011] The neck network is constructed by an upsampling module and an MFWF module; the upsampling module is used to upsample each local feature and global feature to obtain multi-scale sampling features; the MFWF module is used to perform weighted fusion on each sampling feature through a fast normalization fusion mechanism to obtain a fusion feature;

[0012] The head network is constructed by a target detection layer; the target detection layer is used to perform target classification and bounding box regression prediction on the fused features, and output target detection results;

[0013] Step S3, training the target detection model with the training set until the loss value of the loss function is less than a preset loss threshold, continuously optimizing the hyperparameters and optimizer function of the target detection model during the training process, and then verifying the trained target detection model with the verification set, and testing the verified target detection model with the test set;

[0014] Step S4: deploy the target detection model that has passed the test, and perform target detection using the deployed target detection model.

[0015] Furthermore, the step S1 is specifically as follows:

[0016] Acquire a large number of historical aerial images taken by drones from different regions and containing different target types, wherein the target types include at least pedestrians, crowds, cars, trucks, buses, vans, motorcycles, bicycles, covered tricycles, and ordinary tricycles;

[0017] A data set is constructed after labeling the target type and target position of each of the historical aerial images and preprocessing the sample expansion, and the data set is divided into a training set, a validation set and a test set based on a preset segmentation ratio.

[0018] Furthermore, in step S2, the calculation process of the SIP module is:

[0019] Downsample the representation map in space by a multiple of 2 to obtain four representation sub-maps, connect the representation sub-maps along the channel dimension to obtain an intermediate feature map, and perform a convolution operation on the intermediate feature map to obtain the first convolution sub-feature U 1 And the second convolutional sub-feature U 2 , connect the first convolutional sub-feature U 1 And the second convolutional sub-feature U 2 The convolution feature U is obtained, and the average pooling operation and the maximum pooling operation are performed on the convolution feature U to obtain the average pooling spatial feature and the maximum pooling spatial feature:

[0020] SA avg =P avg (U);

[0021] SA max =P max (U);

[0022] Among them, P avg () represents the average pooling function; P max () represents the maximum pooling function; SA avg Represents the average pooling spatial feature; SA max Represents the maximum pooling spatial feature;

[0023] The average pooled spatial features and the maximum pooled spatial features are respectively converted into a first spatial attention map and a second spatial attention map, and a first spatial selection mask SA is respectively extracted from the first spatial attention map and the second spatial attention map through a sigmoid activation function 1 and the second space selection mask SA 2 ;

[0024] Based on the first space, a mask SA is selected 1 , the second space selection mask SA 2 , the first convolution sub-feature U 1 And the second convolutional sub-feature U 2 Compute fused attention features:

[0025] S=Conv 1×1 (SA 1 ·U 1 +SA 2 ·U 2 );

[0026] Among them, S represents the fusion attention feature; Conv 1×1 Represents a convolution kernel of size 1×1;

[0027] Calculate fine-grained features based on the fused attention features and the intermediate feature map:

[0028] Y = S·X';

[0029] Among them, Y represents the fine-grained features; X' represents the intermediate feature map.

[0030] Furthermore, in step S2, the formula of the fast normalization fusion mechanism is:

[0031]

[0032] Among them, X” represents the fusion feature; ω 1 And ω 2 Both represent learnable weights; X 1 and X 2 Sampling characteristics for different contents.

[0033] Furthermore, the step S3 is specifically as follows:

[0034] The target detection model is trained by the training set until the loss value of the loss function is less than a preset loss threshold, and the hyperparameters and optimizer function of the target detection model are continuously optimized during the training process until a hyperparameter combination with the best performance is found; the hyperparameters include at least a learning rate, a batch size, a number of iterations, a convolution kernel size, a pooling window size, and a random dropout rate;

[0035] The trained target detection model is then verified using the verification set to determine whether the detection accuracy of the target detection model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification succeeds, and:

[0036] The verified target detection model is tested using the test set to determine whether the confidence is greater than a preset confidence threshold. If not, the test fails and the training set is expanded to continue training; if so, the test succeeds and the training ends.

[0037] In a second aspect, the present invention provides a target detection system based on SIP and MFWF, comprising the following modules:

[0038] A data set construction module is used to obtain a large number of historical aerial images containing different target types taken by drones, construct a data set after preprocessing each of the historical aerial images, and divide the data set into a training set, a validation set, and a test set;

[0039] A target detection model creation module, used to create a target detection model based on the backbone network, the neck network and the head network, and set the loss function of the target detection model;

[0040] The backbone network is constructed by a convolution module, a C2f module, a SIP module and an SPPF module; the convolution module is used to perform a convolution operation on the input aerial image to extract the underlying features including at least edges, textures and colors; the C2f module is used to perform feature conversion, branch processing and feature fusion on each underlying feature to obtain a representation map; the SIP module is used to extract fine-grained features of small targets and complex backgrounds from the representation map through space-to-depth conversion and a large selective convolution kernel mechanism; the SPPF module is used to perform a multi-scale pooling operation on each fine-grained feature to extract local features and global features;

[0041] The neck network is constructed by an upsampling module and an MFWF module; the upsampling module is used to upsample each local feature and global feature to obtain multi-scale sampling features; the MFWF module is used to perform weighted fusion on each sampling feature through a fast normalization fusion mechanism to obtain a fusion feature;

[0042] The head network is constructed by a target detection layer; the target detection layer is used to perform target classification and bounding box regression prediction on the fused features, and output target detection results;

[0043] The target detection model training module is used to train the target detection model through the training set until the loss value of the loss function is less than a preset loss threshold, continuously optimize the hyperparameters and optimizer function of the target detection model during the training process, and then use the verification set to verify the trained target detection model, and test the verified target detection model through the test set;

[0044] The target detection module is used to deploy the target detection model that has passed the test and perform target detection through the deployed target detection model.

[0045] Furthermore, the data set construction module is specifically used for:

[0046] Acquire a large number of historical aerial images taken by drones from different regions and containing different target types, wherein the target types include at least pedestrians, crowds, cars, trucks, buses, vans, motorcycles, bicycles, covered tricycles, and ordinary tricycles;

[0047] A data set is constructed after labeling the target type and target position of each of the historical aerial images and preprocessing the sample expansion, and the data set is divided into a training set, a validation set and a test set based on a preset segmentation ratio.

[0048] Furthermore, in the target detection model creation module, the calculation process of the SIP module is:

[0049] Downsample the representation map in space by a multiple of 2 to obtain four representation sub-maps, connect the representation sub-maps along the channel dimension to obtain an intermediate feature map, and perform a convolution operation on the intermediate feature map to obtain the first convolution sub-feature U 1 And the second convolutional sub-feature U 2 , connect the first convolutional sub-feature U 1 And the second convolutional sub-feature U 2 The convolution feature U is obtained, and the average pooling operation and the maximum pooling operation are performed on the convolution feature U to obtain the average pooling spatial feature and the maximum pooling spatial feature:

[0050] S avg =P avg (U);

[0051] S max =P max (U);

[0052] Among them, P avg () represents the average pooling function; P max () represents the maximum pooling function; SA avg Represents the average pooling spatial feature; SA max Represents the maximum pooling spatial feature;

[0053] The average pooling spatial features and the maximum pooling spatial features are respectively converted into a first spatial attention map and a second spatial attention map, and a first spatial selection mask SA is respectively extracted from the first spatial attention map and the second spatial attention map through a Sigmoid activation function. 1 and the second space selection mask SA 2 ;

[0054] The mask SA is selected based on the first space 1 , the second space selection mask SA 2 , the first convolution sub-feature U 1 And the second convolutional sub-feature U 2 Compute fused attention features:

[0055] S=Conv 1×1 (SA 1 ·U 1 +SA 2 ·U 2 );

[0056] Among them, S represents the fusion attention feature; Conv 1×1 Represents a convolution kernel of size 1×1;

[0057] Calculate fine-grained features based on the fused attention features and the intermediate feature map:

[0058] Y = S·X';

[0059] Among them, Y represents the fine-grained features; X' represents the intermediate feature map.

[0060] Furthermore, in the target detection model creation module, the formula of the fast normalization fusion mechanism is:

[0061]

[0062] Among them, X” represents the fusion feature; ω 1 And ω 2 Both represent learnable weights; X 1 and X 2 Sampling characteristics for different contents.

[0063] Furthermore, the target detection model training module is specifically used for:

[0064] The target detection model is trained by the training set until the loss value of the loss function is less than a preset loss threshold, and the hyperparameters and optimizer function of the target detection model are continuously optimized during the training process until a hyperparameter combination with the best performance is found; the hyperparameters include at least a learning rate, a batch size, a number of iterations, a convolution kernel size, a pooling window size, and a random dropout rate;

[0065] The trained target detection model is then verified using the verification set to determine whether the detection accuracy of the target detection model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification succeeds, and:

[0066] The verified target detection model is tested using the test set to determine whether the confidence is greater than a preset confidence threshold. If not, the test fails and the training set is expanded to continue training; if so, the test succeeds and the training ends.

[0067] The advantages of the present invention are:

[0068] A data set is constructed by obtaining a large number of historical aerial images of different target types taken by drones and preprocessing them, and the data set is divided into a training set, a validation set and a test set; then a target detection model is created based on the backbone network, the neck network and the head network, and the loss function of the target detection model is set; then the target detection model is trained through the training set until the loss value of the loss function is less than the preset loss threshold, and the hyperparameters and optimizer functions of the target detection model are continuously optimized during the training process, and then the trained target detection model is verified by the validation set, and the verified target detection model is tested by the test set; finally, the target detection model that passes the test is deployed, and target detection is performed through the deployed target detection model; the backbone network of the target detection model is constructed by a convolution module, a C2f module, a SIP module and a SPPF module; the convolution module is used to perform a convolution operation on the input aerial image to extract the underlying features including at least edges, textures and colors; the C2f module is used to perform feature conversion, branch processing and feature fusion on each underlying feature to obtain a representation map; The IP module is used to extract fine-grained features of small targets and complex backgrounds from the representation map through space-to-depth conversion and large selective convolution kernel mechanism; the SPPF module is used to perform multi-scale pooling operations on each fine-grained feature to extract local features and global features; the neck network is constructed by the upsampling module and the MFWF module; the upsampling module is used to upsample each local feature and global feature to obtain multi-scale sampling features; the MFWF module is used to weightedly fuse each sampling feature through a fast normalization fusion mechanism to obtain fusion features; the head network is constructed by the target detection layer; the target detection layer is used to perform target classification and bounding box regression prediction on the fusion features, and output the target detection results; that is, the SIP module retains the fine-grained features of small targets and integrates the contextual information from the complex environment, extracts the feature information related to the target from the surrounding environment, thereby improving the feature semantic representation, and the MFWF module is used to process and fuse the multi-scale sampling features, so as to accurately distinguish the importance of each sampling feature to enhance the feature representation of small targets and retain the detail information of small targets, which ultimately greatly improves the detection accuracy of small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0070] Figure 1 It is a flow chart of a target detection method based on SIP and MFWF of the present invention.

[0071] Figure 2 It is a structural schematic diagram of a target detection system based on SIP and MFWF of the present invention.

[0072] Figure 3It is a structural schematic diagram of the target detection model of the present invention.

[0073] Figure 4 It is a structural diagram of the SIP module of the present invention.

[0074] Figure 5 It is a schematic diagram of the structure of the MFWF module of the present invention. DETAILED DESCRIPTION

[0075] The technical solution in the embodiment of the present application has the following overall idea: target detection is performed through a target detection model created by a backbone network, a neck network and a head network, the backbone network is constructed by a convolution module, a C2f module, a SIP module and an SPPF module; the neck network is constructed by an upsampling module and an MFWF module; the head network is constructed by a target detection layer; the SIP module is used to retain the fine-grained features of small targets and integrate contextual information from complex environments, and to extract feature information related to the target from the surrounding environment, thereby improving the feature semantic representation; the MFWF module is used to process and fuse multi-scale sampling features, thereby accurately distinguishing the importance of each sampling feature to enhance the feature representation of small targets and retain the detail information of small targets, thereby improving the detection accuracy of small targets.

[0076] Please refer to Figures 1 to 5 As shown, a preferred embodiment of a target detection method based on SIP and MFWF of the present invention comprises the following steps:

[0077] Step S1, obtaining a large number of historical aerial images containing different target types taken by drones, constructing a data set after preprocessing each of the historical aerial images, and dividing the data set into a training set, a validation set, and a test set;

[0078] Step S2: creating a target detection model (SF-YOLO) based on the backbone network, the neck network and the head network, and setting the loss function of the target detection model;

[0079] The backbone network is constructed by a convolution module, a C2f module, a SIP module (Spatial Information Perception) and an SPPF module (Spatial Pyramid Pooling-Fast); the convolution module is used to perform a convolution operation on the input aerial image to extract the underlying features including at least edges, textures and colors; the C2f module is used to perform feature conversion, branch processing and feature fusion on each underlying feature to obtain a representation map, that is, to generate an output with more representation capabilities, which helps to improve the performance and representation capabilities of the network, so that the network can better adapt to complex data tasks; the SIP module is used to extract fine-grained features of small targets and complex backgrounds from the representation map through space to depth conversion (Space to Depth) and large selective convolution kernel (Large Selective Kernel) mechanism, that is, the SIP module dynamically adjusts the receptive field of the network by changing the feature representation dimension and a series of dilated convolutions, and can extract clue features of the target from the background information; the SPPF module is used to perform multi-scale pooling operations on each fine-grained feature to extract local features and global features;

[0080] The neck network is constructed by an upsample module (Upsample) and an MFWF module (Multi-scale Feature Weighted Fusion, multi-scale feature weighted fusion module); the upsample module is used to upsample each local feature and global feature to obtain multi-scale sampling features; the MFWF module is used to weightedly fuse each sampling feature through a fast normalized fusion mechanism to obtain a fusion feature, that is, weighted fusion of sampling features at two different levels to effectively fuse target feature information at different levels and improve feature expression capabilities;

[0081] The head network is constructed by a target detection layer (Detect); the target detection layer is used to perform target classification and bounding box regression prediction on the fused features and output the target detection result;

[0082] The target detection model is a model architecture based on a deep convolutional neural network. The SIP module retains the fine-grained features of small targets and complex backgrounds in complex environments. The MFWF module fuses multi-scale sampling features and effectively processes semantic features of different scales, reducing information loss. This allows the target detection model to pay more attention to the background information around the target, thereby improving the accuracy of target detection and having significant advantages in processing small target detection tasks in complex scenes.

[0083] Step S3, training the target detection model with the training set until the loss value of the loss function is less than a preset loss threshold, continuously optimizing the hyperparameters and optimizer function of the target detection model during the training process, and then verifying the trained target detection model with the verification set, and testing the verified target detection model with the test set;

[0084] Step S4: deploy the target detection model that has passed the test, and perform target detection using the deployed target detection model.

[0085] The step S1 is specifically as follows:

[0086] Acquire a large number of historical aerial images taken by drones from different regions and containing different target types, wherein the target types include at least pedestrians, crowds, cars, trucks, buses, vans, motorcycles, bicycles, covered tricycles, and ordinary tricycles;

[0087] A data set is constructed after labeling the target type and target position of each of the historical aerial images and preprocessing the sample expansion, and the data set is divided into a training set, a validation set and a test set based on a preset segmentation ratio.

[0088] For specific implementation, the VisDrone2019 dataset can be selected, which covers a variety of scenes in 14 different cities and includes complex backgrounds (from urban and rural areas).

[0089] In step S2, the calculation process of the SIP module is:

[0090] Downsample the representation map in space by a multiple of 2 to obtain four representation sub-maps, connect the representation sub-maps along the channel dimension to obtain an intermediate feature map, and perform a convolution operation on the intermediate feature map to obtain the first convolution sub-feature U 1 And the second convolutional sub-feature U 2 , connect the first convolutional sub-feature U 1 And the second convolutional sub-feature U 2 The convolution feature U is obtained, and the average pooling operation and the maximum pooling operation are performed on the convolution feature U to obtain the average pooling spatial feature and the maximum pooling spatial feature:

[0091] SA avg =P avg (U);

[0092] SA max =P max (U);

[0093] Among them, P avg () represents the average pooling function; P max() represents the maximum pooling function; SA avg Represents the average pooling spatial feature; SA max Represents the maximum pooling spatial feature;

[0094] The average pooled spatial features and the maximum pooled spatial features are respectively converted into a first spatial attention map and a second spatial attention map, and a first spatial selection mask SA is respectively extracted from the first spatial attention map and the second spatial attention map through a sigmoid activation function 1 and the second space selection mask SA 2 ;

[0095] Based on the first space, a mask SA is selected 1 , the second space selection mask SA 2 , the first convolution sub-feature U 1 And the second convolutional sub-feature U 2 Compute fused attention features:

[0096] S=Conv 1×1 (SA 1 ·U 1 +SA 2 ·U 2 );

[0097] Among them, S represents the fusion attention feature; Conv 1×1 Represents a convolution kernel of size 1×1;

[0098] Calculate fine-grained features based on the fused attention features and the intermediate feature map:

[0099] Y = S·X';

[0100] Among them, Y represents the fine-grained features; X' represents the intermediate feature map.

[0101] In step S2, the formula of the fast normalization fusion mechanism is:

[0102]

[0103] Among them, X” represents the fusion feature; ω 1 And ω 2 Both represent learnable weights; X 1 and X 2 Sampling characteristics for different contents.

[0104] In the specific implementation, the fusion feature is also up-sampled by applying content-aware feature recombination (CARAFE), and the specific process is as follows:

[0105] The channel compressor uses 1×1 convolution to compress the input fusion features; the content encoder uses a kernel size of k encoder The convolutional layer generates a reorganized kernel based on the content of the fused features, k encoder =k up –2, k up represents the reorganization kernel size. Then, the kernel normalizer uses a spatial softmax function for each k up ×k up The kernel is normalized to ensure that the sum of kernel values ​​is 1. Finally, the content-aware feature reconstruction module reconstructs features in local regions using weighted sums. For an object location l′ and its corresponding square region N(X l ,k up ), the reorganization is described as follows:

[0106]

[0107] Among them, X' l' Represents the features after CARAFE upsampling; Represents the radius of the convolution kernel; n and m are used to represent the position of the convolution kernel, that is, the offset of the center position of the convolution kernel; ω l represents the position-aware kernel; i represents the horizontal coordinate of the pixel in the image; j represents the vertical coordinate of the pixel in the image.

[0108] The step S3 is specifically as follows:

[0109] The target detection model is trained by the training set until the loss value of the loss function is less than a preset loss threshold, and the hyperparameters and optimizer function of the target detection model are continuously optimized during the training process until a hyperparameter combination with the best performance is found; the hyperparameters include at least a learning rate, a batch size, a number of iterations, a convolution kernel size, a pooling window size, and a random dropout rate;

[0110] The trained target detection model is then verified using the verification set to determine whether the detection accuracy of the target detection model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification succeeds, and:

[0111] The verified target detection model is tested using the test set to determine whether the confidence is greater than a preset confidence threshold. If not, the test fails and the training set is expanded to continue training; if so, the test succeeds and the training ends.

[0112] A preferred embodiment of a target detection system based on SIP and MFWF of the present invention includes the following modules:

[0113] A data set construction module is used to obtain a large number of historical aerial images containing different target types taken by drones, construct a data set after preprocessing each of the historical aerial images, and divide the data set into a training set, a validation set, and a test set;

[0114] A target detection model creation module, used to create a target detection model (SF-YOLO) based on the backbone network, the neck network and the head network, and set the loss function of the target detection model;

[0115] The backbone network is constructed by a convolution module, a C2f module, a SIP module (Spatial Information Perception) and an SPPF module (Spatial Pyramid Pooling-Fast); the convolution module is used to perform a convolution operation on the input aerial image to extract the underlying features including at least edges, textures and colors; the C2f module is used to perform feature conversion, branch processing and feature fusion on each underlying feature to obtain a representation map, that is, to generate an output with more representation capabilities, which helps to improve the performance and representation capabilities of the network, so that the network can better adapt to complex data tasks; the SIP module is used to extract fine-grained features of small targets and complex backgrounds from the representation map through space to depth conversion (Space to Depth) and large selective convolution kernel (Large Selective Kernel) mechanism, that is, the SIP module dynamically adjusts the receptive field of the network by changing the feature representation dimension and a series of dilated convolutions, and can extract clue features of the target from the background information; the SPPF module is used to perform multi-scale pooling operations on each fine-grained feature to extract local features and global features;

[0116] The neck network is constructed by an upsample module (Upsample) and a MFWF module (Multi-scale Feature Weighted Fusion, multi-scale feature weighted fusion module); the upsample module is used to upsample each local feature and global feature to obtain multi-scale sampling features; the MFWF module is used to perform weighted fusion on each sampling feature through a fast normalized fusion (FastNormalized Fusion) mechanism to obtain a fusion feature, that is, to perform weighted fusion on sampling features of two different levels to effectively fuse target feature information at different levels and improve feature expression capabilities;

[0117] The head network is constructed by a target detection layer (Detect); the target detection layer is used to perform target classification and bounding box regression prediction on the fused features and output the target detection result;

[0118] The target detection model is a model architecture based on a deep convolutional neural network. The SIP module retains the fine-grained features of small targets and complex backgrounds in complex environments. The MFWF module fuses multi-scale sampling features and effectively processes semantic features of different scales, reducing information loss. This allows the target detection model to pay more attention to the background information around the target, thereby improving the accuracy of target detection and having significant advantages in processing small target detection tasks in complex scenes.

[0119] The target detection model training module is used to train the target detection model through the training set until the loss value of the loss function is less than a preset loss threshold, continuously optimize the hyperparameters and optimizer function of the target detection model during the training process, and then use the verification set to verify the trained target detection model, and test the verified target detection model through the test set;

[0120] The target detection module is used to deploy the target detection model that has passed the test and perform target detection through the deployed target detection model.

[0121] The data set construction module is specifically used for:

[0122] Acquire a large number of historical aerial images taken by drones from different regions and containing different target types, wherein the target types include at least pedestrians, crowds, cars, trucks, buses, vans, motorcycles, bicycles, covered tricycles, and ordinary tricycles;

[0123] A data set is constructed after labeling the target type and target position of each of the historical aerial images and preprocessing the sample expansion, and the data set is divided into a training set, a validation set and a test set based on a preset segmentation ratio.

[0124] For specific implementation, the VisDrone2019 dataset can be selected, which covers a variety of scenes in 14 different cities and includes complex backgrounds (from urban and rural areas).

[0125] In the target detection model creation module, the calculation process of the SIP module is:

[0126] Downsample the representation map in space by a multiple of 2 to obtain four representation sub-maps, connect the representation sub-maps along the channel dimension to obtain an intermediate feature map, and perform a convolution operation on the intermediate feature map to obtain the first convolution sub-feature U 1 And the second convolutional sub-feature U 2 , connect the first convolutional sub-feature U 1 And the second convolutional sub-feature U 2The convolution feature U is obtained, and the average pooling operation and the maximum pooling operation are performed on the convolution feature U to obtain the average pooling spatial feature and the maximum pooling spatial feature:

[0127] SA avg =P avg (U);

[0128] SA max =P max (U);

[0129] Among them, P avg () represents the average pooling function; P max () represents the maximum pooling function; SA avg Represents the average pooling spatial feature; SA max Represents the maximum pooling spatial feature;

[0130] The average pooled spatial features and the maximum pooled spatial features are respectively converted into a first spatial attention map and a second spatial attention map, and a first spatial selection mask SA is respectively extracted from the first spatial attention map and the second spatial attention map through a sigmoid activation function 1 and the second space selection mask SA 2 ;

[0131] Based on the first space, a mask SA is selected 1 , the second space selection mask SA 2 , the first convolution sub-feature U 1 And the second convolutional sub-feature U 2 Compute fused attention features:

[0132] S=Conv 1×1 (SA 1 ·U 1 +SA 2 ·U 2 );

[0133] Among them, S represents the fusion attention feature; Conv 1×1 Represents a convolution kernel of size 1×1;

[0134] Calculate fine-grained features based on the fused attention features and the intermediate feature map:

[0135] Y = S·X';

[0136] Among them, Y represents the fine-grained features; X' represents the intermediate feature map.

[0137] In the target detection model creation module, the formula of the fast normalization fusion mechanism is:

[0138]

[0139] Among them, X” represents the fusion feature; ω 1 And ω 2 Both represent learnable weights; X 1 and X 2 Sampling characteristics for different contents.

[0140] In the specific implementation, the fusion feature is also up-sampled by applying content-aware feature recombination (CARAFE), and the specific process is as follows:

[0141] The channel compressor uses 1×1 convolution to compress the input fusion features; the content encoder uses a kernel size of k encoder The convolutional layer generates a reorganized kernel based on the content of the fused features, k encoder =k up –2, k up represents the reorganization kernel size. Then, the kernel normalizer uses a spatial softmax function for each k up ×k up The kernel is normalized to ensure that the sum of kernel values ​​is 1. Finally, the content-aware feature reconstruction module reconstructs features in local regions using weighted sums. For an object location l′ and its corresponding square region N(X l ,k up ), the reorganization is described as follows:

[0142]

[0143] Among them, X' l' Represents the features after CARAFE upsampling; Represents the radius of the convolution kernel; n and m are used to represent the position of the convolution kernel, that is, the offset of the center position of the convolution kernel; ω l represents the position-aware kernel; i represents the horizontal coordinate of the pixel in the image; j represents the vertical coordinate of the pixel in the image.

[0144] The target detection model training module is specifically used for:

[0145] The target detection model is trained by the training set until the loss value of the loss function is less than a preset loss threshold, and the hyperparameters and optimizer function of the target detection model are continuously optimized during the training process until a hyperparameter combination with the best performance is found; the hyperparameters include at least a learning rate, a batch size, a number of iterations, a convolution kernel size, a pooling window size, and a random dropout rate;

[0146] The trained target detection model is then verified using the verification set to determine whether the detection accuracy of the target detection model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification succeeds, and:

[0147] The verified target detection model is tested using the test set to determine whether the confidence is greater than a preset confidence threshold. If not, the test fails and the training set is expanded to continue training; if so, the test succeeds and the training ends.

[0148] In summary, the advantages of the present invention are:

[0149] A data set is constructed by obtaining a large number of historical aerial images of different target types taken by drones and preprocessing them, and the data set is divided into a training set, a validation set and a test set; then a target detection model is created based on the backbone network, the neck network and the head network, and the loss function of the target detection model is set; then the target detection model is trained through the training set until the loss value of the loss function is less than the preset loss threshold, and the hyperparameters and optimizer functions of the target detection model are continuously optimized during the training process, and then the trained target detection model is verified by the validation set, and the verified target detection model is tested by the test set; finally, the target detection model that passes the test is deployed, and target detection is performed through the deployed target detection model; the backbone network of the target detection model is constructed by a convolution module, a C2f module, a SIP module and a SPPF module; the convolution module is used to perform a convolution operation on the input aerial image to extract the underlying features including at least edges, textures and colors; the C2f module is used to perform feature conversion, branch processing and feature fusion on each underlying feature to obtain a representation map; The IP module is used to extract fine-grained features of small targets and complex backgrounds from the representation map through space-to-depth conversion and large selective convolution kernel mechanism; the SPPF module is used to perform multi-scale pooling operations on each fine-grained feature to extract local features and global features; the neck network is constructed by the upsampling module and the MFWF module; the upsampling module is used to upsample each local feature and global feature to obtain multi-scale sampling features; the MFWF module is used to weightedly fuse each sampling feature through a fast normalization fusion mechanism to obtain fusion features; the head network is constructed by the target detection layer; the target detection layer is used to perform target classification and bounding box regression prediction on the fusion features, and output the target detection results; that is, the SIP module retains the fine-grained features of small targets and integrates the contextual information from the complex environment, extracts the feature information related to the target from the surrounding environment, thereby improving the feature semantic representation, and the MFWF module is used to process and fuse the multi-scale sampling features, so as to accurately distinguish the importance of each sampling feature to enhance the feature representation of small targets and retain the detail information of small targets, which ultimately greatly improves the detection accuracy of small targets.

[0150] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A target detection method based on SIP and MFWF, characterized in that: The steps include: Step S1, obtaining a large number of historical aerial images containing different target types taken by drones, constructing a data set after preprocessing each of the historical aerial images, and dividing the data set into a training set, a validation set, and a test set; Step S2: creating a target detection model based on the backbone network, the neck network and the head network, and setting a loss function of the target detection model; The backbone network is constructed by a convolution module, a C2f module, a SIP module and an SPPF module; the convolution module is used to perform a convolution operation on the input aerial image to extract the underlying features including at least edges, textures and colors; the C2f module is used to perform feature conversion, branch processing and feature fusion on each underlying feature to obtain a representation map; the SIP module is used to extract fine-grained features of small targets and complex backgrounds from the representation map through space-to-depth conversion and a large selective convolution kernel mechanism; The SPPF module is used to perform multi-scale pooling operations on each fine-grained feature to extract local features and global features; The neck network is constructed by an upsampling module and an MFWF module; the upsampling module is used to upsample each local feature and global feature to obtain multi-scale sampling features; the MFWF module is used to perform weighted fusion on each sampling feature through a fast normalization fusion mechanism to obtain a fusion feature; The head network is constructed by a target detection layer; the target detection layer is used to perform target classification and bounding box regression prediction on the fused features, and output target detection results; Step S3, training the target detection model with the training set until the loss value of the loss function is less than a preset loss threshold, continuously optimizing the hyperparameters and optimizer function of the target detection model during the training process, and then verifying the trained target detection model with the verification set, and testing the verified target detection model with the test set; Step S4: deploy the target detection model that has passed the test, and perform target detection using the deployed target detection model.

2. The target detection method based on SIP and MFWF as claimed in claim 1, characterized in that: The step S1 is specifically as follows: Acquire a large number of historical aerial images taken by drones from different regions and containing different target types, wherein the target types include at least pedestrians, crowds, cars, trucks, buses, vans, motorcycles, bicycles, covered tricycles, and ordinary tricycles; A data set is constructed after labeling the target type and target position of each of the historical aerial images and preprocessing the sample expansion, and the data set is divided into a training set, a validation set and a test set based on a preset segmentation ratio.

3. The target detection method based on SIP and MFWF as claimed in claim 1, characterized in that: In step S2, the calculation process of the SIP module is: Downsample the representation map in space by a multiple of 2 to obtain four representation sub-maps, connect the representation sub-maps along the channel dimension to obtain an intermediate feature map, perform a convolution operation on the intermediate feature map to obtain a first convolution sub-feature U1 and a second convolution sub-feature U2, connect the first convolution sub-feature U1 and the second convolution sub-feature U2 to obtain a convolution feature U, perform average pooling and maximum pooling operations on the convolution feature U, respectively, to obtain an average pooling spatial feature and a maximum pooling spatial feature: SA avg =P avg (U); SA max =P max (U); Among them, P avg () represents the average pooling function; P max () represents the maximum pooling function; SA avg Represents the average pooling spatial feature; SA max Represents the maximum pooling spatial feature; The average pooled spatial features and the maximum pooled spatial features are converted into a first spatial attention map and a second spatial attention map, respectively, and a first spatial selection mask SA1 and a second spatial selection mask SA2 are extracted from the first spatial attention map and the second spatial attention map respectively through a Sigmoid activation function; The fused attention feature is calculated based on the first spatial selection mask SA1, the second spatial selection mask SA2, the first convolution sub-feature U1, and the second convolution sub-feature U2: S=Conv 1×1 (SA1·U1+SA2·U2); Among them, S represents the fusion attention feature; Conv 1×1 Represents a convolution kernel of size 1×1; Calculate fine-grained features based on the fused attention features and the intermediate feature map: Y = S·X'; Among them, Y represents the fine-grained features; X' represents the intermediate feature map.

4. The target detection method based on SIP and MFWF as claimed in claim 1, characterized in that: In step S2, the formula of the fast normalization fusion mechanism is: Among them, "X" represents the fusion feature; ω1 and ω2 both represent learnable weights; X1 and X2 are sampling features with different contents.

5. The target detection method based on SIP and MFWF as claimed in claim 1, characterized in that: The step S3 is specifically as follows: The target detection model is trained by the training set until the loss value of the loss function is less than a preset loss threshold, and the hyperparameters and optimizer function of the target detection model are continuously optimized during the training process until a hyperparameter combination with the best performance is found; the hyperparameters include at least a learning rate, a batch size, a number of iterations, a convolution kernel size, a pooling window size, and a random dropout rate; The trained target detection model is then verified using the verification set to determine whether the detection accuracy of the target detection model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification succeeds, and: The verified target detection model is tested using the test set to determine whether the confidence is greater than a preset confidence threshold. If not, the test fails and the training set is expanded to continue training; if so, the test succeeds and the training ends.

6. A target detection system based on SIP and MFWF, characterized in that: Includes the following modules: A data set construction module is used to obtain a large number of historical aerial images containing different target types taken by drones, construct a data set after preprocessing each of the historical aerial images, and divide the data set into a training set, a validation set, and a test set; A target detection model creation module, used to create a target detection model based on the backbone network, the neck network and the head network, and set the loss function of the target detection model; The backbone network is constructed by a convolution module, a C2f module, a SIP module and an SPPF module; the convolution module is used to perform a convolution operation on the input aerial image to extract the underlying features including at least edges, textures and colors; the C2f module is used to perform feature conversion, branch processing and feature fusion on each underlying feature to obtain a representation map; the SIP module is used to extract fine-grained features of small targets and complex backgrounds from the representation map through space-to-depth conversion and a large selective convolution kernel mechanism; The SPPF module is used to perform multi-scale pooling operations on each fine-grained feature to extract local features and global features; The neck network is constructed by an upsampling module and an MFWF module; the upsampling module is used to upsample each local feature and global feature to obtain multi-scale sampling features; the MFWF module is used to perform weighted fusion on each sampling feature through a fast normalization fusion mechanism to obtain a fusion feature; The head network is constructed by a target detection layer; the target detection layer is used to perform target classification and bounding box regression prediction on the fused features, and output target detection results; The target detection model training module is used to train the target detection model through the training set until the loss value of the loss function is less than a preset loss threshold, continuously optimize the hyperparameters and optimizer function of the target detection model during the training process, and then use the verification set to verify the trained target detection model, and test the verified target detection model through the test set; The target detection module is used to deploy the target detection model that has passed the test and perform target detection through the deployed target detection model.

7. The target detection system based on SIP and MFWF as claimed in claim 6, characterized in that: The data set construction module is specifically used for: Acquire a large number of historical aerial images taken by drones from different regions and containing different target types, wherein the target types include at least pedestrians, crowds, cars, trucks, buses, vans, motorcycles, bicycles, covered tricycles, and ordinary tricycles; A data set is constructed after labeling the target type and target position of each of the historical aerial images and preprocessing the sample expansion, and the data set is divided into a training set, a validation set and a test set based on a preset segmentation ratio.

8. The target detection system based on SIP and MFWF as claimed in claim 6, characterized in that: In the target detection model creation module, the calculation process of the SIP module is: Downsample the representation map in space by a multiple of 2 to obtain four representation sub-maps, connect the representation sub-maps along the channel dimension to obtain an intermediate feature map, perform a convolution operation on the intermediate feature map to obtain a first convolution sub-feature U1 and a second convolution sub-feature U2, connect the first convolution sub-feature U1 and the second convolution sub-feature U2 to obtain a convolution feature U, perform average pooling and maximum pooling operations on the convolution feature U, respectively, to obtain an average pooling spatial feature and a maximum pooling spatial feature: SA avg =P avg (U); SA max =P max (U); Among them, P avg () represents the average pooling function; P max () represents the maximum pooling function; SA avg Represents the average pooling spatial feature; SA max Represents the maximum pooling spatial feature; The average pooled spatial features and the maximum pooled spatial features are converted into a first spatial attention map and a second spatial attention map, respectively, and a first spatial selection mask SA1 and a second spatial selection mask SA2 are extracted from the first spatial attention map and the second spatial attention map respectively through a Sigmoid activation function; The fused attention feature is calculated based on the first spatial selection mask SA1, the second spatial selection mask SA2, the first convolution sub-feature U1, and the second convolution sub-feature U2: S=Conv 1×1 (SA1·U1+SA2·U2); Among them, S represents the fusion attention feature; Conv 1×1 Represents a convolution kernel of size 1×1; Calculate fine-grained features based on the fused attention features and the intermediate feature map: Y = S·X'; Among them, Y represents the fine-grained features; X' represents the intermediate feature map.

9. The target detection system based on SIP and MFWF as claimed in claim 6, characterized in that: In the target detection model creation module, the formula of the fast normalization fusion mechanism is: Among them, "X" represents the fusion feature; ω1 and ω2 both represent learnable weights; X1 and X2 are sampling features with different contents.

10. The target detection system based on SIP and MFWF as claimed in claim 6, characterized in that: The target detection model training module is specifically used for: The target detection model is trained by the training set until the loss value of the loss function is less than a preset loss threshold, and the hyperparameters and optimizer function of the target detection model are continuously optimized during the training process until a hyperparameter combination with the best performance is found; the hyperparameters include at least a learning rate, a batch size, a number of iterations, a convolution kernel size, a pooling window size, and a random dropout rate; The trained target detection model is then verified using the verification set to determine whether the detection accuracy of the target detection model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification succeeds, and: The verified target detection model is tested by the test set to determine whether the confidence is greater than a preset confidence threshold. If not, the test fails and the training set is expanded to continue training; If yes, the test is successful and the training ends.

Images