Vehicle real-time detection method and system for unmanned aerial vehicle remote sensing multi-spectral image

Abstract

The present application belongs to the technical field of image recognition, and discloses a vehicle real-time detection method and system for unmanned aerial vehicle remote sensing multispectral images. Multi-band images are acquired and segmented to obtain multiple segmented sub-images, the multiple segmented sub-images are super-resolution reconstructed to obtain reconstructed images, vehicle targets of the reconstructed images are manually labeled to obtain a vehicle dataset; the obtained vehicle dataset is divided into a training group and a test group; the training group is input into an initial YOLOV5 image recognition model for training to obtain a target YOLOV5 image recognition model. The present application corrects initial anchor box parameters to adapt to the size of vehicle targets in unmanned aerial vehicle remote sensing multispectral images; a lightweight attention mechanism network is added to the backbone network to improve useful features of vehicle targets and suppress irrelevant features; in the detection network, a WBF algorithm is used to improve the inaccuracy of counting.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, and in particular relates to a method and system for real-time vehicle detection using UAV remote sensing multispectral images. Background Technology

[0002] With the rapid development of economy and technology, the number of vehicles is gradually increasing, and traffic congestion has become a major problem in the field of highway transportation. How to achieve real-time and accurate vehicle detection and solve traffic congestion has become an urgent issue. Due to the flexibility, efficiency, and high mobility of drones, and the fact that 4-band remote sensing multispectral imagery contains richer image information than RGB imagery, using drone-captured remote sensing multispectral imagery for vehicle detection has significant advantages in timely grasping traffic information and understanding vehicle distribution, which is helpful for traffic flow management, scheduling, and the investigation and prediction of traffic congestion.

[0003] The vehicle detection using UAV remote sensing multispectral images has the following technical defects: (1) The resolution of the aerial images taken by the UAV imaging equipment is low, which cannot meet the actual needs of vehicle detection; (2) There are small targets in the UAV remote sensing multispectral images, and the anchor frame size set according to experience cannot match them, making them difficult to detect, resulting in missed detection and false detection; (3) The detection model structure is relatively complex. When detecting many categories, the computational complexity is high, the detection speed is slow, real-time detection is not possible, and the requirements for hardware equipment are high.

[0004] Furthermore, vehicle detection methods based on remote sensing images can be divided into three categories: vehicle detection based on spectral or geometric features, traditional machine learning methods, and deep learning methods. (1) Common methods for vehicle detection based on spectral or geometric features include threshold segmentation and gradient comparison. (2) Traditional machine learning methods detect vehicle targets using methods such as Histogram of Oriented Gradient (HOG). (3) Vehicle detection algorithms based on deep learning can be divided into two categories: one is vehicle detection algorithms based on region extraction, with representative algorithms including R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN. The other is vehicle detection algorithms based on end-to-end, with representative algorithms including YOLO (You Only Look Once) and SSD (Single Shot MultiBoxDetector).

[0005] The existing technologies have the following technical shortcomings: Both existing methods primarily target static images. Vehicle detection based on spectral or geometric features suffers from poor accuracy and weak robustness. Traditional machine learning methods largely rely on manual feature extraction, which may not be suitable for diverse variations and performs poorly under occlusion conditions, resulting in numerous false negatives and missed detections, and thus poor robustness and universality. While existing deep learning methods can detect both video and static images, region-based detection algorithms cannot meet real-time requirements; end-to-end detection algorithms also exhibit low accuracy. Summary of the Invention

[0006] To overcome the problems existing in related technologies, the present invention discloses a method and system for real-time vehicle detection based on UAV remote sensing multispectral imagery; in particular, it relates to a method for real-time vehicle detection based on UAV remote sensing multispectral imagery using super-resolution reconstruction.

[0007] The technical solution is as follows: The real-time vehicle detection method based on super-resolution reconstructed UAV remote sensing multispectral imagery includes the following steps:

[0008] S1. Acquire multi-band images and segment them to obtain multiple segmented sub-images. Perform super-resolution reconstruction on the multiple segmented sub-images and obtain the reconstructed images. Label the vehicle targets in the reconstructed images to obtain the vehicle dataset.

[0009] S2. Divide the obtained vehicle dataset into a training group and a test group;

[0010] S3. Input the training group into the initial YOLOv5 image recognition model for training to obtain the target YOLOv5 image recognition model;

[0011] S4. Input the test group into the target YOLOv5 image recognition model to test the target YOLOv5 image recognition model's ability to recognize the target vehicle and perform real-time vehicle detection.

[0012] In step S1, the multi-band image is the image obtained by merging four images of the scene to be identified.

[0013] In one embodiment, the four images include: a near-infrared monospectral image, a red monospectral image, a blue monospectral image, and a green monospectral image.

[0014] In step S1, multiple segmented sub-images are super-resolution reconstructed to obtain the reconstructed image, specifically including the following steps:

[0015] S1-1, using two 3×3 convolutional layers, shallow feature extraction is performed on the segmented sub-image to obtain a shallow feature map;

[0016] S1-2, The shallow feature map is segmented and then recombined to obtain the deep feature map;

[0017] S1-3, perform feature aggregation on the shallow feature map and the deep feature map to obtain an aggregated feature map;

[0018] S1-4, the aggregated feature map is upsampled using a 3×3 deconvolution layer, and the aggregated feature map is depth-mapped using 3×3×64 and 3×3×1 convolution kernels to obtain the reconstructed image.

[0019] In step S3, the training set is input into the initial YOLOv5 image recognition model for training to obtain the target YOLOv5 image recognition model, including the following steps:

[0020] S3-1, The aspect ratio of manually labeled anchor boxes in several reconstructed images in the training group is calculated using the K-means algorithm. The anchor box parameters in the initial YOLOV5 image recognition model are then corrected to obtain the YOLOV5 image recognition model with corrected anchor box parameters.

[0021] S3-2, Input the training group into the YOLOV5 image recognition model after anchor frame parameter correction, train the YOLOV5 image recognition model after anchor frame parameter correction, extract the vehicle target feature information of several reconstructed images in the training group and fuse them;

[0022] S3-3: Anchor boxes are marked on each vehicle target based on the fused feature information. By comparing the anchor boxes with those marked manually, the difference between the manually marked anchor boxes and the model-marked anchor boxes is calculated. Then, the network parameters are updated in reverse and iterated to obtain the target YOLOv5 image recognition model.

[0023] In step S3-2, the extraction and fusion of vehicle target feature information from several reconstructed images in the training group specifically includes the following steps:

[0024] (a) Assign weights to various features of each target vehicle in the training group according to their importance;

[0025] (b) The various feature information is weighted by multiplication according to the weighted weights to enhance the useful features of each vehicle target and suppress the irrelevant features of each vehicle target; wherein the useful features include category features and size features, and the irrelevant features include color features and orientation features;

[0026] (c) Extract the feature information of each vehicle target.

[0027] In step S4, the ability of the target YOLOv5 image recognition model to recognize the target vehicle is tested, specifically including the following steps:

[0028] On vehicle targets in several reconstructed images of the test group, several first prediction boxes containing location and score are generated. The weighted box fusion (WBF) algorithm is used to merge the several first prediction boxes on the vehicle targets to obtain the final prediction result. Then, a unique second prediction box is generated on the vehicle targets in the several reconstructed images of the test group.

[0029] In one embodiment, generating a unique second prediction box includes:

[0030] The first prediction boxes containing position and score on each target vehicle are calculated using formulas (1) and (2), and the horizontal and vertical coordinates of each vertex of the second prediction box are merged.

[0031]

[0032]

[0033] Where M represents the number of a target vehicle among the X target vehicles in the reconstructed images of the test group; Y represents the number of a first predicted bounding box among the Mth target vehicle in the reconstructed images of the test group; A Mj B represents the x-coordinate of the j-th vertex of the second predicted bounding box of the M-th target vehicle in several reconstructed images of the test group; Mj A represents the ordinate of the j-th vertex of the second predicted bounding box of the M-th target vehicle in several reconstructed images of the test group; Mji B represents the x-coordinate of the j-th vertex of the i-th first predicted bounding box of the M-th target in the reconstructed images of the test group; Mji C represents the ordinate of the j-th vertex of the i-th first predicted bounding box of the M-th target object in the reconstructed images of the test group; Mi This represents the score of the i-th first prediction box of the M-th target object in several reconstructed images of the test group.

[0034] In step S4, the real-time detection of the vehicle specifically includes the following steps:

[0035] Step 1: Acquire four-band imagery;

[0036] Step 2: Segment the image;

[0037] Step 3: Perform super-resolution reconstruction on the segmented image;

[0038] Step 4: Manually annotate the reconstructed images with anchor boxes to create a dataset;

[0039] Step 5: Divide the dataset into training and testing groups;

[0040] Step 6: Calculate the dimensions of the manually labeled anchor boxes in the training group using k-means;

[0041] Step 7: Input the training group into the model after anchor frame parameter correction to obtain the anchor frames marked on each vehicle target by the model after anchor frame parameter correction.

[0042] Step 8: Calculate the difference between the manually annotated anchor boxes and the model annotation boxes;

[0043] Step 9: Update the network parameters in the model after the anchor frame parameter correction based on the difference between the two anchor frames.

[0044] Step 10: Detect the target vehicle in the test group image.

[0045] Another objective of this invention is to provide a real-time vehicle detection system based on super-resolution reconstructed UAV remote sensing multispectral imagery, which implements the aforementioned real-time vehicle detection method based on super-resolution reconstructed UAV remote sensing multispectral imagery. The real-time vehicle detection system based on super-resolution reconstructed UAV remote sensing multispectral imagery includes:

[0046] Image acquisition module, used to acquire four-band images;

[0047] The image segmentation module is used to segment images;

[0048] The super-resolution reconstruction module is used to perform super-resolution reconstruction on the segmented image.

[0049] The manually labeled anchor box module is used to create a dataset from the reconstructed images;

[0050] The training and test grouping module is used to divide the dataset into training and test groups;

[0051] The module for calculating the size of manually labeled anchor boxes is used to calculate the size of manually labeled anchor boxes in the training group using k-means.

[0052] The model bounding box acquisition module is used to input the training group into the model after anchor box parameter correction, and obtain the anchor boxes marked on each vehicle target by the model after anchor box parameter correction.

[0053] The anchor frame comparison module is used to calculate the difference between manually annotated anchor frames and model annotation frames;

[0054] The parameter acquisition module is used to update the network parameters in the model after the anchor frame parameter correction based on the difference between the two anchor frames.

[0055] The vehicle detection module is used to detect target vehicles in the test group images.

[0056] Another object of the present invention is to provide a user input program storage medium, wherein the stored computer program enables an electronic device to execute the aforementioned real-time vehicle detection method based on super-resolution reconstructed UAV remote sensing multispectral imagery.

[0057] Combining all the above technical solutions, the advantages and positive effects of this invention are as follows:

[0058] First, in view of the technical problems existing in the prior art and the difficulty of solving these problems, and closely combining the technical solution to be protected by this invention with the results and data during the research and development process, this paper analyzes in detail how the technical solution of this invention solves the technical problems, and the inventive technical effects brought about after solving the problems, as described in detail below:

[0059] (1) This invention provides a real-time vehicle detection method based on UAV remote sensing multispectral images with super-resolution reconstruction. It can solve the problems of low resolution of UAV remote sensing multispectral images and difficulty in detecting small vehicle targets by traditional target detection methods. It can also solve the problems of low detection accuracy and slow speed in traditional algorithms.

[0060] (2) The present invention modifies the initial anchor frame parameters to adapt them to the size of vehicle targets in UAV remote sensing multispectral images; adds a lightweight attention mechanism network to the backbone network to enhance the useful features of vehicle targets and suppress their irrelevant features; and uses the WBF algorithm in the detection network to improve the situation of inaccurate counting.

[0061] (3) This invention improves the resolution of UAV remote sensing multispectral images through super-resolution reconstruction, enhancing the texture information and features of smaller vehicle targets. Based on this, it provides a method for real-time and accurate detection and identification of vehicle targets.

[0062] (4) This invention provides a real-time vehicle detection method based on super-resolution reconstruction of UAV remote sensing multispectral imagery. Super-resolution reconstruction technology improves the resolution of four-band UAV remote sensing multispectral imagery, enhancing the texture details and features of smaller vehicle targets. In the vehicle detection algorithm stage, this enables the algorithm to learn useful information about vehicle targets more quickly and accurately; moreover, the algorithm has a lightweight network structure, offering a significant advantage in detection speed. Compared with existing technologies, this technology can detect vehicle targets accurately in real time.

[0063] Secondly, considering the technical solution as a whole or from the perspective of the product, the technical effects and advantages of the technical solution to be protected by this invention are specifically described as follows:

[0064] This invention first performs super-resolution reconstruction on four-band (blue, green, red, and near-red) UAV remote sensing multispectral images, and then feeds the reconstructed high-resolution images into a vehicle detection algorithm for vehicle detection. In the super-resolution reconstruction model, this invention adds a convolutional layer to the shallow feature extraction layer to improve its feature learning ability and obtain better feature extraction results. In the image reconstruction module, this invention replaces the previous layer with a deconvolution layer for image reconstruction, improving the network's learning ability and reducing model training time. In the vehicle detection model, this invention first corrects the anchor box size to adapt it to the size of vehicle targets in the UAV remote sensing multispectral images; a lightweight attention mechanism network is added to the backbone network to enhance the useful features of vehicle targets and suppress their irrelevant features; in the detection network, the WBF algorithm is used to improve the situation of inaccurate counting. Attached Figure Description

[0065] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure;

[0066] Figure 1 This is a flowchart of a real-time vehicle detection method based on super-resolution reconstruction of UAV remote sensing multispectral images provided in an embodiment of the present invention.

[0067] Figure 2 This is a flowchart of the process of super-resolution reconstruction of the multiple segmented sub-images and obtaining the reconstructed image provided in the embodiments of the present invention;

[0068] Figure 3 This is a flowchart of the process of inputting the training group into the image recognition model for training to obtain the target YOLOv5 image recognition model, provided by an embodiment of the present invention.

[0069] Figure 4 This is a flowchart of the process for extracting multiple feature information of each vehicle target in several reconstructed images of the training group, provided by an embodiment of the present invention.

[0070] Figure 5 This is a flowchart provided by an embodiment of the present invention, showing how to input the test group into the target YOLOv5 image recognition model to test the target YOLOv5 image recognition model's ability to recognize the target vehicle;

[0071] Figure 6 This is a schematic diagram of the real-time vehicle detection method based on super-resolution reconstruction of UAV remote sensing multispectral images provided in this embodiment of the invention.

[0072] Figure 7(a) is a flowchart of the UAV remote sensing multispectral image super-resolution reconstruction provided in an embodiment of the present invention;

[0073] Figure 7(b) is a flowchart of RSTB in super-resolution reconstruction of UAV remote sensing multispectral images provided in an embodiment of the present invention;

[0074] Figure 7(c) is a flowchart of the STL process for super-resolution reconstruction of UAV remote sensing multispectral images provided in an embodiment of the present invention;

[0075] Figure 8 This is a schematic diagram of the vehicle detection algorithm for UAV remote sensing multispectral imagery provided in this embodiment of the invention;

[0076] Figure 9 This is a diagram of the Squeeze-and-Excitation (SE) network structure provided in this embodiment of the invention;

[0077] Figure 10 This is a schematic diagram of the weighted box fusion (WBF) principle provided in an embodiment of the present invention;

[0078] Figure 11 This is a schematic diagram of the calculation IOU provided in an embodiment of the present invention;

[0079] Figure 12 This is a schematic diagram of a real-time vehicle detection system based on super-resolution reconstruction of UAV remote sensing multispectral images provided in this embodiment;

[0080] In the diagram: 1. Image acquisition module; 2. Image segmentation module; 3. Super-resolution reconstruction module; 4. Manually labeled anchor box module; 5. Training and test group grouping module; 6. Manually labeled anchor box size calculation module; 7. Model box acquisition module; 8. Anchor box comparison module; 9. Parameter acquisition module; 10. Vehicle detection module. Detailed Implementation

[0081] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0082] I. Explanation of the Implementation Example:

[0083] like Figure 1 As shown in the figure, this embodiment provides a real-time vehicle detection method based on super-resolution reconstructed UAV remote sensing multispectral imagery, which includes the following steps:

[0084] S1. Acquire multi-band images and segment them to obtain multiple segmented sub-images. Perform super-resolution reconstruction on the multiple segmented sub-images and obtain the reconstructed images. Manually label the vehicle targets in the reconstructed images to obtain the vehicle dataset.

[0085] S2. Divide the obtained vehicle dataset into a training group and a test group;

[0086] S3. Input the training group into the initial YOLOv5 image recognition model for training to obtain the target YOLOv5 image recognition model;

[0087] S4. Input the test group into the target YOLOv5 image recognition model to test the target YOLOv5 image recognition model's ability to recognize the target vehicle and perform real-time vehicle detection.

[0088] In this embodiment of the invention, in step S1, the multi-band image is a merged image of four images of the scene to be identified, and the four images include monospectral images of near-infrared band, red band, blue band and green band.

[0089] This scheme requires a designated image acquisition device to acquire monochromatic spectral images of the scene to be identified in the near-infrared, red, green, and blue bands. Then, the SIFT algorithm is used to extract image features from these monochromatic spectral images. These monochromatic spectral images are then registered, and the spatial information contained in the extracted image features is used to synthesize a multi-band image. Because the multi-band image includes a near-infrared monochromatic spectral image, the initial YOLOv5 image recognition model can learn more information about the target objects in the image during training using the multi-band image, resulting in better model training performance. Once the trained model is deployed in traffic control and dispatching, it can effectively alleviate traffic congestion.

[0090] In step S1 of this scheme, the target YOLOv5 image recognition model adopts the improved YOLOv5 model. The improvement of the YOLOv5 model lies in the correction of the anchor box size, the addition of the attention mechanism SE network, and the replacement of the non-maximum suppression NMS algorithm with the weighted box fusion WBF algorithm.

[0091] In embodiments of the present invention, such as Figure 2 As shown, in step S1, performing super-resolution reconstruction on the multiple segmented sub-images to obtain the reconstructed image includes the following steps:

[0092] S1-1, shallow feature extraction is performed on the segmented sub-image to obtain a shallow feature map;

[0093] S1-2, The shallow feature map is segmented and then recombined to obtain the deep feature map;

[0094] S1-3, perform feature aggregation on the shallow feature map and the deep feature map to obtain an aggregated feature map;

[0095] S1-4, Upsample and depth map the aggregated feature map to obtain the reconstructed image.

[0096] Furthermore, in step S1-1, when performing shallow feature extraction on the segmented sub-image, two 3×3 convolutional layers are used to perform shallow feature extraction on the segmented sub-image to obtain the shallow feature map.

[0097] Because the super-resolution reconstruction model typically uses only one 3×3 convolutional layer to extract shallow features from the segmented sub-images, it cannot extract more shallow features. Furthermore, using more convolutional layers results in longer processing times. Therefore, this solution uses two 3×3 convolutional layers to extract shallow features from the segmented sub-images, which can extract more shallow features while ensuring model processing efficiency.

[0098] Furthermore, in steps S1-4, the aggregated feature map is upsampled and depth-mapped, including upsampling the aggregated feature map with a 3×3 deconvolution layer and depth-mapping the aggregated feature map with 3×3×64 and 3×3×1 convolution kernels.

[0099] In this scheme, the aggregated feature map is upsampled using a 3×3 deconvolution layer, and depth mapping is performed on the aggregated feature map using 3×3×64 and 3×3×1 convolution kernels. This allows for a more thorough restoration of the detailed features of the image in the aggregated feature map, resulting in a reconstructed image with higher clarity. Thus, when training the initial YOLOv5 image recognition model, a target YOLOv5 image recognition model with stronger object recognition capabilities can be obtained.

[0100] Furthermore, the deep feature map obtained after segmenting and recombining the shallow feature map contains the deep features of the target object in the segmented sub-image. The deep features are then aggregated with the shallow features to obtain the aggregated feature map of the segmented sub-image. In the aggregated feature map, various features of the target object are enhanced. After upsampling and depth mapping of the aggregated feature map, the reconstructed image is obtained. Therefore, when the reconstructed image is used to train the initial YOLOv5 image recognition model, the training effect is better, and the obtained target YOLOv5 image recognition model has a stronger ability to recognize the target object.

[0101] In embodiments of the present invention, such as Figure 3 As shown, in step S3, the training set is input into the initial YOLOv5 image recognition model for training to obtain the target YOLOv5 image recognition model, including the following steps:

[0102] S3-1, Input the training group into the image recognition model, use the K-means algorithm to calculate the aspect ratio of the manually labeled anchor boxes of several reconstructed images in the training group, and correct the anchor box parameters in the image recognition model;

[0103] S3-2, The training group is input into the image recognition model after the anchor frame parameter is corrected, and the image recognition model after the anchor frame parameter is corrected is trained. Multiple feature information of each vehicle target in several reconstructed images of the training group is extracted, and the feature information of each target is fused.

[0104] S3-3, the image recognition model after anchor frame parameter correction marks anchor frames on each vehicle target according to the fused feature information. By comparing with manually marked anchor frames, the difference between manually marked anchor frames and model marked anchor frames is calculated. Then, the network parameters are updated in reverse and iterated to obtain the target YOLOv5 image recognition model.

[0105] In this scheme, by manually annotating the anchor boxes in the reconstructed images of the training group, the target objects in the reconstructed images can be accurately located. Then, the feature map obtained by feature extraction and feature fusion of the target objects will contain more obvious features for identifying the target objects. When inputting it into the image recognition model for training, it will be more targeted, and thus a target YOLOv5 image recognition model with stronger target object recognition ability can be obtained.

[0106] Furthermore, such as Figure 4 As shown, in step S3-2, the extraction of various feature information of each vehicle target in several reconstructed images of the training group includes the following steps:

[0107] (a) Assign weights to various features of each target vehicle in the training group according to their importance;

[0108] (b) The weights of the various feature information after being weighted as described above are multiplied to enhance the useful features of each vehicle target and suppress the irrelevant features of each vehicle target.

[0109] (c) Extract the feature information of each vehicle target.

[0110] This scheme assigns weights to various features of each target object in the training process according to their importance. This allows for the assignment of higher weights to more obvious features that are easier to identify accurately, and lower weights to less obvious features that are difficult to identify accurately. After weighting the various feature information according to the above weights, the proportion of obvious features in the resulting aggregated feature map is more prominent, which is beneficial for subsequent image recognition model training and results in a target YOLOv5 image recognition model with stronger recognition capabilities.

[0111] In one embodiment, such as Figure 5 As shown, in step S4, the ability of the target YOLOv5 image recognition model to recognize the target vehicle is tested, including the following steps:

[0112] The test group is input into the target YOLOV5 image recognition model, and several first prediction boxes containing position and score are generated on the vehicle targets in several reconstructed images of the test group.

[0113] The weighted bounding box fusion (WBF) algorithm is used to merge several first predicted bounding boxes on the vehicle target to obtain the final prediction result, and then a unique second predicted bounding box is generated on the vehicle target in several reconstructed images of the test group.

[0114] In this scheme, when marking first predicted bounding boxes on each target object in the reconstructed image, multiple first predicted bounding boxes are drawn on the same target object. The YOLOv5 image recognition model with anchor box parameter correction filters out the first predicted bounding boxes with low scores and extracts the first predicted bounding boxes with high scores, causing inaccurate counting of second predicted bounding boxes during vehicle detection. Therefore, the target YOLOv5 image recognition model, by merging the first predicted bounding boxes on the same target object, obtains second predicted bounding boxes that can more accurately define the location of the target object, improving the situation of inaccurate counting of second predicted bounding boxes. Furthermore, after inputting the test group dataset into the target YOLOv5 image recognition model, a more accurate target YOLOv5 image recognition model can be obtained, thereby improving its ability to recognize target objects.

[0115] Furthermore, generating a unique second prediction box includes:

[0116] The horizontal and vertical coordinates of each vertex of the second prediction box after merging the first prediction boxes containing position and score on each target vehicle are calculated using formulas (1) and (2).

[0117]

[0118]

[0119] Where M represents the number of a target vehicle among the X target vehicles in the several reconstructed images of the test group, and Y represents the number of a first prediction box among the Mth target vehicle in the several reconstructed images of the test group.

[0120] Where A Mj The x-coordinate of the j-th vertex of the second prediction box of the M-th target vehicle in the several reconstructed images of the test group is given.

[0121] B Mj The value represents the ordinate of the j-th vertex of the second prediction box of the M-th target vehicle in the several reconstructed images of the test group;

[0122] A Mji The x-coordinate of the j-th vertex of the i-th first prediction box of the M-th target in the reconstructed images of the test group is given.

[0123] B Mji The value represents the ordinate of the j-th vertex of the i-th first prediction box of the M-th target in the reconstructed images of the test group;

[0124] C Mi This represents the score of the i-th first prediction box of the M-th target object in several reconstructed images of the test group.

[0125] As can be seen from the coordinate formula of the second predicted box of the Mth target in the reconstructed image, the coordinates of the second predicted box are obtained by weighted summation of the coordinates of each of the first predicted boxes on the target and their corresponding scores. Therefore, the box with the higher score of each of the first predicted boxes has a greater weight when generating the second predicted box. Thus, after merging each of the first predicted boxes according to the above method, the second predicted box obtained is only one box on the target, which improves the phenomenon of multiple predicted boxes appearing on a target and further enhances the target YOLOv5 image recognition model's ability to recognize targets.

[0126] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0127] The information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments.

[0129] II. Application Examples:

[0130] Application Example 1

[0131] like Figure 6 As shown, this embodiment of the invention provides a real-time vehicle detection method based on super-resolution reconstruction of UAV remote sensing multispectral imagery. Specifically, it includes the following steps:

[0132] Step 1: Acquisition and preprocessing of UAV remote sensing multispectral images.

[0133] Furthermore, in step 1, a drone equipped with a multispectral camera is used to collect drone remote sensing multispectral images containing vehicles.

[0134] Furthermore, in step 1, the collected UAV remote sensing multispectral images are processed by image registration, band synthesis, and image segmentation, and images that do not contain vehicle targets are discarded.

[0135] Furthermore, the specific details of image registration, band synthesis, and image segmentation are as follows:

[0136] (1) Image registration and band synthesis of UAV remote sensing multispectral images are performed by using the SIFT algorithm with scale invariant feature transformation to match each band of the UAV remote sensing multispectral image and synthesize it into a 4-band TIF format multispectral image according to spatial location.

[0137] (2) Segmenting UAV remote sensing multispectral images involves cutting the UAV remote sensing multispectral images to create an image dataset containing vehicle targets required for the super-resolution reconstruction model, so that the super-resolution reconstruction model can be trained.

[0138] Step 3: Input the UAV remote sensing multispectral imagery into the super-resolution reconstruction model. For example... Figures 7(a)-7(c) ;

[0139] Furthermore, in step 3, the super-resolution reconstruction model is trained by inputting a low-resolution vehicle target dataset from UAV remote sensing multispectral imagery into the network model to obtain a high-resolution output. This super-resolution reconstruction model is an improved SWinIR network model, as detailed below:

[0140] The SWinIR network mainly includes: a shallow feature extraction module, a deep feature extraction module, and a high-quality image reconstruction module.

[0141] The shallow feature extraction module uses two 3×3 convolutional layers (Conv) to extract shallow features. The first convolutional layer extracts shallow features from low-resolution UAV remote sensing multispectral imagery, which are then used in the second convolutional layer; the second convolutional layer performs further shallow feature extraction, which is then used in the deep feature extraction module.

[0142] The deep feature extraction module consists of several residual SwinTransformer blocks (RSTBs) and 3×3 convolutional layers. Each RSTB contains a Swing Transformer layer (STL) and convolutional layers. The STL includes a LayerNorm normalization layer, a Multi-head Self Attention (MSA) module, a Multi-Layer Perceptron (MLP), and residual connections. This module first divides the feature map from the shallow feature extraction module into several non-overlapping blocks, then recombines them using multiple concatenated residual SwinTransformer blocks, and finally outputs the result through convolutional layers.

[0143] The image reconstruction module comprises upsampling and depth mapping. First, it aggregates shallow and deep features. Second, it performs upsampling using a 3×3 deconvolutional layer. Finally, it performs depth mapping using a residual network structure with a single residual unit, consisting of several identical convolutional layers. The last layer has a 3×3×1 kernel, while the others have 3×3×64 kernels. This module uses the PReLu activation function, which adds a very small number of parameters to the ReLu activation function, improving accuracy and resolving the neuron death issue caused by the ReLu function. This method is used to reconstruct high-resolution UAV remote sensing multispectral imagery.

[0144] Step 4: Label vehicle targets and create a vehicle dataset.

[0145] Cars, trucks, and buses were labeled using a tool in high-resolution UAV remote sensing multispectral imagery. Each labeled image was stored in an XML file, where the position and size of the label boxes could be retrieved. The XML file was stored in the Annotations folder, and the corresponding UAV remote sensing multispectral image was stored in the JPEGImages folder. The XML file was then converted to a TXT file and stored in the labels folder.

[0146] Step 5: Feed the vehicle target dataset into the deep learning model for training.

[0147] The deep learning model in step 5 is an improved YOLOv5 model, as detailed below:

[0148] (1) Determine the size of the anchor frame: Input the vehicle target dataset into the YOLOv5 model and use the K-means clustering algorithm to analyze and determine the size of the anchor frame.

[0149] Furthermore, the specific details of the K-means clustering algorithm are as follows:

[0150] The algorithm divides the data into k groups, randomly selects k objects as initial cluster centers, calculates the distance between each object and each seed cluster center, and assigns each object to the nearest cluster center. The value of each cluster center is continuously recalculated during the iteration process, and the algorithm stops when the best clustering result is found.

[0151] (2) The improved YOLOv5 model is trained end-to-end.

[0152] Furthermore, the improved YOLOv5 model structure in step (2) is as follows:

[0153] The improved YOLOv5 network structure mainly includes: a backbone network, a neck network, and a detection network.

[0154] The backbone network is used to extract some general features of the image. The first layer of the backbone network is the Focus module, which slices the input image and then concatenates the slices to achieve deep connectivity. The result is then fed into a convolutional layer (Conv) consisting of 32 convolutions. After batch normalization (BN) and the LeakyReLU activation function, the result is fed into the BottleneckCSP module. The Bottleneck module is the main component of the BottleneckCSP module. It is a residual network architecture that connects a 1×1 convolutional layer (Conv2d+BN+LeakyReLU activation function, CBL) and a 3×3 CBL. The final output of the Bottleneck module is added to the initial input through the residual structure. The initial input of the BottleneckCSP module is fed into two branches, and the number of feature map channels is halved through the convolution operation between the two branches. Stacking operations are used to deeply concatenate the feature maps output from the two branches. Then, a BN layer and a Conv2d layer are used to output the same feature map as the input to the BottleneckCSP module. The tenth layer of the backbone network is a Spatial Pyramid Pooling (SPP) module, which transforms feature maps of arbitrary size into fixed-size feature vectors to improve the network's receptive field. The feature maps output from the convolutional layers are deeply concatenated with the output feature maps downsampled by three parallel max-pooling layers, and then passed through a convolutional layer to obtain the final output feature map. The backbone network also incorporates an attention mechanism (SE) network, which learns to automatically determine the importance of different features of the vehicle target, enhancing useful features and suppressing unimportant features accordingly. The specific process is as follows: Figure 8 As shown.

[0155] like Figure 9As shown, the specific steps of the lightweight attention mechanism network, the Squeeze-and-Excitation (SE) network, are as follows: Given an input X, a feature map U is output through convolution. The feature map U of size W×H×C is compressed (Squeeze) along the spatial dimensions, i.e., global pooling is performed on the feature map U to obtain a feature map of size 1×1×C. The compressed feature map is then activated (Excitation) through two fully connected layers (FC), generating weights for each feature channel. Finally, a weighting (Scale) operation is performed, i.e., multiplication is used to apply weights channel-wise to each channel of the feature map U to complete the channel-dimensional recalibration, obtaining the final output result. This network learns the importance of different feature channels, enhances useful features of the vehicle target, and suppresses irrelevant features of the vehicle.

[0156] The neck network is a feature aggregation layer that fuses image features, primarily used to generate the Feature Pyramid Network (FPN), which then transmits the output feature map to the detection network. The novel FPN structure enhances the bottom-up path, improves the transmission of low-level features, and strengthens the detection of objects at different scales. Therefore, the same target object of varying sizes and scales can be accurately identified.

[0157] The detection network, the final part of the model, consists of three detection layers for detecting image objects of varying sizes. Each detection layer outputs a vector, which is then used to generate and label predicted bounding boxes and categories for vehicle targets in the original image. This module merges the predicted boxes using a weighted box fusion (WBF) algorithm.

[0158] like Figure 10 As shown, the steps of the Weighted Boxes Fusion (WBF) are as follows: First, each predicted box is sorted in descending order according to its confidence score and added to the list.

[0159] Then, a list of box "clusters" and a list of box "merges" are created. The Intersection of Union (IoU) ratio is compared with a specified threshold to determine the degree of matching between them and the original boxes. Figure 11 As shown.

[0160] Finally, the coordinates and confidence scores of all boxes in the box list are adjusted. The average confidence score of all merged boxes is used as the new confidence score. Similar fusion and weighting are then applied to the new coordinates to generate the most suitable predicted boxes, using quantitative information to generate correct coordinates. This method merges predicted boxes, improving upon inaccurate counting.

[0161] This invention utilizes the WBF algorithm to merge these boxes, resulting in only one box on the final object. This avoids the phenomenon of multiple boxes potentially existing on a single object, thus improving the accuracy of the final detection.

[0162] Application Example 2

[0163] This embodiment provides a real-time vehicle detection method based on super-resolution reconstructed UAV remote sensing multispectral imagery, comprising the following steps:

[0164] Step 1: Acquire four-band imagery;

[0165] Step 2: Segment the image;

[0166] Step 3: Perform super-resolution reconstruction on the segmented image;

[0167] Step 4: Manually annotate the reconstructed images with anchor boxes to create a dataset;

[0168] Step 5: Divide the dataset into training and testing groups;

[0169] Step 6: Calculate the dimensions of the manually labeled anchor boxes in the training group using k-means;

[0170] Step 7: Input the training group into the model after anchor frame parameter correction to obtain the anchor frames marked on each vehicle target by the model after anchor frame parameter correction.

[0171] Step 8: Calculate the difference between the manually annotated anchor boxes and the model annotation boxes;

[0172] Step 9: Update the network parameters in the model after the anchor frame parameter correction based on the difference between the two anchor frames.

[0173] Step 10: Detect the target vehicle in the test group image.

[0174] Application Example 3

[0175] This invention also provides a computer device comprising: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0176] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps described in the various method embodiments above.

[0177] This invention also provides an information data processing terminal, which, when executed on an electronic device, provides a user input interface to implement the steps described in the above method embodiments. The information data processing terminal is not limited to mobile phones, computers, or switches.

[0178] This invention also provides a server that, when executed on an electronic device, provides a user input interface to implement the steps described in the above method embodiments.

[0179] This invention provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.

[0180] Application Example 4

[0181] like Figure 12 As shown in the figure, this embodiment provides a real-time vehicle detection system based on super-resolution reconstructed UAV remote sensing multispectral imagery, which includes the following steps:

[0182] Image acquisition module 1 is used to acquire four-band images;

[0183] Image segmentation module 2 is used to segment the image;

[0184] Super-resolution reconstruction module 3 is used to perform super-resolution reconstruction on the segmented image;

[0185] The manually labeled anchor box module 4 is used to create a dataset from the reconstructed images;

[0186] Training and test grouping module 5 is used to divide the dataset into training and test groups;

[0187] The module 6 for calculating the size of manually labeled anchor frames is used to calculate the size of manually labeled anchor frames in the training group using k-means.

[0188] The model bounding box acquisition module 7 is used to input the training group into the model after the anchor box parameters are corrected, and to obtain the anchor boxes marked on each vehicle target by the model after the anchor box parameters are corrected.

[0189] Anchor frame comparison module 8 is used to calculate the difference between manually annotated anchor frames and model annotation frames;

[0190] The parameter acquisition module 9 is used to update the network parameters in the model after the anchor frame parameters are corrected based on the difference between the two anchor frames.

[0191] The vehicle detection module 10 is used to detect target vehicles in the test group images.

[0192] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0193] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0194] III. Evidence of the relevant effects of the embodiments:

[0195] In the above embodiments, it is shown that this solution requires a set image acquisition device to acquire monochromatic spectral images of the scene to be identified in the near-infrared, red, green, and blue bands. Then, the SIFT algorithm is used to extract the image features of the monochromatic spectral images respectively. The monochromatic spectral images are then registered, and the spatial information contained in the extracted image features is used to synthesize the multi-band image. Since the near-infrared monochromatic spectral image is added to the multi-band image, the target YOLOv5 image recognition model can learn more information about the target objects in the image when it is trained using the multi-band image, thus improving the model training effect. After the trained model is put into traffic command and dispatch, it can play a better role in alleviating traffic congestion.

[0196] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention and within the spirit and principles of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for real-time vehicle detection based on UAV remote sensing multispectral imagery reconstructed from super-resolution, characterized in that, The method includes the following steps: S1. Acquire multi-band images and segment them to obtain multiple segmented sub-images. Perform super-resolution reconstruction on the multiple segmented sub-images and obtain the reconstructed images. Label the vehicle targets in the reconstructed images to obtain the vehicle dataset. S2. Divide the obtained vehicle dataset into a training group and a test group; S3. Input the training group into the initial YOLOv5 image recognition model for training to obtain the target YOLOv5 image recognition model; S4. Input the test group into the target YOLOv5 image recognition model to test the target YOLOv5 image recognition model's ability to recognize the target vehicle and perform real-time vehicle detection. In step S1, multiple segmented sub-images are super-resolution reconstructed to obtain the reconstructed image, specifically including the following steps: S1-1, using two 3×3 convolutional layers, shallow feature extraction is performed on the segmented sub-image to obtain a shallow feature map; S1-2, The shallow feature map is segmented and then recombined to obtain the deep feature map; S1-3, perform feature aggregation on the shallow feature map and the deep feature map to obtain an aggregated feature map; S1-4, the aggregated feature map is upsampled using a 3×3 deconvolution layer, and the aggregated feature map is depth-mapped using 3×3×64 and 3×3×1 convolution kernels to obtain the reconstructed image; In step S3, the training set is input into the initial YOLOv5 image recognition model for training to obtain the target YOLOv5 image recognition model, including the following steps: S3-1, The aspect ratio of manually labeled anchor boxes in several reconstructed images in the training group is calculated using the K-means algorithm. The anchor box parameters in the initial YOLOV5 image recognition model are then corrected to obtain the YOLOV5 image recognition model with corrected anchor box parameters. S3-2, Input the training group into the YOLOV5 image recognition model after anchor frame parameter correction, train the YOLOV5 image recognition model after anchor frame parameter correction, extract the vehicle target feature information of several reconstructed images in the training group and fuse them; S3-3: Anchor boxes are marked on each vehicle target based on the fused feature information. By comparing with manually marked anchor boxes, the difference between manually marked anchor boxes and model-marked anchor boxes is calculated. Then, the network parameters are updated in reverse and iterated to obtain the target YOLOv5 image recognition model.

2. The method for real-time vehicle detection based on UAV remote sensing multispectral imagery using super-resolution reconstruction as described in claim 1, characterized in that, In step S1, the multi-band image is the image obtained by merging four images of the scene to be identified.

3. The method for real-time vehicle detection based on UAV remote sensing multispectral imagery using super-resolution reconstruction as described in claim 2, characterized in that, The four images include: a near-infrared monospectral image, a red monospectral image, a blue monospectral image, and a green monospectral image.

4. The method for real-time vehicle detection based on UAV remote sensing multispectral imagery using super-resolution reconstruction as described in claim 1, characterized in that, In step S3-2, the extraction and fusion of vehicle target feature information from several reconstructed images in the training group specifically includes the following steps: (a) Assign weights to various features of each target vehicle in the training group according to their importance; (b) The various feature information is weighted by multiplication according to the weighted weights to enhance the useful features of each vehicle target and suppress the irrelevant features of each vehicle target; wherein the useful features include category features and size features, and the irrelevant features include color features and orientation features; (c) Extract the feature information of each vehicle target.

5. The method for real-time vehicle detection based on UAV remote sensing multispectral imagery using super-resolution reconstruction as described in claim 1, characterized in that, In step S4, the ability of the target YOLOv5 image recognition model to recognize the target vehicle is tested, specifically including the following steps: On vehicle targets in several reconstructed images of the test group, several first prediction boxes containing location and score are generated. The weighted box fusion (WBF) algorithm is used to merge the several first prediction boxes on the vehicle targets to obtain the final prediction result. Then, a unique second prediction box is generated on the vehicle targets in several reconstructed images of the test group.

6. The method for real-time vehicle detection based on UAV remote sensing multispectral imagery using super-resolution reconstruction as described in claim 5, characterized in that, The generation of the unique second prediction box includes: The first prediction boxes containing position and score on each target vehicle are calculated using formulas (1) and (2), and the horizontal and vertical coordinates of each vertex of the second prediction box are merged. （1） （2） Where M represents the number of a target vehicle among X target vehicles in the reconstructed images of the test group; Y represents the number of a first predicted bounding box among the Mth target vehicle in the reconstructed images of the test group; A Mj B represents the x-coordinate of the j-th vertex of the second predicted bounding box of the M-th target vehicle in several reconstructed images of the test group; Mj A represents the ordinate of the j-th vertex of the second predicted bounding box of the M-th target vehicle in several reconstructed images of the test group; Mji B represents the x-coordinate of the j-th vertex of the i-th first predicted bounding box of the M-th target in the reconstructed images of the test group; Mji C represents the ordinate of the j-th vertex of the i-th first predicted bounding box of the M-th target object in the reconstructed images of the test group; Mi This represents the score of the i-th first prediction box of the M-th target object in several reconstructed images of the test group.

7. The method for real-time vehicle detection based on UAV remote sensing multispectral imagery using super-resolution reconstruction as described in claim 1, characterized in that, In step S4, the real-time detection of the vehicle specifically includes the following steps: Step 1: Acquire four-band imagery; Step 2: Segment the image; Step 3: Perform super-resolution reconstruction on the segmented image; Step 4: Manually annotate the reconstructed images with anchor boxes to create a dataset; Step 5: Divide the dataset into training and testing groups; Step 6: Calculate the dimensions of the manually labeled anchor boxes in the training group using k-means; Step 7: Input the training group into the model after anchor frame parameter correction to obtain the anchor frames marked on each vehicle target by the model after anchor frame parameter correction. Step 8: Calculate the difference between the manually labeled anchor boxes and the model labeled boxes; Step 9: Update the network parameters in the model after the anchor frame parameter correction based on the difference between the two anchor frames. Step 10: Detect the target vehicle in the test group image.

8. A real-time vehicle detection system based on super-resolution reconstructed UAV remote sensing multispectral imagery, implementing the real-time vehicle detection method based on super-resolution reconstructed UAV remote sensing multispectral imagery as described in any one of claims 1-7, characterized in that, The real-time vehicle detection system based on super-resolution reconstructed UAV remote sensing multispectral imagery includes: Image acquisition module (1) is used to acquire four-band images; Image segmentation module (2) is used to segment images; The super-resolution reconstruction module (3) is used to perform super-resolution reconstruction on the segmented image; The manually labeled anchor box module (4) is used to create a dataset from the reconstructed images; The training and test grouping module (5) is used to divide the dataset into training and test groups; The module for calculating the size of manually labeled anchor frames (6) is used to calculate the size of manually labeled anchor frames in the training group using k-means; The model box acquisition module (7) is used to input the training group into the model after the anchor box parameters are corrected, and to obtain the anchor boxes marked on each vehicle target by the model after the anchor box parameters are corrected. Anchor frame comparison module (8) is used to calculate the difference between manually annotated anchor frames and model annotation frames; The parameter acquisition module (9) is used to update the network parameters in the model after the anchor frame parameter correction based on the difference between the two anchor frames. The vehicle detection module (10) is used to detect target vehicles in the test group images.

Images