A mine detection method based on multimodal image fusion carried by unmanned aerial vehicle

Abstract

The present invention belongs to the technical field of unmanned aerial vehicle detection, and discloses a multimodal image fusion mine detection method based on unmanned aerial vehicle, wherein a plurality of modal images are collected by an unmanned aerial vehicle platform and the modal images are input into an image registration module for registration, and the positions of the same targets between the registered modal images are kept aligned on the modal images; a small target multiscale prediction algorithm is used to perform feature extraction and multiscale prediction on a single modal image and obtain the classification and rough prediction frame of the ground environment; a weight of the rough prediction frame is assigned, and the setting of the weight depends on the classification information of the ground environment and the physical properties of the preset target; all the rough prediction frames with assigned weights are subjected to non-maximum suppression processing to obtain the final prediction frame. The unmanned aerial vehicle mine detection method of the present invention greatly improves the accuracy of mine detection and reduces false alarms and missed alarms.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle detection, and in particular relates to an unmanned aerial vehicle-based multi-modal image fusion mine detection method. Background Art

[0002] The tactical application of ground-scattered mines in modern warfare has put forward new requirements for the speed and accuracy of mine detection, and has also spawned new mine detection operation methods. Traditional manual mine detection, vehicle-mounted mine detection and other methods can no longer meet the requirements for rapid detection of large minefields. With the development of highly maneuverable, low-cost and high-precision unmanned aerial survey and remote sensing technology, integrating imaging sensors with unmanned aerial vehicles to obtain low-altitude remote sensing images has become a new research hotspot.

[0003] However, the existing technology of mine detection based on unmanned aerial vehicle platforms mainly focuses on the use of ground penetrating radars, which have high requirements on the flight stability of the unmanned aerial vehicle and are greatly affected by clutter interference and inhomogeneous media, making it difficult to achieve high-precision detection. For spectral detection, it mainly collects and uses single spectral data such as thermal infrared or near-infrared, which cannot cover all scenes and has a significant problem of missed detection. The existing mine detection method based on a single sensor has a high false alarm rate and loses its effectiveness in some difficult scenes such as humid and low-lying areas.

[0004] The target detection technology based on deep learning has reached a new height in recent years. Different sensors have complementary information, can adapt to various environmental conditions and improve the accuracy of decision-making. The fusion decision of multi-source images does not limit the number of sensors, can be flexibly expanded horizontally, and is universal for the detection of various types of mines in various scenarios. In addition, the deep learning network can naturally be compatible with data of different modalities, and a sufficient number of training samples can reach the peak value of the modal data. However, due to the limitations of the information content of the data itself, in some scenarios, a single modality image cannot provide effective information, and different modal data are complementary. Therefore, how to design a multi-modal image fusion mine detection method carried by unmanned aerial vehicles, through the reasonable fusion of multi-modal information, comprehensively utilize the detection advantages of multiple sensors, achieve better results in the target detection network based on deep learning, improve the accuracy of mine detection, and reduce false alarms and missed alarms, has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-modal image fusion mine detection method based on an unmanned aerial vehicle to solve the problems mentioned in the above background technology.

[0006] In order to achieve the above invention purpose, the technical solution adopted is:

[0007] A multi-modal image fusion mine detection method based on an unmanned aerial vehicle comprises: collecting a plurality of modal images through an unmanned aerial vehicle platform and inputting the modal images into an image registration module for registration, keeping the positions of the same targets between the registered modal images aligned on the modal images; performing feature extraction and multi-scale prediction on a single modal image through a small target multi-scale prediction algorithm and obtaining a classification and a rough prediction frame of the ground environment; allocating weights to the rough prediction frames, wherein the setting of the weights depends on the classification information of the ground environment and the preset physical properties of the target; performing non-maximum suppression processing on all the rough prediction frames to which weights have been allocated, and obtaining a final prediction frame.

[0008] The present invention is further configured to: collect a plurality of modal images through an unmanned aerial vehicle platform and input the modal images into an image registration module for registration, including: acquiring the modal images of the same target through a plurality of sensors of the unmanned aerial vehicle platform and mapping the plurality of modal images onto a reference image.

[0009] The present invention is further configured as follows: the modal images of the same target are acquired through several sensors of the unmanned aerial vehicle platform and several modal images are mapped to a reference image, including: inputting the modal image to be registered and the reference image into a convolutional neural network; calculating the semantic feature matching degree on the deep convolutional graph and selecting the block with the highest matching degree to map to a shallow convolutional layer; calculating the feature matching degree of the block on the shallow convolutional layer and performing transformation model calculation on the pixel area of ​​the block with the highest matching degree to obtain the transformation model.

[0010] The present invention is further configured as follows: the small target multi-scale prediction algorithm is used to perform feature extraction and multi-scale prediction on a single modal image respectively and obtain a classification and a rough prediction frame of the ground environment, including: performing feature extraction on the modal image respectively through a convolutional neural network to generate several feature maps of different levels; aggregating the feature maps of several different levels, obtaining multiple feature maps of the same size through 1x1 convolution calculation and performing cascade processing to obtain a multi-scale feature map; predicting the target and vegetation classification information of the multi-scale feature map to obtain the prediction content.

[0011] The present invention is further configured such that: the prediction content includes mine location coordinates, mine category, target confidence and terrain category.

[0012] The present invention is further configured as follows: the size of the weight is allocated according to the terrain classification result and the physical properties of the mine, and the product of the weight and the target confidence is the score of the final prediction frame.

[0013] The present invention is further configured as follows: performing non-maximum suppression processing on all the coarse prediction frames with assigned weights to obtain a final prediction frame, including: filtering the coarse prediction frames with assigned weights according to a score threshold; merging the coarse prediction frames with assigned weights of the same target to remove redundant frames; and filtering the coarse prediction frames with assigned weights by an intersection-over-union threshold.

[0014] The present invention is further configured to: filter the coarse prediction frames to which weights have been assigned by an intersection-and-union ratio threshold, including: determining the highest-scoring prediction frame according to the ranking of the scores of the coarse prediction frames to which weights have been assigned; traversing the remaining coarse prediction frames to which weights have been assigned, and deleting the coarse prediction frames to which weights have been assigned whose intersection-and-union ratio with the highest-scoring prediction frame is greater than a set intersection-and-union ratio threshold; and repeating the screening until the intersection-and-union ratio threshold is maximized.

[0015] The present invention is further configured as follows: the convolutional neural network is composed of convolutional layers, activation functions and pooling layers that are alternately connected.

[0016] In summary, compared with the prior art, the present invention discloses a multi-modal image fusion mine detection method based on unmanned aerial vehicles, wherein a plurality of modal images are collected by an unmanned aerial vehicle platform and the modal images are input into an image registration module for registration, and the positions of the same targets between the registered modal images are kept aligned on the modal images; a small target multi-scale prediction algorithm is used to perform feature extraction and multi-scale prediction on a single modal image and obtain the classification and rough prediction frame of the ground environment; a rough prediction frame weight is assigned, and the setting of the weight depends on the classification information of the ground environment and the preset target physical properties; all the rough prediction frames with assigned weights are subjected to non-maximum suppression processing to obtain the final prediction frame. That is, through this setting, the mine detection accuracy is greatly improved, and false alarms and missed alarms are reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is a flow chart of a multi-modal image fusion mine detection method based on an unmanned aerial vehicle provided in this embodiment. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] In addition, the technical features involved in the different embodiments of the present invention described above can be combined with each other as long as they do not conflict with each other.

[0021] A multi-modal image fusion mine detection method based on unmanned aerial vehicle, such as Figure 1 As shown, the following steps are included:

[0022] S101, collecting a plurality of modal images through an unmanned aerial vehicle platform and inputting the modal images into an image registration module for registration, so as to keep the positions of the same targets between the registered modal images aligned on the modal images.

[0023] In the specific implementation process, several modal images are multimodal images. Image registration of multimodal images refers to obtaining images of the same scene from different sensors, mapping one or more images to another reference image, so that the points in the two images corresponding to the same position in the real space correspond one to one. Image registration is often a preprocessing step for image fusion. Image pairs that have undergone precise image registration can usually obtain better fusion effects. Traditional image registration usually manually designs features, and image registration tasks of different modalities require redesign of feature descriptions. Image registration based on neural networks can eliminate the need for feature design and selection, greatly reducing the workload of manual participation.

[0024] In this step, modal images of the same target are acquired through several sensors of the UAV-mounted platform and the modal images are mapped onto the reference image, which specifically includes:

[0025] Input the modality image to be registered and the reference image into the convolutional neural network;

[0026] Calculate the semantic feature matching degree on the deep convolutional graph and select the block with the highest matching degree to map to the shallow convolutional layer;

[0027] The feature matching degree of the blocks on the shallow convolution layer is calculated and the pixel area of ​​the block with the highest matching degree is subjected to transformation model calculation to obtain the transformation model.

[0028] It should be noted that the modal image to be registered and the reference image are respectively input into the convolutional neural network. First, the matching degree of the semantic features is calculated on the deep convolutional map, and the block with the highest matching degree is selected and backward mapped to the shallow convolutional layer. Then, a more refined feature matching is calculated for the corresponding block on the shallow convolutional map. Finally, the pixel area with the highest matching degree is selected to calculate the transformation model. After obtaining the transformation model, the image to be registered is transformed, and the image to be registered can be mapped to the reference image space.

[0029] S102, performing feature extraction and multi-scale prediction on a single modality image respectively through a small target multi-scale prediction algorithm and obtaining a classification and a rough prediction frame of the ground environment.

[0030] This step specifically includes:

[0031] The convolutional neural network is used to extract features from the modal images to generate several feature maps at different levels;

[0032] Aggregate several feature maps of different levels, calculate multiple feature maps of the same size through 1x1 convolution and perform cascade processing to obtain multi-scale feature maps;

[0033] Predict the target and vegetation classification information of the multi-scale feature map to obtain the predicted content.

[0034] The prediction content includes mine location coordinates, mine category, target confidence and terrain category.

[0035] In this implementation, since the pixel area of ​​the ground scattering mine target is limited for low-altitude remote sensing images, it belongs to small target detection, and multi-scale prediction can effectively improve the detection accuracy of small targets. It should be noted that the convolutional neural network is composed of convolutional layers, activation functions and pooling layers connected alternately. The convolutional neural network extracts features from multimodal images respectively to generate feature maps of different levels, and converges the features of shallow feature maps and deep feature maps. The shallow feature map has rich detail information and positioning information, and the deep feature map has a larger receptive field and more comprehensive semantic features. After 1x1 convolution calculation, multiple feature maps of the same size are obtained and cascaded, and multi-scale features that integrate low-level details and high-level abstract features can be obtained. For the feature maps of multiple scales obtained after fusion, targets of different sizes and vegetation classification information are predicted respectively. The prediction results on the small-size feature map include global information, and the positioning accuracy on the large-size feature map is more accurate.

[0036] S103, assigning a weight to the rough prediction frame, wherein the weight setting depends on the classification information of the ground environment and the preset physical properties of the target.

[0037] In this embodiment, weights are assigned to the coarse prediction frames obtained from different modal images respectively, and the size of the weight is allocated according to the terrain classification result and the physical properties of the mine. For a flat gravel road surface, the mine shape is highly distinguishable from the surrounding environment, and the high-resolution RGB image can clearly capture the obvious mine features, so the prediction frame of the RGB image is assigned a large weight. For a sandy or muddy ground environment, the ground penetrating radar image can capture the mine features on the shallow surface. Assigning a large weight to it is helpful to survey old mines that are partially buried under the sand. For a ground environment with low and dense vegetation, the mines are seriously obscured by the vegetation. The thermal effect of metal mine targets on thermal infrared images is more significant. Assigning a large weight can extract the mines hidden under the vegetation. For metal mines, sensors such as ground penetrating radar and thermal infrared that are more sensitive to metal targets can be assigned a large weight; for extremely small targets.

[0038] In this step, the size of the weight is allocated according to the terrain classification result and the physical properties of the mine, and the product of the weight and the target confidence is the score of the final prediction box.

[0039] S104, performing non-maximum suppression processing on all the coarse prediction boxes with assigned weights to obtain the final prediction box.

[0040] In this step, all the coarse prediction boxes with assigned weights are subjected to non-maximum suppression to obtain the final prediction box, which specifically includes:

[0041] Filter the coarse prediction boxes with assigned weights according to the score threshold;

[0042] Merge the coarse prediction boxes with assigned weights for the same target to remove redundant boxes;

[0043] The coarse prediction boxes with assigned weights are filtered by the intersection-over-union ratio threshold.

[0044] Furthermore, the coarse prediction boxes with assigned weights are filtered through the intersection-over-union ratio threshold, including:

[0045] Determine the highest scoring prediction box according to the ranking of the scores of the coarse prediction boxes to which the weights have been assigned;

[0046] Traverse the remaining coarse prediction boxes with assigned weights, and delete the coarse prediction boxes with assigned weights whose intersection-over-union ratio with the highest-scoring prediction box is greater than the set intersection-over-union ratio threshold;

[0047] Repeat the screening until the intersection-union ratio threshold is maximized.

[0048] In this embodiment, the prediction frames with calculated scores are fused by decision through non-maximum suppression. First, all prediction frames with scores below the score threshold are discarded. Considering the requirement for high recall rate in the mine detection scenario, the score threshold is generally set to a low level of 0.1-0.2. Non-maximum suppression is used to merge all prediction frames belonging to the same target, thereby removing a large number of redundant frames. Non-maximum suppression is based on the assumption that highly overlapping detections belong to the same object. The mine laying method of scattering mines makes it unlikely that they are tightly fixed in extremely adjacent positions. In the mine detection scenario, even if two or more mines are in close contact, it is not very useful for mine clearance. The recognition of the mine position will not be affected. The intersection-and-union ratio between the prediction frames determines the overlap degree of the two frames. A high intersection-and-union ratio threshold is helpful for the detection of dense targets, but it will cause a large number of false alarms in sparse target scenarios. Considering the sparsity of mine detection scenarios, the intersection-and-union ratio threshold is generally set to 0.6. Non-maximum suppression accepts the detection with the highest score belonging to the same object, and then removes all detections with an intersection-and-union ratio exceeding the threshold, and repeats the screening process, greedily accepting the local maximum value and discarding its neighbors. Finally, the remaining prediction frames can completely contain the effective mine targets that can be detected by each sensor while reducing a large number of redundant detections.

[0049] It should be noted that the fusion method proposed in this embodiment has the characteristics of plug-and-play, can be easily embedded in the industry's most advanced target detection model, and has the characteristics of easy operation, easy implementation and easy deployment. The details of the fusion include the addition of the multi-scale prediction part to the classification of terrain and landforms, and the weight distribution is based on the mine model with the highest score. The weight distribution can be set as a hyperparameter or trained as a learnable parameter. Secondly, since the non-maximum suppression of the prediction of each sensor is avoided separately, but the weights are redistributed and then performed uniformly, the number of redundant frames and the time cost of post-processing can be greatly reduced.

[0050] It is understandable that in this embodiment, the multimodal images may be fused by pixel-level fusion or feature-level fusion, and then general target detection processing may be performed.

[0051] In summary, the present invention has the following beneficial effects:

[0052] It makes full use of the complementarity between multimodal data, and improves the accuracy and recall rate of detection by mobilizing the information complementarity between each modality, and can adapt to different minefield environments;

[0053] As a post-processing method for typical target detection models, the decision fusion module based on non-maximum suppression is plug-and-play and can be flexibly applied to different target detection models.

[0054] In practical application, the multimodal image fusion mine detection method based on drones disclosed in this embodiment solves the problems of high risk factor and slow detection speed of traditional manual or vehicle-mounted mine detection, and has better mobility and speed advantages, and strong environmental adaptability. On the other hand, multimodal fusion detection can adapt to different terrain areas and mines of various materials and models to the greatest extent, and meet the requirements of complete and accurate detection through the complementarity of information. The drone is equipped with a multi-sensor imaging platform and a deep learning computing module, which can ensure the real-time detection.

[0055] In addition, the multimodal image fusion detection strategy proposed in this embodiment is fully deployable end-to-end. The so-called end-to-end means that the input is the unprocessed original image pixels and the output is the mine location coordinates. The end-to-end integrated operation process avoids the time-consuming and tedious image transmission back to the ground station and manual processing.

[0056] Furthermore, the multi-scale prediction part proposed in this embodiment adds the classification of terrain and landforms, and distributes weights based on the mine model with the highest score. The weight distribution can be set as a hyperparameter or trained as a learnable parameter. Secondly, since non-maximum suppression is avoided for the prediction of each sensor individually, and the weights are redistributed and then performed uniformly, the number of redundant frames and the time cost of post-processing can be greatly reduced.

[0057] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A multi-modal image fusion mine detection method based on unmanned aerial vehicle, characterized in that: include: Collecting a plurality of modal images through an unmanned aerial vehicle platform and inputting the modal images into an image registration module for registration, so as to keep the positions of the same objects in the modal images aligned after registration; Acquiring the modal images of the same target through a plurality of sensors of the unmanned aerial vehicle platform and mapping the plurality of modal images onto a reference image, including: Inputting the modality image to be registered and the reference image into a convolutional neural network; Calculate the semantic feature matching degree on the deep convolutional graph and select the block with the highest matching degree to map to the shallow convolutional layer; Calculating the feature matching degree of the block on the shallow convolution layer and performing transformation model calculation on the pixel area of ​​the block with the highest matching degree to obtain the transformation model; Performing feature extraction and multi-scale prediction on each of the modal images through a small target multi-scale prediction algorithm and obtaining a classification and a rough prediction frame of the ground environment; Allocating a weight to the rough prediction frame, wherein the weight is set depending on the classification information of the ground environment and the preset target physical properties; All the coarse prediction frames to which weights have been assigned are subjected to non-maximum suppression processing to obtain a final prediction frame.

2. The multi-modal image fusion mine detection method based on unmanned aerial vehicle according to claim 1, characterized in that: The method of performing feature extraction and multi-scale prediction on a single modal image by using a small target multi-scale prediction algorithm and obtaining a classification and a rough prediction frame of the ground environment includes: Extracting features of the modal images by convolutional neural networks to generate feature maps of several different levels; Aggregating the feature maps of several different levels, obtaining multiple feature maps of the same size through 1x1 convolution calculation and performing cascade processing to obtain a multi-scale feature map; The target and vegetation classification information of the multi-scale feature map are predicted to obtain predicted content.

3. The multi-modal image fusion mine detection method based on unmanned aerial vehicle according to claim 2, characterized in that: The prediction content includes mine location coordinates, mine category, target confidence and terrain category.

4. The multi-modal image fusion mine detection method based on unmanned aerial vehicle according to claim 3, characterized in that: The size of the weight is allocated according to the terrain classification result and the physical properties of the mine, and the product of the weight and the target confidence is the score of the final prediction box.

5. The multi-modal image fusion mine detection method based on unmanned aerial vehicle according to claim 4, characterized in that: The non-maximum suppression process is performed on all the coarse prediction frames to which weights have been assigned to obtain a final prediction frame, including: Filter the coarse prediction boxes to which weights have been assigned according to a score threshold; Merging the coarse prediction boxes with assigned weights of the same target to remove redundant boxes; The coarse prediction boxes to which weights have been assigned are filtered by an intersection-over-union ratio threshold.

6. The multi-modal image fusion mine detection method based on unmanned aerial vehicle according to claim 5, characterized in that: The coarse prediction frame to which the weight has been assigned is screened by an intersection-over-union ratio threshold, including: Determine a prediction frame with a highest score according to the ranking of the scores of the coarse prediction frames to which weights have been assigned; Traversing the remaining coarse prediction frames to which weights have been assigned, and deleting the coarse prediction frames to which weights have been assigned whose intersection-over-union ratio with the highest-scoring prediction frame is greater than a set intersection-over-union ratio threshold; The screening is repeated until the intersection-over-union ratio threshold is maximized.

7. The method for detecting mines based on multimodal image fusion carried by an unmanned aerial vehicle according to claim 1, characterized in that: The convolutional neural network is composed of convolutional layers, activation functions and pooling layers connected alternately.

Images