Method for detecting explosive objects and marking minefields using multispectral camera and augmented reality

Abstract

FIELD: military.SUBSTANCE: invention relates to detecting explosive objects. The technical result consists in increasing the accuracy of detection of explosive objects and reducing the time of their detection by processing data in real time. This result is achieved by surveying the area using an unmanned aerial vehicle, obtaining images of the earth's surface using a multispectral camera, and processing the resulting images using a machine learning algorithm. From the unmanned aerial vehicle, several video streams corresponding to different spectral channels of the multispectral camera are simultaneously transmitted to the ground computing cluster, said video streams are hardware synchronized with each other by means of a protocol for precise time synchronization and a common sensor synchronization signal, the video streams are multiplexed and transmitted via a wideband digital radio channel while maintaining the temporal correlation between the channels, after which they are processed in real time on the ground computing cluster containing at least one graphic computing processor. The processing is performed by a deep learning algorithm that extracts features from each spectral channel and combines these features to identify anomalies corresponding to explosive objects. After an anomaly is detected, the coordinates of the detected object are determined by comparing the position of the image pixels with the telemetry data of the unmanned aerial vehicle.EFFECT: increased accuracy of detection of explosive objects and reduced time of their detection.1 cl, 1 dwg

Description

[0001] FIELD OF TECHNOLOGY

[0002] The claimed technical solution generally relates to the field of detection of explosive objects, primarily mines, using unmanned aerial vehicles (UAVs), multispectral imaging and machine learning algorithms.

[0003] STATE OF THE ART

[0004] Existing mine detection methods using UAVs are often limited by the following problems:

[0005] Data processing is performed on board the UAV, which limits computing resources and reduces accuracy;

[0006] Multispectral image streams are not synchronized at the pixel level, which reduces the detection quality;

[0007] The results are post-processed after the flight, creating a delay between the survey and the receipt of the map;

[0008] Insufficient visibility for the operator and sapper makes it difficult to move safely around the area.

[0009] Current solutions based on "gluing" images from different spectral channels (early fusion) or voting based on detection results (decision fusion) do not allow for the detection of complex combinations of features, such as thermal anomalies, vegetation changes, and microrelief.

[0010] Public patent application EP4425445A1, "REMOTE AERIAL MINEFIELD SURVEY," published September 4, 2024, discloses a method for remotely surveying areas to detect explosive devices using aerial multispectral or hyperspectral imaging. Images are pre-processed and georeferenced, then analyzed using neural network algorithms to identify potential explosive devices. The results can be displayed as heat maps or markers on an orthophotomap.

[0011] The prior art includes patent application AU2023342520A1, "Systems and methods for detecting and identifying explosives," published April 17, 2025. It describes a system for detecting and identifying explosives, including a UAV platform with optical sensors. The obtained data is transmitted to a processing module, where orthomosaics are formed and objects are classified using machine learning algorithms.

[0012] Furthermore, patent RU2327189C2, "Method and System for Remote Detection of Metal Objects, Such as Weapons," published June 20, 2008, discloses a system for remote detection of metal objects, including mines, using radar and other sensor methods. This source confirms the existence of remote detection approaches, but does not cover the combined analysis of multispectral data with georeferencing and augmented reality visualization.

[0013] Thus, the above existing solutions allow for the detection of mines and explosive hazards using UAVs and various sensors, but they do not provide:

[0014] hardware-synchronized multispectral shooting;

[0015] transmitting multiple spectral streams to the ground computing cluster for real-time processing;

[0016] Combine features from different channels using deep learning algorithms to improve detection accuracy;

[0017] Creation of a geo-referenced map of a minefield with display of detected objects in augmented reality for the operator and sapper.

[0018] The present invention aims to eliminate these disadvantages of the prior art and provides improved accuracy and speed of detection of explosive objects, as well as safe visualization of results for users.

[0019] SUMMARY OF THE INVENTION

[0020] The technical problem that the declared technical solution is aimed at solving is the creation of a new method for detecting explosive objects and marking a minefield.

[0021] The technical result of the solution is to increase the accuracy of detection of explosive objects and reduce the time of their detection by processing data in real time.

[0022] Additional technical results are:

[0023] - reducing the number of false alarms;

[0024] - increasing the reliability of georeferencing of detected objects;

[0025] - providing clear visualization of results for the operator and sapper.

[0026] The specified technical result is achieved through a method for detecting explosive objects and marking a minefield, which contains the following stages:

[0027] - carry out a survey of the territory using an unmanned aerial vehicle;

[0028] - obtain images of the earth's surface using a multispectral camera and process the resulting images using a machine learning algorithm, while:

[0029] • the unmanned aerial vehicle simultaneously transmits several video streams corresponding to different spectral channels of the multispectral camera to the ground computing cluster;

[0030] • synchronize the specified video streams with each other using a precise time synchronization protocol and a common sensor synchronization signal;

[0031] • video streams are multiplexed and transmitted via a wideband digital radio channel while maintaining temporal correlation between channels;

[0032] • after which they are processed in real time on a ground-based computing cluster containing at least one graphics computing processor;

[0033] • the processing is performed using a deep learning algorithm that extracts features from each spectral channel and combines these features to identify anomalies corresponding to explosive objects;

[0034] • After detecting an anomaly, determine the coordinates of the detected object by comparing the position of the image pixels with the telemetry data of the unmanned aerial vehicle;

[0035] • and display detected objects on the video stream for the operator in the form of virtual augmented reality marks;

[0036] • As the area is surveyed, a geo-referenced map of the location of the discovered explosive objects is created.

[0037] In a particular embodiment of the method, the multispectral camera contains at least six spectral channels.

[0038] In a particular embodiment of the method, the spectral channels include: a visible RGB channel, a RedEdge channel, a near infrared channel, a shortwave infrared channel, a longwave infrared channel, and a panchromatic channel.

[0039] In a particular embodiment of the method, image processing is performed by a hybrid neural network containing: a convolutional neural network for extracting local features; a transformer architecture for combining features from different spectral channels.

[0040] In a particular embodiment of the method, the combination of information from different spectral channels is performed at the feature level (feature fusion).

[0041] In a particular embodiment of the method, the result of processing is a segmentation mask of the detected object and the probability of its presence.

[0042] In a particular embodiment of the method, false positives are eliminated by temporal filtering, in which a detected object is confirmed only when it is detected in several consecutive frames.

[0043] In a particular embodiment of the method, the coordinates of the detected objects are determined using a reverse photogrammetric projection method, taking into account the coordinates of the unmanned aerial vehicle, the orientation angles of the camera, and a digital terrain model.

[0044] In a particular embodiment of the method, an orthophotomap of the surveyed area is additionally formed.

[0045] In a particular embodiment of the method, the coordinates of the detected explosive objects are applied to the orthophotoplan.

[0046] DESCRIPTION OF DRAWINGS

[0047] The invention will be further described in accordance with the accompanying drawing, which is provided to clarify the essence of the invention and in no way limits the scope of the invention. The following drawing is attached to the application:

[0048] Fig. 1 illustrates the structural diagram of the claimed solution.

[0049] DETAILED DESCRIPTION OF THE INVENTION

[0050] The following detailed description of the invention includes numerous implementation details to provide a clear understanding of the present invention. However, it will be apparent to one skilled in the art how the present invention may be used with or without these implementation details. In other instances, well-known methods, procedures, and components have not been described in detail to avoid obscuring the features of the present invention.

[0051] Furthermore, it will be clear from the foregoing description that the invention is not limited to the embodiment described. Numerous possible modifications, changes, variations, and substitutions, while preserving the spirit and form of the present invention, will be apparent to those skilled in the art.

[0052] The proposed technical solution proposes a method for detecting explosive ordnance and marking minefields. This method improves the accuracy and speed of detecting hazardous objects in a surveyed area and generates a minefield map with augmented reality visualization.

[0053] The method includes the following main steps:

[0054] 1. Preparing for the UAV flight.

[0055] The multispectral camera is calibrated to precisely align the pixels of different channels (transition matrix).

[0056] The sensors, especially the thermal imaging sensor, are warming up to stabilize their readings. The flight mission is being loaded into the UAV.

[0057] The ground computing cluster is activated and a secure connection is established with the aircraft.

[0058] 2. Survey of the territory.

[0059] The UAV moves along a grid route at an altitude of 50,100 meters (for optimal visibility and accurate georeferencing).

[0060] The multispectral camera generates six video streams:

[0061] visible RGB, about RedEdge,

[0062] near infrared (NIR), shortwave infrared (SWIR), longwave infrared (LWIR), panchromatic.

[0063] Video streams are hardware synchronized using PTR and Genlock.

[0064] Streams are transmitted in RAW format at 12 14 bits / pixel via a wideband digital radio channel at 5 GHz with adaptive modulation with minimal intra-frame compression (JPEG2000), maintaining correlation between channels.

[0065] On a ground-based computing cluster containing multiple GPUs, the following is performed:

[0066] - Merging and pre-processing of streams.

[0067] - Processing by deep learning algorithm (CNN+Transformer) to extract features and fuse them between channels.

[0068] - Detection of anomalies corresponding to explosive objects.

[0069] It should be further noted that the processing of video streams is performed using a deep learning algorithm implemented in the form of a hybrid neural network, including at least one convolutional neural network (CNN), configured with the ability to isolate local spatial features in images, and at least one transformer architecture module (Transformer), configured with the ability to take into account global dependencies and combine features obtained from different spectral channels.

[0070] As a convolutional neural network, architectures selected from the group including ResNet, EfficienfNet, YOLO or their modifications can be used, ensuring the selection of informative features of objects in an image.

[0071] Models that implement a self-attention mechanism that enables the fusion of information from different spectral channels can be used as a transformer architecture.

[0072] The fusion of information from different spectral channels is mainly carried out at the feature level (feature fusion), in which the features extracted from each spectral channel are combined into a common representation used for subsequent object detection.

[0073] The deep learning algorithm is trained using a training set containing multispectral images of terrain areas known to contain and not contain explosive ordnance. This training set includes images obtained in various spectral ranges, including visible, near-infrared, and thermal.

[0074] During training, images are labeled, identifying areas corresponding to explosive objects and generating target masks or bounding boxes. Training is performed using optimization methods aimed at minimizing the loss function characterizing the detection and / or segmentation error of objects.

[0075] To improve the algorithm's resilience to changes in shooting conditions, data augmentation methods can be used during the training process, including changing illumination, adding noise, geometric transformations, and modeling various types of terrain surfaces.

[0076] The trained model is used to process incoming real-time video streams and generate a detection result in the form of a segmentation mask and / or a probabilistic assessment of the presence of an explosive object.

[0077] 3. Detection and georeferencing of objects.

[0078] Evaluates the segmentation mask of an object and its probability of presence. False positives are eliminated using temporal filtering (an object is only confirmed if it is detected across multiple frames).

[0079] The coordinates of the object are determined by the back projection method, which takes into account:

[0080] coordinates and orientation angles of the UAV,

[0081] digital terrain model.

[0082] Each detection is recorded in the database with the following information: coordinates, object size, spectral signature, source frame.

[0083] 4. Forming a minefield map.

[0084] - Based on the received data, an orthophotomap of the surveyed area (GeoTIFF) is created.

[0085] - Detected objects are plotted in Shapefile (points, hazard polygons).

[0086] - Data from multiple spans is combined using SLAM.

[0087] - The results are visualized:

[0088] for the operator - video stream with 2D marks;

[0089] For the sapper - an AR interface (tablet or AR glasses) with 3D virtual markers and danger zones with a radius of 15 m.

[0090] - The map is updated in real time as the UAV flies.

[0091] Signs of detection of min.

[0092] The object is identified by a combination of: thermal anomaly (mines heat up / cool down differently than the ground), spectral contrast of vegetation (RedEdge / NIR), microrelief (soil deformations), and the presence of foreign material (metal / plastic). The combination of these features and stream synchronization ensures high detection accuracy in open areas.

[0093] As a result of using the claimed method, it is possible to increase the accuracy of mine detection through multispectral analysis, reduce processing time due to data transmission to the ground cluster and real-time processing, reduce the number of false alarms through combined feature analysis and time filtering, and provide operational information to the operator and sapper through visualization in augmented reality.

[0094] Advantages of the declared solution:

[0095] Using a powerful ground cluster instead of an onboard computer to process “heavy” deep learning models.

[0096] Hardware stream synchronization allows for correct processing of multispectral data.

[0097] Segmentation mask allows you to determine the exact shape, size and position of the min.

[0098] Real-time map and marker updates speed up decision making.

[0099] The method is applicable for surveying various types of territories (deserts, forests, snow) and allows for safe detection of minefields with high accuracy and efficiency.

[0100] The claimed method is implemented using a system for detecting explosive objects and marking a minefield (Fig. 1), containing:

[0101] Unmanned aerial vehicle;

[0102] multispectral camera;

[0103] transmitting module;

[0104] a ground-based computing cluster comprising at least one graphics processing unit;

[0105] while:

[0106] The multispectral camera is designed with the ability to generate multiple spectral video streams;

[0107] The system is equipped with hardware synchronization of video streams;

[0108] The transmitting module is capable of transmitting synchronized video streams to the ground cluster;

[0109] The computing cluster is capable of processing video streams in real time using a deep learning algorithm with feature fusion;

[0110] The system is designed with the ability to determine the coordinates of objects and display them in augmented reality.

[0111] These application materials present a preferred implementation of the claimed technical solution, which should not be used as limiting other particular embodiments of its implementation that do not go beyond the scope of the requested scope of legal protection and are obvious to specialists in the relevant field of technology.

Claims

1. A method for detecting explosive objects and marking a minefield, comprising the following steps: - carry out a survey of the territory using an unmanned aerial vehicle; - obtain images of the earth's surface using a multispectral camera and process the resulting images using a machine learning algorithm, while: - from the unmanned aerial vehicle, several video streams corresponding to different spectral channels of the multispectral camera are simultaneously transmitted to the ground-based computing cluster; - synchronize the specified video streams with each other using a precise time synchronization protocol and a common sensor synchronization signal; - video streams are multiplexed and transmitted via a broadband digital radio channel while maintaining temporal correlation between channels; - after which they are processed in real time on a ground-based computing cluster containing at least one graphics computing processor; - in this case, the processing is performed using a deep learning algorithm that extracts features from each spectral channel and combines these features to identify anomalies corresponding to explosive objects; - after detecting an anomaly, the coordinates of the detected object are determined by comparing the position of the image pixels with the telemetry data of the unmanned aerial vehicle; - and display detected objects on the video stream for the operator in the form of virtual augmented reality markers; - while surveying the area, a geo-referenced map of the location of the discovered explosive objects is created.

2. The method according to claim 1, characterized in that the multispectral camera contains at least six spectral channels.

3. The method according to paragraph 2, characterized in that said spectral channels include: visible RGB channel, RedEdge channel, near infrared channel, shortwave infrared channel, longwave infrared channel, panchromatic channel.

4. The method according to paragraph 1, characterized in that the image processing is performed by a hybrid neural network containing: convolutional neural network for local feature extraction; transformer architecture for combining features from different spectral channels.

5. The method according to paragraph 1, characterized in that the combination of information from different spectral channels is performed at the feature level (feature fusion).

6. The method according to paragraph 1, characterized in that the result of the processing is a segmentation mask of the detected object and the probability of its presence.

7. The method according to paragraph 1, characterized in that false alarms are eliminated by means of temporal filtering, in which the detected object is confirmed only when it is detected in several consecutive frames.

8. The method according to paragraph 1, characterized in that the coordinates of the detected objects are determined by the method of reverse photogrammetric projection, taking into account the coordinates of the unmanned aerial vehicle, the orientation angles of the camera and the digital terrain model.

9. The method according to paragraph 1, characterized in that an orthophotomap of the surveyed territory is additionally formed.

10. The method according to paragraph 9, characterized in that the coordinates of the detected explosive objects are applied to the orthophotoplan.

Images