Method and system for aerial patrol

Abstract

Provided is a method for air patrol comprising the steps of: - determining a patrol route for at least one aircraft apparatus; - directing at least one aircraft apparatus along a predetermined route and, obtaining aerial image data of a terrain with reference to the coordinates and time of image acquisition; - processing, while in flight, image data using a pre-trained machine learning algorithm; - performing, using a machine learning algorithm, image segmentation, pattern recognition and classification of recognized objects along a predetermined patrol route; - identifying at least one type of incident and forming an incident log entry; - verifying at least one of the identified incidents; - transmitting, while in flight, incident log entries to the information and analytical platform server. The technical result is increased efficiency and speed of identifying incidents during aerial patrol, reduced traffic transmission and load on the data transmission network.

Description

[Method and system for aerial patrol]

[0001] The present technical solution relates to the field of aerial monitoring of territories as well as linear infrastructures located thereon including, in particular, pipelines, electric power transmission lines, railway lines, roadways, and may be used for automated detection of incidents.

[0002] To date, there are many methods and systems for aerial patrol by means of either manned or unmanned aircraft apparatuses. The objects subject to aerial patrol may be areas of terrain and objects of various sizes and lengths, in particular, an object may be a section of a pipeline of 500 km long or a forest area of 10,000 km2. For patrolling such objects, important criteria are the flight range of the aircraft apparatus, the accuracy and speed of monitoring and detecting incidents, and the speed of response to detected incidents.

[0003] For example, Russian patent No. RU 2699940 (published on 11.09.2019) discloses a method for monitoring the state of a main pipeline, comprising performing aerial photography with concurrent recording of satellite coordinates of photographing points; linking photographs to the route and objects of the main pipeline in a geographic information system; building orthophotomaps or digital terrain models by using triangulation of photographic data. Recording of satellite coordinates of photographing points is performed by using a geodetic-class global satellite navigation system followed by processing the obtained measurement results using the precise positioning method and assigning, to each photograph, the satellite coordinates obtained as a result of processing. The photographs are referenced to the route and main pipeline objects by way of importing same into a geographic information system having a pre-imported main pipeline route diagram. Next, the method comprises selecting in an automated mode, from the array of georeferenced photographs, photographs of objects of the main pipeline including facts of violations and unauthorized access attempts along the main pipeline route, localizing and noting the detected facts in the geographic information system; according to the built orthophotomaps or digital models of the terrain, detecting and noting, in the geoinformation system, dangerous geological processes occurring along the main pipeline route; overlaying images of orthophotomaps or digital models of the terrain having hazardous geological processes occurring along the main pipeline route and made in different time periods and having the same coordinates; measuring the geometric parameters of hazardous geological processes occurring along the main pipeline route in order to further monitor the magnitude and speed of altering thereof. Based on the measurements obtained, comprised are forecasting the dynamics of the development of dangerous geological processes occurring along the main pipeline route, and making decisions on elimination thereof. The equipment required to implement the method of monitoring the condition of the main pipeline route is disposed on board a helicopter by using vibration reduction structures. The helicopter takes off in the air to fly over the main pipeline route. The condition of the main pipeline route is monitored at a flying altitude of an aircraft apparatus of 200-300 m above the ground at a speed of 100-180 km / h. Concurrently, there is being carried out visual inspection of the main pipeline route by an observer and aerial photography.

[0004] The disadvantage of this known method is the use of post-processing of patrol results, whereas the operational monitoring is carried out using an observer-operator located on board a helicopter. Further, generating a patrol report and detecting incidents that the operator could not detect during the visual inspection takes a significant amount of time. Furthermore, given a significant length of the patrol area, for example, 400 km, the operator may fail to detect some incidents requiring prompt response because of the human factor and the flying altitude of 200-300 m above ground level. A further disadvantage of this method may be an unstable communication system at the time of detection of an incident, not allowing the operator to accurately and promptly report the coordinates of the incident, whereas information from the array of geo-referenced photographs may be obtained and processed on the ground, that is, upon completing the patrol, landing the aircraft apparatus, transferring the data to the server and processing the data. Thus, the prior-art method does not allow for prompt and accurate monitoring allowing patrol reports while in flight.

[0005] Russian patent No. RU2281534 (published 10.08.2006) discloses a method of condition inspection of product pipelines, comprising scanning the area of the product pipeline by means of thermal imaging and visual cameras from an aircraft apparatus (for example, Mi-8T), laser-scanning of the terrain, and positioning the aircraft apparatus. The obtained data are used to generate a comprehensive model of thermal and visual image of the surface of the product pipeline area and to detect gas leaks. The flight is carried out at an altitude of 100-150 meters along a route imported into the navigation program.

[0006] The thermal information data obtained during the image acquisition process is viewed in the original state thereof, without any processing, to detect leaks. Therefore, the search for leaks may be carried out immediately after the flight, and upon gaining practical experience, this operation may be assigned to the flight operator and carried out while in flight on the control monitor.

[0007] This prior-art method is intended to detect leaks, however it does not enable accurate and prompt detection of other possible incidents that may be typical for pipelines and other possible patrol objects. Monitoring accuracy substantially comes down to operator's experience and attention, who may fail to detect some incidents, misidentify individual incidents, or fail to identify some incidents that require prompt response. Detailed processing may be carried out only after patrol completion, thus also not making it possible to form a patrol report while in flight as the aircraft apparatus moves along the patrol route which may be hundreds of kilometers.

[0008] Russian patent No. 2780304 (published on 21.09.2022) discloses a method for monitoring the technical condition of gas transport objects based on a geographic information system. This prior-art system for aerial patrol includes an unmanned aircraft apparatus by means of which there are performed: conducting aerial photography with concurrent recording of the coordinates of the photographing area, then building orthophotomaps using the method of triangulation of photographic data, coordinate-referencing, using the aerial photograph, gas transport objects in a geographic information system, comparing orthophotomap images to operational data in the same coordinates, searching for violations by visual method on aerial photographs, and recording the violations in the system. Data analysis includes post-processing with visual inspection, as well as obtaining additional data from sensors located on the personnel.

[0009] The disadvantage of this prior-art system is the low speed of response to incidents due to the use of post-processing methods upon completion of patrolling an object. Furthermore, obtaining additional data for incident identification involves installing sensors on personnel and having personnel in the area of the object being monitored, on the ground. Said technology has low versatility and may be used in industrial conditions but does not enable the inspection of large areas and linear infrastructures, such as, for example, a gas pipeline. The length of a gas pipeline may be hundreds and thousands of kilometers, in which case the presence of personnel or the installation of sensors along the entire length of the pipeline may be difficult and impractical from the point of view of patrol speed and the promptness of obtaining data on gas leaks, as well as from the point of view of the possibility of collecting data, as the entire length of the pipeline may not have a stable network signal for data transmission.

[0010] Chinese patent No. CN115457411 (A) (published on 09.12.2022) discloses a system for aerial monitoring of pipelines, comprising an aircraft apparatus that provides for the obtaining of photo / video data on the condition of a pipeline while moving along a predetermined route and for transmission of data to a ground station for processing and transmitting same to a central server by means of a data transmission network followed by generation of alarm signals for timely maintenance of the pipeline. The data may include obtained images and image acquisition coordinates. In this prior-art solution, data processing is performed not on board the aircraft apparatus but at a ground station situated in the pipeline patrol area and accompanying the aircraft apparatus.

[0011] The disadvantage of this prior-art solution is the significant traffic transmitted over the data transmission network; in addition, the linear infrastructures such as pipelines, roads, and railways may typically include sections of limited data transmission speed or no network at all, not allowing for prompt transmission of data to be processed, and reducing the speed / quality of processing, and the efficiency of response to incidents. Moreover, if there is insufficient data for analysis, it will be required to launch an additional aircraft apparatus to re-patrol the area, thus also reducing the speed of incident identification. If the data to be processed is insufficient or of low quality, there may be a false identification of an incident, or some of the existing incidents may not be identified. This may be due to detection deficiencies as well as low speed or errors in real-time data transmission from the aircraft apparatus to the ground station.

[0012] Therefore, there is a need to improve existing systems and methods for aerial patrol.

[0013] The object of the present technical solution is to overcome at least some of the prior-art disadvantages.

[0014] The first object of the present invention is a system for aerial patrol, including at least one aircraft apparatus configured to traverse a predetermined patrol route, said aircraft apparatus comprising:

[0015] - a detection module configured to obtain aerial image data of a terrain with reference to the coordinates and time of image acquisition;

[0016] - a processing module configured to obtain image data of a terrain from the detection module, process same while in flight and identify incidents on the terrain using at least one pre-trained machine learning algorithm so as to form an incident log including at least the type and location of the incident on the patrol route, wherein the processing module is configured to further verify an incident, which has been identified by means of the pre-trained machine learning algorithm, while in flight, by way of obtaining and processing additional data from the detection module and / or by way of obtaining user input from the operator;

[0017] - a data transmission module configured to transmit, while in flight, incident log entries by means of a data transmission network to at least one information and analytical platform server.

[0018] In one possible embodiment of the system, the detection module is configured to obtain images in the visible and infrared spectra, and the processing module is configured to identify incidents based on terrain image data in the visible and infrared ranges.

[0019] In one possible embodiment of the system, the detection module further comprises one or more of: a laser scanner, object auto-tracking device, gas analyzer, radiation-chemical reconnaissance device, and the processing module is configured to obtain additional data from the detection module and use same to identify incidents using a pre-trained machine learning algorithm.

[0020] According to one possible embodiment, the system further comprises at least one database comprising at least one set of historical image data of a terrain along a predetermined patrol route, the processing module is further configured to obtain the historical data and identify incidents on the terrain when comparing, by means of a pre-trained machine learning algorithm, between current data obtained from the detection module and historical data.

[0021] According to one possible embodiment, the system comprises a data storage module associated with the detection module and the processing module and configured to at least partially store data obtained by the detection module and / or data processed by the processing module.

[0022] According to one possible embodiment, the system further comprises an automated operator workstation configured to communicate with the processing module and verify the incident by way of outputting to the operator the terrain image data from the detection module and / or the result of data processing and of incident identification, and to obtain user input from the operator.

[0023] In one possible embodiment of the system, the aircraft apparatus is a manned or unmanned aircraft apparatus, and the automated operator workstation is disposed in the aircraft apparatus.

[0024] In one possible embodiment of the system, the aircraft apparatus is a manned or unmanned aircraft apparatus, and the automated operator workstation is configured to remotely communicate with the processing module by means of a data transmission network.

[0025] According to one possible embodiment of the system, the processing module, by means of the data transmission module, is further configured to transmit to the information and analytical platform server at least one frame of terrain image data, which contains the identified incident, for verification of the incident.

[0026] According to one possible embodiment of the system, the aircraft apparatus is configured to at least partially alter the predetermined patrol route for additional verification of the incident by means of the obtaining of a command from the processing module and / or from the automated operator workstation and / or from the information and analytical platform server.

[0027] According to one possible embodiment of the system, the aircraft apparatus is configured to alter at least one patrol parameter for additional verification of the incident by way of obtaining a command from the processing module and / or from the automated operator workstation and / or from the information and analytical platform server.

[0028] According to one possible embodiment of the system, the patrol parameter of an aircraft apparatus includes one or more of: speed of movement, altitude, direction of movement, maneuvering, pitch.

[0029] According to one possible embodiment, the system includes at least one additional aircraft apparatus configured to traverse a route at least partially coinciding with a predetermined patrol route, wherein the processing module of the first aircraft apparatus is configured to further verify the incident by means of the obtaining, by means of the data transmission network, of additional data from the detection module installed on the additional aircraft apparatus.

[0030] According to one possible embodiment, the system includes at least one information and analytical platform server, which is configured to obtain incident log entries by means of the data transmission network from at least one aircraft apparatus being present in flight and traversing a predetermined patrol route, wherein the server is configured to retrieve the log entries and form digital models of incidents identified on the terrain while at least one aircraft apparatus is traversing a predetermined patrol route.

[0031] According to one possible embodiment of the system, the information and analytical platform server is further configured to obtain at least one frame of terrain image data comprising an identified incident for verifying the incident and / or forming a digital model of at least one incident identified on the terrain while at least one aircraft apparatus is traversing a predetermined patrol route.

[0032] Another object of the present invention is a method for aerial patrol, comprising the following steps:

[0033] - determining a patrol route for at least one aircraft apparatus comprising a detection module, a processing module and a data transmission module;

[0034] - directing at least one aircraft apparatus along a predetermined route and, by means of the detection module, obtaining aerial image data of a terrain with reference to the coordinates and time of image acquisition;

[0035] - processing, while in flight, image data using the processing module comprising a pre-trained machine learning algorithm;

[0036] - performing, using a machine learning algorithm, image segmentation, pattern recognition and classification of recognized objects along a predetermined patrol route;

[0037] - identifying at least one type of incident and forming an incident log entry comprising at least data on the type and location of the identified incident;

[0038] - verifying at least one of the identified incidents by way of obtaining and processing additional data from the detection module and / or obtaining an operator command;

[0039] - by means of the data transmission module, transmitting, while in flight, incident log entries by means of the data transmission network to the information and analytical platform server.

[0040] In one possible embodiment of the method, the step of determining the route further comprises determining incident types to be identified, and the step of data processing comprises identifying the predetermined incident types.

[0041] In one possible embodiment of the method, the method comprises verifying an identified incident corresponding to at least one of the predetermined types of incidents.

[0042] In one possible embodiment of the method, the step of determining a route further comprises determining the type of patrol object, and the step of data processing comprises identifying the types of incidents related to the predetermined type of patrol object.

[0043] In one possible embodiment of the method, the step of determining a route comprises recording at least one set of historical terrain image data along a predetermined patrol route, the step of processing further comprises obtaining historical data and identifying incidents on the terrain by comparing, using a pre-trained machine learning algorithm, current data obtained from the detection module and historical data of the corresponding area of the terrain.

[0044] In one possible embodiment of the method, the types of incidents to be identified include at least one or more of the following: smoke, fire, fire effects, landslide, avalanche, pipeline rupture, pipeline exposure, washaway, flooding, icing, leak, gas emission, unauthorized dumping, man-made accident, water body pollution, deforestation, regenerated forest corridors, stationary foreign objects, mobile foreign objects, line break, damage to elements of linear infrastructures, presence of defects in linear infrastructures and coatings.

[0045] In one possible embodiment of the method, the type of patrol object includes at least one or more of the following: pipeline, railway line, electric power transmission line, communication line, forest area, settlement area, agricultural land, body of water, mountain range, industrial zone.

[0046] In one possible embodiment of the method, the step of obtaining aerial image data of a terrain with reference to the coordinates and time of image acquisition comprises obtaining images in the visible and infrared spectrum, and the step of processing comprises identifying at least one type of incident based on the obtained image in the visible and / or infrared spectrum.

[0047] In one possible embodiment of the method, the method comprises obtaining, by means of the detection module, additional data by means of one or more means selected from: a laser scanner, object auto-tracking device, gas analyzer, radiation-chemical reconnaissance device, and the processing step comprises identifying at least one type of incident based on the additional data obtained.

[0048] In one possible embodiment of the method, the method comprises recording at least a portion of the image data which is obtained by the detection module.

[0049] In one possible embodiment of the method, the method comprises mapping a predetermined patrol route in the visible and / or infrared ranges by means of the detection module and the processing module on board the aircraft apparatus.

[0050] In one possible embodiment of the method, the step of verifying an identified incident comprises transmitting an incident log entry together with image data of the corresponding identified incident by means of the data transmission network to the information and analytical platform server and / or the automated operator workstation.

[0051] In one possible embodiment of the method, the step of verifying an incident comprises outputting the image data of a terrain from the detection module and / or the result of data processing from the processing module to the automated operator workstation disposed in the aircraft apparatus.

[0052] In one possible embodiment of the method, the step of verifying an incident comprises obtaining, from the information and analytical platform server, user input or a command including confirmation or denial of the incident.

[0053] In one possible embodiment of the method, the step of verifying an identified incident comprises at least partially altering the predetermined patrol route and obtaining additional image data for additional verification of the identified incident by way of obtaining a command from the processing module and / or from the automated operator workstation and / or from the information and analytical platform server.

[0054] In one possible embodiment of the method, altering the predetermined patrol route includes at least one or more of the following: altering the course, returning to the area of the identified incident, performing additional patrols of the area of the identified incident with an expansion of the patrol radius, descending in the area of the identified incident, landing an aircraft apparatus in the area of the identified incident.

[0055] In one possible embodiment of the method, the step of verifying an identified incident comprises altering at least one patrol parameter and obtaining additional image data for additional verification of the identified incident by way of obtaining a command from the processing module and / or from the automated operator workstation and / or from the information and analytical platform server.

[0056] According to one possible embodiment of the method, the patrol parameter of an aircraft apparatus includes one or more of: speed of movement, altitude, direction of movement, maneuvering, pitch.

[0057] In one possible embodiment of the method, the step of verifying an identified incident comprises transmitting a request by means of the data transmission module, the request comprising the coordinates and type of the identified incident, to at least one additional aircraft apparatus traversing a route that at least partially coincides with the predetermined patrol route, and obtaining, in response to the request, additional image data associated with the identified incident from the detection module installed on an additional aircraft apparatus.

[0058] In one possible embodiment of the method, the method comprises processing additional image data associated with the identified incident by means of the processing module, and confirming or denying the identified incident.

[0059] In one possible embodiment of the method, the method comprises forming, in response to the confirmation of the identified incident, an incident log entry comprising at least data on the type and location of the verified identified incident.

[0060] In one possible embodiment of the method, the method comprises, in response to the denial of an identified incident, deleting an entry from the incident log or adding an entry about a falsely identified incident.

[0061] In one possible embodiment of the method, the method further comprises transmitting, at the step of transmitting log entries while in flight, at least one image or a sequence of multiple images for the verified identified incident.

[0062] In one possible embodiment of the method, the method further comprises determining, at the step of determining a patrol route, the coordinates of the control areas, and transmitting data on incident log entries when the aircraft apparatus reaches the control area.

[0063] In one possible embodiment of the method, the method comprises obtaining the incident log entries by means of the data transmission network on at least one information and analytical platform server and further processing the obtained data and forming a report on the identified incidents along the patrol route.

[0064] In one possible embodiment of the method, the method comprises obtaining incident log entries by means of a data transmission network on at least one information and analytical platform server and forming a digital model of incidents identified on the terrain while at least one aircraft apparatus is traversing a predetermined patrol route.

[0065] In one possible embodiment of the method, the method further comprises processing said digital model using a decision support algorithm and automatically generating a scenario for responding to incidents identified on the terrain while at least one aircraft apparatus is traversing a predetermined patrol route.

[0066] In one possible embodiment of the method, the method comprises obtaining at least one image or a sequence of multiple images for a verified identified incident by means of a data transmission network on at least one information and analytical platform server, processing the obtained data and performing additional training of the machine learning algorithm based on the verified identified incident.

[0067] The technical result of the subject technology may be, in particular, increased efficiency and speed of identifying incidents during aerial patrol, reduced traffic transmission and load on the data transmission network, as well as overcoming of at least some of the prior art disadvantages.

[0068] Further and / or alternative features, aspects and advantages of the embodiments of subject technology will be apparent from the description to follow, accompanying figures and accompanying claims.

[0069] is a non-limiting embodiment of a system suitable for implementing the subject technology.

[0070] is a block diagram of the subject method for aerial patrol.

[0071] is an illustrative example of obtaining images in the visible and infrared spectrum.

[0072] Figs. 4A-4B are an illustrative example of semantic segmentation of a terrain image on a patrol route.

[0073] is an illustrative example of incident identification.

[0074] All examples and conditional language recited herein are intended for aiding the reader in understanding the concepts and main aspects of the present technology and are to be construed as being without limitation.

[0075] Furthermore, the following describes rather simplified embodiments of the technology in order to provide a clear understanding. As those skilled in the art would understand, many embodiments of the present technology may be of a greater complexity.

[0076] Terms and definitions used in the description herein

[0077] As used herein, "aircraft apparatus" means a manned aircraft apparatus or an unmanned aircraft apparatus (UAA). Examples of manned aircraft apparatuses are, for example, an airplane, a helicopter, a glider, a hang glider, an airship, a hot air balloon, and the like. The unmanned aircraft apparatus may be configured to fly through the air in automatic mode, i.e. without human participation or external control sources, or may be configured to move through the air in semi-automatic mode, i.e. to obtain at least some control commands from a human being (e.g., a pilot, operator, and the like) or an external source (e.g., a control panel, control server, external control device, and the like) by means of predetermined communication channels. Unlimited examples of UAAs are a variety of multi-rotor UAAs, for example, multicopter drones; single-rotor UAAs, for example, unmanned helicopter; hybrid UAAs, for example, rotary wing drones; and the like. In the context of the claimed technical solution, the type and design of the aircraft apparatus are not specifically limited in any manner and will not be described in detail. It is however worth noting that in order to implement the subject method and patrol system, the aircraft apparatus must be configured to carry a payload, specifically modules, units and other elements, which will be described in more detail below. In certain embodiments, the aircraft apparatus (whether unmanned or manned) may be configured to accommodate an automated operator workstation and the operator (user) themselves in the fuselage; in other embodiments, the presence of an automated operator workstation and the operator themselves on board the aircraft apparatus may be optional. In certain embodiments of the manned aircraft apparatus, the automated operator workstation may be combined with the aircraft apparatus control elements, and the operator function may be at least partially performed by a pilot or a passenger or other subject present on board the aircraft apparatus.

[0078] As used in the present invention, the term "module", unless the description herein clearly dictates otherwise, refers to a functional element or a combination of functional elements of a device in the form of a component portion, node, block or other assembly unit that performs certain technical functions that provide for the functioning of the device. The module may generally be implemented in practice using a combination of known structural elements, a combination of known structural elements and known hardware, a combination of known structural elements and known software and hardware or a combination of known hardware and known software. So, for example, the detection module is configured to obtain aerial image data of a terrain with reference to the coordinates and time of image acquisition, wherein the detection module may be implemented using a combination of known structural elements, a combination of structural elements and hardware, a combination of structural elements and software and hardware or a combination of hardware and software. In particular, a combination of a photo / video camera, an infrared camera (thermal imager), means of location determination (GPS, GLONASS, GSM Galileo, Beidou, and the like) or means of access to the respective aircraft apparatus location data. The processing module may be implemented using software and hardware, and may be a physical device, an apparatus, or a plurality of modules implemented using hardware, for example, using an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA), or a combination of hardware and software, for example, using a microprocessor system and a set of instructions implementing the functionality of the processing module, which (when executed) transform the microprocessor system into an application-specific device or system (for example, on-board data processing / air patrol server). The data storage module refers to any type and nature of storage media, including RAM, ROM, disks (CD-ROM, DVD, floppy disk, hard disk, and the like), USB flash drives, solid state drives, magnetic tape drives, and the like. Furthermore, each of the modules, or at least one of them, may also be implemented in the form of a combination of hardware and software, wherein some of the functionality described herein with respect to one of the modules may be implemented by means of hardware only, whereas other functionality described herein with respect to the same module or other module may be implemented by way of hardware in combination with software.

[0079] As used in the present description, "database" refers to any structured data set that does not depend on a specific structure, database management software, hardware of the computer that stores the data, uses the data or otherwise makes the data available for use. The database may be present on the same hardware running the process that stores or uses the information stored in the database, or it may be present on separate hardware, for example, a dedicated server or plurality of servers.

[0080] As used in the present patent application, the words "first", "second", "third", etc. are used as ordinal numbers only to show the difference between nouns that differ from each other, and not for the purpose of describing any particular relationship of data between these nouns. For example, "first aircraft apparatus" and "additional aircraft apparatus" may substantially represent the same aircraft apparatuses or at least partially different from one another aircraft apparatuses, whereas the words "first" and "additional" are used to improve the clarity in the description of individual embodiments of the claimed technical solution.

[0081] In the context of the present description, "image data" may represent static or dynamic images (video) or a sequence of multiple frames and / or video recordings. The image data may contain data of image acquisition in the visible and / or other spectrum, in particular infrared spectrum. The image data may be subjected to processing and may also further comprise a set of metadata, such as metadata on recognized objects or image acquisition coordinates. The image data may be provided in any format that may be stored in a database and transmitted over a data network. The image data may comprise, inter alia, audiovisual content (images, videos), data (location data, digital data on image acquisition parameters, on timestamps, and the like), text (descriptions / locations of recognized objects), and the like.

[0082] In the context of the present description, "information and analytical system server" means a computer program running on the corresponding equipment, which program is capable of obtaining requests or data (for example, from a processing module installed on at least one aircraft apparatus, an automated operator workstation, a data transmission module, and the like) over a data transmission network and executing said requests or initiating the execution of said requests or initiating the processing of the obtained data. The hardware may be a single physical computer or a single physical computer system, but neither one is required for the subject technology. In the present context, the use of the term "information analysis system server" or "server" does not mean that every task (e.g. obtained instructions or requests) or any particular task will be obtained, executed or initiated to be executed by the same server (i.e. the same software and / or hardware); it means that any number of pieces of software or hardware may be involved in obtaining / transmitting, executing or initiating the execution of any request or the consequences of any request, and all of this software and hardware may be one server or multiple servers, and the both alternatives are included in the term "at least one server".

[0083] As used herein, "automated operator workstation" (operator AWS) means any computer device capable of operating with software suitable for solving the respective task and providing for obtaining of user input by means of at least one input means or input-output means, as well as for outputting information to the user by means of at least one output means or input-output means. Thus, examples of operator AWSs may be, inter alia, personal computers (desktop computers, laptops, netbooks, and the like), smartphones, tablets, smart watches, augmented reality glasses / helmets. Of note, a device that behaves as an operator AWS in this context may behave as a server with respect to other electronic devices and may, in particular, be a server or one of the servers of an information and analytical system. The use of the expression "automated operator workstation" does not exclude the possible use of a plurality of electronic devices or a plurality of automated operator workstations to obtain / transmit, perform or initiate the performance of any task or request, or the consequences of any task or request, or the steps of the claimed method.

[0084] In the context of the present description, the term "incident" may refer to one or more factors or events whose detection is the purpose of the aerial patrol and area monitoring being conducted. The nature, type and other characteristics of the incident within the framework of the subject technical solution are not specifically limited in any manner and may be determined at the step of determining the patrol route, training the machine learning algorithm, and the like. Furthermore, in certain embodiments, identified as an incident may be any anomaly, for example, by comparing, by means of a pre-trained machine learning algorithm, between current data obtained from the detection module and historical data obtained previously for the same patrol area. Non-limiting examples of incidents include smoke, fire, fire effects, landslide, avalanche, pipeline rupture, pipeline exposure, washaway, flooding, icing, leak, gas emission, unauthorized dumping, man-made accident, water body pollution, deforestation, regenerated forest corridors, stationary foreign objects, mobile foreign objects, line break, damage to elements of linear infrastructures, presence of defects in linear infrastructures and coatings. An incident may be an ongoing event, such as a forest fire, an event that has already occurred (burnt areas resulted from a forest fire), and an event that may be predicted to occur based on individual factors, such as the detection of individuals making a fire in a forest area (it may contribute to a forest fire). Furthermore, in the context of the claimed technical solution, an incident may be either a negative event (examples of which are provided above) or a positive event, in particular the detection of a person or other objects during a search and rescue operation.

[0085] In the context of the present description, the term "incident identification" refers to the automated detection of a suspected incident while processing data being obtained by the detection module and being processed by the machine learning algorithm of the processing module. The accuracy of incident identification may depend on the quality of the data being obtained and on the quality of the pre-trained machine learning algorithm. In the context of the present description, the term "verification of an identified incident" means further verification of an identified incident by way of obtaining additional data on the suspected incident to confirm or deny same by one or more of: an operator located in the aircraft apparatus, a remote operator, the same or a distinct machine learning algorithm.

[0086] In the context of the present description, the term "incident log" is understood to mean a file with incident entries, the file comprising at least the type and location of an identified incident. The file format is not specifically limited in any manner and may be a .log, .txt, .xml, or other file format, including a special format developed for a specific aerial patrol system. As additional information, the file may also comprise characteristics and an additional description of the incident that were obtained by the detection module or recognized by the processing module or entered by the operator by means of the automated operator workstation. Information from the incident log may be transmitted over a data transmission network, extracted and processed on other equipment, in particular on the information and analytical system server.

[0087] In the context of the present description, the term "machine learning algorithm" may represent one or more algorithms capable of performing the respective steps of the subject method, in particular, processing the obtained image data, segmenting the images, recognizing patterns and classifying the recognized objects along a predetermined patrol route. Non-limiting examples of known algorithms include Fast R-CNN, Faster R-CNN, R-CNN, Mask R-CNN, YOLO, all of which are based on convolutional neural networks. The algorithm may be trained on pre-labeled data of a variety of identified incidents or on unlabeled data in a supervised, reinforcement-learning or unsupervised manner.

[0088] In the context of the present description, the term "digital incident model" means a digital representation of an object within a specific area of terrain; the digital model may include one or more visual and / or semantic layers comprising data on identified incidents and locations thereof. The digital incident model may be one or more layers of a digital terrain map which includes one or more of: a satellite imagery layer of the terrain, an aerial photography layer in the visible range, an aerial photography layer of the terrain in the infrared range, a semantic layer containing a description of objects on the terrain, a semantic layer containing a terrain plan, a historical layer containing a retrospective of the terrain map.

[0089] In the context of the present description, "location" means, in particular, a point or area defined by specific geographic coordinates with a predetermined degree of accuracy. For example, as a geographic location there may be selected a point or area along a patrol route at which an incident has been identified. The above examples have been provided solely for understanding purposes and should not be construed as limiting.

[0090] The list of illustrative examples provided herein is not intended to be exhaustive, and those skilled in the art may conceive other modifications without departing from the scope of the present technology. Further, where no examples of modifications have been set forth, it should not be interpreted that no modifications are possible and / or that what is described is the sole manner of implementing that element of the present technology.Description of the system

[0091] Hereinafter, some exemplary embodiments of the present invention will be described with reference to the accompanying drawings; however, it should be understood that the description below does not define or limit the scope of the present invention.

[0092] In the following description, a detailed description of known functions and designs will be omitted as this unimportant information may obscure the concept of the present invention.

[0093] is a schematic diagram illustrating an aerial patrol system 100 in accordance with one embodiment of the subject technology. The system 100 comprises at least one aircraft apparatus 102. The design of the aircraft apparatus 102 is not limiting with respect to the implementation of the claimed invention. In particular, as the aircraft apparatus 102 used may be a known manned or unmanned aircraft apparatus capable of carrying a payload and travel over a predetermined distance along a patrol route. For example, the aircraft apparatus 102 may be Airbus AS350, MI171, Robinson R44 helicopter, and the like; in certain embodiments, the aircraft apparatus 102 may be a manned single-engine aircraft, such as Cessna 172, and the like. As the aircraft apparatus 102 also used may be unmanned aircraft apparatuses of helicopter, multicopter, airplane, or other type, of various designs, such as the Aerostar manufactured by Aeronautics LTD, the Alpha A900, and the like.

[0094] The system 100 further comprises the following modules, which may be installed on board the aircraft apparatus 102, for example by means of a hanger or may be at least partially integrated into the housing of the aircraft apparatus 102.

[0095] A detection module 104 configured to obtain aerial image data of a terrain with reference to the coordinates and time of image acquisition. The detection module 104 is preferably installed on an external side of the housing of the aircraft apparatus 102 and comprises at least one sensor, for example one camera for obtaining photo / video images. In certain embodiments, the detection module 104 is configured to obtain images in the visible and infrared spectrum. In certain embodiments, the detection module 104 may comprise multiple cameras, such as two IR cameras for image acquisition in the infrared spectrum and one camera for image acquisition in the visible spectrum. The cameras may be installed at different angles, or one or more cameras may be installed on a rotating base so as to controllably alter the direction and angle of image acquisition. In certain embodiments, the detection module 104 further comprises one or more of: a laser scanner, object auto-tracking device, gas analyzer, radiation-chemical reconnaissance device. Thus, the detection module 104 may be configured to obtain additional data and measurements that may be used in subsequent processing in certain embodiments of the claimed method for aerial patrol.

[0096] The processing module 106 installed interiorly to the housing of the aircraft apparatus 102 or secured onto the housing of the aircraft apparatus 102 is configured to obtain image data of a terrain from the detection module 104. The processing module 106 comprises at least one processor that has access to software instructions, while executing same, the processing module 106 is configured to process while in flight and identify incidents on the terrain by means of at least one pre-trained machine learning algorithm to form an incident log, and also to carry out the corresponding steps of the method, which method will be described in more detail below. Further, the formed incident log comprises at least the type and location of the incident along the patrol route.

[0097] The processing module 106 is configured to further verify an incident, which has been identified by means of the pre-trained machine learning algorithm, while in flight, by way of obtaining and processing additional data from the detection module 104 and / or by way of obtaining user input from the operator.

[0098] The system comprises a data transmission module 108 configured to transmit, while in flight, incident log entries by means of a data transmission network 400 to at least one information and analytical platform server 200. The data transmission module 108 may be one or more of: a GSM module, 2G, 3G, 4G, 5G communication modules, a specialized carrier-grade wireless broadband data transmission device, a satellite communication system. In certain embodiments, the data transmission module 108 may be coupled, for example, by means of Wi-Fi, ZigBee, Bluetooth to a standard data transmission module of the aircraft apparatus 102. The data transmission module 108 may be a distinct independent wireless data transmission device, for example, at least one 4G LTE modem with one or more antennas installed on the housing of the aircraft apparatus 102. In certain embodiments, the data transmission module 108 may be a GSM module configured to transmit incident log entries, for example, in the form of text messages (SMS). Depending on the intended operating area of the system 100 and the coverage area of the data transmission networks in the aerial patrol area, the system may use one or more data transmission modules 108 of the same or different types. Inter alia, the data transmission module 108 may be configured to transmit data by means of multiple communication channels concurrently or selectively by means of one of the available communication channels.

[0099] In certain embodiments, the system 100 further comprises at least one database comprising at least one set of historical image data of a terrain along a predetermined patrol route, the processing module 106 is further configured to obtain the historical data and identify incidents on the terrain when comparing, by means of a pre-trained machine learning algorithm, between current data obtained from the detection module 104 and historical data. Given the historical data that may be loaded into the memory of the processing module 106 together with the instructions on the predetermined patrol route, the processing module 106 may not only identify already known types of incidents, taking into account the pre-trained algorithm, but also detect any abnormalities and potential new types of incidents that the machine learning algorithm has not yet been trained to identify.

[0100] In certain embodiments, the system further comprises a data storage module associated with the detection module 104 and the processing module 106 and configured to at least partially store data obtained by the detection module 104 and / or data processed by the processing module 106. Saving data allows for carrying out of additional post-processing of results and generation of historical data further to operational processing and transmitting of incident log entries while in flight. This may eliminate the need to transmit significant amounts of data and ensure prompt response to incidents, while maintaining the ability to subsequently use the collected data for additional analysis, training machine learning algorithms, providing complete data to the customer, generating historical data for a predetermined patrol route, and the like.

[0101] In certain embodiments, the system further comprises an automated operator workstation 300 (operator AWS) configured to communicate with the processing module 106 and verify the incident by way of outputting to the operator the terrain image data from the detection module 104 and / or the result of data processing and of incident identification, and to obtain user input from the operator. As shown in, there may be a variety of options for the location of the automated operator workstation 300, or there may be present multiple operator AWSs 300 having distinct or identical functionality and location. The operator AWS 300 may be disposed in the housing of the aircraft apparatus 102, for example, in the cabin of a manned or unmanned aircraft apparatus, for example, a manned helicopter. Further, in certain embodiments, the operator may also be the pilot of the aircraft apparatus 102, and the operator AWS 300 may be installed in the area of the pilot's seat, or the operator AWS 300 may be installed in the area of the passenger compartment and the operator may be a passenger of the aircraft apparatus 102, whether manned or unmanned. The operator AWS 300 may be disposed in the area of the aerial patrol object and installed in a stationary manner, or in a movable manner, for example, in a vehicle. For example, while performing the aerial patrol of a railway, the operator's AWS 300 may be installed in the train's driving cab, whereas while performing roadway patrol, it may be installed in an automobile. It is possible to install one or more stationary operator AWSs 300 along the patrol route. The operator AWS 300 may be disposed as part of the information and analytical platform server 200 or another object accessible by means of one or more data transmission networks, and configured to remotely communicate with the processing module 106 by means of the data transmission network 400.

[0102] Given reduced traffic when transmitting data for verifying an incident, the data transmission may be carried out for any one of the provided locations of the operator AWS 300. Those skilled will appreciate that it may be preferable to dispose the operator AWS 300 in the aircraft apparatus 102 given no restrictions on the data transmission speed from the processing module 106, which is also disposed in the aircraft apparatus 102; in addition, when it is disposed in the aircraft apparatus 102, the operator may carry out additional visual control and verify incidents not only on the basis of data from the processing module 106 and the detection module 104 and other data, including historical data, but also on the basis of their own visual data.

[0103] In certain embodiments, the processing module 106, by means of the data transmission module 108, is further configured to transmit to the information and analytical platform server 200 at least one frame of terrain image data, which contains the identified incident, for verification of the incident. Thus, an incident report, further to information about incident type and location, may comprise additional data in the form of one or more images or video fragments in the visible and / or infrared range. The data transmission network 400 or other network or communication channel may be used to transmit additional data.

[0104] The aircraft apparatus 102 is configured to at least partially alter the predetermined patrol route for additional verification of the incident by means of the obtaining of a command from the processing module 106 and / or from the automated operator workstation 300 and / or from the information and analytical platform server 200. When using an unmanned aircraft apparatus 102, the commands may be transmitted directly to the control unit of the unmanned aircraft apparatus; in the case of using a manned aircraft apparatus 102, the commands may be transmitted to the pilot by means of the navigation system of the aircraft apparatus 102 or other commands, in particular audio / visual commands transmitted to the pilot. Altering the predetermined route may be understood to mean, in particular, at least one or more of the following: altering of the course, return to the area of an identified incident, performance of additional patrol of the area of an identified incident with an expansion of the patrol radius, descent in the area of an identified incident, landing of the aircraft apparatus in the area of an identified incident. Depending on the design and type of the aircraft apparatus 102 used, various options for altering the predetermined patrol route for verification of the identified incident are possible. When using the aircraft apparatus 102 of an airplane type, it may be preferable to re-fly over a particular section of the patrol route, and when using the aircraft apparatus 102 of a helicopter or multi-rotor type, it may be sufficient to reduce the speed and altitude in the area of the identified incident.

[0105] The aircraft apparatus 102 is configured to alter at least one patrol parameter for additional verification of the incident by way of obtaining a command from the processing module 106 and / or from the automated operator workstation 300 and / or from the information and analytical platform server 200. The patrol parameter of an aircraft apparatus includes one or more of: speed of movement, altitude, direction of movement, maneuvering, pitch.

[0106] The system 100 may include at least one additional aircraft apparatus 102 configured to traverse a route at least partially coinciding with a predetermined patrol route, wherein the processing module of the first aircraft apparatus 102 is configured to further verify the incident by means of the obtaining, by means of the data transmission network 400, of additional data from the detection module 104 installed on the additional aircraft apparatus 102. Thus, a group of multiple aircraft apparatuses 102 may carry out aerial patrol and traverse a predetermined patrol route at least partially so as to enable the verification of the identified incident. The communication between the aircraft apparatuses 102 may be carried out by means of corresponding data transmission modules 108 coupled to one another by means of a data transmission network 400 or by means of another short-range network (not shown).

[0107] The system 100 in certain embodiments includes at least one information and analytical platform server 200, which is configured to obtain incident log entries by means of the data transmission network 400 from at least one aircraft apparatus 102 being present in flight and traversing a predetermined patrol route, wherein the server 200 is configured to retrieve the log entries and form digital models of incidents identified on the terrain while at least one aircraft apparatus 102 is traversing a predetermined patrol route. The server 200 may comprise a database of patrol routes, historical data on patrol routes, and a database of incidents. The server 200, based on the reports obtained from the aircraft apparatus 102, may be configured to organize a response to identified incidents even before the completion of the patrol route, form tasks and reports for the customer, emergency repair services, security and law enforcement services, fire services, rescue services and other services. The data obtained from the aircraft apparatus 102 on verified or unverified incidents may be subjected to additional checking (additional verification) by means of one or multiple operator AWSs 300 or other devices and complexes coupled to the information and analytical platform server 200.

[0108] In certain embodiments, the information and analytical platform server 200 is further configured to obtain at least one frame of terrain image data comprising an identified incident for verifying the incident and / or forming a digital model of at least one incident identified on the terrain while at least one aircraft apparatus is traversing a predetermined patrol route.

[0109] The server 200 may also be configured to automatically form or adjust patrol routes based on one or more of the obtained incident log entries, terrain image data, route deviation data upon verification of an identified incident, or based on parameters determined by the operator. Additionally, by means of the information and analytical platform server, there may be organized access to aerial patrol reports, image data, data on digital models of incidents and on patrol routes and other data by means of the data transmission network 400, and there may also be organized interaction between various services and organizations in terms of prevention, identification, elimination and forecasting of possible incidents, accidents and emergencies.Description of the method

[0110] Next,shows a flow chart of the steps of a method 1000 for aerial patrol performed in a system 100 comprising at least one aircraft apparatus 102.

[0111] Step 1002 – determining a patrol route for at least one aircraft apparatus comprising a detection module, a processing module and a data transmission module;

[0112] The method 1000 begins at the step 1002 comprising determining a patrol route for at least one aircraft apparatus 102. The list of possible routes may be selected based on the location of the object or objects being monitored, geographical locations, and is not specifically limited in any manner. In general, the patrol route may include a sequence of geographic coordinate points to be flown over by at least one aircraft apparatus 102.

[0113] The step 1002 of the route determination may further comprise determining incident types to be identified, and during subsequent data processing, the predetermined incident types are identified. At least one or more of the following may be determined as incident types to be identified: smoke, fire, fire effects, landslide, avalanche, pipeline rupture, pipeline exposure, washaway, flooding, icing, leak, gas emission, unauthorized dumping, man-made accident, water body pollution, deforestation, regenerated forest corridors, stationary foreign objects, mobile foreign objects, line break, damage to elements of linear infrastructures, presence of defects in linear infrastructures and coatings.

[0114] Depending on the monitoring object, the location of the object and other factors, there may be formed "white" and / or "black" lists of incident types. The conditional "white" list may include incident types that should not be identified as incidents; these are ignored in data processing and are not included in the log of identified incidents. The conditional "black" list may include types of incidents that should be identified as important for a predetermined patrol route and monitoring object and these are included in the log of identified incidents.

[0115] It is important to note that the same type of incident, depending on the object and patrol route, may be attributed to both the "white" list and the "black" list. So, for example, a means of transport identified in the pipeline's security zone when monitoring a pipeline running in a forest strip may be considered an incident that needs a response, whereas a means of transport identified when monitoring a pipeline running within the boundaries of populated areas or near roads, may be ignored and this type of incident may be included in the "white" list to exclude false identification of incidents. Thus, defining incident types at the determining step allows to further increase the efficiency of aerial patrol, increase data processing speed, and reduce the load on the data transmission network, since only data on selected incident types is directed to the server, whereas the remaining data may be ignored.

[0116] The step 1002 of determining a route may further comprise determining the type of a patrol object, and the step of data processing comprises identifying the types of incidents related to the determined type of a patrol object. In this case, for each type of patrol / monitoring object, types of incidents may be determined that are important and / or, conversely, individual types of incidents may be excluded that are not significant for a specific patrol object and may be ignored. The type of a patrol object includes at least one or more of the following: pipeline, railway line, electric power transmission line, communication line, forest area, settlement area, agricultural land, body of water, mountain range, industrial zone.

[0117] The step 1002 of determining a route may comprise recording at least one set of historical data of terrain images along a predetermined patrol route, and then the processing step further comprises obtaining historical data and identification of incidents on the terrain while comparing, by means of a pre-trained machine learning algorithm, current data obtained from the detection module to historical data of the corresponding terrain section. Using historical data makes it possible to identify not only known incident types but also to reveal any anomalies that may relate to a known incident type or new incident type.

[0118] The step 1002 may further comprise determining the coordinates of control areas, and transmitting data on incident log entries when an aircraft apparatus reaches the control area. Furthermore, suspected incident areas that require additional verification may also be selected as control areas. Thus, the patrol efficiency and speed may be further improved.

[0119] The simplified illustrative example inshows only one aircraft apparatus 102, however, in the context of the subject technology, the amount of aircraft apparatuses 102 and of distinct or coinciding patrol routes is not limited in any manner. The step 1002 may comprise determining a plurality of patrol routes for a plurality of aircraft apparatuses.

[0120] The method 1000 then proceeds to a step 1004.

[0121] Step 1004 – directing at least one aircraft apparatus along a predetermined route and, by means of the detection module, obtaining aerial image data of a terrain with reference to the coordinates and time of image acquisition;

[0122] The step 1004 comprises directing at least one aircraft apparatus 102 along the route determined at step 1002 and performing, by means of the detection module 104, the obtaining of aerial image data (photo / video data) of the terrain with reference to the coordinates and time of image acquisition. Said data may be sufficient for implementing the method further. However, in certain embodiments, the step 1004 may also comprise obtaining additional data, in particular, images may be obtained in the visible and infrared spectrum, and the processing step comprises performing the identification of at least one type of incident based on the obtained image in the visible and / or infrared spectrum.shows an illustrative example of obtaining images in the visible and infrared spectrum by means of the detection module 104.

[0123] In certain embodiments, the step 1004 may comprise, by means of the detection module 104, obtaining additional data by one or more means selected from: a laser scanner, object auto-tracking device, gas analyzer, radiation-chemical reconnaissance device, and the processing step comprises identifying at least one type of incident based on the additional data obtained. Those skilled will appreciate that certain types of incidents may require the presence of appropriate equipment in the detection module 104 for identification thereof.

[0124] The data being obtained or at least a portion of the image data or of other data may be recorded and stored on board the aircraft apparatus 102 for later use, including for forming historical data along a predetermined patrol route.

[0125] The method then proceeds to a step 1006.

[0126] Step 1006 – processing, while in flight, the image data by means of the processing module comprising a pre-trained machine learning algorithm;

[0127] The step 1006 comprises processing the image data while in flight using the processing module 106 comprising a pre-trained machine learning algorithm. In certain embodiments, provided there is present the historical image data of a terrain along a predetermined patrol route, it further comprises obtaining the historical data and identifying incidents on the terrain by way of comparing, by means of a pre-trained machine learning algorithm, the current data obtained from the detection module 104 and the historical data of the corresponding section of the terrain.

[0128] The step 1006 may further comprise mapping the predetermined patrol route in the visible and / or infrared ranges, provided there is respective data, by means of the detection module 104 and the processing module 106 on board the aircraft apparatus.

[0129] The method then proceeds to a step 1008.

[0130] Step 1008 – performing, using a machine learning algorithm, image segmentation, pattern recognition and classification of recognized objects along a predetermined patrol route;

[0131] The step 1008 comprises performing, using a machine learning algorithm, image segmentation, pattern recognition and classification of recognized objects along a predetermined patrol route. Figs. 4A-4B are an illustrative example of semantic segmentation of a terrain image on a patrol route.shows the original image obtained in the step 1004, andon the right shows the processing result in the form of a segmented representation including classification and description of the recognized objects.

[0132] The method then proceeds to a step 1010.

[0133] Step 1010 – identifying at least one type of incident and forming an incident log entry comprising at least data on the type and location of the identified incident;

[0134] The step 1010 comprises identifying at least one type of incident and forming an incident log entry comprising at least data on the type and location of the identified incident.

[0135] shows an illustrative example of incident identification, in particular the presence of equipment and people in the protected area of the patrol object. Further, the incident log entry may be made in text format indicating the coordinates and type of incidents. In certain embodiments, the number of identified incidents, such as the number of vehicles, may also be indicated. An image of the identified incident may be saved individually to be used in further steps.

[0136] The method then proceeds to a step 1012.

[0137] Step 1012 - verifying at least one of the identified incidents by way of obtaining and processing additional data from the detection module and / or obtaining an operator command;

[0138] The step 1012 comprises verifying at least one of the identified incidents by way of obtaining and processing additional data from the detection module 104 and / or obtaining an operator command. Thus, according to method 1000, additional verification may be performed for some or all of the identified incidents. Thanks to processing of data during the flight of the aircraft apparatus 102, the need for verification may be determined in the process of traversing along the patrol route, which allows, if necessary, verifying and thus confirming or denying an identified incident.

[0139] In a number of prior-art methods that use post-processing for verification, it will be necessary to either re-patrol, or verification is carried out by technicians visiting the site of the suspected incident, while in some cases such an unverified incident may not be confirmed.

[0140] In certain embodiments, wherein the step 1002 of determining a route further comprises determining incident types to be identified, the step 1012 may comprise verifying an identified incident corresponding to at least one of the predetermined incident types. Further, other types of incidents may not be verified, whereas the identification thereof by means of the processing module 106 may be reliable. This may apply, for example, to leaks of a heat transfer fluid, which may be clearly identified using infrared images, or to a forest fire. Incidents such as detection of people and other objects during search and rescue operations, exposed pipelines, broken electric power transmission lines, and the like may require additional verification. The examples given should not be construed as limiting and the reasonability of verifying certain types of incidents may be determined individually with respect to the selected monitoring object and the route of aerial patrol thereof.

[0141] Verification of the identified incident in certain embodiments may be carried out by transmitting an incident log entry with image data of the corresponding identified incident by means of the data transmission network 400 to the information and analytical platform server 200 and / or the automated operator workstation 300. Thus, without significantly increasing traffic and load on computation power, there may be performed the remote verification of a suspected incident, thus further increasing the speed and efficiency of aerial patrol, since already at the verification step of an identified incident, if it is confirmed, there may be taken prompt measures to eliminate the incident even before the completion of aerial patrol along a predetermined route. For example, emergency repair teams, security service personnel, and the like may be dispatched.

[0142] In certain embodiments, wherein the operator AWS 300 is disposed directly in the cabin of the aircraft apparatus, the step 1012 comprises outputting the image data of a terrain from the detection module 104 and / or the result of data processing from the processing module 106 to the operator AWS 300 without the need to use the data transmission network 400. In case of direct connection, traffic will not be significantly limited and not only individual image data of the identified incident may be output to the operator. Furthermore, while present on board the aircraft apparatus 102, the operator may further verify individual types of incidents through visual inspection, thereby complementing the work of the machine learning algorithms and the data obtained by the detection module 104.

[0143] The step 1012 of incident verification on the processing module may comprise obtaining user input from the operator AWS 300 or a command from the information and analytical platform server 200 including confirmation or denial of the incident. Thus, a previously identified incident may be included in the incident log or, on the contrary, excluded therefrom as an unconfirmed one.

[0144] The processing of additional image data associated with the identified incident may be performed by the processing module 106 using the same or an additional machine learning algorithm. Those skilled will appreciate that, provided there are additional input image data and / or additional data, the processing result may be clarified with the respective confirmation or denial of the previously identified incident.

[0145] In certain embodiments of the method 1000, the step 1012 of verifying an identified incident may comprise at least partially altering the predetermined patrol route so as to obtain additional image data for additional verification of the identified incident by way of obtaining a command from the processing module 106 and / or from the automated operator workstation 300 and / or from the information and analytical platform server 200. The altering of the patrol route allows for obtaining additional data enabling confirmation or denial of the identified incident, wherein altering the predetermined route may include at least one or more of the following: altering the course, returning to the area of the identified incident, performing additional patrols of the area of the identified incident with an expansion of the patrol radius, descending in the area of the identified incident, landing an aircraft apparatus in the area of the identified incident. Landing an aircraft apparatus may be a last resort, but it is relevant for certain objects and patrol purposes, for example, during a search and rescue operation, where the detection of an incident (search object) may substantially mean the end of the predetermined aerial patrol route.

[0146] Furthermore, the step 1012 of verifying an identified incident in certain embodiments of the method 1000 may comprise altering at least one patrol parameter and obtaining additional image data for additional verification of the identified incident by way of obtaining a command from the processing module 106 and / or from the automated operator workstation 300 and / or from the information and analytical platform server 200. The patrol parameter of the aircraft apparatus 102 includes one or more of: speed of movement, altitude, direction of movement, maneuvering, pitch. Thus, for example, there may be obtained a command from the operator AWS 300 to decrease or increase the flight altitude during patrolling, to alter the flight speed of the aircraft apparatus 102, and the like. In particular, when the number of incidents to be identified is significant, it allows for optimizing the speed of data processing enabling the verification of the incident without the need to subsequently alter the patrol route and, for example, without the need to return to the area of the identified incident.

[0147] In certain embodiments of the method 1000, wherein more than one aircraft apparatus 102 is employed, the step 1012 of verifying an identified incident may comprise transmitting, by means of the data transmission module 108, a request comprising the coordinates and type of the identified incident to at least one additional aircraft apparatus 102 traversing a route that at least partially coincides with the predetermined patrol route. Thus, incident verification may be performed without the need to alter the patrol route or patrol parameters. In response to the request, obtained are additional image data associated with the identified incident from the detection module 104 installed on the additional aircraft apparatus 102, wherein the first aircraft apparatus 102 continues patrolling along the predetermined route without altering the speed and efficiency of identifying and verifying incidents.

[0148] The method then proceeds to a step 1014.

[0149] Step 1014 - by means of the data transmission module, transmitting, while in flight, incident log entries by means of the data transmission network to the information and analytical platform server.

[0150] The step 1014 comprises, by means of the data transmission module 108, transmitting, while in flight, incident log entries by means of the data transmission network 400 to the information and analytical platform server 200.

[0151] In addition to the log entries, by means of the same or an additional data transmission network 400, there may be transmitted at least one image or a sequence of multiple images, in particular a sequence of frames or a fragment of a video recording for a verified identified incident. Said information may be used to obtain additional data on the identified incident and to take appropriate measures.

[0152] In certain embodiments, wherein the step 1002 of determining a patrol route further comprises determining coordinates of the control areas, the transmission of data on incident log entries is carried out by means of the data transmission module 108 when the aircraft apparatus 102 reaches the control area. Control areas may comprise, for example, potential incidents, or said areas may have a stable connection facilitating more efficient and fast data transmission while in flight. The selection of control areas may consider other criteria and characteristics taking into account the characteristics of the object and the predetermined route of aerial patrol.

[0153] After obtaining the incident log entries by means of the data transmission network 400, on at least one information and analytical platform server 200, in certain embodiments, further performed is the processing of obtained data and formed is a report on the identified incidents on the patrol route. The report may be based on all or at least a portion of the identified incidents. In particular, a portion of the identified incidents may not be included in the final report, while, on the contrary, individual incidents may be provided with additional information based on the results of processing the obtained data, for example, including accident prediction, planning of repair work, calling rescue or security services, and the like. Thus, the information and analytical platform server 200 may further include a decision support system that enables rapid response, planning of routine maintenance and recording of violations at aerial patrol objects. It is worth noting that said steps in the subject method may be carried out as and when the aircraft apparatus 102 moves along a predetermined patrol route, that is, while still in flight but not after completing the patrol along a route that may be hundreds of kilometers long and take hours and, in some cases, tens of hours. This may be particularly important for a number of specific types of incidents that require immediate response.

[0154] Certain embodiments comprise further forming, on at least one information and analytical platform server 200, a digital model of incidents identified on the terrain while at least one aircraft apparatus 102 is traversing a predetermined patrol route. Forming a digital model may provide additional information, as well as improve the accuracy and efficiency of patrols and subsequent incident response measures by way of analyzing multiple factors, either through automated algorithms or by one or more operators. Further, said digital model may be fully processed by means of a decision support algorithm so as to provide for automatic generation (determination) of a scenario for responding to incidents identified on the terrain while at least one aircraft apparatus 102 is traversing a predetermined patrol route.

[0155] On at least one information and analytical platform server 200, in certain embodiments, performed are obtaining at least one image or a sequence of multiple images for a verified identified incident by means of a data transmission network, processing the obtained data and performing additional training of the machine learning algorithm based on the verified identified incident. After additional training based on a plurality of data from distinct aircraft apparatuses 102 traversing the same or at least partially different aerial patrol routes, the algorithm being used by the processing module 106 of the aircraft apparatus 102 may be updated.

[0156] Next, the method 1000 is multiply repeated as and when the aircraft apparatus 102 moves along a predetermined patrol route and is then terminated. After completing the patrol route, the obtained image data may be transferred to the server 200 for additional data processing, forming historical data, and forecasting; further, it is important to note that the main report on the identified incidents was obtained while in flight before the aircraft apparatus 102 completed the predetermined aerial patrol route and, therefore, the necessary response measures could be taken promptly, so the post-processing in this method 1000 is only to supplement the results obtained.

[0157] The subject system and method for aerial patrol provide for prompt data collection and transmission, data analysis is performed while in flight on board the aircraft apparatus so as to provide for the transmission, over the data transmission network, of only necessary and sufficient information while providing for reduced traffic transmitted and a decreased load on the network, thus allowing interested users to promptly obtain information necessary for making informed decisions.

[0158] Those skilled in the art may appreciate modifications and improvements to the above-described embodiments of the subject system and method for aerial patrol. The preceding description is intended to be illustrative rather than limiting. Accordingly, the scope of the present technology is limited only by the scope of the appended claims.

Claims

A system for aerial patrol, including at least one aircraft apparatus configured to traverse a predetermined patrol route, said aircraft apparatus comprising:- a detection module configured to obtain aerial image data of a terrain with reference to the coordinates and time of image acquisition;- a processing module configured to obtain image data of a terrain from the detection module, process same while in flight and identify incidents on the terrain using at least one pre-trained machine learning algorithm so as to form an incident log including at least the type and location of the incident on the patrol route, wherein the processing module is configured to further verify an incident, which has been identified by means of the pre-trained machine learning algorithm, while in flight, by way of obtaining and processing additional data from the detection module and / or by way of obtaining user input from the operator;- a data transmission module configured to transmit, while in flight, incident log entries by means of a data transmission network to at least one information and analytical platform server;- the system further comprises at least one database comprising at least one set of historical image data of a terrain along a predetermined patrol route, the processing module is further configured to obtain the historical data and identify incidents on the terrain when comparing, by means of a pre-trained machine learning algorithm, between current data obtained from the detection module and historical data.The system according to claim 1, wherein the detection module is configured to obtain images in the visible and infrared spectrum, and the processing module is configured to identify incidents based on terrain image data in the visible and infrared ranges.The system according to claim 1, wherein the detection module further comprises one or more of: a laser scanner, object auto-tracking device, gas analyzer, radiation-chemical reconnaissance device, and the processing module is configured to obtain additional data from the detection module and use same to identify incidents using a pre-trained machine learning algorithm.The system according to claim 1, which comprises a data storage module associated with the detection module and the processing module and configured to at least partially store data obtained by the detection module and / or data processed by the processing module.The system according to claim 1, which further comprises an automated operator workstation configured to communicate with the processing module and verify the incident by way of outputting to the operator the terrain image data from the detection module and / or the result of data processing and of incident identification, and to obtain user input from the operator.The system according to claim 5, wherein the aircraft apparatus is a manned or unmanned aircraft apparatus, and the automated operator workstation is disposed in the aircraft apparatus.The system according to claim 5, wherein the aircraft apparatus is a manned or unmanned aircraft apparatus, and the automated operator workstation is configured to remotely communicate with the processing module by means of a data transmission network.The system according to claim 1, wherein the processing module, by means of the data transmission module, is further configured to transmit to the information and analytical platform server at least one frame of terrain image data, which contains the identified incident, for verification of the incident.The system according to claim 1, wherein the aircraft apparatus is configured to at least partially alter the predetermined patrol route for additional verification of the incident by means of the obtaining of a command from the processing module and / or from the automated operator workstation and / or from the information and analytical platform server.The system according to claim 1, wherein the aircraft apparatus is configured to alter at least one patrol parameter for additional verification of the incident by way of obtaining a command from the processing module and / or from the automated operator workstation and / or from the information and analytical platform server.The system according to claim 10, wherein the patrol parameter of an aircraft apparatus includes one or more of: speed of movement, altitude, direction of movement, maneuvering, pitch.The system according to claim 1, comprising at least one additional aircraft apparatus configured to traverse a route at least partially coinciding with a predetermined patrol route, wherein the processing module of the first aircraft apparatus is configured to further verify the incident by means of the obtaining, by means of the data transmission network, of additional data from the detection module installed on the additional aircraft apparatus.The system according to claim 1, comprising at least one information and analytical platform server, which is configured to obtain incident log entries by means of the data transmission network from at least one aircraft apparatus being present in flight and traversing a predetermined patrol route, wherein the server is configured to retrieve the log entries and form digital models of incidents identified on the terrain while at least one aircraft apparatus is traversing a predetermined patrol route.The system according to claim 13, wherein the information and analytical platform server is further configured to obtain at least one frame of terrain image data comprising an identified incident for verifying the incident and / or forming a digital model of at least one incident identified on the terrain while at least one aircraft apparatus is traversing a predetermined patrol route.A method for air patrol comprising the steps of:- determining a patrol route for at least one aircraft apparatus comprising a detection module, a processing module and a data transmission module, at the step of determining a route, recording at least one set of historical terrain image data along a predetermined patrol route;- directing at least one aircraft apparatus along a predetermined route and, by means of the detection module, obtaining aerial image data of a terrain with reference to the coordinates and time of image acquisition;- processing, while in flight, image data using the processing module comprising a pre-trained machine learning algorithm;- at the step of processing, further obtaining historical data and identifying incidents on the terrain by comparing, using a pre-trained machine learning algorithm, current data obtained from the detection module and historical data of the corresponding area of the terrain;- performing, using a machine learning algorithm, image segmentation, pattern recognition and classification of recognized objects along a predetermined patrol route;- identifying at least one type of incident and forming an incident log entry comprising at least data on the type and location of the identified incident;- verifying at least one of the identified incidents by way of obtaining and processing additional data from the detection module and / or obtaining an operator command;- by means of the data transmission module, transmitting, while in flight, incident log entries by means of the data transmission network to the information and analytical platform server.The method according to claim 15, wherein the step of determining the route further comprises determining incident types to be identified, and the step of data processing comprises identifying the predetermined incident types.The method according to claim 15, wherein the method comprises verifying an identified incident corresponding to at least one of the predetermined types of incidents.The method according to claim 15, wherein the step of determining a route further comprises determining the type of patrol object, and the step of data processing comprises identifying the types of incidents related to the predetermined type of patrol object.The method according to claim 15, wherein the types of incidents to be identified include at least one or more of the following: smoke, fire, fire effects, landslide, avalanche, pipeline rupture, pipeline exposure, washaway, flooding, icing, leak, gas emission, unauthorized dumping, man-made accident, water body pollution, deforestation, regenerated forest corridors, stationary foreign objects, mobile foreign objects, line break, damage to elements of linear infrastructures, presence of defects in linear infrastructures and coatings.The method according to claim 18, wherein the type of patrol object includes at least one or more of the following: pipeline, railway line, electric power transmission line, communication line, forest area, settlement area, agricultural land, body of water, mountain range, industrial zone.The method according to claim 15, wherein the step of obtaining aerial image data of a terrain with reference to the coordinates and time of image acquisition comprises obtaining images in the visible and infrared spectrum, and the step of processing comprises identifying at least one type of incident based on the obtained image in the visible and / or infrared spectrum.The method according to claim 15, wherein the method comprises obtaining, by means of the detection module, additional data by means of one or more means selected from: a laser scanner, object auto-tracking device, gas analyzer, radiation-chemical reconnaissance device, and the processing step comprises identifying at least one type of incident based on the additional data obtained.The method according to claim 15, wherein the method comprises recording at least a portion of the image data which is obtained by the detection module.The method according to claim 23, wherein the method comprises mapping a predetermined patrol route in the visible and / or infrared ranges by means of the detection module and the processing module on board the aircraft apparatus.The method according to claim 23, wherein the step of verifying an identified incident comprises transmitting an incident log entry together with image data of the corresponding identified incident by means of the data transmission network to the information and analytical platform server and / or the automated operator workstation.The method according to claim 15, wherein the step of verifying an incident comprises outputting the image data of a terrain from the detection module and / or the result of data processing from the processing module to the automated operator workstation disposed in the aircraft apparatus.The method according to any one of claims 25-26, wherein the step of verifying an incident comprises obtaining, from the information and analytical platform server, user input or a command including confirmation or denial of the incident.The method according to claim 15, wherein the step of verifying an identified incident comprises at least partially altering the predetermined patrol route and obtaining additional image data for additional verification of the identified incident by way of obtaining a command from the processing module and / or from the automated operator workstation and / or from the information and analytical platform server.The method according to claim 28, wherein altering the predetermined patrol route comprises at least one or more of the following: altering the course, returning to the area of the identified incident, performing additional patrols of the area of the identified incident with an expansion of the patrol radius, descending in the area of the identified incident, landing an aircraft apparatus in the area of the identified incident.The method according to claim 15, wherein the step of verifying an identified incident comprises altering at least one patrol parameter and obtaining additional image data for additional verification of the identified incident by way of obtaining a command from the processing module and / or from the automated operator workstation and / or from the information and analytical platform server.The method according to claim 30, wherein the patrol parameter of an aircraft apparatus includes one or more of: speed of movement, altitude, direction of movement, maneuvering, pitch.The method according to claim 15, wherein the step of verifying an identified incident comprises transmitting a request by means of the data transmission module, the request comprising the coordinates and type of the identified incident, to at least one additional aircraft apparatus traversing a route that at least partially coincides with the predetermined patrol route, and obtaining, in response to the request, additional image data associated with the identified incident from the detection module installed on an additional aircraft apparatus.The method according to any one of claims 28-32, wherein the method comprises processing additional image data associated with the identified incident by means of the processing module, and confirming or denying the identified incident.The method according to claim 33, wherein the method comprises forming, in response to the confirmation of the identified incident, an incident log entry comprising at least data on the type and location of the verified identified incident.The method according to claim 33, wherein the method comprises, in response to the denial of an identified incident, deleting an entry from the incident log or adding an entry about a falsely identified incident.The method according to claim 15, wherein the method further comprises transmitting, at the step of transmitting log entries while in flight, at least one image or a sequence of multiple images for the verified identified incident.The method according to claim 15, wherein the method further comprises determining, at the step of determining a patrol route, the coordinates of the control areas, and transmitting data on incident log entries when the aircraft apparatus reaches the control area.The method according to claim 15, wherein the method comprises obtaining the incident log entries by means of the data transmission network on at least one information and analytical platform server and further processing the obtained data and forming a report on the identified incidents along the patrol route.The method according to claim 15, wherein the method comprises obtaining incident log entries by means of a data transmission network on at least one information and analytical platform server and forming a digital model of incidents identified on the terrain while at least one aircraft apparatus is traversing a predetermined patrol route.The method according to claim 39, wherein the method further comprises processing said digital model using a decision support algorithm and automatically generating a scenario for responding to incidents identified on the terrain while at least one aircraft apparatus is traversing a predetermined patrol route.The method according to claim 36, wherein the method comprises obtaining at least one image or a sequence of multiple images for a verified identified incident by means of a data transmission network on at least one information and analytical platform server, processing the obtained data and performing additional training of the machine learning algorithm based on the verified identified incident.

Images